Local development

Windows: .NET SDK 10.0.401, Aspire 13.6, Docker 28+, Node 24.15+ recommended. Use PowerShell 5.1 and run commands separately (or chain with ; and exit-code checks).

Set-Location src\admin-ui
npm ci
Set-Location ..\..
.\scripts\Start-Dev.ps1          # with demo data (default)
.\scripts\Start-Dev.ps1 -Empty   # empty database, only the Keycloak login users

The script wraps aspire start + aspire wait admin-ui; you can still run those directly (demo data is then on unless Seed__Enabled=false is set in the environment).

#Demo data or an empty environment

Seeding (Seed:Enabled) runs in the migration service and only when the database has no departments. Demo data is fictional förvaltningar, teams, providers, models, prices, routes and a dev key and three example routing rules (see Routing rules). In the empty variant you sign in with a Keycloak test user (gateway-admin) and create everything yourself. To switch an existing local environment, stop the stack, delete the AppHost's Postgres Docker volume (docker volume ls, then docker volume rm <name>; or the volume named in DevelopmentVolumes:Postgres) and start again.

#Fake LLM toggle

The fake provider (fake-llm) starts with the AppHost by default. Set "FakeLlm:Enabled": false in the AppHost user secrets (or FakeLlm__Enabled=false in the environment) to skip it. The admin UI badge shows Development or Production from the admin API's environment.

By default the fake LLM answers instantly. To make chat, responses and messages answers behave like a real provider, set FakeLlm:Latency in the AppHost user secrets (or FakeLlm__Latency__* on the fake LLM itself). All values default to 0 (off):

With stream_options.include_usage, the fake LLM's chat chunks carry "usage":null as OpenAI's do.

#Running from Visual Studio

1. Install Visual Studio 2022 17.14+ (or newer) with the .NET 10 SDK, Docker Desktop (running), Node 24+, and run dotnet dev-certs https --trust once.
2. Open UmeLlmGateway.slnx, set Ume.LlmGateway.AppHost as startup project, profile https.
3. Right-click the AppHost project > Manage User Secrets and paste the dev secrets JSON (Parameters:* for pepper, dev-key, OIDC secret, dev user password, Redis and Postgres passwords, plus DevelopmentVolumes:Postgres). Keep them stable: the Postgres volume stores the password it was created with, and the key pepper hashes all virtual keys, so changing either later breaks the existing data. With these set, data survives restarts and git pull (migrations run automatically). Aspire generates the passwords itself if you skip this step.
4. "Seed:Enabled": false = empty environment; true or omitted = demo data. It only takes effect on an empty database (delete the Postgres Docker volume to switch).
5. Press F5. The Aspire dashboard opens; click the admin-ui endpoint. Keycloak test users are created from dev/keycloak; the shared dev password is the dev-user-password secret.

#Environments (dev / test / production)

There is no separate "test" configuration; the mode decides the environment:

Production inputs (OIDC authority/client, certificates, secrets, provider endpoints) are generated or prompted for by deploy/init-deployment.sh (see Quick production setup below) and described in the runbook. Other settings can be overridden with Section__Key environment variables (for example Ollama__Enabled=true, development only).

#Admin UI code

src/admin-ui is Vue 3 + Pinia + Vue Router with Reka UI primitives and Tailwind CSS v4, built from the design prototype in Design prototype/ (tokens in src/admin-ui/src/styles/tokens.css, Tailwind config in tailwind.config.cjs). The UI is English only, with light, dark and Lumen themes (data-theme, remembered in localStorage). Reusable building blocks live in src/components/ui (buttons, tables, drawers, dialogs, form fields, meters), the page shell in src/components/layout, and pages in src/views. Icons are a subset of Material Symbols; after adding an icon name run python src/admin-ui/scripts/subset-icons.py (it needs the Python package fonttools[woff]).

#Build and test

The UI is https://localhost:5173. Local Keycloak test users/roles are provisioned by the realm import; get development passwords through Aspire's secret store without printing them, enabling transcripts or saving browser auth state. Ollama is opt-in: set Ollama__Enabled=true before starting. FakeLlm is the default test provider. Stop this exact AppHost before backend/AppHost edits or builds:

aspire stop --apphost src\Ume.LlmGateway.AppHost\Ume.LlmGateway.AppHost.csproj --non-interactive
dotnet build UmeLlmGateway.slnx -v q -nologo
dotnet test --solution UmeLlmGateway.slnx
Set-Location src\admin-ui
npm run lint
npm run typecheck
npm test

npm run lint fails on warnings, exactly like CI (npm run lint:fix fixes most). To catch it before pushing, enable the pre-commit hook once per clone: git config core.hooksPath .githooks. Debug builds of the admin API (Visual Studio, F5) also run the lint and show problems as a build warning (skipped when node_modules is missing or CI=true).

#Tests that do not flake

Each backend test project shares one fixture (Postgres, the gateway or admin API host, the mock provider) across all its test classes, and xUnit runs the classes in parallel. A test that passes alone can therefore fail in CI, where two CPUs make every race wider. The rules that keep them stable:

• Make your own data. Create the key, team or route the test needs, and find your requests by a unique marker (a Guid in the prompt), never by "the latest" or "the only" row. Do not assert on global state another class may change (settings, global routing rules); if a test must change it, restore it in finally.
• Wait for asynchronous effects, do not sleep. The mock provider logs a request only after answering it: read it with fixture.UpstreamRequestsAsync / UpstreamBodyAsync, never Upstream.LogEntries. Usage records: fixture.UsageAsync (flushes the writer). Changes that reach the gateway over Redis (revocation, rotation, budgets): retry the request until it has the expected status, as WaitForStatusAsync does.
• Stay inside one time window. Rate limits count per calendar minute and budget periods start at midnight in Stockholm. Call TestClock.InOneRateLimitMinuteAsync() before a test that needs several requests in the same minute (the fixtures already keep every test away from midnight). Use a FakeTimeProvider where the code takes one.

Verify POC (.github/workflows/verify.yml) runs on every pull request (on its merge with main), on pushes to main and on release tags; a newer push to a pull request cancels the run in progress. A branch without a pull request is not verified automatically: open a (draft) pull request, or run Verify POC from the Actions tab on the branch.

To hunt for flakes, run the workflow Flake hunt (Actions tab): it runs the backend tests on a CI runner several times and lists every test that failed. It also runs once a week.

Known flakes stay out of the pull request pipeline. A test that has failed intermittently because of timing or a race is marked [Trait(TestCategory.Name, TestCategory.Flaky)] (tests/Ume.LlmGateway.TestKit/TestCategory.cs) with a comment saying why. Verify POC runs dotnet test ... --filter-not-trait "Category=Flaky"; locally dotnet test runs every test, and Flake hunt runs the marked ones too, so they are still watched. Run only the pipeline's set with dotnet test --solution UmeLlmGateway.slnx --filter-not-trait "Category=Flaky", or only the flaky ones with --filter-trait "Category=Flaky". The trait is a to-do, not a home: fix the test and remove it. Marked now:

#Browser tests

Browser tests (tests\e2e, Playwright) need a running stack. Use the separate test database so your development data is never touched:

.\scripts\Start-E2E.ps1          # own Docker volume (ume-e2e-postgres), reset on every start, demo data, fake LLM on
Set-Location tests\e2e
npm ci
npx playwright install chromium
npx tsc --noEmit
npx playwright test
Set-Location ..\..
.\scripts\Start-E2E.ps1 -Stop    # stops the stack and removes the test database

The test stack uses the same ports as the dev stack, so stop the dev stack first. The tests cover sign-in, keys (show-once secret, rotation, revocation), organisation, usage, audit, budgets, the demo scenario, routing rules (create, validate, test, reorder, disable, delete, team and department deletion and reassignment, key rotation, budget_used/tokens_used, with real gateway requests) and accessibility (axe at WCAG 2.2 AA in the three themes, drawers and dialogs, reflow, keyboard, forced colours, target size). They do not save screenshots, videos, traces or storage state. No manual screen reader testing is planned; open follow-ups are listed in HANDOVER.md.

To regenerate the routing screenshots in docs/images after a UI change, run npx playwright test -c playwright.docs.config.ts screenshots in tests\e2e against the test stack. The script builds the two routing examples through the UI, so it changes the (disposable) test database only.

#Website

site/ builds the public website (https://thekim1.github.io/llmgateway/): the product page, Getting started, Benchmarks and every documentation page listed in site/content.mjs, rendered from the Markdown files in docs/. The workflow .github/workflows/pages.yml builds it on every pull request that touches the site or the docs, and publishes it to GitHub Pages on main. To add a page, write the Markdown file in docs/ and add it to DOCS in site/content.mjs.

Set-Location site
npm ci
npm run serve        # builds into site/dist and serves http://localhost:4321/

The build fails on a link to a page, file or heading that does not exist. Links between Markdown files and to docs/images and docs/reports become site links; links to other repository files go to GitHub.

The admin UI pictures on the website (site/public/screenshots, one per theme) come from the real UI. To regenerate them, start the test stack and run npx playwright test -c playwright.docs.config.ts website in tests\e2e. Only demo data appears; the secret in the key dialog is masked and the keys the script creates are revoked.