Close to zero added latency.
The gateway sits on the path of every LLM call, so it's built to add as little as possible. These runs use a fake LLM that answers instantly, so only the gateway's own work is measured. Every gateway figure on this page comes from one run (27 Oct 2026); the fake LLM, Postgres and Redis had cores of their own and stayed well below their limits.
Throughput as load grows
Streaming chat, messages per second, 4 vCPUs unless notedWhat one request costs
With warm caches, a request makes no Postgres calls on the request path. Keys and the routing catalogue live in memory; usage is written in batches in the background.
| Step | When | Redis round trips | How |
|---|---|---|---|
| Key and catalogue lookup | Before first byte | 0 | In memory; refreshed in the background |
| Rate limit + circuit state | Before first byte | 1 | One Lua script and one MGET, pipelined |
| Budget reservation | Before first byte | 1 | One script checks key, team and department budgets |
| Budget reconciliation | After response | 1 | Pipelined, off the client's critical path |
| Usage record | Background | 0 | Batched, about ten records per Postgres COPY statement |
Next to eneo, on the same fake LLM
Same four pinned cores for each platform. Both are called the way an external application calls them: the gateway with a virtual key, eneo with a service API key. These are different kinds of product: eneo is a chat application that stores every conversation; the gateway is a stateless proxy. The numbers show what each adds on top of the model call, not which is better.
What to run it on
For a quiet installation, up to about 50 messages per second with room for bursts. Measured with the gateway limited to 0.5 vCPU and 512 MB, PII detection and the content guard on, a fake LLM with a real provider's timing (about 5.6 s per streamed answer) and mixed traffic: 55 000 messages, 0 failed.
| Recommended | CPU | Memory | Disk |
|---|---|---|---|
| Gateway | 0.5 vCPU, up to 1 | 256 MB, up to 512 MB | – |
| Admin API and UI | 0.25 vCPU | 256 MB, up to 512 MB | – |
| Postgres | 0.5 vCPU, up to 1 | 512 MB, up to 1–2 GB | Usage data |
| Redis | 0.1 vCPU | 64 MB, up to 256 MB | None |
| One host for everything | 2 vCPU | 4 GB | 20 GB + usage data |
| Gateway and admin API only | 1–1.5 vCPU | 1.5 GB | 5 GB |
| Usage data in Postgres, 365 days kept | Per year |
|---|---|
| 5 msg/s average | ≈ 90 GB |
| 10 msg/s average | ≈ 180 GB |
| 50 msg/s around the clock | ≈ 890 GB |
About 560 bytes per request, kept for the retention period (90–3 650 days). Size disk by the average rate over the day, not the peak. With your own Postgres and Redis, 50 messages/s is about 50 batched inserts and 450 Redis commands per second. The admin API was not measured under load. Sizing in Production setup
Full results
The platform comparison is also a PDF.
Run it yourself
Absolute numbers depend on the machine; compare runs on the same host. Postgres and Redis ran in Docker on WSL2 here, so Redis round trips cost more than on a production network.
How to measure# All BenchmarkDotNet benchmarks (~10 min)
dotnet run -c Release --project benchmarks\Ume.LlmGateway.Benchmarks -- --filter *
# Postgres/Redis calls per request + load test
dotnet run -c Release --project benchmarks\Ume.LlmGateway.Benchmarks -- audit
# Throughput and scaling against a fake LLM, pinned cores
cd benchmarks/compare && PLATFORMS="fake ume" ./run.sh && PLATFORMS="fake ume" ./scale.sh