Enterprise API rate limits: 4 algorithms and the quota traps

TL;DR: Enterprise API rate limits protect shared capacity and control cost. Use a rate limit for short bursts and a quota for the month. Pick token bucket for bursty clients, leaky bucket for steady backends. Keep counters shared across gateway nodes. For AI calls, limit tokens per minute as well as requests.

A client sends too many requests. Your API says no. That's a rate limit.

Simple enough. But at enterprise scale it gets tricky fast. You share capacity across many tenants. The bill grows with use. And AI calls now cost very different amounts.

This post covers the four algorithms, why counters go wrong across servers, and how to limit LLM calls by tokens, not just requests.

Rate limits and quotas are not the same thing

A rate limit caps requests in a short window. Seconds or minutes. It stops one client from hogging shared capacity during a spike.

A quota caps use over a longer period. A day or a month. It usually ties to a plan or a bill. Think "10,000 calls a month".

Enterprises run both. One protects uptime right now. The other protects revenue and fairness over time. Azure API Management treats them as separate policies for rate limits and quotas, so you can tune each on its own.

When a client goes over, the standard reply is a 429 status. The 429 code is defined in RFC 6585, and the server may add a Retry-After header to say when to come back. The header itself is defined in RFC 9110.

The four algorithms

Each one handles bursts differently.

1. Fixed window. Count requests in set blocks of time, like each minute. Reset at the boundary. Simple, but a client can fire a burst at the end of one window and again at the start of the next.
2. Sliding window. Count across overlapping periods. Fairer at the edges. Costs a little more to run.
3. Token bucket. A client builds up a reserve of tokens and spends them in bursts. The bucket refills slowly. Good for traffic that comes in spikes.
4. Leaky bucket. Requests leave at a steady rate, however they arrive. Burst tolerance is low, but the load on your backend stays flat.

Azure shows both in practice. Its v2 tiers use a token bucket. Its classic tiers use a sliding window. Microsoft's docs call out that difference.

Here's the rule of thumb. Pick token bucket for APIs where users burst. Pick leaky bucket when the thing behind the API needs a flat, steady load.

Why separate servers break your limit

Here's the trap. You test with one server. The limit works. Then you add a second gateway node for uptime.

Now a client can hit both. If each node counts on its own, that client gets two allowances, not one.

Google's Apigee docs name this as an antipattern. They describe a non-distributed quota where each message processor keeps its own counter, so the quota is likely to be exceeded. The fix is in the Apigee Quota policy: set it up to use a shared, distributed counter.

AWS has a related catch. Its usage plans say throttling and quotas are not hard limits and are applied on a best-effort basis. So don't lean on them alone to cap spend.

What to do about it:

• Keep counters in shared state, like Redis, not in each server's memory.
• Enforce limits at the gateway, so a policy change doesn't mean redeploying every service.
• Scope each limit to the right thing: API key, user, IP address, or agent identity.
• Pair plan-level quotas with per-user limits, so one user can't drain a whole company account.

What clients should do on a 429

Read Retry-After first. Treat it as an instruction.

If there isn't one, wait and try again with exponential backoff and jitter. Jitter means a small random delay, so a thousand clients don't all retry at the same second.

Wrap calls in a circuit breaker too. It stops a wave of retries from hitting a struggling backend. For more on retry and proxy tactics, this guide to rate limiting and large-scale web automation goes deeper.

Limiting LLM calls by tokens

Request counts miss the real cost of an AI call. One request might use a few tokens. Another might use tens of thousands.

That's why LLM providers limit both. Anthropic's rate limit docs list requests per minute, input tokens per minute and output tokens per minute. A 429 comes back when you pass any of them, with a retry-after header. OpenAI works the same way, and this walkthrough of OpenAI rate limits shows how to check your limits and ask for more.

Your own platform needs the same two layers. Here's what works:

• Set requests-per-minute and tokens-per-minute limits per agent, per app and per team. Not one blanket cap.
• Add a budget cap that stops a workflow once spend crosses a line, even if no rate limit was hit.
• Limit across several windows at once: minute, day and month. A quiet morning shouldn't pay for a wild afternoon.
• Reserve tokens before you send the request, not after. That avoids surprise throttling from the provider.
• Fall back to another model or provider when a limit is hit, through the gateway.

Agents raise the stakes. One agent task can fire dozens of model calls with no human in the loop. A tokens-per-minute ceiling is what stands between a bug and a big bill.

Track token spend per tenant, 429 rates by endpoint, and quota use over time. Our post on API usage analytics covers the dashboard side. For the money side, see LLM API pricing compared and FinOps for AI. And if repeat prompts are eating your token budget, LLM caching cuts the calls before they hit a limit. The governance side is covered in AI agent governance, and this LLM integration guide covers wiring a model into your stack.

A rollout checklist

Don't switch limits on all at once. Go in order.

1. Pick the goal. Are you protecting capacity, enforcing a plan, or capping AI spend? Each needs a different control.
2. Map your endpoints. A search call and a write call don't carry the same risk.
3. Choose scoping keys. API key, user or agent. Line them up with plan tiers.
4. Pick the algorithm and where it runs. Gateway, sidecar or upstream. Make counters shared wherever more than one node sees traffic.
5. Test before launch. Simulate bursts. Write tests that check the 429 status and the Retry-After header. Run them against the same multi-node setup you'll use in production.
6. Set up monitoring. Dashboards for 429 rates and quota use. Alerts before limits hit real traffic. A runbook for raising a limit in a hurry.

Publish your limits too. Clear thresholds, headers and error messages cut support tickets.

Where teams go wrong

Two mistakes come up again and again.

The first is trusting one node's counter to speak for the whole system. Add a second node and the allowance doubles without anyone noticing.

The second is treating token budgets as optional. They're not. Once agents act without a person approving each call, the token cap is your safety net.

Need a hand?

Devwiz builds the control layer into AI platforms. That covers shared counters, token budgets for LLM calls, and the monitoring that keeps incidents short. See our AI programs work, or look at how we approach AI platform architecture.

Scoping a rate limiting or AI governance rebuild? Let's talk and we'll map it out.

Frequently asked questions

How do you rate limit API requests?

Cap requests in a time window using token bucket or sliding window. When a client goes over, return a 429 status with a Retry-After header. Enterprises enforce this at the gateway, with shared counters, not per-server memory.

What is the difference between a rate limit and a quota?

A rate limit caps requests over a short window, like a minute, to protect capacity. A quota caps use over a longer period, like a month, and usually ties to a plan or bill. Most enterprise APIs use both.

Does an API gateway support rate limiting?

Yes. Gateways offer it as policy. AWS API Gateway uses usage plans. Azure API Management has rate-limit-by-key and quota-by-key policies. Apigee has a Quota policy. Check each vendor's docs, as some apply limits on a best-effort basis.

Why limit LLM calls by tokens as well as requests?

One request can use a handful of tokens or tens of thousands. Providers such as Anthropic limit requests, input tokens and output tokens per minute. Your own platform needs the same two layers, plus a budget cap.

How should clients handle a 429 response?

Read the Retry-After header and wait. If there isn't one, use exponential backoff with jitter. Add a circuit breaker so retries don't pile onto a struggling backend.