This is the question we get asked most often about Azure OpenAI, usually after the first month's invoice: should we move to provisioned throughput? The honest answer is that it depends on one number almost nobody has measured, and that most teams who switch early end up paying more for worse flexibility. Here is how to work it out for your own workload.
The two billing models, stated plainly
Pay-as-you-go bills per token, input and output priced separately, with no floor and no ceiling. You pay for exactly what you consume, and you share capacity with everyone else in the region — which means throughput is best-effort and latency varies with someone else's traffic.
Provisioned throughput reserves capacity in units. You pay by the hour for the units you hold, whether or not a single token flows through them. In exchange you get predictable latency, a throughput guarantee, and a bill that does not move when a feature goes viral.
The trap is that these are not two prices for the same thing. Pay-as-you-go is a variable cost, provisioned throughput is a fixed one, and the comparison is only meaningful once you know your utilisation.
The one number that decides it: sustained utilisation
Provisioned capacity only wins when you keep it busy. The break-even point is the utilisation at which the hourly cost of the units you hold equals what the same traffic would have cost per token. Below that line pay-as-you-go is cheaper, above it provisioned wins, and the line moves with the model, the region and whatever reservation term you commit to.
So measure before you model. Pull per-minute token throughput for 30 days from your own telemetry, not a daily average — a daily average hides the fact that most internal tools do 80% of their work in a four-hour window and sit idle overnight. What you need is the shape of the curve, because provisioned capacity is sized against your peak and billed against your idle.
- Peak tokens per minute, input and output separately, at the 95th percentile rather than the maximum
- The ratio of that peak to your median — anything above roughly four to one makes provisioned capacity expensive
- How many hours a day traffic is genuinely near peak
- Whether traffic is human-driven (spiky, daytime) or batch (schedulable, and therefore flattenable)
When provisioned throughput is clearly right
- Latency is in the product. Anything interactive and customer-facing, where a slow response is a defect rather than an inconvenience, benefits from capacity nobody else can crowd out.
- Traffic is flat and high. Document pipelines, classification at scale, anything batch that you can schedule to run around the clock rather than in bursts.
- You need a predictable number for a budget. Sometimes a fixed cost that is slightly higher is worth more than a variable one that is slightly lower, particularly when finance has to sign off a year ahead.
- You are hitting rate limits on shared capacity. If pay-as-you-go throttling is already shaping your architecture, you are paying for it in engineering time rather than tokens.
When it is clearly wrong
If you are still changing models every few weeks, do not reserve capacity. Provisioned units are tied to a model and a version, and the reason a team switches models is almost always that the newer one is better or cheaper. Committing to capacity in the middle of that is buying certainty you do not want yet.
If your traffic is spiky and human-driven, the peak you must provision for and the average you actually use are far apart, and you pay for the gap every hour of every night. And if your monthly spend is still in the low thousands, the engineering time spent modelling this is worth more than the saving. Come back when the invoice justifies the meeting.
Three mistakes that make provisioned capacity cost more
- 01Sizing against the absolute peak. One unusual Tuesday should not set your capacity floor for a year. Size against the 95th percentile and let the rest spill over to pay-as-you-go, which is exactly what the spillover configuration exists for.
- 02Reserving before optimising the workload. Caching duplicate prompts, trimming context windows and routing simple requests to a smaller model routinely cut token volume by a third. Do that first — otherwise you reserve capacity for waste and lock the waste in for the term.
- 03Forgetting that idle capacity is still billed. Non-production environments pointed at provisioned endpoints, or units left in place after a feature was retired, are the GenAI equivalent of a VM nobody turned off. We find these on almost every AI-heavy estate we review.
The hybrid answer most mature teams land on
The pattern that survives contact with reality is neither one nor the other. Provision for the steady-state floor, let bursts spill over to pay-as-you-go, keep non-production entirely on pay-as-you-go, and route by workload rather than by team: interactive traffic to provisioned capacity, batch and background work to whatever is cheapest at the time.
That requires a routing layer, per-workload budgets and enough observability to tell which requests went where. It is a few days of platform work, and it is the difference between a bill you can explain and one you can only apologise for.
How to run the numbers this week
- 01Export 30 days of token consumption at per-minute granularity, split by input and output.
- 02Compute the 95th-percentile throughput and the median. Note the ratio.
- 03Price the units needed for that percentile at current published rates, including any reservation discount you would realistically commit to.
- 04Price the same 30 days of traffic at pay-as-you-go rates.
- 05Compare — then re-run step 4 after applying caching and model routing, because that is the version of the number you should actually decide on.
Published rates change often enough that any figure written into an article is wrong by the time you read it, so take the method rather than the numbers and check current pricing when you run it.
The decision is not really pay-as-you-go versus provisioned throughput; it is whether you have measured your own traffic well enough to have an opinion. Teams that measure first usually find the answer is 'partly, later, and less than we thought' — and that the optimisation work they do to find out pays for itself regardless of which way they go.