API Costs & Rate Limits
Gen AI · Ethics, Cost & Careers
Theory
Most hosted LLM APIs charge by tokens processed, with input tokens (what you send) and output tokens (what the model writes) usually priced separately, and output tokens typically costing more each. Providers also set rate limits: caps on requests or tokens per minute or per day, to share capacity fairly. Hitting a limit returns an error, so apps must retry with a short, growing wait (exponential backoff).
Cost is easy to estimate: tokens per call times calls per day. The main levers are shorter prompts, sending only the relevant context instead of whole documents, a sensible maximum output length, a smaller model for simple steps, caching repeated context or answers, and batching non-urgent work. Limits and prices vary by provider and account, so always check current documentation rather than relying on remembered numbers.
Why we use it
An app that looks cheap in testing can become expensive at production volume, or fail under load when it hits rate limits. Estimating tokens and handling limits early avoids surprise bills and outages.
When to use it
Estimate cost before launching any LLM feature and monitor it after. Plan for rate limits whenever you process many items, such as tagging thousands of reviews, or serve many users at once.
How it works
- Estimate input and output tokens per call
- Multiply by expected calls per day
- Apply your provider's current rates to get daily cost
- Is it above budget? Yes: Trim prompts, cap output, cache, smaller model. No: Proceed to build.
- Handle rate-limit errors with retry and backoff
- Monitor token usage in production
Example
Pricing example (illustrative): $3 per million input tokens, $15 per million output tokens.
A chatbot sending 500 input tokens + 200 output tokens per message, at 10,000 messages/day:
Input cost/day: 5M tokens × $3/1M = $15
Output cost/day: 2M tokens × $15/1M = $30
Total: ~$45/day — trimming output length or caching repeated context can meaningfully cut this.Key points
- Input and output tokens are usually billed separately; output often costs more
- Shorter prompts and lower max-output settings directly cut cost
- Caching repeated context and batching non-urgent work helps at scale
- Handle rate-limit errors with retries and exponential backoff
- Prices and limits change: check current provider documentation