API Costs & Rate Limits

Gen AI · Ethics, Cost & Careers

Theory

Most hosted LLM APIs charge by tokens processed, with input tokens (what you send) and output tokens (what the model writes) usually priced separately, and output tokens typically costing more each. Providers also set rate limits: caps on requests or tokens per minute or per day, to share capacity fairly. Hitting a limit returns an error, so apps must retry with a short, growing wait (exponential backoff).

Cost is easy to estimate: tokens per call times calls per day. The main levers are shorter prompts, sending only the relevant context instead of whole documents, a sensible maximum output length, a smaller model for simple steps, caching repeated context or answers, and batching non-urgent work. Limits and prices vary by provider and account, so always check current documentation rather than relying on remembered numbers.

Why we use it

An app that looks cheap in testing can become expensive at production volume, or fail under load when it hits rate limits. Estimating tokens and handling limits early avoids surprise bills and outages.

When to use it

Estimate cost before launching any LLM feature and monitor it after. Plan for rate limits whenever you process many items, such as tagging thousands of reviews, or serve many users at once.

How it works

  1. Estimate input and output tokens per call
  2. Multiply by expected calls per day
  3. Apply your provider's current rates to get daily cost
  4. Is it above budget? Yes: Trim prompts, cap output, cache, smaller model. No: Proceed to build.
  5. Handle rate-limit errors with retry and backoff
  6. Monitor token usage in production

Example

Pricing example (illustrative): $3 per million input tokens, $15 per million output tokens.

A chatbot sending 500 input tokens + 200 output tokens per message, at 10,000 messages/day:
Input cost/day: 5M tokens × $3/1M = $15
Output cost/day: 2M tokens × $15/1M = $30
Total: ~$45/day — trimming output length or caching repeated context can meaningfully cut this.

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic