ZARAI — AI tools and community

By the end of this lesson you will be able to pick a model tier for a feature on purpose instead of by default, turn token estimates into per-request and per-month costs you can defend in a pricing meeting, know the two situations where local models actually win, and have a runnable cost-model script you can point at any feature idea before you build it.

The three tiers, and what you're actually trading

Model catalogs change monthly; the structure of the market doesn't. Every major provider ships roughly three tiers:

Frontier — the biggest, smartest model. Best at multi-step reasoning, ambiguous instructions, hard code generation, nuanced judgment. Slowest and by far the most expensive — often 10–30× the small tier per token.
Mid — 80–90% of frontier quality on most tasks at a fraction of the price. The workhorse tier for production features.
Small/fast — cheap and quick. Excellent at classification, extraction, routing, formatting, and simple summarization. Falls apart on long-horizon reasoning.

The trap is anchoring on the demo. You prototype with the frontier model because you want your idea to look good, it does, and now frontier pricing is silently baked into your unit economics. The professional move is the opposite: build the feature, write ten test cases, then walk DOWN the tiers until quality breaks. The tier just above the breaking point is your production model. Most features that founders assume need frontier — triage, extraction, summarization, routing — run fine on mid or small. Save frontier spend for the steps that genuinely need judgment.

A second habit: decompose before you upgrade. A pipeline of "small model classifies → mid model drafts → frontier model reviews only the flagged 10%" routinely beats "frontier does everything" on both cost and quality.

From price-per-token to unit economics

Providers price per million tokens, input and output separately. All numbers below are illustrative — check current provider pricing before you commit real forecasts; rates change often and vary by provider. We'll use deliberately round numbers:

| Tier (illustrative) | Input / 1M tokens | Output / 1M tokens |
|---|---|---|
| Frontier | $10.00 | $30.00 |
| Mid | $1.00 | $4.00 |
| Small | $0.10 | $0.40 |

Now a worked example. Feature: "summarize this support conversation" button in a helpdesk product.

Estimate the token shape of one request:

System prompt + instructions: ~500 tokens
Conversation transcript: ~3,000 tokens
Output summary: ~300 tokens

Per-request cost on the mid tier:

```
input: 3,500 tokens × ($1.00 / 1,000,000) = $0.0035
output: 300 tokens × ($4.00 / 1,000,000) = $0.0012
per request ≈ $0.0047
```

Under half a cent. Feels free, right? Now scale it. Say each active user clicks the button 30 times a month:

```
per user / month: 30 × $0.0047 = $0.141
per 1,000 users: 1,000 × $0.141 = $141 / month
per 10,000 users: = $1,410 / month
```

Run the same shape on frontier ($10 in / $30 out): $0.044 per request → $1,320 per 1,000 users per month — nearly 10× your mid-tier cost for the entire 10k-user base. And on small ($0.10 / $0.40): $0.00047 per request → $14 per 1,000 users. Three tiers, same feature: $14 vs $141 vs $1,320 per 1k users. That two-orders-of-magnitude spread is the difference between a feature you bundle free and a feature that eats your margin.

Two multipliers people forget:

Conversation history compounds input tokens. In a chat feature, turn 10 resends turns 1–9. Average input per request can be 5–10× your single-turn estimate.
Retries and regenerations are real traffic. Users click regenerate; your retry logic refires on timeouts. Budget 1.2–1.5× your naive request count.

When local models make sense

"Local" means open-weight models (Llama, Mistral, Qwen and friends) on hardware you control — your GPU server, or on-device. Two situations where they genuinely win:

Privacy and data governance. If your customers are in healthcare, defense, legal, or finance, "your data never leaves our infrastructure" isn't a preference — it's a procurement requirement. A local model turns an impossible sale into a checkbox. This is a capability argument, and it's often decisive regardless of cost.
The cost floor at sustained high volume. API pricing is linear forever: 100M requests cost 100M × per-request. Self-hosted inference is a step function: fixed hardware/ops cost, near-zero marginal cost. If you're running millions of small-model calls a day (classification, moderation, embedding-adjacent tasks), the lines cross and local wins.

Be honest about the other side of the ledger: you inherit GPU provisioning, model updates, quantization tradeoffs, scaling, and on-call. Below serious volume, API-tier small models are almost always cheaper than one engineer-week of infrastructure work per month. The mature pattern is hybrid — local for the high-volume simple calls or the privacy-bound path, API frontier for the low-volume hard reasoning.

Fallback chains: model choice is an availability strategy

Providers have outages, rate limits, and bad latency days. If your product hard-depends on exactly one model from one provider, that provider's status page is your status page.

The fix is a fallback chain per feature: an ordered list of (provider, model) pairs, tried in sequence on failure:

```
summarize_ticket:

mid-tier model, provider A (primary — best cost/quality)
mid-tier model, provider B (provider A outage)
small model, provider A (degraded mode: worse summary > no summary)
```

Three design rules. Chain across providers, not just models — a provider outage takes out all its models at once. Decide what "degraded" means per feature — a worse summary is acceptable; a worse legal-compliance answer may not be, so some chains should end in "return an error" rather than a weaker model. This is why you standardize on the OpenAI-compatible request shape (as in Lesson 1's script): when every provider speaks the same protocol, failover is a config change, not a rewrite. You'll build exactly this into your client in the next lesson.

The cost model, as code

Stop estimating in your head. This script is the artifact for this lesson: describe a feature's token shape, get per-request and per-1k-users monthly cost across tiers. Rates are illustrative — check current provider pricing and edit the table.

```typescript
// cost-model.ts — run with: npx tsx cost-model.ts

// ILLUSTRATIVE rates in $ per 1M tokens — replace with current provider pricing.
const TIERS = {
frontier: { input: 10.0, output: 30.0 },
mid: { input: 1.0, output: 4.0 },
small: { input: 0.1, output: 0.4 },
} as const;

interface FeatureShape {
name: string;
inputTokens: number; // avg tokens sent per request (prompt + context)
outputTokens: number; // avg tokens generated per request
requestsPerUserMonth: number;
retryMultiplier?: number; // regenerations + retries, default 1.3
}

cost({
name: "Summarize support conversation",
inputTokens: 3_500,
outputTokens: 300,
requestsPerUserMonth: 30,
});

cost({
name: "AI chat assistant (history compounds input)",
inputTokens: 12_000, // avg across a session's turns, history included
outputTokens: 400,
requestsPerUserMonth: 120,
});
```

Run it and look at the chat assistant line. Chat features with compounding history are routinely 20–50× the cost of single-shot features — knowing that before you promise "unlimited AI chat" on your pricing page is the whole point of this lesson.

Exercise

Model the flagship AI feature of the product you want to build. Estimate its token shape honestly (write one real prompt, paste a realistic input, count the tokens — Lesson 1's script prints them), set a realistic requests-per-user-month, and run the cost model across all three tiers. Success looks like: one sentence you could say to a co-founder with a straight face: "This feature costs about $X per 1,000 users per month on the [tier] model, so at a $Y/month price point our AI gross margin is Z%." If the margin is ugly, redesign the token shape — shorter context, smaller tier, decomposed pipeline — and run it again.

You now know what to say to the model, and what saying it costs — next we build the production-shaped project skeleton that every feature in the rest of this course plugs into: typed client, timeouts, retries, failover, and logging you'll thank yourself for.