How LLMs Actually Work — For Builders

By the end of this lesson you will be able to reason about a language model the way you reason about a database or a network: as a component with a measurable cost per call, a hard capacity limit, a latency profile you can predict, and failure modes you can architect around — and you will have a working script that measures token throughput and latency against any OpenAI-compatible endpoint.

The model is a token machine, and tokens are your unit of everything

Forget the theory-course framing for a moment. As a builder, here is the only mental model you need to start: an LLM is a function that takes a sequence of tokens in and produces tokens out, one at a time. A token is a chunk of text — roughly 4 characters or about 0.75 words in English. "authentication" might be one token; "auth" + "entication" might be two, depending on the tokenizer.

Why does this matter to you and not just to researchers? Because every dimension of your product's LLM behavior is denominated in tokens:

• Cost is billed per token — input tokens and output tokens, usually at different rates.
• Capacity (the context window) is measured in tokens.
• Latency scales with tokens: a fixed-ish time to process the input, then a per-token generation time for the output.

If you internalize one thing today, make it this: when you design an LLM feature, you are designing a token budget. Everything else follows.

The context window is a budget, not a feature

Every model has a context window — the maximum number of tokens it can consider in a single call, covering your system prompt, conversation history, retrieved documents, and the response it generates. Modern models advertise windows from ~128k tokens to over a million.

New builders read "1M token context" and think "great, I'll just stuff everything in." That is the first expensive mistake, for three reasons:

1. You pay for every input token, every call. If you send a 50,000-token document with every request in a chat session, you are paying for those 50,000 tokens again on every turn. A 20-turn conversation just cost you a million input tokens.
2. Latency grows with input size. More input tokens means longer time-to-first-token. Your "instant" feature becomes a 6-second spinner.
3. Attention quality degrades. Models are demonstrably better at using information near the start and end of the context than material buried in the middle. Relevant-but-buried is functionally invisible more often than you would like.

So treat the context window like RAM on an embedded device. Ask of every feature: what is the smallest set of tokens that lets the model do this job well? That question — not "how do I fit more in" — is the origin of retrieval, summarization pipelines, and conversation truncation strategies, all of which you will build later in this course.

Sampling: why temperature is a product decision

The model doesn't output a token — it outputs a probability distribution over every possible next token, and then a sampler picks one. The main knob is temperature:

• Temperature 0 (or near it): the sampler almost always picks the most likely token. Output is close to deterministic — same input, essentially same output. Use this for extraction, classification, structured JSON output, code transforms — anything where you want repeatability and testability.
• Temperature ~0.7–1.0: the sampler takes chances on lower-probability tokens. Output varies between runs. Use this for brainstorming, copywriting variants, conversational personality.

This is not a tuning nicety; it is a product decision. If your feature is "generate 5 headline options," temperature 0 will give you five nearly identical headlines. If your feature is "extract the invoice total as JSON," temperature 1 will occasionally hand you creative arithmetic. Match the knob to the job, and write it down per-feature in your config — you'll build exactly that config in Lesson 4.

The latency and cost mental model

Here's the back-of-envelope model that will serve you in every design review:

```
total_latency ≈ time_to_first_token + (output_tokens × time_per_token)
total_cost ≈ (input_tokens × input_rate) + (output_tokens × output_rate)
```

Two practical consequences:

• Output tokens dominate latency. Generation happens token-by-token, often at tens of tokens per second. A 100-token answer might stream in 2–3 seconds; a 2,000-token answer might take 40+ seconds. If your UX needs speed, constrain output length ("answer in under 50 words") before you shop for a faster model.
• Output tokens usually cost more than input tokens — often 3–5× more per token. A summarization feature (huge input, small output) has a very different cost shape than a generation feature (small input, huge output). We do the full arithmetic in Lesson 3.

And one UX truth: streaming changes perceived latency more than any model choice. A response that starts appearing in 400ms feels fast even if it takes 8 seconds to finish. Build streaming in from day one.

What the model cannot do — and what that forces architecturally

The model is a frozen snapshot of training data with no clock, no calculator, and no network connection. Three hard limits, each with an architectural implication:

1. No fresh facts. It doesn't know today's news, your database contents, or your user's account state. It will often confidently generate plausible-looking answers anyway — this is hallucination, and it is not a bug you can prompt away. Implication: any feature that depends on current or private data needs you to retrieve the data and put it in the context. The model reasons over what you supply; your retrieval layer is the actual source of truth.
2. No exact math. It predicts tokens; it does not compute. It will get 23 × 17 right most of the time and 4,382.19 × 0.0725 wrong often enough to lose you a customer. Implication: the model decides what to calculate, your code does the calculating. This is the core idea behind tool calling, which you'll implement in the paid modules.
3. No memory between calls. Every API call is stateless. The "conversation" your users experience exists because you resend the history each turn. Implication: conversation state is your data model, your storage problem, and — see the budget discussion above — your token bill.

Builders who accept these limits early build tool-calling, retrieval-backed, verified systems. Builders who don't spend three months prompt-tweaking a hallucination problem that was never prompt-shaped.

Measure it yourself

Talk is cheap; measurements are yours. This script calls any OpenAI-compatible chat endpoint (OpenAI, Anthropic's compatibility endpoint, Together, Groq, a local Ollama server — anything speaking the same protocol) and reports token counts, time-to-first-token, and tokens per second.

```typescript
// measure-llm.ts — run with: npx tsx measure-llm.ts
// Env: LLM_BASE_URL (e.g. https://api.openai.com/v1), LLM_API_KEY, LLM_MODEL

const BASE_URL = process.env.LLM_BASE_URL ?? "https://api.openai.com/v1";
const API_KEY = process.env.LLM_API_KEY!;
const MODEL = process.env.LLM_MODEL ?? "gpt-4o-mini";

async function measure(prompt: string) {
const start = performance.now();
let firstTokenAt: number | null = null;
let output = "";

const res = await fetch(${BASE_URL}/chat/completions, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: Bearer ${API_KEY},
},
body: JSON.stringify({
model: MODEL,
messages: [{ role: "user", content: prompt }],
stream: true,
stream_options: { include_usage: true },
}),
});
if (!res.ok || !res.body) throw new Error(HTTP ${res.status}: ${await res.text()});

let usage: { prompt_tokens: number; completion_tokens: number } | undefined;
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "";

while (true) {
const { done, value } = await reader.read();
if (done) break;
buf += decoder.decode(value, { stream: true });
const lines = buf.split("\n");
buf = lines.pop() ?? "";
for (const line of lines) {
if (!line.startsWith("data: ") || line.includes("[DONE]")) continue;
const chunk = JSON.parse(line.slice(6));
const delta = chunk.choices?.[0]?.delta?.content;
if (delta) {
if (firstTokenAt === null) firstTokenAt = performance.now();
output += delta;
}
if (chunk.usage) usage = chunk.usage;
}
}

const end = performance.now();
const genMs = end - (firstTokenAt ?? end);
console.log(model: ${MODEL});
console.log(input tokens: ${usage?.prompt_tokens ?? "n/a"});
console.log(output tokens: ${usage?.completion_tokens ?? "n/a"});
console.log(time to 1st tok: ${((firstTokenAt ?? end) - start).toFixed(0)} ms);
console.log(total time: ${(end - start).toFixed(0)} ms);
if (usage?.completion_tokens && genMs > 0)
console.log(tokens/sec: ${(usage.completion_tokens / (genMs / 1000)).toFixed(1)});
console.log(---\n${output.slice(0, 200)}...);
}

measure("Explain what a token is in one short paragraph, then in one long paragraph.");
```

Run it a few times. Notice the variance between runs — that variance is why you'll add timeouts and retries in Lesson 4.

Exercise

Run the measurement script against one endpoint with three prompts of increasing requested output length: "answer in one sentence," "answer in one paragraph," and "answer in 500 words" — same question each time. Record time-to-first-token, total latency, and output token count for each. Success looks like: a small table of your three runs, plus one sentence stating the relationship you observed between output tokens and total latency (it should be roughly linear — and now you've proven the lesson's core claim on your own hardware and wallet).

You now think in tokens, budgets, and failure modes — next, we weaponize that by dissecting exactly what goes into the context: the anatomy of a prompt that behaves like an engineered interface instead of a wish.