The last guide mentioned "attention" and "Q, K, V" in passing. This one opens that box slowly and carefully — no prior math needed. By the end you'll understand the engine inside every modern LLM.
0. Why attention matters
To predict the next word, a model has to understand how words relate. In the sentence "The trophy didn't fit in the suitcase because it was too big," what does "it" mean — the trophy or the suitcase? A human knows instantly. The model needs a way for the word "it" to look at the other words and figure out which one it depends on.
That mechanism is called attention. It was introduced in a famous 2017 paper ("Attention Is All You Need") and it's the beating heart of the Transformer — the design behind GPT, Llama, Gemma, and essentially every modern LLM. Understand attention and you understand the core of the whole field.
1. Step 1: words become vectors (embeddings)
Computers can't do math on the letters "c-a-t." So the very first thing a model does is turn each token into a list of numbers called a vector (also called an embedding). A vector might be a few hundred to a few thousand numbers long.
The magic is which numbers. During training, the model arranges these vectors so that words with similar meanings get similar numbers. "Cat" and "kitten" end up close together; "cat" and "airplane" end up far apart.
Every token becomes a vector. Similar meanings → similar vectors.
Analogy: Think of a map where every word is a pin. Related words are pinned near each other. The vector is just that pin's coordinates — except instead of 2 numbers (latitude, longitude) there are hundreds.
This is exactly the embeddings idea from the first guide's "where to go next" list — and it's a real EdgeLM feature (/v1/embeddings). Attention is what happens to these vectors next. → Full companion guide on embeddings.
2. Step 2: the core idea of attention
Now every word is a vector. Attention lets each word update its own vector by mixing in information from the other words it cares about.
Back to "…because it was too big." The vector for "it" is initially generic. Attention lets "it" gather meaning from "trophy" and "suitcase," weigh which one matters more, and absorb it — so afterwards, the vector for "it" quietly means "the trophy."
The one-sentence version: attention = each word looks at every other word, decides how relevant each one is, and pulls in a weighted blend of their information.
The only question left is: how does a word decide who's relevant, and what exactly does it pull in? That's where Query, Key, and Value come in.
3. Step 3: Query, Key, Value — in plain words
From each word's vector, the model creates three new vectors by multiplying it with three sets of trained weights. They have job-titles:
Query (Q) — "what am I looking for?" The current word's search request.
Key (K) — "what do I offer?" A label each word advertises about itself.
Value (V) — "what will I hand over if you pick me?" The actual information a word contributes.
Every word turns its vector into three role-vectors. Queries match Keys; the winners' Values get pulled in.
Analogy — a library search: Your Query is what you type into the search box. Each book has a Key on its spine (its label). You compare your query to every spine to see how well it matches. Then from the best-matching books you take the Value — the actual contents inside. Attention does this for every word at once.
So each word issues a Query, every word offers a Key and a Value, and the matching between Queries and Keys decides how much of each Value gets pulled in.
4. Step 4: one attention step, walked through
Here's the whole thing for a single word, start to finish. Don't worry about the formulas — read the plain-English label on each step.
Make Q, K, V. Turn each word's vector into its Query, Key, and Value vectors.
Score. Compare this word's Query against every word's Key. A closer match = a higher score. (The comparison is a "dot product" — just a number saying how aligned two vectors are.)
Normalize into weights. Push all the scores through softmax, which turns them into percentages that add up to 100%. Now you have "pay 70% attention to this word, 20% to that one…"
Blend the Values. Take each word's Value, multiply by its weight, and add them all up. That weighted blend is the word's new, context-aware vector.
"it" pays 70% attention to "trophy" — so it absorbs the trophy's meaning.
That's self-attention: every word in the sentence does this at the same time, each ending up with a richer vector that "knows" about its context.
For the curious, the entire operation is one tidy formula — but every symbol in it is a step you just read:
Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V
Q·Kᵀ is the scoring, √d just keeps the numbers from getting too large, softmax makes weights, and ·V blends the Values. That's it.
5. Step 5: multi-head attention
One round of attention can only focus on one kind of relationship at a time. But language has many at once: who did what, which adjective describes which noun, what "it" refers to. So models run several attention operations in parallel, called heads.
Each head has its own Q, K, V weights and learns to notice a different pattern. Their results are then combined. This is multi-head attention.
Different heads catch different relationships; their views are merged.
Analogy: Several proofreaders read the same sentence — one checks grammar, one tracks who "it" refers to, one watches the adjectives. You pool their notes to fully understand the sentence.
6. Step 6: how the model knows word order
There's a subtle gap. Attention, on its own, treats a sentence like a bag of words — it has no built-in sense of order. But "dog bites man" and "man bites dog" are very different!
The fix is positional encoding: before attention, the model adds a little "position signal" to each word's vector — a numeric stamp saying "I'm word #1," "I'm word #2," and so on. Now attention can tell where each word sits.
A position stamp is added to each word vector, so the model can't mix up word order.
Analogy: Numbered seats at a theater. The people (words) are the same, but the seat numbers (positions) let everyone tell who's sitting where.
7. Step 7: the full Transformer block
Attention is the star, but a full Transformer block wraps it with a few helpers. Stacking many of these blocks (modern models have dozens) is what makes an LLM. Each block does:
1 · Multi-head attention — words share context (everything above).
2 · A feed-forward network — a small standalone step applied to each word to "think" about what it just gathered. This is where a lot of the model's raw knowledge lives.
3 · Residual connections — each step adds its result on top of the input instead of replacing it, so nothing important gets lost as data flows through many layers.
4 · Layer normalization — a tidy-up that keeps the numbers in a healthy range so training stays stable.
One block = attention + feed-forward (each with add & normalize). Stack many to build an LLM.
8. Step 8: how the next word is chosen
After the data flows through all the blocks, the model has a final, deeply-informed vector for the most recent position. One last step turns that into an actual next word:
The final vector is compared against every word in the model's vocabulary, producing a score for each possible next token.
Softmax (the same normalizer from before) turns those scores into probabilities — e.g. "big" 61%, "heavy" 22%, "large" 9%, …
The model picks from that distribution. Pick the top one every time for safe, predictable text; allow some randomness (the temperature setting) for more creative text.
The final vector becomes probabilities over every possible next word; the model picks one, then repeats.
Then the chosen word is fed back in and the whole process repeats for the next word. That "one token at a time" loop is exactly the behavior the first guide built everything around.
9. Tying back to the KV cache
Now the KV cache from the first guide should click completely. Notice that when generating each new word, every earlier word's Key and Value don't change — they only depend on that word, not on the new one being written.
So: instead of recomputing the K and V for the whole sentence every step, the model computes each word's K and V once and stores them — the KV cache. Only the newest word needs fresh Q, K, V. That's why the cache saves so much work, and why "paged-KV" (storing it efficiently) matters on a memory-tight device.
10. Test yourself
Pick an answer and it'll explain why. Score updates as you go.
1. What is an embedding?
A vector. Each token becomes a list of numbers arranged so similar meanings sit close together.
2. In one sentence, what does attention let a word do?
Correct. Each word scores every other word's relevance, then absorbs a weighted mix of their Values.
3. Which is the best plain-English match?
Value = the contents handed over. Query is the search request; Key is the label being matched against.
4. What does softmax do in attention?
Turns scores into weights. Those weights say how much attention to pay to each word.
5. Why use multiple attention heads?
Different relationships. One head might track grammar, another what "it" refers to, and so on — then they're combined.
6. Why does the model need positional encoding?
Word order. Without a position signal, "dog bites man" and "man bites dog" would look the same to attention.
7. Why can the KV cache save so much work?
Reused, not recomputed. Only the newest token needs fresh Q, K, V; the rest are already stored.
Score: 0 / 7
Mini-glossary
Vector / embedding
A list of numbers representing a token's meaning; similar meanings sit close together.
Attention
Each word looks at every other word and absorbs a weighted blend of their information.
Self-attention
Attention where words in the same sentence attend to each other.
Query / Key / Value
Q = the search request; K = a word's advertised label; V = the info a word contributes.
Softmax
Turns a set of raw scores into weights that add up to 100%.
Multi-head attention
Several attention operations in parallel, each catching a different relationship.
Positional encoding
A position signal added to each word so the model knows word order.
Feed-forward network
A per-word step inside each block where much of the model's knowledge lives.
Residual connection
Adding a step's output to its input so information isn't lost across many layers.
Transformer block
Attention + feed-forward (plus add & normalize); stacked many times to build an LLM.
Temperature
A knob for how random the next-word choice is — low = safe, high = creative.