You've just started learning AI. This walks you from "what is a model?" all the way to the clever tricks that squeeze a giant AI onto a small device — one idea at a time, in plain language.
Before any of the fancy stuff, let's nail down four words. Everything later is built on these.
A language model is a computer program that predicts the next word. That's really all it does — you give it some text, and it guesses what word comes next, then the next, then the next. Do that over and over and you get sentences, answers, essays. A Large Language Model (LLM) is just a very big version of this, like ChatGPT.
Inside the model are billions of numbers. These numbers are called weights (or parameters — same thing). They were tuned during training, when the model read huge amounts of text. The weights are the model's knowledge. A "3-billion-parameter model" literally means 3 billion of these numbers.
The model doesn't read whole words — it reads tokens, which are word-pieces. "Running" might be one token, or split into "run" + "ning". Roughly, one token ≈ ¾ of a word. The model predicts one token at a time. Remember that phrase — it comes back constantly.
Inference is just the act of using a trained model to generate an answer (as opposed to training it). When you type a question and get a reply, that's inference. This whole guide is about making inference fast and small.
Most AI today runs in the cloud — on powerful servers in a data center. Your phone sends your question over the internet, a big machine does the work, and the answer comes back.
Edge means the opposite: running the model directly on your own device — your phone, laptop, or a small gadget — with no server involved. This is what a project like EdgeLM is about.
Why bother? Three reasons: it works offline, it's private (your data never leaves the device), and there's no server bill. The catch is that a phone is tiny compared to a data center, so we have to be very clever. That cleverness is the rest of this guide.
If you remember only one technical idea, make it this one, because it explains why almost every trick exists.
So the two winning moves are almost always: (a) read fewer bytes, or (b) avoid redoing work. Watch for these two themes below.
Freshly trained, each weight is stored as a very precise number using 16 or 32 bits (bits are the 0s and 1s computers store). A 3-billion-weight model at 16 bits is about 6 gigabytes — too big for most phones, and slow to read.
Quantization means storing each weight with fewer bits — often just 4. You lose a little precision, but the model gets ~4× smaller.
You take a small group of weights (say 32 of them, called a block), look at the biggest value in that block, and compute a single shared scale factor. Then each weight is stored as a tiny 4-bit whole number, and at run time you multiply it by the block's scale to get back an approximate original value. That simple scheme is exactly what Q4_0 means: 4-bit weights, one scale per block.
Two payoffs: the model now fits in the phone's memory, and because generation is memory-bound, reading 4× fewer bytes makes it roughly correspondingly faster. The cost — a hair of accuracy from the rounding — is usually barely noticeable.
Go deeper (reference): the exact quantization math — block scales and the q = round(w / scale) arithmetic, the per-block overhead (~4.5 bits/weight), symmetric vs asymmetric formats (Q4_0 / Q4_1 / Q8_0), outliers, and the smarter K-quants — is covered in guide 6 — Advanced Edge Internals.
Q4_0. That's not a bug — since decode is memory-bound, making the math faster buys nothing. The golden rule in action.To pick the next token, the model looks back at everything said so far — a mechanism called attention. For each token it builds three little vectors nicknamed Q, K, V (Query, Key, Value). The current token's Query compares itself against the Keys of every earlier token to decide which past words matter, then pulls in their Values.
Here's the problem: when writing token #500, you'd need the K and V of tokens #1–499. Recomputing all of those every single step would be painfully slow.
So the model saves the Keys and Values of every token it has already processed. That saved store is the KV cache. Now each new token only computes its own K and V and adds them to the pile — the past is already done.
The cache grows as the conversation gets longer, and it can eat a lot of memory. Paged-KV borrows a trick from your operating system: instead of one big rigid block of memory, it chops the cache into small fixed-size pages. A conversation grabs pages as it grows and hands them back when finished — no wasted space, and several conversations can share one pool neatly. (This is a piece EdgeLM built.)
A phone has more than one chip that can do math, and each is good at different things. The software that runs the model on a particular chip is called a backend.
CPU — the general-purpose brain. Always available and flexible, but only moderately parallel. A reliable baseline (and memory-bound during generation).
GPU — thousands of tiny cores that do the same math on lots of data at once. Great for the big matrix multiplications inside a model. On phones these are called Adreno (Qualcomm) or Mali (ARM).
NPU — a chip built only for neural-network math. Most power-efficient when it works, but fussier about which models and formats it accepts. Qualcomm's software stack for it is called QNN.
The engineering job is to detect what the device actually has and route the work to the fastest option that works.
Remember: tokens come out one at a time, and each one costs a full read of the big model. Speculative decoding tries to get several tokens per big-model pass.
The catch: this only wins if the big model has spare compute to absorb the checking cheaply. A GPU does. A CPU — already memory-bound and compute-starved — just drowns under the extra work.
Normally, to use a model, a program loads the entire file into its own memory. If three apps each want the model, that's three copies — three times the RAM. Impossible on a phone.
Memory-mapping (or mmap) tells the operating system: "make this file look like it's in my memory, and quietly load pieces from storage as needed." The OS keeps the file's pages in one shared area. When a second app maps the same file, the OS points it at the same pages — no second copy is made.
One shared model in memory is great, but what if several apps want to use it at the same moment? You need a traffic controller.
A broker or scheduler sits in front of the model. Apps send their requests to it; it lines them up, decides the order, and batches compatible requests so the model handles several in a single pass — far more efficient than one at a time.
This is what turns a single loaded model into a small on-device service that many apps can call at once.
One last idea that ties everything together. Generating an answer happens in two phases with very different personalities:
Prefill — the model reads your whole prompt at once. Lots of math in parallel, so it's compute-heavy. This phase loves GPUs/NPUs and fast-math tricks.
Decode — the model writes the answer one token at a time. This is memory-bound (the golden rule), so it loves quantization and the KV cache.
Most of what a user feels as "slowness" is the decode phase — the answer trickling out word by word. That's why so much edge work targets decode specifically.
No pressure — pick an answer and it'll tell you why. Your score updates as you go.
1. When a model generates text, what is usually the main speed bottleneck?
Memory speed. Generation is "memory-bound" — the chip mostly waits for weights to arrive from RAM, not for math.
2. What does quantization do?
Fewer bits per weight. That shrinks the file ~4× and, because decode is memory-bound, speeds it up too. Q4_0 is one such scheme.
3. Why does a model keep a KV cache?
To avoid redoing work. Each new token reuses the saved K,V of all earlier tokens instead of recomputing them.
4. Which chip is built only for neural-network math?
The NPU. Most power-efficient when it works, but pickier about models and formats. Qualcomm's stack for it is called QNN.
5. Speculative decoding was a net loss on CPU in EdgeLM. Why?
No spare compute. The trick only wins when the big model has room to verify guesses cheaply — true on GPU, not on a memory-bound CPU.
6. What does mmap let several apps do?
Share one copy. The OS maps the same file's pages for every app — the "one copy in RAM" result EdgeLM validated.
7. Which phase does a user usually feel as "slowness"?
Decode. The answer trickles out token by token, so that's where most edge optimization is aimed.
You now understand how a model is squeezed onto a device. Here's a sensible path for what to learn next, roughly easiest to hardest. Each card says what it is and why it's worth your time.
What: The math that lets a token "look back" at earlier tokens — the Query/Key/Value idea from the KV-cache section, one level deeper. This is the heart of the Transformer, the architecture behind every modern LLM.
Why now: It's the single most important mechanism in LLMs, and you've already met its vocabulary here. → Read the companion deep-dive on attention & Transformers (same beginner style). Also search: "illustrated transformer".
What: Turning words (or sentences, images, audio) into lists of numbers — vectors — where similar meanings sit close together. They power search, recommendations, and "chat with your documents."
Why now: They're simpler than full generation and immediately useful. Bonus: EdgeLM has an on-device /v1/embeddings feature, so this connects straight to the project. → Read the companion deep-dive on embeddings (same beginner style).
What: Fine-tuning means taking a pre-trained model and training it a little more on your own data to specialize it. LoRA is a cheap, popular way to do that by training only a tiny set of extra weights instead of all billions.
Why now: It's how people customize models without a data center — very relevant to small/edge setups. Search: "fine-tuning vs prompting", "LoRA explained".
What: The loop that creates the weights in the first place: gradient descent and backpropagation — nudging billions of numbers to reduce error, over and over.
Why now: Everything in this guide is about using a finished model; this explains where it came from. A bit more math-heavy, so save it for once the above feels comfortable. Search: "gradient descent intuition", "backpropagation explained".
What: Quantization schemes beyond Q4_0 (like Q8_0 and "K-quants"), how NPU stacks (QNN) and runtimes (llama.cpp, LiteRT-LM) are built, and continuous batching in the scheduler.
Why now: These are the natural next steps once the fundamentals click — and they map directly onto EdgeLM's own roadmap notes.