The deep end. Now that the fundamentals are solid, here's what really happens inside a fast on-device runtime: smarter quantization, squeezing the CPU, the reality of NPUs, and serving many requests at once. This is EdgeLM's actual engineering frontier.
From guide 1: generating a token means reading the model's weights out of memory, so decode speed is usually limited by memory bandwidth, not math. Every technique here is another angle on the same fight — move fewer bytes, waste no compute, keep every core busy. If that sentence makes sense, you're ready.
Guide 1 explained Q4_0: chop weights into blocks, store each as a 4-bit integer with one shared scale. It's simple but blunt — every weight in a block is treated as equally important, and it uses just one scale.
Before the K-quants, the simplest upgrade is Q4_1. Both are 4-bit and block-based; the difference is how each block's 16 levels are placed.
Q4_0 is symmetric: it stores just a scale per block and assumes the weights are centred on zero, so w ≈ q × scale with a signed integer q. Q4_1 is asymmetric: it stores a scale and a minimum (offset), so w ≈ q × scale + min with an unsigned q. That extra min lets the levels line up with a lopsided block instead of forcing them around zero — more accurate for skewed weights, at the cost of a bit more storage (~5 bits/weight vs ~4.5).
Newer schemes do better still. Q8_0 keeps 8 bits — bigger and more accurate, often used for the KV cache. But the clever family is the K-quants (you'll see names like Q4_K_M, Q5_K_M, Q6_K).
Two upgrades. First, sub-blocks with their own scales: instead of one scale per big block, they add finer-grained scales (and offsets) so the rounding fits the local data much better. Second, mixed precision: not all weights matter equally, so the important ones get more bits and the rest get fewer. The _M ("medium") in Q4_K_M refers to which layers are bumped up.
Q4_0 compresses a photo at one quality setting for the whole image. K-quants spend more bits on the faces and fewer on the flat sky — same file size, better where it matters.Q4_0, because decode is bandwidth-bound and the runtime already repacks Q4_0 efficiently. Picking the format (e.g. Q4_K_M vs Q4_0 vs Q8_0 KV) is often a bigger lever than accelerating the math.Most phones do inference on the CPU, so wringing more out of it matters. Two levers stand out.
A CPU core can multiply several numbers in one instruction using SIMD ("single instruction, multiple data"), and modern ARM chips have an i8mm extension for fast 8-bit integer matrix math. But it only pays off if the weights are laid out in exactly the order the instruction wants. So the runtime repacks the quantized weights into that friendly layout up front. This is why extra math libraries can add nothing — the fast path is already taken.
Phone CPUs are big.LITTLE: a few fast "big" cores and several slow, efficient "little" ones. If work drifts onto the little cores, throughput drops. A tuned runtime builds a threadpool pinned to the big cores, picks the best thread count by measuring, and can request a sustained performance mode so the chip doesn't throttle mid-generation.
Guide 1 introduced the NPU — a chip built only for neural-net math, the most power-efficient option when it works. Qualcomm's software stack for it is QNN. So why isn't everything on the NPU already?
Because NPUs are rigid by design. Where a CPU runs any code on the fly, an NPU wants the whole model compiled ahead of time into a fixed graph of operations it supports, in a specific quantization format. Three friction points follow:
Operator coverage — if the model uses an operation the NPU doesn't implement, that part falls back to CPU, and the handoffs can erase the win.
Format constraints — the NPU accepts particular quantization layouts, not whatever .gguf you happen to have.
Ahead-of-time compilation — the model must be converted and compiled per chip family, which is real engineering, not a flag you flip.
/v1/edge/device endpoint) and an honest integration plan, but no QNN backend yet — it's groundwork. That candor is the point: "supports NPU" is a long road, not a checkbox.Guide 1's scheduler batches requests so one model serves several at once. But there's a naive way and a smart way.
Static batching: gather a group, run them together, and don't start new work until the whole group finishes. The problem: requests have different lengths, so everyone waits for the slowest one, and the model sits half-idle.
Continuous batching fixes this. The moment any request in the batch finishes, its slot is freed and a waiting request drops in — the batch is refilled every step instead of drained and refilled. The model stays full, and new users don't wait for a whole group to clear.
Continuous batching needs to add and remove requests from the running batch at any moment — each with its own growing KV cache (guide 1). If each cache were one big fixed block, you couldn't cleanly slot conversations in and out without wasting or fragmenting memory.
Paged-KV is what unlocks it: because each cache is split into small reusable pages, a finishing request hands its pages back to the pool and an incoming request grabs what it needs. The scheduler and paged-KV are two halves of the same capability.
Pulling the honest status together, so you can tell built-and-proven from work-in-progress:
| Piece | Status |
|---|---|
| mmap sharing | Built & proven on device — "one copy in RAM" across apps. |
| paged-KV | Built in increments, integrated with a batched service. |
| CPU perf levers | Built (big-core threadpool, Q8_0 KV, thread sweep, sustained mode) — unmeasured, needs on-device build. |
| spec. decoding | Gated GPU-only; net loss on CPU. |
| GPU backend | Integrated engine; unsupported on Mali-G615 — needs a flagship Adreno. |
| NPU / QNN | Detection + plan only; no QNN backend yet. |
Pick an answer and it'll explain why. Score updates as you go.
1. How do K-quants (like Q4_K_M) beat plain Q4_0?
Finer scales + mixed precision. Important weights get more bits; local scales fit the data better — near-Q8 quality at near-Q4 size.
2. Why can a math-acceleration library add ~0% to decode on Q4_0?
Bandwidth-bound. Faster math doesn't help when you're waiting on memory and the SIMD path is already taken.
3. What is weight "repacking" for?
SIMD-friendly layout. Instructions like i8mm are only fast if the data is pre-arranged in their expected order.
4. Why are threads pinned to the "big" cores?
Keep the heavy work fast. big.LITTLE chips mix fast and slow cores; pinning keeps generation on the fast ones.
5. What makes NPUs hard to use?
Rigid by design. Unsupported ops fall back to CPU, and the model must be converted/compiled per chip — real work, not a flag.
6. How does continuous batching beat static batching?
No idle gaps. As each request finishes, a waiting one drops in, so the model stays full every step.
7. Why does continuous batching depend on paged-KV?
Two halves of one capability. Reusable pages let conversations slot in and out without wasting or fragmenting memory.