A 200 OK does not mean your LLM used the GPU
A low tok/s number cannot tell whether the model is slow or never made it onto the GPU. With 0/33 layers on the GPU, prefill fell 22x while the server kept returning 200.
A local LLM server can return 200, generate text, and still run 0/33 layers on the GPU.
A low tok/s number cannot tell whether the model itself is slow or never made it onto the GPU. If the model is slow, different hardware may help. If it did not fit, a different quant or context size can change the result on the same machine.
Both placements answered requests normally.
Same model, three different gaps
I forced the same Qwen3.5-9B Q4_K_M file through both paths on an M5 with 32GB of unified memory.
| Placement | Prefill | Decode | Wall-clock rate |
|---|---|---|---|
| Metal, 33/33 layers | 588.0 tok/s | 21.1 tok/s | 15.5 tok/s |
| CPU, 0/33 layers | 26.8 tok/s | 12.2 tok/s | 3.0 tok/s |
Prefill fell 22x. Decode fell 1.7x. The end-to-end rate fell 5.2x.
Both runs used the same 8.95B-parameter, 5.28 GiB file with llama.cpp build 9430 on macOS 26.5.1. The context was 4096 tokens with 4 of 10 CPU threads. Picchio’s mp1 protocol used roughly 730 prompt tokens and 128 generated tokens per pass, three passes with the first one cold. The table holds the warm medians.
The saved verdict blocks are 33/33 on Metal and 0/33 on CPU. The raw engine logs replay in Picchio’s self-test.
Prefill set the wait before the first token on a long prompt. Decode set the pace once text started moving. A tok/s result without the lane could describe either.
A correct signal on the wrong label
The app’s health check called /v1/models and turned a 200 response into ready. llama.cpp documents that endpoint as its model-info API.
GPU placement was reported somewhere else. --n-gpu-layers controlled the requested offload, and the startup log printed the result. A CPU run could report 0/33 layers offloaded while the server remained available.
Both signals were correct. I had merged them into one status.
The endpoint answered whether the service was up. The startup log answered where the model was running. I displayed the first beside a label about the second.
Unified memory did not remove the path
Apple lists 153GB/s of memory bandwidth for the 14-inch M5 MacBook Pro. The CPU and GPU share that memory pool, so the model does not need a second copy moved into discrete VRAM over PCIe.
That describes where the bytes live. It does not make CPU and GPU execution equivalent. Metal still has explicit resource storage and access choices for CPU and GPU work. Kernels, caches, scheduling, and the two inference lanes can respond differently. Apple’s Metal documentation keeps storage mode as a choice, including on Apple GPUs.
The 22x and 1.7x results came from the same memory pool, model file, and machine.
The three states I keep now
| State | Evidence |
|---|---|
| Service availability | The endpoint answers and inference completes |
| Model placement | The runtime reports CPU, GPU, or a split |
| Measured execution | Prefill, decode, and wall-clock timing agree with the OS GPU meter |
For llama.cpp, placement comes from the startup offload line. /v1/models stays a service and model-info check.
For Ollama, ollama ps reports a PROCESSOR split such as 100% GPU, 100% CPU, or a CPU/GPU memory split. It does not expose llama.cpp’s per-layer placement, but it does expose where the model memory landed.
MLX exposes CPU and GPU device types, and set_default_device changes the execution device. Picchio handles MLX through the OS-side watch path rather than giving it the full llama.cpp lane verdict.
Picchio grew out of this separation. It keeps prefill, decode, and wall-clock time apart, then reads engine placement beside the OS GPU meter. Its monitor mode compares a running server against that engine’s own healthy baseline.
/v1/models still returns the same 200. I just stopped asking it a GPU question.