A model that scores well on a workstation GPU is rarely the model that ships. To run on a camera, a phone or a microcontroller it usually has to get smaller and faster, and every change puts accuracy at risk. This guide explains the main compression techniques, how to test them, and how to decide which to use, one step at a time.
Set the bar before you compress anything
Compression is a trade, so you need a way to tell whether the trade was worth it. That means an evaluation set and a written acceptance bar, agreed before anyone touches the model. The Australian Government's Guidance for AI adoption makes the same point for AI generally: define acceptance criteria and test methods that reflect the intended use, then test before you deploy.
For an edge model, a useful evaluation set has:
- Field data, not just lab data. Images from the real camera, mount and lighting, or audio from the real microphone in the real room.
- Named slices. Night versus day, dust, rain, glare, unusual objects. Average accuracy hides the slice that matters.
- The error that costs most. A missed person near a machine is worse than a false alarm, but a false alarm every ten minutes makes the system useless. Choose the metric that reflects both.
- Hardware budgets. Maximum latency per input, peak memory and model file size on the target device.
The main levers
Quantisation changes the number format of weights and activations. Pruning removes weights. Distillation trains a different, smaller model. Underneath all three sits architecture choice: starting from a model family designed for mobile or embedded use. In practice they are often combined.
Post-training quantisation: try it first
Post-training quantisation converts a trained model without retraining it. Google's LiteRT documentation describes several options:
| Option | Calibration data | What LiteRT says you get | Notes |
|---|---|---|---|
| Dynamic range | Not needed | 4x smaller, 2x–3x speedup on CPU | The simplest starting point |
| Full integer | Around 100–500 representative samples | 4x smaller, 3x+ speedup; runs on CPU, Edge TPU and microcontrollers | Required for integer-only hardware; accuracy can drop, particularly in smaller networks |
| Float16 | Not needed | 2x smaller, GPU acceleration | Minimal accuracy loss, but less latency gain than integer formats |
| 16x8 (experimental) | Needed | Can improve accuracy significantly with only a slight size increase | Noticeably slower than 8-bit integer; not supported by existing hardware-accelerated delegates |
The calibration samples matter more than people expect. Full-integer quantisation sets its numeric ranges from them, so if they come only from daytime images, the night-time slice can suffer. Draw them from the same conditions as the evaluation set, without reusing evaluation images.
Other toolchains follow similar patterns. Apple's Core ML Tools groups its workflows into data-free compression, post-training compression with a small calibration set, and training-time compression, which it says gives the best accuracy and compression trade-offs. NVIDIA's TensorRT supports INT8 and FP8, plus INT4 for weight-only quantisation.
Quantisation-aware training: when post-training isn't enough
If post-training quantisation costs too much accuracy, quantisation-aware training (QAT) is the next step. TensorFlow's QAT guide explains that it emulates inference-time quantisation during training, so the model learns weights that survive conversion to 8-bit. Google advises starting with post-training quantisation because it's easier, and notes that QAT is often better for accuracy.
Google's published ImageNet results give a feel for the effect: MobileNetV2 224 went from 70.77% top-1 accuracy unquantised to 70.01% with 8-bit QAT, and ResNet v1 50 from 76.3% to 76.1%. Those are Google's numbers on a public benchmark, not a forecast for your data. The cost of QAT is a training pipeline: you need the training data, code and compute, not just the finished model file. DiUS used QAT to bring its insurer phone model under 1 MB.
Pruning
Pruning zeroes out weights that contribute little. TensorFlow's pruning guide describes magnitude-based pruning applied gradually during training, and reports up to 6x improvements in model compression with minimal loss of accuracy. Read the fine print: the main gain is a smaller compressed file. Speed gains depend on whether the runtime and hardware can skip the zeros, and TensorFlow lists broader latency support as future work. Structured patterns that hardware can exploit tend to cost more accuracy; in Google's published image results, the structured 2-by-4 pattern lost more accuracy than unstructured sparsity.
Distillation: a small model that learns from a big one
Knowledge distillation trains a compact "student" model to reproduce the behaviour of a large "teacher". The technique takes its name from the 2015 paper Distilling the Knowledge in a Neural Network by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, which built on earlier work showing that the knowledge in an ensemble of models can be compressed into a single model. In practice the student learns from the teacher's full output probabilities as well as the labels, which carry more information than labels alone.
Distillation differs from quantisation in one important way: it changes the architecture. The student can be designed for the target chip from the start, using only operations the accelerator supports. It also costs more, because you are training a new model. A large teacher can also help label unlabelled field data, which is often the scarcer resource.
Vedametric describes its approach as a large teacher training a small student until it keeps the skill you need, followed by pruning and quantising until the model fits a GPU, a phone or a microcontroller. Distil first, quantise second is a common order, because each step removes a different kind of waste.
A worked example of the decision process
Here is how the decisions typically flow on a hypothetical project: a camera above a conveyor must flag foreign objects, and the chosen accelerator runs integer models only. No thresholds are given, because they belong to the project, not to a guide.
- Agree the bar. The site sets a minimum detection rate for foreign objects, a maximum number of false stops per shift, a latency limit per frame and a memory limit. The evaluation set covers night shifts, wet product and dust.
- Measure the baseline. Run the full-precision model on the evaluation set on a workstation. If it misses the bar here, compression won't help; fix the data or the model first.
- Apply full-integer post-training quantisation. This accelerator requires it. Calibrate with samples from every shift and condition.
- Compare slice by slice. If every slice still meets the bar, move to on-device testing. If the loss is concentrated in one slice, check that the calibration set covers it before changing anything else.
- Escalate to QAT if a slice still fails. Fine-tune with quantisation simulated, then rerun the same evaluation.
- Distil if it's too slow or too big. If the quantised model meets the accuracy bar but misses the latency or memory budget, train a smaller student built from the accelerator's supported operations, then quantise the student.
- Verify on the device. Run the evaluation set through the compiled model on the real hardware and compare outputs with the reference. Compilers and runtimes don't always match the desktop simulation.
- Freeze and record. Version the model, calibration set, compiler and runtime together, so the result can be reproduced when the model is retrained.
Questions to ask a compression partner
- Which evaluation set will you use, and which slices will you report?
- Are accuracy and latency measured on the target hardware, or only in simulation?
- Which toolchain and versions, and can we reproduce the build ourselves?
- When the model is retrained next year, is compression an automated pipeline step or a manual job?
- What happens if the accelerator's compiler doesn't support a layer we need?
For the hardware side of these decisions, see the edge AI hardware guide.
FAQ
- Does quantisation always reduce accuracy?
- Usually a little, sometimes not measurably. Google's LiteRT documentation says float16 quantisation causes minimal accuracy loss, while full-integer post-training quantisation can lose accuracy, particularly in smaller networks. The only reliable answer comes from running your own evaluation set, slice by slice, on the quantised model.
- What is the difference between post-training quantisation and quantisation-aware training?
- Post-training quantisation converts a model that is already trained and needs, at most, a small calibration set. Quantisation-aware training simulates quantisation during training or fine-tuning so the model adapts to it. QAT needs the training pipeline and data, but it is often better for accuracy.
- How much calibration data does full-integer quantisation need?
- Google's LiteRT documentation says around 100 to 500 samples from training or validation data is usually enough. Coverage matters more than count: include every lighting, weather and operating condition the device will see.
- Can distillation and quantisation be used together?
- Yes, and they often are. Distillation produces a smaller architecture, then quantisation reduces the number format of that smaller model. Quantising the student with quantisation-aware training is a common final step.