Most edge AI hardware decisions are made in the wrong order. A team picks a board it has heard of, then finds the model doesn't fit, the enclosure overheats or the unit cost breaks the business case. A better order is constraints first, then the model, then the chip, then the board. This guide works through that order for the four hardware families you will meet on Australian projects.
Start with the constraints, not the chip
Before comparing spec sheets, write down the numbers that will rule options out:
- Power source and budget. Mains, a vehicle supply, a solar panel or a battery that has to last for months. This one number eliminates more options than anything else.
- Thermal envelope. A sealed enclosure on a sun-exposed mast behaves very differently from a ventilated cabinet in a control room.
- Latency. How quickly a result must arrive, and whether that is a hard deadline (a machine stops) or a soft one (a dashboard refreshes).
- Model size and inputs. One low-resolution camera, several high-resolution streams, audio, vibration or radar each imply very different memory and compute.
- Volume and service life. Ten units for a trial and ten thousand units over a decade lead to different answers.
The four hardware families
GPU system-on-modules
NVIDIA's Jetson modules put an Arm CPU and an NVIDIA GPU on a compact module that plugs into a carrier board. The range is wide. NVIDIA lists the Jetson Orin NX at up to 157 TOPS, and its Jetson Orin page rates AGX Orin modules from 15 W to 60 W. At the top, the Jetson Thor series quotes up to 2070 FP4 TFLOPS with configurable power from 40 W to 130 W.
These modules suit robots, vehicles and multi-camera systems where models change often and the team wants a full Linux environment and the CUDA toolchain. The price is power and heat. A useful reference point is Sydney's Spiral Blue, whose SE-2 satellite computer pairs a Jetson Orin NX 16GB (up to 157 TOPS at INT8) with a 25 W cap and lower 10 W and 15 W settings. Power modes like these let one hardware design serve several power budgets, at the cost of throughput when throttled.
Check the lifecycle too: NVIDIA lists a 5-year operating lifetime for AGX Orin commercial modules and 10 years for the industrial version.
Dedicated accelerators
An accelerator, often called an NPU, runs the neural network while a host processor handles everything else: camera capture, pre-processing, communications and application logic. Hailo-8 is rated at 26 TOPS with typical power consumption of 2.5 W, comes in M.2 and PCIe form factors, and accepts models from TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX. The newer Hailo-10H quotes 40 TOPS at INT4 (20 TOPS at INT8), also at 2.5 W typical, and adds a direct DDR memory interface aimed at generative models such as LLMs and VLMs. Google's Coral Edge TPU is rated at 4 TOPS, using 0.5 W for each TOPS.
The catch is the toolchain. Many accelerators execute only integer arithmetic, so the model must be fully quantised before it will run. Google's LiteRT documentation notes that integer-only accelerators such as the Coral Edge TPU need full-integer models. Each vendor has its own compiler and list of supported operations, and one unsupported layer can force a fallback to the host CPU or a change to the model. ASX-listed BrainChip takes a different route: Akida is a neuromorphic, event-based design that chip and device makers can build into their own products when power is the hardest limit.
Phones and tablets
A phone already has a camera, screen, GPS, connectivity and a battery, and the person who needs the answer is usually holding it. Google's LiteRT runtime supports Android and iOS with CPU, GPU and NPU acceleration, and lists NPU support for chips from Google Tensor, Intel, Qualcomm, MediaTek and Samsung. On Apple devices, the Core ML Tools documentation notes that weight palettisation typically works best on the Neural Engine.
Two cautions. Android acceleration has shifted: the Neural Networks API (NNAPI) was deprecated in Android 15, and Google recommends that performance-critical workloads migrate to alternatives. And consumer phones aren't built for constant dust, heat and vibration: fine in a field worker's hand, less so bolted to a machine.
Microcontrollers
At the small end, models run on microcontrollers with no operating system at all. LiteRT for Microcontrollers needs a 32-bit platform but no operating system, standard C or C++ libraries or dynamic memory allocation. Google lists supported boards including the Arduino Nano 33 BLE Sense, STM32F746 Discovery kit, Espressif ESP32-DevKitC and Sony Spresense. Only a subset of operations is supported, and there's no on-device training. Microcontrollers suit small, fixed tasks such as wake-word detection, vibration anomaly detection or low-resolution classification on a battery-powered sensor.
Side-by-side comparison
| Family | Example power figures | Strengths | Watch for |
|---|---|---|---|
| GPU module | 7–25 W (Orin Nano) up to 40–130 W (Thor) | Flexible, full Linux, runs most models, multi-camera | Heat, unit cost, battery drain |
| Accelerator plus host | 2.5 W typical for Hailo-8 or Hailo-10H, plus the host | High performance per watt, small form factors | Compiler support, integer-only models, vendor lock-in |
| Phone or tablet | Shares the device battery | Camera, screen and connectivity built in; no new hardware to make | OS and chip fragmentation, ruggedness |
| Microcontroller | Lowest of the four | Unit cost, battery life, tiny size | Kilobytes of memory, limited operations |
Power and heat are one problem
Almost every watt a board draws ends up as heat inside the enclosure. In a sealed box in full sun, processors respond by throttling, and the frame rate quietly drops on the hottest afternoon. Three habits prevent surprises:
- Measure sustained throughput over hours, not a 30-second benchmark.
- Test at the highest ambient temperature the site will see, inside the real enclosure.
- Choose power modes deliberately. A module run at a lower setting may deliver fewer frames per second but keep delivering them all day.
Unit cost versus engineering cost
The bill of materials is what each unit costs to build. The engineering cost is the one-off work to design boards, port the model, write firmware and pass certification. For a pilot of a few units, engineering dominates and an off-the-shelf module is almost always right. Across thousands of units, cents and watts per unit start to outweigh the engineering bill, and a cheaper accelerator on a custom board can repay the extra design work. Component lifetime belongs in the same sum.
Off-the-shelf board or custom board?
Developer kits and partner carrier boards are the fastest way to start. NVIDIA's Orin Nano Super Developer Kit, for example, includes a reference carrier board compatible with all Orin NX and Orin Nano modules, so a prototype can move between modules as requirements firm up.
A custom board starts to make sense when one of these is true:
- The device must fit a particular shape, mount or connector layout.
- The power input is unusual (wide-range vehicle supply, battery, solar) or the budget is tight.
- Shock, vibration, dust or temperature exceed what commercial boards are rated for.
- Volumes are high enough that removing unused parts lowers the unit cost.
- You want to own the design and control the bill of materials.
Several Australian firms design boards around edge AI chips. Vedametric in Sydney designs custom boards built around edge AI accelerators, covering schematics, layout, firmware and power budgets, and trains and compresses the model for the same board. Robotic Systems in Newcastle builds hardware optimised for the power, temperature and vibration conditions of mining. AICRAFT in Adelaide designs tailored onboard computers, board support packages and firmware for defence and remote sensing.
A selection sequence that works
- Write down the task, the accuracy you need and the constraints above.
- Pick a candidate model and run it on a developer kit with representative data.
- Compress it for the target and check accuracy again (see the quantisation and distillation guide).
- Measure sustained speed and power at worst-case temperature.
- Decide between an off-the-shelf carrier and a custom board.
- Plan software updates, spares and component lifetime before ordering in volume.
When the model and the hardware are being designed together, bring both disciplines in from step one. Changing a model architecture to suit an accelerator is usually far cheaper than respinning a board to suit a model, which is why Vedametric specifies the model, hardware and software together as one team.
FAQ
- What does a TOPS figure actually tell me?
- TOPS (trillions of operations per second) is a peak figure for a particular number format, so it only compares like with like. Hailo quotes the Hailo-10H at 40 TOPS for INT4 but 20 TOPS for INT8, and NVIDIA quotes Jetson Thor in FP4 TFLOPS. Real throughput depends on your model, memory bandwidth and which layers the compiler supports, so benchmark your own model on the target.
- Should I choose a Jetson module or a Hailo accelerator?
- A Jetson module is the more flexible option when models change often, you need several camera streams or you want the CUDA toolchain. An accelerator such as Hailo-8 suits a stable model where power and size matter, provided the vendor's compiler supports every layer in your model. Many teams prototype on one and test the other before committing.
- Can large language models run on edge hardware?
- Some can, in reduced form. Hailo positions the Hailo-10H for generative models such as LLMs and VLMs, and Google's LiteRT supports open-weight LLMs running on mobile devices. Expect smaller, quantised models than those served from data centres, and test answer quality on your own tasks.
- Do I need a custom PCB for an edge AI product?
- Not for a pilot. Developer kits and partner carrier boards are faster and cheaper for early units. A custom board pays off when size, power input, ruggedness or unit cost at volume rule out commercial boards.