On-device inference earns its place when the device must act before a remote service can reliably answer. The placement question starts with the decision deadline, the available evidence and the consequence of a wrong answer. Model size comes later. A small classifier that fits in memory can still be the wrong component if its inputs arrive too late or its failure mode is undefined.

Not every decision can wait for the network

A vibration monitor may classify a sample window as ordinary or unusual. A battery sensor may score whether an observation deserves an immediate transmission. A gateway may choose which readings need further analysis. These are bounded decisions with identifiable inputs and outputs; none requires a conversational explanation before the next step can begin.

Separate that work from diagnosing an unfamiliar fault across maintenance records and several machines. The latter can require broader context, more computation and human judgment. Giving both tasks the same remote inference dependency ties a small, frequent decision to infrastructure designed for a different problem. First describe the output contract: a class, a score, a routing choice or an abstention.

Also distinguish inference from control. A model can propose an anomaly class while a deterministic controller retains authority over motor shutdown. That boundary lets engineers constrain actuation, test timing independently and replace the model without redesigning the safety policy. A local prediction is evidence for a decision, not automatic permission to operate equipment.

Four reasons to decide locally

Latency is a deadline problem. Consider an illustrative design exercise: allocate 5–20 milliseconds to local preprocessing and inference, versus 50–500 milliseconds for a remote request and response. These are hypothetical planning ranges, not measured capabilities or network guarantees. A loaded device can miss the first budget; reconnects and retries can push the second far beyond its range. Measure both on the intended hardware and network.

The comparison must include sensor window duration, scheduling delay, preprocessing and actuator response. If a vibration window takes longer to collect than the decision deadline, moving inference onto the processor cannot repair the design. Conversely, when a local warning must appear before a network exchange could plausibly finish, a remote service cannot be the only path.

Offline operation is a separate requirement. A remote pump monitor can continue classifying locally while it stores a bounded queue of summaries. A device dependent on cloud inference needs an explicit outage policy: expire the request, reject late answers, enter a defined degraded mode and apply the local rule or request operator intervention. Repeating the last cloud decision indefinitely can turn stale information into a new command.

Privacy can favor extracting features near the sensor. An occupancy device might transmit counts rather than recordings. That reduces what leaves the enclosure, but features, logs and diagnostic samples can still reveal sensitive patterns. Review what is retained, who can access it and how long it survives; local execution alone does not establish privacy.

Fleet economics also changes placement. A frequently sampled sensor may save radio activity by transmitting only selected events. Compare compute energy, transmission energy, service charges and maintenance effort per useful decision. Include retries and diagnostic traffic. A cheaper prediction can become an expensive system if it requires a larger battery, a different processor or frequent field support.

Where the model can run

A microcontroller suits small models with predictable execution and modest inputs. TinyML deployment requires budgeting flash for weights and code, and RAM for activations, tensor arenas, stacks, communication buffers and the application itself. A model file fitting in flash says little about peak working memory. Measure the complete workload with connectivity enabled rather than testing an isolated inference function.

Int8 quantization represents supported values with eight-bit integers plus scaling information. It can reduce weight storage relative to wider representations and enable integer kernels, but its effect on accuracy and execution depends on the model and runtime. Activations and temporary buffers still consume RAM. Validate quantized behavior on representative field inputs, including rare classes and values near decision thresholds.

A system-on-chip with an NPU can support workloads that exceed an MCU's practical envelope. The accelerator brings its own operator support, conversion pipeline and memory movement costs. Unsupported operations may fall back to another processor. Evaluate the compiled graph and end-to-end power draw, not just the accelerator's advertised arithmetic capability. More capable silicon also affects the bill of materials, thermal design and software maintenance.

An edge gateway shares computation across several devices and can combine their observations. It can preserve operation during an upstream outage while keeping endpoint firmware small. Its costs include another managed computer, installation effort and a local network dependency. If endpoints cannot reach it, they still need a defined behavior. Concentrating decisions on a gateway also concentrates failure impact.

These choices belong in the product architecture before board selection. Moving from a development board to a product includes provisioning, power and serviceability, all of which can outweigh the convenience of a model demonstration.

A model is firmware now

Ship a model as a versioned executable dependency. Identify the weights, runtime, input normalization, feature ordering, label mapping and output policy together. A change in units or sensor calibration can invalidate a previously successful model without altering its architecture. Reject incompatible combinations explicitly instead of hoping that equal tensor dimensions imply compatibility.

Treat delivery as an update transaction. Download and verify the candidate without replacing the active artifact prematurely. Preserve a known working version, switch only after integrity and compatibility checks, and make interrupted activation recoverable. Whether model and firmware occupy one image or separate storage, the activation record must identify a coherent combination.

Stage deployment through representative cohorts before broadening it. Watch boot failures, inference errors, memory pressure and decision distributions alongside application outcomes. Define rollback triggers in advance. ESP32 OTA updates in production covers the surrounding firmware lifecycle; model delivery needs the same attention to recovery and interrupted operations.

Rollback also includes persistent data. If a new feature pipeline rewrites calibration or cached state, the older model must still understand what remains. Test the reverse transition, and separate reversible migrations from changes that need an explicit recovery procedure. A retained binary is insufficient when its input contract has disappeared.

Observing decisions in the field

Record the model version, input-pipeline version, selected class, score and reason for abstention. Add timing and reset context where useful. A confidence distribution shifting toward the rejection threshold is a concrete investigation signal. So is a change in class balance among comparable devices. Neither proves model degradation: a new environment or sensor fault can produce similar symptoms.

Compare cohorts by hardware revision, installation conditions and software version. Aggregate counters can reveal changes without uploading every input. Keep diagnostic queues bounded and sample deliberately; otherwise, a burst of unusual readings can exhaust storage or bandwidth precisely when investigation matters most. Decide how timestamps behave when the device loses its clock source.

Obtaining representative ground truth remains harder than collecting predictions. Review selected events with operators or independent measurements. High confidence does not establish correctness, especially after environmental change. MQTT versus HTTP for connected devices helps frame transport choices, but transport delivery is distinct from evidence that a decision was useful.

When not to infer on the device

Prefer a gateway when the useful evidence spans neighboring devices or the endpoint cannot afford the working memory and power. Prefer cloud analysis when decisions can wait, depend on large shared datasets and benefit from centralized maintenance. Local inference is a poor choice if the model's operating conditions cannot be validated or updated within the product's support model.

Sometimes a threshold, state machine or physical measurement solves the problem more transparently. Evaluate that baseline before accepting model maintenance obligations. What makes an IoT device production ready includes recovery and diagnostics; a successful prediction demo does not replace either.

Before you choose a placement

Write down the deadline, outage behavior, memory budget and recovery contract before choosing placement. The next part of Edge AI Systems examines what bounded AI outputs can honestly promise when a decision moves a physical device.