Edge and embedded deployment: NVIDIA Jetson, mobile (Core ML, TFLite), WebAssembly
Edge and embedded deployment is the practice of optimizing and running machine learning models directly on local hardware devices-like Jetson boards, smartphones, and browsers-bypassing cloud dependency for real-time, offline-capable inference.
It enables ultra-low latency, enhanced data privacy, and reduced operational costs by processing data locally, which is critical for applications in autonomous systems, mobile apps, and on-device AI where cloud connectivity is unreliable or insecure. This skill directly translates to competitive product advantage in IoT, consumer electronics, and automotive industries.
1Careers
1Categories
9.0 Avg Demand
15%
Avg AI Risk
How to Learn Edge and embedded deployment: NVIDIA Jetson, mobile (Core ML, TFLite), WebAssembly
1. Understand model quantization fundamentals: Learn INT8 vs FP16, post-training quantization vs quantization-aware training. 2. Get hands-on with one platform: Start with TensorFlow Lite on a Raspberry Pi or a simple Android app. 3. Master the conversion pipeline: Practice converting a standard PyTorch or TensorFlow model to ONNX, then to the target format (TFLite, Core ML).
Focus on platform-specific optimization. For Jetson, learn to use TensorRT for kernel fusion and precision calibration. For mobile, master Core ML's performance tiers and TFLite's delegate system (GPU, NNAPI). Common mistake: Ignoring the target device's memory constraints during model architecture design, leading to OOM errors during deployment.
Architect cross-platform deployment pipelines. Design a single model (e.g., using PyTorch) that is optimized via different paths for Jetson (TensorRT), iOS (Core ML), Android (TFLite), and web (ONNX Runtime Web). At this level, you evaluate trade-offs between latency, accuracy, and power consumption across a device fleet, and mentor teams on building reproducible MLOps workflows for edge.
Practice Projects
Beginner
Project
Deploy a TFLite Object Detection Model on Android
Scenario
You have a pre-trained SSD-MobileNet model from the TensorFlow Model Zoo. Your goal is to create a basic Android application that uses the phone's camera to detect objects in real-time.
How to Execute
1. Convert the SavedModel to a .tflite file using the TFLite Converter with integer quantization. 2. Set up a new Android Studio project with the TensorFlow Lite Android Support Library. 3. Integrate the TFLite Interpreter into the camera preview activity, feeding each frame and rendering bounding boxes on a Canvas overlay. 4. Profile inference time per frame using Android Studio's profiler.
Intermediate
Project
Optimize a PyTorch Model for NVIDIA Jetson Nano with TensorRT
Scenario
You need to deploy a custom image classification model (e.g., ResNet-18) trained in PyTorch onto a Jetson Nano for a real-time industrial quality inspection system. The model must run at >30 FPS.
How to Execute
1. Export the PyTorch model to ONNX format. 2. Use the JetPack SDK's TensorRT Python API to parse the ONNX file and build an optimized engine, specifying FP16 precision. 3. Write a C++ or Python inference loop that uses the TensorRT engine with CUDA memory management for zero-copy input. 4. Benchmark FPS and accuracy against the PyTorch baseline, then iterate by simplifying layers if needed.
Advanced
Project
Build a Cross-Platform ML Inference Pipeline for a Smart Home Device
Scenario
You are the lead ML engineer for a new smart camera that must support voice command recognition (audio) and person detection (video) offline. The hardware is a Jetson Xavier NX, but the same models need to work on companion mobile apps for configuration.
How to Execute
1. Design modular model architectures (e.g., a small Transformer for audio, MobileNetV3 for vision) using PyTorch. 2. Implement a CI/CD pipeline (GitHub Actions) that, for each commit, converts models to ONNX, then builds and tests: a) TensorRT engines for Jetson, b) Core ML models for iOS, c) TFLite models for Android. 3. Develop a unified C++ inference library (using ONNX Runtime) that abstracts platform-specific calls, compiled via CMake for each target. 4. Conduct power and thermal stress testing on the Jetson device under sustained load.
Use PyTorch/TensorFlow for model training. ONNX is the critical interoperability format for moving models between training frameworks and deployment targets (TensorRT, Core ML, TFLite, ONNX Runtime Web).
Edge & Mobile SDKs
NVIDIA JetPack SDK (TensorRT, CUDA)Apple Core ML ToolsTensorFlow Lite (with XNNPACK, GPU/NNAPI delegates)ONNX Runtime Mobile/Web
JetPack provides the full stack for NVIDIA Jetson devices. Core ML Tools optimize models for Apple silicon. TFLite is the standard for Android and microcontrollers. ONNX Runtime provides a unified runtime across mobile, desktop, and web (via WebAssembly).
Profiling & Debugging
NVIDIA Nsight SystemsAndroid Studio ProfilerInstruments (Xcode)TensorBoard
Nsight Systems is essential for profiling GPU/CUDA workloads on Jetson. Mobile profilers track CPU, GPU, and memory usage. TensorBoard helps visualize model graphs and quantization effects.
Web Deployment
ONNX Runtime WebTensorFlow.jsWebAssembly (Wasm)
ONNX Runtime Web and TensorFlow.js allow models to run in browsers using WebGL, WebGPU, or WebAssembly backends, enabling private, no-server AI applications.
Interview Questions
Answer Strategy
The interviewer is testing for practical, hands-on knowledge of the conversion pipeline and resource constraints. Structure your answer linearly: 1) Export to ONNX, 2) Convert to TFLite, 3) Apply post-training quantization (specify dynamic range or full integer for CPU), 4) Test on representative hardware using the TFLite benchmark tool, 5) Discuss fallback strategies if latency is too high (e.g., model pruning, using a smaller backbone). Sample answer: 'First, I'd export the model to ONNX using torch.onnx.export, ensuring opset version compatibility. Then, using the TFLite Converter, I'd convert it and apply full integer quantization with a representative dataset to minimize memory footprint. I'd rigorously profile on the target Android device, focusing on both latency and peak memory usage. If needed, I'd explore architecture modifications or TFLite's GPU delegate for acceleration.'
Answer Strategy
This tests deep debugging and optimization skills. Use the STAR method (Situation, Task, Action, Result). Focus on systematic analysis: profiling with Nsight, checking for precision-sensitive layers, and kernel timing. Sample answer: 'Situation: Our object detector showed a 15% accuracy drop with FP16 TensorRT. Task: Identify and resolve the precision loss without sacrificing performance. Action: I used Nsight Systems to trace the execution, isolating a custom activation function that wasn't being fused and had high numerical instability in FP16. I rewrote it as a TensorRT plugin with mixed-precision logic. Result: Accuracy recovered to baseline with only a 2% latency increase from FP32, meeting our real-time requirements.'
Careers That Require Edge and embedded deployment: NVIDIA Jetson, mobile (Core ML, TFLite), WebAssembly