Deploy a PyTorch Model to Qualcomm Device

Quantize and deploy a MobileNetV2 model to a Snapdragon device via AI Hub.

Introduction

30 Second Summary

Every AI model running on your phone has been through a compression pipeline. Without optimization, even a simple image classifier would drain your battery and take seconds to respond.

In this project, you will take a pre-trained PyTorch model and optimize it for a real mobile device using Qualcomm AI Hub. You will quantize the model from FP32 to INT8, compile it for a Samsung Galaxy S24, and run inference on cloud-hosted hardware.

What You'll Build

You will build a complete edge AI optimization pipeline that exports, quantizes, compiles, profiles, and runs a MobileNetV2 model on a real Snapdragon device through Qualcomm AI Hub.

By the end of this project, you'll have:

  • A quantized INT8 model compiled for a Samsung Galaxy S24, ready for on-device deployment.
  • On-device profiling results showing inference latency, memory usage, and NPU compute unit mapping.
  • Validated inference predictions comparing on-device INT8 output against the original PyTorch FP32 model.
  • Secret Mission: Compare TFLite portable format vs QNN Context Binary NPU-optimized format for performance trade-offs.

Are there any prerequisites?

This project assumes basic Python and PyTorch familiarity. You should be comfortable running Python scripts and understand what a pre-trained model is.

Before We Start

Before you can optimize a model for a real mobile device, you need the right tools in place. Qualcomm AI Hub is a cloud-based platform that provisions real Snapdragon devices for compiling, profiling, and running inference on your models.

In this step, you will create an isolated Python environment, install the Qualcomm AI Hub SDK, and connect to the platform so you can submit jobs to cloud-hosted devices like the Samsung Galaxy S24.

In this step, get ready to:
  • Create a conda environment with Python 3.10 and install project dependencies.
  • Create a free Qualcomm AI Hub account and configure your API token.
  • Verify your setup by listing available cloud-hosted devices.
Create a conda environment and install dependencies

Conda lets you create isolated Python environments so your project's packages don't conflict with anything else on your machine. You will create a fresh environment pinned to Python 3.10, which is the minimum version required by qai-hub.

  • Press Cmd+Space to open Spotlight Search.
  • Type Terminal and press Enter to open it.
  • Check if conda is already installed by running this command:
conda --version

✔️ I see a version number

Conda is ready to go. Continue to the next instruction below.

ⓧ Command not found

You need to install Miniconda first. Miniconda is a lightweight installer that gives you the conda command.

  • Install Miniconda using Homebrew by running this command:
brew install --cask miniconda
  • After the install finishes, initialize conda for your shell by running:
conda init "$(basename "${SHELL}")"
  • Close and reopen your Terminal window for the changes to take effect.
  • Confirm conda is working by running conda --version again. You should now see a version number.

Don't have Homebrew either?

If brew is also not found, visit https://brew.sh and follow the one-line install command. Then return here and run brew install --cask miniconda.

help me install Homebrew and Miniconda on my Mac

  • Create a new conda environment named qai_hub with Python 3.10 by running this command:
conda create python=3.10 -n qai_hub
  • When prompted to proceed, type y and press Enter.
  • Activate the new environment by running:
conda activate qai_hub

Your terminal prompt should now show (qai_hub) at the beginning, confirming you are inside the new environment.

  • Install the project's Python packages by running:
pip3 install "qai-hub[torch]" torchvision Pillow numpy requests

What are these packages?

  • qai-hub[torch] is the Qualcomm AI Hub SDK with PyTorch integration for submitting compile, quantize, profile, and inference jobs.
  • torchvision provides pre-trained models like MobileNetV2.
  • Pillow and numpy handle image loading and array processing.
  • requests downloads sample images and class labels from Qualcomm's public assets.

The install may take a few minutes as PyTorch is a large package. Wait until you see the terminal prompt return with no errors.

Seeing dependency conflicts?

Make sure your (qai_hub) environment is active. If you accidentally installed into your base environment, deactivate it with conda deactivate, then run conda activate qai_hub and try the pip install again.

help me resolve dependency conflicts when installing qai-hub packages

Create a Qualcomm AI Hub account and configure your API token

Qualcomm AI Hub is free to use. You need a Qualcomm ID to authenticate and submit jobs to their cloud-hosted devices.

  • Open your browser and navigate to myaccount.qualcomm.com.
  • Click Sign Up and create a free Qualcomm ID using your email address.
  • Once your account is created, navigate to workbench.aihub.qualcomm.com.
  • Sign in with your new Qualcomm ID.
  • Click Account in the navigation menu.
  • Select Settings, then API Token.
  • Copy the API token displayed on this page.

Why do I need an API token?

The API token authenticates your local CLI with Qualcomm's cloud infrastructure. Every compile, profile, and inference job you submit from your terminal is routed through this token to real Snapdragon hardware in Qualcomm's data centers.

  • Back in your Terminal (with the (qai_hub) environment active), configure the CLI with your token by running:
qai-hub configure --api_token your-api-token-here
  • Replace your-api-token-here with the token you just copied from the Workbench settings page.

Getting a permission or authentication error?

Double-check that you copied the full token with no trailing spaces. Make sure you are signed into workbench.aihub.qualcomm.com (not a different Qualcomm portal) when copying the token.

help me troubleshoot my Qualcomm AI Hub API token configuration

Verify your setup by listing available devices

The final check confirms that your token works and that you can see the cloud-hosted Snapdragon devices available for running jobs.

  • List the available devices by running this command:
qai-hub list-devices

You should see a table of available devices including Samsung Galaxy S24, S25, and other Snapdragon-powered hardware.

What are these devices?

These are real physical phones and laptops provisioned in Qualcomm's cloud. When you submit a profile or inference job later in this project, your model actually runs on one of these devices. You are not using a simulator.

Empty list or connection error?

Run qai-hub configure --api_token your-api-token-here again and make sure the token is correct. Also check that your internet connection is active, since this command reaches Qualcomm's API servers.

help me fix my qai-hub list-devices command not working

Your environment is ready and authenticated. Next up, you will load a pre-trained PyTorch model and export it for optimization on Qualcomm hardware.

Load and Export the PyTorch Model

Your environment is configured and your terminal can talk to Qualcomm's cloud-hosted devices. Now it's time to work with the actual model you'll deploy.

In this step, you'll load a pre-trained MobileNetV2 image classifier from torchvision, export its computation graph, and submit it to Qualcomm AI Hub for compilation into an optimized ONNX model.

In this step, get ready to:
  • Load MobileNetV2 with pre-trained ImageNet weights.
  • Export the model graph using torch.export.
  • Submit a compile job to AI Hub targeting ONNX format.
Load MobileNetV2 and prepare for export

MobileNetV2 is a lightweight image classification model built for mobile and edge devices. Loading it with pre-trained weights gives you a model that already classifies 1000 ImageNet categories.

  • Create a new file called edge_ai_pipeline.py in your project folder by running:
touch edge_ai_pipeline.py
  • Open edge_ai_pipeline.py in your IDE and add the following imports and model-loading code:
import os

import numpy as np
import requests
import torch
import torchvision
from PIL import Image

import qai_hub as hub

# --- Step 2: Load and Export the PyTorch Model ---

# 1. Load pre-trained MobileNetV2 from torchvision
torch_model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()

What does this code do?

  • The imports bring in all the libraries you installed in Step 1.
  • torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1") loads MobileNetV2 with weights pre-trained on 1.2 million ImageNet images.
  • .eval() switches the model from training mode to inference mode, disabling dropout and batch normalization updates.
  • Save edge_ai_pipeline.py.
  • Confirm the file exists by running:
ls edge_ai_pipeline.py

You should see edge_ai_pipeline.py listed in the output.

File not found?

  • Make sure your terminal is in the same directory where you created the file.
  • If you created the file through your IDE, check that it saved to the expected location.

Help me find where my file was created.

Export the model and compile to ONNX on AI Hub

torch.export captures the model's computation graph as a flat, serializable representation. AI Hub then takes this exported graph and compiles it into an optimized ONNX model with hardware-aware optimizations applied.

  • In edge_ai_pipeline.py, add the following code below the model loading line to export the graph:
# 2. Export the model using torch.export
input_shape = (1, 3, 224, 224)
with torch.no_grad():
    pt2_model = torch.export.export(torch_model, (torch.rand(input_shape),))

What does the input shape mean?

The tuple (1, 3, 224, 224) describes a single image: batch size 1, 3 color channels (RGB), and 224x224 pixels.

torch.export.export() traces the model's operations using a random tensor of this shape. The result is a portable graph that AI Hub can optimize for specific hardware.

  • Below the export code, add the compile job submission:
# 3. Connect to AI Hub and compile to ONNX
client = hub.Client()
device = hub.Device("Samsung Galaxy S24 (Family)")

compile_onnx_job = client.submit_compile_job(
    model=pt2_model,
    device=device,
    input_specs=dict(image_tensor=input_shape),
    options="--target_runtime onnx",
)
assert isinstance(compile_onnx_job, hub.CompileJob)
print(f"Compile to ONNX job URL: {compile_onnx_job.url}")

What does this code do?

  • hub.Client() connects to Qualcomm AI Hub using the API token you configured in Step 1.
  • hub.Device("Samsung Galaxy S24 (Family)") sets your target device to the Snapdragon 8 Gen 3 chipset inside the S24 series.
  • submit_compile_job() sends the exported model to AI Hub's cloud servers for compilation.
  • The --target_runtime onnx option tells AI Hub to produce an ONNX representation. You'll quantize this before compiling to a device-ready format.
  • Save edge_ai_pipeline.py.

You won't see output yet. The next substep adds the retrieval code and runs the script.

Verify the compile job and retrieve the ONNX model

When you call get_target_model(), AI Hub waits for the compile job to finish and returns the resulting model object. This is the unquantized ONNX model you'll feed into the quantization step next.

  • In edge_ai_pipeline.py, add the following code at the bottom of the file:
unquantized_onnx_model = compile_onnx_job.get_target_model()
assert isinstance(unquantized_onnx_model, hub.Model)
print("ONNX model ready for quantization.")

What does this code do?

  • get_target_model() blocks until the cloud compile job completes, then returns the compiled model as a hub.Model object.
  • The assert verifies the returned object is the correct type, catching any job failures early.
  • Save edge_ai_pipeline.py.
  • Run the script by executing:
python edge_ai_pipeline.py

The compile job takes 1-2 minutes to complete on AI Hub's servers. You should see output similar to:

Compile to ONNX job URL: https://workbench.aihub.qualcomm.com/jobs/...
ONNX model ready for quantization.

What should I see?

The first line prints a URL you can open in your browser to view the compile job details on the AI Hub Workbench dashboard. The second line confirms the ONNX model was retrieved successfully.

Seeing an authentication error?

  • Make sure your conda environment qai_hub is active. Run conda activate qai_hub if you opened a new terminal.
  • Verify your API token is still configured by running qai-hub list-devices. If it fails, reconfigure with qai-hub configure --api_token YOUR_API_TOKEN.
  • Check that you copied the token from Account > Settings > API Token in the Workbench, not from another page.

Help me debug my AI Hub authentication error.

✔️ Awesome, I've got everything!

Great. Double check you've saved edge_ai_pipeline.py and your compile job URL printed successfully.

ⓧ I'd like to double check the full code

import os

import numpy as np
import requests
import torch
import torchvision
from PIL import Image

import qai_hub as hub

# --- Step 2: Load and Export the PyTorch Model ---

# 1. Load pre-trained MobileNetV2 from torchvision
torch_model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()

# 2. Export the model using torch.export
input_shape = (1, 3, 224, 224)
with torch.no_grad():
    pt2_model = torch.export.export(torch_model, (torch.rand(input_shape),))

# 3. Connect to AI Hub and compile to ONNX
client = hub.Client()
device = hub.Device("Samsung Galaxy S24 (Family)")

compile_onnx_job = client.submit_compile_job(
    model=pt2_model,
    device=device,
    input_specs=dict(image_tensor=input_shape),
    options="--target_runtime onnx",
)
assert isinstance(compile_onnx_job, hub.CompileJob)
print(f"Compile to ONNX job URL: {compile_onnx_job.url}")

unquantized_onnx_model = compile_onnx_job.get_target_model()
assert isinstance(unquantized_onnx_model, hub.Model)
print("ONNX model ready for quantization.")

Your MobileNetV2 model is now an optimized ONNX representation sitting on AI Hub's servers. Next up, you'll quantize it from FP32 to INT8 to unlock the full speed of the Snapdragon's Hexagon NPU.

Quantize the Model with Calibration Data

Your ONNX model is ready, but it stores every weight and activation as a 32-bit floating point number. Mobile NPU hardware like the Snapdragon Hexagon Tensor Processor performs best with quantized operations.

In this step, you will quantize the model from FP32 to INT8 using Qualcomm AI Hub's AIMET-powered quantization pipeline. This can deliver up to a 3x improvement in performance.

In this step, get ready to:
  • Prepare calibration data for the quantization process.
  • Submit a quantization job to convert the model from FP32 to INT8.
  • Retrieve the quantized model ready for device compilation.
Prepare calibration data for quantization

Quantization maps FP32 values to a smaller INT8 range. The quantizer needs sample data to understand how values are distributed across the model's layers. This sample data is called calibration data.

  • In edge_ai_pipeline.py, find the line that says print("ONNX model ready for quantization."), then copy the following code below it:
# --- Step 3: Quantize the Model with Calibration Data ---

# 4. Prepare calibration data (random sample for quick latency benchmark)
calibration_data = dict(
    image_tensor=[np.random.randn(1, 3, 224, 224).astype(np.float32)]
)

What does this code do?

calibration_data is a dictionary with a key image_tensor containing a list of one random NumPy array shaped like a single image input: batch 1, 3 color channels, 224x224 pixels.

For this project, random data is enough to measure on-device latency. While the resulting quantized model's accuracy will be poor, it will have the same on-device latency as an accurate model. In production, you would use representative real images.

  • Save edge_ai_pipeline.py.

You won't see a change yet. The next chunk uses this calibration data to submit the quantization job.

Submit the quantization job

With calibration data ready, you can submit a quantization job to AI Hub. The platform performs post-training quantization (PTQ), converting FP32 weights and activations to INT8 using AIMET (AI Model Efficiency Toolkit) under the hood.

  • Add the following code directly below the calibration_data block you just added:
# 5. Submit quantization job (FP32 -> INT8)
quantize_job = client.submit_quantize_job(
    model=unquantized_onnx_model,
    calibration_data=calibration_data,
    weights_dtype=hub.QuantizeDtype.INT8,
    activations_dtype=hub.QuantizeDtype.INT8,
)
assert isinstance(quantize_job, hub.QuantizeJob)
print(f"Quantize job URL: {quantize_job.url}")

What does this code do?

  • client.submit_quantize_job() sends the ONNX model and calibration data to AI Hub for quantization.
  • hub.QuantizeDtype.INT8 for both weights_dtype and activations_dtype means every weight and every intermediate activation gets quantized to 8-bit integers.
  • The output is a quantized ONNX model in QDQ (quantize-dequantize) "fake quantization" format, ready for compilation to a device-specific runtime.
  • Save edge_ai_pipeline.py.

You won't see a change yet. The next chunk retrieves the result of this job.

Retrieve the quantized model
  • Add this code directly below the print(f"Quantize job URL: {quantize_job.url}") line:
quantized_onnx_model = quantize_job.get_target_model()
assert isinstance(quantized_onnx_model, hub.Model)
print("Model quantized to INT8 successfully.")

What does this code do?

get_target_model() waits for the quantization job to finish, then returns the resulting INT8 model as a hub.Model object. This quantized model is what you will compile for the Samsung Galaxy S24 in the next step.

  • Save edge_ai_pipeline.py.
  • Run the script in your terminal by running:
python edge_ai_pipeline.py

You should see output that includes a quantize job URL and the message Model quantized to INT8 successfully. at the end. The quantization job may take a minute or two to complete.

Seeing an error or the job taking too long?

  • If you see an authentication error, re-run qai-hub configure --api_token with a fresh token from Account > Settings > API Token in the Workbench.
  • If the assertion fails on hub.QuantizeJob, check that you are passing the unquantized_onnx_model variable (not pt2_model or torch_model) as the model argument.
  • The job can take 1-3 minutes. The get_target_model() call blocks until the job finishes.

Help me debug my quantization job error

Why does the Hexagon Tensor Processor need INT8?

The Snapdragon Hexagon Tensor Processor performs best with quantized operations. By converting weights and activations from 32-bit floats to 8-bit integers, you reduce memory bandwidth by 4x and unlock the NPU's most efficient compute paths.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

import os

import numpy as np
import requests
import torch
import torchvision
from PIL import Image

import qai_hub as hub

# --- Step 2: Load and Export the PyTorch Model ---

# 1. Load pre-trained MobileNetV2 from torchvision
torch_model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()

# 2. Export the model using torch.export
input_shape = (1, 3, 224, 224)
with torch.no_grad():
    pt2_model = torch.export.export(torch_model, (torch.rand(input_shape),))

# 3. Connect to AI Hub and compile to ONNX
client = hub.Client()
device = hub.Device("Samsung Galaxy S24 (Family)")

compile_onnx_job = client.submit_compile_job(
    model=pt2_model,
    device=device,
    input_specs=dict(image_tensor=input_shape),
    options="--target_runtime onnx",
)
assert isinstance(compile_onnx_job, hub.CompileJob)
print(f"Compile to ONNX job URL: {compile_onnx_job.url}")

unquantized_onnx_model = compile_onnx_job.get_target_model()
assert isinstance(unquantized_onnx_model, hub.Model)
print("ONNX model ready for quantization.")

# --- Step 3: Quantize the Model with Calibration Data ---

# 4. Prepare calibration data (random sample for quick latency benchmark)
calibration_data = dict(
    image_tensor=[np.random.randn(1, 3, 224, 224).astype(np.float32)]
)

# 5. Submit quantization job (FP32 -> INT8)
quantize_job = client.submit_quantize_job(
    model=unquantized_onnx_model,
    calibration_data=calibration_data,
    weights_dtype=hub.QuantizeDtype.INT8,
    activations_dtype=hub.QuantizeDtype.INT8,
)
assert isinstance(quantize_job, hub.QuantizeJob)
print(f"Quantize job URL: {quantize_job.url}")

quantized_onnx_model = quantize_job.get_target_model()
assert isinstance(quantized_onnx_model, hub.Model)
print("Model quantized to INT8 successfully.")

Your model is now quantized from FP32 to INT8. Next up, you will compile this quantized model for the Samsung Galaxy S24 and profile its on-device performance.

Compile for Target Device and Profile On-Device Performance

Your model is quantized to INT8, slashing its memory footprint and enabling faster computation. The next challenge is converting this quantized model into a format that actually runs on the device's hardware accelerator.

In this step, you will compile the quantized model into a TFLite binary optimized for the Samsung Galaxy S24's NPU. Then you will profile it on a real cloud-hosted device to measure inference latency, memory usage, and compute unit mapping.

In this step, get ready to:
  • Compile the quantized INT8 model to TFLite format for Samsung Galaxy S24.
  • Profile the compiled model on a real cloud-hosted device.
  • Download and examine on-device performance metrics.
Compile the quantized model to TFLite for the Samsung Galaxy S24

The quantized ONNX model cannot run directly on a phone. It needs to be compiled into a device-specific runtime format. The TFLite runtime is a mobile-optimized inference engine that maps INT8 operations to the Hexagon Tensor Processor inside Snapdragon chips.

  • In edge_ai_pipeline.py, add the following code below the line that says print("Model quantized to INT8 successfully."):
# --- Step 4: Compile for Target Device and Profile ---

# 6. Compile quantized model to TFLite
compile_tflite_job = client.submit_compile_job(
    model=quantized_onnx_model,
    device=device,
    options="--target_runtime tflite --quantize_io",
)
assert isinstance(compile_tflite_job, hub.CompileJob)
print(f"Compile to TFLite job URL: {compile_tflite_job.url}")

target_model = compile_tflite_job.get_target_model()
assert isinstance(target_model, hub.Model)
print("Quantized TFLite model compiled successfully.")

What does this code do?

client.submit_compile_job() sends your quantized ONNX model to Qualcomm's cloud compiler. The --target_runtime tflite flag tells it to produce a TFLite binary.

The --quantize_io flag ensures the model's inputs and outputs use integer types instead of floating point. This maximizes NPU utilization because the Snapdragon Hexagon Tensor Processor performs best with quantized operations.

get_target_model() is a blocking call that waits for the compile job to finish, then returns the compiled model as a hub.Model object ready for profiling or inference.

  • Save edge_ai_pipeline.py.
  • You won't run this file yet. The next chunk adds profiling, and you will run everything together.

Seeing a compile error?

If the job fails with a model format error, confirm that quantized_onnx_model is a hub.Model from the previous step's quantize job. Re-run your script from the top to ensure all variables are populated.

If you see an authentication error, run qai-hub configure --api_token again to refresh your token.

help me debug a compile job failure on Qualcomm AI Hub

Profile the compiled model on a real cloud-hosted device

Now that you have a deployment-ready TFLite model, you need to know how it actually performs on hardware. A profile job runs your model on a physical Samsung Galaxy S24 provisioned in Qualcomm's cloud and measures real-world metrics.

  • Add the following code directly below the line that says print("Quantized TFLite model compiled successfully."):
# 7. Profile on cloud-hosted device
profile_job = client.submit_profile_job(
    model=target_model,
    device=device,
)
assert isinstance(profile_job, hub.ProfileJob)
print(f"Profile job URL: {profile_job.url}")

What does this code do?

submit_profile_job() uploads your compiled TFLite model to a real Samsung Galaxy S24 in Qualcomm's device cloud. It runs inference 100 times in a tight loop to produce stable, repeatable latency measurements.

The profile job measures inference latency, model load time, and peak memory usage. It also maps which compute units (NPU, GPU, or CPU) execute each layer.

  • Save edge_ai_pipeline.py.
  • You won't see a change yet. The next chunk downloads the profile results.
Download and interpret profiling results

The profile job collects detailed hardware metrics while running on the physical device. Now you will download those results and print them.

  • Add the following code directly below the line that says print(f"Profile job URL: {profile_job.url}"):
profile = profile_job.download_profile()
print("\n--- On-Device Profile Results ---")
print(profile)

What does this code do?

download_profile() is a blocking call that waits until the profile job completes on the device, then returns a dictionary of performance metrics.

Key fields in the output include:

  • estimated_inference_time (in microseconds). This is the minimum observed latency across multiple iterations.
  • estimated_inference_peak_memory (in bytes). The peak memory used during inference.
  • first_load_time and warm_load_time showing how long the model takes to load initially versus on subsequent runs.
  • Save edge_ai_pipeline.py.
  • Run your script by executing:
python edge_ai_pipeline.py
  • The script will take a few minutes to complete as it submits jobs to Qualcomm's cloud and waits for each one to finish.
  • You should see compile job URLs, quantize job URLs, and finally the profile results dictionary printed in your terminal.
  • Look for estimated_inference_time in the output. This shows your model's inference latency in microseconds on the Samsung Galaxy S24's NPU.

What should I see in the profile results?

The output is a Python dictionary containing timing values in microseconds and memory values in bytes. You will see fields like estimated_inference_time, first_load_time, warm_load_time, and memory peak ranges.

The profile confirms that your INT8 quantized operations are mapped to the Hexagon NPU for maximum throughput. You can also visit the profile job URL printed in your terminal to see a visual breakdown of per-layer compute unit assignments in the Workbench UI.

Profile job taking too long or failing?

Profile jobs typically take 1-3 minutes. If the job times out, check that your internet connection is stable. The download_profile() call blocks until the job finishes, so a slow connection can make it appear stuck.

If the profile job fails, open the job URL in your browser to check the error details in the Workbench UI. Common issues include using a device that is temporarily unavailable.

help me debug a profile job failure on Qualcomm AI Hub Workbench

✔️ Awesome, I've got everything!

Great. Double check you have saved edge_ai_pipeline.py and confirmed the profile results printed in your terminal.

ⓧ I'd like to double check the full code

import os

import numpy as np
import requests
import torch
import torchvision
from PIL import Image

import qai_hub as hub

# --- Step 2: Load and Export the PyTorch Model ---

# 1. Load pre-trained MobileNetV2 from torchvision
torch_model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()

# 2. Export the model using torch.export
input_shape = (1, 3, 224, 224)
with torch.no_grad():
    pt2_model = torch.export.export(torch_model, (torch.rand(input_shape),))

# 3. Connect to AI Hub and compile to ONNX
client = hub.Client()
device = hub.Device("Samsung Galaxy S24 (Family)")

compile_onnx_job = client.submit_compile_job(
    model=pt2_model,
    device=device,
    input_specs=dict(image_tensor=input_shape),
    options="--target_runtime onnx",
)
assert isinstance(compile_onnx_job, hub.CompileJob)
print(f"Compile to ONNX job URL: {compile_onnx_job.url}")

unquantized_onnx_model = compile_onnx_job.get_target_model()
assert isinstance(unquantized_onnx_model, hub.Model)
print("ONNX model ready for quantization.")

# --- Step 3: Quantize the Model with Calibration Data ---

# 4. Prepare calibration data (random sample for quick latency benchmark)
calibration_data = dict(
    image_tensor=[np.random.randn(1, 3, 224, 224).astype(np.float32)]
)

# 5. Submit quantization job (FP32 -> INT8)
quantize_job = client.submit_quantize_job(
    model=unquantized_onnx_model,
    calibration_data=calibration_data,
    weights_dtype=hub.QuantizeDtype.INT8,
    activations_dtype=hub.QuantizeDtype.INT8,
)
assert isinstance(quantize_job, hub.QuantizeJob)
print(f"Quantize job URL: {quantize_job.url}")

quantized_onnx_model = quantize_job.get_target_model()
assert isinstance(quantized_onnx_model, hub.Model)
print("Model quantized to INT8 successfully.")

# --- Step 4: Compile for Target Device and Profile ---

# 6. Compile quantized model to TFLite
compile_tflite_job = client.submit_compile_job(
    model=quantized_onnx_model,
    device=device,
    options="--target_runtime tflite --quantize_io",
)
assert isinstance(compile_tflite_job, hub.CompileJob)
print(f"Compile to TFLite job URL: {compile_tflite_job.url}")

target_model = compile_tflite_job.get_target_model()
assert isinstance(target_model, hub.Model)
print("Quantized TFLite model compiled successfully.")

# 7. Profile on cloud-hosted device
profile_job = client.submit_profile_job(
    model=target_model,
    device=device,
)
assert isinstance(profile_job, hub.ProfileJob)
print(f"Profile job URL: {profile_job.url}")

profile = profile_job.download_profile()
print("\n--- On-Device Profile Results ---")
print(profile)

Your model is compiled and profiled on real hardware. Next up, you will run actual inference on the cloud-hosted Samsung Galaxy S24 and validate that the INT8 predictions match the original PyTorch output.

Run On-Device Inference and Validate Results

Your quantized model is compiled and profiled. You know it runs fast on the NPU. But does it still produce correct predictions after all that optimization?

In this step, you will send a real image to the cloud-hosted Samsung Galaxy S24, run inference with your INT8 model, and compare the results against the original FP32 PyTorch model. This is the final validation that your optimization pipeline preserved model quality.

In this step, get ready to:
  • Prepare a sample image for inference with proper preprocessing.
  • Run inference on the cloud-hosted device and decode the predictions.
  • Compare on-device INT8 results with PyTorch FP32 predictions and download the compiled model.
Prepare a sample image for inference

Neural networks expect input in a very specific format. A raw JPEG file is not usable directly. You need to resize, normalize, and transpose the image into the shape the model was trained on.

  • In edge_ai_pipeline.py, find the line that says print(profile) at the bottom of your file.
  • Add the following code below it to download and preprocess a sample image:
# --- Step 5: Run On-Device Inference and Validate ---

# 8. Prepare sample image
sample_image_url = (
    "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/apidoc/input_image1.jpg"
)
response = requests.get(sample_image_url, stream=True)
response.raw.decode_content = True
image = Image.open(response.raw).resize((224, 224))
input_array = np.expand_dims(
    np.transpose(np.array(image, dtype=np.float32) / 255.0, (2, 0, 1)), axis=0
)

What does this code do?

This downloads a sample image from Qualcomm's public assets and transforms it into the exact format MobileNetV2 expects.

  • resize((224, 224)) matches the model's trained input resolution.
  • / 255.0 normalizes pixel values from 0-255 to 0-1.
  • np.transpose(..., (2, 0, 1)) converts from HWC (height, width, channels) to CHW format, which PyTorch models expect.
  • np.expand_dims(..., axis=0) adds the batch dimension, making the shape (1, 3, 224, 224).
  • Save edge_ai_pipeline.py.
  • You won't see a change yet. The next chunk submits this image to the device for inference.
Run inference on the cloud-hosted device

With the image preprocessed, you can now send it to the physical Samsung Galaxy S24 for inference. The device runs your compiled INT8 TFLite model and returns raw output logits.

  • Add the following code below the image preprocessing block to submit the inference job:
# 9. Run inference on cloud-hosted device
inference_job = client.submit_inference_job(
    model=target_model,
    device=device,
    inputs=dict(image_tensor=[input_array]),
)
assert isinstance(inference_job, hub.InferenceJob)
print(f"Inference job URL: {inference_job.url}")

on_device_output = inference_job.download_output_data()
assert isinstance(on_device_output, dict)

What does this code do?

submit_inference_job() sends your preprocessed image to the cloud-hosted Samsung Galaxy S24. The device runs the compiled TFLite model and returns the output tensor.

download_output_data() retrieves the raw output as a dictionary. The keys correspond to the model's output tensor names.

  • Save edge_ai_pipeline.py.
  • You won't see a change yet. The next chunk decodes these raw outputs into human-readable class predictions.

Now you need to convert the raw logits into probabilities and map them to ImageNet class labels.

  • Add the following code below the inference download block to post-process and display predictions:
# 10. Post-process on-device output
output_name = list(on_device_output.keys())[0]
out = on_device_output[output_name][0]
on_device_probabilities = np.exp(out) / np.sum(np.exp(out), axis=1)

# 11. Load ImageNet class labels
sample_classes = "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/apidoc/imagenet_classes.txt"
response = requests.get(sample_classes, stream=True)
response.raw.decode_content = True
categories = [str(s.strip()) for s in response.raw]

# 12. Print top-5 on-device predictions
print("\nTop-5 On-Device predictions (INT8 quantized):")
top5_classes = np.argsort(on_device_probabilities[0], axis=0)[-5:]
for c in reversed(top5_classes):
    print(f"  {categories[c]:30s} {on_device_probabilities[0][c]:>6.1%}")

What does this code do?

  • Softmax (the np.exp(out) / np.sum(np.exp(out)) formula) converts raw logits into probabilities that sum to 1.0.
  • The ImageNet class labels file maps each output index to a human-readable category name (e.g., "golden retriever", "sports car").
  • np.argsort(...)[-5:] grabs the indices of the 5 highest-probability classes, giving you the top-5 predictions.
  • Save edge_ai_pipeline.py.
  • You won't see a change yet. The next chunk adds PyTorch comparison and runs the full script.

Seeing a connection error when downloading the image?

The sample image and class labels are hosted on Qualcomm's public S3 bucket. If you see a timeout or connection refused error, check that your internet connection is active and try running the script again.

Why can't I download the sample image from Qualcomm's assets?

Compare on-device results with PyTorch predictions and download the compiled model

The ultimate test of your quantization pipeline is whether the INT8 model still agrees with the original FP32 model. If both predict the same top class, your optimization preserved the model's decision-making ability.

  • Add the following code below the top-5 on-device predictions block to compare with PyTorch and download the model:
# 13. Compare with PyTorch (FP32) predictions
with torch.no_grad():
    pytorch_input = torch.from_numpy(input_array)
    pytorch_output = torch_model(pytorch_input)
    pytorch_probs = torch.nn.functional.softmax(pytorch_output, dim=1).numpy()

print("\nTop-5 PyTorch predictions (FP32 original):")
top5_pytorch = np.argsort(pytorch_probs[0], axis=0)[-5:]
for c in reversed(top5_pytorch):
    print(f"  {categories[c]:30s} {pytorch_probs[0][c]:>6.1%}")

# 14. Download the compiled model
target_model.download("mobilenet_v2.tflite")
print("\nCompiled model downloaded as 'mobilenet_v2.tflite'")
print("\nPipeline complete!")

What does this code do?

  • torch.no_grad() disables gradient computation since you are only running inference, not training.
  • torch.nn.functional.softmax() converts the PyTorch logits to probabilities, matching what you did with NumPy for the on-device output.
  • target_model.download("mobilenet_v2.tflite") saves the compiled INT8 TFLite model to your local disk. This is the deployment-ready artifact you could ship to any Android device with a Snapdragon chipset.
  • Save edge_ai_pipeline.py.
  • Run the full pipeline by running this command in your terminal:
python edge_ai_pipeline.py
  • Wait for all jobs to complete. You should see the inference job URL printed, followed by the top-5 on-device predictions and the top-5 PyTorch predictions. Both lists should show the same top class.
  • Confirm that mobilenet_v2.tflite was downloaded by running:
ls mobilenet_v2.tflite
  • You should see mobilenet_v2.tflite listed in the output. This is your deployment-ready quantized model.

Top classes don't match between PyTorch and on-device?

If the top-1 class differs between FP32 and INT8, this can happen when the quantized model's confidence is close between two classes. Check if the top-1 on-device class appears anywhere in the PyTorch top-5 list.

If predictions look completely random (uniform probabilities), your calibration data may have caused issues during quantization. Re-running the script often resolves this since the random calibration seed changes.

Help me debug why my on-device predictions don't match my PyTorch predictions.

Why might the confidence percentages differ?

Quantization compresses 32-bit floating point values into 8-bit integers. This introduces small rounding errors that accumulate across layers. The top predicted class usually stays the same, but the exact confidence percentages will differ. This is expected behavior.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file and confirmed the mobilenet_v2.tflite download.

ⓧ I'd like to double check the full code

import os

import numpy as np
import requests
import torch
import torchvision
from PIL import Image

import qai_hub as hub

# --- Step 2: Load and Export the PyTorch Model ---

# 1. Load pre-trained MobileNetV2 from torchvision
torch_model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()

# 2. Export the model using torch.export
input_shape = (1, 3, 224, 224)
with torch.no_grad():
    pt2_model = torch.export.export(torch_model, (torch.rand(input_shape),))

# 3. Connect to AI Hub and compile to ONNX
client = hub.Client()
device = hub.Device("Samsung Galaxy S24 (Family)")

compile_onnx_job = client.submit_compile_job(
    model=pt2_model,
    device=device,
    input_specs=dict(image_tensor=input_shape),
    options="--target_runtime onnx",
)
assert isinstance(compile_onnx_job, hub.CompileJob)
print(f"Compile to ONNX job URL: {compile_onnx_job.url}")

unquantized_onnx_model = compile_onnx_job.get_target_model()
assert isinstance(unquantized_onnx_model, hub.Model)
print("ONNX model ready for quantization.")

# --- Step 3: Quantize the Model with Calibration Data ---

# 4. Prepare calibration data (random sample for quick latency benchmark)
calibration_data = dict(
    image_tensor=[np.random.randn(1, 3, 224, 224).astype(np.float32)]
)

# 5. Submit quantization job (FP32 -> INT8)
quantize_job = client.submit_quantize_job(
    model=unquantized_onnx_model,
    calibration_data=calibration_data,
    weights_dtype=hub.QuantizeDtype.INT8,
    activations_dtype=hub.QuantizeDtype.INT8,
)
assert isinstance(quantize_job, hub.QuantizeJob)
print(f"Quantize job URL: {quantize_job.url}")

quantized_onnx_model = quantize_job.get_target_model()
assert isinstance(quantized_onnx_model, hub.Model)
print("Model quantized to INT8 successfully.")

# --- Step 4: Compile for Target Device and Profile ---

# 6. Compile quantized model to TFLite
compile_tflite_job = client.submit_compile_job(
    model=quantized_onnx_model,
    device=device,
    options="--target_runtime tflite --quantize_io",
)
assert isinstance(compile_tflite_job, hub.CompileJob)
print(f"Compile to TFLite job URL: {compile_tflite_job.url}")

target_model = compile_tflite_job.get_target_model()
assert isinstance(target_model, hub.Model)
print("Quantized TFLite model compiled successfully.")

# 7. Profile on cloud-hosted device
profile_job = client.submit_profile_job(
    model=target_model,
    device=device,
)
assert isinstance(profile_job, hub.ProfileJob)
print(f"Profile job URL: {profile_job.url}")

profile = profile_job.download_profile()
print("\n--- On-Device Profile Results ---")
print(profile)

# --- Step 5: Run On-Device Inference and Validate ---

# 8. Prepare sample image
sample_image_url = (
    "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/apidoc/input_image1.jpg"
)
response = requests.get(sample_image_url, stream=True)
response.raw.decode_content = True
image = Image.open(response.raw).resize((224, 224))
input_array = np.expand_dims(
    np.transpose(np.array(image, dtype=np.float32) / 255.0, (2, 0, 1)), axis=0
)

# 9. Run inference on cloud-hosted device
inference_job = client.submit_inference_job(
    model=target_model,
    device=device,
    inputs=dict(image_tensor=[input_array]),
)
assert isinstance(inference_job, hub.InferenceJob)
print(f"Inference job URL: {inference_job.url}")

on_device_output = inference_job.download_output_data()
assert isinstance(on_device_output, dict)

# 10. Post-process on-device output
output_name = list(on_device_output.keys())[0]
out = on_device_output[output_name][0]
on_device_probabilities = np.exp(out) / np.sum(np.exp(out), axis=1)

# 11. Load ImageNet class labels
sample_classes = "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/apidoc/imagenet_classes.txt"
response = requests.get(sample_classes, stream=True)
response.raw.decode_content = True
categories = [str(s.strip()) for s in response.raw]

# 12. Print top-5 on-device predictions
print("\nTop-5 On-Device predictions (INT8 quantized):")
top5_classes = np.argsort(on_device_probabilities[0], axis=0)[-5:]
for c in reversed(top5_classes):
    print(f"  {categories[c]:30s} {on_device_probabilities[0][c]:>6.1%}")

# 13. Compare with PyTorch (FP32) predictions
with torch.no_grad():
    pytorch_input = torch.from_numpy(input_array)
    pytorch_output = torch_model(pytorch_input)
    pytorch_probs = torch.nn.functional.softmax(pytorch_output, dim=1).numpy()

print("\nTop-5 PyTorch predictions (FP32 original):")
top5_pytorch = np.argsort(pytorch_probs[0], axis=0)[-5:]
for c in reversed(top5_pytorch):
    print(f"  {categories[c]:30s} {pytorch_probs[0][c]:>6.1%}")

# 14. Download the compiled model
target_model.download("mobilenet_v2.tflite")
print("\nCompiled model downloaded as 'mobilenet_v2.tflite'")
print("\nPipeline complete!")

Secret mission

Compare TFLite vs QNN Context Binary Performance

You've deployed your model using TFLite, a portable runtime that works across many Android devices. But Qualcomm offers a second compilation target: QNN Context Binary. This format is SOC-specific and targets the NPU exclusively. Your mission is to compile the same quantized model to QNN Context Binary and compare the performance metrics head-to-head.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally and uses Qualcomm AI Hub's free tier, so there are no ongoing costs.

Resources you used:

  • qai_hub conda environment (with qai-hub, torchvision, Pillow, numpy, and requests packages).
  • Local project files: edge_ai_pipeline.py, edge_ai_pipeline_qnn.py, and mobilenet_v2.tflite.
  • Qualcomm AI Hub account (free, no charges).

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend the pipeline with more models.

  • Your qai_hub conda environment stays active with all packages installed.
  • Your project files remain in place for future runs.
  • Your Qualcomm AI Hub account and API token remain configured for submitting new jobs.

Pause - I'll come back to this later

Deactivate the conda environment to free up your terminal session. Your files and environment stay on disk for when you return.

  • Deactivate the conda environment by running:
conda deactivate
  • When you're ready to resume, reactivate with conda activate qai_hub.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the conda environment and all local files created during the project.

  • Deactivate the conda environment first (if it is currently active):
conda deactivate
  • Remove the qai_hub conda environment and all its installed packages by running:
conda env remove -n qai_hub
  • Delete the local project files. On macOS/Linux:
rm edge_ai_pipeline.py edge_ai_pipeline_qnn.py mobilenet_v2.tflite
  • On Windows:
del edge_ai_pipeline.py edge_ai_pipeline_qnn.py mobilenet_v2.tflite
  • Your Qualcomm AI Hub account is free and does not incur charges. You can leave it as-is or delete it from your Qualcomm account settings at myaccount.qualcomm.com if you no longer need it.

That's a Wrap!

That's a Wrap!

Nice work. You just built a complete edge AI optimization pipeline that takes a PyTorch model from research all the way to optimized on-device inference on a real Snapdragon chip.

You've learned how to:

  • Quantize a PyTorch model from FP32 to INT8 using Qualcomm's AIMET-powered pipeline with calibration data, then compile it for a Samsung Galaxy S24 targeting the TFLite runtime.
  • Profile on-device performance on real cloud-hosted hardware, measuring inference latency, peak memory usage, and NPU compute unit mapping.
  • Run on-device inference on a cloud-hosted Snapdragon device and validate INT8 predictions against the original FP32 PyTorch output.
  • Secret Mission: Compared the portable TFLite format against the NPU-optimized QNN Context Binary format to analyze performance trade-offs between portability and hardware-specific acceleration.

Ready to quiz yourself?