Run a Private AI Assistant on Android

Run Google's Gemma model fully on-device with zero cloud dependency.

Introduction

30 Second Summary

Your phone is powerful enough to run its own AI assistant, no internet or subscription required.

In this project, you will build a personal AI assistant called "NanoClaw" that runs entirely on your Android phone using Google's Gemma model. You will customize its personality, test it offline in airplane mode, and benchmark exactly how fast your phone can run AI.

What You'll Build

A fully private, offline AI chat assistant powered by on-device LLM inference through the Google AI Edge Gallery app.

By the end of this project, you'll have:

  • A working on-device AI assistant running Gemma on your Android phone with zero cloud dependency.
  • A custom "NanoClaw" persona built through system prompts that responds in your preferred style.
  • Performance benchmark data showing exactly how fast Gemma runs on your specific hardware.
  • A custom Agent Skill that gives your NanoClaw the ability to use tools like Wikipedia for fact-grounded answers.

Are there any prerequisites?

None. You just need an Android phone running Android 12 or higher with at least 4 GB of RAM. Wi-Fi is needed only for the initial model download. Everything else runs offline.

Install the Google AI Edge Gallery

Before you can run an AI model on your phone, you need the app that makes it possible.

The Google AI Edge Gallery is a free app by Google that lets you download and run AI models directly on your Android device. It handles everything from model downloads to chat interfaces to performance benchmarking. In this step, you will install the app, verify your phone meets the requirements, and explore what it offers.

In This Step, Get Ready To:
  • Install the Google AI Edge Gallery app on your Android phone.
  • Verify your phone meets the minimum Android version requirement.
  • Explore the app's home screen and understand the tools available to you.
Check Your Android Version

The Google AI Edge Gallery requires Android 12 or higher. You also need at least 4 GB of RAM to run the models you will download later. Let's confirm your phone is ready before installing anything.

  • Open Settings on your phone.
  • Scroll down and tap About Phone.
  • Look for Android version.

You should see a number of 12 or higher. If your phone is running an older version, you will need to update your operating system or use a different device.

Download the App
  • Open the Google Play Store on your Android phone.
  • Search for AI Edge Gallery by Google.
  • Tap Install.

The app is about 22 MB, so it should download quickly.

I found it on Google Play

You are all set. Wait for the install to finish, then move on to the next substep.

I can't find it on Google Play

If you do not have Google Play access, you can install the app manually using the APK file from GitHub.

  • Open your phone's browser and go to https://github.com/google-ai-edge/gallery/releases/latest/.
  • Download the ai-edge-gallery.apk file.
  • Your phone may ask you to enable Install from Unknown Sources. You can find this in Settings under Security or Apps.
  • Tap the downloaded APK file to install it.

Still having trouble installing?

Some phones place the Unknown Sources toggle in different locations. Help me find the Unknown Sources setting on my phone.

Open the App and Explore the Home Screen

The Google AI Edge Gallery is not just a chatbot. It is a collection of AI-powered tools, each designed for a different type of task. Let's see what is available.

  • Launch the AI Edge Gallery app from your home screen or app drawer.

You will see several tiles on the home screen, each representing a different AI capability.

  • AI Chat is for multi-turn conversations, just like chatting with ChatGPT.
  • Prompt Lab is for single-turn tasks like summarization or text generation.
  • Ask Image handles image-to-text tasks using multimodal models.
  • Audio Scribe transcribes audio recordings.
  • Agent Skills lets the AI use external tools like Wikipedia to answer questions.
  • Mobile Actions gives the AI control over device functions.

You will use AI Chat, Prompt Lab, and Agent Skills throughout this project.

What is happening under the hood?

The Google AI Edge Gallery uses LiteRT (formerly TensorFlow Lite) and MediaPipe to run AI models directly on your phone's CPU or GPU. No data is sent to any server. Once a model is downloaded, the app works completely offline.

This is the same technology stack that powers on-device AI features in production Android apps.

Your app is installed and you know what tools are at your disposal. Next up, you will download a Gemma model and have your first on-device AI conversation.

Download and Load Your Gemma Model

With the AI Edge Gallery installed, it is time to load a Gemma model onto your phone. This is the brain of your NanoClaw assistant. Once downloaded, the model lives entirely on your device and never needs the internet again.

In This Step, Get Ready To:
  • Download a Gemma model to your phone for fully on-device AI inference.
  • Send your first message and see the model generate a response locally.
  • Read your phone's performance stats to understand how fast your hardware runs the model.
Choose and Download Your Model

The AI Edge Gallery supports several Gemma model sizes. Smaller models download faster and respond quicker, but larger models produce higher-quality answers. You need to pick the right one for your phone's hardware.

  • Tap the AI Chat tile on the home screen.
  • Browse the list of available models.

You will see several models listed. Here are the two best options for this project:

  • Gemma 4 E2B (recommended). About 2.58 GB download. This is the best balance of quality and speed for phones with 4 GB or more RAM. It also supports Agent Skills, which you will need for the Secret Mission.
  • Gemma 3 1B (lightweight alternative). About 529 MB download. Faster and lighter, good for phones with exactly 4 GB RAM or if you want quicker responses.
  • Tap on your chosen model to expand it.
  • Tap the Download button.

The download will take a few minutes depending on your Wi-Fi speed. You can switch apps or lock your phone during the download. The app sends a notification when it is done.

What Is Quantization?

Gemma 4 E2B has 5.1 billion total parameters, but only 2.3 billion activate during any single response. This technique, combined with quantization (compressing the model's numbers to use less memory), is what makes it possible to run a powerful language model on a phone with just a few gigabytes of RAM.

Being Asked to Log In to Hugging Face?

Some Gemma models are gated and may require you to log in with a Hugging Face account and accept the license terms. The app will prompt you through a browser flow if needed. Gemma 4 models have streamlined downloads that typically do not require a Hugging Face login. If you are stuck, help me get past the Hugging Face login screen.

Start Your First Chat

Once the download finishes, the Download button changes to Try It. This means the model is loaded and ready to go.

  • Tap Try It to open the AI Chat interface.
  • Type a simple message like Hello! Who are you? and tap Send.

Watch the response appear. The text generates token by token. This is happening entirely on your phone's processor. No internet involved.

What Is On-Device Inference?

When you use ChatGPT or Google Gemini online, your message travels to a data center, gets processed on powerful servers, and the response travels back. On-device inference skips all of that. The model runs directly on your phone's CPU or GPU. Your data never leaves the device, which means complete privacy and zero latency from network round trips.

Check the Performance Stats

Every time the model responds, the app displays performance statistics. These numbers tell you exactly how your phone handles AI workloads.

  • Look below the model's response for the performance stats.

You should see three key metrics:

  • Time to First Token (TTFT): How long before the first word appears on screen.
  • Prefill Speed: How fast the model processes your input, measured in tokens per second.
  • Decode Speed: How fast the model generates its output, measured in tokens per second.

What Do These Numbers Mean?

Decode speed is the most important number for chat. It determines how fast text appears on your screen. For reference, a Samsung S24 Ultra achieves roughly 56 tokens per second with Gemma 3 1B on GPU. A Samsung S26 Ultra gets about 52 tokens per second with Gemma 4 E2B on GPU. Your numbers will vary based on your phone's hardware.

  • Write down your TTFT, prefill speed, and decode speed numbers.

You will compare these numbers later when you adjust inference parameters and run formal benchmarks.

You now have a working AI model running entirely on your phone. Next up, you will give it a personality by crafting a custom NanoClaw persona with prompt engineering.

Build Your NanoClaw Persona with Prompt Engineering

You have a working Gemma model on your phone and you have seen it respond to a basic greeting. Now it is time to give your assistant a personality. In this step, you will use prompt engineering to transform a generic language model into your own custom assistant called NanoClaw.

In This Step, Get Ready To:
  • Craft a system prompt that defines your NanoClaw persona's personality and behavior.
  • Test your persona in multi-turn conversations to see how it maintains context.
  • Tune inference parameters like temperature and top-k to control how your assistant responds.
Craft a System Prompt for NanoClaw

A system prompt is a set of instructions that shapes how a language model behaves. It defines the assistant's personality, tone, and boundaries. Without one, the model gives generic responses. With a good system prompt, it feels like a purpose-built assistant.

You will use Prompt Lab to draft and test your system prompt before bringing it into a full conversation.

  • Go back to the home screen of the Google AI Edge Gallery app.
  • Tap the Prompt Lab tile.
  • Select the same Gemma model you downloaded in the previous step.
  • Choose the Freeform Prompt template.
  • Type the following system prompt and send it:
You are NanoClaw, a personal AI assistant that runs entirely on-device. You are privacy-focused, concise, and helpful. You always remind the user that their data never leaves their phone. When asked about your capabilities, explain that you run on the Gemma model locally on Android hardware. Keep responses under 3 sentences unless asked for more detail.

What is a system prompt doing here?

In Prompt Lab, you are sending a single-turn prompt that includes persona instructions. The model reads these instructions and shapes its response accordingly. This is different from a dedicated system prompt field that some chat apps offer. You are essentially telling the model "act like this" as part of your message.

Read the response carefully. Does it feel like a distinct assistant, or does it sound generic? Try tweaking parts of your prompt to make NanoClaw feel more personal. For example, you could add a rule like "Always start your response with 'Hey there!'" or "Use bullet points whenever listing more than two items."

Pro tip: make it yours

The example prompt above is just a starting point. The best system prompts reflect how you actually want your assistant to talk. Try adding constraints like a preferred language style, a specific area of expertise, or even a quirky personality trait.

Test NanoClaw in Multi-Turn Chat

Prompt Lab is great for testing single prompts, but a real assistant needs to handle back-and-forth conversations. AI Chat supports multi-turn conversations where the model remembers what you said earlier in the session.

  • Go back to the home screen and tap AI Chat.
  • Start a new conversation by pasting your system prompt as the first message.
  • Wait for the model to respond, then follow up with these questions one at a time:
What can you help me with?

Summarize this paragraph for me: The Internet of Things refers to the network of physical objects embedded with sensors, software, and other technologies for the purpose of connecting and exchanging data with other devices and systems over the internet.

Write me a short to-do list for learning mobile development.

What does multi-turn mean?

Each time you send a message in AI Chat, the model receives the entire conversation history as context. This means it can reference things you said earlier. If you told it to keep responses under 3 sentences in your system prompt, it should maintain that rule throughout the conversation.

Notice how the model maintains context across turns. Each response builds on the previous conversation. If NanoClaw drifts from its persona after a few messages, that is normal for smaller models. You can re-anchor it by reminding it of its role.

Experiment with Inference Parameters

Your system prompt controls what the model says. Inference parameters control how it generates each word. These settings determine whether your assistant plays it safe or gets creative.

  • While in AI Chat, look for the settings or parameters option to adjust inference parameters.
  • Find the Temperature setting.
  • Set it to 0.2 and ask: "Give me three creative names for a pet robot."
  • Now change the temperature to 0.9 and ask the exact same question again.

Compare the two responses. The low-temperature response should be more predictable and focused. The high-temperature response should be more varied and surprising.

What do temperature and top-k actually do?

When the model generates each word, it calculates a probability for every possible next word. Temperature controls how spread out those probabilities are. A low value like 0.2 makes the model strongly favor the most likely word. A high value like 0.9 flattens the probabilities so less likely words have a better chance of being picked.

Top-k limits how many candidate words the model even considers. Setting top-k to 10 means only the 10 most likely words are in the running. Setting it to 40 opens the door to more diverse word choices.

  • Now try adjusting Top-k. Set it to 10 and ask: "Explain why privacy matters for AI."
  • Change top-k to 40 and ask the same question again.

Understanding these parameters is a critical skill for tuning any AI assistant's behavior. For factual tasks like summarization, lower temperature and top-k keep the model accurate. For creative tasks like brainstorming, higher values produce more interesting results.

Your NanoClaw now has a defined personality and you know how to fine-tune its behavior with inference parameters. Next up, you will put it to the ultimate test by going fully offline and benchmarking exactly how fast your phone can run AI.

Test Offline Capability and Benchmark Your Hardware

You have a working AI assistant with a custom persona and tuned inference parameters. Now it is time to prove the core promise of on-device AI: it works without any internet connection. You will also benchmark your phone's hardware to see exactly how fast Gemma runs on your specific device.

In This Step, Get Ready To:
  • Verify your NanoClaw assistant works fully offline in airplane mode.
  • Run a benchmark to measure your phone's AI inference performance.
  • Compare CPU vs GPU performance to understand mobile AI trade-offs.
Go Fully Offline

The whole point of on-device AI is that it works without a server. Let's put that claim to the test by cutting off all network access.

  • Turn on Airplane Mode on your phone.
  • Confirm that both Wi-Fi and mobile data are off.
  • Open the Google AI Edge Gallery app.
  • Tap AI Chat and start a new conversation with your Gemma model.
  • Type this message and hit Send:
Are you still working without internet? Explain how on-device AI works.

The model responds just like it did when you were online. Every token is generated by your phone's processor. No data leaves your device.

Why does this matter?

Cloud-based AI assistants like ChatGPT send your prompts to remote servers for processing. That means they need internet, they cost money per request, and your data passes through third-party infrastructure. Your NanoClaw assistant avoids all three of those constraints. It works on a plane, in a tunnel, or anywhere else with zero connectivity.

Run a Benchmark on Your Hardware

Chatting gives you a feel for speed, but a benchmark gives you exact numbers. The AI Edge Gallery has a built-in benchmark tool that runs standardized tests on your downloaded model.

  • Turn Airplane Mode off.
  • Go back to the home screen of the AI Edge Gallery app.
  • Tap the hamburger menu (three lines) in the top-left corner.
  • Tap Models to see your downloaded models.
  • Find the benchmark button next to your Gemma model and tap it.

The benchmark runs your model through standardized tests and reports three key metrics:

  • Prefill Speed (tokens/sec): how fast the model processes your input prompt.
  • Decode Speed (tokens/sec): how fast the model generates its response.
  • Time to First Token (TTFT): how long you wait before the first word appears.

What should I see?

Your numbers will vary based on your phone's hardware. For reference, a Samsung S24 Ultra running Gemma 3 1B on GPU achieves roughly 56 tokens/sec decode speed and 2585 tokens/sec prefill speed. A Samsung S26 Ultra running Gemma 4 E2B on GPU achieves roughly 52 tokens/sec decode speed. Your results may be higher or lower depending on your phone's processor and available RAM.

Pro tip: close other apps first

Close background apps before running the benchmark. Other apps competing for RAM and CPU cycles can lower your scores and give you inaccurate results.

Compare CPU vs GPU Performance

Your phone has two different processors that can run AI inference: the CPU (general-purpose) and the GPU (optimized for parallel math). Switching between them can dramatically change performance.

  • Look for a backend or acceleration setting in the model settings within the AI Edge Gallery.
  • If your phone supports GPU inference, switch to the other backend (e.g., from GPU to CPU or vice versa).
  • Run the benchmark again on the new backend.
  • Compare your two sets of results side by side.

What Is the Difference Between CPU and GPU Inference?

GPU inference is typically much faster for prefill (processing your input) because GPUs excel at the parallel matrix math that neural networks require. CPU inference is more universally compatible across devices. GPU inference may also use more battery. Understanding this trade-off is important for anyone building mobile AI applications.

I can switch backends

Compare your CPU and GPU benchmark numbers. You should see a noticeable difference in at least one of the three metrics (prefill speed, decode speed, or TTFT). Write down both sets of results so you can reference them later.

I don't see a backend option

Not all phones expose a backend switching option in the AI Edge Gallery. Some devices only support CPU inference. This is completely fine. Your benchmark results from the CPU run are still valuable. You can skip the comparison and move on.

You now have hard data on your phone's AI performance and proof that your assistant works without any network connection. Next up, you will take on the Secret Mission and transform NanoClaw from a simple chatbot into a tool-using AI agent.

Secret mission

Build a Custom Agent Skill with Wikipedia Grounding

Your NanoClaw can do more than chat. Agent Skills let your assistant use external tools like Wikipedia to ground its answers in real data. Transform your on-device chatbot into a tool-augmented AI agent.

Before you go

Clean Up Your Resources

This project runs entirely on your Android phone, so there are no cloud costs or subscriptions to worry about. Decide whether to keep your resources as they are, free up storage space, or remove everything entirely.

Resources you used:

  • Google AI Edge Gallery app (~22 MB).
  • Downloaded Gemma model (Gemma 4 E2B ~2.58 GB, or Gemma 3 1B ~529 MB).
  • Chat history and system prompts stored locally in the app.

Keep everything running

No action needed. Choose this if you plan to keep using NanoClaw as your on-device AI assistant.

  • The app and model stay on your phone ready to use anytime.
  • There are no ongoing costs. The model runs locally with no subscriptions or API fees.
  • The downloaded model takes up storage space (up to ~2.58 GB for Gemma 4 E2B), but uses no data or battery when idle.

Pause - I'll come back to this later

Free up storage space by removing the downloaded model while keeping the app installed. You can re-download the model later when you want to use NanoClaw again.

  • Open the Google AI Edge Gallery app.
  • Tap the hamburger menu (three lines) in the top-left corner.
  • Tap Models to see your downloaded models.
  • Delete the downloaded Gemma model to reclaim storage space.
  • Keep the app installed so you can re-download the model later without needing to set up again.

Delete - I don't want to use this again

Remove all project resources from your phone completely.

  • Open your phone's Settings.
  • Go to Apps and find AI Edge Gallery.
  • Tap Uninstall to remove the app and all downloaded models.
  • This frees up all storage used by the app and models (up to ~2.6 GB).

Mission accomplished

That's a Wrap!

Nice work! You just built a personal AI assistant that runs entirely on your Android phone. No cloud. No API keys. No subscription fees. No data ever leaving your device.

Here's what you accomplished:

  • Deployed an on-device Gemma model on your Android phone and proved it works fully offline in airplane mode.
  • Built a custom NanoClaw persona using prompt engineering and tuned its behavior with inference parameters like temperature and top-k.
  • Benchmarked your phone's AI performance by measuring decode speed, prefill speed, and time to first token across CPU and GPU backends.
  • Built a tool-augmented AI agent using Agent Skills with Wikipedia grounding, turning your NanoClaw from a chatbot into something that can use external tools.

Ready to quiz yourself?