Deploy a GPU Chatbot on Runpod

Deploy an authenticated GPU chatbot with vLLM and Gradio on Runpod.

Introduction

30 Second Summary

Typing into a chat box feels effortless. Behind every quick reply, a machine is doing expensive work that needs to stay private and available.

In this project, you will deploy an authenticated chatbot on a Runpod GPU. Gradio then streams Qwen replies from a vLLM server.

What You'll Build

You open a secure browser page where a GPU-hosted Qwen model streams replies while remembering the earlier turns in your conversation.

By the end of this project, you'll have:

A streaming chatbot that lets you submit a prompt from your browser. You watch the reply arrive as the model generates it.
A context-aware conversation where the second answer uses information from the first turn.
An authenticated public page delivered over HTTPS. You can share the chat URL while the raw inference endpoint stays private.
Secret Mission: Add starter prompts and browser-saved conversation history.

Are there any prerequisites?

You need a browser, comfort with terminal commands, and a Runpod account funded for less than one hour of on-demand GPU time. All code runs inside the Pod, so you do not need local Python or CUDA installed.

Before We Start

A clear purpose gives every chatbot response something useful to prove. Before any GPU credits are spent, you will choose who your chatbot helps and plan a two-turn conversation that tests whether it retains context.

Launch and Prepare the GPU Pod

Your planned two-turn conversation gives every minute of paid GPU time a clear purpose. A Runpod Pod provides a reproducible CUDA-capable runtime for the model server and browser UI.

In this step, you will launch the paid Pod with private inference networking. Then you will use JupyterLab to install pinned versions of vLLM, Gradio, and the OpenAI Python client.

In this step, get ready to:

Launch an on-demand Runpod Pod with an NVIDIA GPU and persistent storage.
Create the project directory and install the pinned Python packages.
Verify the GPU and prepare two JupyterLab terminal tabs.

Launch the on-demand GPU Pod

Your Pod starts billing

Launching this Pod begins paid compute. Runpod bills Pods by the second at the live rate shown during deployment.

Keep the project runtime below one hour to control the compute total. The volume disk costs $0.10/GB/month while the Pod is running.

Stopped volume-disk storage costs $0.20/GB/month. You will choose what to keep during project cleanup.

Go to the Runpod console in your browser.
Create a Runpod account through the console if you do not have one.
Sign in to your Runpod account.

Why use the official PyTorch image?

The image provides a shared Linux environment for the GPU tools and Python packages. Using the documented image makes the Pod configuration reproducible.

Click + New in the Runpod console.
Select Pod.
Select one available NVIDIA T4 or newer GPU.
Choose on-demand pricing.
Set the container image to runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404.

How the network stays private

An exposed HTTP port lets Runpod's proxy reach a service inside the Pod. Only port 7860 needs that route because it hosts the browser interface.

The inference server later uses port 8000 inside the Pod. Keeping that port unexposed limits direct access to the model server.

Configure a volume disk for the Pod.
Confirm its mount path is /workspace.
Enter 7860 under Expose HTTP Ports (Max 10).
Confirm 8000 is absent from the exposed HTTP ports.

Pod provisioning can take a few minutes while Runpod allocates the GPU. A short wait at this point is expected.

Review the live hourly total in the Summary.
Confirm your credit balance covers one hour at that total.
Add enough credit through Runpod's billing flow if required.
Choose Deploy Pod or Deploy On-Demand based on the console flow shown.
Wait for the Pod status to show that it is running.

That is the costly infrastructure piece complete. Your GPU runtime is now running with persistent storage and one public UI port.

Open JupyterLab through the running Pod's connection controls.

Create the project and install dependencies

The volume disk preserves files in /workspace across Pod stops. The project directory keeps your dependency file separate from other Pod files.

Navigate to /workspace in the JupyterLab file browser.
Create a folder named qwen-gradio-chat with the file browser's new-folder control.

Your persistent workspace now contains the qwen-gradio-chat folder.

Open the qwen-gradio-chat folder from the JupyterLab file browser.
Create a text file named requirements.txt with the file browser's new-file control.

You should see requirements.txt listed inside /workspace/qwen-gradio-chat.

Open requirements.txt from the JupyterLab file browser.
Replace its contents with these pinned dependencies:

gradio==6.21.0
openai==3.13.0

What do these dependencies provide?

The gradio==6.21.0 pin provides the browser chat interface used later in the project.
The openai==3.13.0 pin lets Gradio communicate with the OpenAI-compatible model endpoint.
Exact pins keep the project on the package versions tested together.

Save requirements.txt.
Confirm the saved file contains both version pins.

File saved in the wrong folder?

Check the JupyterLab file browser path. The file must be located at /workspace/qwen-gradio-chat/requirements.txt.

Move the file into the qwen-gradio-chat folder if it appears directly under /workspace.

Help me place requirements.txt in the correct JupyterLab folder.

✔️ Awesome, I've got everything!

Your dependency file now matches the required version pins.

ⓧ I'd like to double check the full code

gradio==6.21.0
openai==3.13.0

The vLLM wheel is a large download, so installation can take several minutes. Quiet stretches in the terminal are expected while packages download.

Open the first terminal tab with JupyterLab's terminal control.
Install the complete project environment by running these commands:

cd /workspace/qwen-gradio-chat
python -m pip install "https://github.com/vllm-project/vllm/releases/download/v0.29.0/vllm-0.29.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl" --extra-index-url https://download.pytorch.org/whl/cu129
python -m pip install -r requirements.txt

What do these installs add?

The first command places the terminal in /workspace/qwen-gradio-chat.
The wheel installs vLLM 0.29.0 with dependencies from the CUDA 12.9 package index.
The final command installs the Gradio and OpenAI pins from requirements.txt.

Wait for each installation to return to the terminal prompt.
Confirm the installation output finishes without a package failure.

Package installation failed?

Confirm the Pod is still running with an NVIDIA T4 or newer GPU. A stopped Pod or incompatible GPU prevents this environment from working.

Check that the terminal started inside /workspace/qwen-gradio-chat. Then rerun the installation block above.

Help me troubleshoot the failed Python package installation.

Verify the GPU and prepare both terminals

The model server and chat interface later run as separate long-lived processes. Two terminal tabs let each service keep running while you work with the other.

Open a second terminal tab with JupyterLab's terminal control.

You should now see two terminal tabs available in JupyterLab.

Return to the first terminal tab.

Before you run the final checks, predict the GPU details and package versions the terminal will reveal.

Verify the NVIDIA GPU and pinned Python packages by running these commands:

nvidia-smi
python -m pip show vllm gradio openai

What should you see?

The first check displays the NVIDIA GPU assigned to your Pod. You should see an NVIDIA T4 or newer GPU.

The package details should show vLLM 0.29.0, Gradio 6.21.0, and OpenAI 3.13.0.

Missing a GPU or package version?

If the GPU is missing, return to the Runpod Pod configuration and confirm the selected Pod has an NVIDIA T4 or newer GPU.

If a package is missing or has the wrong version, rerun the installation block from the previous substep.

Help me diagnose the missing GPU or package version.

Your GPU environment is ready with persistent project files and two terminal tabs. Next, you will load Qwen on the GPU and prove the local inference endpoint works.

Serve Qwen with vLLM

Your Runpod Pod is ready with the pinned packages from the previous step. The next goal is to use its NVIDIA GPU for GPU inference.

A browser interface needs a working model server before it can deliver useful chat. In this step, vLLM gives you the first visible win through a private OpenAI-compatible API.

In this step, get ready to:

Start Qwen3-0.6B on the Pod GPU through vLLM.
Confirm the local server advertises the expected model.
Receive a generated assistant response from the chat endpoint.

Start the Qwen server

The server loads Qwen/Qwen3-0.6B onto the Pod GPU. The qwen3 reasoning parser helps vLLM interpret the model's reasoning output.

The first launch can take a few minutes while the model loads. Terminal A may pause between log messages during this process.

Switch back to terminal A from the previous step.
Start the inference server by running this command:

vllm serve Qwen/Qwen3-0.6B --reasoning-parser qwen3

What does this command do?

The vllm command starts an HTTP server for GPU-backed model requests.
The Qwen/Qwen3-0.6B value identifies the model loaded by the server.
The --reasoning-parser qwen3 option selects the parser mapped to the Qwen3 model series.
The API becomes available inside the Pod at http://127.0.0.1:8000/v1.

Wait for terminal A to finish loading the model.
Confirm the final logs indicate that the API server is ready to accept requests.
Leave terminal A running.

Did the server stop during startup?

Confirm terminal A still has access to the NVIDIA GPU from the previous step. A GPU allocation problem can stop the model before the API starts.

Check that the command uses Qwen/Qwen3-0.6B with the qwen3 reasoning parser.

Ask for help with the startup logs.

Confirm the served model

The /v1/models endpoint reports which model the server currently exposes. A successful response proves terminal B can reach the private service running in terminal A.

Switch back to terminal B from the previous step.
Query the private model list by running this command:

curl http://127.0.0.1:8000/v1/models

What does this response prove?

The request asks vLLM which model is available for inference. It travels through localhost inside the Pod.

A response containing Qwen/Qwen3-0.6B proves the server registered the expected model.

You will see a JSON response whose model list contains Qwen/Qwen3-0.6B.

Could terminal B not reach the model list?

Return to terminal A to confirm the vLLM process is still running. The local endpoint disappears when that process stops.

Check that the request uses port 8000. Port 7860 remains reserved for the browser interface.

Ask for help with the connection problem.

Generate an assistant response

The /v1/chat/completions endpoint accepts structured chat messages. The max_tokens setting keeps this first response short.

Before you send the request, do you expect a model-generated answer or only server metadata?

Send a short prompt to Qwen by running this request in terminal B:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Explain GPU inference in one short sentence."}
    ],
    "max_tokens": 64
  }'

What does this request do?

The Content-Type: application/json header tells the server how to read the request body.
The model field routes the request to Qwen/Qwen3-0.6B.
The messages field supplies the user prompt in chat format.
The max_tokens field limits the generated response to 64 tokens.

You will see a JSON response containing assistant text generated by Qwen/Qwen3-0.6B.

That first answer proves your Pod GPU can serve a real chat request through the private vLLM endpoint.

Did the chat request fail?

Confirm terminal A still shows the running vLLM process. Restart the same server command if the process has stopped.

Compare the request's model value with Qwen/Qwen3-0.6B from the model-list response. The values must match exactly.

Ask for help with the request output.

Your private model server is running on the Pod GPU. Next, you will connect it to a browser chat interface with Gradio.

Build the Gradio Chat Interface

Your Qwen model already answers requests through the private vLLM endpoint. Now you need a browser interface that is easier to demonstrate.

Gradio turns that endpoint into a streaming chat page. The first browser test uses Runpod's HTTP proxy to reveal how the default network binding behaves.

In this step, get ready to:

Build a Gradio chat interface around the private Qwen endpoint.
Start the interface with its default localhost binding.
Compare the local response with the public Runpod proxy route.

Create the chat interface

The interface connects directly to the private OpenAI-compatible endpoint. Gradio handles the chat layout plus response streaming with a small Python file.

Why Gradio for the interface?

The inference server already owns model execution. Gradio adds the browser experience without requiring a separate frontend or another API layer.

This separation keeps the GPU server focused on inference. The interface stays lightweight.

Return to the JupyterLab file browser from earlier.
Navigate to /workspace/qwen-gradio-chat.
Create app.py inside /workspace/qwen-gradio-chat.

You’ll see app.py listed beside requirements.txt in the file browser.

Paste the following code into app.py:

import gradio as gr


demo = gr.load_chat(
    "http://127.0.0.1:8000/v1",
    model="Qwen/Qwen3-0.6B",
    file_types=None,
    system_message="You are a concise, helpful assistant.",
    streaming=True,
    title="Qwen3 on Runpod",
    description="A GPU-hosted Qwen3 chatbot served by vLLM.",
)


if __name__ == "__main__":
    demo.launch()

What does this code do?

The import makes Gradio available through the shorter gr name.
gr.load_chat() connects the interface to the private OpenAI-compatible endpoint.
model="Qwen/Qwen3-0.6B" selects the model already running in terminal A.
file_types=None disables file uploads.
system_message gives the assistant a concise response style.
streaming=True displays each response as it is generated.
demo.launch() starts the web server with Gradio's default network settings.

Save app.py.
Switch back to terminal B from earlier.
Move terminal B into /workspace/qwen-gradio-chat by running this command:

cd /workspace/qwen-gradio-chat

What does this command do?

This moves terminal B into the folder containing app.py. Python can now find the application file by name.

Start the Gradio server by running this command:

python app.py

What does this command do?

Python runs app.py and starts the Gradio web server. The temporary demo.launch() call uses the default local binding on port 7860.

Good progress. You’ll see Gradio report a local address on port 7860, which confirms the interface process is running.

Gradio did not report a local address?

Confirm terminal B is inside /workspace/qwen-gradio-chat.
Check that app.py ends with demo.launch().
Confirm terminal A still shows the running vLLM server.

Help me diagnose why the Gradio server does not start from app.py.

Leave terminal B running so the Gradio server remains available for the network checks.

✔️ Awesome, I've got everything!

Your app.py file is saved. Terminal B is now running the Gradio server.

ⓧ I'd like to double check the full code

import gradio as gr


demo = gr.load_chat(
    "http://127.0.0.1:8000/v1",
    model="Qwen/Qwen3-0.6B",
    file_types=None,
    system_message="You are a concise, helpful assistant.",
    streaming=True,
    title="Qwen3 on Runpod",
    description="A GPU-hosted Qwen3 chatbot served by vLLM.",
)


if __name__ == "__main__":
    demo.launch()

What should match?

Compare this reference with your saved app.py. Every setting shown here belongs in the file before you continue.

Prove the app works inside the Pod

A local request contacts Gradio from inside the same Pod. This isolates the interface from the public proxy so you can prove the server itself is healthy.

Create a temporary JupyterLab terminal tab for the local health check.

Before you run the check, what kind of response would prove that a web page is listening on port 7860?

Request the local Gradio page by running this command:

curl http://127.0.0.1:7860

What does this check prove?

The request goes straight to port 7860 inside the Pod. A page response proves that Gradio is running independently of the external proxy.

You’ll see HTML for the Gradio page in the terminal. That response confirms the interface is healthy on the Pod's local network.

Local request cannot connect?

Confirm terminal B still shows the running Gradio process.
Check that terminal B reported port 7860 when the server started.
Confirm app.py still contains demo.launch().

Help me troubleshoot why curl cannot reach the local Gradio server.

Test the Runpod proxy

The public proxy sends traffic to the exposed container port. This final test asks whether Gradio's default listener accepts traffic arriving through that route.

Before you open the route, do you think the same page that answered inside the Pod will also load through the public proxy?

Return to the Runpod console from earlier.
Open the HTTPS connection link for HTTP port 7860.

You’ll see that the public route cannot reach the Gradio page. This failure is the designed result for the first browser attempt.

Why do the local and public checks disagree?

Gradio's default listener uses 127.0.0.1. That address accepts requests originating inside the same Pod.

The Runpod proxy sends traffic through the container network. A proxied service must listen on 0.0.0.0 to accept that traffic.

Port 7860 is already exposed correctly. The local listener is the remaining gap.

You have proved that the interface works inside the Pod. Next, you’ll change its network binding and protect the public page with authentication.

Expose and Protect the Chat UI

Your chatbot already works inside the Pod. However, the Runpod HTTP proxy cannot reach Gradio while it listens only on localhost.

This step changes the bind address so the proxy can connect. You will also add basic authentication so the public URL does not invite strangers to spend your GPU credits.

In this step, get ready to:

Bind Gradio to an address the Runpod proxy can reach.
Protect the public chat interface with sign-in credentials.
Verify streamed conversational context while keeping inference private.

Configure public access

The current launch call uses Gradio's default network binding. The replacement sets the reachable address plus the port plus the sign-in requirement.

Stop the current Gradio process in terminal B.
Return to JupyterLab from earlier.
Open /workspace/qwen-gradio-chat/app.py in the file editor.
Find this block:

if __name__ == "__main__":
    demo.launch()

Why did the proxy fail?

The current demo.launch() call uses Gradio's default 127.0.0.1 binding. That address accepts requests from inside the Pod only.

This explains why the local curl check worked while the Runpod proxy could not connect.

The credentials are stored as plain text in app.py. A unique demo-only password keeps a real account password out of the file.

Choose a username for this short-lived demonstration.
Choose a unique password that you do not use for another account.
Replace the existing block with the configuration below. Customize the auth tuple with the credentials you chose while entering it:

if __name__ == "__main__":
    demo.launch(
        server_name="0.0.0.0",
        server_port=7860,
        share=False,
        auth=("your-username-here", "your-password-here"),
    )

What does this configuration do?

The server_name value binds Gradio to every container interface so the Runpod proxy can reach it.
The server_port value keeps the web interface on exposed port 7860.
The share value prevents Gradio from creating a separate sharing tunnel.
The auth tuple adds a credential check before the chat page loads.
The two placeholder strings show where your chosen credentials belong.

Save app.py.

The reference below keeps neutral credential placeholders. Your saved copy should contain the username and password you selected.

✔️ Awesome, I've got everything!

Your launch settings are in place. Your saved file is ready for the restart.

ⓧ I'd like to double check the full code

import gradio as gr


demo = gr.load_chat(
    "http://127.0.0.1:8000/v1",
    model="Qwen/Qwen3-0.6B",
    file_types=None,
    system_message="You are a concise, helpful assistant.",
    streaming=True,
    title="Qwen3 on Runpod",
    description="A GPU-hosted Qwen3 chatbot served by vLLM.",
)


if __name__ == "__main__":
    demo.launch(
        server_name="0.0.0.0",
        server_port=7860,
        share=False,
        auth=("your-username-here", "your-password-here"),
    )

What should match?

The structure of your file should match this reference. Your auth tuple should contain the username and password you selected.

Apply the protected network settings

The stopped process still used the old launch configuration. Starting a new process loads the reachable bind address plus the authentication settings.

Before you restart the server, do you expect the proxy to reach the same app that was already healthy inside the Pod?

Restart the Gradio server in terminal B by running:

python app.py

What does this restart prove?

The new process loads the saved launch configuration from app.py. Its 0.0.0.0 binding accepts traffic forwarded through the Runpod proxy.

The authentication check runs before Gradio reveals the chat interface.

You should see Gradio report a server running on port 7860. The process remains active in terminal B while the web interface is available.

Return to the Runpod HTTPS proxy URL from the previous step.
Confirm that a login prompt appears before the chatbot.
Enter the username you chose.
Enter the password you chose.
Submit the sign-in form.

You cleared the networking hurdle. The proxy now reaches your protected Gradio interface.

Still unable to reach the login prompt?

Check that terminal B still shows the Gradio process running.
Confirm that server_name is set to 0.0.0.0.
Confirm that server_port is set to 7860.

Ask for help with the bind address or proxy connection: help me diagnose why my Gradio login page is unreachable through the Runpod proxy.

Test the protected chat

The browser now talks to Gradio through Runpod. Gradio sends each turn to the private vLLM endpoint running on port 8000.

Your first reply proves that streaming still works through the proxy. Your second reply tests whether the chat retains conversational context.

Send the first prompt from the two-turn demonstration you planned in Step 0.
Watch the assistant response stream into the chat.

Before you send the second prompt, do you expect the answer to reuse a detail from the first turn?

Send the second prompt from your planned demonstration.
Check the second answer for information carried over from the first turn.
Return to the Pod configuration in the Runpod console from earlier.
Check Expose HTTP Ports (Max 10).

You should see the first answer stream into the page. The second answer should use context from the first turn.

You should see 7860 as the only exposed HTTP port. Port 8000 remains absent from the public network configuration.

Chat not streaming or missing context?

Confirm that terminal A still shows the Qwen server running.
Keep both prompts inside the same open conversation.
Check terminal B for a request error after sending a prompt.

Ask for help with the connection between both services: help me troubleshoot why my Gradio chat is not streaming contextual responses from the private vLLM endpoint.

That is the full deployment working: the Runpod proxy reaches your authenticated Gradio page. Gradio streams contextual replies from vLLM without exposing port 8000.

Secret mission

Add Saved History and Starter Prompts

Turn the blank chat into a guided experience with three clickable prompts. Then keep conversations available in the same browser after a refresh.

Clean Up Your Resources

Choose whether to keep the chatbot live, stop the Pod for later, or terminate it completely. Runpod charges for the active GPU plus volume storage, so your choice changes what keeps billing.

Cost warning

Your running on-demand Pod is billed by the second at the live GPU rate shown during deployment. The volume disk costs $0.10/GB/month while the Pod is running.

Stopped volume-disk storage costs $0.20/GB/month. Choose Pause or Delete if you do not need the chatbot online.

Resources you used:

One paid Runpod GPU Pod running the chatbot services.
One volume disk mounted at /workspace that stores the project data.
One public HTTPS proxy route for port 7860.

Keep everything running

No action is needed. Choose this if you want the authenticated chatbot to stay available.

Leave the Runpod Pod running.
Keep only port 7860 exposed through the HTTP proxy.
Keep port 8000 absent from the exposed HTTP ports.
Review the live billing total in the Runpod console before leaving the Pod active.

The authenticated chatbot remains reachable through its proxy route. GPU compute charges plus volume storage charges continue.

Pause - I'll come back to this later

Pausing releases the GPU while preserving the volume disk for a later return. Stopped storage remains billed at $0.20/GB/month.

Return to the Pods page in the Runpod console.
Select the running Pod.
Choose Stop.
Wait until the Pod status shows stopped.

Your files under /workspace remain on the volume disk. Data stored only on the container disk is erased.

Delete - I don't want to use this again

Complete removal ends GPU compute charges. It also ends volume storage charges.

Termination Removes the Volume Disk

Terminating the Pod deletes the volume disk. Everything under /workspace disappears with it.

Copy anything you want to keep from /workspace/qwen-gradio-chat to a separate location.
Return to the Pods page in the Runpod console.
Select the running Pod.
Choose Terminate.
Confirm the termination when prompted.
Wait until the Pod disappears from the Pods page.
Refresh the browser tab that held the Runpod HTTPS proxy URL.

You should no longer reach the chatbot. That confirms the public proxy route has ended with the Pod.

Nice Work!

You pulled it off! Your authenticated chatbot now streams Qwen responses from a cloud GPU into the browser.

You've learned how to:

Serve Qwen3-0.6B through a private OpenAI-compatible API powered by vLLM on Runpod.
Resolve a localhost binding failure so the Runpod HTTPS proxy can reach Gradio. Basic authentication protects the public interface while port 8000 stays private.
Demonstrate streaming multi-turn chat where the second response uses context from the first turn.
Secret Mission: Turn the interface into a more usable product with starter prompts plus browser-saved conversation history.

Ready to quiz yourself?