Build AI Chat Moderator with Gemini

Create a production-grade content moderation API that filters toxic messages from chats.

Introduction

This project has been archived

We're no longer updating this project - we recommend checking out our active projects instead!

If you've already completed this project, don't worry! Your documentation is still available for you to check out and share.

If you have any questions, leave a question in our Discord and we'll get back to you ASAP!

⚡️ 30 second Summary

Every platform with user chat faces the same challenge: toxic messages, spam, and abuse can destroy a community in seconds. With millions, or even billions of messages sent everyday, manual moderation simply can't keep up.

That's why platforms like Twitch, YouTube, and Discord use AI to screen content in real-time. AI can classify messages instantly, blocking toxic content, flagging spam, and letting clean messages through.

You'll build this same system using Google's Gemini API and Azure Functions. Your API will analyze every message, block the bad ones automatically, and save violations for moderator review.

What you'll build

In this project, you'll add AI-powered content moderation to your streaming backend from the first project.

By the end of this project, you'll have:

  • 🔗 A Gemini API integration for real-time content analysis.
  • ☢️ Moderation logic that classifies messages as toxic, spam, harassment, or clean.
  • ⛳️ Violations tracking in Cosmos DB that stores flagged messages for review.
  • 💎 Secret Mission: A moderator dashboard API to review and act on flagged content.

Want a complete demo of how to do this project, from start to finish? Check out our 🎬 walkthrough with Kahu

Prerequisite

This project builds on Build a Streaming Backend on Azure. You'll need an existing Function App with a message endpoint and Cosmos DB connection. If you haven't completed that project yet, start there first.

Not sure if this project is right for you? Check if it matches your goals

If you're up for a bit of a challenge, quiz yourself on the key concepts up ahead in this project.

This project is part of a series:

  1. Part 1: Build Your First Azure Function
  2. Part 2: You are here!
  3. Part 3: Orchestrating Parallel Event Streams with Azure
  4. Part 4: GitHub to Azure DevOps Pipeline

Connect Your App to Gemini AI

Every message your users send is about to get screened by the same AI technology that powers Google's content safety systems. You'll grab an API key from Google AI Studio and wire up Gemini to analyze messages in real-time.

First, let's make sure you have everything from Part 1.

✅ I completed Part 1 + Secret Mission

You should have:

  • An Azure Function App deployed and running
  • A Cosmos DB account with a messages container
  • A working POST /message endpoint

If all of this is in place, continue to the next section!

⚡ I completed Part 1 but not Cosmos DB

No problem! You just need to set up Cosmos DB. Complete the steps below, then continue with this project.

Create a Cosmos DB Account

  • In Azure Portal: Create a resource → Search Azure Cosmos DB → Create.
  • Select Azure Cosmos DB for NoSQL → Create.
  • Fill in:
    • Resource group: streaming-backend-rg
    • Account name: streaming-backend-cosmos (globally unique)
    • Location: Same as your resource group
    • Capacity mode: Serverless
  • Click Review + create → Create.

Wait for deployment (2-3 minutes).

Create the Database and Container

  • Click Go to resource → In sidebar, click Data Explorer.
  • Click New Container.
  • Fill in:
    • Database id: Create new → streamingdb
    • Container id: messages
    • Partition key: /username
  • Click OK.

Get the Connection String

  • In sidebar: Settings → Keys.
  • Copy the PRIMARY CONNECTION STRING.

Add the Connection String to Azure

  • Search Function App → Select yours.
  • Settings → Environment variables → + Add.
  • Name: COSMOS_CONNECTION_STRING
  • Value: Paste your connection string.
  • Click Apply → Apply → Confirm.

Update Your Function Code

  • Open function_app.py in Cursor.
  • Send this prompt in Cursor chat:
Update this Azure Function app to use Cosmos DB instead of in-memory storage:

1. Add a get_container() helper that connects to Cosmos DB using the COSMOS_CONNECTION_STRING environment variable, database "streamingdb", and container "messages"
2. Update POST /api/message to store messages in Cosmos DB using container.create_item()
3. Update GET /api/messages to read from Cosmos DB using container.read_all_items(), removing Cosmos metadata fields (_rid, _self, _etag, _attachments, _ts)
4. Handle the case where Cosmos DB isn't configured (return empty array for GET, log warning for POST)

Use the azure.cosmos SDK (CosmosClient).

What's happening in this prompt?

This prompt asks Cursor to add database persistence to your function app. Instead of storing messages in a Python list (which disappears when the function restarts), it will:

  • Create a helper function to connect to Cosmos DB using your connection string
  • Modify POST to save messages to the database
  • Modify GET to read messages from the database (filtering out internal Cosmos metadata)
  • Handle the case where the database isn't set up yet (graceful degradation)

Update Dependencies

Update requirements.txt:

azure-functions
azure-cosmos

Deploy and Test

  • Open the terminal in your code editor (in Cursor: View → Terminal or press Ctrl+`).
  • Run this command to deploy your changes:
func azure functionapp publish streaming-backend

Replace streaming-backend with your Function App name if different. Test that messages persist, then continue to the next section!

❌ I need to catch up

No worries! Complete the streaming backend setup below, then continue with this project.

Create an Azure Account

  • Go to azure.microsoft.com/free and sign up (or log in if you have one).
  • Azure gives you $200 in free credits for 30 days.

Create a Resource Group

  • Go to the Azure Portal.
  • Click Create a resource → Search Resource group → Create.
  • Fill in:
    • Subscription: Select yours.
    • Resource group: streaming-backend-rg.
    • Region: Select a region close to you.
  • Click Review + create → Create.

Install Azure Functions Core Tools

  • Open a terminal on your computer (on macOS: search for "Terminal", on Windows: search for "PowerShell").
  • Check if Azure Functions Core Tools is already installed:
func --version

If not installed:

macOS:

brew tap azure/functions
brew install azure-functions-core-tools@4

Windows:

npm install -g azure-functions-core-tools@4 --unsafe-perm true

Install the Azure CLI

  • In the same terminal, check if the Azure CLI is installed:
az --version

If not installed:

macOS: brew install azure-cli

Windows: Download from aka.ms/installazurecliwindows

Log in to Azure

  • Run this command to log in to your Azure account:
az login

A browser will open - sign in with your Azure account.

Create Your Azure Function Project

  • Create a folder called streaming-backend and open it in your code editor (like Cursor).
  • Open the terminal in your editor (in Cursor: View → Terminal or press Ctrl+`).
  • Initialize the project by running:
func init --python
  • Create a new HTTP trigger function by running:
func new --name message --template "HTTP trigger" --authlevel "anonymous"
  • Open function_app.py and use Cursor chat to generate code with this prompt:
Create an Azure Function app with three HTTP endpoints:

1. POST /message - receives JSON with "content" and optional "username", stores in an in-memory list, returns the created message with a UUID and timestamp
2. GET /messages - returns all stored messages as JSON
3. GET /health - returns "Healthy" status

Use the azure.functions SDK with @app.route decorators. Include proper error handling.

What's happening in this prompt?

This prompt asks Cursor to create a basic message API with three endpoints:

  • POST /message - Receives a chat message, assigns it a unique ID and timestamp, stores it in memory
  • GET /messages - Returns all stored messages as JSON (for displaying chat history)
  • GET /health - A simple health check endpoint (useful for monitoring)

The @app.route decorators tell Azure Functions which URL paths trigger which functions.

Test Locally

  • In your terminal, start the function by running:
func start
  • Open a new terminal by clicking the + button in the terminal panel (keep the first terminal running):

macOS:

curl -X POST http://localhost:7071/api/message \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello from the stream!", "username": "testuser"}'

Windows:

Invoke-WebRequest -Uri http://localhost:7071/api/message `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body '{"content": "Hello from the stream!", "username": "testuser"}'
  • Test GET:
curl http://localhost:7071/api/messages
  • Press Ctrl+C to stop.

Create a Storage Account

  • In Azure Portal: Create a resource → Storage account → Create.
  • Resource group: streaming-backend-rg.
  • Storage account name: streamingbackendstorage (globally unique, lowercase).
  • Performance: Standard.
  • Redundancy: LRS.
  • Click Review + create → Create.

Upgrade Your Subscription

Azure Functions requires Pay-As-You-Go (still only pay for what you use).

  • Search Subscriptions → Select yours → Upgrade subscription.
  • Select Basic → Upgrade to pay as you go.

Create the Function App

  • Search Function App → Create.
  • Hosting option: Consumption.
  • Resource group: streaming-backend-rg.
  • Function App name: streaming-backend (globally unique).
  • Runtime: Python 3.11.
  • Region: Same as your resource group.
  • Storage: Select the storage account you created.
  • Review + create → Create.

Deploy Your Function

func azure functionapp publish streaming-backend

Copy your Function App URL:

[[FUNCTION_URL="https://streaming-backend-abc123.azurewebsites.net"]]

Test the Live Endpoint

macOS:

curl -X POST [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello from Azure!", "username": "clouduser"}'

Windows:

Invoke-WebRequest -Uri [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body '{"content": "Hello from Azure!", "username": "clouduser"}'
curl [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/messages

Add Persistent Storage with Cosmos DB

Your messages are stored in memory - they're gone when the function restarts. Connect to Azure Cosmos DB for persistent storage.

Build a moderator dashboard API that lets humans review, approve, and reject AI-flagged content.

In this secret mission:

  • Create a GET endpoint to list flagged violations
  • Add filtering by category, date range, and review status
  • Build a PATCH endpoint to approve or reject violations
  • Track moderator actions with audit logging

The human-in-the-loop pattern

Twitch's AutoMod, YouTube's comment filter, and Discord bots all have appeal systems. AI moderation will make mistakes - your dashboard lets moderators catch false positives and feed corrections back.

Create the GET /violations Endpoint

  • Create violations_api.py and send this prompt:
Create a Python module called violations_api.py with a function to list violations for moderator review.

Requirements:
1. Create a get_violations(req, violations_container) function that:
   - Accepts query parameters: category (toxic/spam/harassment), status (pending/approved/rejected), days (default 7), limit (default 50)
   - Queries Cosmos DB violations container with filters
   - Returns JSON with violations array, count, and applied filters
   - Uses parameterized queries to prevent SQL injection
   - Orders results by timestamp DESC
2. Handle the case where no filters are provided (return all pending violations)
3. Return proper error responses for exceptions

Click Accept to apply the changes.

Add the Review Action Endpoint

Send this prompt to add the review function:

Add a review_violation function to violations_api.py that lets moderators approve or reject flagged messages.

Requirements:
1. Create a review_violation(req, violations_container) function that:
   - Gets violation ID from route params
   - Accepts JSON body with: action ("approve" or "reject"), moderatorId, and optional notes
   - Validates that action is either "approve" or "reject"
   - Queries the violation by ID (use cross-partition query)
   - Updates the violation with: reviewStatus, reviewedBy, reviewedAt timestamp, reviewNotes
   - If action is "reject", also set falsePositive=True for tracking
   - Uses upsert_item to save the updated document
2. Return 400 for missing/invalid action or moderatorId
3. Return 404 if violation not found
4. Return the updated violation on success

Click Accept to apply the changes.

Wire Up the New Endpoints

  • Open function_app.py and send this prompt:
Update function_app.py to add two new endpoints for the moderation dashboard:

Requirements:
1. Import get_violations and review_violation from violations_api
2. Add GET /violations endpoint that calls get_violations with the violations container (use route="violations" - Azure adds the /api/ prefix automatically)
3. Add PATCH /violations/{id} endpoint that calls review_violation with the violations container (use route="violations/{id}")
4. Use the existing get_violations_container() helper to get the container
5. Keep all existing endpoints (message, messages, health)

Click Accept to apply the changes.

Deploy and Test

func azure functionapp publish streaming-backend

Test the GET Endpoint

🍎 macOS / Linux

curl [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/violations

🪟 Windows (PowerShell)

Invoke-WebRequest -Uri [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/violations

You should see a JSON response with your violations.

Test the Review Endpoint

Get a violation ID from the GET response, then:

🍎 macOS / Linux

curl -X PATCH [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/violations/[[VIOLATION_ID="your-violation-id"]] \
  -H "Content-Type: application/json" \
  -d '{"action": "approve", "moderatorId": "mod-123", "notes": "Correctly identified as toxic"}'

🪟 Windows (PowerShell)

Invoke-WebRequest -Uri [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/violations/[[VIOLATION_ID="your-violation-id"]] `
  -Method PATCH `
  -Headers @{"Content-Type"="application/json"} `
  -Body '{"action": "approve", "moderatorId": "mod-123", "notes": "Correctly identified as toxic"}'

You should see the updated violation with reviewStatus, reviewedBy, and reviewedAt fields.

You built the complete content moderation loop! Violations are caught automatically, humans review edge cases, and the feedback improves the system!

Once you've completed these steps and have a working Function App with Cosmos DB, continue with Step #1 above!

In this step, get ready to:

  • Create a Gemini API key from Google AI Studio
  • Add the API key to your Azure Function App settings
  • Understand prompt engineering for content moderation

Get Your Google AI API Key

First, let's get an API key from Google AI Studio. The free tier gives you plenty of requests to build and test your moderation system.

✅ I see the Google AI Studio dashboard

You're already signed in. You'll see the Google AI Studio homepage.

  • In the left sidebar, click Get API key

🚀 I see a Get started button

You're not signed in yet. You'll see a welcome page with a Get started button.

  • Click Get started.

✅ I have a Google account

  • Click on your existing Google account to sign in.

After signing in, you'll be on the Google AI Studio dashboard.

➕ I need a Google account

  • Click Use another account.
  • Click Create account.
✅ Account created successfully

After creating your account, you'll be on the Google AI Studio dashboard.

❓ I am struggling to create an account

If you're having trouble:

  • Make sure you're using a valid email address.
  • Check your email for a verification link if prompted.
  • Try using a different browser if the form isn't working.
Still having issues?

Get help with Google account creation

  • In the top right, click Create API key.

A dialog will appear to create your API key.

  • In the name field, enter My API Key (you can call it whatever you want).
  • Click on the Choose an imported project dropdown.
  • Select Create project.
  • In the project name field, enter My New Project (you can call it whatever you want).
  • Click Create project.
  • Click Create key.

Nice! API key set up and ready to go.

  • Click the Copy icon next to your key to copy it.

What is an API key?

An API key is a unique identifier that authenticates your requests to a service. Think of it like a password that tells Google "this request is coming from an authorized user." When you call the Gemini API, you include this key so Google knows who's making the request and can track usage.

Google's free tier gives you 1500 requests/day. If someone uses your key, their requests count against YOUR quota, potentially blocking your access or incurring charges.

⚠️ Keep your API key secure!

Don't share your API key publicly or commit it to version control.

Add the API Key to Azure Function App

Your Function App needs the Gemini API key to call the moderation service. We'll store it securely as an Application Setting.

  • Go to the Azure Portal.
  • Search for Function App and select your streaming backend Function App (from the previous project).

✅ I found my Function App

  • In the left sidebar, scroll down to Settings and select Environment variables.
  • Click + Add.
  • Add the following setting:
    • Name: GEMINI_API_KEY
    • Value: Paste your Gemini API key
  • Click Apply.
  • Back on the Environment variables page, click Apply to save your changes.
  • Click Confirm to restart the app.

✅ Setting saved successfully

Your API key is now securely stored. The Function App will restart to load the new setting.

Why use Application Settings instead of hardcoding?

Storing API keys in Application Settings keeps them secure and separate from your code. This means you can share your code on GitHub without exposing secrets, and you can easily rotate keys without redeploying. Azure encrypts these settings at rest and only exposes them to your running function as environment variables.

Continue to understand how we'll use this key in our moderation prompts!

⚠️ I'm getting an error

If you have trouble saving the application setting:

  • "Unauthorized" - Make sure you have Contributor or Owner role on the Function App
  • "Invalid setting name" - Setting names can't have spaces. Use GEMINI_API_KEY exactly
  • Setting disappeared after save - Make sure you clicked Save at the top (not just OK on the dialog)

Try refreshing the portal page and checking Configuration again to verify your setting was saved.

Get help with configuration errors or share your error with the NextWork community!

❌ I can't find my Function App

If you don't see your Function App from Part 1:

  • Type "Function App" in the Azure Portal search bar
  • Check that you're in the correct subscription (top right corner)
  • Look for a Function App with a name like streaming-backend-{random} or whatever you named it

If you truly don't have a Function App:

Understand Content Moderation Prompts

Before we write code, let's understand how to design prompts for AI classification tasks.

For content moderation, you want the AI to:

  1. Analyze the message content
  2. Classify it into categories (toxic, spam, harassment, clean)
  3. Return a structured response you can parse

Here's the prompt strategy we'll use:

You are a content moderator for a live streaming chat platform.
Analyze the following message and classify it.

Message: "{user_message}"

Respond with ONLY a JSON object in this format:
{
  "is_allowed": true/false,
  "category": "clean" | "toxic" | "spam" | "harassment",
  "confidence": 0.0-1.0,
  "reason": "brief explanation"
}

Why structure the prompt this way?

This prompt design follows three key principles for AI classification:

  1. Role assignment - "You are a content moderator" gives the AI context about what kind of judgment to make
  2. Clear task - "Analyze and classify" tells it exactly what to do
  3. Structured output - "Respond with ONLY a JSON object" ensures the response is machine-readable

The JSON format with specific fields (is_allowed, category, confidence, reason) makes it easy to parse programmatically. Without this structure, the AI might return free-form text that's harder to act on.

💡 What do the response fields mean?

  • is_allowed - Boolean that your code uses to decide: save the message or block it
  • category - Classification label for analytics and moderator review
  • confidence - How sure the AI is (0.0-1.0). You could use this for borderline cases
  • reason - Human-readable explanation to show users or moderators

This structured approach makes it easy to parse the AI's response and take action.

Add AI Moderation to Your API

Here's where the magic happens. You're about to transform your basic message endpoint into a real-time content filter - every message will pass through Gemini for classification before it touches your database.

In this step, get ready to:

  • Create a violations container in Cosmos DB
  • Add Gemini API integration to your Function
  • Modify the POST /message endpoint to check messages before saving

Create the Violations Container

First, let's create a separate container to store flagged messages for review.

  • Go to the Azure Portal.
  • Search for Cosmos DB and select your account.
  • Select Data Explorer from the sidebar.
  • Click New Container.
  • Fill in the form:
    • Database id: Select Use existing and choose your database (e.g., streamingdb)
    • Container id: violations
    • Partition key: /streamId
  • Click OK.

✅ Container created

Great! You should see the violations container appear under your database in the Data Explorer tree.

⚠️ I'm getting an error

If you have trouble creating the container:

  • "Container already exists" - Great! You already have it. Continue to the next section.
  • "Invalid partition key" - Make sure you typed /streamId with the forward slash
  • "Database not found" - Make sure you selected your existing database, not "Create new"
  • "Unauthorized" - Check that you have Contributor role on the Cosmos DB account

Get help with container creation or share your error with the NextWork community!

Why create a separate violations container?

Separating violations from regular messages serves two purposes:

  1. Security - Flagged content may be inappropriate. Keeping it separate prevents accidental exposure to regular users
  2. Review workflow - Moderators can query just the violations container to review flagged messages without sifting through millions of clean messages

The /streamId partition key matches your messages container, making it easy to correlate violations with their source stream.

Open Your Project in Cursor

Time to write some code! Let's open your streaming backend project in Cursor.

  • Open Cursor on your computer.
  • Click File → Open Folder.
  • Navigate to your streaming-backend project folder and select it.

You should see your project files in the sidebar, including function_app.py and requirements.txt from Part 1.

Don't have Cursor?

Download it from cursor.com. Cursor is an AI-powered code editor that can generate code from prompts. You can also use VS Code with GitHub Copilot if you prefer.

Create the Moderation Module

  • Open the Cursor chat panel (press Ctrl+L or Cmd+L).
  • Send this prompt:
Create a Python module called moderation.py that checks messages for toxic content using the Gemini API.

Requirements:
1. Get the API key from environment variable GEMINI_API_KEY
2. Use the Gemini API endpoint: https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent
3. Create a check_message(message_content: str) function that:
   - Sends the message to Gemini with a prompt asking it to classify as clean/toxic/spam/harassment
   - Returns a dict with: is_allowed (bool), category (str), confidence (float 0-1), reason (str)
   - Uses "fail-open" pattern: if the API fails, return is_allowed=True
   - Handles markdown code blocks in Gemini's response (strip them before parsing JSON)
4. Use the requests library for HTTP calls

What's happening in this prompt?

This prompt asks Cursor to create a moderation module that:

  • Connects to the Gemini API using your API key from environment variables
  • Sends each message to Gemini with a classification prompt
  • Parses Gemini's JSON response to determine if the message is allowed
  • Returns a structured result your main code can act on
  • Uses "fail-open" so a Gemini outage doesn't break your entire chat

💡 What is the "fail-open" pattern?

If AI moderation fails (API down, parsing error, etc.), messages get through rather than being blocked. This prevents a Gemini outage from breaking all chat. Production systems might queue messages for later review instead.

Cursor will analyze your prompt and create the moderation.py file. Click Accept to apply the changes.

Cursor might respond with something like the above. This means it:

  • Created moderation.py with a check_message() function
  • Set up the Gemini API integration with proper request formatting
  • Added the fail-open pattern (returns is_allowed=True if the API fails)

Update the POST /message Endpoint

Now modify your existing message endpoint to check messages before saving them.

  • In Cursor chat, send this prompt:
Update my Azure Function POST /message endpoint to add content moderation.

Requirements:
1. Import check_message from the moderation module I just created
2. Before saving a message, call check_message(message_content)
3. If is_allowed is False:
   - Save the message to a "violations" container in Cosmos DB (include category, confidence, reason, timestamp)
   - Return HTTP 403 with the category and reason in the response
4. If is_allowed is True:
   - Save to the messages container as before
   - Return HTTP 200 success
5. Keep my existing Cosmos DB connection code

What's happening in this prompt?

This prompt asks Cursor to add moderation to your existing endpoint:

  • Import the check_message function you just created
  • Call it before saving any message to the database
  • If blocked: save to the violations container and return HTTP 403
  • If allowed: save to messages container and return HTTP 200
  • Keep all your existing database code intact

💡 Why return 403 for blocked messages?

HTTP 403 (Forbidden) indicates the server understood the request but refuses to authorize it. This is semantically correct for moderation - the user's request was valid, but the content was not allowed. The response body includes the category and reason so clients can show helpful feedback like "Your message was flagged as spam."

Cursor will modify your function_app.py. Click Accept to apply the changes.

Cursor might respond with something like the above. This means it:

  • Added from moderation import check_message at the top of your file
  • Created a get_violations_container() helper to connect to the violations container
  • Modified the POST endpoint to call check_message() before saving
  • If is_allowed is False: saves to the violations container and returns HTTP 403
  • If is_allowed is True: saves to messages and returns HTTP 200 as before

Update requirements.txt

Cursor may have already added the requests library to your requirements. Let's check!

  • Open requirements.txt.

✅ Cursor added requests for me

If you see requests in your requirements.txt, you're good to go! Cursor automatically added it when generating the moderation code.

❌ I don't see requests

No problem! Add requests to the list manually:

azure-functions
azure-cosmos
requests

Why do we need the requests library?

The requests library is Python's most popular HTTP client. We use it to make POST requests to the Gemini API. Azure Functions will install this when you deploy.

Watch Your AI Block Toxic Messages

The moment of truth. You're about to deploy your moderation system and watch it catch toxic content in real-time - just like the systems protecting millions of users on platforms like Twitch and Discord.

In this step, get ready to:

  • Deploy the updated Function App
  • Test with toxic messages (should be blocked)
  • Test with clean messages (should be saved)
  • Review flagged messages in the violations container

Deploy to Azure

Time to deploy your updated code to Azure Functions.

  • Open the terminal in your code editor (in Cursor: View → Terminal or press Ctrl+`).
  • Make sure you're in your streaming-backend project folder.
  • Run this command:
func azure functionapp publish streaming-backend

What does this command do?

This command packages your Python code (including the new moderation.py file) and uploads it to your Azure Function App. Azure will install the dependencies from requirements.txt and restart the function with your changes.

Replace streaming-backend with your Function App name if it's different.

✅ Deployment succeeded

You should see output ending with "Deployment completed successfully".

⚠️ Deployment failed

If your deployment failed:

  • "Not logged in" - Run az login first to authenticate
  • "Function App not found" - Replace streaming-backend with your actual Function App name (case-sensitive)
  • "Build failed" - Check your Python syntax. Run python -m py_compile moderation.py locally to find errors
  • Missing requirements - Make sure you updated requirements.txt with the requests library

Try deploying again after fixing any issues.

Get help with deployment or share your error with the NextWork community!

  • Copy the Function URL from the output.
  • Paste it below:

[[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]

Test with Toxic Messages

Let's test the moderation with messages that should be blocked.

  • Open a new terminal in your code editor by clicking the + button in the terminal panel.
  • Run this command to send a toxic test message:

🍎 macOS / Linux

curl -X POST [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message \
  -H "Content-Type: application/json" \
  -d '{"content": "Terrible choice!", "username": "testuser"}'

🪟 Windows (PowerShell)

Invoke-WebRequest -Uri [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body '{"content": "Terrible choice!", "username": "testuser"}'

✅ Message was blocked (403)

You should receive a 403 response like this:

This means your moderation is working! The AI correctly identified and blocked the toxic message.

What just happened?

When you sent that request, your Function App:

  1. Received the message via HTTP POST
  2. Sent it to Gemini with your moderation prompt
  3. Gemini analyzed the content and returned is_allowed: false
  4. Your code stored the violation in Cosmos DB
  5. Returned 403 with the rejection reason

All of this happened in typically less than a second - fast enough for real-time chat moderation.

⚠️ Message was allowed (200)

If the message wasn't blocked:

  • AI classified differently - Gemini may have interpreted the message differently. Try more explicit test cases
  • API key not set - Check Configuration in Azure Portal for GEMINI_API_KEY
  • Code not deployed - Make sure your latest code is deployed

Try testing with more obviously toxic content or check your Function App logs.

Get help with moderation not working or share your error with the NextWork community!

❌ I got an error

If you received an error (500 or other):

  • "GEMINI_API_KEY not found" - Add the API key in Azure Portal → Configuration
  • "Connection refused" - Your Function App may still be restarting. Wait a minute and try again
  • "Function not found" - Make sure you deployed the latest code with moderation.py
  • "Invalid JSON" - Check your curl command syntax (quotes, escaping)

Check the Function App logs in Azure Portal → Monitor for more details.

Get help with function errors or share your error with the NextWork community!

Test with Clean Messages

Now test with messages that should pass moderation. Run this command in the same terminal:

🍎 macOS / Linux

curl -X POST [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message \
  -H "Content-Type: application/json" \
  -d '{"content": "Great stream! Love the content!", "username": "gooduser"}'

🪟 Windows (PowerShell)

Invoke-WebRequest -Uri [[FUNCTION_URL="https://streaming-backend.azurewebsites.net"]]/api/message `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body '{"content": "Great stream! Love the content!", "username": "gooduser"}'

✅ Message was saved (200)

You should receive a 200 response:

Your moderation correctly identified this as a clean message and saved it to Cosmos DB!

⚠️ Message was blocked (403)

If a clean message was blocked:

  • False positive - AI moderation isn't perfect. Try different phrasing
  • Overly strict prompt - The AI may be too aggressive. You can adjust the prompt later
  • Check the reason field in the response to understand why it was blocked

This is normal - even production systems have false positives. The Secret Mission covers how to handle these!

Get help with false positives or share your error with the NextWork community!

Review Violations in Data Explorer

Let's check the flagged messages stored in Cosmos DB.

  • Go to the Azure Portal.
  • Search for Cosmos DB and select your account.
  • Select Data Explorer from the sidebar.
  • Expand your database and select the violations container.
  • Click Items to see all flagged messages.

You should see your test violations with:

  • The original message content
  • Category (toxic, spam, etc.)
  • Confidence score
  • Reason for blocking
  • Timestamp

Why store all this metadata?

Each violation document captures the full context:

  • message - The actual content (for moderator review)
  • category - What type of violation (for filtering and analytics)
  • confidence - How certain the AI was (for prioritizing review)
  • reason - AI's explanation (helps moderators understand the decision)
  • userId/username - Who sent it (for repeat offender tracking)
  • timestamp - When it happened (for time-based analysis)

This data enables dashboards, analytics, and appeal workflows - which you'll build in the Secret Mission!

Secret mission

No AI is perfect. Twitch's AutoMod lets streamers adjust sensitivity. Discord bots have appeal systems. YouTube's filters can be overridden by channel owners. Why? Because sometimes AI blocks a sarcastic joke or misses subtle harassment.

You've built the AI side. Now build the human side - an API that lets moderators review flagged content, overturn false positives, and track what the AI is catching. This is the human-in-the-loop pattern used by every major platform.

Give Moderators the Power to Override AI

Clean Up Your Resources

Clean Up Your Resources

Before you close your laptop, let's make sure you're not paying for resources you don't need.

🟢 Keep everything running

If you plan to continue with more projects in this series, keep your resources running!

  • Function App - Will continue to cost a small amount (usually free tier eligible)
  • Cosmos DB - Free tier includes 1000 RU/s and 25 GB storage
  • GEMINI_API_KEY - Free tier has generous limits for development

No action needed - just continue to the next project when you're ready!

🟡 Remove API key only

If you want to keep the Function App but remove the Gemini integration:

  • Go to your Function App in the Azure Portal.
  • Select Configuration under Settings.
  • Find GEMINI_API_KEY and click the delete icon (trash can).
  • Click Save at the top.
  • Click Continue to confirm the app restart.

Your Function App will keep running, but moderation calls will fail (with the fail-open behavior returning "allowed").

🔴 Delete all resources

If you're completely done with the streaming backend and want to remove everything:

  • Go to the Azure Portal.
  • Type "Resource groups" in the search bar.
  • Select your resource group (the one containing your Function App).
  • Click Delete resource group at the top.
  • Type the resource group name to confirm.
  • Click Delete.

This will remove:

  • Function App
  • Cosmos DB account (including all containers and data)
  • Application Insights (if created)
  • Storage account (used by Functions)

Wait for the deletion to complete - this may take a few minutes.

You Built Production-Grade AI Moderation

You Built Production-Grade AI Moderation

You just built the same content moderation pattern used by platforms processing millions of messages daily. Twitch, Discord, Match Group - they all use AI to screen content in real-time. Now you know how it works.

What you learned:

  • 🤖 How to integrate Gemini AI for real-time content classification
  • ☁️ Azure Functions for serverless API development
  • 🗄️ Cosmos DB for storing messages and violations
  • 🔐 API key management with Azure Application Settings
  • 🛡️ The fail-open pattern for resilient AI systems
  • ✍️ Designing prompts for AI classification tasks

Ready to quiz yourself? 💪

What's next?

Want to keep building AI-powered cloud applications? Check out our other Azure AI projects, or dive into the Secret Mission to add human moderator oversight to your system!

Make sure to join the NextWork Community to share what you've built and get help along the way 🤝