Build a Smart Vision Assistant

Introduction

30 Second Summary

Imagine navigating a busy room without being able to see what's around you. For visually impaired people, everyday tasks like identifying objects, recognizing money, and avoiding obstacles require constant assistance from others.

In this project, you will build a Smart Vision Assistant prototype that uses a webcam to detect objects in real time, recognize currency denominations, and measure obstacle distance. Every detection is announced through audio so the user never needs to look at a screen.

What You'll Build

A unified assistive technology system that combines YOLOv8 object detection, MobileNetV2-based currency classification via transfer learning, and simulated ultrasonic distance sensing with text-to-speech feedback.

By the end of this project, you'll have:

• A real-time object detection script that identifies 80+ object classes through your webcam and announces them aloud.
• A currency recognition module that classifies banknote denominations using a convolutional neural network.
• A distance measurement system with proximity-based audio alerts, designed for deployment on a Raspberry Pi with an HC-SR04 sensor.
• A unified assistant with mode switching that coordinates all three modules into one seamless experience.
• Secret Mission: Add a navigation mode that combines object detection with distance measurement to give directional audio cues like "chair detected, 2 meters ahead."

Are there any prerequisites?

You need basic Python knowledge (variables, functions, and loops). No machine learning experience is required. The project runs on your PC using a webcam, so no special hardware is needed to get started.

Before We Start

Before We Start...

Set Up Your Development Environment

Before you can build any machine learning features, you need the right foundation. Your Smart Vision Assistant relies on several Python libraries for object detection, image classification, and audio feedback.

In this step, you will install Python, set up VS Code, and get all the dependencies in place. Once everything is ready, you can focus purely on building the assistant modules.

In this step, get ready to:

• Install and verify Python 3.10+.
• Set up VS Code and create the project folder.
• Create a virtual environment and install all dependencies.

Install Python 3.10+ and verify the installation

The assistant uses TensorFlow and YOLOv8 for its ML capabilities. Both require Python 3.10 or higher to run correctly.

• Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), then type Terminal and press Enter.
• Check your Python version by running this command:

macOS

python3 --version

Windows

python --version

✔️ I see version 3.10 or higher

You are all set. Move on to the next substep.

ⓧ I see an older version

macOS often ships with an older Python version by default. You need to install Python 3.10 or higher alongside it.

• Go to https://www.python.org/downloads and download the latest Python 3.10+ installer for your operating system.
• Run the installer and follow the prompts.
• On Windows, check the box that says Add Python to PATH before clicking Install Now.
• Close and reopen your terminal, then run the version check command again.

macOS users with an older default Python

After installing a newer Python, your system's python3 command may still point to the old version. Try running python3.10 --version or python3.11 --version to confirm the newer version is available.

ⓧ Command not found

Python is not installed on your system yet. Follow these steps to install it.

• Go to https://www.python.org/downloads and download the latest Python 3.10+ installer for your operating system.
• Run the installer and follow the prompts.
• On Windows, check the box that says Add Python to PATH before clicking Install Now.
• Close and reopen your terminal after the installation completes.
• Run the version check command again to confirm it shows 3.10 or higher.

Still seeing the wrong version?

Make sure you completely closed and reopened your terminal after installing Python. Existing terminal sessions cache the old PATH and will not pick up the new installation.

Help me troubleshoot my Python installation

Install VS Code and set up a project folder

VS Code is the code editor you will use to write and organize all your Python files. You will also create a dedicated folder on your Desktop to keep everything in one place.

• If you do not have VS Code installed, download it from https://code.visualstudio.com and run the installer.

Now create your project folder. This is where all your assistant scripts and model files will live.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir smart_vision_assistant
cd smart_vision_assistant

• Confirm the folder was created by running:

pwd

You should see a path ending in smart_vision_assistant.

• Open VS Code by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Visual Studio Code and press Enter to open it.
• In VS Code, click File in the top menu bar.
• Click Open Folder.
• Navigate to your Desktop and select the smart_vision_assistant folder, then click Open.

Don't see the folder on your Desktop?

Make sure you ran cd ~/Desktop before the mkdir command. Without that step, the folder was created wherever your terminal happened to open.

Help me find or recreate my project folder

Create a virtual environment and install dependencies

A virtual environment keeps your project's packages isolated from the rest of your system. This prevents version conflicts between different Python projects on your machine.

• Open a terminal inside VS Code by pressing Ctrl+` (backtick key, located below Escape on most keyboards).
• Create a virtual environment by running this command:

macOS

python3 -m venv venv

Windows

python -m venv venv

• Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

.\venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

Think of it as a clean room for your project's packages. Anything you install with pip while the environment is active stays inside that folder.

If something goes wrong, you can delete the venv folder and start fresh without affecting the rest of your system.

• Install all the project dependencies by running:

pip install ultralytics opencv-python pyttsx3 numpy tensorflow

What are these packages?

• ultralytics provides the YOLOv8 object detection model.
• opencv-python handles webcam capture and image processing.
• pyttsx3 converts text to speech offline, so the assistant works without internet.
• numpy powers numerical operations used by the ML models.
• tensorflow provides the MobileNetV2 model for currency classification.

This installation may take a few minutes, especially for tensorflow. Wait until you see the terminal return to your prompt before continuing.

Linux users need an extra step for audio

The pyttsx3 library requires the espeak engine on Linux. Install it by running:

sudo apt update && sudo apt install espeak-ng libespeak1

On macOS and Windows, the text-to-speech engine works out of the box.

Seeing installation errors?

Make sure you see (venv) in your terminal prompt. If it is missing, the virtual environment is not active and packages will install globally (or fail due to permissions).

If tensorflow fails, confirm your Python version is 3.10 or higher. TensorFlow does not support older Python versions.

Help me fix my pip install errors

• Create the four Python files your project needs by running:

macOS

touch object_detector.py currency_classifier.py distance_sensor.py smart_assistant.py

Windows

New-Item object_detector.py, currency_classifier.py, distance_sensor.py, smart_assistant.py

• Confirm the files exist by running:

ls

You should see object_detector.py, currency_classifier.py, distance_sensor.py, and smart_assistant.py in the output, along with the venv folder.

What are these files for?

• object_detector.py will handle real-time object detection using YOLOv8.
• currency_classifier.py will classify banknote denominations using transfer learning.
• distance_sensor.py will measure obstacle distance with audio warnings.
• smart_assistant.py will bring all three modules together into one unified assistant.

Your development environment is locked and loaded. Next up, you will write your first real module: a real-time object detector that sees through your webcam and announces what it finds.

Build Real-Time Object Detection with YOLOv8

Your environment is set up and ready to go. Now it's time to build the core feature of your Smart Vision Assistant: real-time object detection.

Object detection is how your assistant "sees" the world. Using a pre-trained YOLOv8 model, you'll capture video frames from your webcam, identify objects in each frame, draw bounding boxes around them, and announce what's detected through audio.

In this step, get ready to:

• Download the YOLOv8 nano pre-trained model and run inference on webcam frames.
• Add bounding box visualization and object label overlay on the video feed.
• Integrate text-to-speech to announce detected objects with a cooldown timer.

Download the YOLOv8 nano pre-trained model and write inference code

YOLOv8 nano (yolov8n.pt) is a lightweight object detection model that can identify 80 different object classes. It's only about 6MB, making it fast enough for real-time webcam processing.

• Open object_detector.py in VS Code by clicking on it in the Explorer sidebar.
• Add the following imports and class initialization to set up the YOLO model and text-to-speech engine:

from ultralytics import YOLO
import cv2
import pyttsx3
import time
import threading

class ObjectDetector:
    def __init__(self):
        self.model = YOLO("yolov8n.pt")
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.last_announced = {}
        self.cooldown = 5  # seconds between repeat announcements

What does this code do?

• YOLO("yolov8n.pt") loads the YOLOv8 nano model. The first time you run this, it automatically downloads the model weights from the internet.
• pyttsx3.init() initializes the text-to-speech engine. It works fully offline using your system's built-in speech synthesis.
• last_announced and cooldown work together to prevent the assistant from repeating the same object announcement every frame.

• Save object_detector.py by pressing Cmd+S (macOS) or Ctrl+S (Windows).

Seeing an import error?

If VS Code underlines the imports with red, make sure your virtual environment is selected as the Python interpreter. Click the Python version in the bottom status bar and select the interpreter inside your smart_vision_assistant folder.

help me fix import errors in my object_detector.py file

Add bounding box visualization and object label overlay on the video feed

The detection loop captures frames from your webcam, runs each frame through YOLO, and draws green rectangles around every detected object. Each rectangle also gets a label showing what the object is and how confident the model is.

• Below the __init__ method, add the detect_and_display method by copying this code:

    def detect_and_display(self):
        """Run object detection on webcam feed with audio announcements."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        print("Object Detection Active. Press 'q' to quit.")

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            results = self.model(frame, conf=0.5, verbose=False)

            detected_objects = []
            for result in results:
                for box in result.boxes:
                    x1, y1, x2, y2 = map(int, box.xyxy[0])
                    confidence = float(box.conf[0])
                    class_id = int(box.cls[0])
                    label = self.model.names[class_id]

                    cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
                    text = f"{label} {confidence:.2f}"
                    cv2.putText(frame, text, (x1, y1 - 10),
                                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)

                    if label not in detected_objects:
                        detected_objects.append(label)

            for obj in detected_objects:
                if self.should_announce(obj):
                    self.announce(f"{obj} detected")

            cv2.imshow("Smart Vision Assistant - Object Detection", frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

        cap.release()
        cv2.destroyAllWindows()

What does this code do?

• cv2.VideoCapture(0) opens your default webcam. The 0 refers to the first available camera.
• self.model(frame, conf=0.5, verbose=False) runs YOLO inference on each frame. Only objects detected with 50% or higher confidence are kept.
• cv2.rectangle draws a green bounding box around each detected object. cv2.putText overlays the object label and confidence score above the box.
• The last section calls should_announce and announce for each unique object. We'll add these methods next.

• Save object_detector.py.
• You won't be able to run the script yet. The detect_and_display method calls announce and should_announce which we'll add in the next substep.

Indentation looking off?

Make sure the entire detect_and_display method is indented one level inside the ObjectDetector class. In Python, every method inside a class needs consistent indentation (4 spaces).

help me fix indentation in my object_detector.py

Integrate pyttsx3 to announce detected objects with a cooldown timer

Without a cooldown, the assistant would try to say "person detected" 30 times per second. The cooldown timer ensures each object is only announced once every 5 seconds, keeping the audio useful instead of overwhelming.

• Between the __init__ method and the detect_and_display method, add the announce method:

    def announce(self, text):
        """Announce text in a separate thread to avoid blocking."""
        def speak():
            self.engine.say(text)
            self.engine.runAndWait()
        thread = threading.Thread(target=speak)
        thread.daemon = True
        thread.start()

Why use threading here?

The pyttsx3 engine's runAndWait() blocks the main thread until speech finishes. Without threading, the video feed would freeze every time the assistant speaks.

By running speech in a daemon thread, the webcam continues capturing frames while audio plays in the background.

• Save object_detector.py.
• Directly below the announce method and above detect_and_display, add the should_announce method:

    def should_announce(self, label):
        """Check if enough time has passed to re-announce this object."""
        current_time = time.time()
        if label not in self.last_announced:
            self.last_announced[label] = current_time
            return True
        if current_time - self.last_announced[label] > self.cooldown:
            self.last_announced[label] = current_time
            return True
        return False

What does should_announce do?

This method tracks the last time each object label was spoken. If the same object (like "person") was announced less than 5 seconds ago, it returns False and the assistant stays quiet.

When a new object appears or enough time passes, it returns True and updates the timestamp. This prevents the assistant from being annoyingly repetitive.

• At the very bottom of object_detector.py (outside the class, with no indentation), add the entry point:

if __name__ == "__main__":
    detector = ObjectDetector()
    detector.detect_and_display()

What is __name__ == "__main__"?

This Python pattern means "only run this code if the file is executed directly." When you later import ObjectDetector into the unified assistant script, this block won't execute. It only runs when you call python object_detector.py directly.

• Save object_detector.py.
• Run the object detector by entering this command in your terminal (make sure your virtual environment is active):

python object_detector.py

• You should see a webcam window open titled "Smart Vision Assistant - Object Detection". Green bounding boxes appear around detected objects (people, cups, monitors, chairs). Your computer's speakers announce each new object aloud.
• Press q while the webcam window is focused to quit the script.

First run downloads the model

The very first time you run this script, ultralytics downloads the yolov8n.pt model file (about 6MB). This only happens once. Future runs use the cached file.

Webcam not opening or script crashing?

• If you see "Error: Could not open webcam", check that no other application (Zoom, Teams, FaceTime) is using your camera. Close those apps and try again.
• On macOS, your terminal or VS Code may need camera permissions. Go to System Settings, then Privacy & Security, then Camera to grant access.
• If pyttsx3 gives an error on macOS, you may need to install pyobjc. Run pip install pyobjc in your virtual environment.

help me fix my object_detector.py webcam or audio issues

✔️ Awesome, I've got everything!

Great. Double check you've saved your file and that the webcam window appeared with green bounding boxes around detected objects.

ⓧ I'd like to double check the full code

Here is the complete object_detector.py file. Compare it to yours and fix any differences:

from ultralytics import YOLO
import cv2
import pyttsx3
import time
import threading

class ObjectDetector:
    def __init__(self):
        self.model = YOLO("yolov8n.pt")
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.last_announced = {}
        self.cooldown = 5  # seconds between repeat announcements

    def announce(self, text):
        """Announce text in a separate thread to avoid blocking."""
        def speak():
            self.engine.say(text)
            self.engine.runAndWait()
        thread = threading.Thread(target=speak)
        thread.daemon = True
        thread.start()

    def should_announce(self, label):
        """Check if enough time has passed to re-announce this object."""
        current_time = time.time()
        if label not in self.last_announced:
            self.last_announced[label] = current_time
            return True
        if current_time - self.last_announced[label] > self.cooldown:
            self.last_announced[label] = current_time
            return True
        return False

    def detect_and_display(self):
        """Run object detection on webcam feed with audio announcements."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        print("Object Detection Active. Press 'q' to quit.")

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            results = self.model(frame, conf=0.5, verbose=False)

            detected_objects = []
            for result in results:
                for box in result.boxes:
                    x1, y1, x2, y2 = map(int, box.xyxy[0])
                    confidence = float(box.conf[0])
                    class_id = int(box.cls[0])
                    label = self.model.names[class_id]

                    cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
                    text = f"{label} {confidence:.2f}"
                    cv2.putText(frame, text, (x1, y1 - 10),
                                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)

                    if label not in detected_objects:
                        detected_objects.append(label)

            for obj in detected_objects:
                if self.should_announce(obj):
                    self.announce(f"{obj} detected")

            cv2.imshow("Smart Vision Assistant - Object Detection", frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

        cap.release()
        cv2.destroyAllWindows()


if __name__ == "__main__":
    detector = ObjectDetector()
    detector.detect_and_display()

Your assistant can now see and announce objects in real time. Next up, you'll build a currency recognition module that uses transfer learning to classify banknote denominations from camera images.

Build the Currency Recognition Module

Your object detector can identify 80 different objects in real time. Now it is time to tackle a specific challenge that visually impaired users face every day: identifying currency.

In this step, you will build a currency recognition module using transfer learning. Instead of training a model from scratch (which requires millions of images), you will reuse a pre-trained MobileNetV2 model and teach it to recognize banknote denominations with just a small dataset.

In this step, get ready to:

• Prepare a sample currency dataset folder structure.
• Build a MobileNetV2-based transfer learning classifier for currency denomination recognition.
• Write an inference script that classifies banknotes from camera frames and announces the denomination.

Prepare a sample currency dataset

Transfer learning works best when your training data is organized into folders, where each folder name is a class label (e.g., a denomination like 10, 20, 50). The classifier will learn to map images in each folder to that denomination.

• Create the dataset folder structure inside your smart_vision_assistant folder by running these commands:

mkdir -p dataset/10 dataset/20 dataset/50 dataset/100

What does this folder structure do?

Each subfolder (10/, 20/, 50/, 100/) represents a banknote denomination. When you add images to these folders, the classifier uses the folder name as the label for all images inside it.

You can add your own banknote photos (10-20 per denomination works well for a prototype), or use Microsoft's BankNote-Net open dataset which covers 17 currencies with 112 denominations.

• Confirm the folders exist by running:

ls dataset

You should see 10, 20, 50, and 100 listed as subfolders.

No banknote images yet?

You do not need actual training images to complete this step. The script runs in demo mode without a trained model. You can add images and train later when you have collected photos.

Build the MobileNetV2-based classifier

MobileNetV2 is a lightweight CNN architecture designed for mobile and edge deployment. Microsoft's Seeing AI app for the blind uses this same architecture for currency recognition. By freezing the pre-trained layers and adding your own classification head on top, you get a powerful classifier with minimal training data.

• Create a new file called currency_classifier.py in your smart_vision_assistant folder.
• Add the imports, class definition, and model-building method by copying the following code:

import numpy as np
import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import cv2
import pyttsx3
import os

class CurrencyClassifier:
    def __init__(self, model_path=None, class_names=None):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.img_size = (224, 224)

        if model_path and os.path.exists(model_path):
            self.model = tf.keras.models.load_model(model_path)
            self.class_names = class_names or []
        else:
            self.model = None
            self.class_names = []

    def build_model(self, num_classes):
        """Build a MobileNetV2-based transfer learning model."""
        base_model = MobileNetV2(
            weights='imagenet',
            include_top=False,
            input_shape=(224, 224, 3)
        )
        base_model.trainable = False

        x = base_model.output
        x = GlobalAveragePooling2D()(x)
        x = Dense(128, activation='relu')(x)
        x = Dropout(0.5)(x)
        predictions = Dense(num_classes, activation='softmax')(x)

        model = Model(inputs=base_model.input, outputs=predictions)
        model.compile(
            optimizer='adam',
            loss='categorical_crossentropy',
            metrics=['accuracy']
        )
        self.model = model
        return model

What does this code do?

The __init__ method sets up the text-to-speech engine and optionally loads a previously trained model. If no model file exists, the classifier starts empty (ready for training or demo mode).

The build_model() method creates the transfer learning architecture. MobileNetV2(weights='imagenet', include_top=False) loads the pre-trained feature extractor without its original classification layer. Setting trainable = False freezes these learned features so they are not overwritten during training.

On top, a new classification head is added: GlobalAveragePooling2D compresses the feature maps, Dense(128) learns denomination-specific patterns, Dropout(0.5) prevents overfitting, and the final Dense(num_classes, softmax) outputs a probability for each denomination.

• Save currency_classifier.py.
• Verify the imports work by running this command in your terminal:

python -c "from currency_classifier import CurrencyClassifier; print('Import successful')"

You should see Import successful printed in your terminal. TensorFlow may also print some informational messages about GPU availability.

Seeing an import error?

If you see an error about tensorflow not being found, make sure your virtual environment is activated. You should see the environment name in your terminal prompt.

If pyttsx3 gives an error on Linux, you may need to install the espeak backend.

help me fix my TensorFlow import error in currency_classifier.py

Now add the training method that teaches the model to recognize denominations from your dataset.

• In currency_classifier.py, add the following train() method inside the CurrencyClassifier class, below the build_model() method:

    def train(self, data_dir, epochs=10, batch_size=16):
        """Train the currency classifier on a dataset directory."""
        train_datagen = ImageDataGenerator(
            rescale=1.0/255,
            rotation_range=20,
            width_shift_range=0.2,
            height_shift_range=0.2,
            horizontal_flip=True,
            validation_split=0.2
        )

        train_generator = train_datagen.flow_from_directory(
            data_dir,
            target_size=self.img_size,
            batch_size=batch_size,
            class_mode='categorical',
            subset='training'
        )

        val_generator = train_datagen.flow_from_directory(
            data_dir,
            target_size=self.img_size,
            batch_size=batch_size,
            class_mode='categorical',
            subset='validation'
        )

        self.class_names = list(train_generator.class_indices.keys())
        num_classes = len(self.class_names)

        if self.model is None:
            self.build_model(num_classes)

        self.model.fit(
            train_generator,
            epochs=epochs,
            validation_data=val_generator
        )

        self.model.save("currency_model.h5")
        print(f"Model saved. Classes: {self.class_names}")

What does the train method do?

ImageDataGenerator handles loading images from your dataset folders and applying data augmentation (rotation, shifting, flipping). Augmentation artificially increases dataset diversity so the model generalizes better from a small number of photos.

The validation_split=0.2 reserves 20% of images for validation, so you can track whether the model is actually learning or just memorizing. After training, the model is saved to currency_model.h5 for later use.

• Save currency_classifier.py.

You will not see a visible change yet. The next section adds the inference code that brings the camera feed to life.

Write the inference and camera classification script

The model architecture and training pipeline are ready. Now you need the code that takes a live camera frame, feeds it through the model, and announces the detected denomination aloud.

• In currency_classifier.py, add the classify_frame() and announce_currency() methods below the train() method:

    def classify_frame(self, frame):
        """Classify a currency note from a camera frame."""
        if self.model is None:
            return "No model loaded", 0.0

        img = cv2.resize(frame, self.img_size)
        img = img.astype('float32') / 255.0
        img = np.expand_dims(img, axis=0)

        predictions = self.model.predict(img, verbose=0)
        class_idx = np.argmax(predictions[0])
        confidence = predictions[0][class_idx]

        if confidence > 0.7 and class_idx < len(self.class_names):
            return self.class_names[class_idx], confidence
        return "Unknown", confidence

    def announce_currency(self, denomination):
        """Announce the detected currency denomination."""
        self.engine.say(f"This is a {denomination} note")
        self.engine.runAndWait()

What does classify_frame do?

The method resizes the camera frame to 224x224 pixels (the input size MobileNetV2 expects), normalizes pixel values to 0-1 range, and passes it through the model. If the model's confidence exceeds 0.7 (70%), it returns the denomination name. Otherwise it returns "Unknown" to avoid false positives.

The announce_currency() method uses pyttsx3 to speak the detected denomination aloud, giving the visually impaired user immediate audio feedback.

• Now add the camera loop and main entry point. Paste the following code below announce_currency(), still inside the class:

    def run_camera_classification(self):
        """Run currency classification on webcam feed."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        print("Currency Recognition Active. Press 'c' to capture, 'q' to quit.")

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            cv2.putText(frame, "Press 'c' to classify currency",
                        (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 0), 2)
            cv2.imshow("Smart Vision Assistant - Currency Mode", frame)

            key = cv2.waitKey(1) & 0xFF
            if key == ord('c'):
                denomination, confidence = self.classify_frame(frame)
                print(f"Detected: {denomination} (confidence: {confidence:.2f})")
                if denomination != "Unknown":
                    self.announce_currency(denomination)
            elif key == ord('q'):
                break

        cap.release()
        cv2.destroyAllWindows()

• Finally, add the main entry point at the bottom of the file, outside the class (no indentation):

if __name__ == "__main__":
    classifier = CurrencyClassifier()
    print("To train: prepare a dataset folder with subfolders for each denomination")
    print("Example structure: dataset/10/ dataset/20/ dataset/50/ dataset/100/")
    print("\nRunning in demo mode (no trained model)...")
    classifier.run_camera_classification()

What does run_camera_classification do?

This method opens your webcam and displays a live feed with an instruction overlay. When you press the c key, it captures the current frame and runs it through the classifier. If a denomination is recognized with high confidence, it announces the result aloud.

The __main__ block runs the classifier in demo mode. Without a trained model, pressing c will return "No model loaded". Once you add images to your dataset folders and call the train() method, the classifier will start recognizing denominations.

• Save currency_classifier.py.
• Run the currency classifier in demo mode:

python currency_classifier.py

You should see your webcam feed open in a window titled "Smart Vision Assistant - Currency Mode" with yellow text saying "Press 'c' to classify currency". Press q to close the window.

Webcam window not appearing?

Make sure no other application is using your webcam (close your object detector if it is still running).

If you see "Error: Could not open webcam", check that your webcam is connected and not blocked by system privacy settings.

help me fix my webcam not opening in currency_classifier.py

✔️ Awesome, I've got everything!

Great. Double check you have saved currency_classifier.py before moving on.

ⓧ I'd like to double check the full code

import numpy as np
import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import cv2
import pyttsx3
import os

class CurrencyClassifier:
    def __init__(self, model_path=None, class_names=None):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.img_size = (224, 224)

        if model_path and os.path.exists(model_path):
            self.model = tf.keras.models.load_model(model_path)
            self.class_names = class_names or []
        else:
            self.model = None
            self.class_names = []

    def build_model(self, num_classes):
        """Build a MobileNetV2-based transfer learning model."""
        base_model = MobileNetV2(
            weights='imagenet',
            include_top=False,
            input_shape=(224, 224, 3)
        )
        base_model.trainable = False

        x = base_model.output
        x = GlobalAveragePooling2D()(x)
        x = Dense(128, activation='relu')(x)
        x = Dropout(0.5)(x)
        predictions = Dense(num_classes, activation='softmax')(x)

        model = Model(inputs=base_model.input, outputs=predictions)
        model.compile(
            optimizer='adam',
            loss='categorical_crossentropy',
            metrics=['accuracy']
        )
        self.model = model
        return model

    def train(self, data_dir, epochs=10, batch_size=16):
        """Train the currency classifier on a dataset directory."""
        train_datagen = ImageDataGenerator(
            rescale=1.0/255,
            rotation_range=20,
            width_shift_range=0.2,
            height_shift_range=0.2,
            horizontal_flip=True,
            validation_split=0.2
        )

        train_generator = train_datagen.flow_from_directory(
            data_dir,
            target_size=self.img_size,
            batch_size=batch_size,
            class_mode='categorical',
            subset='training'
        )

        val_generator = train_datagen.flow_from_directory(
            data_dir,
            target_size=self.img_size,
            batch_size=batch_size,
            class_mode='categorical',
            subset='validation'
        )

        self.class_names = list(train_generator.class_indices.keys())
        num_classes = len(self.class_names)

        if self.model is None:
            self.build_model(num_classes)

        self.model.fit(
            train_generator,
            epochs=epochs,
            validation_data=val_generator
        )

        self.model.save("currency_model.h5")
        print(f"Model saved. Classes: {self.class_names}")

    def classify_frame(self, frame):
        """Classify a currency note from a camera frame."""
        if self.model is None:
            return "No model loaded", 0.0

        img = cv2.resize(frame, self.img_size)
        img = img.astype('float32') / 255.0
        img = np.expand_dims(img, axis=0)

        predictions = self.model.predict(img, verbose=0)
        class_idx = np.argmax(predictions[0])
        confidence = predictions[0][class_idx]

        if confidence > 0.7 and class_idx < len(self.class_names):
            return self.class_names[class_idx], confidence
        return "Unknown", confidence

    def announce_currency(self, denomination):
        """Announce the detected currency denomination."""
        self.engine.say(f"This is a {denomination} note")
        self.engine.runAndWait()

    def run_camera_classification(self):
        """Run currency classification on webcam feed."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        print("Currency Recognition Active. Press 'c' to capture, 'q' to quit.")

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            cv2.putText(frame, "Press 'c' to classify currency",
                        (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 0), 2)
            cv2.imshow("Smart Vision Assistant - Currency Mode", frame)

            key = cv2.waitKey(1) & 0xFF
            if key == ord('c'):
                denomination, confidence = self.classify_frame(frame)
                print(f"Detected: {denomination} (confidence: {confidence:.2f})")
                if denomination != "Unknown":
                    self.announce_currency(denomination)
            elif key == ord('q'):
                break

        cap.release()
        cv2.destroyAllWindows()


if __name__ == "__main__":
    classifier = CurrencyClassifier()
    print("To train: prepare a dataset folder with subfolders for each denomination")
    print("Example structure: dataset/10/ dataset/20/ dataset/50/ dataset/100/")
    print("\nRunning in demo mode (no trained model)...")
    classifier.run_camera_classification()

Your currency recognition module is ready to classify banknotes the moment you train it with real images. Next up, you will build the distance measurement module that warns users about nearby obstacles.

Build the Distance Measurement Module

Your assistant can see objects and recognize currency. Now it needs spatial awareness to warn users about obstacles in their path.

In a real deployment, an ultrasonic sensor measures the distance to nearby objects. For our PC prototype, you will build a simulated sensor that generates distance readings, then layer on proximity-based audio alerts. You will also document the hardware wiring so the module is ready for Raspberry Pi deployment later.

In this step, get ready to:

• Write a simulated distance sensor module for PC testing.
• Implement proximity-based audio alerts with urgency levels.
• Document the HC-SR04 ultrasonic sensor wiring for Raspberry Pi deployment.

Write the simulated distance sensor module

A visually impaired user needs to know when obstacles are nearby. On hardware, an ultrasonic sensor sends a pulse and measures how long it takes to bounce back. On your PC, you will simulate those readings so you can test the full alert pipeline without any hardware.

• Create a new file called distance_sensor.py in your smart_vision_assistant folder.
• Add the imports and class initializer by pasting this code:

import pyttsx3
import time
import random
import threading

class DistanceSensor:
    """
    Distance measurement module.
    On PC: Uses simulated readings for testing.
    On Raspberry Pi: Uses HC-SR04 ultrasonic sensor via RPi.GPIO.
    """

    def __init__(self, use_simulation=True):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 160)
        self.use_simulation = use_simulation
        self.running = False

        # Distance thresholds in centimeters
        self.danger_zone = 30    # Very close - immediate warning
        self.warning_zone = 100  # Close - caution
        self.notice_zone = 200   # Moderate - awareness

        if not use_simulation:
            try:
                import RPi.GPIO as GPIO
                self.GPIO = GPIO
                self.TRIG = 23
                self.ECHO = 24
                self.setup_gpio()
            except ImportError:
                print("RPi.GPIO not available. Falling back to simulation.")
                self.use_simulation = True

What does this code do?

The DistanceSensor class initializes with use_simulation=True by default. This means it generates mock distance readings on your PC without any hardware.

Three distance thresholds define urgency zones: danger_zone at 30cm, warning_zone at 100cm, and notice_zone at 200cm. If running on a Raspberry Pi, it attempts to import the RPi.GPIO library and falls back to simulation if unavailable.

• Add the distance measurement methods below the __init__ method:

    def setup_gpio(self):
        """Set up GPIO pins for HC-SR04 sensor on Raspberry Pi."""
        self.GPIO.setmode(self.GPIO.BCM)
        self.GPIO.setup(self.TRIG, self.GPIO.OUT)
        self.GPIO.setup(self.ECHO, self.GPIO.IN)
        self.GPIO.output(self.TRIG, False)
        time.sleep(2)

    def get_distance_simulated(self):
        """Return a simulated distance reading for PC testing."""
        return random.uniform(10, 400)

    def get_distance_hcsr04(self):
        """Get distance reading from HC-SR04 ultrasonic sensor."""
        self.GPIO.output(self.TRIG, True)
        time.sleep(0.00001)
        self.GPIO.output(self.TRIG, False)

        start_time = time.time()
        stop_time = time.time()

        while self.GPIO.input(self.ECHO) == 0:
            start_time = time.time()

        while self.GPIO.input(self.ECHO) == 1:
            stop_time = time.time()

        elapsed = stop_time - start_time
        distance = elapsed * 17150
        distance = round(distance, 2)

        return distance

    def get_distance(self):
        """Get distance reading from the appropriate source."""
        if self.use_simulation:
            return self.get_distance_simulated()
        else:
            return self.get_distance_hcsr04()

What does this code do?

These four methods handle distance measurement from both sources:

• setup_gpio() configures the Raspberry Pi pins for the ultrasonic sensor.
• get_distance_simulated() returns a random value between 10 and 400 centimeters, mimicking real sensor noise.
• get_distance_hcsr04() sends an ultrasonic pulse and calculates distance from the echo time using the formula elapsed * 17150 (speed of sound divided by 2 for the round trip).
• get_distance() dispatches to the correct source based on the simulation flag.

• Save distance_sensor.py.

Seeing a syntax error?

Make sure the methods inside the class are indented with 4 spaces. Each def keyword should align with the def __init__ line above.

Help me find the indentation issue

Implement proximity-based audio alerts

Distance readings alone are not useful to a visually impaired user. The assistant needs to translate numbers into spoken warnings with different urgency levels.

• Add the alert logic and monitoring loop below the get_distance method in distance_sensor.py:

    def get_alert_level(self, distance):
        """Determine alert level based on distance."""
        if distance <= self.danger_zone:
            return "danger", f"Warning! Obstacle very close at {distance:.0f} centimeters"
        elif distance <= self.warning_zone:
            return "warning", f"Caution. Obstacle at {distance:.0f} centimeters"
        elif distance <= self.notice_zone:
            return "notice", f"Object detected at {distance:.0f} centimeters"
        else:
            return "clear", ""

    def announce(self, text):
        """Announce alert via text-to-speech."""
        self.engine.say(text)
        self.engine.runAndWait()

What does this code do?

The get_alert_level method maps distance values to four urgency tiers. Anything within 30cm triggers a danger alert. Between 30-100cm is a warning. Between 100-200cm is a notice. Beyond 200cm is clear (no announcement).

Each tier returns a tuple of the alert level name and a spoken message that includes the actual distance. This gives the user both urgency context and precise information.

• Add the continuous monitoring loop, cleanup method, and main block below the announce method:

    def monitor_distance(self, interval=1.0):
        """Continuously monitor distance and provide audio alerts."""
        self.running = True
        print("Distance Monitoring Active. Press Ctrl+C to stop.")
        print(f"Thresholds - Danger: {self.danger_zone}cm, "
              f"Warning: {self.warning_zone}cm, Notice: {self.notice_zone}cm")

        last_alert_level = "clear"

        try:
            while self.running:
                distance = self.get_distance()
                alert_level, message = self.get_alert_level(distance)

                print(f"Distance: {distance:.1f} cm | Alert: {alert_level}")

                if alert_level != "clear" and alert_level != last_alert_level:
                    self.announce(message)

                last_alert_level = alert_level
                time.sleep(interval)

        except KeyboardInterrupt:
            print("\nDistance monitoring stopped.")
        finally:
            if not self.use_simulation:
                self.GPIO.cleanup()

    def cleanup(self):
        """Clean up GPIO resources."""
        self.running = False
        if not self.use_simulation:
            self.GPIO.cleanup()


if __name__ == "__main__":
    sensor = DistanceSensor(use_simulation=True)
    sensor.monitor_distance(interval=2.0)

What does this code do?

The monitor_distance method runs a continuous loop that reads the sensor at a configurable interval. It only announces when the alert level changes (not on every reading), which prevents repetitive speech from overwhelming the user.

The last_alert_level variable tracks state between readings. If the user moves from clear to warning, they hear a caution. If they stay at the same level, no repeat announcement fires. KeyboardInterrupt handling ensures clean shutdown with Ctrl+C.

• Save distance_sensor.py.
• Test the distance module by running this command in your terminal:

python distance_sensor.py

You should see distance readings printing every 2 seconds with alert levels. Some readings will trigger audio announcements when the simulated distance falls within a threshold zone. Press Ctrl+C to stop.

Not hearing any audio alerts?

The simulated readings are random. If all readings happen to land above 200cm (the notice zone), no audio plays. Run the script a few times to get readings in the lower ranges.

On Linux, pyttsx3 requires espeak-ng installed. If you hear nothing on any run, verify espeak is available on your system.

Help me troubleshoot text-to-speech

Document the HC-SR04 wiring for Raspberry Pi deployment

The code you just wrote already includes the full Raspberry Pi implementation. Now let's document the hardware wiring so this module is ready for physical deployment.

What is the HC-SR04 ultrasonic sensor?

The HC-SR04 is an ultrasonic distance sensor that measures from 2cm to 400cm with approximately 3mm accuracy. It sends a 40kHz sound pulse from its trigger pin and listens for the echo on a separate pin. The time between trigger and echo determines the distance.

The sensor operates at 5V, but the Raspberry Pi GPIO pins only tolerate 3.3V. Connecting the Echo pin directly would damage the Pi. A voltage divider circuit using resistors drops the voltage to a safe level.

HC-SR04 wiring to Raspberry Pi

Here is the pin connection guide for deploying this module on a Raspberry Pi:

• VCC (sensor) connects to 5V (Pi pin 2).
• GND (sensor) connects to GND (Pi pin 6).
• TRIG (sensor) connects directly to GPIO 23 (BCM numbering).
• ECHO (sensor) connects through a voltage divider to GPIO 24 (BCM numbering).

The voltage divider circuit for the Echo pin uses a 1kOhm resistor from the Echo pin to GPIO 24, and a 2kOhm resistor from GPIO 24 to ground. This reduces the 5V echo signal to approximately 3.3V, protecting the Pi.

How would I switch to real hardware?

To deploy on a Raspberry Pi, change the initialization to DistanceSensor(use_simulation=False). The class automatically attempts to import RPi.GPIO and configure pins 23 and 24. If the library is not installed, it gracefully falls back to simulation mode.

The get_distance_hcsr04 method sends a 10-microsecond pulse on the trigger pin, then times how long the echo pin stays high. The formula elapsed * 17150 converts that time to centimeters using the speed of sound (34300 cm/s divided by 2 for the round trip).

✔️ Awesome, I've got everything!

Great. Double check you have saved distance_sensor.py and confirmed it runs with simulated readings.

ⓧ I'd like to double check the full code

import pyttsx3
import time
import random
import threading

class DistanceSensor:
    """
    Distance measurement module.
    On PC: Uses simulated readings for testing.
    On Raspberry Pi: Uses HC-SR04 ultrasonic sensor via RPi.GPIO.
    """

    def __init__(self, use_simulation=True):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 160)
        self.use_simulation = use_simulation
        self.running = False

        # Distance thresholds in centimeters
        self.danger_zone = 30    # Very close - immediate warning
        self.warning_zone = 100  # Close - caution
        self.notice_zone = 200   # Moderate - awareness

        if not use_simulation:
            try:
                import RPi.GPIO as GPIO
                self.GPIO = GPIO
                self.TRIG = 23
                self.ECHO = 24
                self.setup_gpio()
            except ImportError:
                print("RPi.GPIO not available. Falling back to simulation.")
                self.use_simulation = True

    def setup_gpio(self):
        """Set up GPIO pins for HC-SR04 sensor on Raspberry Pi."""
        self.GPIO.setmode(self.GPIO.BCM)
        self.GPIO.setup(self.TRIG, self.GPIO.OUT)
        self.GPIO.setup(self.ECHO, self.GPIO.IN)
        self.GPIO.output(self.TRIG, False)
        time.sleep(2)

    def get_distance_simulated(self):
        """Return a simulated distance reading for PC testing."""
        return random.uniform(10, 400)

    def get_distance_hcsr04(self):
        """Get distance reading from HC-SR04 ultrasonic sensor."""
        self.GPIO.output(self.TRIG, True)
        time.sleep(0.00001)
        self.GPIO.output(self.TRIG, False)

        start_time = time.time()
        stop_time = time.time()

        while self.GPIO.input(self.ECHO) == 0:
            start_time = time.time()

        while self.GPIO.input(self.ECHO) == 1:
            stop_time = time.time()

        elapsed = stop_time - start_time
        distance = elapsed * 17150
        distance = round(distance, 2)

        return distance

    def get_distance(self):
        """Get distance reading from the appropriate source."""
        if self.use_simulation:
            return self.get_distance_simulated()
        else:
            return self.get_distance_hcsr04()

    def get_alert_level(self, distance):
        """Determine alert level based on distance."""
        if distance <= self.danger_zone:
            return "danger", f"Warning! Obstacle very close at {distance:.0f} centimeters"
        elif distance <= self.warning_zone:
            return "warning", f"Caution. Obstacle at {distance:.0f} centimeters"
        elif distance <= self.notice_zone:
            return "notice", f"Object detected at {distance:.0f} centimeters"
        else:
            return "clear", ""

    def announce(self, text):
        """Announce alert via text-to-speech."""
        self.engine.say(text)
        self.engine.runAndWait()

    def monitor_distance(self, interval=1.0):
        """Continuously monitor distance and provide audio alerts."""
        self.running = True
        print("Distance Monitoring Active. Press Ctrl+C to stop.")
        print(f"Thresholds - Danger: {self.danger_zone}cm, "
              f"Warning: {self.warning_zone}cm, Notice: {self.notice_zone}cm")

        last_alert_level = "clear"

        try:
            while self.running:
                distance = self.get_distance()
                alert_level, message = self.get_alert_level(distance)

                print(f"Distance: {distance:.1f} cm | Alert: {alert_level}")

                if alert_level != "clear" and alert_level != last_alert_level:
                    self.announce(message)

                last_alert_level = alert_level
                time.sleep(interval)

        except KeyboardInterrupt:
            print("\nDistance monitoring stopped.")
        finally:
            if not self.use_simulation:
                self.GPIO.cleanup()

    def cleanup(self):
        """Clean up GPIO resources."""
        self.running = False
        if not self.use_simulation:
            self.GPIO.cleanup()


if __name__ == "__main__":
    sensor = DistanceSensor(use_simulation=True)
    sensor.monitor_distance(interval=2.0)

Your distance measurement module is complete. It works in simulation mode on your PC today, and it is wired up for real hardware deployment when you are ready to move to a Raspberry Pi. Next up, you will bring all three modules together into a unified assistant with mode switching.

Integrate All Modules into the Unified Assistant

You now have three independent modules that each solve one piece of the assistive technology puzzle. The object detector finds things. The currency classifier identifies money. The distance sensor warns about obstacles.

A visually impaired user needs all three working together in one seamless experience. In this step, you'll create a unified assistant script that coordinates all modules and lets the user switch between modes with a single keypress.

In this step, get ready to:

• Create a main assistant script that initializes and coordinates all three modules.
• Add keyboard-driven mode switching between navigation, currency, and identify modes.
• Test the full pipeline end-to-end with your webcam and audio output.

Create the main assistant script

The unified assistant needs to import all three modules, initialize them, and run a single webcam loop that delegates to the active mode.

• Create a new file called smart_assistant.py in your smart_vision_assistant folder.
• Add the imports and the class initialization by copying this code into the file:

from object_detector import ObjectDetector
from currency_classifier import CurrencyClassifier
from distance_sensor import DistanceSensor
import cv2
import pyttsx3
import threading
import time

class SmartVisionAssistant:
    def __init__(self):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.mode = "navigation"  # navigation, currency, identify
        self.running = True

        # Initialize modules
        self.object_detector = ObjectDetector()
        self.currency_classifier = CurrencyClassifier()
        self.distance_sensor = DistanceSensor(use_simulation=True)

    def announce(self, text):
        """Announce text via TTS."""
        self.engine.say(text)
        self.engine.runAndWait()

    def switch_mode(self, new_mode):
        """Switch between assistant modes."""
        self.mode = new_mode
        self.announce(f"Switching to {new_mode} mode")
        print(f"Mode changed to: {new_mode}")

What does this code do?

The SmartVisionAssistant class acts as the central controller. It creates instances of all three modules you built in previous steps.

The switch_mode() method changes the active mode and announces the switch through the speaker. This audio confirmation is critical for a visually impaired user who cannot see the on-screen mode indicator.

Now you need the main loop that captures webcam frames and dispatches them to the active mode.

• Add the run() method directly below switch_mode() in the same class:

    def run(self):
        """Run the unified smart vision assistant."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        self.announce("Smart Vision Assistant activated. Navigation mode.")
        print("\n=== Smart Vision Assistant ===")
        print("Controls:")
        print("  'n' - Navigation mode (object detection + distance)")
        print("  'c' - Currency recognition mode")
        print("  'i' - Identify mode (detailed object detection)")
        print("  'q' - Quit")
        print("=============================\n")

        while self.running:
            ret, frame = cap.read()
            if not ret:
                break

            # Display current mode
            cv2.putText(frame, f"Mode: {self.mode.upper()}",
                        (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 255), 2)

            if self.mode == "navigation":
                frame = self.navigation_mode(frame)
            elif self.mode == "currency":
                frame = self.currency_mode(frame)
            elif self.mode == "identify":
                frame = self.identify_mode(frame)

            cv2.imshow("Smart Vision Assistant", frame)

            key = cv2.waitKey(1) & 0xFF
            if key == ord('n'):
                self.switch_mode("navigation")
            elif key == ord('c'):
                self.switch_mode("currency")
            elif key == ord('i'):
                self.switch_mode("identify")
            elif key == ord('q'):
                self.running = False

        cap.release()
        cv2.destroyAllWindows()
        self.announce("Smart Vision Assistant deactivated")

What does this code do?

The run() method is the main event loop. It opens the webcam, reads frames continuously, and routes each frame to the currently active mode method.

Keyboard input is captured with cv2.waitKey(1) on every frame. Pressing n, c, or i switches modes instantly. The mode label is overlaid on the video feed so sighted observers can see which mode is active.

Add mode-switching handlers

Each mode processes the camera frame differently. Navigation mode combines object detection with distance sensing. Currency mode waits for a capture command. Identify mode runs a more sensitive detection sweep.

• Add the navigation_mode() method below run():

    def navigation_mode(self, frame):
        """Navigation mode: detect large obstacles and measure distance."""
        results = self.object_detector.model(frame, conf=0.5, verbose=False)

        for result in results:
            for box in result.boxes:
                x1, y1, x2, y2 = map(int, box.xyxy[0])
                class_id = int(box.cls[0])
                label = self.object_detector.model.names[class_id]

                cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
                cv2.putText(frame, label, (x1, y1 - 10),
                            cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)

                if self.object_detector.should_announce(label):
                    self.object_detector.announce(f"{label} ahead")

        # Simulated distance reading
        distance = self.distance_sensor.get_distance()
        alert_level, message = self.distance_sensor.get_alert_level(distance)
        color = (0, 0, 255) if alert_level == "danger" else (0, 255, 255)
        cv2.putText(frame, f"Distance: {distance:.0f}cm ({alert_level})",
                    (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.6, color, 2)

        return frame

What does this code do?

Navigation mode is the primary assistive mode. It runs YOLOv8 at confidence 0.5 to detect objects and announces them with the word "ahead" appended (e.g., "chair ahead").

It also reads the simulated distance sensor on every frame and overlays the reading on the video. If an obstacle is in the danger zone, the distance text turns red.

• Add the currency_mode(), identify_mode(), and the entry point block below navigation_mode():

    def currency_mode(self, frame):
        """Currency recognition mode."""
        cv2.putText(frame, "Hold currency note to camera. Press 'space' to scan.",
                    (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 0), 2)
        return frame

    def identify_mode(self, frame):
        """Detailed identification mode."""
        results = self.object_detector.model(frame, conf=0.3, verbose=False)

        detected = []
        for result in results:
            for box in result.boxes:
                x1, y1, x2, y2 = map(int, box.xyxy[0])
                confidence = float(box.conf[0])
                class_id = int(box.cls[0])
                label = self.object_detector.model.names[class_id]

                cv2.rectangle(frame, (x1, y1), (x2, y2), (255, 0, 255), 2)
                text = f"{label} ({confidence:.0%})"
                cv2.putText(frame, text, (x1, y1 - 10),
                            cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 255), 2)
                detected.append(label)

        if detected:
            count_text = f"Objects: {len(detected)}"
            cv2.putText(frame, count_text, (10, 60),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 255, 0), 2)

        return frame


if __name__ == "__main__":
    assistant = SmartVisionAssistant()
    assistant.run()

What does this code do?

Currency mode displays an instruction overlay and waits for the user to press space to capture. This is intentionally simple because the user needs a still frame for accurate classification.

Identify mode lowers the confidence threshold to 0.3, catching more objects at the cost of some false positives. It draws bounding boxes in magenta to visually distinguish it from navigation mode. The object count is shown in the top-left corner.

• Save smart_assistant.py.

Seeing a red underline on the imports?

If VS Code shows an import error for object_detector, currency_classifier, or distance_sensor, make sure all four .py files are in the same folder. These are local imports that Python resolves at runtime.

Help me fix import errors in my smart_assistant.py file.

✔️ Awesome, I've got everything!

Great. Make sure you have saved smart_assistant.py before moving on to the test.

ⓧ I'd like to double check the full code

from object_detector import ObjectDetector
from currency_classifier import CurrencyClassifier
from distance_sensor import DistanceSensor
import cv2
import pyttsx3
import threading
import time

class SmartVisionAssistant:
    def __init__(self):
        self.engine = pyttsx3.init()
        self.engine.setProperty('rate', 150)
        self.mode = "navigation"  # navigation, currency, identify
        self.running = True

        # Initialize modules
        self.object_detector = ObjectDetector()
        self.currency_classifier = CurrencyClassifier()
        self.distance_sensor = DistanceSensor(use_simulation=True)

    def announce(self, text):
        """Announce text via TTS."""
        self.engine.say(text)
        self.engine.runAndWait()

    def switch_mode(self, new_mode):
        """Switch between assistant modes."""
        self.mode = new_mode
        self.announce(f"Switching to {new_mode} mode")
        print(f"Mode changed to: {new_mode}")

    def run(self):
        """Run the unified smart vision assistant."""
        cap = cv2.VideoCapture(0)
        if not cap.isOpened():
            print("Error: Could not open webcam")
            return

        self.announce("Smart Vision Assistant activated. Navigation mode.")
        print("\n=== Smart Vision Assistant ===")
        print("Controls:")
        print("  'n' - Navigation mode (object detection + distance)")
        print("  'c' - Currency recognition mode")
        print("  'i' - Identify mode (detailed object detection)")
        print("  'q' - Quit")
        print("=============================\n")

        while self.running:
            ret, frame = cap.read()
            if not ret:
                break

            # Display current mode
            cv2.putText(frame, f"Mode: {self.mode.upper()}",
                        (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 255), 2)

            if self.mode == "navigation":
                frame = self.navigation_mode(frame)
            elif self.mode == "currency":
                frame = self.currency_mode(frame)
            elif self.mode == "identify":
                frame = self.identify_mode(frame)

            cv2.imshow("Smart Vision Assistant", frame)

            key = cv2.waitKey(1) & 0xFF
            if key == ord('n'):
                self.switch_mode("navigation")
            elif key == ord('c'):
                self.switch_mode("currency")
            elif key == ord('i'):
                self.switch_mode("identify")
            elif key == ord('q'):
                self.running = False

        cap.release()
        cv2.destroyAllWindows()
        self.announce("Smart Vision Assistant deactivated")

    def navigation_mode(self, frame):
        """Navigation mode: detect large obstacles and measure distance."""
        results = self.object_detector.model(frame, conf=0.5, verbose=False)

        for result in results:
            for box in result.boxes:
                x1, y1, x2, y2 = map(int, box.xyxy[0])
                class_id = int(box.cls[0])
                label = self.object_detector.model.names[class_id]

                cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
                cv2.putText(frame, label, (x1, y1 - 10),
                            cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)

                if self.object_detector.should_announce(label):
                    self.object_detector.announce(f"{label} ahead")

        # Simulated distance reading
        distance = self.distance_sensor.get_distance()
        alert_level, message = self.distance_sensor.get_alert_level(distance)
        color = (0, 0, 255) if alert_level == "danger" else (0, 255, 255)
        cv2.putText(frame, f"Distance: {distance:.0f}cm ({alert_level})",
                    (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.6, color, 2)

        return frame

    def currency_mode(self, frame):
        """Currency recognition mode."""
        cv2.putText(frame, "Hold currency note to camera. Press 'space' to scan.",
                    (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 0), 2)
        return frame

    def identify_mode(self, frame):
        """Detailed identification mode."""
        results = self.object_detector.model(frame, conf=0.3, verbose=False)

        detected = []
        for result in results:
            for box in result.boxes:
                x1, y1, x2, y2 = map(int, box.xyxy[0])
                confidence = float(box.conf[0])
                class_id = int(box.cls[0])
                label = self.object_detector.model.names[class_id]

                cv2.rectangle(frame, (x1, y1), (x2, y2), (255, 0, 255), 2)
                text = f"{label} ({confidence:.0%})"
                cv2.putText(frame, text, (x1, y1 - 10),
                            cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 255), 2)
                detected.append(label)

        if detected:
            count_text = f"Objects: {len(detected)}"
            cv2.putText(frame, count_text, (10, 60),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 255, 0), 2)

        return frame


if __name__ == "__main__":
    assistant = SmartVisionAssistant()
    assistant.run()

Test the full pipeline end-to-end

With all four files in place, you can now launch the unified assistant and verify that each mode works correctly with your webcam.

• Make sure your virtual environment is activated, then run the assistant by entering this command:

python smart_assistant.py

What should I see?

Your terminal prints the controls menu. A webcam window opens with "Mode: NAVIGATION" in cyan text at the top. You hear the audio announcement "Smart Vision Assistant activated. Navigation mode." through your speakers.

Green bounding boxes appear around detected objects. A distance reading with the alert level is shown below the mode label. Objects are announced aloud with the word "ahead" (e.g., "person ahead").

• Press i on your keyboard while the video window is focused.
• Confirm you hear "Switching to identify mode" and see magenta bounding boxes with confidence percentages.
• Press c to switch to currency mode.
• Confirm you see the instruction text "Hold currency note to camera. Press 'space' to scan." overlaid on the feed.
• Press n to return to navigation mode.
• Press q to quit. You should hear "Smart Vision Assistant deactivated" before the window closes.

Webcam not opening or no audio?

If you see "Error: Could not open webcam", make sure no other application (like a video call) is using the camera. Close other apps and try again.

If video works but you hear no audio, check that your system volume is turned up. On Linux, make sure espeak-ng is installed.

Help me debug my Smart Vision Assistant not working.

Your Smart Vision Assistant is fully operational. You have object detection, currency recognition, distance sensing, and mode switching all working together in a single coordinated system. Next up, extend this into a navigation mode that gives directional audio cues.

Secret mission

Navigation Mode with Directional Audio Cues

Your assistant can detect objects and measure distance separately. But what if it could combine both to tell you exactly where obstacles are relative to you? In this secret mission, you'll build a navigation mode that divides the camera frame into zones and gives directional audio cues like "Warning: chair on your left, very close."

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your machine, so there are no ongoing cloud costs. Decide whether to keep your resources, pause them to come back later, or delete them entirely.

Resources you used:

• Project folder with Python scripts (object_detector.py, currency_classifier.py, distance_sensor.py, smart_assistant.py).
• Python virtual environment with installed packages (ultralytics, opencv-python, pyttsx3, numpy, tensorflow).
• Downloaded model files (yolov8n.pt and any trained currency_model.h5).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the Smart Vision Assistant.

• Your project folder, virtual environment, and model files will remain on your Desktop ready to use.
• The virtual environment and downloaded models take up approximately 2-3 GB of disk space.
• To resume working, open your terminal, navigate to your project folder, and activate the virtual environment.

Pause - I'll come back to this later

Stop any running processes to free up system memory. Your files and virtual environment stay intact for when you return.

• Close any running Python scripts by pressing q in the OpenCV window or Ctrl+C in the terminal.
• Deactivate your virtual environment by running:

deactivate

• Close VS Code if you are done for now.
• Your project folder and all files will remain on your Desktop when you are ready to return.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes your project files, virtual environment, and downloaded models.

• Close any running Python scripts by pressing q in the OpenCV window or Ctrl+C in the terminal.
• Deactivate your virtual environment if it is still active:

deactivate

• Delete the entire project folder. On macOS or Linux, run:

rm -rf ~/Desktop/smart-vision-assistant

• On Windows, run:

rmdir /s /q %USERPROFILE%\Desktop\smart-vision-assistant

• This removes all Python scripts, the virtual environment, the yolov8n.pt model, and any trained currency_model.h5 file.

Nice Work!

Nice Work!

Congratulations! You just built a Smart Vision Assistant that brings together machine learning, deep learning, and IoT concepts into a real-world assistive technology prototype.

What you learned:

• Built a real-time object detection system using YOLOv8 that identifies 80+ object classes through a webcam feed and announces them via text-to-speech.
• Trained a currency recognition model using transfer learning with MobileNetV2 to classify banknote denominations from camera images.
• Integrated all modules into a unified assistant with mode switching that combines object detection, currency recognition, and distance measurement with proximity-based audio alerts.
• Secret Mission: Added a navigation mode that combines object detection with distance measurement to give directional audio cues and proximity warnings.

Ready to quiz yourself?