Analyze Product Reviews with NLP

Build a Python CLI that reveals NLP features and sentiment in product reviews.

Introduction

30 Second Summary

Product reviews often mix praise with criticism in the same sentence. A single verdict can hide which words shaped the result.

In this project, you will build ReviewLens, a local command-line tool that reveals how Natural Language Processing changes an English product review. Its report traces the text from tokenization through TF-IDF keywords to VADER sentiment.

What You'll Build

Your finished terminal tool turns a pasted product review into a visible trail from raw text through cleaned words to ranked keywords plus a sentiment label.

By the end of this project, you'll have:

  • Trace one review through tokens, filtered words, stems, then lemmas.
  • Inspect a Bag-of-Words readout where each surviving term appears beside its count.
  • Spot the five strongest TF-IDF keywords before checking four VADER scores plus a clear sentiment label.
  • Secret Mission: Switch keyword extraction between stems and lemmas to compare how the rankings change.

Are there any prerequisites?

Beginner familiarity with Python variables, lists, functions, loops, and imports plus a Windows PC running Python 3.11+ with Visual Studio Code installed is enough to begin.

Step 1 prepares Git for Windows, the Python extension, a virtual environment, plus the pinned packages.

Before We Start

This first checkpoint helps you commit to building ReviewLens as a transparent local command-line analyzer for English product reviews. Every preprocessing stage stays visible so you can inspect how the text changes.

Set Up a Reproducible Workspace

ReviewLens grows through several NLP experiments. Isolating its packages prevents one experiment from breaking another when a dependency changes.

You will prepare a project workspace in Visual Studio Code. You will use a virtual environment to hold pinned versions of NLTK and scikit-learn. You will also configure Git so every later project snapshot carries your identity.

In this step, get ready to:
  • Install Git for Windows and configure your commit identity.
  • Create the ReviewLens workspace with pinned dependency files.
  • Build an isolated Python environment and verify its packages.
Install and configure Git

Git records the history behind your portfolio project. The first check reveals whether Git for Windows is already available on your computer.

This check may print a version or report that the command is unavailable. Both outcomes have a path below.

  • Press the Windows key to open Windows search.
  • Type PowerShell.
  • Press Enter to open PowerShell.
  • Check whether Git is installed by running this command:
git --version

What does this command do?

The command asks Git to print its installed version. A version string confirms that PowerShell can find Git.

✔️ I see a Git version

Your Git installation is available in PowerShell. That clears the first workspace dependency.

ⓧ Command not found

Git needs to be installed before PowerShell can run its commands.

  • Open the official Git for Windows installer page in your browser.
  • Choose Git for Windows/x64 Setup for an x64 computer.
  • Choose Git for Windows/ARM64 Setup for an ARM64 computer.
  • Run the downloaded standalone installer.
  • Keep the beginner-safe default selections in the installer.
  • Finish the installation.
  • Close PowerShell.
  • Reopen PowerShell through Windows search.
  • Confirm Git is now available by running:
git --version

What should I see?

PowerShell prints a Git version string after the installer adds Git to your command path.

Git for Windows is ready when the version string appears.

Every Git commit stores an author name plus an email address. Global configuration applies your identity to repositories created by your Windows account.

  • Set your Git identity by running these commands with your own values:
git config --global user.name "[[GIT_NAME="your name"]]"
git config --global user.email [[GIT_EMAIL="your GitHub email"]]

Why does Git need this identity?

Git writes the configured name plus email address into each commit. The global scope saves these values for future repositories under your Windows account.

  • Verify the saved name plus email address by running:
git config --global user.name
git config --global user.email

What should I see?

The first line prints your configured name. The second line prints your configured email address.

Your future ReviewLens commits now carry the correct author details.

Git details look wrong?

Run the two configuration commands again with the correct values. The new global values replace the previous ones.

If Git still prints unexpected details, help me check where Git is reading my user name and email from.

Create the workspace and dependency files

The project folder gives ReviewLens one predictable home. Its dependency file records exact package versions so the same environment can be recreated later.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code.
  • Press Enter to open Visual Studio Code.
  • Select File from the top menu.
  • Select Open Folder....
  • Select your Windows Desktop location.
  • Select New Folder.
  • Name the folder reviewlens-nlp.
  • Select Select Folder to open the workspace.

What is a workspace?

Visual Studio Code treats an opened folder as a workspace. The Explorer view should now show reviewlens-nlp at the top.

The Python extension connects Visual Studio Code to installed interpreters. It also discovers project-local virtual environments.

  • Press Ctrl+Shift+X to show Extensions.
  • Search for Python.
  • Select the Python extension published by Microsoft.
  • Click Install if the extension is missing.
  • Press Ctrl+Shift+E to return to the Explorer view.

The dependency file is the recipe for rebuilding ReviewLens. Each package uses an exact version so a future installation resolves to the same foundation.

  • Select the New File... button in the Explorer view.
  • Name the file requirements.txt.
  • Add the pinned dependencies by copying this file content:
nltk==3.10.3
scikit-learn==1.9.1

What does this file do?

  • The first line pins NLTK to version 3.10.3 for text preprocessing plus VADER sentiment.
  • The second line pins scikit-learn to version 1.9.1 for Bag-of-Words plus TF-IDF features.
  • Exact pins make later installations reproducible.
  • Save requirements.txt.
  • Confirm requirements.txt appears directly under reviewlens-nlp in the Explorer view.

Can't see requirements.txt?

Confirm the Explorer heading shows reviewlens-nlp. A different heading means another folder is open.

If the file still does not appear, help me create requirements.txt in the open Visual Studio Code workspace.

✔️ Awesome, I've got everything!

Your pinned dependency recipe is saved in the project folder.

ⓧ I'd like to double check the full code

nltk==3.10.3
scikit-learn==1.9.1

The virtual environment plus generated Python cache files belong to your local machine. A Git ignore file keeps those machine-specific files out of future commits.

  • Select the New File... button in the Explorer view.
  • Name the file .gitignore.
  • Add the project exclusions by copying this file content:
.venv/
__pycache__/
*.py[cod]
.vscode/

What does this file do?

  • The .venv/ rule excludes the installed virtual environment.
  • The Python cache rules exclude generated bytecode files.
  • The .vscode/ rule excludes local editor settings.
  • Save .gitignore.
  • Confirm .gitignore appears beside requirements.txt in the Explorer view.

Is .gitignore missing?

Check that the filename begins with a dot. Confirm that it has no added .txt extension.

If the filename keeps changing, help me create a dotfile named .gitignore in Visual Studio Code on Windows.

✔️ Awesome, I've got everything!

Your local environment files are now excluded from future Git snapshots.

ⓧ I'd like to double check the full code

.venv/
__pycache__/
*.py[cod]
.vscode/
Build and verify the virtual environment

A project-local environment stores ReviewLens packages away from your shared Python installation. Activating it ensures the package installer targets .venv.

  • Press Ctrl+Shift+P to open the Command Palette.
  • Enter Terminal: Create New Terminal.
  • Select Terminal: Create New Terminal to open the integrated terminal.
  • Select PowerShell from the terminal profile menu if another shell opens.
  • Create the project-local environment by running:
python -m venv .venv

What does this command do?

Python creates an isolated environment inside the .venv folder. Packages installed after activation stay scoped to ReviewLens.

  • Confirm .venv appears in the Explorer view.

No .venv folder?

Confirm the terminal path ends with reviewlens-nlp. Confirm that Python 3.11 or newer is installed.

If environment creation still fails, help me diagnose a failed Python venv creation in PowerShell.

  • Activate the new environment by running:
.venv\Scripts\Activate.ps1

What does activation change?

Activation points the current PowerShell session at the Python executable inside .venv. The terminal prompt should begin with (.venv).

  • Confirm the PowerShell prompt begins with (.venv).

Activation blocked in PowerShell?

Confirm that the terminal is inside reviewlens-nlp. Confirm that .venv was created successfully.

If PowerShell reports a script execution restriction, help me activate .venv safely in Windows PowerShell.

The pinned dependency file now turns the empty environment into the exact NLP workspace this project expects.

  • Install the pinned project dependencies by running:
pip install -r requirements.txt

What does this command install?

The installer reads requirements.txt from the open project folder. It installs the two pinned libraries plus their required dependencies inside .venv.

  • Wait for the installation to return control to the PowerShell prompt.
  • Press Ctrl+Shift+P to open the Command Palette.
  • Enter Python: Select Interpreter.
  • Select Python: Select Interpreter.
  • Choose the interpreter whose path contains .venv\Scripts\python.exe.

Dependency installation failed?

Confirm the prompt begins with (.venv). Confirm that requirements.txt contains both pinned package lines.

For an environment-specific installation error, help me debug pip install in my ReviewLens virtual environment.

Before the final check, what version details plus package names do you expect the three commands to reveal?

  • Verify Python, Git, and the active environment packages by running:
py -3 --version
git --version
python -m pip list

What does the final check prove?

  • The first command prints the installed Python 3 version.
  • The second command prints the Git version available to PowerShell.
  • The third command lists packages from the active Python environment.

You should see Python 3.11 or newer plus a Git version string. The package list should include nltk 3.10.3 plus scikit-learn 1.9.1.

That is the workspace foundation complete. ReviewLens now has isolated packages plus reproducible dependency files.

Your Windows workspace is ready. Next, you will run a naive tokenizer and expose the text problems that the real NLP pipeline must solve.

Expose Naive Tokenization

Your reproducible workspace is ready. Now you need a baseline that shows how a product review looks after the simplest possible split.

Product reviews mix capital letters with punctuation. A whitespace-only rule must prove that it can separate both cleanly.

In this step, get ready to:
  • Create a Python script that splits one mixed-case review on whitespace.
  • Run the script to compare the original review with its token list.
  • Identify the case, punctuation, and common-word problems in the output.
Create the naive tokenizer

Naive tokenization uses a simple rule to divide text into smaller units. The built-in split() method gives you a quick version based only on whitespace.

  • Create analyzer.py inside the reviewlens-nlp folder using the left file sidebar in Visual Studio Code.
  • Paste the code below into analyzer.py:
review = "This battery is AMAZING, but the buttons feel cheap!"
naive_tokens = review.split()

print(f"Original: {review}")
print(f"Naive tokens: {naive_tokens}")

What does this code do?

  • The review variable holds one product review with mixed capitalization and punctuation.
  • The review.split() call divides the review wherever it finds whitespace.
  • The final two lines print the source text followed by the resulting token list.
  • Save analyzer.py.
  • Confirm analyzer.py appears in the left file sidebar under reviewlens-nlp.

File missing from the sidebar?

Confirm that you created analyzer.py inside the open reviewlens-nlp folder. Check that the filename does not end with an extra text-file extension.

Help me find why my analyzer file is missing in Visual Studio Code.

✔️ Awesome, I've got everything!

Your saved file matches the first ReviewLens script.

ⓧ I'd like to double check the full code

review = "This battery is AMAZING, but the buttons feel cheap!"
naive_tokens = review.split()

print(f"Original: {review}")
print(f"Naive tokens: {naive_tokens}")
Run the split and inspect the result

The terminal output shows exactly where each whitespace boundary landed. This makes the naive rule easy to test against the original sentence.

Before you run the script, predict whether spaces alone can separate every word cleanly.

  • Test the naive tokenizer by running this command:
python analyzer.py

You'll see Original: followed by the sample review. The next line begins with Naive tokens: and shows the whitespace-split list.

Your first analyzer is running. You now have a visible baseline for judging the quality of its tokens.

Script not running?

If the terminal cannot find analyzer.py, confirm that the terminal is using the reviewlens-nlp folder. If no output appears, save the file before running the command again.

Help me run analyzer.py from the correct folder.

  • Compare the original sentence with the token list.
  • Find the tokens that still include punctuation.
  • Find the word that keeps its uppercase letters.
  • Find the common words that remain in the list.

What did the split miss?

Punctuation stays attached to AMAZING,. It also stays attached to cheap!.

The token AMAZING, remains uppercase. A lowercase form from another review would become a different token.

Common words such as This, is, but, and the remain in the output.

This shortfall is the point of the experiment. Whitespace alone does not produce clean features for product reviews.

You have proved where naive tokenization falls short. Next, you'll turn that rough list into cleaner language features.

Build the Preprocessing Pipeline

You already exposed the weakness in whitespace splitting by catching case differences and attached punctuation. Those inconsistencies prevent related words from sharing a clean representation.

Now you will use NLTK for proper tokenization. The pipeline also makes stemming and lemmatization visible.

In this step, get ready to:
  • Add the NLTK tools and language resources used by the analyzer.
  • Build a preprocessing function that exposes five text transformations.
  • Run the analyzer and compare its stems with its lemmas.
Add the NLTK building blocks

The NLTK package provides the Python APIs for this pipeline. Its language data is downloaded separately because corpora and lexicons live outside the installed package.

  • Switch back to analyzer.py in the Visual Studio Code editor.
  • Place your cursor above the existing review line.
  • Paste the imports and reusable processing objects from this code:
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize.treebank import TreebankWordTokenizer

TOKENIZER = TreebankWordTokenizer()
STEMMER = PorterStemmer()
LEMMATIZER = WordNetLemmatizer()

What do these objects provide?

  • The stopwords corpus supplies common English words for the filtering stage.
  • TreebankWordTokenizer separates most punctuation from nearby words.
  • PorterStemmer applies rules that shorten related word forms.
  • WordNetLemmatizer uses WordNet to find dictionary-backed forms.
  • Save analyzer.py.
  • Confirm the new imports load by running this command in the existing PowerShell terminal:
python analyzer.py

What does this check prove?

Python loads every new import before it reaches your existing review code. A complete naive report confirms that the installed NLTK package is available to this interpreter.

Your original review and naive token list should print again. That is your first upgrade in place: the analyzer remains runnable while its new NLP tools load successfully.

Seeing an import error?

  • Confirm the Visual Studio Code status bar still shows the interpreter inside .venv.
  • Confirm requirements.txt still contains the pinned NLTK dependency.

Ask for help with the active environment if the import still fails: Help me diagnose why NLTK cannot be imported from my active .venv interpreter.

The first data download contacts the NLTK data server. A brief quiet pause is expected while the resources are stored outside your project folder.

  • Place your cursor below the LEMMATIZER assignment.
  • Add the downloader function and its first call by pasting this code:
def download_nltk_data():
    """Download the three small NLTK resources used by this project."""
    for package in ("stopwords", "wordnet", "vader_lexicon"):
        nltk.download(package, quiet=True)


download_nltk_data()

What does this code do?

The loop downloads stopwords, wordnet, and vader_lexicon by package identifier. The quiet=True setting suppresses routine downloader messages.

Later runs reuse the stored resources. The analyzer can then access its language data without adding those files to the repository.

  • Save analyzer.py.
  • Download the three resources by running the analyzer again:
python analyzer.py

What does this run confirm?

The download call finishes before the original review is printed. Seeing the naive report confirms that Python reached the existing analyzer code after preparing all three resources.

Good progress. Your installed library now has the separate language data that the preprocessing stages require.

Does the download fail?

Check that your computer can reach the internet because the first run contacts the NLTK data server. Run the analyzer again after restoring the connection.

For help with the downloader, ask: Help me diagnose an NLTK data download failure in this project.

Build the five-stage transformation

Lemmatization works best when it knows a word's grammatical role. This beginner pipeline uses a transparent noun pass followed by a verb pass.

  • Place your cursor directly above the download_nltk_data() call.
  • Add the noun-then-verb helper by pasting this function:
def lemmatize_as_noun_and_verb(word):
    """Apply a simple noun pass followed by a verb pass."""
    noun_form = LEMMATIZER.lemmatize(word, pos="n")
    return LEMMATIZER.lemmatize(noun_form, pos="v")

Why use two passes?

The first pass asks WordNet for a noun lemma. The second pass treats that result as a possible verb.

This keeps the heuristic visible inside one small function. Full part-of-speech tagging remains outside this pipeline.

  • Save analyzer.py.
  • Check the helper's syntax by running the analyzer:
python analyzer.py

What does this syntax check prove?

Python reads the new helper before it reaches the downloader call. The original review output confirms that the function definition loaded successfully.

Seeing a syntax error in the helper?

  • Check that both lines inside lemmatize_as_noun_and_verb() use the same four-space indentation.
  • Check that the part-of-speech values remain "n" and "v".

Ask for a focused review if the function still fails: Help me find the syntax problem in my lemmatization helper.

The main preprocessing function turns one review into named intermediate lists. Returning every list keeps each transformation available for the terminal report.

  • Place your cursor below lemmatize_as_noun_and_verb().
  • Keep the new function above the download_nltk_data() call.
  • Build the transformation pipeline by pasting this code:
def preprocess(text):
    """Return each visible stage of the review-cleaning pipeline."""
    tokens = TOKENIZER.tokenize(text)
    lowercase = [token.lower() for token in tokens]
    alphabetic = [token for token in lowercase if token.isalpha()]
    english_stopwords = set(stopwords.words("english"))
    filtered = [token for token in alphabetic if token not in english_stopwords]
    stems = [STEMMER.stem(token) for token in filtered]
    lemmas = [lemmatize_as_noun_and_verb(token) for token in filtered]

    return {
        "tokens": tokens,
        "lowercase": lowercase,
        "filtered": filtered,
        "stems": stems,
        "lemmas": lemmas,
    }

How does the pipeline work?

  • TOKENIZER.tokenize() separates the review into processable units.
  • lower() gives case variants a consistent form.
  • isalpha() keeps tokens made entirely from letters.
  • stopwords.words("english") provides the common words removed from keyword features.
  • STEMMER.stem() creates mechanically shortened forms.
  • lemmatize_as_noun_and_verb() creates dictionary-backed forms using the two-pass heuristic.
  • Save analyzer.py.
  • Confirm the growing file remains runnable by running the analyzer:
python analyzer.py

What does this run confirm?

Python loads the complete preprocessing function before executing the existing report. The original review and naive token list confirm that every new definition has valid syntax.

Does the pipeline stop loading?

  • Check that every statement inside preprocess() uses four spaces of indentation.
  • Check that the five dictionary keys match tokens, lowercase, filtered, stems, and lemmas.

Ask for help with this specific function if needed: Help me debug the preprocessing function in analyzer.py.

Print and compare every stage

Your current report still calls the whitespace split. Replacing that report with the staged output makes every transformation observable in one run.

  • Find the existing review line beneath download_nltk_data().
  • Select that line through the existing naive-token print line.
  • Replace the selected report code with this staged report:
review = "The battery is AMAZING, but the buttons feel cheap!"
stages = preprocess(review)
print(f"Original: {review}")
print(f"Tokens: {stages['tokens']}")
print(f"Lowercase: {stages['lowercase']}")
print(f"Filtered: {stages['filtered']}")
print(f"Stems: {stages['stems']}")
print(f"Lemmas: {stages['lemmas']}")

What does this report expose?

The stages dictionary holds each list returned by preprocess(). Each print statement reveals one point in the cleaning pipeline.

The original review remains visible at the top. That gives every transformed list a clear reference point.

  • Save analyzer.py.

Before you run the finished pipeline, which punctuation marks and common words do you expect to disappear from the filtered list?

  • Reveal the five preprocessing stages by running this command:
python analyzer.py

What should you see?

  • The Tokens list contains the comma and exclamation mark as separate tokens.
  • The Lowercase list changes AMAZING to amazing.
  • The Filtered list excludes punctuation and common words such as the.
  • The Stems list includes shortened forms such as batteri.
  • The Lemmas list includes dictionary-backed forms such as battery.

Missing a preprocessing list?

  • Check that stages = preprocess(review) appears before the five staged print statements.
  • Check that every print statement uses the matching key from the returned dictionary.
  • Check that the downloader call remains above the review report.

Ask for help comparing your file with the expected pipeline: Help me find why one preprocessing stage is missing from my ReviewLens output.

Your terminal now shows how one raw sentence becomes several feature-ready representations. Each list exposes a distinct NLP decision.

How do stems and lemmas differ?

Stemming applies mechanical rules that can produce forms outside a dictionary. That is why battery becomes batteri.

Lemmatization aims for dictionary-backed words. Its quality depends on grammatical context, so this noun-then-verb heuristic remains intentionally simple.

✔️ Awesome, I've got everything!

Your preprocessing pipeline is complete. Double-check that you saved analyzer.py before continuing.

ⓧ I'd like to double check the full code

import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize.treebank import TreebankWordTokenizer

TOKENIZER = TreebankWordTokenizer()
STEMMER = PorterStemmer()
LEMMATIZER = WordNetLemmatizer()


def download_nltk_data():
    """Download the three small NLTK resources used by this project."""
    for package in ("stopwords", "wordnet", "vader_lexicon"):
        nltk.download(package, quiet=True)


def lemmatize_as_noun_and_verb(word):
    """Apply a simple noun pass followed by a verb pass."""
    noun_form = LEMMATIZER.lemmatize(word, pos="n")
    return LEMMATIZER.lemmatize(noun_form, pos="v")


def preprocess(text):
    """Return each visible stage of the review-cleaning pipeline."""
    tokens = TOKENIZER.tokenize(text)
    lowercase = [token.lower() for token in tokens]
    alphabetic = [token for token in lowercase if token.isalpha()]
    english_stopwords = set(stopwords.words("english"))
    filtered = [token for token in alphabetic if token not in english_stopwords]
    stems = [STEMMER.stem(token) for token in filtered]
    lemmas = [lemmatize_as_noun_and_verb(token) for token in filtered]

    return {
        "tokens": tokens,
        "lowercase": lowercase,
        "filtered": filtered,
        "stems": stems,
        "lemmas": lemmas,
    }


download_nltk_data()
review = "The battery is AMAZING, but the buttons feel cheap!"
stages = preprocess(review)
print(f"Original: {review}")
print(f"Tokens: {stages['tokens']}")
print(f"Lowercase: {stages['lowercase']}")
print(f"Filtered: {stages['filtered']}")
print(f"Stems: {stages['stems']}")
print(f"Lemmas: {stages['lemmas']}")

Your analyzer now exposes every cleaning stage. Next, you will turn those cleaned words into numerical features.

Turn Reviews into Features

Your analyzer now exposes every cleaning stage from raw review text to normalized lemmas. Those lemmas give you a consistent vocabulary for measuring what each review discusses.

Clean words are still strings. Machine-learning models need numerical features that can be compared across documents.

This step uses scikit-learn to build two representations. Bag-of-Words counts surviving terms. TF-IDF highlights terms that stand out within a review corpus.

In this step, get ready to:
  • Add a six-review corpus for feature comparisons.
  • Create nonzero Bag-of-Words counts for the current review.
  • Rank the current review's five strongest TF-IDF terms.
Add the comparison corpus

TF-IDF needs multiple documents to judge how distinctive a term is. A small built-in corpus provides that comparison without adding a separate dataset.

  • In analyzer.py, find the TreebankWordTokenizer import near the top of the file.
  • Add the vectorizer import plus the sample corpus directly below that import by pasting this code:
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

SAMPLE_REVIEWS = [
    "The headphones sound clear and the battery lasts all day.",
    "This keyboard feels cheap and several keys stopped working.",
    "The camera is compact, but its low-light photos look noisy.",
    "Setup was easy and the smart speaker responds quickly.",
    "The blender is powerful, although the lid is difficult to clean.",
    "Delivery was slow, but the laptop itself performs very well.",
]

What does this code add?

  • The vectorizer import provides the two feature-building classes used later in this step.
  • The SAMPLE_REVIEWS list supplies six comparison documents with varied product terms.
  • The fixed corpus makes repeated runs comparable because the reference documents stay the same.
  • Save analyzer.py.
  • Confirm the new import loads by running this command:
python analyzer.py

What does this command check?

Python loads the new scikit-learn import before running the existing preprocessing report. A complete report confirms that the import and corpus contain valid Python.

You should still see the original review followed by the token, lowercase, filtered, stem, and lemma lists.

Good progress. Your analyzer now has a fixed comparison corpus ready for numerical feature extraction.

Does the import fail?

  • Confirm that the selected Visual Studio Code interpreter points to .venv.
  • Check that requirements.txt still contains the pinned scikit-learn dependency.
  • Check that the import appears before SAMPLE_REVIEWS.

Help me diagnose the scikit-learn import failure.

Build numerical feature representations

A document-term matrix gives each review one row. Each vocabulary term receives one column containing a count or weight.

The new function preprocesses every review into lemma strings. Both vectorizers therefore measure the same normalized vocabulary.

  • In analyzer.py, find the closing brace of the dictionary returned by preprocess().
  • Add vectorize_review() directly below preprocess() by pasting this code:
def vectorize_review(review):
    documents = SAMPLE_REVIEWS + [review]
    cleaned_documents = [" ".join(preprocess(document)["lemmas"]) for document in documents]
    count_vectorizer = CountVectorizer(analyzer=str.split)
    count_matrix = count_vectorizer.fit_transform(cleaned_documents)
    count_terms = count_vectorizer.get_feature_names_out()
    review_counts = count_matrix[-1].toarray()[0]
    bag_of_words = sorted([(term, int(value)) for term, value in zip(count_terms, review_counts) if value > 0], key=lambda item: (-item[1], item[0]))
    tfidf_vectorizer = TfidfVectorizer(analyzer=str.split)
    tfidf_matrix = tfidf_vectorizer.fit_transform(cleaned_documents)
    tfidf_terms = tfidf_vectorizer.get_feature_names_out()
    review_weights = tfidf_matrix[-1].toarray()[0]
    top_keywords = sorted([(term, float(weight)) for term, weight in zip(tfidf_terms, review_weights) if weight > 0], key=lambda item: item[1], reverse=True)[:5]
    return bag_of_words, top_keywords

What does this function do?

  • The documents list places the current review after the six sample reviews.
  • The cleaned_documents list converts each review into a space-separated lemma string.
  • CountVectorizer builds a sparse matrix of term counts from the cleaned strings.
  • TfidfVectorizer calculates weights from the same cleaned corpus.
  • The function returns every nonzero count plus the five highest nonzero TF-IDF weights.
  • Save analyzer.py.
  • Confirm Python can load the new function by running this command:
python analyzer.py

What does this run prove?

Python parses vectorize_review() before running the existing report. Seeing the preprocessing lists confirms that the new function has valid syntax.

You should see the same five preprocessing lists complete without an interruption.

Does the script stop running?

  • Check that every line inside vectorize_review() uses consistent indentation.
  • Confirm that vectorize_review() begins at the left edge of the file.
  • Compare both vectorizer class names with the import near the top of analyzer.py.

Help me find the problem in the feature function.

Display and compare the features

The feature function returns two inspectable lists. Printing them beside the preprocessing stages connects each numerical value to the cleaned words that produced it.

  • In analyzer.py, find download_nltk_data() at the bottom of the file.
  • Replace that line plus everything below it by pasting this code:
download_nltk_data()
review = "The battery is excellent but the case feels cheap"
stages = preprocess(review)
bag_of_words, top_keywords = vectorize_review(review)
print(f"Tokens: {stages['tokens']}")
print(f"Lowercase: {stages['lowercase']}")
print(f"Filtered: {stages['filtered']}")
print(f"Stems: {stages['stems']}")
print(f"Lemmas: {stages['lemmas']}")
print(f"Bag of Words: {bag_of_words}")
print(f"Top TF-IDF: {top_keywords}")

What does this report add?

The call to vectorize_review() calculates both feature representations for the sample review. Each tuple keeps a term beside its numerical value.

The preprocessing stages remain visible above the feature lines. You can trace each displayed feature back to its cleaned lemma.

  • Save analyzer.py.

Before you run the report, which review terms do you expect to receive the highest TF-IDF weights?

  • Generate the complete feature report by running this command:
python analyzer.py

What does this command run?

The script preprocesses the current review before fitting both vectorizers to the seven cleaned documents. It prints the numerical features after the five visible text transformations.

You should see nonzero (term, count) pairs on the Bag of Words line. The Top TF-IDF line should contain five (term, TF-IDF score) pairs.

That is the key leap. Your review now appears as inspectable numerical features.

How do these features differ?

Bag-of-Words counts each surviving term. It ignores most word order.

TF-IDF lowers the influence of terms found across several corpus documents. Distinctive terms receive more prominence.

Word embeddings represent words or larger text units as dense meaning-oriented vectors. A later transformer project uses contextual representations to capture distinctions that count matrices miss.

Are the feature lines missing?

  • Confirm that the call to vectorize_review(review) appears before the two feature print statements.
  • Check that both feature print statements remain at the left edge of analyzer.py.
  • Verify that the current review contains alphabetic words that remain after stopword removal.

Help me debug the missing feature output.

✔️ Awesome, I've got everything!

Your analyzer now prints its preprocessing stages plus Bag-of-Words counts and five TF-IDF keywords. Make sure analyzer.py is saved.

ⓧ I'd like to double check the full code

  • Compare every line in analyzer.py with this reference.
  • Update any lines that differ from the reference.
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize.treebank import TreebankWordTokenizer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

SAMPLE_REVIEWS = [
    "The headphones sound clear and the battery lasts all day.",
    "This keyboard feels cheap and several keys stopped working.",
    "The camera is compact, but its low-light photos look noisy.",
    "Setup was easy and the smart speaker responds quickly.",
    "The blender is powerful, although the lid is difficult to clean.",
    "Delivery was slow, but the laptop itself performs very well.",
]

TOKENIZER = TreebankWordTokenizer()
STEMMER = PorterStemmer()
LEMMATIZER = WordNetLemmatizer()


def download_nltk_data():
    for package in ("stopwords", "wordnet", "vader_lexicon"):
        nltk.download(package, quiet=True)


def lemmatize_as_noun_and_verb(word):
    noun_form = LEMMATIZER.lemmatize(word, pos="n")
    return LEMMATIZER.lemmatize(noun_form, pos="v")


def preprocess(text):
    tokens = TOKENIZER.tokenize(text)
    lowercase = [token.lower() for token in tokens]
    alphabetic = [token for token in lowercase if token.isalpha()]
    english_stopwords = set(stopwords.words("english"))
    filtered = [token for token in alphabetic if token not in english_stopwords]
    stems = [STEMMER.stem(token) for token in filtered]
    lemmas = [lemmatize_as_noun_and_verb(token) for token in filtered]
    return {"tokens": tokens, "lowercase": lowercase, "filtered": filtered, "stems": stems, "lemmas": lemmas}


def vectorize_review(review):
    documents = SAMPLE_REVIEWS + [review]
    cleaned_documents = [" ".join(preprocess(document)["lemmas"]) for document in documents]
    count_vectorizer = CountVectorizer(analyzer=str.split)
    count_matrix = count_vectorizer.fit_transform(cleaned_documents)
    count_terms = count_vectorizer.get_feature_names_out()
    review_counts = count_matrix[-1].toarray()[0]
    bag_of_words = sorted([(term, int(value)) for term, value in zip(count_terms, review_counts) if value > 0], key=lambda item: (-item[1], item[0]))
    tfidf_vectorizer = TfidfVectorizer(analyzer=str.split)
    tfidf_matrix = tfidf_vectorizer.fit_transform(cleaned_documents)
    tfidf_terms = tfidf_vectorizer.get_feature_names_out()
    review_weights = tfidf_matrix[-1].toarray()[0]
    top_keywords = sorted([(term, float(weight)) for term, weight in zip(tfidf_terms, review_weights) if weight > 0], key=lambda item: item[1], reverse=True)[:5]
    return bag_of_words, top_keywords


download_nltk_data()
review = "The battery is excellent but the case feels cheap"
stages = preprocess(review)
bag_of_words, top_keywords = vectorize_review(review)
print(f"Tokens: {stages['tokens']}")
print(f"Lowercase: {stages['lowercase']}")
print(f"Filtered: {stages['filtered']}")
print(f"Stems: {stages['stems']}")
print(f"Lemmas: {stages['lemmas']}")
print(f"Bag of Words: {bag_of_words}")
print(f"Top TF-IDF: {top_keywords}")

What should match?

This reference contains the complete feature-enabled analyzer for this step. The imports, corpus, feature function, and report code should match exactly.

Your analyzer can now reveal what a review discusses through counts and keyword weights. Next, you will add VADER sentiment so the report can also describe how the writer feels.

Add Sentiment and Package the Project

Your Python analyzer now exposes each preprocessing stage. It also turns cleaned reviews into visible Bag-of-Words counts and TF-IDF keyword weights.

Keywords reveal what a review discusses. VADER adds the missing emotional signal through a fast rule-based sentiment baseline.

You will turn the analyzer into an interactive command-line tool. A detailed README explains its decisions. A Git commit records the finished project.

In this step, get ready to:
  • Add VADER scores with a transparent sentiment-label rule.
  • Package the interactive analyzer with project documentation.
  • Record the reproducible project in a local Git repository.
Add VADER sentiment to the report

VADER is part of NLTK. It scores the original review so capitalization and punctuation remain available to its rules. The final label stays transparent because it depends only on the sign of the compound score.

  • In Visual Studio Code, switch back to analyzer.py.
  • Find from nltk.corpus import stopwords near the top of the file.
  • Add the VADER import directly below that line by inserting:
from nltk.sentiment.vader import SentimentIntensityAnalyzer

What does this import provide?

The imported class reads the downloaded VADER lexicon. Its polarity_scores() method returns negative, neutral, positive, and compound scores.

  • Find the end of vectorize_review().
  • Add the label function below return bag_of_words, top_keywords by pasting:
def sentiment_label(scores):
    """Apply the project's simple sign-based label rule."""
    if scores["compound"] > 0:
        return "POSITIVE"
    if scores["compound"] < 0:
        return "NEGATIVE"
    return "NEUTRAL"

How does the label rule work?

  • A positive compound score returns POSITIVE.
  • A negative compound score returns NEGATIVE.
  • A compound score of zero returns NEUTRAL.
  • Save analyzer.py.
  • Check that the updated imports and label function run by executing:
python analyzer.py

What does this check prove?

Python loads the new VADER import before running the existing analyzer. A completed numerical report confirms that the import and label function contain no syntax errors.

You should still see the preprocessing lists, Bag-of-Words counts, and five TF-IDF keywords.

Does the analyzer stop before printing?

Confirm that the new import sits below the existing NLTK corpus import. Check that every line inside sentiment_label() uses four spaces of indentation.

Ask for help with the failing import or indentation:

  • Add a formatting helper below sentiment_label() by pasting:
def format_items(items, decimals=None):
    """Format term-value pairs for readable terminal output."""
    if not items:
        return "(none)"
    if decimals is None:
        return ", ".join(f"{term}: {value}" for term, value in items)
    return ", ".join(f"{term}: {value:.{decimals}f}" for term, value in items)

What does this helper format?

  • An empty feature list becomes (none).
  • Bag-of-Words counts keep their integer values.
  • TF-IDF weights can be rounded to a chosen number of decimal places.
  • Save analyzer.py.
  • Confirm that the formatting helper contains valid Python by running:
python analyzer.py

Why run the analyzer again?

This run makes Python parse the new helper alongside the existing pipeline. The complete numerical output confirms that the file still executes successfully.

You should see the existing report reach its final TF-IDF line without a syntax error.

Does Python point to the formatting helper?

Check the closing parentheses on both join() calls. Confirm that the final return line stays inside format_items().

Ask for help checking the helper:

  • Add the complete report function below format_items() by pasting:
def print_report(review):
    """Run the complete analysis and print a transparent report."""
    stages = preprocess(review)
    bag_of_words, top_keywords = vectorize_review(review)
    sentiment_analyzer = SentimentIntensityAnalyzer()
    scores = sentiment_analyzer.polarity_scores(review)
    print("\nREVIEWLENS REPORT")
    print("=" * 60)
    print(f"Original:    {review}")
    print(f"Tokens:      {stages['tokens']}")
    print(f"Lowercase:   {stages['lowercase']}")
    print(f"Filtered:    {stages['filtered']}")
    print(f"Stems:       {stages['stems']}")
    print(f"Lemmas:      {stages['lemmas']}")
    print(f"Bag of Words: {format_items(bag_of_words)}")
    print(f"Top TF-IDF:   {format_items(top_keywords, decimals=3)}")
    print("VADER scores: " f"neg={scores['neg']:.3f}, " f"neu={scores['neu']:.3f}, " f"pos={scores['pos']:.3f}, " f"compound={scores['compound']:.3f}")
    print(f"Sentiment:    {sentiment_label(scores)}")

What does the report combine?

  • The original review feeds the visible preprocessing pipeline.
  • The cleaned lemma features feed Bag-of-Words and TF-IDF.
  • The original review feeds VADER so its punctuation and negation rules remain available.
  • The formatting helper keeps counts and decimal weights readable.
  • Save analyzer.py.
  • Confirm that def print_report(review): now appears below the formatting helper.

The report function now holds every analysis stage in one reusable place.

Does the report function show an editor warning?

Check that every report line uses four spaces of indentation. Confirm that each stages key matches the dictionary returned by preprocess().

Ask for help checking the report function:

  • Select the current script-level calls from the final download_nltk_data() call through the last print statement.
  • Replace the selected code with the interactive entry point below:
def main():
    download_nltk_data()
    print("ReviewLens: NLP Foundations")
    review = input("Paste an English product review: ").strip()
    if not review:
        print("Please enter a non-empty review.")
        return
    print_report(review)


if __name__ == "__main__":
    main()

How does the entry point behave?

  • The data download reuses existing NLTK resources after the first successful run.
  • The input prompt accepts one English product review.
  • The empty-input check exits with a clear message.
  • The entry-point guard runs main() when you execute analyzer.py.
  • Save analyzer.py.

Before you run the interactive analyzer, which parts of the report do you expect to describe the topic? Which part should describe the writer's feeling?

  • Start the interactive analyzer by running:
python analyzer.py

What happens after this command?

The script downloads any missing NLTK data. It then pauses at the product-review prompt.

  • Paste The battery is excellent but the case feels cheap at the prompt.
  • Press Enter.

You should see the original review, every preprocessing list, Bag-of-Words counts, five TF-IDF keywords, four VADER scores, and one sign-based sentiment label.

Does the report stop at VADER?

Confirm that vader_lexicon remains in the package tuple inside download_nltk_data(). Check your internet connection if this is the first run because NLTK may still need to download that resource.

Ask for help with the sentiment stage:

Document the analyzer

A reusable project explains its choices as clearly as it presents its output. Function docstrings describe the code while README.md documents setup, limitations, and the role of word embeddings.

  • Find the existing download_nltk_data() function in analyzer.py.
  • Replace that function with the documented version below:
def download_nltk_data():
    """Download the three small NLTK resources used by this project."""
    for package in ("stopwords", "wordnet", "vader_lexicon"):
        nltk.download(package, quiet=True)

Why add this docstring?

The docstring records why the function exists. It also names the exact scope of the local language-data download.

  • Save analyzer.py.
  • Confirm that the docstring appears immediately below def download_nltk_data():.

You should see the new description inside the function body.

Does the docstring sit outside the function?

Indent the docstring with four spaces. Keep the package loop directly below it at the same indentation level.

Ask for help placing the docstring:

  • Find the existing lemmatize_as_noun_and_verb() function.
  • Replace that function with the documented version below:
def lemmatize_as_noun_and_verb(word):
    """Apply a simple noun pass followed by a verb pass."""
    noun_form = LEMMATIZER.lemmatize(word, pos="n")
    return LEMMATIZER.lemmatize(noun_form, pos="v")

What limitation does this document?

The docstring makes the two-pass heuristic explicit. Readers can see that this function applies fixed noun and verb passes without full part-of-speech tagging.

  • Save analyzer.py.
  • Confirm that the noun pass and verb pass remain unchanged below the docstring.

You should see the new description followed by the original two lemmatization lines.

Did the two-pass function change behavior?

Confirm that the first pass uses pos="n". Confirm that the returned second pass uses pos="v".

Ask for help checking the two passes:

  • Find the existing preprocess() function.
  • Replace that function with the documented version below:
def preprocess(text):
    """Return each visible stage of the review-cleaning pipeline."""
    tokens = TOKENIZER.tokenize(text)
    lowercase = [token.lower() for token in tokens]
    alphabetic = [token for token in lowercase if token.isalpha()]
    english_stopwords = set(stopwords.words("english"))
    filtered = [token for token in alphabetic if token not in english_stopwords]
    stems = [STEMMER.stem(token) for token in filtered]
    lemmas = [lemmatize_as_noun_and_verb(token) for token in filtered]
    return {"tokens": tokens, "lowercase": lowercase, "filtered": filtered, "stems": stems, "lemmas": lemmas}

What does this docstring promise?

The function returns every stage that the report displays. That promise matches the project's goal of keeping each text transformation visible.

  • Save analyzer.py.
  • Verify that the documented analyzer still runs by executing:
python analyzer.py

What does this documentation check prove?

Python parses all three new docstrings before starting the interactive prompt. Reaching that prompt confirms that the function bodies remain valid.

  • Paste The battery is excellent but the case feels cheap at the prompt.
  • Press Enter.

You should see the same complete report because docstrings document the functions without changing their results.

Did a docstring cause a syntax problem?

Confirm that each docstring begins and ends with three double quotation marks. Check that every docstring sits inside its function body.

Ask for help checking the documentation edits:

✔️ Awesome, I've got everything!

Great. Save analyzer.py before creating the README.

ⓧ I'd like to double check the full code

import nltk
from nltk.corpus import stopwords
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from nltk.stem.porter import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize.treebank import TreebankWordTokenizer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

SAMPLE_REVIEWS = [
    "The headphones sound clear and the battery lasts all day.",
    "This keyboard feels cheap and several keys stopped working.",
    "The camera is compact, but its low-light photos look noisy.",
    "Setup was easy and the smart speaker responds quickly.",
    "The blender is powerful, although the lid is difficult to clean.",
    "Delivery was slow, but the laptop itself performs very well.",
]

TOKENIZER = TreebankWordTokenizer()
STEMMER = PorterStemmer()
LEMMATIZER = WordNetLemmatizer()


def download_nltk_data():
    """Download the three small NLTK resources used by this project."""
    for package in ("stopwords", "wordnet", "vader_lexicon"):
        nltk.download(package, quiet=True)


def lemmatize_as_noun_and_verb(word):
    """Apply a simple noun pass followed by a verb pass."""
    noun_form = LEMMATIZER.lemmatize(word, pos="n")
    return LEMMATIZER.lemmatize(noun_form, pos="v")


def preprocess(text):
    """Return each visible stage of the review-cleaning pipeline."""
    tokens = TOKENIZER.tokenize(text)
    lowercase = [token.lower() for token in tokens]
    alphabetic = [token for token in lowercase if token.isalpha()]
    english_stopwords = set(stopwords.words("english"))
    filtered = [token for token in alphabetic if token not in english_stopwords]
    stems = [STEMMER.stem(token) for token in filtered]
    lemmas = [lemmatize_as_noun_and_verb(token) for token in filtered]
    return {"tokens": tokens, "lowercase": lowercase, "filtered": filtered, "stems": stems, "lemmas": lemmas}


def vectorize_review(review):
    documents = SAMPLE_REVIEWS + [review]
    cleaned_documents = [" ".join(preprocess(document)["lemmas"]) for document in documents]
    count_vectorizer = CountVectorizer(analyzer=str.split)
    count_matrix = count_vectorizer.fit_transform(cleaned_documents)
    count_terms = count_vectorizer.get_feature_names_out()
    review_counts = count_matrix[-1].toarray()[0]
    bag_of_words = sorted([(term, int(value)) for term, value in zip(count_terms, review_counts) if value > 0], key=lambda item: (-item[1], item[0]))
    tfidf_vectorizer = TfidfVectorizer(analyzer=str.split)
    tfidf_matrix = tfidf_vectorizer.fit_transform(cleaned_documents)
    tfidf_terms = tfidf_vectorizer.get_feature_names_out()
    review_weights = tfidf_matrix[-1].toarray()[0]
    top_keywords = sorted([(term, float(weight)) for term, weight in zip(tfidf_terms, review_weights) if weight > 0], key=lambda item: item[1], reverse=True)[:5]
    return bag_of_words, top_keywords


def sentiment_label(scores):
    """Apply the project's simple sign-based label rule."""
    if scores["compound"] > 0:
        return "POSITIVE"
    if scores["compound"] < 0:
        return "NEGATIVE"
    return "NEUTRAL"


def format_items(items, decimals=None):
    """Format term-value pairs for readable terminal output."""
    if not items:
        return "(none)"
    if decimals is None:
        return ", ".join(f"{term}: {value}" for term, value in items)
    return ", ".join(f"{term}: {value:.{decimals}f}" for term, value in items)


def print_report(review):
    """Run the complete analysis and print a transparent report."""
    stages = preprocess(review)
    bag_of_words, top_keywords = vectorize_review(review)
    sentiment_analyzer = SentimentIntensityAnalyzer()
    scores = sentiment_analyzer.polarity_scores(review)
    print("\nREVIEWLENS REPORT")
    print("=" * 60)
    print(f"Original:    {review}")
    print(f"Tokens:      {stages['tokens']}")
    print(f"Lowercase:   {stages['lowercase']}")
    print(f"Filtered:    {stages['filtered']}")
    print(f"Stems:       {stages['stems']}")
    print(f"Lemmas:      {stages['lemmas']}")
    print(f"Bag of Words: {format_items(bag_of_words)}")
    print(f"Top TF-IDF:   {format_items(top_keywords, decimals=3)}")
    print("VADER scores: " f"neg={scores['neg']:.3f}, " f"neu={scores['neu']:.3f}, " f"pos={scores['pos']:.3f}, " f"compound={scores['compound']:.3f}")
    print(f"Sentiment:    {sentiment_label(scores)}")


def main():
    download_nltk_data()
    print("ReviewLens: NLP Foundations")
    review = input("Paste an English product review: ").strip()
    if not review:
        print("Please enter a non-empty review.")
        return
    print_report(review)


if __name__ == "__main__":
    main()
  • Use the new-file control in the VS Code Explorer sidebar to create README.md inside reviewlens-nlp.
  • Add the project overview and Windows commands by pasting:
# ReviewLens: NLP Foundations

ReviewLens is a transparent command-line NLP project for analyzing English product reviews. It shows each text transformation instead of returning only a final label.

## What it shows

- Tokenization and lowercasing
- Alphabetic filtering and stopword removal
- Stemming versus lemmatization
- Bag-of-Words term counts
- TF-IDF keyword weights
- Rule-based VADER sentiment scores

## Setup on Windows

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

## Run

```powershell
python analyzer.py
```

Paste a non-empty English product review. The report prints the original review, every preprocessing stage, the nonzero Bag-of-Words features, five TF-IDF keywords, four VADER scores, and a sentiment label.

What does this first README section cover?

The opening explains the finished tool in plain language. The setup and run sections make the project reproducible on Windows.

  • Save README.md.
  • Confirm that README.md appears beside analyzer.py in the Explorer sidebar.

You should see the new documentation file listed in the open workspace.

Is the README missing from the workspace?

Confirm that the filename is exactly README.md. Check that you created it inside reviewlens-nlp.

Ask for help locating the file:

  • Append the technique table and design explanations to README.md by pasting:
## How the NLP pipeline works

| Technique | Purpose | Main limitation |
| --- | --- | --- |
| Tokenization | Splits raw text into processable units | Rules differ across languages and domains |
| Stopword removal | Removes many high-frequency function words | Some removed words can matter for the task |
| Stemming | Mechanically strips word endings | Can produce forms that are not dictionary words |
| Lemmatization | Maps words toward dictionary base forms | Good results depend on linguistic context |
| Bag-of-Words | Counts terms in each document | Ignores most word order and meaning |
| TF-IDF | Emphasizes terms that are distinctive in a corpus | Depends strongly on the corpus used to fit it |
| VADER | Applies a sentiment lexicon and hand-built rules | It is not trained on this product-review corpus |

## What word embeddings add

Bag-of-Words and TF-IDF give each vocabulary term its own feature position. Word embeddings instead represent words or larger text units as dense numerical vectors designed to capture aspects of meaning and context. A later project will use a pretrained transformer, whose contextual representations can distinguish usages that a simple count matrix cannot.

## Sentiment-label rule

This learning project keeps its label rule visible: a positive VADER compound value becomes `POSITIVE`, a negative value becomes `NEGATIVE`, and zero becomes `NEUTRAL`. The four raw scores are always displayed so the label is not a black box.

Why document the trade-offs?

Each technique solves a specific problem. The limitation column shows where a future model or richer linguistic pipeline can improve the analyzer.

The embeddings section connects sparse count features to dense meaning-oriented vectors. It prepares the path toward pretrained transformer models.

  • Save README.md.
  • Scroll to the technique table in README.md.

You should see seven technique rows followed by the embeddings and sentiment-label explanations.

Does the Markdown table look uneven?

Confirm that every table row starts and ends with a vertical bar. Keep the separator row directly below the three column headings.

Ask for help checking the table:

  • Append the limitations and project roadmap to README.md by pasting:
## Limitations

- The built-in corpus is intentionally tiny, so TF-IDF rankings are demonstrations rather than production-quality keywords.
- The lemmatizer uses a simple noun pass followed by a verb pass instead of full part-of-speech tagging.
- Stopword removal can discard useful negation words, so VADER analyzes the original review rather than the filtered words.
- The analyzer is English-only and uses rule-based sentiment rather than a trained product-review classifier.

## Project roadmap

1. Train and evaluate a sentiment classifier with a scikit-learn pipeline.
2. Extract brands, products, and places with spaCy named entity recognition.
3. Compare the classifier with a pretrained model from Hugging Face Transformers.
4. Present all three analyses in a Streamlit app.

What makes these sections useful?

The limitations set honest expectations for the tiny corpus and fixed preprocessing rules. The roadmap turns those limitations into concrete follow-on projects.

  • Save README.md.
  • Scroll to the bottom of README.md.

You should see four limitations followed by four numbered roadmap ideas.

Are the final sections missing?

Confirm that you pasted the new content after the sentiment-label paragraph. Check that both section headings begin with two hash characters.

Ask for help checking the README structure:

✔️ Awesome, I've got everything!

Your README now explains how to run ReviewLens, how the techniques work, and where the project can grow.

ⓧ I'd like to double check the full code

# ReviewLens: NLP Foundations

ReviewLens is a transparent command-line NLP project for analyzing English product reviews. It shows each text transformation instead of returning only a final label.

## What it shows

- Tokenization and lowercasing
- Alphabetic filtering and stopword removal
- Stemming versus lemmatization
- Bag-of-Words term counts
- TF-IDF keyword weights
- Rule-based VADER sentiment scores

## Setup on Windows

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

## Run

```powershell
python analyzer.py
```

Paste a non-empty English product review. The report prints the original review, every preprocessing stage, the nonzero Bag-of-Words features, five TF-IDF keywords, four VADER scores, and a sentiment label.

## How the NLP pipeline works

| Technique | Purpose | Main limitation |
| --- | --- | --- |
| Tokenization | Splits raw text into processable units | Rules differ across languages and domains |
| Stopword removal | Removes many high-frequency function words | Some removed words can matter for the task |
| Stemming | Mechanically strips word endings | Can produce forms that are not dictionary words |
| Lemmatization | Maps words toward dictionary base forms | Good results depend on linguistic context |
| Bag-of-Words | Counts terms in each document | Ignores most word order and meaning |
| TF-IDF | Emphasizes terms that are distinctive in a corpus | Depends strongly on the corpus used to fit it |
| VADER | Applies a sentiment lexicon and hand-built rules | It is not trained on this product-review corpus |

## What word embeddings add

Bag-of-Words and TF-IDF give each vocabulary term its own feature position. Word embeddings instead represent words or larger text units as dense numerical vectors designed to capture aspects of meaning and context. A later project will use a pretrained transformer, whose contextual representations can distinguish usages that a simple count matrix cannot.

## Sentiment-label rule

This learning project keeps its label rule visible: a positive VADER compound value becomes `POSITIVE`, a negative value becomes `NEGATIVE`, and zero becomes `NEUTRAL`. The four raw scores are always displayed so the label is not a black box.

## Limitations

- The built-in corpus is intentionally tiny, so TF-IDF rankings are demonstrations rather than production-quality keywords.
- The lemmatizer uses a simple noun pass followed by a verb pass instead of full part-of-speech tagging.
- Stopword removal can discard useful negation words, so VADER analyzes the original review rather than the filtered words.
- The analyzer is English-only and uses rule-based sentiment rather than a trained product-review classifier.

## Project roadmap

1. Train and evaluate a sentiment classifier with a scikit-learn pipeline.
2. Extract brands, products, and places with spaCy named entity recognition.
3. Compare the classifier with a pretrained model from Hugging Face Transformers.
4. Present all three analyses in a Streamlit app.
Commit the reproducible project

The source code, pinned dependencies, ignore rules, and README now form one reproducible project snapshot. Your Git identity from the earlier setup labels the commit.

  • Switch back to the PowerShell terminal from earlier.
  • Confirm that its prompt is inside reviewlens-nlp.

Before you create the snapshot, what files do you expect Git to record? Which local environment folder should the ignore rules leave out?

  • Initialize the repository, stage the project files, and record the first snapshot by running:
git init
git add .
git commit -m 'Initial project version'

What do these commands record?

  • The first command creates the local repository history.
  • The second command stages the current project files while respecting .gitignore.
  • The third command records the staged files with the message Initial project version.

Git should print a commit summary naming Initial project version. The summary should report the newly recorded project files.

Did Git refuse to create the commit?

Confirm that the terminal is inside reviewlens-nlp before running the commands. If Git asks for an identity, return to the name and email configuration from the workspace setup step.

Ask for help with the first commit:

Before the final run, predict which report lines should stay tied to cleaned text. Which lines should still use the original review?

  • Run the committed analyzer by executing:
python analyzer.py

What does this final run verify?

The command exercises the complete interactive entry point from the committed workspace. It verifies preprocessing, numerical features, keyword ranking, and sentiment in one run.

  • Paste The battery is excellent but the case feels cheap at the prompt.
  • Press Enter.

You should see every preprocessing stage, nonzero Bag-of-Words counts, five TF-IDF keywords, the neg, neu, pos, and compound VADER scores, plus one sentiment label.

Does the final run miss part of the report?

Compare analyzer.py with the full-code tab. Confirm that main() calls print_report(review) after the empty-input check.

Ask for help tracing the report:

That is the complete project working: ReviewLens now explains its text transformations, numerical features, sentiment decision, limitations, and reproducible setup.

Secret mission

Compare Stems and Lemmas as Keyword Features

Turn normalization into a controlled experiment by letting ReviewLens switch between lemma-based and stem-based keywords. You will compare both modes while the original review continues through the same sentiment path.

Clean Up Your Resources

Clean Up Your Resources

All processing stayed on your Windows computer with no cloud resources or ongoing costs. Decide whether to keep the reviewlens-nlp workspace, pause the active environment, or delete the folder.

Resources you used:

  • The reviewlens-nlp workspace in Visual Studio Code containing analyzer.py, requirements.txt, .gitignore, and README.md.
  • The .venv Python virtual environment stored inside reviewlens-nlp.
  • The local Git repository stored in .git inside reviewlens-nlp.

Keep everything running

No action is needed. Choose this option if you want to compare more reviews or extend the analyzer later.

  • Keep the reviewlens-nlp folder with its code, documentation, environment, and Git history.
  • Reuse .venv when you return to the project.
  • Leave the downloaded NLTK data in place for future NLP projects.

Pause - I'll come back to this later

Pause the project by leaving every file in place. Deactivating .venv ends the active environment for the current terminal session.

  • Return to the integrated terminal in Visual Studio Code.
  • Deactivate the environment by running this command:
deactivate

The terminal stops using packages from .venv. Your project files remain unchanged.

  • Close Visual Studio Code.

Delete - I don't want to use this again

Remove the reviewlens-nlp workspace when you want a clean start. This permanently removes the project from its current location.

Deletion is permanent

The folder contains your code, documentation, virtual environment, and Git history. Keep the folder if you want to revisit any of that work.

  • Switch back to the integrated terminal in Visual Studio Code.
  • Deactivate the environment by running this command:
deactivate

The environment leaves the current PowerShell session. The folder is now ready to remove.

  • Close Visual Studio Code.
  • Press the Windows key to open search.
  • Type File Explorer into search.
  • Press Enter to open File Explorer.
  • Locate the reviewlens-nlp folder.
  • Right-click the reviewlens-nlp folder.
  • Select Delete.

You should no longer see reviewlens-nlp in its previous location. The project files and local Git history are gone together.

Keep the Shared NLTK Data

Downloaded NLTK data stays outside the deleted workspace. Keeping it saves another download when you start a later NLP project.

Nice Work!

Nice Work!

You did it! You built ReviewLens, a transparent command-line analyzer that reveals how an English product review becomes features, keywords, and sentiment scores.

You've learned how to:

  • Build a transparent NLP preprocessing pipeline that displays tokenization, lowercasing, stopword removal, stemming, and lemmatization.
  • Transform a review corpus into inspectable Bag-of-Words counts plus TF-IDF keyword weights.
  • Score original reviews with a rule-based VADER sentiment baseline while keeping its four raw scores plus the sign-based label visible.
  • Secret Mission: Added a stem-versus-lemma switch that changes Bag-of-Words counts plus TF-IDF rankings while leaving raw-text VADER scoring unchanged.

Ready to quiz yourself?