Personal Finance Savings Predictor

Build an ML model to predict disposable income from spending habits.

Introduction

30 Second Summary

Everyone has a monthly budget, whether they track it or not. Your rent, groceries, and subscriptions all compete for the same pool of income. But which spending habits actually determine how much you save?

In this project, you will analyze a 20,000-person personal finance dataset, uncover spending patterns across demographics, and train a machine learning model that predicts how much disposable income someone will have based on their profile and habits.

What You'll Build

A complete data science notebook that takes raw financial data through cleaning, exploration, visualization, and predictive modeling to produce actionable savings insights.

By the end of this project, you'll have:

  • A cleaned and analyzed personal finance dataset with insights about spending patterns across demographics.
  • A trained Random Forest regression model that predicts an individual's disposable income based on their profile and spending habits.
  • A feature importance analysis that reveals which spending categories have the most impact on savings potential.
  • Secret Mission: Build an anomaly detection system to flag individuals who are overspending relative to their demographic peers.

Are there any prerequisites?

You should be comfortable with Python and common data science libraries like pandas, numpy, and matplotlib. You will also need access to a Jupyter Notebook environment.

Set Up Libraries and Load Data

Before you can explore spending patterns and build predictions, you need data to work with. This project uses a dataset of 20,000 individuals' financial profiles covering income, occupation, and detailed expense breakdowns across 11 spending categories.

In this step, you'll download the dataset from Kaggle, set up your analysis notebook, and confirm everything loads correctly.

In this step, get ready to:
  • Download the personal finance dataset from Kaggle.
  • Import data science libraries into a new notebook.
  • Load the dataset and confirm its structure.
Download the dataset from Kaggle

The dataset you need is called "Indian Personal Finance and Spending Habits" and contains financial records for 20,000 individuals. Kaggle is a free data science platform that hosts public datasets for analysis.

  • Open the dataset page in your browser:
  • https://www.kaggle.com/datasets/shriyashjagtap/indian-personal-finance-and-spending-habits
  • If you don't have a Kaggle account, click Register and sign up for a free account using your email or Google login.
  • Once signed in, click the Download button near the top-right of the dataset page.
  • The download arrives as a ZIP file. Extract it to find the CSV file inside.
  • Rename the CSV file to indian_personal_finance.csv and place it in the same folder where you'll create your notebook.

Why rename the file?

The filename on Kaggle may differ from what the code expects. Renaming it to indian_personal_finance.csv ensures the pd.read_csv() call in your notebook finds the file without errors.

Don't see a Download button?

Make sure you are signed in to your Kaggle account. The download button only appears for logged-in users.

help me download the dataset from Kaggle

Create your notebook and import libraries

With the dataset ready, you need a notebook to write your analysis in. Each notebook cell runs independently, which makes it ideal for building an analysis piece by piece.

  • In Jupyter Notebook, click New in the top-right corner and select Python 3 to create a new notebook.
  • Click the notebook title at the top (it defaults to "Untitled") and rename it to finance_savings_predictor.
  • Make sure this notebook is saved in the same folder as your indian_personal_finance.csv file.
  • In the first cell of your notebook, paste the following import code:
# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

What do these libraries do?

  • pandas loads and manipulates tabular data (your CSV becomes a DataFrame).
  • numpy provides math operations like square roots and array handling.
  • matplotlib and seaborn create visualizations. Seaborn builds on Matplotlib with statistical chart types.
  • scikit-learn provides machine learning tools: splitting data, training models, and measuring accuracy.
  • Run the cell by pressing Shift+Enter.
  • The cell should complete with no output. If it finishes without errors, all libraries imported successfully.

Seeing a ModuleNotFoundError?

If you see an error like ModuleNotFoundError: No module named 'sklearn', install the missing library by running this in a new cell above your imports:

pip install pandas numpy matplotlib seaborn scikit-learn

After it finishes, re-run your imports cell.

help me fix my import errors in Jupyter Notebook

Load the dataset and verify its structure

Now that your libraries are ready, you can load the CSV file into a DataFrame. A DataFrame is a table-like structure that lets you filter, sort, and analyze data with simple commands.

  • In the next empty cell, paste this code to load and preview the dataset:
# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

What does this code do?

  • pd.read_csv('indian_personal_finance.csv') reads the CSV file into a DataFrame called df.
  • df.shape prints the number of rows and columns as a tuple.
  • df.head() displays the first 5 rows so you can see the column names and sample values.
  • Run the cell by pressing Shift+Enter.
  • You should see Dataset shape: followed by a tuple showing approximately 20,000 rows. Below that, a table preview displays columns including Income, Age, Dependents, Occupation, and City_Tier.

Seeing a FileNotFoundError?

This means Python cannot find indian_personal_finance.csv in the same folder as your notebook. Double-check two things:

  • The CSV file is in the exact same folder as finance_savings_predictor.ipynb.
  • The filename is spelled exactly indian_personal_finance.csv (underscores, not hyphens, all lowercase).

help me fix the FileNotFoundError when loading my CSV

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook by pressing Ctrl+S (Windows) or Cmd+S (macOS).

ⓧ I'd like to double check the full code

Here is the complete code you should have across your two notebook cells so far:

# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

Your environment is ready and the data is loaded. Next up, you'll clean this dataset and engineer features to prepare it for machine learning.

Clean and Prepare the Data

Your notebook is set up and the raw dataset is loaded. Now it's time to understand what's actually in the data and prepare it for analysis.

Raw datasets almost always contain issues that would distort your results. Duplicate rows inflate your statistics, missing values crash your model, and text-based categories can't be fed into a machine learning algorithm directly. This step handles all of those problems so you can trust the numbers in the steps ahead.

In this step, get ready to:
  • Inspect the dataset structure, data types, and missing values.
  • Clean the data by removing duplicates and missing values, then create a Total_Expenses column.
  • Encode categorical variables so they can be used in a machine learning model.
Inspect the dataset structure

Before fixing anything, you need to know what you're working with. How many columns are numeric vs text? Are there missing values? What do the distributions look like?

  • Create a new cell in your notebook and add this inspection code:
# Cell 3: Data Inspection
print("=== Data Types ===")
print(df.dtypes)
print("\n=== Missing Values ===")
print(df.isnull().sum())
print("\n=== Basic Statistics ===")
df.describe()

What does this code do?

This cell runs three diagnostic checks on your DataFrame.

  • df.dtypes shows the data type of every column. You want numeric types (int64, float64) for calculations and object types for categorical text.
  • df.isnull().sum() counts how many missing values exist per column. Any non-zero value means data is incomplete.
  • df.describe() gives summary statistics (mean, min, max, percentiles) for every numeric column so you can spot outliers or impossible values.
  • Run the cell by pressing Shift+Enter.
  • You should see three sections of output: a list of column data types, a count of missing values per column, and a statistics table covering Income, Age, Dependents, and all expense columns.

Not seeing any output?

  • Make sure you ran Cell 2 (the data loading cell) before this one. If df doesn't exist yet, the kernel throws a NameError.
  • Check that your CSV file is in the same folder as your notebook. If the path is wrong, Cell 2 would have failed silently or errored.
  • help me debug my data inspection cell
Clean the data and engineer a total expenses column

Now that you know the dataset's shape, it's time to fix the problems. Duplicates and missing values would skew your model's training data. You'll also create a new column that sums all expense categories into one number.

  • Create a new cell and add this data cleaning code:
# Cell 4: Data Cleaning
# Check for duplicates
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()

# Handle missing values
df = df.dropna()

# Verify no negatives in expense columns
expense_cols = ['Rent', 'Loan_Repayment', 'Insurance', 'Groceries', 'Transport',
                'Eating_Out', 'Entertainment', 'Utilities', 'Healthcare',
                'Education', 'Miscellaneous']

print(f"\nNegative values in expenses: {(df[expense_cols] < 0).sum().sum()}")

# Create Total_Expenses column
df['Total_Expenses'] = df[expense_cols].sum(axis=1)
print(f"\nTotal_Expenses range: {df['Total_Expenses'].min():.0f} to {df['Total_Expenses'].max():.0f}")

What does this code do?

  • df.duplicated().sum() counts identical rows, and drop_duplicates() removes them. Duplicate rows would give certain individuals more weight in your model.
  • df.dropna() removes any row with a missing value. Machine learning models can't handle blanks.
  • The negative value check confirms that no expense column contains impossible values (you can't spend negative money on groceries).
  • expense_cols defines the 11 spending categories. Summing them into Total_Expenses gives you a single number representing each person's total monthly outflow.
  • Run the cell with Shift+Enter.
  • You should see the duplicate count, a confirmation that negative expense values are 0, and the range of your new Total_Expenses column.

Seeing a KeyError on an expense column?

  • Double check that the column names in your expense_cols list match the exact names from the dataset. Column names are case-sensitive. Run df.columns.tolist() to see the actual column names.
  • If you accidentally ran Cell 4 twice without re-running Cell 2, the DataFrame may be empty from the double cleaning pass. Re-run Cell 2 first to reload the data.
  • help me fix my data cleaning cell
Encode categorical variables for machine learning

Your dataset has two text-based columns that describe each individual: their occupation and their city tier. Machine learning models only understand numbers, so you need to convert these categories into numeric columns.

  • Create a new cell and add the encoding code:
# Cell 5: Encode Categorical Variables
df_encoded = pd.get_dummies(df, columns=['Occupation', 'City_Tier'], drop_first=True)
print(f"Shape after encoding: {df_encoded.shape}")

What is one-hot encoding?

pd.get_dummies() converts each unique category into its own column filled with 0s and 1s. For example, if Occupation has values like "Engineer" and "Doctor", you get columns Occupation_Engineer and Occupation_Doctor.

The drop_first=True parameter removes one category from each group to avoid multicollinearity. The dropped category becomes the implicit "baseline" that all others are compared against.

  • Run the cell with Shift+Enter.
  • You should see the new shape of your DataFrame. It will have more columns than before because each category became its own column.

Shape looks the same as before?

  • Make sure you're printing df_encoded.shape and not df.shape. The encoded DataFrame is stored in a new variable called df_encoded.
  • If you see an error about column names not being found, confirm that Occupation and City_Tier still exist in df. Running Cell 4 multiple times shouldn't remove them, but double check with df.columns.tolist().
  • help me fix my categorical encoding
  • Verify your cleaning was thorough by running this quick check in a new cell:
# Quick verification
print(f"Missing values remaining: {df.isnull().sum().sum()}")
print(f"Total_Expenses column exists: {'Total_Expenses' in df.columns}")
print(f"df_encoded columns: {df_encoded.shape[1]}")

You should see zero missing values, confirmation that the Total_Expenses column exists, and a column count higher than the original dataset.

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here's everything your notebook should contain up to this point:

# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

# Cell 3: Data Inspection
print("=== Data Types ===")
print(df.dtypes)
print("\n=== Missing Values ===")
print(df.isnull().sum())
print("\n=== Basic Statistics ===")
df.describe()

# Cell 4: Data Cleaning
# Check for duplicates
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()

# Handle missing values
df = df.dropna()

# Verify no negatives in expense columns
expense_cols = ['Rent', 'Loan_Repayment', 'Insurance', 'Groceries', 'Transport',
                'Eating_Out', 'Entertainment', 'Utilities', 'Healthcare',
                'Education', 'Miscellaneous']

print(f"\nNegative values in expenses: {(df[expense_cols] < 0).sum().sum()}")

# Create Total_Expenses column
df['Total_Expenses'] = df[expense_cols].sum(axis=1)
print(f"\nTotal_Expenses range: {df['Total_Expenses'].min():.0f} to {df['Total_Expenses'].max():.0f}")

# Cell 5: Encode Categorical Variables
df_encoded = pd.get_dummies(df, columns=['Occupation', 'City_Tier'], drop_first=True)
print(f"Shape after encoding: {df_encoded.shape}")

Your data is clean, complete, and encoded. Next up, you'll use this prepared dataset to create visualizations that reveal how different demographics spend and save their money.

Explore Data and Build Visualizations

Your data is clean, encoded, and ready for analysis. Now it is time to ask the data questions and let the patterns reveal themselves visually.

In this step, you will build five professional visualizations that uncover how income, expenses, and demographics interact. These charts form the core of your exploratory data analysis and will guide the machine learning model you build in the next step.

In this step, get ready to:
  • Create a correlation heatmap to identify relationships between financial variables.
  • Build demographic comparison visualizations across city tiers and occupations.
  • Calculate and visualize savings rates to identify which groups save the most.
Create the correlation heatmap

A correlation heatmap shows how strongly each pair of numeric variables moves together. Values close to 1 or -1 indicate a strong relationship. Values near 0 mean the variables are unrelated.

  • Create a new cell in your notebook (Cell 6) and build the correlation heatmap by pasting the following code:
# Cell 6: Correlation Heatmap
plt.figure(figsize=(12, 8))
numeric_cols = ['Income', 'Age', 'Dependents'] + expense_cols + ['Disposable_Income', 'Total_Expenses']
correlation_matrix = df[numeric_cols].corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, fmt='.2f')
plt.title('Correlation Heatmap: Income, Expenses, and Savings')
plt.tight_layout()
plt.show()

What does this code do?

This code selects only the numeric financial columns and calculates a correlation matrix using .corr(). Each cell in the resulting heatmap shows how strongly two variables correlate.

The annot=True parameter prints the correlation values inside each cell. The coolwarm colormap highlights positive correlations in red and negative ones in blue.

  • Run the cell.
  • You should see a large color-coded grid where Income and Disposable_Income are strongly correlated (a deep red cell). Expense categories should show moderate positive correlations with Income.

Seeing a KeyError about column names?

Make sure expense_cols was defined in Cell 4 and that you ran Cell 4 before this one. Jupyter cells must be run in order for variables to be available.

Also confirm that your CSV loaded correctly in Cell 2. Run print(df.columns.tolist()) to verify the column names match exactly.

Help me debug my correlation heatmap

Build demographic comparison visualizations

Now you will compare how spending patterns differ across city tiers and occupations. These charts reveal whether location or career has a bigger impact on financial habits.

  • Create a new cell (Cell 7) and build the spending-by-city-tier bar chart by pasting this code:
# Cell 7: Spending by City Tier
plt.figure(figsize=(10, 6))
city_spending = df.groupby('City_Tier')[expense_cols].mean()
city_spending.plot(kind='bar', figsize=(12, 6))
plt.title('Average Spending by Category and City Tier')
plt.xlabel('City Tier')
plt.ylabel('Average Amount')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()

What does this code do?

This groups the data by City_Tier and calculates the average spending in each expense category. The grouped bar chart lets you compare spending patterns across urban tiers at a glance.

The legend is positioned outside the chart using bbox_to_anchor so it does not overlap the bars.

  • Run the cell.
  • You should see a grouped bar chart with clusters of colored bars for each city tier, showing how average spending differs across categories like Rent, Groceries, and Entertainment.

Chart looks squished or legend overlaps?

Make sure both plt.tight_layout() and plt.show() are at the end of the cell. The tight_layout() call adjusts spacing so the legend and labels fit properly.

Help me fix my bar chart layout

  • Create a new cell (Cell 8) and build the disposable income box plot by pasting this code:
# Cell 8: Disposable Income by Occupation
plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='Occupation', y='Disposable_Income')
plt.title('Disposable Income Distribution by Occupation')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

What does this code do?

A box plot shows the distribution of disposable income for each occupation group. The box represents the middle 50% of values (the interquartile range). The line inside the box is the median.

Dots beyond the whiskers are outliers. These are individuals whose savings are unusually high or low compared to their peers.

  • Run the cell.
  • You should see a series of box-and-whisker plots, one per occupation, with x-axis labels rotated at 45 degrees for readability.

Labels are cut off at the bottom?

Confirm that plt.xticks(rotation=45) and plt.tight_layout() are both present. Without tight_layout(), rotated labels can extend past the figure boundary.

Help me fix cut-off labels on my plot

  • Create a new cell (Cell 9) and build the income vs. expenses scatter plot by pasting this code:
# Cell 9: Income vs Total Expenses
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='Income', y='Total_Expenses', hue='City_Tier', alpha=0.5)
plt.title('Income vs Total Expenses by City Tier')
plt.xlabel('Monthly Income')
plt.ylabel('Total Monthly Expenses')
plt.tight_layout()
plt.show()

What does this code do?

This scatter plot places every individual as a dot, with income on the x-axis and total expenses on the y-axis. The hue='City_Tier' parameter colors each dot by city tier so you can spot geographic spending patterns.

The alpha=0.5 makes dots semi-transparent. This helps you see density in areas where many points overlap.

  • Run the cell.
  • You should see a cloud of colored dots trending upward. This confirms that higher income generally leads to higher total expenses, with color clusters showing city tier differences.

All dots are the same color?

Make sure hue='City_Tier' is spelled exactly as it appears in your DataFrame. Run df['City_Tier'].unique() to confirm the column exists and has multiple values.

Help me fix my scatter plot coloring

Calculate and visualize the savings rate

The final visualization ties everything together. You will calculate each person's savings rate as a percentage of income, then examine which occupation groups are the strongest savers.

  • Create a new cell (Cell 10) and calculate the savings rate and its distribution by pasting this code:
# Cell 10: Savings Rate Analysis
df['Savings_Rate'] = (df['Disposable_Income'] / df['Income']) * 100

plt.figure(figsize=(10, 6))
sns.histplot(df['Savings_Rate'], bins=50, kde=True)
plt.title('Distribution of Savings Rate (%)')
plt.xlabel('Savings Rate (%)')
plt.ylabel('Count')
plt.tight_layout()
plt.show()

# Savings rate by occupation
print("\n=== Average Savings Rate by Occupation ===")
print(df.groupby('Occupation')['Savings_Rate'].mean().sort_values(ascending=False))

What does this code do?

First, it creates a new column Savings_Rate by dividing Disposable_Income by Income and multiplying by 100. This tells you what percentage of their income each person keeps after expenses.

The histogram shows how savings rates are distributed across the entire dataset. The kde=True parameter overlays a smooth density curve that makes the distribution shape easier to read.

The final print statement ranks occupations by average savings rate, answering which demographic group saves the most and which saves the least.

  • Run the cell.
  • You should see a histogram with a smooth curve overlay, plus a printed ranking of occupations by average savings rate. The occupation at the bottom of the list is the demographic group that saves the least.

Seeing inf or NaN values in savings rate?

This happens if any rows have an Income value of 0 (dividing by zero produces infinity). Run df[df['Income'] == 0].shape to check. If there are zero-income rows, filter them out before calculating the savings rate.

Help me fix inf values in my savings rate calculation

✔️ Awesome, I've got everything!

Great. Double check you have run all cells from Cell 6 through Cell 10 and that each visualization is displaying correctly.

ⓧ I'd like to double check the full code

Here is the full notebook code up to the end of this step. Compare your cells against this:

# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

# Cell 3: Data Inspection
print("=== Data Types ===")
print(df.dtypes)
print("\n=== Missing Values ===")
print(df.isnull().sum())
print("\n=== Basic Statistics ===")
df.describe()

# Cell 4: Data Cleaning
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()
df = df.dropna()
expense_cols = ['Rent', 'Loan_Repayment', 'Insurance', 'Groceries', 'Transport',
                'Eating_Out', 'Entertainment', 'Utilities', 'Healthcare',
                'Education', 'Miscellaneous']
print(f"\nNegative values in expenses: {(df[expense_cols] < 0).sum().sum()}")
df['Total_Expenses'] = df[expense_cols].sum(axis=1)
print(f"\nTotal_Expenses range: {df['Total_Expenses'].min():.0f} to {df['Total_Expenses'].max():.0f}")

# Cell 5: Encode Categorical Variables
df_encoded = pd.get_dummies(df, columns=['Occupation', 'City_Tier'], drop_first=True)
print(f"Shape after encoding: {df_encoded.shape}")

# Cell 6: Correlation Heatmap
plt.figure(figsize=(12, 8))
numeric_cols = ['Income', 'Age', 'Dependents'] + expense_cols + ['Disposable_Income', 'Total_Expenses']
correlation_matrix = df[numeric_cols].corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, fmt='.2f')
plt.title('Correlation Heatmap: Income, Expenses, and Savings')
plt.tight_layout()
plt.show()

# Cell 7: Spending by City Tier
plt.figure(figsize=(10, 6))
city_spending = df.groupby('City_Tier')[expense_cols].mean()
city_spending.plot(kind='bar', figsize=(12, 6))
plt.title('Average Spending by Category and City Tier')
plt.xlabel('City Tier')
plt.ylabel('Average Amount')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()

# Cell 8: Disposable Income by Occupation
plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='Occupation', y='Disposable_Income')
plt.title('Disposable Income Distribution by Occupation')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Cell 9: Income vs Total Expenses
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='Income', y='Total_Expenses', hue='City_Tier', alpha=0.5)
plt.title('Income vs Total Expenses by City Tier')
plt.xlabel('Monthly Income')
plt.ylabel('Total Monthly Expenses')
plt.tight_layout()
plt.show()

# Cell 10: Savings Rate Analysis
df['Savings_Rate'] = (df['Disposable_Income'] / df['Income']) * 100
plt.figure(figsize=(10, 6))
sns.histplot(df['Savings_Rate'], bins=50, kde=True)
plt.title('Distribution of Savings Rate (%)')
plt.xlabel('Savings Rate (%)')
plt.ylabel('Count')
plt.tight_layout()
plt.show()
print("\n=== Average Savings Rate by Occupation ===")
print(df.groupby('Occupation')['Savings_Rate'].mean().sort_values(ascending=False))

Your EDA is complete. You have five visualizations that tell a clear story about how income, spending, and demographics interact. Next up, you will use these insights to train a machine learning model that predicts disposable income.

Train the Random Forest Model

Your EDA revealed clear patterns in spending behavior across demographics. Now it's time to turn those insights into a predictive model.

In this step, you'll train a Random Forest regression model to predict how much disposable income a person will have based on their profile and spending habits. This is the core of your data science notebook: a model that learns from data and makes predictions.

In this step, get ready to:
  • Prepare features and split data into training and test sets.
  • Train a Random Forest regression model on the training data.
  • Evaluate model accuracy using R-squared and RMSE metrics.
Prepare features and split the data

Before training a model, you need to separate your data into inputs (features) and the output you want to predict (target). You also need to hold back some data for testing so you can measure how well the model performs on data it has never seen.

  • Create a new cell in your notebook and add the following code to define your features and split the data:
# Cell 11: Prepare Data for ML
feature_cols = [col for col in df_encoded.columns if col not in
                ['Disposable_Income', 'Desired_Savings', 'Desired_Savings_Percentage',
                 'Savings_Rate', 'Total_Expenses']]
X = df_encoded[feature_cols]
y = df_encoded['Disposable_Income']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Training set: {X_train.shape}")
print(f"Test set: {X_test.shape}")

What does this code do?

This cell prepares your data for machine learning in three steps:

  • feature_cols selects all columns EXCEPT the ones that leak the answer. You exclude Disposable_Income (that's your target), plus Desired_Savings, Desired_Savings_Percentage, Savings_Rate, and Total_Expenses because they are derived from the target or would give the model an unfair shortcut.
  • X holds all the feature columns (inputs) and y holds the target variable (what you want to predict).
  • train_test_split reserves 20% of the data for testing. The random_state=42 ensures your split is reproducible every time you run the cell.
  • Run the cell.
  • You should see output showing the training set shape (approximately 80% of your rows) and the test set shape (approximately 20%).

Seeing a NameError for df_encoded?

This means the encoding cell (Cell 5) has not been run in your current session. Jupyter Notebooks lose state if the kernel restarts.

  • Go to the menu and click **Kernel**, then **Restart & Run All** to re-execute every cell from the top.

Why is my df_encoded variable not found even though I defined it earlier?

Train the Random Forest model

Why Random Forest over linear regression?

Random Forest handles non-linear relationships and feature interactions automatically. It is also robust to outliers and provides built-in feature importance rankings that you will use in the next step.

Linear regression assumes a straight-line relationship between inputs and output. Real financial data is messier than that. Random Forest combines many decision trees to capture complex patterns without manual feature engineering.

  • Create a new cell and add the following code to train the model and generate predictions:
# Cell 12: Train Random Forest Model
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

What does this code do?

  • RandomForestRegressor(n_estimators=100, random_state=42) creates a forest of 100 decision trees. Each tree learns a slightly different pattern from the data, and their combined prediction is more accurate than any single tree.
  • model.fit(X_train, y_train) trains the model on your training data. This is where the model learns the relationship between features and disposable income.
  • model.predict(X_test) asks the trained model to predict disposable income for the test data it has never seen before.
  • Run the cell.
  • This cell may take a few seconds to run as the model trains 100 decision trees. No output is printed, but the model is now stored in the model variable and predictions are stored in y_pred.

Cell taking a very long time?

Training 100 trees on around 20,000 rows should finish in under 30 seconds on most machines. If it hangs for minutes, your kernel may be stuck.

  • Click **Kernel**, then **Interrupt** to stop the current cell. Try running it again.

Why is my Random Forest model taking so long to train?

Evaluate model performance

A model is only useful if it makes accurate predictions. You need to measure how close your predictions are to the actual values in the test set.

  • Create a new cell and add the following code to calculate and print your model's evaluation metrics:
# Cell 13: Model Evaluation
r2 = r2_score(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
print("=== Model Performance ===")
print(f"R-squared: {r2:.4f}")
print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")

What do these metrics mean?

  • R-squared measures how much of the variation in disposable income your model explains. A score of 1.0 means perfect predictions. A score above 0.90 indicates an excellent fit.
  • Mean Squared Error (MSE) is the average of squared differences between predicted and actual values. Lower is better.
  • Root Mean Squared Error (RMSE) is the square root of MSE, expressed in the same units as your target (currency). It tells you the typical size of your prediction errors.
  • Run the cell.
  • You should see printed output with your R-squared score (likely above 0.90), MSE, and RMSE values.

Why is the R-squared so high?

Disposable income is directly calculated from income minus expenses in this dataset. The model has access to both income and all expense categories as features, so the mathematical relationship is almost fully captured. In real-world datasets with noisier relationships, R-squared scores above 0.80 are considered strong.

R-squared is very low or negative?

  • Check that your feature_cols list in Cell 11 excludes the correct columns. Including Total_Expenses or Savings_Rate as features should not cause a low score, but accidentally including Disposable_Income itself or excluding Income from features will.
  • Verify that you ran Cell 4 (cleaning) and Cell 5 (encoding) before Cell 11. Stale data from an earlier kernel state can produce unexpected results.

Help me debug my low R-squared score in my Random Forest model.

✔️ Awesome, I've got everything!

Great. Double check that you have saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the full notebook up to this point. Compare it against your cells:

# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

# Cell 3: Data Inspection
print("=== Data Types ===")
print(df.dtypes)
print("\n=== Missing Values ===")
print(df.isnull().sum())
print("\n=== Basic Statistics ===")
df.describe()

# Cell 4: Data Cleaning
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()
df = df.dropna()
expense_cols = ['Rent', 'Loan_Repayment', 'Insurance', 'Groceries', 'Transport',
                'Eating_Out', 'Entertainment', 'Utilities', 'Healthcare',
                'Education', 'Miscellaneous']
print(f"\nNegative values in expenses: {(df[expense_cols] < 0).sum().sum()}")
df['Total_Expenses'] = df[expense_cols].sum(axis=1)
print(f"\nTotal_Expenses range: {df['Total_Expenses'].min():.0f} to {df['Total_Expenses'].max():.0f}")

# Cell 5: Encode Categorical Variables
df_encoded = pd.get_dummies(df, columns=['Occupation', 'City_Tier'], drop_first=True)
print(f"Shape after encoding: {df_encoded.shape}")

# Cell 6: Correlation Heatmap
plt.figure(figsize=(12, 8))
numeric_cols = ['Income', 'Age', 'Dependents'] + expense_cols + ['Disposable_Income', 'Total_Expenses']
correlation_matrix = df[numeric_cols].corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, fmt='.2f')
plt.title('Correlation Heatmap: Income, Expenses, and Savings')
plt.tight_layout()
plt.show()

# Cell 7: Spending by City Tier
plt.figure(figsize=(10, 6))
city_spending = df.groupby('City_Tier')[expense_cols].mean()
city_spending.plot(kind='bar', figsize=(12, 6))
plt.title('Average Spending by Category and City Tier')
plt.xlabel('City Tier')
plt.ylabel('Average Amount')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()

# Cell 8: Disposable Income by Occupation
plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='Occupation', y='Disposable_Income')
plt.title('Disposable Income Distribution by Occupation')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Cell 9: Income vs Total Expenses
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='Income', y='Total_Expenses', hue='City_Tier', alpha=0.5)
plt.title('Income vs Total Expenses by City Tier')
plt.xlabel('Monthly Income')
plt.ylabel('Total Monthly Expenses')
plt.tight_layout()
plt.show()

# Cell 10: Savings Rate Analysis
df['Savings_Rate'] = (df['Disposable_Income'] / df['Income']) * 100
plt.figure(figsize=(10, 6))
sns.histplot(df['Savings_Rate'], bins=50, kde=True)
plt.title('Distribution of Savings Rate (%)')
plt.xlabel('Savings Rate (%)')
plt.ylabel('Count')
plt.tight_layout()
plt.show()
print("\n=== Average Savings Rate by Occupation ===")
print(df.groupby('Occupation')['Savings_Rate'].mean().sort_values(ascending=False))

# Cell 11: Prepare Data for ML
feature_cols = [col for col in df_encoded.columns if col not in
                ['Disposable_Income', 'Desired_Savings', 'Desired_Savings_Percentage',
                 'Savings_Rate', 'Total_Expenses']]
X = df_encoded[feature_cols]
y = df_encoded['Disposable_Income']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Training set: {X_train.shape}")
print(f"Test set: {X_test.shape}")

# Cell 12: Train Random Forest Model
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Cell 13: Model Evaluation
r2 = r2_score(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
print("=== Model Performance ===")
print(f"R-squared: {r2:.4f}")
print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")

Your model is trained and evaluated. Next up, you'll extract feature importance scores to discover which spending categories matter most for predicting savings potential.

Analyze Feature Importance and Summarize Insights

Your model can predict disposable income with high accuracy. But a prediction alone does not tell a stakeholder why someone saves more or less than their peers.

In this step, you will extract the features your model relies on most, visualize them in a ranked chart, and write a narrative summary that turns raw numbers into actionable financial insights.

In this step, get ready to:
  • Create a horizontal bar chart ranking the top 10 most important features.
  • Interpret the results in markdown cells with business context.
  • Summarize key insights programmatically to make the notebook portfolio-ready.
Extract feature importances and create the bar chart

A Random Forest model tracks how much each input feature reduces prediction error across all its decision trees. The feature_importances_ attribute exposes these values as an array you can rank and plot.

  • Create a new cell in your notebook (Cell 14) and add the following code to extract importances and plot them:
# Cell 14: Feature Importance
importances = model.feature_importances_
feature_importance_df = pd.DataFrame({
    'Feature': feature_cols,
    'Importance': importances
}).sort_values('Importance', ascending=False)

plt.figure(figsize=(10, 8))
top_features = feature_importance_df.head(10)
plt.barh(top_features['Feature'], top_features['Importance'])
plt.xlabel('Feature Importance')
plt.title('Top 10 Most Important Features for Predicting Disposable Income')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.show()

What does this code do?

model.feature_importances_ returns an array where each value represents how much that feature contributed to reducing prediction error across all 100 trees in the forest.

The code wraps these values into a DataFrame alongside the feature names, sorts them in descending order, and then plots only the top 10 as a horizontal bar chart. plt.gca().invert_yaxis() flips the y-axis so the most important feature appears at the top.

  • Run the cell.
  • You should see a horizontal bar chart with 10 features ranked by importance. The longest bar at the top indicates the feature your model relies on most to predict disposable income.

Chart looks empty or features are unnamed?

Make sure you ran Cell 11 (which defines feature_cols) and Cell 12 (which trains the model) before this cell. Jupyter cells must be executed in order for variables to exist in memory.

If the bars are all the same length, check that your model was fitted with model.fit(X_train, y_train) and not on the full dataset.

Why isn't my feature importance chart displaying correctly?

Interpret results in markdown cells

A chart without context is just a picture. Hiring managers reviewing your notebook want to see that you can translate technical output into business language.

  • Click the + Markdown button (or change a new cell's type to Markdown) directly below Cell 14.
  • Write 2-3 sentences explaining which features rank highest and what that means for personal finance. For example, if Rent and Loan_Repayment dominate, note that fixed costs are the strongest predictors of low savings and suggest these are where financial planning should focus.
  • Connect your interpretation to the EDA findings from Step 3. If your correlation heatmap already hinted that certain expenses were closely linked to disposable income, state that the model confirms it.
  • Run the markdown cell to render it.

Why markdown cells matter for portfolios

A notebook full of code cells shows you can write Python. A notebook with markdown narrative between code cells shows you can think like an analyst. Reviewers skim the markdown first to see if you understand the business problem, then check the code to confirm technical competence.

Create the business insights summary

Your final code cell pulls together the headline numbers from your entire analysis into a single printout. This gives readers a quick-reference summary without scrolling through the full notebook.

  • Create a new cell (Cell 15) and add the following code:
# Cell 15: Business Insights Summary
print("=== Key Insights ===")
print(f"1. Top spending predictor: {feature_importance_df.iloc[0]['Feature']}")
print(f"2. Model accuracy (R-squared): {r2:.4f}")
print(f"3. Average savings rate: {df['Savings_Rate'].mean():.1f}%")
print(f"4. Occupation with highest savings: {df.groupby('Occupation')['Savings_Rate'].mean().idxmax()}")

What does this code do?

feature_importance_df.iloc[0]['Feature'] grabs the name of the single most important feature from the sorted DataFrame you created in Cell 14.

The remaining lines pull the R-squared score from Cell 13, compute the dataset-wide average savings rate, and identify which occupation group saves the most using idxmax() (which returns the index label of the maximum value in a Series).

  • Run the cell.
  • You should see four printed lines under the heading === Key Insights ===, each showing a concrete finding from your analysis.

Seeing a NameError for feature_importance_df or r2?

These variables are defined in earlier cells (Cell 14 and Cell 13 respectively). Use Kernel > Restart & Run All to re-execute every cell in order. This ensures all variables are in memory.

Why am I getting a NameError in my Jupyter Notebook summary cell?

  • Add one final markdown cell below Cell 15 with a heading like ## Conclusion and 3-4 sentences summarizing your actionable insights in plain language.
  • Focus on what a non-technical reader would care about. For example: which expense categories should someone reduce first if they want to save more, and which demographic groups are most at risk of low savings.

Make it skimmable

Use bullet points in your markdown conclusion. A hiring manager scanning your notebook will spend about 10 seconds on the summary. Bullets let them grab the key takeaways instantly.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all 15 code cells plus your markdown interpretation cells.

ⓧ I'd like to double check the full code

Here is the complete notebook code from Cell 1 through Cell 15. Compare it against your notebook and fix any differences.

# Cell 1: Setup and Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Cell 2: Load Data
df = pd.read_csv('indian_personal_finance.csv')
print(f"Dataset shape: {df.shape}")
df.head()

# Cell 3: Data Inspection
print("=== Data Types ===")
print(df.dtypes)
print("\n=== Missing Values ===")
print(df.isnull().sum())
print("\n=== Basic Statistics ===")
df.describe()

# Cell 4: Data Cleaning
print(f"Duplicate rows: {df.duplicated().sum()}")
df = df.drop_duplicates()
df = df.dropna()
expense_cols = ['Rent', 'Loan_Repayment', 'Insurance', 'Groceries', 'Transport',
                'Eating_Out', 'Entertainment', 'Utilities', 'Healthcare',
                'Education', 'Miscellaneous']
print(f"\nNegative values in expenses: {(df[expense_cols] < 0).sum().sum()}")
df['Total_Expenses'] = df[expense_cols].sum(axis=1)
print(f"\nTotal_Expenses range: {df['Total_Expenses'].min():.0f} to {df['Total_Expenses'].max():.0f}")

# Cell 5: Encode Categorical Variables
df_encoded = pd.get_dummies(df, columns=['Occupation', 'City_Tier'], drop_first=True)
print(f"Shape after encoding: {df_encoded.shape}")

# Cell 6: Correlation Heatmap
plt.figure(figsize=(12, 8))
numeric_cols = ['Income', 'Age', 'Dependents'] + expense_cols + ['Disposable_Income', 'Total_Expenses']
correlation_matrix = df[numeric_cols].corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, fmt='.2f')
plt.title('Correlation Heatmap: Income, Expenses, and Savings')
plt.tight_layout()
plt.show()

# Cell 7: Spending by City Tier
plt.figure(figsize=(10, 6))
city_spending = df.groupby('City_Tier')[expense_cols].mean()
city_spending.plot(kind='bar', figsize=(12, 6))
plt.title('Average Spending by Category and City Tier')
plt.xlabel('City Tier')
plt.ylabel('Average Amount')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()

# Cell 8: Disposable Income by Occupation
plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='Occupation', y='Disposable_Income')
plt.title('Disposable Income Distribution by Occupation')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Cell 9: Income vs Total Expenses
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='Income', y='Total_Expenses', hue='City_Tier', alpha=0.5)
plt.title('Income vs Total Expenses by City Tier')
plt.xlabel('Monthly Income')
plt.ylabel('Total Monthly Expenses')
plt.tight_layout()
plt.show()

# Cell 10: Savings Rate Analysis
df['Savings_Rate'] = (df['Disposable_Income'] / df['Income']) * 100
plt.figure(figsize=(10, 6))
sns.histplot(df['Savings_Rate'], bins=50, kde=True)
plt.title('Distribution of Savings Rate (%)')
plt.xlabel('Savings Rate (%)')
plt.ylabel('Count')
plt.tight_layout()
plt.show()
print("\n=== Average Savings Rate by Occupation ===")
print(df.groupby('Occupation')['Savings_Rate'].mean().sort_values(ascending=False))

# Cell 11: Prepare Data for ML
feature_cols = [col for col in df_encoded.columns if col not in
                ['Disposable_Income', 'Desired_Savings', 'Desired_Savings_Percentage',
                 'Savings_Rate', 'Total_Expenses']]
X = df_encoded[feature_cols]
y = df_encoded['Disposable_Income']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Training set: {X_train.shape}")
print(f"Test set: {X_test.shape}")

# Cell 12: Train Random Forest Model
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Cell 13: Model Evaluation
r2 = r2_score(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
print("=== Model Performance ===")
print(f"R-squared: {r2:.4f}")
print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")

# Cell 14: Feature Importance
importances = model.feature_importances_
feature_importance_df = pd.DataFrame({
    'Feature': feature_cols,
    'Importance': importances
}).sort_values('Importance', ascending=False)
plt.figure(figsize=(10, 8))
top_features = feature_importance_df.head(10)
plt.barh(top_features['Feature'], top_features['Importance'])
plt.xlabel('Feature Importance')
plt.title('Top 10 Most Important Features for Predicting Disposable Income')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.show()

# Cell 15: Business Insights Summary
print("=== Key Insights ===")
print(f"1. Top spending predictor: {feature_importance_df.iloc[0]['Feature']}")
print(f"2. Model accuracy (R-squared): {r2:.4f}")
print(f"3. Average savings rate: {df['Savings_Rate'].mean():.1f}%")
print(f"4. Occupation with highest savings: {df.groupby('Occupation')['Savings_Rate'].mean().idxmax()}")

Secret mission

Anomaly Detection for Overspenders

Your Random Forest model predicts savings for the average person. But what about the outliers? Build an anomaly detection system that flags individuals who are spending way more than their demographic peers. This introduces unsupervised learning concepts and gives your notebook a real-world fraud-detection flavor.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files for future reference, or delete them to free up disk space.

Resources you used:

  • finance_savings_predictor.ipynb notebook file.
  • indian_personal_finance.csv dataset file.
  • Python libraries installed via pip (pandas, numpy, matplotlib, seaborn, scikit-learn).

Keep everything running

No action needed. Choose this if you want to revisit your analysis, tweak the model, or use the notebook as a portfolio piece.

  • Your notebook and dataset remain in the folder where you saved them.
  • The installed Python libraries stay available for other projects that use the same packages.

Pause - I'll come back to this later

Shut down your Jupyter Notebook server to free up memory. Your files stay on disk for when you return.

  • In the terminal where Jupyter is running, press Ctrl+C and confirm the shutdown.
  • Your finance_savings_predictor.ipynb and indian_personal_finance.csv files remain saved in their folder.
  • To resume later, navigate to the folder in your terminal and run jupyter notebook again.

Delete - I don't want to use this again

Remove all project files and uninstall the libraries you added. This frees up disk space completely.

  • Shut down Jupyter Notebook by pressing Ctrl+C in the terminal where it is running.
  • Delete the notebook and dataset files by running the following command from the folder where they are saved:
rm finance_savings_predictor.ipynb indian_personal_finance.csv

On Windows, use this command instead:

del finance_savings_predictor.ipynb, indian_personal_finance.csv
  • Uninstall the Python libraries you installed for this project:
pip uninstall pandas numpy matplotlib seaborn scikit-learn -y

Should I uninstall these libraries?

Only uninstall if you are sure no other projects on your machine depend on these packages. If you use pandas or numpy elsewhere, skip this step and just delete the notebook and dataset files.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete data science notebook that takes raw personal finance data and transforms it into predictive insights and actionable recommendations.

You've learned how to:

  • Clean, explore, and visualize a 20,000-row financial dataset to uncover demographic spending patterns across occupations and city tiers.
  • Train a Random Forest regression model that predicts disposable income based on an individual's profile and spending habits.
  • Extract and interpret feature importance to derive business recommendations about which spending categories most affect savings potential.
  • Secret Mission: Built an anomaly detection system using statistical z-scores to flag individuals overspending relative to their demographic peer groups.

Ready to quiz yourself?