Weekly Job Scraper with GitHub Actions

Scrape UK visa sponsorship jobs and automate weekly data collection.

Introduction

30 Second Summary

Job hunting is stressful enough without having to manually check multiple job boards every week for visa sponsorship roles. Imagine fresh results waiting for you every Monday morning, collected automatically.

In this project, you will build a Python web scraper that collects UK visa sponsorship job listings from NHS Jobs and saves them to a CSV file. You will then automate the whole process with GitHub Actions so it runs every Monday morning.

What You'll Build

A Python script that scrapes NHS Jobs for visa sponsorship roles, filters the results, and saves them to a CSV file updated weekly by a GitHub Actions workflow.

By the end of this project, you'll have:

A working Python script (scraper.py) that fetches NHS Jobs search results and filters for visa sponsorship roles.
A CSV output file with job title, employer, salary, location, closing date, and link for each sponsored role.
A GitHub Actions workflow (.github/workflows/scrape.yml) that runs the scraper on a weekly cron schedule every Monday at 7:00 AM UTC.
Secret Mission: Add the Ministry of Justice jobs site as a second source and merge results into your CSV.

Are there any prerequisites?

No prior experience with web scraping or automation is needed. Step 1 walks you through installing all required tools from scratch.

Before We Start

Before you start building your job scraper, you need a working development environment. That means having Python ready, a code editor open, and a place to store your code online.

In this step, you'll create a project folder, install the Python packages your scraper needs, and connect everything to a GitHub repository.

In this step, get ready to:

Create a project folder and open it in VS Code.
Set up a virtual environment and install Python packages.
Create a GitHub repository and connect it to your project.

Create your project folder and open it in VS Code

Your scraper is a Python project, so the first thing to confirm is that Python is installed on your machine.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
Check if Python is installed by running this command:

python3 --version

✔️ I see a version number

You should see something like Python 3.x.x. Any version 3.9 or higher works perfectly for this project.

ⓧ Command not found

If you are on Windows, try running python --version instead (without the 3). If that shows a version number, you are all set.

If neither command works, download Python from the official site:

Go to python.org/downloads.
Download the latest version for your operating system.
Run the installer. On Windows, check the box that says Add Python to PATH before clicking Install Now.
Close and reopen your terminal, then run python3 --version again to confirm.

Now create a project folder on your Desktop so you can find it easily later.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir uk-visa-scraper
cd uk-visa-scraper

Confirm the folder was created by running:

ls

The folder is empty for now, so ls shows no output. That is expected.

Next, open VS Code so you have a proper code editor for the project.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type Visual Studio Code and press Enter to open it.
In VS Code, click File in the top menu bar.
Click Open Folder.
Navigate to your Desktop, select the uk-visa-scraper folder, then click Open.

You should see UK-VISA-SCRAPER in the Explorer sidebar on the left. The folder is empty, which is exactly what you want.

Don't see the Explorer sidebar?

Click View in the top menu bar, then click Explorer to toggle it on. You can also press Cmd+Shift+E (macOS) or Ctrl+Shift+E (Windows).

help me find the Explorer sidebar in VS Code

Set up a virtual environment and install packages

Python projects use a virtual environment to keep their packages separate from other projects on your machine. This prevents version conflicts between different projects.

Open VS Code's built-in terminal by pressing Ctrl+` (backtick key, next to the number 1 on your keyboard).
Create a virtual environment by running this command:

python3 -m venv venv

What does this command do?

python3 -m venv tells Python to create a new virtual environment. The second venv is the folder name where the environment lives.

Confirm the venv folder was created by running:

ls

You should see a venv folder in the output.

Now you need to activate the virtual environment so any packages you install go into it.

Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the beginning of your terminal prompt. This means the virtual environment is active.

Don't see (venv) in your prompt?

On Windows, you might get an error about execution policies. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in PowerShell, then try the activate command again.

Make sure you ran the correct command for your operating system. The macOS and Windows commands are different.

help me activate my Python virtual environment

Your scraper needs two Python packages: requests for fetching web pages, and BeautifulSoup for reading the HTML. You will define these in a requirements.txt file, which is the standard way Python projects list their dependencies.

In VS Code's Explorer sidebar, hover over the UK-VISA-SCRAPER heading and click the New File icon (it looks like a page with a plus sign).
Name the file requirements.txt and press Enter.
Paste the following into the file:

requests==2.33.1
beautifulsoup4==4.13.4

What does this file do?

requirements.txt lists the exact packages and versions your project needs. The == pins each package to a specific version so your scraper behaves the same way every time it runs.

requests handles sending HTTP requests to websites. beautifulsoup4 parses the HTML that comes back so you can extract specific pieces of data.

Save requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
Install the packages from your requirements file by running this command in the terminal:

pip install -r requirements.txt

You should see output ending with Successfully installed followed by a list of package names. This confirms both packages are installed in your virtual environment.

Seeing an error during installation?

Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If it is missing, run the activate command for your operating system again.

Check that requirements.txt is saved and contains the two package lines. A typo in the file name or package name will cause the install to fail.

help me fix my pip install error

✔️ Awesome, I've got everything!

Make sure you have saved requirements.txt before moving on.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file. Make sure yours matches exactly.

requests==2.33.1
beautifulsoup4==4.13.4

Create a GitHub repository and connect it

Your project needs a GitHub repository so you can store your code online. Later in the project, you will also use this repo to run your scraper automatically with GitHub Actions.

First, confirm that Git is installed on your machine.

Check if Git is installed by running this command in your VS Code terminal:

git --version

✔️ I see a version number

You should see something like git version 2.x.x. Any recent version works for this project.

ⓧ Command not found

Install Git from the official site:

Go to git-scm.com/downloads.
Download and run the installer for your operating system.
Accept all the default settings during installation.
Close and reopen your VS Code terminal, then run git --version again to confirm.

Now create a new repository on GitHub to store your project.

Open github.com in your browser and sign in to your account.
Click the + icon in the top right corner, then click New repository.
Enter uk-visa-scraper as the Repository name.
Select Public for the visibility.
Leave all other options unchecked. Do not add a README, .gitignore, or license.
Click Create repository.

Why leave the repo empty?

You already have files locally (your requirements.txt and venv folder). Starting with an empty GitHub repo makes it simple to push your local files up without merge conflicts.

GitHub shows a setup page with commands you can copy. You will use these to connect your local folder to the remote repository.

Initialize a Git repository in your project folder by running this command in your VS Code terminal:

git init

Connect your local repository to the GitHub repo by running this command:

git remote add origin [[YOUR_REPO_URL="https://github.com/your-username/uk-visa-scraper.git"]]

Where do I find my repo URL?

On the GitHub setup page that appeared after you created the repo, look for the HTTPS URL near the top. It looks like https://github.com/your-username/uk-visa-scraper.git. Copy that URL and use it in the command above.

Stage and commit your files, then push them to GitHub by running these commands:

git add requirements.txt
git commit -m "Initial commit"
git branch -M main
git push -u origin main

What do these commands do?

git add requirements.txt stages the file, marking it to be included in the next commit.
git commit saves a snapshot of your staged files with a message describing the change.
git branch -M main renames the default branch to main so it matches GitHub's default.
git push uploads your local commits to the remote GitHub repository.

Only push requirements.txt

Notice you are only adding requirements.txt to Git, not the entire venv folder. Virtual environments are large and machine-specific, so they should never be pushed to GitHub.

GitHub may prompt you to sign in. If a browser window opens asking you to authorize Git, follow the prompts and allow access.

Go back to your repository page on GitHub and refresh the page.

You should see requirements.txt listed in your repository with the commit message Initial commit. Your local project and GitHub repo are now connected.

Push failed or asked for a password?

If you see an authentication error, make sure you are signed in to GitHub in your browser. Git should open a browser window for authentication. If it asks for a username and password in the terminal instead, you need to set up a personal access token or use the GitHub CLI.

Double check that your remote URL is correct by running git remote -v. It should show your GitHub repo URL.

help me push my code to GitHub

Your development environment is ready to go. Next up, you will write the Python script that fetches real job listings from the NHS Jobs website.

Set Up Your Project

Before writing any scraping code, you need a clean project workspace. A dedicated folder, the right Python packages, and a GitHub repository will keep everything organized from day one.

In this step, you'll create your project folder, install your Python dependencies inside a virtual environment, and push your first commit to GitHub.

In this step, get ready to:

Create your project folder and open it in VS Code.
Set up a virtual environment and install the Python packages you need.
Connect your project to a GitHub repository.

Create your project folder

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir uk-visa-scraper
cd uk-visa-scraper

Confirm the folder was created by running:

pwd

You should see a path ending in uk-visa-scraper, like /Users/yourname/Desktop/uk-visa-scraper.

Now open VS Code so you can work inside this folder.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type VS Code and press Enter to open it.
Click File in the top menu bar.
Click Open Folder.
Navigate to your Desktop and select the uk-visa-scraper folder.
Click Open.

You should see uk-visa-scraper in the VS Code title bar and an empty Explorer sidebar on the left.

VS Code asking you to trust the folder?

If VS Code shows a "Do you trust the authors of the files in this folder?" dialog, click Yes, I trust the authors. This is normal for any new folder you open.

help me open my project folder in VS Code

Set up a virtual environment and install packages

A virtual environment is an isolated space where Python packages live separately from your system-wide Python installation. This prevents version conflicts between projects.

Open VS Code's built-in terminal by pressing Ctrl+` (backtick key, usually above Tab).

Use VS Code's terminal from now on

This integrated terminal opens directly inside your project folder. You can close the system terminal you used earlier.

Create and activate a virtual environment by running these commands:

macOS

python3 -m venv venv
source venv/bin/activate

Windows

python -m venv venv
venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms your virtual environment is active.

What is a virtual environment?

Think of it as a sealed box for your project's packages. When you install something inside the virtual environment, it only exists there.

The venv folder that just appeared in your Explorer sidebar holds this isolated Python copy. You never need to edit anything inside it.

Don't see (venv) in your prompt?

Make sure you ran the source (macOS) or activate (Windows) command as a separate step after creating the environment.

On Windows, you may need to allow PowerShell to run scripts first. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned and then try activating again.

help me activate my Python virtual environment

Now you'll create a requirements.txt file to pin your project's dependencies and then install them.

In VS Code's Explorer sidebar, click the New File icon at the top.
Type requirements.txt and press Enter.
Paste the following content into the new file:

requests==2.33.1
beautifulsoup4==4.13.4

Save requirements.txt by pressing Cmd+S (macOS) or Ctrl+S (Windows).
Install the packages by running this command in the terminal:

pip install -r requirements.txt

What does this do?

requirements.txt lists every Python package your project needs, pinned to an exact version. This makes sure anyone (or any automated workflow) can recreate your environment.

requests (2.33.1) lets your script send HTTP requests to websites.
beautifulsoup4 (4.13.4) parses raw HTML so you can search through it like a structured document.

pip install -r requirements.txt reads the file and installs everything listed.

Your terminal should show Successfully installed beautifulsoup4-4.13.4 requests-2.33.1 (along with some sub-dependencies).

Seeing an error during install?

Make sure your virtual environment is active. You should see (venv) in your terminal prompt. If not, re-run the activation command from the platform tab above.

If you see a pip: command not found error, try pip3 instead of pip.

help me install my Python packages

✔️ Awesome, I've got everything!

Great. Double check you've saved requirements.txt before moving on.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file. Make sure yours matches exactly.

requests==2.33.1
beautifulsoup4==4.13.4

Create a GitHub repository and connect it

Your scraper will eventually run automatically using GitHub Actions, so your code needs to live in a GitHub repository.

Open your browser and go to github.com.
Click the + icon in the top right corner.
Click New repository.
Enter uk-visa-scraper in the Repository name field.
Leave all other settings as their defaults. Do not check "Add a README file."
Click Create repository.

Why skip the README?

You already have files locally (requirements.txt and the venv folder). Initializing GitHub with a README creates a commit on the remote that conflicts with your local files. Starting with an empty repo makes the first push clean.

GitHub now shows a "Quick setup" page with your repository URL. Keep this page open.

Before you push, you need a .gitignore file to keep the bulky venv folder out of your repository.

In VS Code's Explorer sidebar, click the New File icon.
Type .gitignore and press Enter.
Paste this line into the file:

venv/

Save .gitignore.

What is .gitignore?

Any file or folder listed in .gitignore will be completely invisible to Git. The venv/ folder contains thousands of installed package files that anyone can recreate from requirements.txt.

Copy your repository URL from the GitHub "Quick setup" page. It looks like https://github.com/your-username/uk-visa-scraper.git.
Initialize Git and push your code by running these commands in the VS Code terminal:

git init
git add .
git commit -m "Initial commit"
git branch -M main
git remote add origin [[YOUR_REPO_URL="https://github.com/your-username/uk-visa-scraper.git"]]
git push -u origin main

What do these commands do?

git init turns your folder into a Git repository.
git add . stages all your files (except what's in .gitignore) for a commit.
git commit saves a snapshot of your staged files.
git remote add origin links your local repo to the GitHub repo you just created.
git push uploads your code to GitHub.

GitHub may ask you to sign in

When you run git push, a browser window or dialog may pop up asking you to authenticate with GitHub. Sign in with your GitHub credentials and authorize access.

Go back to your GitHub repository page in the browser and refresh.

You should see requirements.txt and .gitignore listed in your repository. The venv folder should not appear.

Push failed or seeing an error?

If you see fatal: remote origin already exists, run git remote set-url origin YOUR_URL with your actual repo URL instead of the git remote add command.

If you see Permission denied, make sure you signed in to the correct GitHub account during the authentication step.

help me push my code to GitHub

Your workspace is ready. You have a clean project folder, isolated dependencies, and version control set up. Next up, you'll write the Python code that actually fetches job listings from NHS Jobs.

Fetch NHS Jobs Search Results

Your project environment is set up and your dependencies are installed. Time to write the code that actually does something.

NHS Jobs is the largest public jobs board in the UK, listing thousands of roles that offer visa sponsorship. In this step, you'll write a Python script that reaches out to the NHS Jobs search page and pulls back the raw HTML response.

In this step, get ready to:

Create scraper.py and configure the NHS Jobs search URL and parameters.
Fetch live search results from NHS Jobs and verify the response in your terminal.

Create your scraper script

Before your scraper can fetch anything, it needs to know where to look and what to search for. The NHS Jobs site accepts keyword searches as URL parameters, just like when you type a search into your browser.

Why not use an API instead?

NHS Jobs does not offer a public API for searching job listings. The only way to get this data programmatically is to fetch the same HTML pages a browser would load, then extract the information you need.

This is exactly what web scraping is: send an HTTP request, get back HTML, and parse it for data.

Create a new file called scraper.py in your project folder, then paste the following code:

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime, timezone

def scrape_nhs_jobs():
    """Scrape NHS Jobs for visa sponsorship listings."""
    # The NHS Jobs search results URL
    base_url = "https://www.jobs.nhs.uk/candidate/search/results"
    # Search parameters that mimic a keyword search for "visa sponsorship"
    params = {
        "keyword": "visa sponsorship",
        "language": "en",
        "searchFormType": "main",
        "skipPhraseSuggester": "true",
    }
    # Identify our scraper with a custom User-Agent header
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; UKJobScraper/1.0)"
    }

What does this code do?

The four import lines load the libraries your scraper needs. requests handles HTTP requests. BeautifulSoup, csv, and datetime will be used in later steps.
base_url points to the NHS Jobs search results page.
params is a dictionary of search parameters. requests appends these to the URL automatically, so you don't have to build the query string by hand.
headers includes a User-Agent string that identifies your scraper to the NHS Jobs server.

Save scraper.py.

You won't see any output yet. The next chunk adds the actual HTTP request that fetches the page.

Fetch the search results page

With the URL and parameters defined, you can now send the request and see what comes back. Three lines of code are all it takes to hit the NHS Jobs server and print the response.

In scraper.py, add the following code below the headers dictionary, still inside the scrape_nhs_jobs() function:

    # Send the GET request to NHS Jobs
    response = requests.get(base_url, params=params, headers=headers, timeout=30)
    # Stop the script if the server returns an error
    response.raise_for_status()
    # Print the raw HTML so we can inspect the page structure
    print(response.text)

At the very bottom of scraper.py, outside the function (no indentation), add the entry point that runs your scraper:

if __name__ == "__main__":
    scrape_nhs_jobs()

What does this code do?

requests.get() sends an HTTP GET request to the NHS Jobs URL with your search parameters. The timeout=30 argument stops the script from hanging if the server doesn't respond within 30 seconds.
raise_for_status() checks if the server returned an error code (like 404 or 500). If it did, the script stops immediately instead of continuing with bad data.
print(response.text) dumps the raw HTML to your terminal so you can see what the server sent back.
The if __name__ == "__main__" block makes sure the scraper only runs when you execute the file directly, not when it's imported by another script.

Save scraper.py.
Make sure your virtual environment is still active (you should see (venv) at the start of your terminal prompt).
Run the script by entering this command in your terminal:

python scraper.py

You should see a wall of HTML text filling your terminal. This is the raw source code of the NHS Jobs search results page.

Seeing an error instead of HTML?

ModuleNotFoundError: No module named 'requests' means your virtual environment isn't active. Run source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows), then try again.
ConnectionError or Timeout means your internet connection dropped or the NHS Jobs site is temporarily down. Check your connection and retry.
HTTPError: 403 means the server blocked your request. Double check that your headers dictionary matches the code above exactly.

Help me debug my scraper script

✔️ Awesome, I've got everything!

Double check you've saved scraper.py and you're ready to move on.

ⓧ I'd like to double check the full code

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime, timezone

def scrape_nhs_jobs():
    """Scrape NHS Jobs for visa sponsorship listings."""
    base_url = "https://www.jobs.nhs.uk/candidate/search/results"
    params = {
        "keyword": "visa sponsorship",
        "language": "en",
        "searchFormType": "main",
        "skipPhraseSuggester": "true",
    }
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; UKJobScraper/1.0)"
    }

    response = requests.get(base_url, params=params, headers=headers, timeout=30)
    response.raise_for_status()
    print(response.text)

if __name__ == "__main__":
    scrape_nhs_jobs()

You've confirmed that your scraper can reach NHS Jobs and pull back live HTML. Next, you'll parse that HTML to extract the actual job listings.

Parse Job Listings from the HTML

In the previous step, you confirmed that your scraper can fetch raw HTML from NHS Jobs. The problem is that raw HTML is a massive wall of text with tags, attributes, and layout code mixed in with the actual job data.

In this step, you will use BeautifulSoup to parse that HTML into a structured tree you can search. You will walk through each job listing, pull out the fields you care about, and filter out the ones that explicitly say "no sponsorship."

In this step, get ready to:

Parse the raw HTML response with BeautifulSoup and select individual job cards.
Extract each job's title, employer, salary, location, closing date, and link.
Filter out listings that say "no sponsorship" and print the clean results.

Set up pagination and parse job cards

Right now, your scraper fetches a single page and dumps all the HTML to the terminal. NHS Jobs returns 10 results per page, so you need a loop that keeps requesting the next page until there are no more results.

BeautifulSoup turns that raw HTML into a searchable tree. You give it a CSS selector and it hands back every matching element.

In scraper.py, find these three lines inside scrape_nhs_jobs() (right after the headers dictionary):

    response = requests.get(base_url, params=params, headers=headers, timeout=30)
    response.raise_for_status()
    print(response.text)

Delete those three lines and replace them with the following code:

    # Start with an empty list and track the current page
    jobs = []
    page = 1

    while True:
        params["page"] = page
        print(f"Fetching NHS Jobs page {page}...")
        response = requests.get(base_url, params=params, headers=headers, timeout=30)
        response.raise_for_status()

        # Parse the HTML and find all job cards on this page
        soup = BeautifulSoup(response.text, "html.parser")
        job_cards = soup.select("[data-test='search-result']")

        # Stop if this page has no results
        if not job_cards:
            print(f"No more results found on page {page}. Stopping.")
            break

What does this code do?

jobs = [] creates an empty list where you will collect each job's data. page = 1 tracks which results page you are fetching.

The while True loop requests one page at a time. BeautifulSoup(response.text, "html.parser") converts the raw HTML string into a tree you can search.

soup.select("[data-test='search-result']") finds every element with that data-test attribute. Each one is a single job card. When the list comes back empty, the loop stops.

Do not save yet. The while loop still needs the extraction logic and a closing section. You will add those next.

Extract and filter each job listing

Each job card is a chunk of HTML containing nested tags. The job title lives in an h2 tag, the employer is in an h3 tag, and salary and closing date details appear as li list items inside the card.

Directly below the if not job_cards: block you just added (still inside the while True loop), paste the following code:

        for card in job_cards:
            # Extract the job title from the h2 tag
            title_tag = card.select_one("h2")
            title = title_tag.get_text(strip=True) if title_tag else "N/A"

            # Skip listings that explicitly say "no sponsorship"
            if "no sponsorship" in title.lower():
                continue

            # Build the full job URL from the first link in the card
            link_tag = card.select_one("a")
            link = ""
            if link_tag and link_tag.get("href"):
                href = link_tag["href"]
                if href.startswith("/"):
                    link = f"https://www.jobs.nhs.uk{href}"
                else:
                    link = href

            # Extract employer name from the h3 tag
            employer_tag = card.select_one("h3")
            employer_text = employer_tag.get_text(strip=True) if employer_tag else "N/A"

            # Set defaults for fields that may not appear on every card
            location = "N/A"
            salary = "N/A"
            closing_date = "N/A"

            # Scan list items for salary (starts with a pound sign) and closing date
            description_items = card.select("li")
            for item in description_items:
                text = item.get_text(strip=True)
                if text.startswith("\u00a3"):
                    salary = text
                elif "closing" in text.lower():
                    closing_date = text

            # Try to split location from the employer text
            if employer_text != "N/A" and location == "N/A":
                parts = employer_text.rsplit(" ", 1)
                if len(parts) == 2 and len(parts[1]) >= 5:
                    employer = parts[0]
                    location = parts[1]
                else:
                    employer = employer_text
            else:
                employer = employer_text

            # Add the cleaned job data to our list
            jobs.append({
                "Title": title,
                "Employer": employer,
                "Location": location,
                "Salary": salary,
                "Closing Date": closing_date,
                "Link": link,
                "Source": "NHS Jobs",
            })

        # Move to the next page
        page += 1

        # Safety cap to avoid hammering the server
        if page > 10:
            print("Reached page limit. Stopping.")
            break

    return jobs

What does this code do?

The for card in job_cards loop walks through every job card on the current page. For each card, it extracts the title from the h2 tag and immediately checks for "no sponsorship" in the title. If it finds that phrase, continue skips the listing entirely.

The link extraction prepends https://www.jobs.nhs.uk to relative URLs (those starting with /) so every link is a full, clickable URL.

The salary and closing date are found by scanning the li elements inside each card. The location is split from the employer text using rsplit, because NHS Jobs often combines the employer name and postcode into one string.

After the for loop finishes for one page, page += 1 moves to the next page. The cap at page 10 prevents the scraper from making hundreds of requests in one go.

Why filter by title instead of the full description?

Some NHS Jobs listings include "NO SPONSORSHIP" right in the job title to warn applicants up front. Checking the title is a quick first pass that catches the most obvious exclusions without needing to visit each job's detail page.

Do not save yet. The if __name__ block at the bottom of your file still needs to be updated so you can see the results.

Print the filtered results to the terminal

Your scrape_nhs_jobs() function now returns a list of job dictionaries. You need to update the bottom of the file to call the function, capture the returned list, and print each job.

Find the existing if __name__ block at the bottom of scraper.py:

if __name__ == "__main__":
    scrape_nhs_jobs()

Replace it with the following code:

if __name__ == "__main__":
    print("Starting UK Visa Sponsorship Job Scraper...")
    print("=" * 50)

    # Run the scraper and capture the results
    nhs_jobs = scrape_nhs_jobs()
    print(f"Found {len(nhs_jobs)} NHS sponsorship jobs")

    # Print each job dictionary so we can inspect the data
    for job in nhs_jobs:
        print(job)

What does this code do?

The updated main block stores the returned list in nhs_jobs and prints the total count. The for job in nhs_jobs: print(job) loop dumps each dictionary to the terminal so you can spot-check the extracted fields.

Save scraper.py.
Run the scraper by entering this command in your terminal:

python scraper.py

You should see output that looks something like this:

Starting UK Visa Sponsorship Job Scraper...
==================================================
Fetching NHS Jobs page 1...
Fetching NHS Jobs page 2...
...
Found 85 NHS sponsorship jobs
{'Title': 'Salaried GP', 'Employer': 'The Cottons Medical Centre', 'Location': '...', 'Salary': '...', 'Closing Date': '...', 'Link': 'https://www.jobs.nhs.uk/...', 'Source': 'NHS Jobs'}

What should I see?

The scraper prints a "Fetching NHS Jobs page X..." line for each page it processes. The exact number of jobs will vary depending on what is currently listed on NHS Jobs.

You should see multiple dictionaries printed, each with a Title, Employer, Link, and the other fields. Listings that contained "no sponsorship" in the title should be absent from the output.

Seeing zero jobs or an error?

If you see Found 0 NHS sponsorship jobs, the CSS selector may not be matching any elements on the page. Websites occasionally change their HTML structure. Double-check by opening the NHS Jobs search page in your browser, right-clicking a job card, and choosing Inspect to see if the data-test attribute is still present.
If you see a ConnectionError or Timeout, check your internet connection and try running the script again.
If you see a SyntaxError, make sure your indentation is consistent. Every line inside the while True loop should be indented 8 spaces (two levels). Lines inside the for card loop should be indented 12 spaces (three levels).
Still stuck? Help me debug why my NHS Jobs scraper returns zero results or throws an error.

✔️ Awesome, I've got everything!

Double check you have saved scraper.py before moving on.

ⓧ I'd like to double check the full code

Here is the complete scraper.py file as it should look at the end of this step.

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime, timezone

def scrape_nhs_jobs():
    """Scrape NHS Jobs for visa sponsorship listings."""
    base_url = "https://www.jobs.nhs.uk/candidate/search/results"
    params = {
        "keyword": "visa sponsorship",
        "language": "en",
        "searchFormType": "main",
        "skipPhraseSuggester": "true",
    }
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; UKJobScraper/1.0)"
    }

    jobs = []
    page = 1

    while True:
        params["page"] = page
        print(f"Fetching NHS Jobs page {page}...")
        response = requests.get(base_url, params=params, headers=headers, timeout=30)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        job_cards = soup.select("[data-test='search-result']")

        if not job_cards:
            print(f"No more results found on page {page}. Stopping.")
            break

        for card in job_cards:
            title_tag = card.select_one("h2")
            title = title_tag.get_text(strip=True) if title_tag else "N/A"

            if "no sponsorship" in title.lower():
                continue

            link_tag = card.select_one("a")
            link = ""
            if link_tag and link_tag.get("href"):
                href = link_tag["href"]
                if href.startswith("/"):
                    link = f"https://www.jobs.nhs.uk{href}"
                else:
                    link = href

            employer_tag = card.select_one("h3")
            employer_text = employer_tag.get_text(strip=True) if employer_tag else "N/A"

            location = "N/A"
            salary = "N/A"
            closing_date = "N/A"

            description_items = card.select("li")
            for item in description_items:
                text = item.get_text(strip=True)
                if text.startswith("\u00a3"):
                    salary = text
                elif "closing" in text.lower():
                    closing_date = text

            if employer_text != "N/A" and location == "N/A":
                parts = employer_text.rsplit(" ", 1)
                if len(parts) == 2 and len(parts[1]) >= 5:
                    employer = parts[0]
                    location = parts[1]
                else:
                    employer = employer_text
            else:
                employer = employer_text

            jobs.append({
                "Title": title,
                "Employer": employer,
                "Location": location,
                "Salary": salary,
                "Closing Date": closing_date,
                "Link": link,
                "Source": "NHS Jobs",
            })

        page += 1

        if page > 10:
            print("Reached page limit. Stopping.")
            break

    return jobs

if __name__ == "__main__":
    print("Starting UK Visa Sponsorship Job Scraper...")
    print("=" * 50)
    nhs_jobs = scrape_nhs_jobs()
    print(f"Found {len(nhs_jobs)} NHS sponsorship jobs")
    for job in nhs_jobs:
        print(job)

Your scraper now pulls structured job data from multiple pages and filters out irrelevant listings. Next up, you will save all of those results into a clean CSV file so you can open and review them in a spreadsheet.

Save Results to CSV

Your scraper can now fetch and filter NHS Jobs listings. Printing results to the terminal is great for testing, but you need a proper file you can open, share, and track over time.

In this step, you will write a function that saves your scraped jobs to a CSV file. Each row gets a timestamp so you know exactly when the data was collected.

In this step, get ready to:

Write a save_to_csv() function that writes job data to a CSV file with a timestamp.
Update the main block to call your new function instead of printing results.
Run the full scraper end-to-end and verify the CSV output.

Create the save_to_csv function

Python's built-in csv module can write dictionaries directly to rows in a CSV file. Each job dictionary your scraper returns maps perfectly to a row.

In scraper.py, add the following function directly below the scrape_nhs_jobs() function (after its return jobs line) and before the if __name__ block:


def save_to_csv(jobs, filename="sponsorship_jobs.csv"):
    """Save job listings to a CSV file."""
    # Generate a UTC timestamp for when this scrape ran
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
    fieldnames = ["Title", "Employer", "Location", "Salary", "Closing Date", "Link", "Source", "Scraped At"]

    # Open the CSV file and write each job as a row
    with open(filename, "w", newline="", encoding="utf-8") as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()
        for job in jobs:
            job["Scraped At"] = timestamp
            writer.writerow(job)

    print(f"Saved {len(jobs)} jobs to {filename}")

What does this code do?

The function takes your list of job dictionaries and an optional filename (defaulting to sponsorship_jobs.csv).

datetime.now(timezone.utc) captures the current time in UTC, so your timestamps are consistent no matter where the script runs.
csv.DictWriter maps each dictionary key to the matching CSV column. You define the column order once in fieldnames and every row follows it automatically.
The Scraped At column is added to each row inside the loop, stamping every job with the same scrape time.

Save scraper.py.

You will not see a change yet. The next substep wires this function into the main block so it actually runs.

Update the main block to save CSV instead of printing

The current main block prints every job dictionary to the terminal. Now that you have save_to_csv(), you can replace that print loop with a single function call.

Scroll to the if __name__ == "__main__": block at the bottom of scraper.py.
Find the existing block, which currently looks like this:

if __name__ == "__main__":
    print("Starting UK Visa Sponsorship Job Scraper...")
    print("=" * 50)
    nhs_jobs = scrape_nhs_jobs()
    print(f"Found {len(nhs_jobs)} NHS sponsorship jobs")
    for job in nhs_jobs:
        print(job)

Replace the entire if __name__ block with this updated version:

if __name__ == "__main__":
    print("Starting UK Visa Sponsorship Job Scraper...")
    print("=" * 50)

    # Run the NHS Jobs scraper
    nhs_jobs = scrape_nhs_jobs()
    print(f"Found {len(nhs_jobs)} NHS sponsorship jobs")

    # Collect all jobs into one list
    all_jobs = nhs_jobs

    # Save everything to CSV
    save_to_csv(all_jobs)
    print("Done!")

Why the all_jobs variable?

Right now all_jobs just holds the NHS results. This variable is a placeholder for combining multiple sources later. If you add another scraper, you can merge both lists into all_jobs before saving.

Save scraper.py.

Run the scraper and verify the CSV

Time to run the full pipeline end-to-end. The scraper will fetch pages, filter jobs, and write everything to a CSV file in your project folder.

Make sure your virtual environment is still active in the terminal. You should see (venv) at the start of your terminal prompt.
Run the scraper by entering this command:

python scraper.py

What should I see?

The terminal prints page-by-page progress as it fetches results from NHS Jobs. After a minute or so, you should see a summary line like:

Found 87 NHS sponsorship jobs
Saved 87 jobs to sponsorship_jobs.csv
Done!

Your exact number will differ depending on which listings are live when you run it.

Confirm the CSV file was created by running:

ls

You should see sponsorship_jobs.csv listed alongside scraper.py and requirements.txt.

Open sponsorship_jobs.csv in VS Code by clicking on it in the Explorer sidebar.
Verify that the first row contains your column headers: Title, Employer, Location, Salary, Closing Date, Link, Source, Scraped At.
Check that the Scraped At column has a UTC timestamp on every row.

CSV file is empty or missing columns?

If the file only has the header row and no data, check the terminal output. If it says Found 0 NHS sponsorship jobs, the NHS Jobs site may be temporarily down. Wait a few minutes and try again.
If you see a NameError: name 'save_to_csv' is not defined, make sure the save_to_csv function is defined above the if __name__ block, not inside it.
If the Scraped At column is missing, check that "Scraped At" appears in your fieldnames list.

Still stuck? Help me debug why my CSV file is empty or has missing columns.

Before you move on, double check that your full scraper.py matches the code below.

✔️ Awesome, I've got everything!

Great. Double check you have saved scraper.py before moving on.

ⓧ I'd like to double check the full code

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime, timezone

def scrape_nhs_jobs():
    """Scrape NHS Jobs for visa sponsorship listings."""
    base_url = "https://www.jobs.nhs.uk/candidate/search/results"
    params = {
        "keyword": "visa sponsorship",
        "language": "en",
        "searchFormType": "main",
        "skipPhraseSuggester": "true",
    }
    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; UKJobScraper/1.0)"
    }

    jobs = []
    page = 1

    while True:
        params["page"] = page
        print(f"Fetching NHS Jobs page {page}...")
        response = requests.get(base_url, params=params, headers=headers, timeout=30)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        job_cards = soup.select("[data-test='search-result']")

        if not job_cards:
            print(f"No more results found on page {page}. Stopping.")
            break

        for card in job_cards:
            title_tag = card.select_one("h2")
            title = title_tag.get_text(strip=True) if title_tag else "N/A"

            if "no sponsorship" in title.lower():
                continue

            link_tag = card.select_one("a")
            link = ""
            if link_tag and link_tag.get("href"):
                href = link_tag["href"]
                if href.startswith("/"):
                    link = f"https://www.jobs.nhs.uk{href}"
                else:
                    link = href

            employer_tag = card.select_one("h3")
            employer_text = employer_tag.get_text(strip=True) if employer_tag else "N/A"

            location = "N/A"
            salary = "N/A"
            closing_date = "N/A"

            description_items = card.select("li")
            for item in description_items:
                text = item.get_text(strip=True)
                if text.startswith("\u00a3"):
                    salary = text
                elif "closing" in text.lower():
                    closing_date = text

            if employer_text != "N/A" and location == "N/A":
                parts = employer_text.rsplit(" ", 1)
                if len(parts) == 2 and len(parts[1]) >= 5:
                    employer = parts[0]
                    location = parts[1]
                else:
                    employer = employer_text
            else:
                employer = employer_text

            jobs.append({
                "Title": title,
                "Employer": employer,
                "Location": location,
                "Salary": salary,
                "Closing Date": closing_date,
                "Link": link,
                "Source": "NHS Jobs",
            })

        page += 1

        if page > 10:
            print("Reached page limit. Stopping.")
            break

    return jobs


def save_to_csv(jobs, filename="sponsorship_jobs.csv"):
    """Save job listings to a CSV file."""
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
    fieldnames = ["Title", "Employer", "Location", "Salary", "Closing Date", "Link", "Source", "Scraped At"]

    with open(filename, "w", newline="", encoding="utf-8") as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()
        for job in jobs:
            job["Scraped At"] = timestamp
            writer.writerow(job)

    print(f"Saved {len(jobs)} jobs to {filename}")


if __name__ == "__main__":
    print("Starting UK Visa Sponsorship Job Scraper...")
    print("=" * 50)

    nhs_jobs = scrape_nhs_jobs()
    print(f"Found {len(nhs_jobs)} NHS sponsorship jobs")

    all_jobs = nhs_jobs

    save_to_csv(all_jobs)
    print("Done!")

Your scraper now produces a clean CSV file you can open in any spreadsheet app. Next up, you will automate this entire process with GitHub Actions so fresh job data appears in your repo every Monday morning.

Automate with GitHub Actions

Your scraper works on your machine, but right now you have to remember to run it yourself every single week. That defeats the whole point of building an automated pipeline.

GitHub Actions can run scripts on a schedule directly from your repository. In this step, you'll create a workflow file that runs your scraper every Monday morning and commits the fresh CSV back to your repo automatically.

In this step, get ready to:

Create a GitHub Actions workflow file with a weekly cron schedule and a manual trigger.
Configure the workflow to install dependencies, run the scraper, and commit results.
Push your code to GitHub, trigger the workflow, and verify the CSV appears in your repo.

Create the workflow file with triggers

GitHub Actions workflows live in a special folder inside your repository. Each workflow is a YAML file inside .github/workflows/.

Create the .github/workflows/ directory inside your project folder by running this command:

mkdir -p .github/workflows

On Windows PowerShell?

If mkdir -p doesn't work, run mkdir .github\workflows instead. PowerShell creates parent directories automatically without the -p flag.

Confirm the directory was created by running:

ls .github/workflows

You should see an empty directory with no errors. The folder is ready for your workflow file.

Create a new file called scrape.yml inside .github/workflows/ and add the following trigger configuration:

name: Scrape UK Sponsorship Jobs

on:
  schedule:
    - cron: "0 7 * * 1"
  workflow_dispatch:

permissions:
  contents: write

What does this configuration do?

name gives your workflow a human-readable label that appears in the GitHub Actions tab.
schedule with cron: "0 7 * * 1" runs the workflow every Monday at 7:00 AM UTC. The five fields represent minute, hour, day of month, month, and day of week.
workflow_dispatch adds a manual "Run workflow" button in the GitHub Actions tab so you can trigger it yourself any time.
permissions: contents: write gives the workflow permission to push commits back to your repository.

Save scrape.yml.

What is cron?

Cron is a scheduling syntax used across Linux, macOS, and many cloud services. Each cron expression has five fields that define when a task runs. GitHub Actions uses cron to trigger workflows on a repeating schedule.

Configure the workflow job steps

The triggers tell GitHub when to run your workflow. Now you need to tell it what to run.

In scrape.yml, add the following code below the permissions: write line:

jobs:
  scrape:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v6

      - name: Set up Python
        uses: actions/setup-python@v6
        with:
          python-version: "3.13"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run scraper
        run: python scraper.py

      - name: Commit and push results
        run: |
          git config --local user.email "github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add sponsorship_jobs.csv
          git commit -m "Update sponsorship jobs data [skip ci]" || exit 0
          git push

What does each step do?

Checkout repository clones your repo onto the GitHub Actions runner so the workflow has access to your files.
Set up Python installs Python 3.13 on the runner. The runner is a fresh virtual machine, so nothing is pre-installed.
Install dependencies reads your requirements.txt and installs requests and beautifulsoup4.
Run scraper executes your scraper.py script, which fetches NHS Jobs and writes sponsorship_jobs.csv.
Commit and push results stages the CSV, commits it with a bot identity, and pushes it to your repo. The || exit 0 prevents the workflow from failing if there are no changes to commit.

Save scrape.yml.

YAML indentation errors?

YAML is very sensitive to indentation. Every level must use exactly 2 spaces. Tabs will break the file. Make sure each - name: line is indented 6 spaces from the left edge (under steps:).

Help me fix YAML indentation in my GitHub Actions workflow

✔️ Awesome, I've got everything!

Double check you've saved scrape.yml before moving on.

ⓧ I'd like to double check the full code

Here is the complete scrape.yml file. Make sure yours matches exactly, including indentation.

name: Scrape UK Sponsorship Jobs

on:
  schedule:
    - cron: "0 7 * * 1"
  workflow_dispatch:

permissions:
  contents: write

jobs:
  scrape:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v6

      - name: Set up Python
        uses: actions/setup-python@v6
        with:
          python-version: "3.13"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run scraper
        run: python scraper.py

      - name: Commit and push results
        run: |
          git config --local user.email "github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add sponsorship_jobs.csv
          git commit -m "Update sponsorship jobs data [skip ci]" || exit 0
          git push

Push to GitHub and trigger the workflow

Your workflow file is ready locally. Now you need to push everything to GitHub so the Actions runner can find it.

Stage and push all your project files by running these commands:

git add .
git commit -m "Add scraper and GitHub Actions workflow"
git push

What does this push include?

This pushes your scraper.py, requirements.txt, sponsorship_jobs.csv, and the .github/workflows/scrape.yml workflow file to your GitHub repository. GitHub automatically detects the workflow file and registers it in the Actions tab.

You should see output ending with a branch reference like main -> main confirming the push succeeded.

Push rejected or authentication error?

If you see a permission or authentication error, make sure your GitHub remote URL is correct and your credentials are set up. Run git remote -v to check your remote URL.

Help me fix a git push authentication error

Now you can trigger the workflow manually to test it without waiting until Monday.

Open your repository on GitHub in your browser.
Click the Actions tab near the top of the page.
In the left sidebar, click Scrape UK Sponsorship Jobs.
Click the Run workflow dropdown button on the right side of the page.
Click the green Run workflow button to start the run.

A new workflow run appears in the list. Click on it to watch the progress. Each step (checkout, Python setup, install, scrape, commit) runs in sequence.

How long does the workflow take?

The workflow typically takes 1 to 3 minutes. Most of the time is spent installing Python and your pip packages on the fresh runner. The scraping itself is fast.

Wait for the workflow run to show a green checkmark.
Click the Code tab to go back to your repository's file list.
You should see sponsorship_jobs.csv in the file list with a recent commit message of Update sponsorship jobs data [skip ci].

Click on sponsorship_jobs.csv to open it. You should see rows of job listings with titles, employers, salaries, and links scraped fresh by the GitHub Actions runner.

Workflow failed with a red X?

Click on the failed run, then click the scrape job to see the logs. The step that failed is highlighted in red.

If Install dependencies failed, check that requirements.txt is pushed to your repo and the package names are spelled correctly.
If Run scraper failed, check the error message in the logs. A common cause is an indentation issue in scraper.py.
If Commit and push results failed, make sure your workflow has permissions: contents: write set in scrape.yml.

Help me debug my failed GitHub Actions workflow

Why is this free?

GitHub Actions provides unlimited free minutes for public repositories. Your weekly scraper only runs for a couple of minutes each time, so you won't hit any usage limits.

Your scraper now runs every Monday at 7:00 AM UTC without any input from you. Fresh visa sponsorship job data will appear in your GitHub repo week after week.

Secret mission

Add Ministry of Justice as a Second Source

Your scraper pulls from one source right now. Real-world scrapers often combine multiple sites into a single dataset. In this Secret Mission, you will add the Ministry of Justice jobs site as a second data source and merge the results into your CSV.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs locally and uses GitHub Actions on a public repository, so there are no ongoing costs.

Resources you used:

Local project folder with scraper.py, requirements.txt, and sponsorship_jobs.csv.
Python virtual environment (venv) with requests and beautifulsoup4 installed.
GitHub repository with a GitHub Actions workflow (.github/workflows/scrape.yml).

Keep everything running

No action needed. Choose this if you want the scraper to keep running every Monday and collecting fresh job data in your repo.

Your GitHub Actions workflow will continue to run on its weekly schedule.
The sponsorship_jobs.csv file in your repo will update automatically each Monday.
GitHub Actions is free for public repositories, so this costs nothing to maintain.

Pause - I'll come back to this later

Disable the scheduled workflow so it stops running, but keep all your code and data in place for when you return.

Go to your GitHub repository and click the Actions tab.
Click Scrape UK Sponsorship Jobs in the left sidebar.
Click the ... menu (three dots) and select Disable workflow.
To resume later, return to the same page and click Enable workflow.
Deactivate your local virtual environment by running deactivate in your terminal.

Delete - I don't want to use this again

Remove all project resources and start fresh.

Delete the GitHub repository. Go to your repo on GitHub, click Settings, scroll to the bottom, and click Delete this repository.
Delete the local project folder from your Desktop.
The virtual environment lives inside your project folder, so deleting the folder removes it too.

Mission accomplished

That's a Wrap!

Nice work! You just built a fully automated job scraper that delivers fresh UK visa sponsorship listings to your GitHub repo every single week.

You've learned how to:

Built a Python web scraper using requests and BeautifulSoup to fetch and parse real-world HTML from NHS Jobs.
Exported structured job data to a clean CSV file, filtering out irrelevant listings along the way.
Automated the entire pipeline with GitHub Actions to run on a weekly cron schedule and commit results back to your repo.
Secret Mission: Added the Ministry of Justice jobs site as a second data source and merged results from multiple sites into a single CSV.

Ready to quiz yourself?