Build a PubMed Research Paper Monitor
Search PubMed, download open-access PDFs, and track papers in a CSV.
Introduction
30 Second Summary
Researchers and students spend hours manually checking databases for new papers in their field. What if you could automate that entire process with a single script?
In this project, you will build a Python script that automatically searches PubMed for recent research papers, downloads open-access PDFs, and tracks your growing collection in a CSV spreadsheet.
What You'll Build
A command-line Python tool that connects to PubMed's REST APIs to search for papers, fetch metadata, and download freely available PDFs.
By the end of this project, you'll have:
- A working Python script that searches PubMed for recent studies in any research field you choose.
- A local folder of open-access PDFs automatically downloaded from PubMed Central.
- A CSV spreadsheet tracking your entire paper collection with metadata like title, authors, date, and DOI.
- Secret Mission: Add multi-topic monitoring to search across multiple research fields simultaneously.
Are there any prerequisites?
None. This guide includes setup instructions for all required tools. You do not need any prior experience with APIs or Python libraries to get started.
Set Up Your Project
Before you can search PubMed for research papers, you need a workspace with the right tools installed.
In this step, you will install the Python HTTP library your script needs, create the project folder structure, and verify that your machine can talk to the PubMed API.
In this step, get ready to:
- Install the requests library for making HTTP calls.
- Create the project folder with your script file and a papers directory.
- Test your connection to the PubMed API.
Install the requests library
Python's standard library can handle basic HTTP, but the requests library makes it much simpler to work with web APIs. It handles URL encoding, headers, and error responses for you.
- Open a terminal in VS Code by pressing Ctrl+` (Windows) or Cmd+` (macOS). The backtick key is above Tab.
- Install the requests library by running this command:
python -m pip install requestsYou should see output ending with Successfully installed requests-... (the version number may vary).
Seeing 'python' is not recognized?
Try python3 -m pip install requests instead. On some systems the Python executable is named differently.
If both fail, make sure Python is added to your system PATH. You can verify by running python --version first.
help me fix a 'python not recognized' error in my VS Code terminal on Windows
Create the project folder structure
Your project needs a main script file and a folder to store downloaded PDFs.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir pubmed-monitor
cd pubmed-monitor- Create the papers/ directory that will hold your downloaded PDFs by running:
mkdir papers- Create the main script file by running:
New-Item pubmed_monitor.py- Confirm the files and folder exist by running:
lsYou should see papers and pubmed_monitor.py in the output.
- Now open this folder in VS Code. Click File in the top menu bar.
- Click Open Folder.
- Navigate to your Desktop, select the pubmed-monitor folder, then click Select Folder.
Why open the folder in VS Code?
Opening the project folder makes VS Code's file explorer show your project structure. It also ensures the integrated terminal starts in the correct directory for running your script.
Write a test request to the PubMed API
Before building the full monitor, you will write a quick test that proves your machine can reach PubMed's servers. This catches network issues early.
- In VS Code's file explorer on the left, click pubmed_monitor.py to open it.
- Add the following imports and configuration constants at the top of the file:
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"What does this code do?
The first six lines import libraries you will use throughout the project. requests handles HTTP calls, xml.etree.ElementTree parses XML responses, and the rest handle files, timing, and dates.
The configuration section defines your search topic, how far back to look (30 days), and where to save results. BASE_URL points to NCBI's E-utilities REST API and OA_API_URL points to the PMC Open Access service.
- Below the configuration, add this test block that sends a single search request to PubMed:
if __name__ == "__main__":
# Test connection to PubMed
url = f"{BASE_URL}esearch.fcgi"
params = {"db": "pubmed", "term": "hematology", "retmax": 1, "retmode": "xml"}
response = requests.get(url, params=params)
response.raise_for_status()
print("Successfully connected to PubMed API!")What does this test do?
This block only runs when you execute the script directly. It builds a URL for PubMed's ESearch endpoint and asks for just 1 result to keep the test fast.
raise_for_status() will throw an error if the server returns a problem (like a 404 or 500 status code). If you see the success message, your network, Python install, and the requests library are all working.
- Save pubmed_monitor.py by pressing Ctrl+S (Windows) or Cmd+S (macOS).
- Open a terminal in VS Code (press Ctrl+` if it closed) and run your script:
python pubmed_monitor.pyYou should see:
Successfully connected to PubMed API!Seeing a connection error?
Make sure you are connected to the internet. PubMed's API requires a live network connection.
If you see ModuleNotFoundError: No module named 'requests', the library did not install correctly. Run python -m pip install requests again and check for errors in the output.
help me debug why my PubMed test script is failing
✔️ Awesome, I've got everything!
Great. Make sure you have saved pubmed_monitor.py before moving on.
ⓧ I'd like to double check the full code
Here is the complete pubmed_monitor.py file so far:
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"
if __name__ == "__main__":
url = f"{BASE_URL}esearch.fcgi"
params = {"db": "pubmed", "term": "hematology", "retmax": 1, "retmode": "xml"}
response = requests.get(url, params=params)
response.raise_for_status()
print("Successfully connected to PubMed API!")Your environment is ready and PubMed is responding. Next up, you will write a proper search function that finds recent papers in your chosen field.
Search PubMed for Recent Papers
Your project folder is ready and Python can talk to the PubMed API. Now it's time to turn that raw connection into a proper search function.
PubMed's ESearch API accepts a keyword query and returns a list of matching article IDs called PMIDs. You'll write a function that sends a filtered search and extracts those IDs from the XML response.
In this step, get ready to:
- Write a function that queries PubMed's ESearch API.
- Filter results to papers published in the last 30 days.
- Parse the XML response to extract PubMed IDs (PMIDs).
Write the search_pubmed() function
The ESearch API returns XML containing a <Count> element (total matches) and an <IdList> element (the article IDs). Your function will build the request, send it, and parse both pieces out of the response.
- In pubmed_monitor.py, find the if __name__ == "__main__": block at the bottom of the file.
- Delete everything from that line to the end of the file.
- Paste the following code in its place:
def search_pubmed(query, days_back=30, max_results=20):
"""Search PubMed for recent articles matching the query."""
url = f"{BASE_URL}esearch.fcgi"
params = {
"db": "pubmed",
"term": query,
"retmax": max_results,
"reldate": days_back,
"datetype": "edat",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
# Parse the XML response to pull out article IDs
root = ET.fromstring(response.text)
pmids = [id_elem.text for id_elem in root.findall(".//Id")]
count = root.findtext(".//Count", default="0")
print(f"Found {count} results for '{query}', retrieved {len(pmids)} PMIDs")
return pmids
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
print(pmids)What does this code do?
The function builds a request to PubMed's ESearch endpoint with several filtering parameters:
- reldate restricts results to papers indexed within the last N days (30 by default).
- datetype set to edat means the date filter uses the Entrez date (when PubMed indexed the article).
- retmax caps how many IDs come back in one response (20 by default).
After the request, Python's built-in xml.etree.ElementTree parses the response. It finds every <Id> element inside the <IdList> and collects their text values into a Python list.
What is a PMID?
A PMID (PubMed Identifier) is a unique number assigned to every article in the PubMed database. You'll use these IDs in the next step to fetch full article details like titles, authors, and abstracts.
- Save pubmed_monitor.py.
- Run the script by entering this command in your terminal:
python pubmed_monitor.pyYou should see output similar to this:
Found 347 results for 'hematology', retrieved 20 PMIDs
['39878234', '39877991', '39877543', ...]What should I see?
The first line shows the total number of hematology papers indexed in the last 30 days, followed by how many IDs your script retrieved (capped at 20). The second line is the Python list of PMID strings. Your exact numbers and IDs will differ because new papers are added to PubMed daily.
Not seeing any results?
- Check that you deleted the entire old if __name__ == "__main__": block. If any of the old test code remains, Python may skip your new function call.
- Make sure your function definition starts at the same indentation level as the old block (no extra spaces). Python is whitespace-sensitive.
- If you see a ConnectionError, check your internet connection. The script needs to reach NCBI's servers.
My PubMed search function isn't returning results. Can you help me debug it?
✔️ Awesome, I've got everything!
Great. Double check you've saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the complete pubmed_monitor.py file at this point:
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"
def search_pubmed(query, days_back=30, max_results=20):
"""Search PubMed for recent articles matching the query."""
url = f"{BASE_URL}esearch.fcgi"
params = {
"db": "pubmed",
"term": query,
"retmax": max_results,
"reldate": days_back,
"datetype": "edat",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
pmids = [id_elem.text for id_elem in root.findall(".//Id")]
count = root.findtext(".//Count", default="0")
print(f"Found {count} results for '{query}', retrieved {len(pmids)} PMIDs")
return pmids
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
print(pmids)Your search function is working. Next up, you'll use those PMIDs to fetch full article metadata like titles, authors, and publication dates.
Fetch Article Metadata
In the previous step, you found a list of PubMed IDs (PMIDs) matching your search. PMIDs are just reference numbers. To make them useful, you need the actual article details behind each one.
PubMed's EFetch API accepts a batch of PMIDs and returns full XML records containing titles, authors, dates, abstracts, and identifiers. You will write a function that calls this API and parses the response into clean Python dictionaries.
In this step, get ready to:
- Write the fetch_article_details() function that calls the EFetch API with your PMIDs.
- Parse the XML response to extract title, authors, publication date, DOI, abstract, and PMC ID.
- Print a formatted summary of each article found.
Call the EFetch API with your PMIDs
The EFetch endpoint takes a comma-separated list of PMIDs and returns detailed XML for each article. Your function will send this request and set up parsing of the response.
- In pubmed_monitor.py, find the end of the search_pubmed() function (after the return pmids line).
- Add the first part of the fetch_article_details() function by copying this code below it:
def fetch_article_details(pmids):
"""Fetch detailed metadata for a list of PMIDs."""
if not pmids:
return []
url = f"{BASE_URL}efetch.fcgi"
params = {
"db": "pubmed",
"id": ",".join(pmids),
"rettype": "xml",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
articles = []What does this code do?
- if not pmids: return [] is a guard clause that avoids making a pointless API call when the search found nothing.
- The id parameter joins all PMIDs with commas so a single request fetches all articles at once.
- ET.fromstring() parses the XML text into a tree you can query with .find() and .findall() methods.
- articles = [] creates the list where you will store each parsed article dictionary.
You will not see a change yet. The next section completes the parsing loop that fills the articles list.
Parse the XML response to extract metadata
PubMed's XML nests each article inside a PubmedArticle element. Inside that, you find ArticleTitle for the title, Author elements with LastName and ForeName fields, and ArticleId elements with an IdType attribute distinguishing DOIs from PMC IDs.
- Directly below the articles = [] line you just added, copy the parsing loop that extracts every field:
for article in root.findall(".//PubmedArticle"):
pmid = article.findtext(".//PMID")
title = article.findtext(".//ArticleTitle", default="No title")
authors = []
for author in article.findall(".//Author"):
last_name = author.findtext("LastName", default="")
fore_name = author.findtext("ForeName", default="")
if last_name:
authors.append(f"{last_name} {fore_name}".strip())
authors_str = "; ".join(authors[:5])
if len(authors) > 5:
authors_str += " et al."
pub_date = article.find(".//PubDate")
year = pub_date.findtext("Year", default="") if pub_date is not None else ""
month = pub_date.findtext("Month", default="") if pub_date is not None else ""
day = pub_date.findtext("Day", default="") if pub_date is not None else ""
date_str = f"{year} {month} {day}".strip()
abstract_parts = article.findall(".//AbstractText")
abstract = " ".join(part.text or "" for part in abstract_parts)
doi = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "doi":
doi = article_id.text or ""
break
pmc_id = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "pmc":
pmc_id = article_id.text or ""
break
articles.append({
"pmid": pmid,
"title": title,
"authors": authors_str,
"date": date_str,
"abstract": abstract[:500],
"doi": doi,
"pmc_id": pmc_id
})
return articlesWhat does this parsing loop do?
- root.findall(".//PubmedArticle") iterates over every article record in the XML response.
- Author names are combined as "LastName ForeName" and capped at 5 entries. If there are more, the list ends with "et al." to keep output readable.
- The PubDate element contains separate Year, Month, and Day children. The code safely handles cases where any of these are missing.
- DOI and PMC ID live in ArticleId elements differentiated by the IdType attribute. Not every article has both, so defaults to empty strings keep things safe.
- The abstract is truncated to 500 characters to keep the CSV manageable later.
You will not see output yet. The next substep updates your script's entry point to call this function and display results.
Print a formatted summary of each article
Your current if __name__ block only prints raw PMIDs. You need to call fetch_article_details() and display the results in a readable format.
- At the bottom of pubmed_monitor.py, find this code:
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
print(pmids)- Replace print(pmids) with the following lines, so the block looks like this:
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
articles = fetch_article_details(pmids)
for a in articles:
print(f"\nTitle: {a['title']}")
print(f"Authors: {a['authors']}")
print(f"Date: {a['date']}")What changed here?
Instead of printing a raw list of IDs, the script now passes those IDs to fetch_article_details() and loops through the results. Each article's title, authors, and date are printed on separate lines for easy scanning.
- Save pubmed_monitor.py.
- Run your script by executing this command:
python pubmed_monitor.pyYou should see a count of results, followed by article titles, author names, and publication dates printed in your terminal.
Not seeing any articles?
- Check that the fetch_article_details() function is indented at the module level (same level as search_pubmed()), not nested inside it.
- Make sure the for loop inside fetch_article_details() is indented one level inside the function (4 spaces), and the lines inside the loop are indented two levels (8 spaces).
- If you see a network error, confirm you have an internet connection. PubMed's servers need to be reachable.
help me debug why my fetch_article_details function is not returning results
✔️ Awesome, I've got everything!
Great. Double check you have saved pubmed_monitor.py before moving on.
ⓧ I'd like to double check the full code
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"
def search_pubmed(query, days_back=30, max_results=20):
"""Search PubMed for recent articles matching the query."""
url = f"{BASE_URL}esearch.fcgi"
params = {
"db": "pubmed",
"term": query,
"retmax": max_results,
"reldate": days_back,
"datetype": "edat",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
pmids = [id_elem.text for id_elem in root.findall(".//Id")]
count = root.findtext(".//Count", default="0")
print(f"Found {count} results for '{query}', retrieved {len(pmids)} PMIDs")
return pmids
def fetch_article_details(pmids):
"""Fetch detailed metadata for a list of PMIDs."""
if not pmids:
return []
url = f"{BASE_URL}efetch.fcgi"
params = {
"db": "pubmed",
"id": ",".join(pmids),
"rettype": "xml",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
articles = []
for article in root.findall(".//PubmedArticle"):
pmid = article.findtext(".//PMID")
title = article.findtext(".//ArticleTitle", default="No title")
authors = []
for author in article.findall(".//Author"):
last_name = author.findtext("LastName", default="")
fore_name = author.findtext("ForeName", default="")
if last_name:
authors.append(f"{last_name} {fore_name}".strip())
authors_str = "; ".join(authors[:5])
if len(authors) > 5:
authors_str += " et al."
pub_date = article.find(".//PubDate")
year = pub_date.findtext("Year", default="") if pub_date is not None else ""
month = pub_date.findtext("Month", default="") if pub_date is not None else ""
day = pub_date.findtext("Day", default="") if pub_date is not None else ""
date_str = f"{year} {month} {day}".strip()
abstract_parts = article.findall(".//AbstractText")
abstract = " ".join(part.text or "" for part in abstract_parts)
doi = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "doi":
doi = article_id.text or ""
break
pmc_id = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "pmc":
pmc_id = article_id.text or ""
break
articles.append({
"pmid": pmid,
"title": title,
"authors": authors_str,
"date": date_str,
"abstract": abstract[:500],
"doi": doi,
"pmc_id": pmc_id
})
return articles
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
articles = fetch_article_details(pmids)
for a in articles:
print(f"\nTitle: {a['title']}")
print(f"Authors: {a['authors']}")
print(f"Date: {a['date']}")Your script can now reach into PubMed's database and pull back structured metadata for any matching papers. Next up, you will download open-access PDFs for the articles that have them.
Download Open-Access PDFs
Your script can now find papers and pull their metadata. But metadata alone does not give you the actual research to read.
Not every PubMed article is freely downloadable. Only papers deposited in PubMed Central's Open Access Subset have PDFs you can grab programmatically. In this step, you will query the PMC OA Web Service API to discover which articles have PDFs available, then download them directly into your papers/ folder.
In this step, get ready to:
- Write a function to check PDF availability via the PMC OA Web Service API.
- Write a function to download PDFs to your local folder.
- Add rate limiting delays to respect NCBI's usage policies.
Write the check_pdf_availability() function
The PMC OA Web Service API accepts a PMC ID and returns XML listing any downloadable resources for that article. Your function will look for a link element with format="pdf" and return the download URL.
- In pubmed_monitor.py, scroll to the bottom and find the if __name__ == "__main__": block.
- Add the following function directly above that if __name__ block:
def check_pdf_availability(pmc_id):
"""Check if a PDF is available for download via PMC OA API."""
params = {"id": pmc_id}
response = requests.get(OA_API_URL, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
# Look for a link element with format="pdf"
for link in root.findall(".//link"):
if link.get("format") == "pdf":
href = link.get("href", "")
# Convert FTP URLs to HTTPS for download via requests
if href.startswith("ftp://ftp.ncbi.nlm.nih.gov"):
href = href.replace(
"ftp://ftp.ncbi.nlm.nih.gov",
"https://ftp.ncbi.nlm.nih.gov"
)
return href
return NoneWhat does this code do?
The PMC OA API returns XML with <link> elements for each available download format. Your function searches specifically for the one with format="pdf".
The API returns FTP URLs like ftp://ftp.ncbi.nlm.nih.gov/.... Since the requests library does not support FTP, the function swaps the protocol to https://ftp.ncbi.nlm.nih.gov so you can download via a normal HTTP request.
- Save pubmed_monitor.py.
Seeing a syntax error?
Make sure your new function is at the same indentation level as fetch_article_details() above it. All def statements should be at the leftmost column.
Help me fix a syntax error in my check_pdf_availability function.
Write the download_pdf() function
Once you have a PDF URL, you need a function that actually downloads the file and saves it to disk. This function will also skip files you have already downloaded to avoid re-fetching them.
- Add the following function directly below check_pdf_availability() and above the if __name__ block:
def download_pdf(url, filename, folder):
"""Download a PDF file to the specified folder."""
os.makedirs(folder, exist_ok=True)
filepath = os.path.join(folder, filename)
# Skip if already downloaded
if os.path.exists(filepath):
print(f" Already downloaded: {filename}")
return filepath
# Stream the download to handle large files
response = requests.get(url, stream=True)
response.raise_for_status()
with open(filepath, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f" Downloaded: {filename}")
return filepathWhat does this code do?
os.makedirs(folder, exist_ok=True) creates the download folder if it does not already exist. The exist_ok=True flag prevents an error if the folder is already there.
The stream=True parameter tells requests to download the file in chunks rather than loading the entire PDF into memory at once. This keeps memory usage low for large files.
The duplicate check at the top means you can run your script multiple times without re-downloading papers you already have.
- Save pubmed_monitor.py.
Not sure where to add the function?
Your file should now have four functions in order: search_pubmed(), fetch_article_details(), check_pdf_availability(), then download_pdf(). The if __name__ block should remain at the very bottom.
Help me organize the functions in pubmed_monitor.py.
Add rate limiting and test PDF downloads
NCBI allows 3 requests per second without an API key. Adding 0.4-second delays between calls keeps your script safely within this limit.
Why 0.4 seconds between requests?
NCBI's rate limiting policy allows a maximum of 3 requests per second per IP address when you do not register for an API key. A 0.4-second sleep between each call ensures you never exceed 2.5 requests per second, giving yourself a comfortable safety margin.
- Find the existing if __name__ == "__main__": block at the bottom of your file.
- Replace the entire block with the following code:
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
time.sleep(0.4)
articles = fetch_article_details(pmids)
time.sleep(0.4)
for article in articles:
if article["pmc_id"]:
print(f"\nChecking PDF for: {article['title'][:60]}...")
time.sleep(0.4)
pdf_url = check_pdf_availability(article["pmc_id"])
if pdf_url:
filename = f"{article['pmc_id']}.pdf"
download_pdf(pdf_url, filename, DOWNLOAD_FOLDER)What does this code do?
time.sleep(0.4) pauses the script for 0.4 seconds between each API call. This respects NCBI's 3-requests-per-second limit.
The loop checks each article for a pmc_id. Only articles deposited in PubMed Central will have one. For those that do, it queries the OA API and downloads the PDF if available.
Each downloaded file is named after its PMC ID (e.g. PMC1234567.pdf), making it easy to identify which paper each file belongs to.
- Save pubmed_monitor.py.
- Run the script by executing this command in your terminal:
python pubmed_monitor.py- Confirm that at least one PDF was downloaded by running:
ls papers/You should see one or more .pdf files listed, each named with a PMC ID like PMC1234567.pdf.
No PDFs appeared in the folder?
Not every hematology article in your results will have a PMC ID, and not every PMC article has an open-access PDF. If zero PDFs download, try increasing MAX_RESULTS to 50 at the top of your file and running again.
If you see a connection error, check that your internet connection is active and that you can reach https://ftp.ncbi.nlm.nih.gov in your browser.
Help me troubleshoot why no PDFs were downloaded.
✔️ Awesome, I've got everything!
Great. Double check you have saved pubmed_monitor.py and that your papers/ folder contains at least one PDF.
ⓧ I'd like to double check the full code
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"
def search_pubmed(query, days_back=30, max_results=20):
"""Search PubMed for recent articles matching the query."""
url = f"{BASE_URL}esearch.fcgi"
params = {
"db": "pubmed",
"term": query,
"retmax": max_results,
"reldate": days_back,
"datetype": "edat",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
pmids = [id_elem.text for id_elem in root.findall(".//Id")]
count = root.findtext(".//Count", default="0")
print(f"Found {count} results for '{query}', retrieved {len(pmids)} PMIDs")
return pmids
def fetch_article_details(pmids):
"""Fetch detailed metadata for a list of PMIDs."""
if not pmids:
return []
url = f"{BASE_URL}efetch.fcgi"
params = {
"db": "pubmed",
"id": ",".join(pmids),
"rettype": "xml",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
articles = []
for article in root.findall(".//PubmedArticle"):
pmid = article.findtext(".//PMID")
title = article.findtext(".//ArticleTitle", default="No title")
authors = []
for author in article.findall(".//Author"):
last_name = author.findtext("LastName", default="")
fore_name = author.findtext("ForeName", default="")
if last_name:
authors.append(f"{last_name} {fore_name}".strip())
authors_str = "; ".join(authors[:5])
if len(authors) > 5:
authors_str += " et al."
pub_date = article.find(".//PubDate")
year = pub_date.findtext("Year", default="") if pub_date is not None else ""
month = pub_date.findtext("Month", default="") if pub_date is not None else ""
day = pub_date.findtext("Day", default="") if pub_date is not None else ""
date_str = f"{year} {month} {day}".strip()
abstract_parts = article.findall(".//AbstractText")
abstract = " ".join(part.text or "" for part in abstract_parts)
doi = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "doi":
doi = article_id.text or ""
break
pmc_id = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "pmc":
pmc_id = article_id.text or ""
break
articles.append({
"pmid": pmid,
"title": title,
"authors": authors_str,
"date": date_str,
"abstract": abstract[:500],
"doi": doi,
"pmc_id": pmc_id
})
return articles
def check_pdf_availability(pmc_id):
"""Check if a PDF is available for download via PMC OA API."""
params = {"id": pmc_id}
response = requests.get(OA_API_URL, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
for link in root.findall(".//link"):
if link.get("format") == "pdf":
href = link.get("href", "")
if href.startswith("ftp://ftp.ncbi.nlm.nih.gov"):
href = href.replace(
"ftp://ftp.ncbi.nlm.nih.gov",
"https://ftp.ncbi.nlm.nih.gov"
)
return href
return None
def download_pdf(url, filename, folder):
"""Download a PDF file to the specified folder."""
os.makedirs(folder, exist_ok=True)
filepath = os.path.join(folder, filename)
if os.path.exists(filepath):
print(f" Already downloaded: {filename}")
return filepath
response = requests.get(url, stream=True)
response.raise_for_status()
with open(filepath, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f" Downloaded: {filename}")
return filepath
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
time.sleep(0.4)
articles = fetch_article_details(pmids)
time.sleep(0.4)
for article in articles:
if article["pmc_id"]:
print(f"\nChecking PDF for: {article['title'][:60]}...")
time.sleep(0.4)
pdf_url = check_pdf_availability(article["pmc_id"])
if pdf_url:
filename = f"{article['pmc_id']}.pdf"
download_pdf(pdf_url, filename, DOWNLOAD_FOLDER)Your monitor can now grab actual research papers. Next up, you will build a CSV tracker so you never lose track of what you have collected.
Track Your Collection with CSV
Your script can search PubMed, fetch metadata, and download PDFs. But every time you run it, there is no memory of what it already found.
In this step, you will add a CSV spreadsheet to track every paper you discover. The script will check the CSV before processing, skip duplicates, and append only new results.
In this step, get ready to:
- Write a function to load previously saved paper IDs from the CSV.
- Write a function to append new papers to the CSV spreadsheet.
- Assemble the full main() function that ties the entire pipeline together with duplicate detection.
Write the load_existing_pmids() function
Before searching PubMed, your script needs to know which papers it has already collected. This function reads the CSV file and returns a set of PMIDs that are already tracked.
- In pubmed_monitor.py, find the download_pdf() function.
- Add the following function below it:
def load_existing_pmids(csv_file):
"""Load PMIDs already in the collection CSV."""
if not os.path.exists(csv_file):
return set()
existing = set()
with open(csv_file, "r", newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
for row in reader:
existing.add(row["pmid"])
return existingWhat does this code do?
- os.path.exists(csv_file) checks if the CSV file already exists. On the very first run, it won't exist yet, so the function returns an empty set.
- csv.DictReader reads each row as a dictionary, using the CSV headers as keys.
- The function collects every pmid value into a Python set. Sets allow instant lookup, so checking whether a PMID already exists is very fast.
- Save pubmed_monitor.py.
You won't see a change yet. This function is used by the main() function you will write in the third substep.
Write the save_to_csv() function
This function takes a list of article dictionaries and appends them to the CSV file. It writes headers only on the first run when the file does not yet exist.
- Directly below the load_existing_pmids() function you just added, paste this function:
def save_to_csv(articles, csv_file):
"""Save article metadata to the collection CSV."""
file_exists = os.path.exists(csv_file)
fieldnames = [
"pmid", "title", "authors", "date", "doi",
"pmc_id", "pdf_downloaded", "pdf_filename", "added_date", "abstract"
]
with open(csv_file, "a", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
if not file_exists:
writer.writeheader()
for article in articles:
writer.writerow(article)What does this code do?
- The file is opened in append mode ("a") so new rows are added to the end without overwriting existing data.
- fieldnames defines the column order. Each article dictionary is written as one row matching these columns.
- The if not file_exists check ensures headers are written only once, on the first run.
- Save pubmed_monitor.py.
You won't see a change yet. The next substep wires everything together.
Assemble the main() function
Right now the bottom of your file has a bare if __name__ == "__main__": block that runs search, fetch, and download one after another with no tracking. You will replace that entire block with a proper main() function that adds duplicate detection, CSV logging, and status reporting.
- In pubmed_monitor.py, scroll to the bottom and find this block:
if __name__ == "__main__":
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
time.sleep(0.4)
articles = fetch_article_details(pmids)
time.sleep(0.4)
for article in articles:
if article["pmc_id"]:
print(f"\nChecking PDF for: {article['title'][:60]}...")
time.sleep(0.4)
pdf_url = check_pdf_availability(article["pmc_id"])
if pdf_url:
filename = f"{article['pmc_id']}.pdf"
download_pdf(pdf_url, filename, DOWNLOAD_FOLDER)- Delete that entire block (from if __name__ down to the last line of the file).
- Replace it with the following main() function and entry point:
def main():
"""Main function to run the PubMed monitor."""
print("=" * 60)
print("PubMed Research Paper Monitor")
print("=" * 60)
# Load PMIDs we already have in our collection
existing_pmids = load_existing_pmids(CSV_FILE)
print(f"\nExisting papers in collection: {len(existing_pmids)}")
print(f"\nSearching for: {SEARCH_TERM}")
print("-" * 40)
# Search PubMed for recent papers
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
time.sleep(0.4)
# Filter out papers we already have
new_pmids = [p for p in pmids if p not in existing_pmids]
print(f"New papers to process: {len(new_pmids)}")
if not new_pmids:
print("\nNo new papers found. Your collection is up to date!")
return
# Fetch metadata for new papers only
articles = fetch_article_details(new_pmids)
time.sleep(0.4)
all_new_articles = []
# Process each article: check for PDFs and download if available
for article in articles:
article["pdf_downloaded"] = "No"
article["pdf_filename"] = ""
article["added_date"] = datetime.now().strftime("%Y-%m-%d")
if article["pmc_id"]:
print(f"\n Checking PDF for: {article['title'][:60]}...")
time.sleep(0.4)
pdf_url = check_pdf_availability(article["pmc_id"])
if pdf_url:
filename = f"{article['pmc_id']}.pdf"
try:
download_pdf(pdf_url, filename, DOWNLOAD_FOLDER)
article["pdf_downloaded"] = "Yes"
article["pdf_filename"] = filename
except Exception as e:
print(f" Failed to download: {e}")
all_new_articles.append(article)
# Save all new articles to the CSV
if all_new_articles:
save_to_csv(all_new_articles, CSV_FILE)
print(f"\n{'=' * 60}")
print(f"Added {len(all_new_articles)} new papers to {CSV_FILE}")
pdfs = sum(1 for a in all_new_articles if a["pdf_downloaded"] == "Yes")
print(f"Downloaded {pdfs} PDFs to '{DOWNLOAD_FOLDER}/' folder")
else:
print(f"\n{'=' * 60}")
print("No new papers found.")
print("=" * 60)
if __name__ == "__main__":
main()What does this code do?
The main() function orchestrates the entire pipeline in five stages:
- Load existing PMIDs from the CSV so the script knows what it has already collected.
- Search PubMed and filter out any PMIDs that are already in the collection.
- Fetch metadata only for new papers, saving API calls.
- Attempt to download PDFs for articles that have a PMC ID. A try/except block catches download failures gracefully.
- Append all new articles to the CSV. The final summary reports how many papers and PDFs were added.
- Save pubmed_monitor.py.
- Run the script by entering this command in your terminal:
python pubmed_monitor.pyYou should see a banner reading PubMed Research Paper Monitor, followed by search results, PDF download attempts, and a summary line reporting how many new papers were added to collection.csv.
- Run the exact same command a second time:
python pubmed_monitor.pyThis time you should see No new papers found. Your collection is up to date! because every PMID from the search is already in your CSV. Duplicate detection is working.
Seeing a KeyError or empty CSV?
- If you see KeyError: 'pmid', make sure your fieldnames list in save_to_csv() starts with "pmid" and matches the keys in the article dictionaries.
- If the second run still reports new papers, check that load_existing_pmids() reads from the same file path as save_to_csv(). Both should use CSV_FILE.
- If you see a permissions error, make sure no other program (like Excel) has collection.csv open.
Help me debug my CSV tracking function.
✔️ Awesome, I've got everything!
Great. Double check you have saved your file.
ⓧ I'd like to double check the full code
Here is the complete pubmed_monitor.py at this point:
import requests
import xml.etree.ElementTree as ET
import csv
import os
import time
from datetime import datetime
# Configuration
SEARCH_TERM = "hematology"
DAYS_BACK = 30
MAX_RESULTS = 20
DOWNLOAD_FOLDER = "papers"
CSV_FILE = "collection.csv"
BASE_URL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
OA_API_URL = "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi"
def search_pubmed(query, days_back=30, max_results=20):
"""Search PubMed for recent articles matching the query."""
url = f"{BASE_URL}esearch.fcgi"
params = {
"db": "pubmed",
"term": query,
"retmax": max_results,
"reldate": days_back,
"datetype": "edat",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
pmids = [id_elem.text for id_elem in root.findall(".//Id")]
count = root.findtext(".//Count", default="0")
print(f"Found {count} results for '{query}', retrieved {len(pmids)} PMIDs")
return pmids
def fetch_article_details(pmids):
"""Fetch detailed metadata for a list of PMIDs."""
if not pmids:
return []
url = f"{BASE_URL}efetch.fcgi"
params = {
"db": "pubmed",
"id": ",".join(pmids),
"rettype": "xml",
"retmode": "xml"
}
response = requests.get(url, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
articles = []
for article in root.findall(".//PubmedArticle"):
pmid = article.findtext(".//PMID")
title = article.findtext(".//ArticleTitle", default="No title")
authors = []
for author in article.findall(".//Author"):
last_name = author.findtext("LastName", default="")
fore_name = author.findtext("ForeName", default="")
if last_name:
authors.append(f"{last_name} {fore_name}".strip())
authors_str = "; ".join(authors[:5])
if len(authors) > 5:
authors_str += " et al."
pub_date = article.find(".//PubDate")
year = pub_date.findtext("Year", default="") if pub_date is not None else ""
month = pub_date.findtext("Month", default="") if pub_date is not None else ""
day = pub_date.findtext("Day", default="") if pub_date is not None else ""
date_str = f"{year} {month} {day}".strip()
abstract_parts = article.findall(".//AbstractText")
abstract = " ".join(part.text or "" for part in abstract_parts)
doi = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "doi":
doi = article_id.text or ""
break
pmc_id = ""
for article_id in article.findall(".//ArticleId"):
if article_id.get("IdType") == "pmc":
pmc_id = article_id.text or ""
break
articles.append({
"pmid": pmid,
"title": title,
"authors": authors_str,
"date": date_str,
"abstract": abstract[:500],
"doi": doi,
"pmc_id": pmc_id
})
return articles
def check_pdf_availability(pmc_id):
"""Check if a PDF is available for download via PMC OA API."""
params = {"id": pmc_id}
response = requests.get(OA_API_URL, params=params)
response.raise_for_status()
root = ET.fromstring(response.text)
for link in root.findall(".//link"):
if link.get("format") == "pdf":
href = link.get("href", "")
if href.startswith("ftp://ftp.ncbi.nlm.nih.gov"):
href = href.replace(
"ftp://ftp.ncbi.nlm.nih.gov",
"https://ftp.ncbi.nlm.nih.gov"
)
return href
return None
def download_pdf(url, filename, folder):
"""Download a PDF file to the specified folder."""
os.makedirs(folder, exist_ok=True)
filepath = os.path.join(folder, filename)
if os.path.exists(filepath):
print(f" Already downloaded: {filename}")
return filepath
response = requests.get(url, stream=True)
response.raise_for_status()
with open(filepath, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f" Downloaded: {filename}")
return filepath
def load_existing_pmids(csv_file):
"""Load PMIDs already in the collection CSV."""
if not os.path.exists(csv_file):
return set()
existing = set()
with open(csv_file, "r", newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
for row in reader:
existing.add(row["pmid"])
return existing
def save_to_csv(articles, csv_file):
"""Save article metadata to the collection CSV."""
file_exists = os.path.exists(csv_file)
fieldnames = [
"pmid", "title", "authors", "date", "doi",
"pmc_id", "pdf_downloaded", "pdf_filename", "added_date", "abstract"
]
with open(csv_file, "a", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
if not file_exists:
writer.writeheader()
for article in articles:
writer.writerow(article)
def main():
"""Main function to run the PubMed monitor."""
print("=" * 60)
print("PubMed Research Paper Monitor")
print("=" * 60)
existing_pmids = load_existing_pmids(CSV_FILE)
print(f"\nExisting papers in collection: {len(existing_pmids)}")
print(f"\nSearching for: {SEARCH_TERM}")
print("-" * 40)
pmids = search_pubmed(SEARCH_TERM, days_back=DAYS_BACK, max_results=MAX_RESULTS)
time.sleep(0.4)
new_pmids = [p for p in pmids if p not in existing_pmids]
print(f"New papers to process: {len(new_pmids)}")
if not new_pmids:
print("\nNo new papers found. Your collection is up to date!")
return
articles = fetch_article_details(new_pmids)
time.sleep(0.4)
all_new_articles = []
for article in articles:
article["pdf_downloaded"] = "No"
article["pdf_filename"] = ""
article["added_date"] = datetime.now().strftime("%Y-%m-%d")
if article["pmc_id"]:
print(f"\n Checking PDF for: {article['title'][:60]}...")
time.sleep(0.4)
pdf_url = check_pdf_availability(article["pmc_id"])
if pdf_url:
filename = f"{article['pmc_id']}.pdf"
try:
download_pdf(pdf_url, filename, DOWNLOAD_FOLDER)
article["pdf_downloaded"] = "Yes"
article["pdf_filename"] = filename
except Exception as e:
print(f" Failed to download: {e}")
all_new_articles.append(article)
if all_new_articles:
save_to_csv(all_new_articles, CSV_FILE)
print(f"\n{'=' * 60}")
print(f"Added {len(all_new_articles)} new papers to {CSV_FILE}")
pdfs = sum(1 for a in all_new_articles if a["pdf_downloaded"] == "Yes")
print(f"Downloaded {pdfs} PDFs to '{DOWNLOAD_FOLDER}/' folder")
else:
print(f"\n{'=' * 60}")
print("No new papers found.")
print("=" * 60)
if __name__ == "__main__":
main()Your PubMed monitor is complete. It searches, fetches metadata, downloads PDFs, and tracks everything in a tidy CSV. Run it whenever you want to catch up on the latest research in your field.
Secret mission
Add Multi-Topic Monitoring
Your PubMed monitor currently searches for just one topic. What if you're researching across multiple fields? Extend the tool to search multiple research topics in a single run, with cross-topic duplicate detection and a collection summary report.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally on your machine. There are no cloud resources, no accounts to close, and no ongoing costs.
Decide whether to keep your resources as they are, pause by freeing up disk space, or delete everything entirely.
Resources you used:
- pubmed_monitor.py (the Python monitoring script).
- papers/ folder containing downloaded open-access PDFs.
- collection.csv spreadsheet tracking your paper collection.
Keep everything running
No action needed. Choose this if you plan to keep running the script to monitor new papers.
- Your script, PDFs, and CSV all stay on your machine at no cost.
- Run python pubmed_monitor.py anytime you want to check for new papers.
- The requests library stays installed in your Python environment. Other Python projects can use it too.
Pause - I'll come back to this later
Free up disk space by removing downloaded PDFs while keeping your script and metadata intact.
- Open the papers/ folder inside your project and delete all the PDF files.
- Keep pubmed_monitor.py and collection.csv so you can resume later.
- When you run the script again, it will re-download any PDFs that are still available from PMC.
Delete - I don't want to use this again
Remove all project files from your machine. There are no cloud resources or accounts to clean up.
- Close VS Code.
- Open your Desktop in Finder (macOS) or File Explorer (Windows).
- Find your project folder (the one containing pubmed_monitor.py, papers/, and collection.csv).
- Delete the entire folder. This removes your script, all downloaded PDFs, and your collection spreadsheet.
Nice Work!
Nice Work!
Nice work! You just built a fully functional research paper monitoring tool that searches PubMed's biomedical literature database, downloads open-access PDFs, and tracks everything in a spreadsheet.
You've learned how to:
- Built a Python script that calls the E-utilities API to search PubMed and parsed XML responses to extract structured article metadata.
- Used the PMC Open Access Web Service to discover and download freely available PDFs with proper rate limiting.
- Created a CSV-based collection tracker with duplicate detection so running the script multiple times never produces repeat entries.
- Secret Mission: Extended the tool to monitor multiple research topics simultaneously with cross-topic deduplication and a collection summary report.
Ready to quiz yourself?