Build a Person Profiler with Gemini
Scrape the web and use Gemini to build structured person profiles.
Introduction
30 Second Summary
Ever Googled someone's name before a meeting to figure out who they are? You end up opening a dozen tabs, skimming profiles, and mentally stitching together a picture of the person.
In this project, you will build a Python module that takes a person's name and returns a structured profile. It searches the web, scrapes the top results, and uses Gemini to synthesize everything into a single clean output.
What You'll Build
A self-contained Python package called person_profiler that converts a person's name into a structured JSON profile by combining web scraping with LLM-powered data synthesis.
By the end of this project, you'll have:
- A multi-query search engine that finds relevant web results across LinkedIn, professional sites, and social media.
- A web scraper that extracts clean text from search result pages and handles failures gracefully.
- A Gemini-powered synthesis pipeline that turns raw scraped text into a structured profile with job title, company, education, social links, and a bio summary.
- A confidence scoring system that rates how reliable each profile field is based on source agreement (Secret Mission).
Are there any prerequisites?
You need Python installed on your machine and basic Python experience. Your existing larpAI project's .env file should already have a GEMINI_API_KEY set up from a previous project.
Install Dependencies and Create the Package
Your person profiler needs four Python packages to handle each stage of its pipeline. They cover web search, page fetching, HTML parsing, and Gemini API calls.
In this step, you'll install those packages and build the person_profiler/ package structure inside your larpAI project. Everything lives in its own directory, so your existing code stays untouched.
In this step, get ready to:
- Install four Python packages for search, scraping, and AI synthesis.
- Create the person_profiler/ package directory with four module files.
- Set up the configuration module and verify the setup works.
Install the required Python packages
Your existing larpAI project has its own dependencies. The person profiler needs four additional packages that are not yet installed.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Visual Studio Code and press Enter to open it.
- Click File in the menu bar.
- Click Open Folder.
- Select your larpAI folder, then click Open.
- Open the integrated terminal by pressing Ctrl+` (backtick key).
- Install all four packages by running this command:
pip install ddgs httpx beautifulsoup4 google-genaiWhat do these packages do?
- ddgs runs web searches through DuckDuckGo. It requires no API key.
- httpx is a modern Python HTTP client for fetching web pages.
- beautifulsoup4 parses HTML and extracts clean text content from web pages.
- google-genai is Google's unified SDK for the Gemini API. It replaces the older google-generativeai package. Both can be installed side by side, so your existing larpAI code is unaffected.
You should see Successfully installed messages in your terminal output for each package.
Seeing a pip error?
- Make sure you're using the correct pip for your Python installation. Try running pip3 install instead.
- If you see a permissions error, try adding --user to the end of the install command.
- Help me troubleshoot my pip install error for the person profiler packages.
Create the package directory and files
In Python, a directory with an __init__.py file becomes a package that other modules can import from. You'll create this structure now so the profiler is importable from anywhere in your larpAI project.
- Create the person_profiler directory by running this command:
mkdir person_profiler- Create the four module files inside it by running:
macOS
touch person_profiler/__init__.py person_profiler/profiler_config.py person_profiler/profiler.py person_profiler/test_profiler.pyWindows
New-Item person_profiler\__init__.py -ItemType File
New-Item person_profiler\profiler_config.py -ItemType File
New-Item person_profiler\profiler.py -ItemType File
New-Item person_profiler\test_profiler.py -ItemType File- Confirm the files exist by running:
ls person_profiler/You should see __init__.py, profiler_config.py, profiler.py, and test_profiler.py in the output.
What are these files for?
- __init__.py tells Python that this directory is a package. Without it, Python treats the folder as a regular directory and won't let you import from it.
- profiler_config.py holds tunable settings like the model name and scraping limits. It is named profiler_config.py instead of config.py to avoid import shadowing with your existing root-level config.py.
- profiler.py is where the core search, scrape, and synthesize pipeline will live.
- test_profiler.py is a CLI test script you'll use to run and validate the module.
Don't see your files?
- On Windows, make sure you used PowerShell (not Command Prompt) for the New-Item commands.
- Double-check you ran mkdir person_profiler first. The file creation commands need the directory to already exist.
- Help me create the person_profiler directory and files.
Set up the configuration module
The person profiler needs its own configuration for API keys, model settings, and scraping parameters. Your existing .env file already sets the GEMINI_API_KEY environment variable, so this config just reads it from the environment.
- Open profiler_config.py in your editor. You'll find it inside the person_profiler/ folder in the Explorer sidebar.
- Paste the following code:
import os
# Read the Gemini API key from environment variables
GEMINI_API_KEY = os.environ.get('GEMINI_API_KEY', 'your-api-key-here')
GEMINI_MODEL = 'gemini-2.5-flash'
# Tunable parameters for search and scraping
MAX_SEARCH_RESULTS = 8
SCRAPE_TIMEOUT = 10
MAX_CONTENT_LENGTH = 3000
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'What does this code do?
- GEMINI_API_KEY reads your API key from the GEMINI_API_KEY environment variable. Your existing .env file already sets this, so you don't need to add it again.
- GEMINI_MODEL sets the model to gemini-2.5-flash, the same model your larpAI project uses.
- MAX_SEARCH_RESULTS caps how many results each search query returns.
- SCRAPE_TIMEOUT sets a 10-second limit for fetching each web page.
- MAX_CONTENT_LENGTH truncates scraped text to 3000 characters per page to avoid overloading Gemini's context.
- USER_AGENT mimics a real browser so websites don't block your requests.
- Save profiler_config.py.
- Verify the config loads correctly by running this command:
python -c "from person_profiler import profiler_config; print('Model:', profiler_config.GEMINI_MODEL)"You should see Model: gemini-2.5-flash printed in your terminal.
Seeing a ModuleNotFoundError?
- Make sure you're running the command from your larpAI project root (the folder containing main.py), not from inside person_profiler/.
- Check that __init__.py exists inside person_profiler/. Python needs this file to recognize the directory as a package.
- I'm getting a ModuleNotFoundError when importing profiler_config. Help me fix this.
Awesome, I've got everything!
Double check you've saved profiler_config.py before moving on.
I'd like to double check the full code
import os
GEMINI_API_KEY = os.environ.get('GEMINI_API_KEY', 'your-api-key-here')
GEMINI_MODEL = 'gemini-2.5-flash'
MAX_SEARCH_RESULTS = 8
SCRAPE_TIMEOUT = 10
MAX_CONTENT_LENGTH = 3000
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'Your environment is ready and the package structure is in place. Next up, you'll build the multi-query search engine that finds information about a person across the web.
Build the Multi-Query Search Engine
Your package structure is in place and your config is wired up. Now it is time to build the first real piece of the pipeline: the search engine that finds information about a person on the web.
A single search query rarely captures everything about a person. Someone might appear on LinkedIn under their full name, on Twitter with a handle, and on a company page under a job title. Your search engine will fire multiple optimized queries to cover all these angles, then merge the results into one deduplicated list.
In this step, get ready to:
- Build a search query generator that creates targeted queries for LinkedIn, professional sites, and social media.
- Implement a search function that runs each query through DuckDuckGo and deduplicates results by URL.
- Test the search engine with a public figure's name and inspect the results.
Add the search imports to profiler.py
Before writing any functions, profiler.py needs its imports and a logging setup. Logging lets you see what the search engine is doing as it runs each query.
- Open person_profiler/profiler.py in your editor and add these imports at the top of the file:
import logging
from ddgs import DDGS
from . import profiler_config as config
# Set up logging so search progress and warnings are visible
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)What does this code do?
- DDGS is the DuckDuckGo search client from the ddgs package. It lets you run web searches without an API key.
- from . import profiler_config as config uses a relative import to pull in your module's settings file. The . means "from the same package."
- logging.basicConfig(level=logging.INFO) turns on info-level logging so you can see progress messages in the terminal.
- Save profiler.py.
You won't see a change yet. The next chunk builds the first function that uses these imports.
Create the search query generator
A single query like "Joy Xia" returns broad, noisy results. A smarter approach fires multiple queries, each targeting a different platform or angle. One query hits LinkedIn. Another targets Twitter/X. A third looks for company or role mentions.
- In profiler.py, add the _generate_search_queries() function below the logging setup:
def _generate_search_queries(name: str) -> list[str]:
# Each query targets a different platform or angle for maximum coverage
queries = [
f'{name} LinkedIn',
f'{name} professional background',
f'{name} site:twitter.com OR site:x.com',
f'"{name}" company OR title OR role',
]
return queriesWhat does each query do?
- LinkedIn query finds professional profiles with job titles and company names.
- professional background query catches bios on company websites, speaker pages, and news articles.
- site: query restricts results to Twitter/X, surfacing social media profiles directly.
- Quoted name with OR keywords forces an exact name match and looks for pages mentioning their company or role.
- Save profiler.py.
You won't see a change yet. The next function uses these queries to actually run the searches.
Implement the deduplicating search function
Running four queries means the same URL might appear in multiple result sets. The search function needs to deduplicate by URL so Gemini does not process the same page twice later.
- In profiler.py, add the _search_person() function below _generate_search_queries():
def _search_person(name: str) -> list[dict]:
queries = _generate_search_queries(name)
seen_urls = set()
results = []
# Run each query through DuckDuckGo and collect unique results
ddgs = DDGS()
for query in queries:
try:
search_results = ddgs.text(
query,
region='wt-wt',
safesearch='moderate',
max_results=config.MAX_SEARCH_RESULTS,
)
for r in search_results:
url = r.get('href', '')
# Skip URLs already found by earlier queries
if url and url not in seen_urls:
seen_urls.add(url)
results.append({
'title': r.get('title', ''),
'url': url,
'snippet': r.get('body', ''),
'query': query,
})
except Exception as e:
logger.warning(f'Search failed for query "{query}": {e}')
return resultsWhat does this code do?
- seen_urls is a set that tracks every URL found so far. Before adding a result, the function checks if the URL is already in the set. This prevents duplicates across queries.
- ddgs.text() runs a DuckDuckGo search and returns a list of dictionaries. Each dictionary has title, href (the URL), and body (the snippet text).
- region='wt-wt' tells DuckDuckGo to return worldwide results instead of filtering by country.
- The try/except block catches network errors or rate limits on individual queries without crashing the whole search.
- Save profiler.py.
Time to test whether the search engine actually finds results.
Test the search engine
You can test the search functions right now without the scraper or Gemini. A quick terminal command lets you verify that queries are generated and results come back.
- Make sure your terminal is in the larpAI root directory (the folder that contains person_profiler/).
- Verify that the query generator works by running this command:
python -c "from person_profiler.profiler import _generate_search_queries; [print(q) for q in _generate_search_queries('Tim Cook')]"You should see four search queries printed, each targeting a different angle: LinkedIn, professional background, Twitter/X, and company or role keywords.
- Run the full search function to test live results by running this command:
python -c "from person_profiler.profiler import _search_person; results = _search_person('Tim Cook'); print(f'Found {len(results)} unique results')"You should see a message like Found 20 unique results (the exact number varies). Any number above zero means your search engine is working.
Getting zero results or an error?
- If you see ModuleNotFoundError: No module named 'ddgs', the ddgs package is not installed. Run pip install ddgs and try again.
- If you see ModuleNotFoundError: No module named 'person_profiler', your terminal is not in the larpAI root directory. Run cd to navigate to the folder that contains the person_profiler/ directory.
- If results come back as zero, DuckDuckGo may be rate-limiting your requests. Wait a minute and try again.
- Why is my search returning zero results?
Why does LinkedIn still show up even though it blocks scraping?
LinkedIn blocks direct page scraping, but DuckDuckGo search results include LinkedIn snippets with the person's title and company. You get useful LinkedIn data from the search snippet itself without ever visiting the page. The scraper in Step 3 will gracefully skip any URL it cannot access.
Awesome, I've got everything!
Great. Double check you have saved profiler.py before moving on.
I'd like to double check the full code
Here is the complete profiler.py so far. Make sure yours matches:
import logging
from ddgs import DDGS
from . import profiler_config as config
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def _generate_search_queries(name: str) -> list[str]:
queries = [
f'{name} LinkedIn',
f'{name} professional background',
f'{name} site:twitter.com OR site:x.com',
f'"{name}" company OR title OR role',
]
return queries
def _search_person(name: str) -> list[dict]:
queries = _generate_search_queries(name)
seen_urls = set()
results = []
ddgs = DDGS()
for query in queries:
try:
search_results = ddgs.text(
query,
region='wt-wt',
safesearch='moderate',
max_results=config.MAX_SEARCH_RESULTS,
)
for r in search_results:
url = r.get('href', '')
if url and url not in seen_urls:
seen_urls.add(url)
results.append({
'title': r.get('title', ''),
'url': url,
'snippet': r.get('body', ''),
'query': query,
})
except Exception as e:
logger.warning(f'Search failed for query "{query}": {e}')
return resultsYour search engine is pulling in results from across the web. Next up, you will build the scraper that visits those URLs and extracts the actual text content from each page.
Build the Web Scraper
Your search engine is returning URLs, titles, and snippets for any name you throw at it. But snippets only give you a preview of what each page contains.
To build a complete profile, you need to visit those pages and extract the full text. In this step, you will build a web scraper that fetches each URL, strips away HTML noise like navigation bars and scripts, and returns clean text ready for Gemini.
In this step, get ready to:
- Build the _scrape_url() function to fetch and parse web pages.
- Build the _scrape_results() function to scrape all search results.
- Add content truncation to protect Gemini's context window.
Fetch and parse web pages with _scrape_url()
Raw HTML pages contain way more than just useful text. They include <script> tags, <style> blocks, navigation bars, footers, and sidebars. You need a function that fetches a page and strips it down to just the meaningful content.
- In profiler.py, add these two imports below your existing import lines at the top of the file:
import httpx
from bs4 import BeautifulSoup- Then, below the _search_person() function, add the _scrape_url() function by copying this code:
def _scrape_url(url: str) -> str:
headers = {'User-Agent': config.USER_AGENT}
try:
# Fetch the page with a timeout and automatic redirect following
with httpx.Client(timeout=config.SCRAPE_TIMEOUT, follow_redirects=True) as client:
response = client.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
# Remove noisy HTML elements that don't contain useful content
for tag in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
tag.decompose()
# Extract clean text and collapse extra whitespace
text = soup.get_text(separator=' ', strip=True)
text = ' '.join(text.split())
# Truncate to keep Gemini's context manageable
return text[:config.MAX_CONTENT_LENGTH]
except Exception as e:
logger.warning(f'Failed to scrape {url}: {e}')
return ''What does this code do?
- httpx.Client fetches the page with a timeout (so your scraper does not hang on slow sites) and a User-Agent header that mimics a real browser. Many websites block requests without a user agent.
- BeautifulSoup parses the raw HTML. The decompose() call permanently removes noisy elements like <script>, <nav>, and <footer> from the page before extracting text.
- get_text(separator=' ', strip=True) pulls all remaining visible text out of the HTML, separated by spaces with leading/trailing whitespace removed.
- The text[:config.MAX_CONTENT_LENGTH] truncation keeps each page's text under 3000 characters. You will explore why this matters in a moment.
- If anything goes wrong (timeout, 404, connection error), the function logs a warning and returns an empty string instead of crashing.
- Save profiler.py.
You will not see any output yet. The next function, _scrape_results(), uses _scrape_url() to scrape all search results at once.
Scrape all search results with _scrape_results()
Your _scrape_url() function handles one page at a time. You need an orchestrator function that loops through all search results, calls _scrape_url() on each one, and collects the successful results.
- In profiler.py, add the _scrape_results() function below _scrape_url():
def _scrape_results(search_results: list[dict]) -> list[dict]:
scraped = []
for result in search_results:
url = result['url']
logger.info(f'Scraping: {url}')
# Scrape the page and keep it only if content was extracted
content = _scrape_url(url)
if content:
scraped.append({
'url': url,
'title': result['title'],
'snippet': result['snippet'],
'content': content,
})
return scrapedWhat does this code do?
- The function loops through every search result and calls _scrape_url() on each URL. It logs each URL as it goes so you can track progress in the terminal.
- The if content: check quietly skips any page that failed to scrape. Since _scrape_url() returns an empty string on failure, only successfully scraped pages make it into the final list.
- Each successful result bundles the URL, title, snippet, and full scraped content together. This gives Gemini everything it needs to synthesize a profile in the next step.
- Save profiler.py.
You will not see output changes yet. You will test the full pipeline in Step 5 after wiring everything together.
Seeing a red underline on the imports?
- Make sure you installed httpx and beautifulsoup4 in Step 1. If not, run pip install httpx beautifulsoup4 in your terminal.
- Check that your import says from bs4 import BeautifulSoup. The package is installed as beautifulsoup4 but imported as bs4.
- Help me fix import errors in profiler.py
Keep scraped content under Gemini's limit
You already added a truncation line in _scrape_url(). This line is short but important to understand.
- Scroll to the _scrape_url() function and find this line near the end:
return text[:config.MAX_CONTENT_LENGTH]Why truncate to 3000 characters?
Your scraper can return up to 8 pages of content (set by MAX_SEARCH_RESULTS in profiler_config.py). Without truncation, a single web page could be tens of thousands of characters long.
By capping each page at 3000 characters, the worst case is about 24K characters total (8 pages times 3000). This is well within Gemini 2.5 Flash's context limit while keeping responses fast and focused.
Most of a person's key details (job title, company, bio) appear in the first few paragraphs of a page. The tail end is usually boilerplate, related links, and ads. Truncating cuts the noise without losing the signal.
Why does LinkedIn still work if it blocks scraping?
LinkedIn blocks most direct scraping attempts. Your scraper does not try to scrape LinkedIn pages directly. Instead, the DDGS search results include LinkedIn snippets (job title, company, headline) as part of the search result text itself.
If _scrape_url() hits a LinkedIn page and gets blocked, it returns an empty string and moves on. The snippet from the search results still provides useful data for Gemini.
Awesome, I've got everything!
Double check you have saved profiler.py with both new functions and the new imports at the top.
I'd like to double check the full code
Here is what your profiler.py should look like so far, including the functions from the previous step.
import logging
import httpx
from bs4 import BeautifulSoup
from ddgs import DDGS
from . import profiler_config as config
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def _generate_search_queries(name: str) -> list[str]:
queries = [
f'{name} LinkedIn',
f'{name} professional background',
f'{name} site:twitter.com OR site:x.com',
f'"{ name}" company OR title OR role',
]
return queries
def _search_person(name: str) -> list[dict]:
queries = _generate_search_queries(name)
seen_urls = set()
results = []
ddgs = DDGS()
for query in queries:
try:
search_results = ddgs.text(
query,
region='wt-wt',
safesearch='moderate',
max_results=config.MAX_SEARCH_RESULTS,
)
for r in search_results:
url = r.get('href', '')
if url and url not in seen_urls:
seen_urls.add(url)
results.append({
'title': r.get('title', ''),
'url': url,
'snippet': r.get('body', ''),
'query': query,
})
except Exception as e:
logger.warning(f'Search failed for query "{query}": {e}')
return results
def _scrape_url(url: str) -> str:
headers = {'User-Agent': config.USER_AGENT}
try:
with httpx.Client(timeout=config.SCRAPE_TIMEOUT, follow_redirects=True) as client:
response = client.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
for tag in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
tag.decompose()
text = soup.get_text(separator=' ', strip=True)
text = ' '.join(text.split())
return text[:config.MAX_CONTENT_LENGTH]
except Exception as e:
logger.warning(f'Failed to scrape {url}: {e}')
return ''
def _scrape_results(search_results: list[dict]) -> list[dict]:
scraped = []
for result in search_results:
url = result['url']
logger.info(f'Scraping: {url}')
content = _scrape_url(url)
if content:
scraped.append({
'url': url,
'title': result['title'],
'snippet': result['snippet'],
'content': content,
})
return scrapedYour scraper is ready to fetch and clean web pages. Next up, you will build the Gemini-powered synthesizer that turns all this raw text into a structured person profile.
Build the Gemini Profile Synthesizer
Your scraper can now pull clean text from web pages. But raw text from a dozen different websites is messy and full of noise.
This is where Gemini comes in. You will define a structured dataclass for person profiles, then build a synthesis function that feeds your scraped data to Gemini 2.5 Flash and gets back a clean JSON profile.
In this step, get ready to:
- Create a PersonProfile dataclass that defines the structured output schema.
- Build a synthesis function that prompts Gemini to turn scraped text into a structured profile.
- Handle edge cases like empty search results and malformed JSON responses.
Create the PersonProfile dataclass
Before you can ask Gemini to produce a profile, you need a structure that defines what a profile looks like. A Python dataclass gives you a clean schema with default values, plus built-in conversion to dictionaries and JSON.
- In profiler.py, add these imports at the top of the file, above your existing imports:
import json
from dataclasses import dataclass, field, asdict
from google import genai
from google.genai import typesWhat are these imports for?
- json lets you parse Gemini's JSON response into a Python dictionary.
- dataclass, field, and asdict come from Python's standard library and let you define structured data with default values.
- google-genai is Google's new unified SDK for the Gemini API. It replaces the older google-generativeai package, which was deprecated in November 2025.
- Now add the PersonProfile dataclass below your logger setup and above the _generate_search_queries() function:
@dataclass
class PersonProfile:
name: str = ''
likely_job_title: str = ''
company: str = ''
education: str = ''
location: str = ''
social_links: list = field(default_factory=list)
bio_summary: str = ''
sources: list = field(default_factory=list)
def to_dict(self):
return asdict(self)
def to_json(self):
return json.dumps(self.to_dict(), indent=2)What does this code do?
- The @dataclass decorator automatically generates __init__ and other methods from the field definitions.
- Each field has a default value (empty string or empty list), so Gemini does not need to fill every field for the dataclass to work.
- field(default_factory=list) creates a fresh empty list for each instance. Without it, all instances would share the same list object.
- to_dict() and to_json() are convenience methods that let other parts of your larpAI system consume the profile as a dictionary or JSON string.
- Save profiler.py.
You will not see a visible change yet. The next substep uses this dataclass as the return type of the synthesis function.
Seeing an import error?
- Make sure you installed google-genai (the new SDK), not google-generativeai (the deprecated one). Both can coexist, but the import paths are different.
- Double check that the new imports are placed at the top of the file, above your existing function definitions.
- Help me fix my import error in profiler.py.
Build the synthesis function
The synthesis function is the core of the profiler. It takes the scraped text snippets, formats them into a prompt, sends them to Gemini 2.5 Flash, and parses the JSON response back into a PersonProfile.
- In profiler.py, add the _synthesize_profile() function below your _scrape_results() function:
def _synthesize_profile(name: str, scraped_data: list[dict]) -> PersonProfile:
if not scraped_data:
return PersonProfile(name=name, bio_summary='No information found online.')
# Build context from all scraped pages
context_parts = []
for i, item in enumerate(scraped_data, 1):
context_parts.append(
f'Source {i} ({item["url"]}):\n'
f'Title: {item["title"]}\n'
f'Content: {item["content"]}\n'
)
context = '\n---\n'.join(context_parts)
# Construct the prompt that tells Gemini exactly what to return
prompt = f"""You are an identity research assistant. Given the following web search results about a person named "{name}", synthesize a structured profile.
Return ONLY valid JSON with these fields:
- "name": the person's full name as found in sources
- "likely_job_title": their current or most recent job title
- "company": their current or most recent company/organization
- "education": their educational background (school and degree if available)
- "location": their city/region/country
- "social_links": a list of social media or professional profile URLs found
- "bio_summary": a 2-3 sentence summary of who this person is
If a field cannot be determined from the sources, use an empty string or empty list. Do not guess or fabricate information.
Web search results:
{context}"""
# Send to Gemini with JSON output mode enabled
client = genai.Client(api_key=config.GEMINI_API_KEY)
response = client.models.generate_content(
model=config.GEMINI_MODEL,
contents=prompt,
config=types.GenerateContentConfig(
response_mime_type='application/json',
thinking_config=types.ThinkingConfig(thinking_budget=0),
),
)
# Parse Gemini's JSON response into a PersonProfile
try:
profile_data = json.loads(response.text)
except json.JSONDecodeError:
logger.error('Gemini returned invalid JSON. Attempting to extract.')
return PersonProfile(name=name, bio_summary='Profile synthesis failed.')
sources = [item['url'] for item in scraped_data]
return PersonProfile(
name=profile_data.get('name', name),
likely_job_title=profile_data.get('likely_job_title', ''),
company=profile_data.get('company', ''),
education=profile_data.get('education', ''),
location=profile_data.get('location', ''),
social_links=profile_data.get('social_links', []),
bio_summary=profile_data.get('bio_summary', ''),
sources=sources,
)What does this function do?
This function has four stages:
- Early exit. If the scraper returned nothing, the function immediately returns a profile with just the name and a "no information found" message. No point calling Gemini with empty data.
- Context building. Each scraped page is formatted into a labeled block (source number, URL, title, content) and joined with --- separators so Gemini can tell sources apart.
- Gemini call. The prompt explicitly lists every JSON field Gemini should return. Setting response_mime_type='application/json' forces Gemini to return valid JSON instead of markdown. Setting thinking_budget=0 disables Gemini's thinking mode for faster responses.
- Parsing. The JSON string is parsed into a dictionary, then each field is extracted with .get() and safe defaults, so a missing field never crashes the program.
- Save profiler.py.
You will not see a change yet. The synthesis function is not called anywhere on its own. In the next substep, you will walk through the edge case handling to make sure it is solid.
Seeing a syntax error in the prompt string?
- The prompt uses a triple-quoted f-string (f"""..."""). Make sure you have three double-quotes at both the start and end.
- The curly braces around {name} and {context} are f-string placeholders. If you see literal {name} in the prompt, check that the f prefix is present before the opening triple quotes.
- Help me debug my synthesis function in profiler.py.
Walk through the edge case handling
The synthesis function you just wrote handles two important failure scenarios. Understanding these helps you debug issues when running the profiler on real names.
What happens when no results are found?
Look at the top of your _synthesize_profile() function. The first thing it checks is whether scraped_data is empty.
If the scraper could not fetch any content (every URL timed out or was blocked), the function skips the Gemini call entirely. It returns a bare PersonProfile with just the name and a fallback bio summary. This avoids wasting an API call on an empty prompt.
What happens when Gemini returns malformed JSON?
Even with response_mime_type='application/json' enabled, there is a small chance Gemini returns something that is not valid JSON.
The try/except json.JSONDecodeError block catches this. Instead of crashing, it logs the error and returns a profile with a "synthesis failed" message. This keeps the pipeline running even if one profile fails.
Why use .get() with defaults instead of direct key access?
The return statement uses profile_data.get('likely_job_title', '') instead of profile_data['likely_job_title']. If Gemini's JSON is missing a field, .get() returns the default instead of raising a KeyError. The profiler stays resilient even when Gemini omits fields.
Awesome, I've got everything!
Double check you have saved profiler.py with both the PersonProfile dataclass and the _synthesize_profile() function.
I'd like to double check the full code
Here is the full profiler.py as it should look at the end of this step. Compare it with your file to spot any differences.
import json
import logging
from dataclasses import dataclass, field, asdict
import httpx
from bs4 import BeautifulSoup
from ddgs import DDGS
from google import genai
from google.genai import types
from . import profiler_config as config
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@dataclass
class PersonProfile:
name: str = ''
likely_job_title: str = ''
company: str = ''
education: str = ''
location: str = ''
social_links: list = field(default_factory=list)
bio_summary: str = ''
sources: list = field(default_factory=list)
def to_dict(self):
return asdict(self)
def to_json(self):
return json.dumps(self.to_dict(), indent=2)
def _generate_search_queries(name: str) -> list[str]:
queries = [
f'{name} LinkedIn',
f'{name} professional background',
f'{name} site:twitter.com OR site:x.com',
f'"{ name}" company OR title OR role',
]
return queries
def _search_person(name: str) -> list[dict]:
queries = _generate_search_queries(name)
seen_urls = set()
results = []
ddgs = DDGS()
for query in queries:
try:
search_results = ddgs.text(
query,
region='wt-wt',
safesearch='moderate',
max_results=config.MAX_SEARCH_RESULTS,
)
for r in search_results:
url = r.get('href', '')
if url and url not in seen_urls:
seen_urls.add(url)
results.append({
'title': r.get('title', ''),
'url': url,
'snippet': r.get('body', ''),
'query': query,
})
except Exception as e:
logger.warning(f'Search failed for query "{query}": {e}')
return results
def _scrape_url(url: str) -> str:
headers = {'User-Agent': config.USER_AGENT}
try:
with httpx.Client(timeout=config.SCRAPE_TIMEOUT, follow_redirects=True) as client:
response = client.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
for tag in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
tag.decompose()
text = soup.get_text(separator=' ', strip=True)
text = ' '.join(text.split())
return text[:config.MAX_CONTENT_LENGTH]
except Exception as e:
logger.warning(f'Failed to scrape {url}: {e}')
return ''
def _scrape_results(search_results: list[dict]) -> list[dict]:
scraped = []
for result in search_results:
url = result['url']
logger.info(f'Scraping: {url}')
content = _scrape_url(url)
if content:
scraped.append({
'url': url,
'title': result['title'],
'snippet': result['snippet'],
'content': content,
})
return scraped
def _synthesize_profile(name: str, scraped_data: list[dict]) -> PersonProfile:
if not scraped_data:
return PersonProfile(name=name, bio_summary='No information found online.')
context_parts = []
for i, item in enumerate(scraped_data, 1):
context_parts.append(
f'Source {i} ({item["url"]}):\n'
f'Title: {item["title"]}\n'
f'Content: {item["content"]}\n'
)
context = '\n---\n'.join(context_parts)
prompt = f"""You are an identity research assistant. Given the following web search results about a person named "{name}", synthesize a structured profile.
Return ONLY valid JSON with these fields:
- "name": the person's full name as found in sources
- "likely_job_title": their current or most recent job title
- "company": their current or most recent company/organization
- "education": their educational background (school and degree if available)
- "location": their city/region/country
- "social_links": a list of social media or professional profile URLs found
- "bio_summary": a 2-3 sentence summary of who this person is
If a field cannot be determined from the sources, use an empty string or empty list. Do not guess or fabricate information.
Web search results:
{context}"""
client = genai.Client(api_key=config.GEMINI_API_KEY)
response = client.models.generate_content(
model=config.GEMINI_MODEL,
contents=prompt,
config=types.GenerateContentConfig(
response_mime_type='application/json',
thinking_config=types.ThinkingConfig(thinking_budget=0),
),
)
try:
profile_data = json.loads(response.text)
except json.JSONDecodeError:
logger.error('Gemini returned invalid JSON. Attempting to extract.')
return PersonProfile(name=name, bio_summary='Profile synthesis failed.')
sources = [item['url'] for item in scraped_data]
return PersonProfile(
name=profile_data.get('name', name),
likely_job_title=profile_data.get('likely_job_title', ''),
company=profile_data.get('company', ''),
education=profile_data.get('education', ''),
location=profile_data.get('location', ''),
social_links=profile_data.get('social_links', []),
bio_summary=profile_data.get('bio_summary', ''),
sources=sources,
)Your synthesizer is ready. In the next step, you will wire search, scrape, and synthesize together into a single profile_person() call and run the full pipeline end-to-end.
Wire It All Together and Test
You have all three stages of the pipeline built: searching, scraping, and synthesizing. Right now they exist as separate private functions inside profiler.py. The module needs a single public function that chains them together in the right order.
In this step, you'll create that public entry point, set up clean package exports so the rest of your larpAI project can import it easily, and build a test script to validate the full pipeline end-to-end.
In this step, get ready to:
- Create the profile_person() function that chains search, scrape, and synthesize into one call.
- Set up __init__.py for clean package imports.
- Build and run test_profiler.py to validate the module end-to-end.
Create the profile_person() function
All the private helper functions are prefixed with underscores (_search_person, _scrape_results, _synthesize_profile). The module needs one public function that the outside world calls.
- In profiler.py, scroll to the very bottom of the file (below the _synthesize_profile() function) and add the profile_person() function:
def profile_person(name: str) -> PersonProfile:
# Public entry point that chains the full pipeline
logger.info(f'Profiling: {name}')
# Stage 1: Run multi-query search
logger.info('Step 1: Searching...')
search_results = _search_person(name)
logger.info(f'Found {len(search_results)} unique results')
# Stage 2: Scrape content from each result
logger.info('Step 2: Scraping...')
scraped_data = _scrape_results(search_results)
logger.info(f'Successfully scraped {len(scraped_data)} pages')
# Stage 3: Send scraped data to Gemini for synthesis
logger.info('Step 3: Synthesizing profile...')
profile = _synthesize_profile(name, scraped_data)
logger.info('Profile complete.')
return profileWhat does this code do?
- profile_person() is the only function without an underscore prefix. This is the public API that other modules in your larpAI project will call.
- It calls _search_person(), _scrape_results(), and _synthesize_profile() in sequence. The output of each stage feeds into the next.
- The logger.info() calls print progress updates so you can see what the module is doing while it runs. This is especially helpful because scraping multiple pages can take a few seconds.
- Save profiler.py.
- You won't see a change yet. The next two substeps set up the package exports and a test script to run this function.
Awesome, I've got everything!
Double check you've saved profiler.py. This file is now complete.
I'd like to double check the full code
Here is the complete profiler.py file with every function you've built across Steps 2 through 5.
import json
import logging
from dataclasses import dataclass, field, asdict
import httpx
from bs4 import BeautifulSoup
from ddgs import DDGS
from google import genai
from google.genai import types
from . import profiler_config as config
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@dataclass
class PersonProfile:
name: str = ''
likely_job_title: str = ''
company: str = ''
education: str = ''
location: str = ''
social_links: list = field(default_factory=list)
bio_summary: str = ''
sources: list = field(default_factory=list)
def to_dict(self):
return asdict(self)
def to_json(self):
return json.dumps(self.to_dict(), indent=2)
def _generate_search_queries(name: str) -> list[str]:
queries = [
f'{name} LinkedIn',
f'{name} professional background',
f'{name} site:twitter.com OR site:x.com',
f'"{name}" company OR title OR role',
]
return queries
def _search_person(name: str) -> list[dict]:
queries = _generate_search_queries(name)
seen_urls = set()
results = []
ddgs = DDGS()
for query in queries:
try:
search_results = ddgs.text(
query,
region='wt-wt',
safesearch='moderate',
max_results=config.MAX_SEARCH_RESULTS,
)
for r in search_results:
url = r.get('href', '')
if url and url not in seen_urls:
seen_urls.add(url)
results.append({
'title': r.get('title', ''),
'url': url,
'snippet': r.get('body', ''),
'query': query,
})
except Exception as e:
logger.warning(f'Search failed for query "{query}": {e}')
return results
def _scrape_url(url: str) -> str:
headers = {'User-Agent': config.USER_AGENT}
try:
with httpx.Client(timeout=config.SCRAPE_TIMEOUT, follow_redirects=True) as client:
response = client.get(url, headers=headers)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
for tag in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
tag.decompose()
text = soup.get_text(separator=' ', strip=True)
text = ' '.join(text.split())
return text[:config.MAX_CONTENT_LENGTH]
except Exception as e:
logger.warning(f'Failed to scrape {url}: {e}')
return ''
def _scrape_results(search_results: list[dict]) -> list[dict]:
scraped = []
for result in search_results:
url = result['url']
logger.info(f'Scraping: {url}')
content = _scrape_url(url)
if content:
scraped.append({
'url': url,
'title': result['title'],
'snippet': result['snippet'],
'content': content,
})
return scraped
def _synthesize_profile(name: str, scraped_data: list[dict]) -> PersonProfile:
if not scraped_data:
return PersonProfile(name=name, bio_summary='No information found online.')
context_parts = []
for i, item in enumerate(scraped_data, 1):
context_parts.append(
f'Source {i} ({item["url"]}):\n'
f'Title: {item["title"]}\n'
f'Content: {item["content"]}\n'
)
context = '\n---\n'.join(context_parts)
prompt = f"""You are an identity research assistant. Given the following web search results about a person named "{name}", synthesize a structured profile.
Return ONLY valid JSON with these fields:
- "name": the person's full name as found in sources
- "likely_job_title": their current or most recent job title
- "company": their current or most recent company/organization
- "education": their educational background (school and degree if available)
- "location": their city/region/country
- "social_links": a list of social media or professional profile URLs found
- "bio_summary": a 2-3 sentence summary of who this person is
If a field cannot be determined from the sources, use an empty string or empty list. Do not guess or fabricate information.
Web search results:
{context}"""
client = genai.Client(api_key=config.GEMINI_API_KEY)
response = client.models.generate_content(
model=config.GEMINI_MODEL,
contents=prompt,
config=types.GenerateContentConfig(
response_mime_type='application/json',
thinking_config=types.ThinkingConfig(thinking_budget=0),
),
)
try:
profile_data = json.loads(response.text)
except json.JSONDecodeError:
logger.error('Gemini returned invalid JSON. Attempting to extract.')
return PersonProfile(name=name, bio_summary='Profile synthesis failed.')
sources = [item['url'] for item in scraped_data]
return PersonProfile(
name=profile_data.get('name', name),
likely_job_title=profile_data.get('likely_job_title', ''),
company=profile_data.get('company', ''),
education=profile_data.get('education', ''),
location=profile_data.get('location', ''),
social_links=profile_data.get('social_links', []),
bio_summary=profile_data.get('bio_summary', ''),
sources=sources,
)
def profile_person(name: str) -> PersonProfile:
logger.info(f'Profiling: {name}')
logger.info('Step 1: Searching...')
search_results = _search_person(name)
logger.info(f'Found {len(search_results)} unique results')
logger.info('Step 2: Scraping...')
scraped_data = _scrape_results(search_results)
logger.info(f'Successfully scraped {len(scraped_data)} pages')
logger.info('Step 3: Synthesizing profile...')
profile = _synthesize_profile(name, scraped_data)
logger.info('Profile complete.')
return profileWire up __init__.py for clean imports
Python treats a folder as a package when it contains an __init__.py file. You created this file empty in Step 1. Now you'll add exports so that other modules in larpAI can import the profiler with a clean one-liner.
- Open __init__.py inside the person_profiler/ folder and add the following code:
# Re-export the public API so callers can do:
# from person_profiler import profile_person, PersonProfile
from .profiler import profile_person, PersonProfile
__all__ = ['profile_person', 'PersonProfile']What does this code do?
- from .profiler import ... pulls profile_person and PersonProfile up to the package level. Without this, callers would have to write from person_profiler.profiler import profile_person instead of the cleaner from person_profiler import profile_person.
- __all__ defines what gets exported when someone writes from person_profiler import *. It keeps the public API explicit.
- Save __init__.py.
- You won't see a change yet. The test script in the next substep will confirm these imports work.
Build the test script and run the pipeline
The module is complete, but you need a way to run it from the command line and see the output. The test script handles path setup, loads your .env file, and pretty-prints the profile.
- Open test_profiler.py inside the person_profiler/ folder and add the path setup and imports:
import sys
import os
# Add the larpAI root to Python's path so we can import person_profiler
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# Load the .env file from the larpAI root to get GEMINI_API_KEY
from dotenv import load_dotenv
load_dotenv(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), '.env'))
from person_profiler.profiler import profile_personWhy does the test script need all this path setup?
When you run the script with python -m person_profiler.test_profiler, Python needs to know where the person_profiler package lives. sys.path.insert() adds the larpAI root directory to Python's search path.
The load_dotenv() call explicitly loads the .env file from the larpAI root. This ensures GEMINI_API_KEY is available as an environment variable before the profiler module tries to read it.
- Below the imports, add the main() function and the entry point guard:
def main():
# Check that the user provided a name argument
if len(sys.argv) < 2:
print('Usage: python -m person_profiler.test_profiler "Person Name"')
sys.exit(1)
# Join all arguments into a single name string
name = ' '.join(sys.argv[1:])
print(f'\nSearching for: {name}')
print('=' * 50)
# Run the full pipeline
profile = profile_person(name)
# Pretty-print the profile
print(f'\n{"=" * 50}')
print(f'PROFILE: {profile.name}')
print(f'{"=" * 50}')
print(f'Job Title : {profile.likely_job_title}')
print(f'Company : {profile.company}')
print(f'Education : {profile.education}')
print(f'Location : {profile.location}')
print(f'\nBio:')
print(f' {profile.bio_summary}')
print(f'\nSocial Links:')
for link in profile.social_links:
print(f' - {link}')
print(f'\nSources ({len(profile.sources)}):')
for src in profile.sources:
print(f' - {src}')
print(f'\nJSON Output:')
print(profile.to_json())
if __name__ == '__main__':
main()What does this code do?
- The argument check at the top prevents the script from running without a name. It prints usage instructions if no name is provided.
- ' '.join(sys.argv[1:]) handles multi-word names like "Joy Xia". Without it, the script would only capture the first word.
- The pretty-print section displays each profile field on its own line, then dumps the raw JSON output at the end. The JSON is what other larpAI modules would consume programmatically.
- Save test_profiler.py.
Awesome, I've got everything!
Double check you've saved test_profiler.py. Time to run it.
I'd like to double check the full code
Here is the complete test_profiler.py file.
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
load_dotenv(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), '.env'))
from person_profiler.profiler import profile_person
def main():
if len(sys.argv) < 2:
print('Usage: python -m person_profiler.test_profiler "Person Name"')
sys.exit(1)
name = ' '.join(sys.argv[1:])
print(f'\nSearching for: {name}')
print('=' * 50)
profile = profile_person(name)
print(f'\n{"=" * 50}')
print(f'PROFILE: {profile.name}')
print(f'{"=" * 50}')
print(f'Job Title : {profile.likely_job_title}')
print(f'Company : {profile.company}')
print(f'Education : {profile.education}')
print(f'Location : {profile.location}')
print(f'\nBio:')
print(f' {profile.bio_summary}')
print(f'\nSocial Links:')
for link in profile.social_links:
print(f' - {link}')
print(f'\nSources ({len(profile.sources)}):')
for src in profile.sources:
print(f' - {src}')
print(f'\nJSON Output:')
print(profile.to_json())
if __name__ == '__main__':
main()Now run the pipeline. Make sure your terminal is in the larpAI root directory (the folder that contains person_profiler/).
- Run the test script by passing a well-known public figure's name as the argument:
python -m person_profiler.test_profiler "Satya Nadella"The script takes 10 to 30 seconds to complete. You'll see logging output as each stage runs: searching, scraping individual URLs, then synthesizing.
What should I see?
Your terminal should print a structured profile that looks something like this:
==================================================
PROFILE: Satya Nadella
==================================================
Job Title : Chairman and CEO
Company : Microsoft
Education : University of Wisconsin-Milwaukee, University of Chicago Booth
Location : Redmond, Washington
Bio:
Satya Nadella is the Chairman and CEO of Microsoft...
Social Links:
- https://twitter.com/sataborella
- https://linkedin.com/in/satyanadella
Sources (6):
- https://...
JSON Output:
{
"name": "Satya Nadella",
...The exact values will vary depending on what search results and pages are available at the time you run it. The important thing is that each field is populated and the JSON output appears at the bottom.
Seeing an error?
- ModuleNotFoundError: No module named 'person_profiler' means your terminal is not in the larpAI root directory. Run ls and confirm you can see the person_profiler/ folder in the output.
- ModuleNotFoundError: No module named 'dotenv' means python-dotenv is not installed. Your larpAI project likely already has it, but you can install it with pip install python-dotenv.
- If Gemini returns an API key error, check that your .env file in the larpAI root contains GEMINI_API_KEY=your-actual-key.
- If most fields are empty, try a more well-known public figure. Obscure names may not have enough web presence for DDGS to find useful results.
Help me debug my person profiler test run
Pro tip: try your own name
Once you've confirmed the pipeline works with a well-known figure, try running it with your own name. It's a fun way to see what your public web presence looks like through the lens of search results and AI synthesis.
Secret mission
Add Confidence Scoring
Your profiler builds a structured profile, but how reliable is each field? When Gemini pulls a job title confirmed by five different sources, that is solid ground. When it finds a location mentioned once on a questionable blog, not so much. Build a confidence scoring system that rates every profile field based on source agreement.
Before you go
Clean Up Your Resources
This project runs entirely locally and uses free services. DDGS requires no API key, and Gemini 2.5 Flash has a generous free tier. There are no ongoing costs to worry about.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- person_profiler/ directory with 4 Python files (__init__.py, profiler_config.py, profiler.py, test_profiler.py).
- Installed Python packages: ddgs, httpx, beautifulsoup4, google-genai.
- Gemini API key (already existed in your .env file, shared with the rest of larpAI).
Keep everything running
No action needed. Choose this if you are still actively building on top of the person profiler or plan to integrate it further into your larpAI project.
- Your person_profiler/ package stays in your larpAI directory, ready to use.
- The installed packages remain available in your Python environment.
- There are no running processes or cloud resources to manage. Everything is on-demand.
Pause - I'll come back to this later
Since this project is entirely local with no running processes, there is nothing to pause. Your files and packages stay where they are until you come back.
- The person_profiler/ directory is self-contained. It will work exactly the same when you return.
- No cloud resources are running, so there are no costs while you are away.
Delete - I don't want to use this again
Remove all project resources and start fresh. This will not affect any other larpAI files.
- Delete the person_profiler/ directory from your larpAI project root.
- Optionally, uninstall the Python packages you installed for this project:
pip uninstall ddgs httpx beautifulsoup4 google-genai- Your .env file and Gemini API key are shared with the rest of your larpAI project. Do not delete them unless you are removing the entire project.
Mission accomplished
Nice Work!
You just built a fully functional person profiler that turns a name into a structured, source-backed profile. The person_profiler package is ready to plug into your larpAI system without a single change to any existing file.
You've learned how to:
- Build a multi-query search engine that optimizes web coverage across LinkedIn, professional sites, and social media for a given person's name.
- Create a web scraper with httpx and BeautifulSoup that extracts clean text from search results and handles failures gracefully.
- Integrate Gemini 2.5 Flash to synthesize raw web data into structured JSON output using a Python dataclass.
- Added confidence scoring to rate how reliable each profile field is based on source agreement.
Ready to quiz yourself?