Reconnaissance is the first phase of any penetration test, and the quality of your recon directly determines the quality of what comes after. theHarvester is a dedicated OSINT tool designed to pull email addresses, subdomains, IP addresses, and employee names from public sources without touching the target directly. It’s fast, scriptable, and works against dozens of data sources.
What theHarvester Does
theHarvester is a Python tool that queries public search engines, certificate transparency logs, DNS datasets, and threat intelligence APIs to build a profile of a target domain. Given a domain like example.com, it can return:
- Employee email addresses
- Subdomains and associated IP addresses
- Virtual hosts sharing an IP
- Open ports (via Shodan)
- URLs indexed in search engines
It does all of this passively — no packets hit the target’s infrastructure.
Installation
theHarvester ships with Kali Linux by default. On other systems:
git clone https://github.com/laramies/theHarvester.git
cd theHarvester
pip3 install -r requirements/base.txt
# Verify
python3 theHarvester.py -h
Basic Syntax
python3 theHarvester.py -d <domain> -b <source> [options]
| Flag | Meaning |
|---|
-d | Target domain |
-b | Data source(s) |
-l | Limit results (default 500) |
-f | Save output to HTML/XML file |
-v | Enable verbose mode |
-s | Start result offset |
Data Sources
theHarvester supports over 40 data sources. The most useful:
Hunter.io
Hunter aggregates professional email addresses scraped from the public web. You need a free API key from hunter.io.
# Add key to ~/.theHarvester/api-keys.yaml
hunter:
key: YOUR_HUNTER_API_KEY
python3 theHarvester.py -d targetcorp.com -b hunter
Hunter often yields formatted email patterns (e.g., [email protected]) that let you extrapolate addresses for employees you find on LinkedIn.
Bing
Bing’s web search is useful for finding indexed pages, subdomains in links, and cached email addresses. No API key required.
python3 theHarvester.py -d targetcorp.com -b bing -l 300
Use the site: operator logic internally — Bing returns pages indexed under the target domain.
Shodan
Shodan scans the internet continuously and indexes banners, ports, and certificates. theHarvester uses Shodan to resolve subdomains and identify open services. Requires a Shodan API key.
shodan:
key: YOUR_SHODAN_API_KEY
python3 theHarvester.py -d targetcorp.com -b shodan
Shodan results often reveal internal-looking hosts that were accidentally exposed, staging servers, and legacy infrastructure.
Certificate Transparency (crtsh)
SSL certificates are public record. crt.sh aggregates them from all major CA logs, making it one of the best subdomain discovery sources.
python3 theHarvester.py -d targetcorp.com -b crtsh
This regularly uncovers subdomains like vpn.targetcorp.com, staging-api.targetcorp.com, or admin.targetcorp.com that don’t appear in DNS brute force dictionaries.
Running Multiple Sources at Once
python3 theHarvester.py -d targetcorp.com -b bing,crtsh,hunter,shodan -f output_report
This generates output_report.html and output_report.xml in the current directory.
Interpreting the Output
A typical run produces sections like:
[*] Emails found: 14
------------------
[email protected]
[email protected]
[email protected]
[*] Hosts found: 31
-------------------
api.targetcorp.com:203.0.113.45
staging.targetcorp.com:203.0.113.12
vpn.targetcorp.com:198.51.100.88
What to look for:
- Email patterns: If you see
j.smith and john.smith, the company may use both formats. Test both.
- Staging/dev subdomains: These often run older software or have weaker security controls.
- IP clustering: Multiple subdomains on the same IP may share a web server — a vulnerability in one may affect others.
- Third-party IPs: A subdomain pointing to an AWS or Heroku IP may indicate a cloud asset that’s part of the attack surface.
theHarvester is a starting point, not an endpoint. Chain it with:
Subfinder + httpx
subfinder -d targetcorp.com -silent | httpx -silent -status-code
Cross-reference subfinder’s DNS brute-force results with theHarvester’s passive results to build a complete subdomain list.
Amass
amass enum -passive -d targetcorp.com
Amass integrates many of the same sources plus BGP data and WHOIS history. Comparing its output with theHarvester’s reveals gaps in each.
Nmap on Discovered Hosts
Once you have a list of IPs:
nmap -iL hosts.txt -sV -p 80,443,8080,8443 -oN web_services.txt
Email Validation
Feed discovered emails into emailverify or use Hunter’s verification API to confirm which addresses are live before using them in phishing simulations.
Automating Recon
For repeated assessments, wrap theHarvester in a shell script:
#!/bin/bash
DOMAIN=$1
mkdir -p recon/$DOMAIN
python3 /opt/theHarvester/theHarvester.py \
-d $DOMAIN \
-b bing,crtsh,hunter,shodan \
-f recon/$DOMAIN/harvest_$(date +%Y%m%d)
echo "[+] Done. Results in recon/$DOMAIN/"
Staying Within Scope
theHarvester operates passively, but the data it collects can guide active testing. Always verify that any subdomain or IP you intend to test falls within your authorized scope. A staging server discovered via certificate transparency is still someone’s production infrastructure.
With theHarvester as your recon foundation, the rest of the engagement starts with real intelligence rather than guesswork.