All Projects
Personal · Job Search Automation

US IT Companies Scraper

US IT Companies Scraper builds a prospect list for backend-engineer job search in the US. It starts from a hand-curated database of 80+ employers (FAANG, cloud, fintech, AI) with salary metadata, filters by backend focus and a configurable minimum salary floor, then enriches each company by scraping career links and regex-matching career emails from homepages. Results export to CSV and Excel via pandas; an optional SMTP outreach module sends templated applications. It complements AI Job Hunter - this tool finds who to target; AI Job Hunter discovers and scores live postings.

Problem

Job search for US backend roles often starts with scattered company lists and manual career-page hunting. AI Job Hunter scores live ATS postings, but you still need a curated employer prospect list with career emails, salary bands, and backend hiring signals before outreach begins.

Before and After

Before

  • Manual spreadsheet of target companies
  • Career emails hunted page by page
  • No salary floor or backend keyword filter
  • Single export format for mail merge
  • Outreach disconnected from prospect data

After

  • 80+ curated US tech employers with metadata
  • Automated career link and email extraction
  • Backend focus + $140k minimum salary gate
  • CSV and Excel export in one batch run
  • SMTP outreach module reads the same export

How Company Leads Get Enriched

A curated seed list of high-paying US tech employers flows through salary and backend filters, per-company website enrichment, and dual CSV/Excel export - with optional SMTP outreach downstream. Pick any block to see the flow.

Input

Orchestration

Seed

Filter

Scrape

Enrich

Export

Outreach

What I Built

Curated company seed database

  • 80+ US employers across FAANG, cloud, fintech, AI, gaming, and e-commerce.
  • Each entry includes website, industry, avg_salary range, and backend_jobs flag.
  • Hand-maintained list beats brittle live discovery for reliability.
  • Extendable Python list - add companies without changing scrape logic.

Dual-mode web extraction

  • Fast path: requests + BeautifulSoup for company homepages (10s timeout).
  • Optional Selenium headless Chrome for JavaScript-heavy Glassdoor browse pages.
  • Rotating User-Agent headers via fake-useragent.
  • Random 1-3s delays between company enrichments.

Career email and backend signal enrichment

  • Eight regex patterns: careers@, jobs@, hr@, talent@, recruiting@, and more.
  • Ten backend keyword scans: platform engineer, microservices, API engineer, etc.
  • Domain fallback emails when none found on the page (careers@domain, jobs@domain).
  • Career link discovery from href and anchor text matching.

Export and outreach workflow

  • pandas exports high_paying_backend_companies.csv and .xlsx in one run.
  • Interactive CLI (run_scraper.py) configures batch size, salary floor, and scrape toggle.
  • JobApplicationEmailer sends templated SMTP applications with rate limiting.
  • Complements AI Job Hunter: prospect list here, live posting discovery there.

Engineering Decisions

  • Curated seed over live job-board crawl - metadata quality and rate-limit safety beat scraping LinkedIn without an API.
  • requests first, Selenium second - static HTML covers most career pages; browser automation reserved for JS-heavy paths.
  • Graceful degradation - per-URL try/except, salary parse failures still include the company, missing emails get domain fallbacks.
  • Separate outreach script - scrape core stays batch-only; email sending is opt-in downstream.
  • Ethical scraping defaults - documented robots.txt guidance, delays, and CAN-SPAM-conscious outreach pacing.
  • LinkedIn path documented as placeholder - real integration would need official API or licensed data provider.

By the Numbers

Curated employers

80+

Email regex patterns

8

Backend keyword patterns

10

Default salary floor

$140k+

Python source lines

~900

Export formats

CSV + Excel

Industry categories

8+

Reliability and Ops

  • Per-company try/except - one failed homepage does not abort the batch.
  • Session reuse with consistent headers across requests in a run.
  • Keyboard interrupt handled in CLI runner.
  • setup.py automates ChromeDriver checks for the optional Selenium path.

Testing

  • Manual batch validation via run_scraper.py summary output.
  • Sample CSV output (high_paying_backend_companies.csv) for spot-checking enrichment quality.
  • Natural next step: unit tests for email extraction, salary parsing, and filter logic.

What I Did

  • 01Built a Python scraping pipeline enriching 80+ US tech companies with career links, contact emails, and salary metadata for backend-engineer targeting.
  • 02Implemented dual-mode extraction: requests + BeautifulSoup for static pages and headless Selenium for JS-heavy job board listings.
  • 03Designed regex-based career email discovery with domain fallbacks and backend keyword scanning across company homepages.
  • 04Added configurable filtering (salary floor, backend focus, batch size) with rate-limited scraping and rotating User-Agent headers.
  • 05Exported enriched datasets to CSV and Excel via pandas for downstream personalized outreach workflows.
  • 06Packaged interactive CLI, venv bootstrap script, and ChromeDriver setup automation for one-command local execution.
Next: HaatGet in Touch