All Projects
Open Source · Data Pipeline

Master Company Database

A master company database built from a ~396k-row Startup India CSV export. Raw records are normalized and deduplicated with union-find before landing in SQLite - the gate every downstream mailer should pull from, never the raw CSV. Enrichment runs in incremental phases: website discovery, crawl, HTML extraction, AI segmentation, contact discovery, email verification, and transparent lead scoring. GitHub Actions pushes rolling database versions to S3 every five minutes.

What I Did

  • 01Built Phases 1-3: field normalization, union-find deduplication with anti-over-merge guardrails, and SQLite master schema with full source traceability.
  • 02Implemented Phases 4-6 enrichment - HTTP probes, gzipped crawls, and structured extraction of emails, socials, tech fingerprints, and company facts.
  • 03Automated scheduled batch enrichment via GitHub Actions and S3 versioning with db_guard protection against truncated publishes.
Next: Docs PNG to JPGGet in Touch