Skip to content

Case study · Personal project

Automated job-market intelligence pipeline

A scheduled Python pipeline that polls roughly 15,800 job-board APIs a day, normalizes about 260,000 postings into one schema, and scores each one on two axes: fit to a skills profile and how AI-forward the employer is.

Period
2026
Role
Designer · sole developer
Results

15,800

Greenhouse, Lever, and Ashby boards polled daily

260K

postings normalized per run

2

scoring axes: role fit · AI-forward

Figures are those published in my resume. Architecture is described at the pattern level: no confidential configuration, pricing logic, or customer data appears on this site.

Job boards are built for browsing, not for a structured search across thousands of employers with a specific, two-part question: does this role fit me, and does this company actually build with AI or just say so?

The three big applicant-tracking systems expose public JSON APIs for every company that uses them, but each has its own schema, pagination, and quirks, and the volume is far past what a person can read.

  1. Enumerated the public job-board endpoints for Greenhouse, Lever, and Ashby plus a remote-jobs feed, and wrote a poller that walks all of them daily with retries and rate awareness, then normalizes every posting into one schema.

  2. Filtered on hard gates first (remote eligibility, compensation, years of experience) so the expensive scoring only runs on postings that could matter, and deduplicated against a persistent ledger so a role is scored once.

  3. Designed a two-axis scoring model. Role fit matches requirement bullets against a skills profile built from my resume. The AI-forward score measures the density of distinct LLM, agent, MCP, and API-tooling language in the posting, kept separate so it cannot inflate fit.

  4. Built the whole thing with an LLM coding agent and MCP connectors, then scheduled it as a daily Claude Code task that writes a ranked report with both views: best fit, and the AI-forward subset.

A day in the pipeline
  1. Poll 15,862 Public Greenhouse, Lever, and Ashby boards, with retries and rate awareness.
  2. Normalize 263K Postings from three schemas and a remote-jobs feed into one.
  3. Hard gates Remote eligibility, compensation, years of experience. gate
  4. Dedupe Against a persistent ledger, so a role is scored once.
  5. Score Role fit against a skills profile; AI-forward language density, kept separate.
  6. Report 1,350 Matches tracked all time; a ranked report lands each morning.

Counts from the November 2026 sweep log. The 263K postings include 260,911 from ATS boards and 2,097 from the remote-jobs feed.

A daily ranked report over the entire public job market in these three systems, with the AI-forward employers surfaced separately, instead of hours of browsing.

The skills profile and scoring weights live in code, so changing what I am looking for is a one-line change that re-ranks the whole market the next morning.

Public job boards polled daily, by applicant-tracking system
Public job boards polled daily, by applicant-tracking system: Greenhouse 8,333; Lever 4,368; Ashby 3,161. 8,333 · Greenhouse Greenhouse 53% 8,333 4,368 · Lever Lever 28% 4,368 3,161 · Ashby Ashby 20% 3,161

15,862 boards in total, each a public JSON endpoint with its own schema and pagination.

Table view
CategoryBoards
Greenhouse8,333
Lever4,368
Ashby3,161

Where it sits

The highlighted nodes are the systems and people this project touched.

issues · intent installs workflow daily runs dispatches waves tool calls branches · PRs DNS · KV · Pages deploy on merge sites · APIs Me, as operator issues in · merges out Skills & plugins my own plugin marketplace Claude Code CLI · desktop · scheduled Scheduled tasks daily pipelines · reports Epic manager · task workers parallel subagents · worktrees MCP connectors Cloudflare · GitHub · Stripe · Semrush GitHub issues · PRs · Actions Cloudflare edge Pages · Functions · D1 · KV · Access Clients & visitors nine production sites