I built a pipeline that generates over $100K a year in qualified leads for a regional bank. Finding leads used to take one person's whole day. The pipeline ran at 12:01 AM each night, read public records with AI, checked them against the bank's criteria, and had qualified leads ready by morning.
- 01Replaced a task that took one person all day with a pipeline that ran every night at 12:01 AM for a regional bank.
- 02Five stages, each logging on its own: AI extracts data from public records, and the bank's rules decide what counts as a lead.
- 03Over $100K a year in qualified leads, 100–300 a day at peak, and more than a year in production without manual work.
- 04Wrote the post-mortem on a 14-hour reprocessing loop and got the $800 charge refunded in full.
- 05Designed and built all of it: the schema, edge functions, batch worker, first dashboard, and the redesign in Figma.
Timeline
2025 → 2026 · Ran nightly for over a year
Role
Pipeline architect, engineer & designer
Technology
Next.js, Supabase, Gemini, Groq, DigitalOcean
Platform
Backend pipeline + client dashboard
$100K+
Annual lead value
0
Manual touches
1+ yr
Running nightly in production
The Outcome
From a daily manual task to a nightly job
One person used to spend every day visiting websites, copying records, and writing outreach emails by hand. The pipeline then ran every night for over a year without manual work.
Nightly run · 12:01 AMCompleted
- Annual lead valuedelivered to the client
- $100K+
- Qualified leads a dayat peak, by my records
- 100–300
- Pipeline tablesin Postgres
- 10
- Pipeline stageseach logged independently
- 5
- Schema migrationsevolved in production
- 49
- Manual interventionsend to end
- 0
A Night's Run
Watch a night's run
Every dot is a filing. They land at Ingest and wait for the batch worker, which lets at most 40 into the AI stages at once and pauses when a provider says to slow down. The bank's rules turn some away at Validate, the fingerprint skips anything already published, and the rest are leads by morning. Replay the bad deploy to see what the code still live during the post-mortem did instead.
Nightly run · 12:01 AM
12:01 AM · new filings arrive
Ingest
Edge function
OCR
Gemini
Validate
Groq · rules
Enrich
Maps
Publish
Fingerprint
Leads
By morning
An animation of one night's run: filings enter, queue for the batch worker, pass OCR and validation forty at a time, and either drop out at the rules or as duplicates, or become leads.
- Filings
- 0
- In flight
- 0/40
- Failed the rules
- 0
- Duplicates skipped
- 0
- Leads
- 0
The stages, the 40-call limit, the backoff, where records drop out, and the bad deploy's loop match the pipeline and its commits. The record counts and rejection rates are simulated, the worker's 500-record page is drawn as 120, and the run is sped up.
Architecture
How the pipeline is built
Five Supabase edge functions, each writing to its own table. Every insert triggers the next stage through a database webhook, and a separate worker paces the two stages that call AI models. Select any component to see what it does and a sketch of how it works.
Example run · 01/0912:01 AM: the nightly run starts and pulls in new filings.
Make.com · 12:01 AM
Nightly trigger
A scheduled Make.com scenario starts each night's run by pulling the day's new public business filings into the pipeline.
49 schema migrations over the project's life, including two that removed dangerous cascading deletes.
Design Principle
“AI extracts the data. The bank's rules make the decision.”
Gemini and Groq turn documents into structured data, and a separate validation stage applies the bank's criteria to it. Because of that split, I can change models or prompts without touching the rules, and a model error can't publish a bad lead on its own.
Post-mortem
Post-mortem: a 14-hour reprocessing loop
A late-night deploy failed without any error, and the pipeline reprocessed the same records for over 14 hours. This summarizes my incident write-up and the lessons I took from it.
14+ hrs
Reprocessing the same records
$800
In looping API calls
100%
Refunded after the write-up
Late night
The deploy didn't go out
Supabase's GitHub integration needs config.toml to redeploy, and I had assumed that file was only for local development. The old code kept running.
Overnight
The old code looped
The batch worker asks for the oldest records still marked pending, 500 at a time. The OCR function still deployed never marked a record done, so every pass returned the same 500, each was sent to Gemini again and wrote another OCR row, and nothing behind them was reached.
+14 hours
Root cause found
The cause was the missing deploy. I also found a Postgres query that scanned the whole table whenever the queue was empty.
The fix
Fixed
I committed config.toml, added explicit status columns for each queue state, and rewrote the query.
3–4 weeks
Full refund
I documented the incident and worked with Google support until the $800 charge was fully refunded.
Lessons from the incident
Verify every deploy
A deploy isn't finished until I've confirmed the new version is running.
Make work idempotent
Every record has an explicit queue state, so re-running a stage can't repeat finished work.
Rate-limit external calls
A loop like this one shouldn't be able to run up a large bill.
Designed for Compliance
Designing for consent
In the same design, each lead opened as a tabbed record, with the tabs in the order the bank reviews them. I used tabs instead of one long page so reviewers would look at one thing at a time.
Lead detail · Contact
The Dashboard
How the dashboard changed
The first version, a weekly progress tracker, was built and used. When the bank planned a direct-mail campaign with QR codes, I designed a campaign view in Figma that extended it: funnel metrics (mailers sent, QR codes scanned, questionnaires submitted) above a lead table the bank could filter and export. It was paid design work planned for the next build, but work on the project stopped before it was built. The design uses mock data, so no real leads are shown.
The Stack
Built by me, end to end
I designed and built all of it: the pipeline architecture, the database schema, the edge functions, the batch worker, the first dashboard, and the dashboard redesign in Figma.
Supabase
Postgres, edge functions
DigitalOcean
Batch orchestration
Gemini
Document OCR
G
Groq
Structured extraction
Next.js
Client dashboard
Zod
Schema validation
Make
Workflow automation
Figma
Dashboard design
Working Togetherexit 0
$ run complete · next job: yours
If your team has a manual process like this
Repetitive daily work like this can often be automated. I would take the same approach: separate stages so failures are easy to trace, AI limited to reading and extracting data, each workload run where it fits best, and ongoing support once it's in production.