I presented Label Buddy at GitLab's Other Interfaces in person in BLR

Process

Put the model last

In Label Buddy the model runs sixth of eight stages, on the smallest possible input. Hashing, dedup, cache, rules, and embeddings resolve the bulk for free, so the slow, costly, non-deterministic step only sees what genuinely needs judgment. Cost discipline is a design decision, not an afterthought.

Policy as product surface

Label Buddy's entire scoring behaviour lives in an editable markdown file. Thresholds, per-label strictness, overlap rules, adjacency map, Valid/Review/Invalid bands, prompt guidelines. A PM or ops person tunes the model in plain language, no code change. The product around a model has to be tunable by the people who own the outcome.

Results

Onboarding QA time

Full week Manual spreadsheet matching
~1 hour Automated pipeline run

Pipeline scale

~140k rows Per customer CSV — only LLM-resolved rows cost money
Semantic policy anyone can edit without touching code — status labels and strictness PMs own
Review 100k+ copy lines in under 10 minutes — analysis summary and detailed results in one pass
View scope, summary, and run costs in one surface — re-runs stay cheap when only revalidation is needed
40 hours to 1 hour monthly — shipped in a week on $200 Replit + Claude
Built-in docs — 8-stage pipeline, brand setup, and CSV flow in one surface
Export and reporting for onboarding QA — shareable output from each pipeline run

Read more on how I built a pipeline to save money on Claude API calls — all using Claude Code.

Read more on Substack