Driftless field notes
Guides for when the model, the labels, or the eval change.
Each post is one situation: a retired model, cheaper inference, new gold labels, RAG, or an agent. If you already know your app type — classifier, summarizer, RAG, or agent — start from the map below.
The provider retires your model. Changing the ID is not enough — compare first, then repair only if quality recovers.
Read guide → Support changed the labeling policySupport changed what “correct” means. Keep the model, update the prompt, and prove it on rows the repair loop did not see.
Read guide → Prompt repair that doesn’t wait on memoryStop relying on someone to remember. Schedule a weekly check that opens a PR or issue with evidence.
Read guide → Finance wants cheaper inference, same barFinance wants a cheaper model. Keep the same quality bar so cheaper is never an automatic yes.
Read guide → Offline F1 is lying — labels conflictIf two similar tickets have different expected labels, no prompt or model can get both right. Audit first.
Read guide → Free-form answers graded by an LLM judgeFree-form answers need a grader. Check that the LLM judge agrees with people before you trust its scores.
Read guide → RAG QA: new answer model, same knowledge baseA new answer model can drop citations or invent policy even when retrieval stays the same.
Read guide → Tool-calling agent: new planner, same toolsA polite final sentence can hide a skipped refund check. Score the tool calls, not just the prose.
Read guide →