When a model or dataset changes, Driftless runs your eval, repairs only the files you allow, and can open a pull request with the evidence — or block the change if quality drops.
You need an eval you can run from the command line. The bundled demo needs no API key. Automatic prompt repair on a real workflow does.
Same harness, current vs target
Schema errors and wrong labels
Edits only the prompt files you allow
Winner must pass untouched holdout
The dependency problem
Models and eval data are dependencies of the prompt that works today. When either moves, Driftless re-tests the resolved workflow, repairs what is stale, and proposes the update—like Dependabot, but quality-gated.
The replacement is cheaper, but your JSON parser starts rejecting fenced output and classification quality drops.
Support changes its refund policy and updates gold labels. The model stays put, but the prompt now targets yesterday’s rules.
Automatic synchronization
Driftless closes the loop from a changed dependency to a reviewable update. Your eval decides what is better, and an untouched holdout decides what can ship.
Run current and target models through the same harness and dataset.
Cluster schema errors, wrong labels, refusals, and other recurring failures.
Generate candidate edits only inside the files you explicitly allow.
Choose on tuning data, then independently verify the winner on holdout.
Your contract
One versioned file tells Driftless how to run your app, where the eval lives, what repair may edit, and what “good enough” means.
Driftless shells out to your existing command. Your parsing, retrieval, tools, and post-processing stay intact.
Prompt files, examples, and config can be writable while schemas and product code remain read-only.
Use F1, schema error rate, pass rate, numeric scores, structured fields, or a calibrated LLM judge.
version: 1
workflows:
support_classifier:
run:
command: python evals/run_eval.py
input_path: evals/tickets.jsonl
model:
current: gpt-3.5-turbo
env_var: CLASSIFIER_MODEL
files:
editable:
- prompts/system.md
- prompts/examples.yml
readonly:
- schemas/ticket.schema.json
thresholds:
min_f1: 0.90
max_schema_error_rate: 0.02
migration:
holdout_required: trueReviewable outcomes
Public draft PR from a 290-label testbed: generated report, untouched holdout, one prompt repair, and a model configuration update.
The published CLI reproduces the four-row pass with --generator fixture. Regenerating PR #4’s exact patch still needs the testbed harness plus --generator llm (or the testbed simulator) and may differ.
Use cases
Gold labels and F1. Start with copy-example support-classifier (no key).
Free-form answers. Calibrate the grader before you trust its scores.
→ 07RAG QANew answer model, same retrieved docs. copy-example rag-qa.
New planner, same tools. copy-example tool-agent.
Try it locally
The bundled demo needs no API key. First compare shows a cheaper target failing the quality gate. --generator none keeps it blocked. --generator fixture applies the known-good patch and can pass. This is separate from public testbed PR #4.
Four-row smoke demo: this proves installation, contract execution, gating, and a key-free pass/block loop. It does not establish production quality, statistical confidence, or provider behavior. --generator llm is refused on this simulator; for a harness that calls OpenAI, use copy-example support-classifier-live. Use representative data and --generator llm before shipping a real workflow.
Adopting an existing app? driftless configure <workflow> --apply writes a draft driftless.yml you can review, then creates or appends the real file without rewriting comments. Fill in any leftover placeholders, then run validate. Add CI with init-ci last. Follow the adoption guide →
$ pip install driftless
$ driftless copy-example support-classifier --out-dir demo
$ cd demo
$ driftless validate -w support_classifier
$ driftless compare -w support_classifier --to gpt-4o-mini
$ driftless migrate -w support_classifier --to gpt-4o-mini --generator none
# Expected: BLOCKED with a non-zero exit; continue below.
$ driftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
# Expected: PASS — bundled known-good patch, still no API key.
$ driftless report -w support_classifier
$ driftless open-pr -w support_classifier
# Dry run: no PR or issue is created without --create.
--generator none exits non-zero because the cheaper target fails the quality gate. --generator fixture applies the bundled known-good patch and can pass. --generator llm is refused here. Both saved runs leave evidence for report and the dry-run open-pr preview.