Keep every LLM workflow in sync.

What Driftless does

Providers retire models. Teams change gold labels. A cheaper model looks tempting. In each case, swapping an ID is rarely enough: the prompt that worked yesterday often fails today, and you need numbers before you ship.

Driftless is a command-line tool (the GitHub Action just runs the same commands). You describe the task once in a driftless.yml file. Driftless then:

1. Runs your existing eval command under the old model and the new one.
2. Repairs only the files you list as editable (usually prompts).
3. Checks the winner on holdout rows it did not tune on.
4. Prints a report and can open a PR — or an issue if the bar is not met.

You own the eval, the quality bar, and the credentials. Driftless orchestrates the loop. It does not reimplement your parser, retrieval, or tools.

Words you'll see

Installation

Python 3.10 or newer. For the quickstart you only need the CLI — no API key:

copypip install driftless

Skip this unless you are developing Driftless itself:

copypython -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Automatic prompt repair on your app needs a provider key. Install the extra and export one key:

copypip install "driftless[llm]"      # openai + anthropic SDKs
export OPENAI_API_KEY=sk-...   # or ANTHROPIC_API_KEY

Quickstart

Copy the bundled ticket classifier. It runs without an API key. -w support_classifier is the workflow name inside driftless.yml.

copypip install driftless
driftless copy-example support-classifier --out-dir driftless-classifier-demo
cd driftless-classifier-demo
driftless validate -w support_classifier
driftless compare -w support_classifier --to gpt-4o-mini

validate checks the wiring. compare runs the current model and a cheaper target on the same four rows and does not edit files. You should see F1 1.000 → 0.000 and cost 0.024 → 0.004.

Read FAIL min_f1: 0.000 >= 0.9 as: the cheap model scored 0.000, you required at least 0.9, so a bare swap is not safe to ship. That failure is the demo working.

Blocked path — no prompt edits, still no API key. migrate is expected to exit non-zero. Run the later commands anyway:

copydriftless migrate -w support_classifier --to gpt-4o-mini --generator none
driftless report -w support_classifier
driftless open-pr -w support_classifier

Passing path — applies the known-good patch shipped with this example:

copydriftless migrate -w support_classifier --to gpt-4o-mini --generator fixture
driftless report -w support_classifier
driftless open-pr -w support_classifier

Expect PASS. open-pr only prints what it would open unless you add --create. Do not use fixture on your own app. This example is a local simulator: --generator llm is refused here.

For a harness that calls OpenAI, use copy-example support-classifier-live (needs OPENAI_API_KEY). Other key-free demos: copy-example rag-qa and copy-example tool-agent. See use cases for which example matches your app.

Use cases

The quickstart is a classifier. Same product loop, different app shape:

Classifier, RAG, and agent copies are local simulators: --generator llm is refused. Use --generator fixture for the key-free pass, or the live classifier / a customer harness for real repair.

Public alpha

Driftless is for teams that already have an eval they can run from the command line. The bundled examples need no API key. Putting it on your own app needs a filled-in contract, a reliable harness, and a plan for provider cost and secrets. There is no hosted bot; see what 1.0 will include.

Adopt Driftless in an existing repository

The bundled example is a tour. For your own app, you still bring the eval — Driftless will not invent one. If you are unsure which command to run, use the command chooser.

1. Discover and configure: find model usage, then turn it into a driftless.yml draft.
2. Validate locally: make sure your eval command runs, then compare one target model.
3. Set boundaries: review which files may be edited, the quality bar, and the provider budget.
4. Add CI last: generate workflows with init-ci, review the files, then commit only the jobs you want.

copycd your-existing-repo
driftless scan
driftless configure <workflow> --apply
# Review the inferred contract; resolve any remaining placeholders.
driftless validate -w <workflow>
driftless compare -w <workflow> --to <model>
driftless init-ci

If discovery does not fit your setup, start from the smallest working contract or run driftless init. The Getting Started walkthrough shows that minimum file, then a fuller draft-to-contract example.

The workflow contract

driftless.yml is the one file you fill in. Each named block under workflows: is one LLM task. Unknown keys are errors, so typos fail fast instead of being ignored.

Smallest working contract

For a classifier with gold labels, this is enough to validate and compare. You do not need config_file, token fields, or a second workflow.

copyversion: 1
workflows:
  my_classifier:
    model:
      current: gpt-4
      env_var: MODEL
    files:
      editable:
        - prompts/system.md
    run:
      command: python evals/run_eval.py
      input_path: evals/inputs.jsonl
      output_path: evals/outputs.jsonl
    eval:
      labels_path: evals/gold.jsonl
      label_field: label
    thresholds:
      min_f1: 0.90

The harness must write one JSON object per line to output_path and read the model from the env var. model.current is Driftless's baseline, not what the app reads at runtime. Skip config_file unless a PR should rewrite an in-repo model ID. Not a classifier? Use eval.score_field, eval.pass_field, or eval.judge instead of labels.

A fuller contract can add holdout, cost gates, schema checks, and config_file once that minimum runs:

copyworkflows:
  support_classifier:
    run:
      command: "python evals/run_eval.py"
      input_path: evals/inputs.jsonl
      output_path: evals/outputs.jsonl
      timeout_seconds: 600
    model:
      current: gpt-3.5-turbo
      target_candidates: [gpt-4o-mini]
      env_var: SUPPORT_CLASSIFIER_MODEL
      config_file: config/llm.yml
      config_path: model
    files:
      editable: [prompts/system.md, prompts/examples.yml]
      context: [src/parser.py]
      readonly: [src/**]
    eval:
      labels_path: evals/labels.jsonl
      schema_path: schemas/ticket.schema.json
      label_field: category
      split: { tuning: 60%, seed: 7 }
    thresholds:
      min_f1: 0.9
      max_schema_error_rate: 0.02
    migration:
      max_iterations: 4
      holdout_required: true
      split_seed_count: 1

You need one way to swap the model at runtime. env_var is enough. A workflow may also set config_file:

• env_var — Driftless sets this environment variable when it runs your command, so the same script can use a different model during eval.
• config_file + config_path — a path into a JSON/YAML file. Used when opening a PR, to write the new model ID into the repo.

Percentages are ergonomic: 60%, 60, and 0.6 all parse to 0.6.

Command chooser

Start here if you know the goal but not the command:

• Try it without keys: copy-example support-classifier, then the quickstart.
• Is my setup wired? validate. Is a cheaper model safe? compare.
• Repair for a new model: migrate. Repair after labels changed: refine.
• Let CI decide what work exists: plan. Preview a PR: open-pr (add --create only to actually open it).

See the complete command chooser for label audits, judge checks, and automation variants.

The migration loop

In everyday terms: split the eval into a tuning set and a holdout set. Try the new model with the old prompt. If that already passes, stop. If not, generate small prompt edits, score them on tuning data, keep the smallest edit that helps, and only accept a winner that also passes on holdout — rows the loop never trained on.

The same loop powers migrate and refine:

• Migrate — the model changed. Repair until your thresholds pass.
• Refine — the eval data changed. Keep the model pinned, chase a better score, then suggest new thresholds.

The sketch below is for readers who want the exact algorithm. You can skip it and still use the CLI. (evaluate means: apply files in a backup/restore sandbox, run your real command, score the split.)

copyrun_migration(W, M_target, generator G, objective O, seed):

  # ── Setup ──────────────────────────────────────────────
  tuning, holdout ← split(W.dataset, seed)        # deterministic, seeded
  baseline     ← evaluate(M_current, current_files, tuning)
  naive_target ← evaluate(M_target,  current_files, tuning)

  # ── Short-circuit: is a bare model swap enough? (migrate) ─
  if O = MEET_THRESHOLDS and passes(baseline, naive_target)
     and passes_on(holdout):
       return model_change_only               # just bump the model ID

  # ── Iterative repair ───────────────────────────────────
  original ← current editable files            # frozen, for diff sizing
  best ← naive_target ; best_files ← {} ; best_size ← 0
  width ← G.num_candidates ; widened ← false   # adaptive search width

  for i in 1 .. W.migration.max_iterations:
      clusters   ← cluster_failures(best.rows)  # group similar errors
      context    ← { clusters, failing & correct examples,
                     attempt history, editable + readonly files }
      candidates ← G.generate(context, escalated_width if widened else width)
      if candidates = ∅: break

      improved ← false
      for patch in candidates:
          size ← diff_size(patch, original)
          try:
              check_scope(patch)               # reject edits outside files.editable
              cand ← evaluate(M_target, apply(patch), tuning)
          except error:                        # patch broke the workflow
              log(failed) ; continue           # skip it — never abort the run

          better ← score(cand, O) > score(best, O)
          tie    ← score(cand, O) = score(best, O) and size < best_size
          if better or tie:                    # tie → the smaller edit wins
              best, best_files, best_size ← cand, patch.files, size
              improved ← true

      if O = MEET_THRESHOLDS and passes(baseline, best)
         and passes_on(holdout, best_files):
           commit(best_files) ; return pass    # validated on never-tuned data

      if improved:         widened ← false             # progress → cheap width
      else if not widened: widened ← true ; continue   # stall → widen once
      else:                break                        # stalled at full width

  # ── Resolve outcome ────────────────────────────────────
  if O = MAXIMIZE:                              # refine
      validate best_files on holdout (no-regression vs. current)
      suggest fresh thresholds from holdout metrics
      return pass if best beats naive_target else no_change
  else:                                         # migrate, thresholds unmet
      return partial if best improved over naive_target else blocked

Invariants the loop guarantees, regardless of what a generator proposes:

• Sandboxed trials — every candidate is applied via backup → run → restore; the working tree is written only on a committed pass.
• Crash isolation — a candidate that breaks the workflow (e.g. emits invalid YAML/JSON) is logged as a failed attempt and skipped; it can't abort the run.
• Minimal-change tie-breaker — on an exact score tie the smaller edit wins; against the no-op baseline (best_size = 0) a same-scoring patch is rejected, so the loop never makes a change that doesn't help.
• Stall-escalation — a stalled iteration widens the candidate pool once (to max(width × 3, 5)) before giving up — cheap when easy, broad when stuck.
• Holdout gate — nothing is committed until it clears a split it never tuned against.

Migration statuses

migrate exits non-zero on partial / blocked, so it gates CI naturally. A blocked migration still produces a full report and files an actionable issue.

Safety guarantees

These are enforced by the engine, not left to the patch generator:

• Edit-scope enforcement — any patch touching a file outside files.editable is rejected before it is applied.
• Sandboxed application — candidate edits are applied with originals backed up and always restored, so evaluation never leaves the repo dirty.
• Holdout gating — nothing is committed unless it passes thresholds on the holdout split.
• No auto-merge, no force-push — open-pr is a dry run by default; --create opens a PR/issue but never merges or pushes to the base branch.

Known limits

• The supported product is the command-line tool plus a GitHub Action you run in CI you control. There is no hosted bot.
• Embedding migrations and vector-index rebuilds are out of scope; RAG migrations keep retrieval fixed.
• Agent tools should be fake or sandboxed, and provider lifecycle data is bundled/CI-refreshed rather than a live hosted catalog.
• Small evals can produce noisy passes. Use representative data, meaningful holdout, and multiple seeds for high-risk changes. See eval confidence.
• --generator fixture only repairs bundled simulators. --generator llm is refused on those; use support-classifier-live or a customer harness.

Read the full limits and ownership boundaries before broad rollout.

Cost & budgets

compare runs current and target workflows. migrate adds candidate evaluations and holdout; judge grading can add one provider call per output on every evaluation. Start with a representative sample, migration.max_iterations: 2–3, one candidate at a time, and a required holdout. Expand only after the report shows useful progress.

Estimate rows × models × candidates × iterations (plus judge/tool calls), set provider-side spend limits, and keep open-pr dry-run until the result is understood. See cost shapes and suggested starting budgets.

Upgrading

Read the changelog before changing release lines. Version 0.3 rejects legacy migration.allow_* fields: replace them with exact files.editable paths, run validate locally, preview one compare, and only then update CI. Follow the complete 0.3 upgrade guide.

Triggers & policy

Most commands answer can we change the model. Policy answers when should we even look — optional, like a Dependabot config. A trigger is only a candidate; your eval still decides whether it is worth it.

A .driftless/policy.yml configures per-trigger thresholds, candidate allow/deny globs (preview models denied by default), and an ignore snooze list. The plan command wires this together as a CI triage step.

CLI commands

Useful flags on migrate:

• --generator llm|none|fixture — repair strategy (LLM-backed by default; none is a dry analysis; fixture reproduces bundled-example patches).
• --to <model> — the target model to migrate to (otherwise the contract's candidates are used).
• --strict-label-audit — block when duplicate/near-duplicate inputs disagree on gold labels (warns by default).

Contract schema reference

Evaluation metrics

compare and migrate load the output JSONL your command writes, align it with gold labels, validate each record against the JSON schema, and compute:

By default they compare current prompt on current model vs current prompt on target. When the prompt was never source-optimized, that delta mixes prompt debt with model drift — see Measuring migration gains.

• Accuracy + macro precision / recall / F1 (per-class breakdown retained for failure clustering).
• Schema error rate — unparseable or schema-invalid records.
• Refusal rate — empty/null labels, a truthy refused field, or values listed in eval.refusal_values.
• Average latency — derived from run duration / record count.
• Total cost — only when the workflow emits a per-record cost_field. Token-based estimates are never fabricated.

Measuring migration gains

compare and migrate score your current prompt on the current model (baseline) and the same prompt on the target (naive_target). That mirrors flipping a model ID in prod without touching the prompt — the right default.

But when the prompt was never tuned to its ceiling on the source model, the delta conflates two effects:

• Prompt debt — under-optimization that would improve on either model.
• Model-induced drift — quality lost because the target behaves differently.

A headline "+0.07 F1 after migration" can be mostly debt, not repair. The refine path (dataset change, model pinned) avoids this; model migration needs a control.

2×2 control

Optimize on the source model first, then switch. Example from the testbed (macro-F1, 290 labels, real API calls, gpt-3.5-turbo → gpt-4o-mini):

• P0 → A = prompt debt on the source (not migration).
• A → B (−0.072) = true model-induced drift from a strong baseline.
• B → D (+0.066) = gain from re-tuning after the switch.

Full write-up and repro steps: Measuring migration gains honestly.

Run viewer

After migrate or refine, inspect the optimization trajectory in a local web UI — iteration metrics chart, failure-cluster trends, and a full attempt log (rationales, scores, diff sizes, accept/reject).

copydriftless view                    # opens http://localhost:8777/runs.html
driftless view -w support_classifier

The viewer reads .driftless/migrations/<workflow>.json from the current project. You can also load any result JSON via file picker or drag-and-drop. Static demo: runs.html → Load saved passing fixture. Reproduce that passing path locally with migrate --generator fixture on a bundled example. The --generator none quickstart still ends BLOCKED.

Repair & custom generators

The engine defines a PatchGenerator protocol; the repair strategy is swappable. Three ship out of the box:

• LLMPatchGenerator (default) — asks an LLM to rewrite the editable files to fix the observed failure clusters. Provider-neutral, requests strict JSON, and varies temperature across candidates.
• NoOpPatchGenerator (--generator none) — proposes nothing; the loop becomes a dry analysis, useful offline and for CI gating.
• FixturePatchGenerator (--generator fixture) — applies the known-good patch for a bundled example so a passing repair is reproducible without provider keys.

You can customize repair via the contract's repair: block — append domain guidance, fully replace the system prompt, or supply a user_template with {{placeholder}} substitution (placeholders include failure_clusters, failing_examples, editable_files, metrics, and target_model).

A generator only ever proposes — the engine owns acceptance. See The migration loop for the full algorithm (sandboxing, crash isolation, the minimal-change tie-breaker, stall-escalation, and holdout gating).

GitHub Action

A composite GitHub Action wraps the CLI so scans and migrations can run in CI. After validating locally, run driftless init-ci, review the generated files (and any inferred setup step), and commit only the automation you need. Override setup with --setup-command when inference does not fit. Refinement remains manually dispatched unless --refine-on-push is explicit.

copy# .github/workflows/driftless-plan-preview.yml
name: Driftless plan preview
on:
  schedule: [{ cron: "0 9 * * 1" }]
  workflow_dispatch: {}

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: driftless-dev/driftless@v0.3.6
        with:
          command: plan

This is a policy plan preview, not the dependency-only scan workflow. A scheduled plan gates CI when a deprecated model needs attention; a manually-triggered migrate opens a PR (or an issue when blocked) with the evidence attached.