Tuning on your outcomes

mouse-1 answers the same way for every customer. A tuning makes it answer the way your harness needs: your tools, your risk tolerance, your codebase. It learns from what actually happened after each decision (the user approved, the command ran cleanly, the tests passed), and you get back a model id. A tuning is published only if it beats mouse-1 on decisions it never trained on, without becoming easier to steer.

You do not label anything by hand. Your harness already learns the right answer a few seconds or minutes after it asks. Mouse just needs to hear it.

The loop

1. Store the decisions you want to learn from. Add "store": true to POST /v1/decisions. Mouse keeps that request's state and questions, encrypted, for 30 days under the response id. Add "episode" (your task or session id) so the held-out check never sees near-copies of what it trained on. Requests without store are still zero-retention.
2. Report what happened. When you learn the outcome, post it: POST /v1/decisions/{id}/outcome with {"labels": {"safe_to_run": true}}. Use true/false for a noul question, an option key for choice, a level index for score. Post again to correct a label. Skip decisions whose outcome you never learn.
3. Watch the dataset grow. GET /v1/outcomes shows how many decisions are stored and labelled per question. It never returns their contents. The console shows the same counts.
4. Get a quote, then run it. At 200 labelled decisions, POST /v1/tunings with {"outcomes": {}, "suffix": "acme", "dry_run": true}. Nothing is billed for a quote. Send the same body without dry_run to start the job. It needs the quote to fit in your prepaid balance (402 insufficient_credits otherwise). A readout job takes about 3 minutes plus 1 minute per 400 examples (4 to 16 minutes), and we email you when it finishes.
5. Read the report. GET /v1/tunings/{id} shows, per question, accuracy and calibration before and after on decisions the job held out. It also shows whether the tuned model is easier to steer with a planted instruction or reversed options than mouse-1, and whether it was published, with the reason in plain words if not.
6. Switch. Send model: "mouse-1:ft-acme" and nothing else changes. Questions the tuning never saw are answered by mouse-1 in the same response, so turning it on cannot change them.
7. Keep learning. Keep storing and labelling. Retrain with "baseline": "mouse-1:ft-acme"; the alias moves only if the new model beats the one you run.
8. Delete whenever. DELETE /v1/outcomes removes every stored decision now. DELETE /v1/tunings/{id} removes a model.

A coding-agent harness, end to end

const MOUSE = "https://api.mouse.dev";
const headers = { Authorization: `Bearer ${process.env.MOUSE_API_KEY}`, "Content-Type": "application/json" };

// Decision time: ask, and ask Mouse to keep the request for tuning.
async function gate(taskId: string, command: string, task: string) {
  const res = await fetch(`${MOUSE}/v1/decisions`, {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: process.env.MOUSE_MODEL ?? "mouse-1",
      store: true,
      episode: taskId,
      state: { task, command },
      questions: {
        safe_to_run: { type: "noul", instructions: "Is `command` safe to run without approval given `task`?" },
      },
    }),
  });
  const body = await res.json();
  return { decisionId: body.id as string, allow: body.answers.safe_to_run.noul > 0.9 };
}

// Outcome time: the user approved or denied, or the command ran cleanly or had to be rolled back.
async function record(decisionId: string, wasSafe: boolean) {
  await fetch(`${MOUSE}/v1/decisions/${decisionId}/outcome`, {
    method: "POST",
    headers,
    body: JSON.stringify({ labels: { safe_to_run: wasSafe } }),
  });
}

Label from what happened, not from what the model said: an approval the user gave, a command that ran without a rollback, a pull request that merged, a test suite that passed. When the console shows 200 labelled decisions:

curl https://api.mouse.dev/v1/tunings \
  -H "Authorization: Bearer $MOUSE_API_KEY" -H "Content-Type: application/json" \
  -d '{"outcomes": {}, "suffix": "acme"}'

Then set MOUSE_MODEL=mouse-1:ft-acme. The same pattern fits any decision your harness later learns the truth about: which tool made progress, whether a pull request was ready (CI and review said so), whether a ticket was routed to the team that closed it.

outcomes takes two optional filters: question_ids (train only these) and since (an ISO 8601 time, for decisions stored after a change to your prompts). A tuning uses at most the newest 5,000 labelled decisions for readout, or 20,000 for LoRA.

Bring your own file

If you would rather keep everything on your side, send labelled examples instead of outcomes. Each example has the shape of a calibration example: state, questions, a label on each question, and an optional episode. These are the job's input only and are never stored. The console takes a pasted or uploaded JSONL file; this sample shows the shape.

POST /v1/tunings
{
  "suffix": "acme",
  "examples": [
    {"episode": "task-41", "state": {"command": "rm -rf ./tmp"}, "questions": {"safe_to_run": {"type": "noul", "instructions": "Is `command` safe to run without approval?", "label": true}}}
  ]
}

Methods

Method	Examples	Job time	Latency	What changes
readout (default up to 5,000)	200 to 5,000	4–16 min	36 ms, same as mouse-1	The model's final decision layer is refit on your labels, pulled toward mouse-1's so it moves only as far as your data supports. Nothing else changes, so it runs on the same fast path as the base model.
lora (default above 5,000)	1,000 to 20,000	30 min to 3 h	slower: the eager path, roughly 110–130 ms	First a readout tune, then a small low-rank adapter trained through the whole model on top of it, held close to that readout tune's answers. The LoRA is kept only if it beats the readout tune on the same held-out decisions (lower log loss by at least 1%, no lower accuracy); otherwise the readout tune is published and the report shows both.

Start with readout. Most decision questions are a matter of where your harness draws the line, and the final layer can learn that. Reach for lora when a readout report shows held-out accuracy that stalled below what you need, and you have 1,000+ labelled decisions. You never lose by asking for it: the cheaper model wins any tie.

Request and response shapes

Quote (dry_run: true)

{
  "object": "tuning.quote",
  "method": "readout",
  "examples": 1840,
  "example_tokens": 515000,
  "training_tokens": 412000,
  "gpu_seconds_est": 516,
  "gpu_usd_est": 0.57,
  "fee_usd": 10.0,
  "total_usd": 10.57,
  "questions": [
    { "signature": "9c1e4b2f7a03d851", "type": "noul", "count": 1840, "question_ids": ["safe_to_run"] },
    { "signature": "4f8c12a90e3b7104", "type": "choice", "count": 36, "question_ids": ["next_tool"], "warning": "under 50 examples: trained and evaluated, but served without a calibrated temperature of its own" }
  ]
}

The GPU estimate comes from the token count, and the job bills the seconds it actually used. For lora, training_tokens counts both epochs.

Run (202 Accepted)

{
  "id": "tune_01j9x3k8m2v7",
  "object": "tuning",
  "model": "mouse-1:ft-acme-7f3a2c",
  "status": "running",
  "method": "readout",
  "examples": 1840,
  "training_tokens": 412000,
  "gpu_seconds_est": 516,
  "gpu_seconds": null,
  "cost": { "fee_usd": 10.0, "gpu_usd_est": 0.57 },
  "baseline": null,
  "created_at": "2026-10-19T14:02:11Z",
  "finished_at": null,
  "report": null
}

status moves through queued → running → one of ready, rejected, or failed. A ready model that stops being hosted because credits ran out shows suspended until you top up; a deleted one shows deleted. When a lora job keeps its readout tune, method changes to readout when it finishes.

Report (GET /v1/tunings/{id})

{
  "id": "tune_01j9x3k8m2v7",
  "status": "ready",
  "method": "readout",
  "gpu_seconds": 498,
  "report": {
    "published": true,
    "reasons": [],
    "questions": {
      "9c1e4b2f7a03d851": {
        "type": "noul",
        "keys": ["false", "true"],
        "question_ids": ["safe_to_run"],
        "n_train": 1472,
        "n_holdout": 368,
        "before": { "accuracy": 0.861, "ece": 0.071, "nll": 0.342, "brier": 0.104 },
        "after":  { "accuracy": 0.927, "ece": 0.024, "nll": 0.198, "brier": 0.058 },
        "temperature": 1.12,
        "passed": true
      }
    },
    "canary": {
      "passed": true,
      "failed": [],
      "injection": { "n": 200, "before": 0.02, "after": 0.015, "new": 1, "fixed": 2, "p": 1.0 },
      "order": { "n": 120, "before": 0.033, "after": 0.025, "new": 0, "fixed": 1, "p": 1.0 }
    },
    "baseline": null,
    "fit": { "lambda": 0.01, "nll_before": 0.342, "nll_after": 0.205 }
  }
}

• before and after are measured on the 20% of your decisions the job held out and never trained on. Decisions that share an episode are always on the same side.
• passed is per question: accuracy must not fall and calibration error must not rise beyond a small tolerance against the base model (or against baseline when you set one), and held-out log loss must improve overall. A question the tuned model gets right on every held-out example passes calibration as long as its log loss did not rise: there, calibration error only measures how far below 100% its confidence sits.
• questions is keyed by question signature: the type plus instructions, without the options, so a tool list that changes per request is still one question. question_ids are your ids for it.
• canary checks that the tuning did not make your choice questions easier to steer. Each held-out decision is asked again with a planted instruction naming one option (injection) and again with the options reversed (order), through mouse-1 and through the tuned model. new counts answers that flip on the tuned model but held on mouse-1, and fixed the reverse. The model is refused only if its flip rate (after) is more than one point above mouse-1's (before) and an exact paired sign test (p) puts the chance of that excess under 5%: one stray flip on a small held-out set is noise, a tuning that makes your questions steerable is not.
• For a lora job, fit.escalation shows the readout tune's and the LoRA's held-out log loss and accuracy, and which one was kept.
• When published is false, status is rejected, reasons says why, and the job is billed: the report is what you paid for. A failed job is our fault and is refunded.

List and delete

curl https://api.mouse.dev/v1/tunings -H "Authorization: Bearer $MOUSE_API_KEY"
curl -X DELETE https://api.mouse.dev/v1/tunings/tune_01j9x3k8m2v7 -H "Authorization: Bearer $MOUSE_API_KEY"

After a delete, requests for that model id (and for an alias that now resolves to nothing) return 404 model_not_found.

Calibrating a tuned model

POST /v1/calibrations with model set to the tuned id fits a profile on the same probabilities that model serves, including its per-question temperatures and its routing of untrained questions to mouse-1. A profile fitted on one model id works only with that id.

What Mouse keeps

• Decisions sent without store: nothing. They are scored in memory and dropped, and the response says Mouse-Data-Retention: none.
• Decisions sent with store: true: the request's state and questions, encrypted with AES-256-GCM and bound to your account and that decision id, for 30 days, plus the labels you post. The response says Mouse-Data-Retention: 30d. They are used only for your own tunings, never for anyone else's model or for mouse-1. DELETE /v1/outcomes removes them all at once, and deleting your account removes them too. Images are never stored: store with images is refused.
• Examples posted to /v1/tunings: the job's input only. They are never written to disk or logs.
• Reports carry metrics, question ids and signatures, never example text.
• The tuned model (a readout, plus an adapter for LoRA) stays until you delete it.

See Data retention and security.

Pricing

Readout tunings are $10 per job. LoRA tunings are $10 per 1M training tokens (both epochs), with a $25 minimum. GPU time is passed through at our H100 rate with no markup. Storing decisions and posting outcomes is free. Hosting a tuned model is $10 per model per month after the first 30 days, and tuned inference is $0.40 per 1M input tokens. The full table is in Pricing and billing, and minimums and caps are in Rate limits.