Every test of your agent hides a judgment about what counts as done.
A test written without the people who own the task ends up measuring how well an answer reads. A weaker model can fake that. So it passes the test while it does the wrong work.
Operant builds the test from your own chats, but a person on your team makes the call. The test grades a cheaper model against your current model's answers to the same messages.
Good depends on the task
None of these definitions comes from a leaderboard:
| Task | It's good when |
|---|---|
| A compliance pack | An auditor accepts it |
| An incident digest | Nothing in the period is missing, and every line traces back to the tracker |
| A codebase search | It names the right call sites without dumping whole files into the chat |
The people who own a task know things no general benchmark captures. They know which errors are embarrassing and which are a disaster. They know which corners can be cut under load, and what a reviewer will reject.
Without them, the failure looks like this. A cheaper model writes an answer that reads well but does the wrong work. It passes a test that was never pointed at the work in the first place.
Operant builds the harness
Operant does the machinery. It captures the calls, groups the work and sets chats aside for testing. Then it runs them on a cheaper model and grades the answers. It also runs the approval flow.
What it doesn't do is invent your definition of good. The test uses your own chats from production. The cheaper model is graded against what your current model answers to those same messages. So the bar comes from your own work, not from a leaderboard.
A person still decides
A test score is evidence, not a verdict. The person who owns the work looks at the score, the example chats and the price, and decides. That person can be wrong too. So turning a change off needs no test: one click puts the task back on the top model.
Starting from your own chats means you don't need a finished dataset or a perfect rubric to begin. Teams often stall for months trying to write the perfect rubric from scratch. Here, the test comes from work you already run, and your current model sets the bar. All you need is work that repeats, and someone who owns the task to make the call.
So a cheaper model takes over only when it holds up on your work and your team says yes.
See how that bar gets applied in How we prove a cheaper model can do the job.