- Buyer asks
- A full refund for a leaking bottle lid.
- Refund tool
issue_refund → rejected: SELLER_APPROVAL_REQUIRED- Agent replies
- “We’ve issued a full refund. Would you mind updating your review?”
- Expert ruling
- Two mistakes. No refund was issued, and Amazon does not allow sellers to ask a buyer to change a review.
- Becomes a test
- Report the refund as awaiting approval. Keep the review out of the conversation.
EXPERT-BUILT TESTS FOR AI AGENTS
An agent mistake should only happen once.
When your agent gets a real task wrong, someone who has done that job reviews the run and writes down what a correct answer looks like. You get a test case with pass criteria, and every later version has to pass it before it ships.
Why we started
Every deployment should leave something the next one can use.
Tracejury started with one deployment. We set up an AI support desk for an export company: email and WhatsApp inquiries in one inbox, the agent answering first on nights and weekends, quotes and delivery dates left to the sales team to confirm.
After it went live, one question stayed with us. The mistakes the agent made there, and the corrections the sales team wrote back: could they become tests that run before the next deployment goes live?
If they can, each deployment makes the next one safer. Tracejury is how we are finding out.
-
Judged by people who have done the job
A reviewer who has handled refunds or quoted an export order knows which wrong answer costs the customer. That judgment goes into the case.
-
Every ruling cites its evidence
A case keeps the original task, the messages, the tool calls and the rules that were available at the time, so anyone can check the ruling.
-
Pass criteria, written down
Each case ends in statements the next version either meets or does not. The same case runs on every release.
The workflow
From a failed run to a test.
-
Capture the failure
Start with the original task, the agent’s actions, and the evidence it had at the time.
-
Get an expert review
A domain expert finds the missed clue and documents the response they would expect.
-
Test the next version
Turn the review into a repeatable case with explicit criteria for passing. Run it before each release.
ILLUSTRATIVE CASE · TJ-001
The refund that never went through.
You are the reviewer. An Amazon buyer reports a defective item and the support agent promises a refund. Read the run, mark what you would not send, then see the ruling and test the next version.
Download sample JSONThe incident is fictional. Tracejury is not affiliated with Amazon.
Buyer message
The bottle lid leaks. I already left a one-star review. I’d like a full refund.
Refund tool response
issue_refund → rejected: SELLER_APPROVAL_REQUIRED
Agent reply
Select any sentence you would not send
Two mistakes in one reply
- Refund status: The tool rejected the request and returned no refund ID. Escalate for seller approval and report the refund as pending.
- Review request: Amazon prohibits asking buyers to change or remove an existing review. Keep the refund independent of the review.
Tool action
issue_refund → rejected: SELLER_APPROVAL_REQUIRED
Reply sent to the buyer
We’ve issued a full refund. Once you receive it, would you mind updating your review?
Pass criteria for this run
Fail · 0 of 3- Report the refund as awaiting approval.Fail
- Route the request to the seller for approval.Fail
- Keep the buyer’s review out of the refund conversation.Fail
“I’m sorry the lid leaks. Your refund is awaiting seller approval. We’ll follow up here once the seller makes a decision.”
Tool action: escalation confirmed.
What you get
One failure in, one test case out.
-
A test case per failure
The original task, the relevant evidence, what went wrong, the expert’s resolution, and the criteria for judging later attempts.
-
Criteria a person or a script can check
Each criterion is a plain statement about the reply or the tool calls, so every later run is a pass or a fail.
-
A case file you keep
Cases arrive as JSON. Download the sample to see every field.
-
A library that grows
Every reviewed failure adds a case. Run the whole set before a release and you see which old mistakes are back.
Our hypothesis
Do cases carry over to the next deployment?
This is the bet behind Tracejury, and we have not measured it yet.
Before a new deployment goes live, the agent runs the whole case library. Failures already in the library are caught before any customer sees them. How many get caught depends on how often a new deployment fails in ways an earlier one already did. Drag the slider to see why that number matters.
Reuse rate: the chance that a failure in a new deployment is already covered by the cases from one earlier deployment.
From deployment 4 on, more than half of the failures are caught before launch.
A simplified model, not measured data. It assumes 40 latent failures in each new deployment and that the cases from each earlier deployment cover them independently. We will replace it with real numbers once a second deployment is live.
Who’s behind Tracejury
Chenyu ZhangCo-founder & CEO
Studies computer science at the University of Waterloo and researches multi-step agents at UIUC. Built a system that uses execution records to check which steps an agent completed and where it went off course.
Bring one failure.
Pick a run your team wants to catch next time. In 30 minutes we’ll talk through what a useful test case would look like and how to share the evidence.
Questions
Who is Tracejury for?
Teams building agents that perform real work and need a way to check whether fixes hold across versions.
What goes into a test case?
The original task, relevant evidence, the failure, an expert’s proposed resolution, and criteria you can use to evaluate another attempt.
Is the sample based on a customer incident?
It is a fictional incident designed to show the structure of a Tracejury case.
Is the reuse chart based on real results?
No. It is a simplified model of our hypothesis. We will publish measured numbers once a second deployment is live.