How to evaluate an AI support agent before you trust it with customers

Most teams "test" an AI agent by chatting with it for ten minutes, coming away impressed, and shipping. Then a real customer phrases something sideways, the agent does something strange, and the incident review begins. The problem isn't the model. It's that a ten-minute vibe check is not a test plan.

Why agents need a different kind of testing

Traditional software is deterministic: same input, same output, one assertion. Agents are probabilistic and open-ended — the same intent arrives in a thousand phrasings, and the agent's behavior emerges from reasoning, not fixed branches. You can't unit-test your way to confidence with a handful of hardcoded strings.

What you can do is simulate: generate realistic customer conversations at volume and grade how the agent handles them.

The three things worth measuring

A serious evaluation checks more than "did it give a good answer":

• Did it resolve the goal? Graded per conversation — resolved, partially, or escalated — ideally by an LLM judge with a rubric, so you can score hundreds of runs, not five.
• Did it use the right tools? An agent can produce a lovely answer while calling the wrong function or none at all. Check the actions, not just the prose.
• Did it stay in bounds? Did it honor guardrails, avoid off-limits topics, and hand off when it should? A confidently-wrong action is the failure mode that matters most.

Test the failures on purpose

Averages hide the incidents that make the news. The evaluations that earn trust deliberately probe the edges:

• Adversarial and ambiguous inputs — the sideways phrasings, the mid-conversation topic changes, the customer who won't give a clear answer.
• Tool failures injected on purpose — what does the agent do when the billing API times out? Recover gracefully, or hallucinate success?
• Regression tests — every real conversation that went wrong becomes a fixed test case, so the same bug can never ship twice.

That last one turns evaluation from a launch gate into a ratchet: the agent can only get more reliable over time, never quietly less.

Evaluation is continuous, not a launch step

The mistake is treating evals as something you do once before go-live. Models change, your knowledge changes, and customer behavior changes. The teams that trust their agents run the suite continuously and watch resolution as a live metric — which is the same feedback loop that improves the agent in the first place.

You wouldn't ship backend code with no tests. Don't ship an agent on a vibe check. This is exactly what the optimize tooling is built for.