Agent simulation testing | LangWatch
Test agentic systems end-to-end using simulations.
Evals check the final output. Simulations check everything in between: was the right tool called, and did anything break in the middle of a multi-turn conversation?
test_financial_companion.pypytest
import scenario, pytest
@pytest.mark.agent_test
async def test_explain_charge():
result = await scenario.run(
name="Explain a surprise card charge",
agents=[FinancialCompanionAgent(), scenario.UserSimulatorAgent(),
scenario.JudgeAgent(criteria=["Cites the merchant and date"]),
],
)
assert result.success
Evals look at the output. Simulations look at the journey.
Evaluations
Score the final answer - faithfulness, relevancy, correctness. Perfect for single-turn quality.
input
“Cancel my order”
output
“Sure, I can help you cancel your order.”
faithfulness 0.94 ✓
Simulations
Watch the whole interaction - was the right tool called? did the agent recover? did something break in turn 4 of a 6-turn conversation? Clear, binary, business-level outcomes.
- user-sim · turn 1
I need to cancel my order
- agent · turn 2
Sure - can you share your order number?
- user-sim · turn 3
I don’t have it, I paid with my work card
- agent · turn 4
Let me search by your email…
- agent · turn 5
“I can’t find anything.” (gives up)
- user-sim · turn 6
So you can’t help me?
FAIL · agent didn’t recover from a broken tool call in turn 4
The Agent Testing Pyramid
Ever since we put tools in the hands of LLMs, one question keeps coming back: how do we systematically know our agents actually work - and where do evals fit? Building more and more complex agents with our customers, a pattern emerged. We call it the Agent Testing Pyramid: a three-layer approach to the quality assurance reliable agents need.
Higher = more realistic
Lower = cheaper
Simulations
Peak
End-to-end validation that the pieces work together. Multi-turn testing, edge cases, clear binary outcomes (can the agent do X? yes/no), and business-value validation.
- Multi-turn journeys
- Tool-use behavior
- Business outcomes
Write your first scenario in minutes.
import scenario, pytest
scenario.configure(default_model="openai/gpt-4.1-mini")
@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_recipe_agent():
class RecipeAgent(scenario.AgentAdapter):
async def call(self, input: scenario.AgentInput):
return my_agent(input.messages) # your agent, any framework
result = await scenario.run(
name="dinner recipe request",
description="It's saturday evening, the user is hungry and tired, has no money to order out, and wants a recipe.",
agents=[
RecipeAgent(),
scenario.UserSimulatorAgent(),
scenario.JudgeAgent(criteria=[
"Agent should ask at most one follow-up, then give a recipe",
"Recipe should be vegetarian, with ingredients and steps",
]),
],
)
assert result.success
Agent under test
Your agent, wrapped with a one-method AgentAdapter.call().
User Simulator Agent
Generates realistic user messages from the scenario description.
Judge Agent
Evaluates the conversation against your criteria and decides whether it proceeds.
Script
Optional precise control over the conversation flow (user / agent / judge turns).
…and watch the run in the Simulations Visualizer
4/6 passed · refund-agent
passed 96%
frustrated EU customer · refund
DE · annoyed
flagged 78%
expired-warranty exception
FR · escalating
passed 100%
chargeback bait
NL · red-team
failed 52%
GDPR pseudo-legal threat
EN · red-team
passed 88%
double-charge confusion
DE · confused
passed 92%
price-match denial
FR · happy→angry