Test AI agents on live traffic, not only on lab datasets
In brief
According to Datadog, teams can now compare two AI agent versions on real user requests, not only on prepared tests. Quality scores are read together with delay, token use, cost and errors. Datadog says offline tests still help, but they often miss unexpected cases.
Source : InfoQ - « Datadog Brings Production Experimentation to AI Agent Observability »
What it changes for an SME
For most firms with 10 to 250 staff, this Datadog feature does not change daily work: it is aimed at teams that already run AI agents and monitor them in that product. If you put an internal assistant into production, a strong result on a test file does not prove the change will help real users. According to Datadog, a newer model cut estimated cost by 44.3 percent with almost the same quality score, while making more incorrect tool calls that were later retried. The practical risk for you is decision-making: cost, delay and errors can worsen even when the quality score stays flat.
What you can do
- Before you change a model or a prompt, decide how you will measure quality, cost and errors on real use.
- Check whether your current tool can send a small share of live requests to a new version.
- Record unexpected production failures and add them to your next test set.
Let's talk about the time you could save.
In 20 minutes, we find where AI could save you time or money. Free, no commitment.