Measure whether your AI agent is useful, not just online
In brief
According to Grafana Labs, classic checks such as response time, technical errors and token use do not show whether an AI agent's answer is correct, grounded or useful. The company proposes evaluating separate behaviours, including request fulfilment, toxicity and cost, then turning them into service-level objectives and an error budget. The aim is also to spot a slow drop in quality, not only a sudden outage.
Source : InfoQ - « Grafana Applies SLOs and Error Budgets to AI Agent Behaviour »
What it changes for an SME
If you already run an internal assistant, the risk is not only an outage: according to Grafana Labs, it can keep answering while becoming less useful, more costly or less safe, a little at a time. Treating each behaviour as a separate target (fulfilment, toxicity, personal data, cost) avoids hiding a failure in one overall score. This is not a legal duty, and it only helps if the measurement is sound: an automatic judge can be wrong. Without a clear definition of acceptable behaviour, you lack a basis to decide when to try changes and when to freeze them.
What you can do
- Write down what you will not accept from your assistant: unmet requests, toxic content, personal data, hijacked instructions, excessive cost.
- Ask your IT team whether you track anything beyond uptime and response time.
- Require checks on real conversations, with human review of doubtful cases, not only offline tests.
Let's talk about the time you could save.
In 20 minutes, we find where AI could save you time or money. Free, no commitment.