From AI prototype to production: how to build evals for reliable agents

We've built a number of AI agents for clients at Nearform. What we've found is that while prototypes can be created at amazing speed and with relative ease, the key challenge in shipping a real application to production is ensuring agent reliability and confidence. In this blog post, we're going to share the principles and processes that help us deploy resilient agents to production faster, highlighting these key takeaways:

1. Build a golden dataset with domain experts that defines what "correct" means for your agent.
2. Combine code-based scorers (fast, cheap, deterministic) with LLM-as-judge scorers (nuanced, flexible) to cover both structured and open-ended outputs.
3. Evaluate tool usage and multi-turn behaviour, not just final text output.
4. Wire evals into your development workflow so they run automatically on every change.
5. Start small: 20-30 curated examples and 3-5 scorers are enough to begin catching regressions.

This article covers golden dataset design, scoring strategies, tool and multi-turn evaluation, and CI integration. While we will not cover live monitoring, model fine-tuning, or retrieval pipeline optimization in depth, each of these topic merits its own deep dive.

Most AI projects get validated through demos where inputs are controlled, the happy path is tested, and edge cases are conveniently skipped. The weaknesses only surface once real users interact with the system.

Traditional testing can't solve this, as AI systems are non-deterministic: the same input can produce different outputs across runs (even when you set temperature to 0). Standard assertions break when the "correct" answer can be phrased a hundred different ways, when tool call sequences vary, or when the agent's reasoning path shifts between runs. You need a different approach to quality assurance, one built around probabilistic scoring rather than binary pass/fail.

A golden dataset is a curated collection of representative tasks and expected behaviours that defines what "correct" looks like for your agent. Think of it as the test suite for a system where outputs vary between runs.

The dataset structure depends on your use case. Input and expected output are essential. You may also add context, expected tool calls, tags, status, or any metadata that helps with evaluation. For agents, tracking tool usage and steps lets you assess whether they are on the right track.

This gives you a stable reference to measure progress and catch regressions over time.

To make this all a bit more concrete, here’s an example golden dataset for a customer support agent:

ID	Input	Context	Expected Output	Expected Tool Call	Tags	Status
order-status	"What's the status of order #4521?"	Customer ID: C-102, has active order	"Your order #4521 is currently in transit and expected to arrive by March 10."	getOrderStatus(orderId: "4521")	difficulty:easy	active
cancel-and-refund	"Cancel my subscription and refund the last charge"	Customer ID: C-205, monthly plan, last charge $49	"Your subscription has been cancelled and a refund of $49 has been initiated. You'll receive a confirmation email shortly."	cancelSubscription(customerId: "C-205"), initiateRefund(amount: 49)	difficulty:medium, regression	active
double-charge	"I was charged twice this month"	Customer ID: C-310, single charge visible in billing system	"I've checked your billing history and can see only one charge this month. If you're seeing a duplicate on your statement, it may be a pending authorization. Would you like me to escalate this to our billing team?"	getBillingHistory(customerId: "C-310")	difficulty:hard, edge-case	active
ambiguous-request	"Can you help me with something?"	No prior context	"Of course! Could you tell me a bit more about what you need help with so I can assist you?"	None	difficulty:easy, ambiguous-input	active

Developers build agents, but domain experts know what correct behaviour actually looks like. Without their input, your dataset will miss realistic workflows, edge cases, and domain-specific constraints that determine whether the agent works in production.

The process works like this:

1. Gather real interactions. Pull from production logs, support tickets, or recorded user sessions. Synthetic examples alone won't capture the messiness of real usage.
2. Identify the scenarios that matter with domain experts. Have them flag common requests, known failure modes, compliance-sensitive tasks, and ambiguous inputs.
3. Define expected behaviour precisely. For each scenario, document what the agent should do, which tools it should call, what arguments it should pass, and what output it should produce.
4. Tag and categorise. Mark each entry with difficulty, category, and edge case flags. This lets you run targeted subsets later.

Something most teams don't understand initially is how much time this process takes. Across multiple teams, we noticed it can take tens of hours of effort by developers and experts working together to build and validate the dataset. Account for that when estimating the project.

You can start small: twenty to thirty well-curated examples are more valuable than hundreds of sloppy ones. Grow the dataset as you discover new failure modes. When your eval scores plateau but production users still report issues the dataset doesn't cover, that's your signal to expand. Most teams find that 50 to 100 entries cover the core behaviour well, with more added for specific feature areas or high-risk scenarios. The goal is coverage of distinct behaviours, not volume for its own sake (especially because you'll pay to run experiments on the dataset).

A golden dataset is only useful if the team can trust it. That means treating it with the same rigor you apply to source code: version it, review changes, and resolve disagreements explicitly. And you ideally want a good tool to leverage it.

In this article, we’ll focus on evals from the Mastra agentic framework, which we’ve been using on client projects. It's comprehensive, flexible, and TypeScript-forward (which we love at Nearform). Other frameworks like Braintrust, Promptfoo, and LangSmith offer similar evaluation features. The same principles apply, whichever tool you use.

The simplest approach is to store the dataset as a CSV file in your codebase. Every change goes through a pull request, just like code. This gives you a clear history of what changed, who changed it, and why. When a domain expert adds a new test case or adjusts an expected output, the change is visible and reviewable. Additionally, it’s often desirable to pick tools that let you manage golden datasets outside the repository for ease of use by domain experts.

One thing we noticed with clients: especially early on, forcing domain experts to open PRs for each change doesn't work well if they aren't technical. We had more success using Google Spreadsheets as the collaboration surface than downloading the CSV from there.

Disagreements between domain experts are inevitable. Two experts might define different expected outputs for the same ambiguous input. When this happens, document both perspectives and decide on a single canonical answer. If the input is genuinely ambiguous, the correct expected behaviour might be to ask the user for clarification rather than guessing.

A side effect we didn't expect: building the dataset surfaces new requirements. It forces people to think about scenarios nobody planned for. You end up with better requirements as a result.

As the dataset grows, structure matters more. For example, organising entries by category, difficulty, or feature area, and using tags consistently lets you run targeted subsets (e.g., all "billing" cases or all "hard" cases) without running the full suite every time.

One thing to avoid: letting the dataset grow unchecked. Periodically audit it with your domain experts. Remove outdated entries, merge duplicates, and update expected outputs that no longer reflect current product behaviour.

Evals are automated checks that measure whether your agent behaves correctly across the scenarios in your golden dataset. They're the equivalent of automated tests, but for non-deterministic systems.

Every time you change a prompt, swap a model, add a tool, or modify your retrieval pipeline, you run evals. They tell you whether the change improved things, broke things, or had no effect.

Because AI systems are non-deterministic, the same test case can pass on one run and fail on the next. One option is to run critical entries multiple times and average the scores. Set score thresholds (for example, relevancy above 0.8) rather than demanding perfect results on every run. This absorbs natural variance while still catching real regressions.

Mastra makes this observable. When tracing is enabled, it captures the full context of each agent execution: prompts, responses, tool calls, latency, and cost. You can trace exactly what happened in each eval run.

Not all outputs can be scored the same way. You need different strategies depending on what you're measuring.

Code-based scorers

These evaluate outputs deterministically using rules. They're similar to normal tests, so they're fast, cheap, and reliable for structured checks:

• Exact match: Did the agent return the correct order ID? Does the answer contain the right link?
• Schema validation: Does the JSON output conform to the expected structure?
• Keyword coverage: Did the response mention the required terms?
• Regex matching: Does the output follow a required format?

Some things can't be scored with rules. Was the response helpful? Did the agent reason correctly? Was the tone appropriate? For these, you use an LLM as a judge: a separate model that scores your agent's output against defined criteria. This approach was formalised in the LLM-as-a-Judge paper by Zheng et al. and has become standard for evaluating open-ended outputs.

Mastra ships built-in LLM-based scorers for common evaluation needs, but you can also build custom ones. Here's a simple example attaching a relevancy scorer to your agent:

import { Agent } from "@mastra/core/agent";
import { openai } from "@ai-sdk/openai";
import {
  createAnswerRelevancyScorer,
  createFaithfulnessScorer,
  createToxicityScorer,
} from "@mastra/evals/scorers/llm";

const agent = new Agent({
  name: "support-agent",
  instructions: "You are a customer support agent...",
  model: openai("gpt-5"),
  scorers: {
    relevancy: {
      scorer: createAnswerRelevancyScorer({ model: openai("gpt-5-mini") }),
    },
  },
});

LLM-as-judge scorers themselves add cost and latency to your eval pipeline. Running five LLM-based scorers on every eval entry can get expensive quickly. This is another reason to combine them with cheap, fast code-based scorers, and to reserve LLM judges for the cases where deterministic scoring genuinely cannot capture what you need.

For agentic systems, scoring the final text output isn't enough. An agent might produce a perfectly worded response while calling the wrong tool, passing incorrect arguments, or skipping a required step entirely. If you only evaluate the output, you miss these failures completely.

Tool usage evaluation checks three things:

1. Did the agent call the right tool? If a user asks to cancel a subscription, the agent should call cancelSubscription. A correct final answer produced without the right tool call is still a bug, because the agent didn't really accomplish the task.
2. Did the agent pass the right parameters? Calling initiateRefund(amount: 0) instead of initiateRefund(amount: 49) is a silent failure that a text-only scorer would never catch.
3. Did the agent execute tools in the right sequence? Some workflows require a specific order. You might need to verify an account before modifying it, or fetch data before acting on it.

Your golden dataset should capture all three. The "Expected Tool Call" column in the dataset table exists for exactly this reason. A well-written scorer compares the agent's actual tool call sequence against the expected one, checking tool names, argument values, and ordering.

Code-based scorers work well here. Tool calls are structured data, so you can write deterministic checks that compare expected versus actual calls without needing an LLM judge.

Evaluation failures tell you one of three things: the agent is wrong, the eval logic is wrong, or the dataset is wrong. Learning to tell the difference is where most of the skill lives.

When a new scorer consistently flags correct outputs as failures, the scorer needs to be tuned. We had one scorer that was too strict with matching, even though domain experts confirmed the results were good enough. After investigation, we loosened the criteria.

When real users surface a failure mode not covered by any existing test case, the dataset needs to be expanded to include it. When the agent genuinely gets something wrong, you fix the prompt, the tool, or the retrieval pipeline, and the eval confirms the fix.

This feedback loop is continuous. The dataset is never "done." It evolves as your agent handles more real traffic and as you discover new ways the system can fail.

In practice, this works through trace analysis. Every interaction from demo users or live users is captured as a trace, including the full sequence of prompts, tool calls, and responses. Domain experts periodically review these traces, looking for failure modes the current dataset doesn't cover. When they spot a new one, they add a corresponding entry with the expected behaviour clearly defined. From there, you iterate on the agent until it handles that case correctly and the eval confirms it.

Scorers evolve alongside the dataset. A new compliance requirement might mean adding a scorer that checks for mandatory disclaimers. A new tool integration means validating tool call arguments. When the brand voice shifts, you adjust the tone criteria. The changes come from both domain experts and engineers, and they don't always agree on priority, which is fine.

On a recent client project, we needed to improve a scorer called link-accuracy, which measured whether the agent returned the expected links in its responses. When we started, link-accuracy sat at 0.35 out of 1.0. Wrong, outdated, or missing links in nearly two-thirds of test cases. We assumed the problem was the agent's reasoning, but it turned out to be spread across the entire stack.

Over three sprints of digging into failing eval entries, we found issues at every layer. About 15% of the entries in the golden dataset itself had incorrect expected links, leftover from an earlier version of the product. Upstream APIs were returning stale or malformed data. Several tool descriptions were misleading, causing the model to call the wrong tool or pass incorrect arguments. The system prompt needed a full restructure to give the agent clearer guidance on when and how to surface links.

The first round of fixes (externalize some context in a RAG pipeline instead of always injecting it) actually dropped link-accuracy from 0.38 to 0.31. This is normal. Restructuring a system prompt can temporarily destabilise behaviour in cases that previously passed by accident. But each subsequent fix compounded: correcting the dataset removed false negatives and brought the score to 0.52, fixing the upstream data pushed it to 0.68, and improved tool descriptions brought us to 0.75. By the end, link-accuracy stabilised at 0.81, with the remaining gap mostly due to genuinely ambiguous cases where the "correct" link was debatable.

Because regression evals ran automatically on every branch, we could validate each pull request in isolation. A PR that fixed a tool description could be checked immediately for whether it improved link-accuracy without regressing other scorers like tone-consistency or answer-relevancy. Without that setup, the same investigation would have taken weeks of guesswork.

If you only run evals manually before major releases, regressions slip through for weeks. There are two types of evaluations worth distinguishing:

• Live evaluations score agent interactions in real time, running asynchronously on production traffic without blocking the response. Useful for continuous monitoring, but not the focus of this article.
• Golden dataset evaluations run your agent against your curated test cases, either locally or in CI. This is where you catch regressions before they reach users.

In Mastra, this is built around the concept of experiments. You create a dataset, then run your agent against it with a set of scorers. Each run is a named experiment, making it easy to compare results across changes:

import { mastra } from "./index";
import { createAnswerRelevancyScorer } from "@mastra/evals/scorers/prebuilt";

const dataset = await mastra.datasets.get({ id: "support-agent-golden" });

const relevancy = createAnswerRelevancyScorer({ model: "openai/gpt-5-mini" });

const summary = await dataset.startExperiment({
  name: "improved-system-prompt-v2",
  targetType: "agent",
  targetId: "support-agent",
  scorers: [relevancy],
});

console.log(summary.status); // 'completed' | 'failed'
console.log(summary.succeededCount);
console.log(summary.failedCount);

for (const item of summary.results) {
  console.log(item.itemId, item.output);
  for (const score of item.scores) {
    console.log(`  ${score.scorerName}: ${score.score} - ${score.reason}`);
  }
}

Naming each experiment (like "improved-system-prompt-v2") lets you compare runs side by side and track whether a change actually moved scores in the right direction. A useful naming convention is "branchId-commitId", especially when running evals in CI.

Treat golden dataset evals like CI tests. A focused "regression" subset of your most critical test cases runs automatically on every PR that touches the agent (prompts, tools, or model changes). For bigger changes, run the full suite to verify nothing else broke.

Building evals and a proper golden dataset takes more time than most teams expect. It requires engineering and domain experts working closely together, and there's no shortcut around that. The more you feed real interaction data back into the dataset, the more resilient the system gets. An AI agent working in natural language breaks in ways that a traditional UI never would, and you need different tools and processes to catch those failures before your users do.

If you're building AI systems and need help getting from demo to production, Nearform can bring expertise in scaling all parts of the stack; evals, prompts, models, and everything in between.