How to Write Evaluation Criteria and Test Sets for a Marketing Agent

To write evaluation criteria and test sets for an AI marketing agent before production, list every decision the agent makes without a person, write one pass or fail statement per decision, and collect 20 to 50 real cases with the right answer recorded for each. Model vendors document that method for developers. Marketing teams need three additions: graders for brand, claims and routing, a test set small enough to keep current, and a launch threshold agreed before the first run.

Which evaluation criteria does a marketing agent need before production?

The evaluation criteria and test sets for a marketing AI agent before production should come straight from the workflow document, where each step in which the agent decides, writes or sends becomes one criterion with a yes or no answer. Anthropic's developer documentation asks for success criteria that are specific, measurable, achievable and relevant, and illustrates the first with this line: "Instead of 'good performance,' specify 'accurate sentiment classification.'" In marketing terms, "qualifies leads well" becomes "assigns the owner and lifecycle stage a senior SDR gave the same lead."

A second test comes from the January 2026 guide to agent evals by Anthropic's engineering team: "A good task is one where two domain experts would independently reach the same pass/fail verdict." If your demand gen lead and your sales ops lead would grade one output differently, rewrite the criterion first.

The table is an illustrative example for an agent that qualifies inbound leads, updates the CRM and drafts the first follow-up.

The routing and write checks correspond to what OpenAI's evaluation guide calls tool selection and data precision, defining the second as "Evaluations that verify the agent calls the tool with the correct arguments." Google's Gen AI evaluation service, whose agent evaluation feature is labeled Preview, separates final response evaluation from trajectory evaluation, meaning the sequence of tool calls. Its trajectory_single_tool_use metric checks only whether a named tool was used, which suits handoffs. For the other rows, grade the final CRM state instead of the order of calls, since Anthropic's guide reports that checking a fixed sequence proved "too rigid" and made tests brittle.

Claims get the strictest grader. The FTC's advertising guide for small businesses states: "Before a company runs an ad, it has to have a 'reasonable basis' for the claims." The same guide says an advertising agency may also be held legally responsible for misleading claims in ads. We apply that standard to any promotional copy an agent drafts, so the approved claims list sits in both the agent's context and the grader.

How should the golden test set be built, and how large?

Best practices for offline test set design when evaluating LLM agents come down to three points across the vendor guides: draw cases from real traffic, include the hard ones, and balance the set in both directions. Anthropic's guide puts the starting size within reach of a small team: "In reality, 20-50 simple tasks drawn from real failures is a great start." OpenAI's guide supplies the mix: "Ensure your test data includes typical cases, edge cases, and adversarial cases." It also lists as an anti-pattern "Creating eval datasets that don't faithfully reproduce production traffic patterns."

For the lead agent above, a 50-case golden set that one marketer can maintain might split as follows (our suggestion, an assumption to adjust to your traffic).

• 25 typical leads, sampled from last quarter's inbound in proportion to source.
• 10 edge cases: missing fields, personal email domains, duplicates, existing customers, non-English submissions.
• 5 adversarial cases: instructions typed into a form's free-text field, discount demands, a competitor posing as a buyer.
• 10 cases where the right move is to hold back, by handing off or doing nothing: enterprise accounts, unsubscribe requests, legal questions, vendor spam.

The last group is there because Anthropic's guide warns that "One-sided evals create one-sided optimization." An agent tested only on leads that deserve an email can end up emailing everyone.

Each case stores the input as the agent would see it (the form fill, the CRM record, or the brief for a content agent), the labeled answer for every criterion, and a reference output that passes every grader. Have two people label independently and rewrite any case where they disagree. Strip personal data before the set leaves the CRM.

In what order do you build the eval suite?

To build an eval suite for an LLM agent before deployment, write the code graders first, the model judges second, and reserve people for calibrating the judges. Anthropic's documentation describes code-based grading as the fastest and most reliable method and human grading as high quality but slow and expensive, and it says of LLM-based grading: "Test to ensure reliability first then scale." Routing, CRM writes, banned terms and handoffs all fit code, while tone and claim support need a judge.

Give each judge one dimension and one rubric, as Anthropic's guide suggests. Ask for a verdict instead of a score, since OpenAI's guide lists "Use pairwise comparison or pass/fail for more reliability" among its recommendations for model judges. Let the judge return "Unknown" when the claims sheet does not cover a statement, and route those cases to a person. Then have a marketer grade the same outputs blind and compare verdicts, because Anthropic's guide says judges "should be closely calibrated with human experts" before teams rely on them.

Run every case more than once, because agents vary between runs. Anthropic's guide gives the arithmetic: "If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%." A lead agent touches every inbound contact, so count a case as passed only when all of its trials pass.

What goes on the launch threshold worksheet?

As an LLM agent evaluation framework, a golden test set works as a pre-production checklist once each criterion carries a threshold written down before anyone sees results. NIST's Generative AI Profile (AI 600-1, July 2024), a companion to its voluntary AI Risk Management Framework, suggests establishing minimum thresholds for performance or assurance criteria and reviewing them as part of deployment approval, the go/no-go decision. It also suggests sharing pre-deployment test results with those who hold release approval authority, and verifying sources and citations in generated outputs during pre-deployment measurement and ongoing monitoring.

We found no primary source that publishes a pass threshold for brand, claims or routing accuracy in marketing. The figures below are placeholders for a hypothetical team running the 50-case set, and yours should follow from what one miss costs. The 10 hold-back cases produce no email and we assume each of the other 40 does, so 40 emails are graded.

Use zero tolerance where one error is expensive and no reviewer stands behind the agent, and a percentage where a person approves the output anyway. On 50 cases a single failure moves the pass rate by two points, so read 47 and 48 of 50 as the same result. A launch decided on a handful of good demos runs against the profile's suggested action to "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments."

Where a criterion misses but the workflow still pays, launch with a human approval step on that action and record it in your governance operating model. Field permissions belong in the same review, and our guide to CRM write access for marketing agents covers where the documented limits sit.

What should you keep measuring after launch?

After launch the golden set becomes the regression suite. Anthropic's guide says regression evals "should have a nearly 100% pass rate" and protect against backsliding, so rerun the full set on every prompt, model or tool change and treat a drop as a defect. Each week, review a sample of production runs by hand and add the misses as new cases.

Whether the agent stays in production depends on pipeline and cost, so track the share of agent-qualified leads that sales accepts, time to first touch, and review minutes per run. Our AI ROI calculator and the cost formula in what a marketing agent costs per contact are further resources for that math. Newcomers to the vocabulary can start with what agentic AI is, and our agentic AI automation service is the place to ask for help with the build.

Send the worksheet, the test set and the first run's results to whoever signs off on launch, as one document. If that person can tell in a single read which misses block launch and which are covered by an approval step, the evaluation has done its job.

Sources

• Anthropic: Demystifying evals for AI agents, published January 9, 2026
• Anthropic: Define success criteria and build evaluations, Claude Platform Docs
• OpenAI: Evaluation best practices
• Google: Evaluate Gen AI agents, a Preview feature
• NIST: AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024
• FTC: Advertising FAQ's: A Guide for Small Business, April 2001, edited January 2025