How to Prove an AI Email Feature Adds Revenue With a Holdout Test
Measure the incremental revenue of AI send-time optimization or AI-written email with a randomized holdout: ESP control group features, sizing math and a worked readout.
On this page
- Why run a randomized trial when the ESP already reports revenue?
- Send time optimization lift and attributed flow revenue are different numbers
- Which control group features do ESPs document?
- How many contacts does each group need?
- A worked readout with hypothetical numbers
- How do you isolate AI-written content from seasonality and promotions?
- What if the feature is already live for everyone?
- Sources
Measuring the incremental revenue of AI send-time optimization in email marketing takes one holdout test: assign contacts to two groups at random, give one group the AI feature and the other your usual send, then compare everything each group spends over the same weeks. The same design covers AI-written content. Both groups live through the same sale, the same payday and the same slow week, so the calendar cancels and the gap that remains belongs to the feature. The work sits in three places: the control group tools your ESP documents, a group size built on revenue per recipient, and a readout you can set beside attributed revenue.
Why run a randomized trial when the ESP already reports revenue?
The cleanest incrementality estimate comes from a randomized controlled trial, while marketing attribution assigns revenue by a crediting rule, which is why a send time optimization holdout and the ESP dashboard can disagree about the same emails. The survey of online controlled experiments by Kohavi, Longbotham, Sommerfield and Henne (Data Mining and Knowledge Discovery, 2009), hosted on ExP Platform, gives the reason: random assignment distributes confounding variables equally across the variants, which establishes a causal relationship between the change and the measure of interest.
Attribution follows a different logic. The Klaviyo article on message attribution (updated Mar 10, 2026) says new accounts default to a last touch model with 5-day lookback windows for email opens and email clicks, and that "Klaviyo will only attribute a customer event (e.g., placed order) to messages within one of these open windows." That rule decides which message gets credit for an order. We found no statement in it about whether the order would have been placed without the message, and that is what finance wants to know before paying for an AI feature.
Send time optimization lift and attributed flow revenue are different numbers
Send time optimization's incremental lift and the attributed revenue on a flow or campaign report can drift apart, because the feature is tuned for engagement and attribution is triggered by engagement. The Braze documentation says Intelligent Timing delivers a message at the time when a user "is most likely to engage (open or click)."
Put the two documented behaviors together and a pattern follows. A feature that earns more opens places more orders inside an attribution window, and those orders move into the email column whether or not the customer would have bought that week anyway. Flows widen the gap, since a cart or browse trigger selects people who were already close to buying. This is our reading of the rules, so test it on your own data: attributed flow revenue shows which message was the last touch, and the holdout estimates what the feature added.
Which control group features do ESPs document?
| Platform and feature | What the vendor's page says | Fit for a single-feature test |
|---|---|---|
| Klaviyo random sample split in flows (updated Feb 23, 2026) | A conditional split set to "Random sample" sends a chosen percentage down the first path, "which you can think of as the control"; 50% gives a close to even split | Good inside one flow |
| Klaviyo global holdout groups (updated Sep 23, 2025) | Holds profiles back from emails, SMS and push notifications sent through campaigns and flows; needs at least 400,000 profiles; about 5% of marketable profiles and a 3-month run recommended; one active group at a time | Measures the whole program |
| Braze Global Control Group | A percentage of all users who shouldn't receive any campaigns or Canvases, with exceptions the page lists; a recommended size of no smaller than 1000 users and no more than 10% of the audience; the report runs on sessions or a custom event | Measures the whole program |
| HubSpot random distribution by percentage branch (updated July 31, 2026) | Splits enrolled records into evenly sized groups; Marketing Hub Professional or Enterprise required; a minimum of 1,000 records for an approximately even split | Good inside one workflow |
The two global groups answer a bigger question (what the whole messaging program adds) and cannot isolate send time or copy. Braze's page adds that a campaign-specific or Canvas-specific control group can run alongside its global one. For a feature test, use a random split and keep each contact in one group for the whole run, since the Kohavi paper calls a consistent experience for each user important. Klaviyo's page says the split "will randomly choose a branch per profile" and we found no statement there on whether a profile keeps its branch across flows. Our practice is to write the group to a profile property once, then branch every campaign and flow on that property.
How many contacts does each group need?
Kohavi's controlled experiments paper supplies the sizing rule for an incremental revenue holdout in marketing, and it shows how costly revenue is as a test metric compared with conversion. The formula is n = 16σ²/Δ², where n is the number of users in each variant, σ² is the variance of the metric and Δ is the change you want to detect, at 95% confidence and 80% power. In the paper's example, a site where 5% of users buy and spend about $75 (a $3.75 average, with an assumed $30 standard deviation) needs over 409,000 users to detect a 5% change in revenue and fewer than 122,000 when the metric is conversion rate.
The same arithmetic for email, using our own illustrative assumptions for a four-week window:
| Metric | Assumed baseline | Assumed spread | Contacts per group, 10% lift | Contacts per group, 5% lift |
|---|---|---|---|---|
| Revenue per recipient | $2.40 | $18 standard deviation | 90,000 | 360,000 |
| Order rate | 3.0% | p(1-p) = 0.0291 | 51,733 | 206,933 |
Pull the real mean and standard deviation from order data per contact, zeros included. Three more rules from the paper shape the plan. Split 50/50, since a 99%/1% test has to run about 25 times longer. Run for at least a week or two and then in multiples of a week so day-of-week effects can be analyzed, a lesson the authors extend to holidays and seasons. Before the real test, run an A/A test in which both groups get the same experience, which checks the split and gives you the variability for the sizing math.
A worked readout with hypothetical numbers
Picture a store with 200,000 emailable contacts, split 50/50 for four weeks. Both groups get the same campaigns and offers on the same days; the treatment group gets AI send time and the control group gets the usual fixed hour. Every figure below is invented for illustration.
| Control, fixed hour | Treatment, AI send time | Difference | |
|---|---|---|---|
| Contacts assigned | 100,000 | 100,000 | |
| ESP-attributed email revenue | $50,000 | $59,000 | +$9,000 (+18%) |
| All order revenue from assigned contacts | $240,000 | $246,000 | +$6,000 (+2.5%) |
| Revenue per recipient | $2.40 | $2.46 | +$0.06 |
| 95% interval on that difference | -$0.10 to +$0.22 |
With an $18 standard deviation, the standard error of the difference is about $0.08, so the t statistic is roughly 0.75 against the 1.96 threshold the Kohavi paper cites for 95% confidence. Three readings sit side by side. The dashboard credits the AI-timed emails with $59,000. The gap between groups on attributed revenue is +18%. On all orders the lift is +2.5%, the interval spans zero, and its upper bound of $0.22 falls short of the $0.24 that a 10% lift would need. So the test argues against a lift of 10% or more and leaves a smaller one unresolved, which is a usable answer when the feature carries a price. Two habits keep the readout clean: count all orders from every assigned contact, opened or not, and fix the duration before launch.
How do you isolate AI-written content from seasonality and promotions?
A test design that isolates the effect of AI content generation from seasonality and promotions changes one thing between two randomized groups and runs both through the same calendar.
- Hold the audience rules, the offer, the send day and the send hour identical, so only the copy differs.
- Send the promotion to both groups. A sale that lifts everyone lifts both lines equally.
- Compare concurrently. Last month's human-written emails against this month's AI emails measures the month.
- Treat human editing of AI drafts as part of the treatment, and record the process you tested.
- Track unsubscribes and spam complaints per group as guardrails. Revenue that arrives with more complaints is a trade to examine before rollout. Our guide to what email deliverability is explains inbox placement, and the email deliverability checker is a further resource.
What if the feature is already live for everyone?
A/B testing handles seasonality and promotions through the randomized experiment itself, and difference in differences is a standard causal inference method marketing teams fall back on when no randomized group exists. Brodersen and colleagues at Google, in a 2015 Annals of Applied Statistics paper available on arXiv, describe it as estimating the difference between the pre-post difference in the treatment group and the pre-post difference in the control group. The assumption underneath is that the control group's level is an adequate proxy for what the treated group would have done without treatment.
In email terms, one brand list or region gets the feature while a comparable one keeps the old setup. The same paper cautions that marketing data are "subject to multiple seasonal variations" and that static models "yield overoptimistic inferences with too narrow uncertainty intervals" when fit to serially correlated data. Report a difference-in-differences figure as second-best evidence, and schedule the randomized test for the next cycle.
A holdout costs you the feature on half the list for a month, and in return it prices the feature in dollars that hold up in a finance review. For context around the test, our email marketing statistics page puts email's share of revenue in perspective, what email marketing costs covers platform and agency rates, and the lifecycle email playbook is a further resource. Our lifecycle and demand generation team designs this kind of test.
Sources
- ExP Platform: Kohavi, Longbotham, Sommerfield and Henne, Controlled experiments on the web: survey and practical guide, Data Mining and Knowledge Discovery 18:140-181 (2009)
- arXiv: Brodersen, Gallusser, Koehler, Remy and Scott, Inferring causal impact using Bayesian structural time-series models, The Annals of Applied Statistics 9(1):247-274 (2015)
- Klaviyo Help Center: Understanding Klaviyo message attribution (updated Mar 10, 2026)
- Klaviyo Help Center: How to A/B test flow branches (updated Feb 23, 2026)
- Klaviyo Help Center: Getting started with global holdout groups (updated Sep 23, 2025)
- Braze: Global Control Group
- Braze: Intelligent Timing
- HubSpot Knowledge Base: Use branches in workflows (updated July 31, 2026)