Which Metric Should Decide a Subject Line Test After Apple MPP

The email marketing metrics to measure subject line performance post Apple MPP, with the open rate inflated by machine opens, are click rate for a single send and placed order rate or revenue per recipient once enough volume has pooled. Many teams already accept that opens are unreliable. What gets skipped is the arithmetic that follows: the further a metric sits from the inbox, the more recipients it needs before a gap between two subject lines means anything, and a 40,000-person list runs short quickly. The winning metric has to be one your list can resolve.

What does Apple say Mail Privacy Protection does?

Email marketing metrics after MPP shifted because of one behavior that Apple Mail Privacy Protection documents in plain terms. On its privacy page (dated 2025-12-12), Apple says that when a message arrives in the Mail app or Mail on iCloud.com, Protect Mail Activity downloads remote content in the background by default and routes it through two separate relays, so that no single entity can identify both the reader and the mail content they receive. The download happens, in Apple's words, "regardless of whether you engage with the email."

A tracking pixel is remote content, so for those recipients the open is recorded whether or not anyone read the subject line. Mailchimp describes the effect on its own reports in the same terms: with Apple MPP enabled, emails present in Apple Mail are reported as opened "regardless of the contact's activity, resulting in inflated and inaccurate open rates."

How do Klaviyo, Mailchimp and HubSpot treat those opens in 2026?

Apple Mail Privacy Protection open rates get different handling in each platform's help pages as of October 2026, and the email marketing metrics each one offers in their place, clicks and conversions, differ too.

The Klaviyo A/B testing article (updated Aug 5, 2025) still recommends open rate for subject line, preview text and sender tests, and in the same section says Klaviyo "should account for these inflated open rates; however, you may need a higher threshold to reach statistical significance." Its placed order rate option requires a Placed Order metric on the account. Mailchimp's FAQ says the results of A/B tests based on open rates "may not be accurate" and suggests click rates, and the Mailchimp testing overview adds a manual pick and notes that access to A/B tests depends on plan. HubSpot excludes suspected bot activity from marketing email performance data on all plans, while the A/B feature needs Marketing Hub Professional or Enterprise according to the HubSpot testing article. We found no statement in either HubSpot article on whether an open-rate winner is picked from filtered opens.

What are the alternatives to open rate?

The alternatives to open rate in email marketing after MPP come down to four numbers, and each asks for a different amount of volume.

• Click rate, clicks as a share of delivered recipients. Mailchimp states that clicks and purchases "aren't impacted by Apple MPP." This is our default decider for a subject line on a single send.
• Open rate with machine opens excluded. A fast early read where the platform can filter, with the caveat that it describes only readers without Protect Mail Activity turned on.
• Placed order rate, the share of recipients who buy. Klaviyo offers it as an automatic winner, and Mailchimp's nearest equivalent is total revenue.
• Revenue per recipient, attributed revenue divided by delivered recipients. It is the number finance cares about and the noisiest of the four.

Among email subject line testing metrics, revenue per recipient and click to open rate respond to Apple Mail Privacy Protection differently. Click to open rate divides by opens, so preloaded opens swell the denominator; Mailchimp warns that reports might show a lower "Clicks per unique open" rate for contacts who enable Apple MPP. Revenue per recipient never touches the open event. HubSpot's A/B article lists click rate and click through rate as separate options, and its HubSpot email analytics article defines click-through rate as clicks out of the number who opened, so choose click rate as the winning metric.

Can a 40,000-email subject line test reach significance?

For email subject line A/B testing, the sample size on a send of 40,000 emails decides which of those metrics can reach statistical significance at all. The example below is hypothetical, and every input is our assumption. A list of 40,000 delivered recipients is split into two variations of 20,000. Half the list reads in Apple Mail with Protect Mail Activity on, and every one of those messages registers an open. The other half opens at 20%, which puts the reported open rate at 60%. Click rate is 2.0% and placed order rate is 0.30%. The required sample comes from the two-sample test for proportions that the R manual hosted by ETH Zurich documents as power.prop.test, run two-sided at its default 0.05 significance level with 80% power.

Lifts are relative changes in real behavior. In the first row, a 10% lift in human opens moves the reported rate from 60% to 61%, because the preloaded half of the list cannot move. In the second row, only half of each variation counts toward the test, so the recipient figures are twice the number of non-Apple readers required.

Machine opens land on both variations in equal proportion, so they dilute the difference: the unfiltered open test needs nearly three times the recipients of the filtered one for the same real lift. Click rate on a full 50/50 split resolves lifts of roughly 21% and larger, and a 10% lift takes 80,682 recipients per variation, about four sends of this size with the same hypothesis repeated (question against statement, for instance). Placed order rate is out of reach for one send, since a 20% lift needs 143,421 per variation, a little over seven sends. Revenue per recipient adds the spread of order values to that, so plan for more volume again. Swap in your own baselines before using any of this; our email marketing statistics page collects sourced figures if you need a reference point.

How should the winner be called?

In email marketing A/B testing of subject lines, sample size belongs in the plan before the send, and the winner is called against it afterward. The platforms' own labels work as floors. Klaviyo tags a campaign test statistically significant when 50 people have received each variation and the win probability is at least 90%, and HubSpot recommends sending an A/B tested email to at least 1,000 contacts. Neither figure says how small a lift the test could have seen, which is the job the table above does.

Slice tests need the same check. Klaviyo's setup example gives 20% of recipients to each variation and picks a winner after 6 hours. On 40,000 that is 8,000 per variation, enough to resolve click lifts of about 33% and up. Letting the platform send the leading line to the remaining 24,000 costs nothing, but record it as a learning only when the gap clears that bar. At this list size we would rather split the whole send 50/50 on click rate and pool the result with the next test of the same idea. Klaviyo also recommends "a test duration based on the A/B test metric" and we give clicks and orders a longer window than opens. In HubSpot, set the fallback version, the one sent when results are inconclusive, to your control.

How do you prove an AI subject line tool is working?

The metrics beyond open rate that prove AI subject line optimization under Apple Mail Privacy Protection are the same ones: click rate per send, then placed order rate or revenue per recipient across a pooled series. Klaviyo's AI option, available on paid accounts, writes variation B from your original line by changing one attribute such as personalization, length, question versus statement, or emoji. Scoring is where the risk sits, because a tool rewarded on unfiltered opens is partly rewarded by Apple's background downloads. Keep a human-written line as variation A on every send, set click rate as the winning metric, and judge the tool on pooled clicks and orders once the volume in the table has accumulated. One limit to plan around: Klaviyo's personalized variations strategy, offered to accounts with more than 400,000 total profiles, cannot use placed order rate.

A subject line test only measures something when the message reaches the inbox, so settle placement first: what email deliverability is covers the basics, and our email deliverability checker is a further resource. Testing also carries a price, since HubSpot keeps A/B tests on Marketing Hub Professional and Enterprise and Mailchimp ties them to plan, which makes it part of what email marketing costs. Our lifecycle and demand generation team plans test programs this way, and the lifecycle email playbook is the wider reference. Pick the metric your volume can resolve, write the required sample beside the hypothesis before you send, and let the open-rate default go.

Sources

• Apple: Mail Privacy Protection & Privacy (dated 2025-12-12)
• Klaviyo: How to A/B test an email campaign (updated Aug 5, 2025)
• Klaviyo: Understanding statistical significance in Klaviyo campaigns (updated Jul 7, 2025)
• Mailchimp: Apple Mail Privacy Protection (MPP) FAQs
• Mailchimp: About A/B Tests
• HubSpot: Understand bot filtering in marketing email analytics
• HubSpot: Run A/B tests for marketing emails
• HubSpot: Analyze marketing email performance (updated October 5, 2026)
• ETH Zurich: R manual, Power Calculations for Two-Sample Test for Proportions (power.prop.test)