How to Validate a Marketing Mix Model Before You Shift Budget
Five checks a marketing mix model should pass before budget moves, mapped to the diagnostics Meridian and Robyn report, with pass and fail readings and a staged shift.
On this page
To validate a marketing mix model's accuracy before shifting budget, treat the holdout test as two separate tests: weeks of the KPI held out of training, and a lift test held out of calibration. The first shows whether the model's structure is overfit. The second is the only check that sets the model's ROI against a measured causal effect. Around them sit convergence, a baseline that makes sense, and a look at how far each posterior moved from its prior. We mapped all five to the diagnostics Google's Meridian and Meta's Robyn report, using each tool's documentation as it read on October 7, 2026.
What does Meridian's documentation say validation can prove?
Google's Meridian marketing mix modeling documentation is plain about causal inference and model validation. The model-fit guide from Google says "directly validating the quality of causal inference is difficult and requires well-designed experiments", and that those experiments "must have the same estimand as the MMM". Short of an experiment, the page says, you rely on indirect measures.
Meridian packages those measures as health checks that each return PASS, REVIEW or FAIL, and rolls them into a model health score from 0 to 100. The scoring guide from Google says a score of 90 or greater "suggests the model does not exhibit serious statistical issues and can be used for decision making", while 70 or less "indicates potential systematic errors". It also asks readers to treat the scores as "directional insights rather than absolute rankings of model quality". The weights say more than the thresholds do. R-squared carries a 7.5% weight, which rises when skipped checks drop out and the remaining weights are renormalized. Negative baseline, the Bayesian posterior predictive p-value (PPP) and the composite calibration score carry 22.5% each, prior-posterior shift and ROI consistency take 12.5% each, and a model that has not converged scores zero.
The five checks, mapped to what Meridian and Robyn report
MMM validation methods fall into three groups: diagnostics the tool prints, calibration against a geo lift test, and a holdout test on data the model never saw. Meridian's readings below come from the health-check guide from Google and the scoring guide above, Robyn's from the features page from Meta.
| Check | Meridian reports | Robyn reports |
|---|---|---|
| Convergence | max_r_hat, the highest R-hat across all parameters. Below 1.2 is PASS; 1.2 or above is FAIL | Two criteria, labeled "UNDER REVISION" in the docs, that compare the last quantile of iterations with the first ones on the standard deviation and median of each error |
| Fit, in and out of sample | R-squared, MAPE and wMAPE, for the full and out-of-sample data when a holdout is set. R-squared above 0 is PASS; 0 or below is REVIEW. Bayesian PPP of 0.05 or above is PASS; below is FAIL | nrmse_test when ts_validation = TRUE, plus actual vs. predicted and fitted vs. residual plots on each model one-pager |
| Baseline | Posterior probability that the baseline is negative. Below 0.2 is PASS; 0.2 to 0.8 is REVIEW; above 0.8 is FAIL | A decomposition waterfall, with baseline defined as "non-steerable variables" |
| Prior versus posterior | Prior-posterior shift for ROI: PASS when every channel shows a significant shift, REVIEW when any channel shows none. ROI consistency: REVIEW when a posterior mean falls outside the 1st to 99th percentiles of a custom prior. Both run only with ROI priors | DECOMP.RSSD, the difference between share of spend and share of effect. Robyn fits a ridge regression, and we found no prior-versus-posterior output |
| Lift test | Composite calibration score: a calibrated channel scores 100; an uncalibrated one scores 0 to 100, lower when it needs calibration more, per the channel guide from Google | MAPE.LIFT, the calibration error against experiments |
Three cautions go with that table. We found no pass threshold for nrmse_test or DECOMP.RSSD on Robyn's features page, so we use them to rank candidate models. The tools also part ways on R-squared: Robyn's Analyst's Guide from Meta offers a rule of thumb in which R-squared above 0.9 is "ideal" and below 0.8 is "not ideal, aim to improve", while Meridian's health-check guide warns that a model can reach a high R-squared by fitting a trend that sales and ad spend share, crediting organic growth to the ads. We side with Meridian on that point. Last, a REVIEW on prior-posterior shift means the data did not move that channel's ROI away from the prior, so any reallocation the optimizer proposes for that channel rests mostly on the prior, which Google's guide accepts when the prior came from solid external evidence such as an earlier experiment.
Which weeks should the holdout test hide?
Google Meridian separates MMM validation into experiments for causal accuracy and a holdout sample for fit, and its holdout advice argues against the simplest design. Per the holdout guide from Google, the holdout argument takes a HoldoutSpec in one of three forms: a random draw of time periods in each geo, the same date ranges in every geo, or given dates in given geos.
The guide recommends a sample "fairly balanced across geos and time periods" and tells users to avoid holding out large, contiguous chunks of time, such as the end of the modeling window. Its reasoning is that Meridian "isn't designed for forecasting the KPI", and that the knots it uses for trend and seasonality need data near them. It also limits what the result means: the holdout is for comparing model specifications on the same sample, and the best out-of-sample fit is no guarantee of the best model for causal inference.
Robyn splits the series itself. With ts_validation = TRUE, its features page describes a three-way split into training, validation and test, sized by a train_size hyperparameter whose default range is 0.5 to 0.8. Validation error becomes the optimizer's objective, and nrmse_test reports out-of-sample performance.
In both tools we read the gap between training and test error first. A test error far above the training error suggests the overparameterization Meridian's fit guide says the check exists to catch. A close match clears the structure and says nothing yet about whether the ROI estimates are causal.
Why keep one lift test out of calibration?
In marketing mix modeling, validation against a holdout geo experiment is the one test of MMM accuracy that calibration cannot replace. A model calibrated on a test will tend to agree with it: Meridian sets the channel's ROI prior from the result, and Robyn's optimizer minimizes MAPE.LIFT against it. Agreement afterward shows the calibration took hold, and only an experiment the model never saw is free to disagree. On the pages we read, we found no report in either tool that makes that comparison, so we do it by hand.
The comparison needs matching estimands before it means anything. The calibration guide from Google asks about a test's recency, duration, media granularity and population, and notes that "The MMM counterfactual is zero spend", so a test that only reduced spend measures marginal lift and will not line up with the model's channel ROI. Robyn's Analyst's Guide asks for the same alignment: "the same level of granularity, same metrics measured, and within the same period".
Our reading is a pass when the experiment's interval and the model's interval (a credible interval in Meridian, a bootstrapped one in Robyn) overlap for the same channel, KPI and weeks, and a fail when they sit apart. A fail is still useful: the test goes into calibration, the model is refit, and the optimizer runs again before any budget moves. Our guide to analyzing an incrementality test result covers the readout side.
Holding a test back has a cost, because the held-out channel is estimated without its best evidence. With two or more tests, we hold out the one on the channel the optimizer wants to move most. With a single test, we calibrate on it and let the staged shift do the validating.
Stage the shift so it tests the recommendation
Meridian's optimizer already bounds the first move. In the fixed-budget scenario, the optimization guide from Google gives spend_constraint_lower and spend_constraint_upper a default of 0.3, which keeps each channel between 70% and 130% of its spend under the starting allocation. Robyn's allocator also works within media-level constraints, and the Budget Allocator section of its Analyst's Guide says: "Please consider validating outputs before the implementation."
A staged shift does that validating with money. Every figure in this example is hypothetical: a model recommends moving $60,000 a month from paid search to paid social and predicts $120,000 in added monthly revenue, with a 90% credible interval of $40,000 to $200,000. We assume the effect scales with each market's share of spend.
| Stage | What moves | Prediction or check | Go signal |
|---|---|---|---|
| 1. Geo pilot, six weeks | The recommended mix in markets carrying 25% of spend, or $15,000 a month | $30,000 a month, interval $10,000 to $50,000 | Measured lift overlaps the prediction and its interval excludes zero |
| 2. National shift | The remaining $45,000 a month | $90,000 a month, interval $30,000 to $150,000 | Stage 1 passed |
| 3. Refit | Nothing | The five checks, rerun on the new weeks | Each passes or has a reviewed explanation |
Stage 1 is a geo experiment on the recommendation itself: the new mix runs in test markets, the others stay on the current mix, and the difference is read the way our quarterly geo-holdout guide describes. The pilot measures a marginal change between two mixes, so we compare it with the model's predicted change for that reallocation and, given the estimand note above, keep it out of a zero-spend ROI prior. If the measured lift falls below the predicted interval, the shift stops at the pilot and the channel earns a go-dark test before more money follows.
Platform-reported conversions need a separate correction, which the multiplier method covers, and our primer on what incrementality means explains why a lift test outranks a fit metric. MMM and attribution work sits with our data and analytics team, and the Attribution Doctor is a further resource on the attribution side.
Sources
- Google, Meridian guide "Assess the model fit and the results", last updated September 28, 2026
- Google, Meridian guide "Model health score", last updated September 2, 2026
- Google, Meridian guide "Model health checks", last updated October 2, 2026
- Google, Meridian guide "Holdout observations (train and test split)", last updated September 28, 2026
- Google, Meridian guide "Calibrate treatment priors", last updated September 24, 2026
- Google, Meridian guide "Budget optimization scenarios", last updated August 3, 2026
- Google, Meridian guide "Channel calibration recommendation and score", last updated September 11, 2026
- Meta, Robyn documentation, "Key Features"
- Meta, Robyn documentation, "An Analyst's Guide to MMM"