Three nested estimators on the California Proposition 99 panel. Each keeps everything the previous one assumed except a single restriction. Switch between them and watch the counterfactual — and the uncertainty around it — change.
California against its synthetic
The gap
What to look for
Every estimator tracks California closely before 1988 — that is what they are fitted to do. They disagree about the level of the counterfactual afterwards, and that disagreement is the whole spread of the ATT. The BSCM path sits apart from the other three because it fits an explicit intercept the others do not; it is answering a slightly different question, not getting the same question wrong.
Every estimator that has been run on this panel
Reading the forest plot honestly
Teal markers target the same estimand as the three stages. Grey markers do not.
SPILLSYNTH(cd) measures against a demeaned leave-one-out baseline,
SpSyDiD reports three effects at once, and ISCM uses a different
normalisation. Putting them in one column without that distinction would be the easiest way to
draw a false conclusion from a tidy-looking table. Hover any row for its estimand.
The simplex forces donor weights to be non-negative and to sum to one. A horseshoe prior only makes zero likely. Same data, same pre-treatment fit — a very different answer to "which states resemble California?"
Simplex against horseshoe, all 38 donors
The question this tab asks
Which states does the constraint invent, and which does it erase? Thirty-three donors receive exactly zero under the simplex. Under the horseshoe most of them receive something — and almost none of those credible intervals exclude zero. The broader pool is not more informative; it is more honest about how little the pre-treatment fit pins down.
The spatial intensity ρ controls how much of California's policy reaches its neighbours. Move it and watch the spillovers and the ATT respond. No MCMC runs in the browser: the ATT is read off a 31-point sweep computed in Python, and the per-state spillovers are the fitted panel at ρ̂ rescaled linearly in ρ. The ranking is exact; the levels away from ρ̂ are an approximation, not a refit.
Where the policy leaked
The ten largest spillovers at this ρ
What to look for
At ρ = 0 every spillover is zero by construction and the ATT equals the plain Bayesian synthetic control. As ρ rises, Nevada moves first and moves furthest — it is the only donor that touches California. Idaho and Utah follow, one step further out on the contiguity graph, and everything beyond that second ring stays flat. The estimated ρ is marked on the slider; the concentration you see there is the finding, not a scaling choice.
The published R analysis of this model reported a 95% credible interval 0.38 packs wide. The corrected configuration reports 12.7. Two different things were wrong, and this tab separates them.
The ρ chain: fixed step against adaptive step
Failure one: the interval was unreliable
Thin the chain and watch what does not happen. Keeping every second draw halves the sample and leaves the effective sample size almost unchanged — that is the point of ESS: it counts information, not rows, and the draws you discard were largely redundant. What the comparison does show is the gap between the two chains. A chain of a quarter of a million highly correlated draws can be worth a few dozen independent ones, and quantiles computed from an ESS in single digits are not posterior summaries — they describe where the chain happened to wander. The orange trace is the R specification's fixed Metropolis step; the teal one is Robbins–Monro adaptation targeting 44% acceptance. (Past k = 20 the retained sample is too small for the estimator itself to be reliable, which is why the slider stops there.)
What each fix buys
Failure two: the interval was incomplete
Running the R specification for a hundred times as many iterations widens the interval by 45%. Propagating the donor-weight uncertainty instead of pinning α at its posterior mean widens it by 1,700%. Effective sample size asks whether an interval is reliable; what you condition on asks whether it is complete. The published interval failed both, which is why it came out 33 times too narrow.
How long must the chain be?
Why half a million draws
The estimand settles early: the ATT is stable from about 100,000 draws. The nuisance parameter does not — ESS(ρ) needs roughly five times that to clear the conventional floor of 100. The binding constraint is the data, not the computer: there is exactly one contiguity channel out of California, so the panel holds one state's worth of evidence about how strongly policies leak.
Glossary
- Potential outcomes under interference
- Each unit has an outcome it would show under every possible assignment of treatment across all units — not just its own. Classical synthetic control assumes a donor's outcome depends only on its own (untreated) status; interference is what happens when that fails.
- ATT
- Average treatment effect on the treated: California's observed sales minus its reconstructed no-treatment sales, averaged over 1988–2000.
- Donor pool
- The 38 states with no large tobacco-control programme of their own, from which the counterfactual is built.
- Simplex
- The constraint that weights be non-negative and sum to one. Guarantees interpolation; forces sparsity.
- Horseshoe prior
- A shrinkage prior with an infinite spike at zero and Cauchy tails. Makes zero the default without making it compulsory.
- SUTVA
- The assumption that one unit's treatment does not affect another unit's outcome. What Stage 3 drops.
- ρ
- Spatial autoregressive intensity — how strongly a unit's outcome depends on its neighbours'. At ρ = 0 the spatial model collapses to the Bayesian one.
- Spillover effect
- A donor's observed outcome minus the outcome it would have had if California had never been treated. Negative means the donor sold fewer packs.
- ESS
- Effective sample size: how many independent draws an autocorrelated chain is worth.
- propagate_alpha
- Whether the credible interval accounts for uncertainty in the donor weights, or holds them fixed at their posterior mean.