Tutorials | Carlos Mendez

Introduction to the Synthetic Control Method in Python with mlsynth

Sun, 04 Oct 2026 00:00:00 +0000

Abstract

Cigarette sales across the United States peaked in the mid-1970s and fell steadily through the 1980s. This tutorial asks how much Proposition 99, the tobacco control program that California began in 1989, reduced cigarette sales per capita. Because sales were already falling everywhere, a simple before-and-after comparison cannot isolate the effect of this single state policy. The tutorial therefore introduces the synthetic control method and the Python library mlsynth. The data, shared with the Stata edition, form a balanced panel of 39 US states from 1970 to 2000 (1,209 observations). The VanillaSC estimator of mlsynth matches California on four covariates and on cigarette sales in 1975, 1980, and 1988. Synthetic California combines five states, led by Utah, and tracks California before 1989 with a root mean squared error of 1.754 packs. The average treatment effect on the treated (ATT) is −18.98 packs per capita per year over 1989–2000, a reduction of 23.9 percent relative to mean synthetic sales. By 2000, sales are 38.2 percent below the synthetic path, a gap to which a second tax increase in 1999 may also contribute. The baseline estimates closely reproduce the Stata benchmark, which reports an ATT of −19.00. In-space placebos, an in-time placebo, and leave-one-out refits then try to break the result. California ranks first among 39 states in the placebo test (p = 0.026), and leave-one-out estimates stay between −19.29 and −17.52 packs. A fake start in 1985, however, yields gaps about one third as large as the real effect. Despite this caveat about timing, the evidence indicates that a program combining a cigarette tax with anti-smoking education can reduce cigarette sales substantially and persistently.

1. Overview

In November 1988, California voters approved Proposition 99. The measure raised the state cigarette tax by 25 cents per pack from January 1989 and funded anti-smoking education. Cigarette sales, however, were already falling across the country, so the decline after 1989 cannot be credited to the program without a comparison. We therefore ask a counterfactual question: how much lower were sales in California because of Proposition 99? The counterfactual is the path of sales that California would have recorded without the program, and no data set observes it directly.

The synthetic control method answers this question with a weighted average of other states. The weights make the average reproduce California before 1989, so the same average after 1989 estimates sales without the program. Abadie, Diamond, and Hainmueller (2010) developed the method with this very case, building on Abadie and Gardeazabal (2003). This tutorial follows their design.

The Python library mlsynth, written by Jared Greathouse, implements the method together with many related estimators behind one configuration dictionary. An estimator, in this sense, is a rule that turns data into an estimate. This tutorial teaches the method and the library together, one step at a time.

The tutorial also replicates every step of the Stata edition, which uses the synth2 command of Yan and Chen (2023). The Stata comparison serves only as a cross-check. Readers who do not use Stata can therefore skip the Stata benchmark boxes and Section 11, and they can ignore the Stata values inside some code blocks.

This post is the entry point to a series of tutorials on synthetic control in Python. The Bayesian spatial post revisits Proposition 99 and models spillovers to neighboring states. The synthetic control ladder applies one mlsynth estimator per rung, from difference-in-differences to synthetic difference-in-differences, to the Brexit referendum. Readers who want a contrasting design can turn to the introduction to difference-in-differences, a companion post in the same format.

1.1 Learning objectives

This tutorial has two aims. It explains the logic of the synthetic control method, and it teaches the mlsynth library one step at a time. By the end of the tutorial, you will be able to:

Explain why a before-and-after comparison and the average of other states give misleading counterfactuals for a single treated state.
Build a synthetic control with the VanillaSC class of mlsynth from one configuration dictionary, with covariates and past values of sales (lagged outcomes) as predictors.
Read the donor weights, predictor weights, and predictor balance from the result object, and judge the pre-treatment fit.
Compute the gap path and the ATT, and express the ATT in packs and in percent.
Assess significance with an in-space placebo test, the ratio of mean squared prediction errors (MSPE ratio), and the permutation p-value. Refine the test with the cut(2) filter, which drops badly fitted placebo states, and with pointwise p-values.
Check robustness with an in-time placebo and leave-one-out refits, and compare the result with outcome-only synthetic control, synthetic difference-in-differences (SDID), and principal component regression (PCR) in mlsynth.

1.2 Study design

The diagram summarizes the study design in three blocks. The first block describes the case: one treated state and 38 possible donor states. The second block builds synthetic California with mlsynth, and the third block tries to break the result with three robustness checks.

graph LR
subgraph SG1["Case study"]
A("<b>39 US states</b><br/>1970–2000, 1,209 rows")
B("<b>California</b><br/>Proposition 99 from 1989")
C("<b>38 donor states</b><br/>no large tobacco program")
A --> B
A --> C
end
subgraph SG2["Synthetic control in mlsynth"]
D("<b>Seven predictors</b><br/>4 covariates, 1980–1988<br/>sales in 1975, 1980, 1988")
E("<b>VanillaSC</b><br/>nested search over<br/>predictor weights V<br/>and donor weights W<br/>5 donors with weight")
F("<b>Gap and ATT</b><br/>ATT = −18.98 packs")
D --> E --> F
end
subgraph SG3["Can we break it?"]
G("<b>In-space placebo</b><br/>rank 1 of 39, p = 0.026")
H("<b>In-time placebo</b><br/>fake start in 1985")
I("<b>Leave-one-out</b><br/>5 refits")
end
J("<b>Estimator tour</b><br/>outcome-only, SDID, PCR")
B --> D
C --> D
F --> G
F --> H
F --> I
F --> J
style SG1 fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style SG2 fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style SG3 fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef violet fill:#1f2b5e,stroke:#a78bfa,stroke-width:3px,color:#e8ecf2
class A,C blue
class B orange
class D,E gray
class F,G,H,I teal
class J violet

The colors of the diagram encode the role of each box. The blue boxes describe the data, the orange box marks the single treated state, and the gray boxes build synthetic California. The teal boxes report or test its result, and the violet box marks the extension, in which the same data feed three other mlsynth estimators.

1.3 Key concepts at a glance

This tutorial relies on a small vocabulary. Each concept below has a definition, an example, and an analogy. The definition is always visible, while the example and the analogy open with a click. Readers who meet an unfamiliar term later, such as the MSPE ratio or the donor weights, can return to this section.

1. Synthetic control method (SCM). The synthetic control method builds a comparison unit from a weighted average of untreated units. The weights make the comparison unit reproduce the treated unit before the policy. After the policy, the comparison unit estimates what would have happened without it.

Example

We build a synthetic California from 38 donor states. Only a few states receive positive weight, and their weighted sales track California closely from 1970 to 1988. The typical miss, the root mean squared error (RMSE) that concept 5 defines, is only 1.754 packs per capita. After 1989, the synthetic path estimates the sales that California would have recorded without Proposition 99.

Analogy

A sound engineer must replace a singer who cannot record the final chorus. The engineer blends recordings of other singers until the blend matches the original voice in every verse. The blend then sings the chorus that the original singer would have sung.

2. Donor pool. The donor pool is the set of untreated units that may enter the synthetic control. Each donor must be free of the treatment and of similar policies during the study period. A contaminated donor pool produces a misleading counterfactual.

Example

Our donor pool contains the 38 states other than California in the data file. Abadie, Diamond, and Hainmueller (2010) had already removed states with large tobacco programs or large tax increases, such as Arizona, Massachusetts, and Oregon. The code therefore uses every remaining state as a potential donor.

Analogy

A casting director auditions only actors who have never played the role. An actor who already played it would bring habits from that production. The audition list must be clean before the casting starts.

3. Donor weights $w_j$ (the vector W). Donor weights state how much each donor contributes to the synthetic control. They cannot be negative, and they must sum to one. These two rules keep the synthetic control inside the range of the donors and make it easy to read.

Example

Suppose that a recipe put three quarters of the weight on a donor state with low sales and one quarter on a donor state with high sales. In every year, synthetic sales would then equal three quarters of the sales of the first state plus one quarter of the sales of the second. Because both weights are nonnegative and sum to one, the synthetic path always lies between the two donors. Section 7.3 reports the weights that mlsynth actually chooses.

Analogy

A smoothie recipe lists the share of each fruit in the glass. No share can be negative, because nobody can remove fruit that was never added. The shares must add up to one full glass.

4. Predictors and predictor weights $v_m$ (the diagonal of the matrix V). Predictors are the characteristics on which the synthetic control must resemble the treated unit. They include covariates and outcome values from chosen pre-treatment years. Covariates are characteristics of a state other than the outcome, such as GDP per capita or the retail price of cigarettes. Predictor weights set how much a mismatch on each predictor counts in the fit.

Example

We use four covariates averaged over 1980–1988 and cigarette sales in 1975, 1980, and 1988. A large weight on sales in 1975, for example, would make any mismatch on that predictor costly, so the fit would match it closely. Section 7.4 compares the predictor weights that mlsynth and Stata choose for these seven predictors.

Analogy

A tailor takes many body measurements before cutting a suit. Two tailors may stress different measurements and still cut the same suit. The customer wears the suit, not the list of priorities.

5. Pre-treatment fit. Pre-treatment fit measures how closely the synthetic control tracks the treated unit before the policy. Its usual summary is the root mean squared error (RMSE) of the pre-treatment gaps, the yearly differences between the treated unit and its synthetic control. The RMSE is the square root of the average squared gap, so it measures the typical miss in packs per capita. A close fit is necessary for a credible counterfactual, but it is not sufficient.

Example

The RMSE of synthetic California over 1970–1988 is 1.754 packs per capita. This miss is about 1.5 percent of the average sales of 116.21 packs in those years. The largest single miss is 5.90 packs, in 1970. By this yardstick, the fit is close, and the miss of 1970 is an exception at the very start of the series.

Analogy

A weather service earns trust by forecasting past days well. A service that missed every past storm deserves little trust tomorrow. A strong record is still no guarantee, because tomorrow can bring a new kind of storm.

6. Gap and ATT. The gap is the difference between the treated unit and its synthetic control in a given year. The average treatment effect on the treated (ATT) is the mean gap over the post-treatment years. It measures the effect on the treated unit only.

Example

The gap is −7.59 packs in 1989 and widens to −25.73 packs by 2000. Averaged over 1989–2000, the ATT is −18.98 packs per capita per year. Relative to the synthetic path, this is a reduction of 23.9 percent.

Analogy

A runner follows a pacer for the first half of a race. In the second half, the runner tries a new drink, and the distance to the pacer changes. The distance at each kilometer is the gap, and its average is the effect of the drink on that runner.

7. In-space placebo and MSPE ratio. An in-space placebo test applies the same method to every donor as if it had been treated. The mean squared prediction error (MSPE) is the average squared gap. The MSPE ratio divides the MSPE after the policy by the MSPE before the policy. The permutation p-value is the share of units whose ratio is at least as large as that of the treated unit. If the policy mattered, the ratio of the treated unit should be unusually large, and its p-value small.

Example

Suppose that one state among the 39 had the largest MSPE ratio of all. It would rank first, and its permutation p-value would be 1/39 = 0.026, the smallest value that 39 states allow. A state that ranked second would have a p-value of 2/39 = 0.051, so the evidence weakens as a state moves down the ranking.

Analogy

A teacher suspects one student because the score of that student jumped. The teacher compares that jump with the jumps of every other student. The more unusual the jump is within the class, the stronger the suspicion.

8. In-time placebo and leave-one-out. An in-time placebo moves the treatment date to a year before the real policy. A leave-one-out check removes one important donor at a time and refits the model. Both tests try to break the result in ways that a real effect should survive.

Example

Both checks have direct counterparts in this tutorial. With a fake start in 1985, the model is fitted to 1970–1984, and the fake gaps for 1985–1988 average −5.97 packs. The same fit gives a mean gap of −18.75 over 1989–2000, so the fake gaps are about one third as large. The leave-one-out check refits the model five times, once without each of the five donors that receive weight, and Section 10 reports how far these estimates move.

Analogy

An engineer tests a bridge in two ways. She first checks the gauges on a day when no truck crosses. She then removes one support at a time and checks that the deck still holds.

These eight concepts cover the vocabulary that the walkthrough assumes. The next section installs mlsynth and records the software stack, because the placebo results depend on it. Section 3 then loads the data and checks their structure.

2. Setup and imports

The analysis needs one specialized library, mlsynth, and a few standard ones. We pin the version of mlsynth, that is, we install one exact release, because the library is young and changes quickly. The following command installs it together with its dependencies:

pip install mlsynth==1.0.0

A version pin alone is not enough, for two reasons. First, the version number does not identify the code. PyPI, the Python Package Index from which pip installs packages, serves release 1.0.0. Later commits on GitHub, that is, saved versions of the code such as 15f168b, print the same version number, and some of them behave differently. Second, the placebo fits of the donor states depend on the optimizer, the numerical search routine that finds the weights. Another software stack can thus move a few numbers in the third decimal. For both reasons, the block below prints the versions of the whole stack and a build fingerprint, and every number in this post comes from that stack.

import platform
import warnings
from importlib import metadata
from pathlib import Path
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from matplotlib.lines import Line2D
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
from scipy.optimize import minimize
from mlsynth import CLUSTERSC, SDID, VanillaSC
from mlsynth.exceptions import MlsynthConfigError, MlsynthDataError
from mlsynth.utils.vanillasc_helpers.config import VanillaSCConfig
warnings.filterwarnings("ignore", category=DeprecationWarning)
RANDOM_SEED = 42 # the seed of the Stata do-file
np.random.seed(RANDOM_SEED)
# The versions that produced every number in this post
print(f"Python {platform.python_version()}")
for package in ["mlsynth", "numpy", "pandas", "scipy", "matplotlib",
"cvxpy", "scs", "statsmodels"]:
print(f" {package:<12} {metadata.version(package)}")
# Build fingerprint: the PyPI release of 1.0.0 has no fit_window field,
# while the GitHub build 15f168b has one
build = "pypi" if "fit_window" not in VanillaSCConfig.model_fields else "git"
print(f"mlsynth build: {build}")

Python 3.13.11
mlsynth 1.0.0
numpy 2.3.5
pandas 3.0.1
scipy 1.17.1
matplotlib 3.10.8
cvxpy 1.8.1
scs 3.2.8
statsmodels 0.14.6
mlsynth build: pypi

The output confirms mlsynth 1.0.0 from PyPI, together with numpy 2.3.5, pandas 3.0.1, and scipy 1.17.1. The fingerprint reads pypi: the line checks whether the installed version has a configuration field, fit_window, that only the GitHub version has. Readers on Google Colab may see other versions of the standard libraries, which can move a few placebo results slightly. The table below summarizes the role of each package in this post.

Package	Purpose
`mlsynth`	Synthetic control estimators (`VanillaSC`, `SDID`, and `CLUSTERSC`) behind one configuration dictionary
`pandas` and `numpy`	Data loading, reshaping, and arithmetic
`matplotlib`	All figures, drawn with the dark theme of the site
`scipy`	The inner weight problem of Exercise 6
`statsmodels`	The two-way fixed effects regression of Exercise 5

All figures in this post use the dark theme of the site. The collapsed block below sets the colors and the font sizes once, so every later figure inherits them. It also defines a helper, signed(), that writes negative numbers in figure labels with a true minus sign.

Show the dark figure theme

# Site color palette
STEEL_BLUE = "#6a9bcc"
WARM_ORANGE = "#d97757"
NEAR_BLACK = "#141413"
TEAL = "#00d4c8"
# Dark theme palette
DARK_NAVY = "#0f1729"
GRID_LINE = "#1f2b5e"
LIGHT_TEXT = "#c8d0e0"
WHITE_TEXT = "#e8ecf2"
# Extra colors for donors, Stata values, and the robustness checks
GREY_DONOR = "#54618a"
GOLD = "#e8b04b"
LIGHT_ORANGE = "#e8956a"
LAVENDER = "#a58fd8"
SAGE = "#8fbf7f"
plt.rcParams.update({
"figure.facecolor": DARK_NAVY,
"axes.facecolor": DARK_NAVY,
"axes.edgecolor": DARK_NAVY,
"axes.linewidth": 0,
"axes.labelcolor": LIGHT_TEXT,
"axes.titlecolor": WHITE_TEXT,
"axes.spines.top": False,
"axes.spines.right": False,
"axes.spines.left": False,
"axes.spines.bottom": False,
"axes.grid": True,
"grid.color": GRID_LINE,
"grid.linewidth": 0.6,
"grid.alpha": 0.8,
"xtick.color": LIGHT_TEXT,
"ytick.color": LIGHT_TEXT,
"xtick.major.size": 0,
"ytick.major.size": 0,
"text.color": WHITE_TEXT,
"font.size": 12,
"legend.frameon": False,
"legend.fontsize": 11,
"legend.labelcolor": LIGHT_TEXT,
"figure.edgecolor": DARK_NAVY,
"savefig.facecolor": DARK_NAVY,
"savefig.edgecolor": DARK_NAVY,
})
# The same colors for the native res.plot() figures of mlsynth
MLSYNTH_DARK_THEME = {
"figure.facecolor": DARK_NAVY, "axes.facecolor": DARK_NAVY,
"savefig.facecolor": DARK_NAVY, "text.color": WHITE_TEXT,
"axes.labelcolor": LIGHT_TEXT, "axes.titlecolor": WHITE_TEXT,
"xtick.color": LIGHT_TEXT, "ytick.color": LIGHT_TEXT,
"grid.color": GRID_LINE, "grid.alpha": 0.8,
"legend.facecolor": DARK_NAVY, "legend.edgecolor": GRID_LINE,
"legend.labelcolor": LIGHT_TEXT, "font.sans-serif": ["DejaVu Sans"],
"font.weight": "normal", "axes.titlesize": 13, "axes.labelsize": 12,
"xtick.labelsize": 11, "ytick.labelsize": 11, "legend.fontsize": 11,
}
def signed(x, nd=2):
"""Format a number for figure text, with a Unicode minus sign."""
x = round(float(x), nd) + 0.0 # adding 0.0 turns -0.0 into 0.0
return f"{x:.{nd}f}".replace("-", "−")

The environment is now fixed, and every figure shares one theme. The next section loads the data and checks their structure. These checks come first, because every later number depends on them.

3. Data loading and exploration

3.1 Load the data

The data are the panel of Abadie, Diamond, and Hainmueller (2010), distributed by the QuaRCS Lab as a Stata file. The helper load_data() reads a local CSV copy when one exists and falls back to the Stata file on GitHub otherwise. Both paths return identical numbers, because the CSV stores the exact values of the Stata file.

DATA_URL = "https://github.com/quarcs-lab/data-open/raw/master/isds/smoking_sc.dta"
CSV_CANDIDATES = (Path("data") / "smoking_sc.csv", Path("smoking_sc.csv"))
NUMERIC = ["cigsale", "lnincome", "beer", "age15to24", "retprice"]
def load_data(candidates=CSV_CANDIDATES, url=DATA_URL):
"""Load the panel: a local CSV copy first, the Stata file otherwise."""
for path in candidates:
if Path(path).exists():
return pd.read_csv(path, float_precision="round_trip"), str(path)
raw = pd.read_stata(url) # state is a labeled categorical
df = raw.assign(state=raw["state"].astype(str), year=raw["year"].astype(int))
df[NUMERIC] = df[NUMERIC].astype("float64") # exact float32 to float64
return df[["state", "year", *NUMERIC]], url
df, source = load_data()
print(f"Loaded {len(df)} rows from {source}")
print(f"Shape: {df.shape}")
print(df.head().round(3))

Loaded 1209 rows from data/smoking_sc.csv
Shape: (1209, 7)
state year cigsale lnincome beer age15to24 retprice
0 Alabama 1970 89.8 NaN NaN 0.179 39.6
1 Alabama 1971 95.4 NaN NaN 0.180 42.7
2 Alabama 1972 101.1 9.498 NaN 0.181 42.3
3 Alabama 1973 102.9 9.550 NaN 0.182 42.1
4 Alabama 1974 108.2 9.537 NaN 0.183 43.1

The panel has 1,209 rows and seven columns: the state, the year, the outcome cigsale, and four covariates. The source line names the local CSV file, and on Google Colab it shows the GitHub URL instead, with the same numbers. Missing values (NaN) appear already in the first rows, because lnincome and beer are not observed in the early years. The next two blocks measure how much of each variable is missing.

Summary statistics reveal the scale and the coverage of each variable. The count column is especially useful here, because it shows how many state-year cells each variable fills. We print the statistics with the describe() method of pandas.

print(df[NUMERIC].describe().T.round(3))

 count mean std min 25% 50% 75% max
cigsale 1209.0 118.893 32.767 40.700 100.900 116.300 130.500 296.200
lnincome 1014.0 9.862 0.171 9.397 9.739 9.861 9.973 10.487
beer 546.0 23.430 4.223 2.500 20.900 23.300 25.100 40.400
age15to24 819.0 0.175 0.015 0.129 0.166 0.178 0.187 0.204
retprice 1209.0 108.342 64.382 27.300 50.000 95.500 158.400 351.200

Cigarette sales average 118.89 packs per capita and range from 40.70 to 296.20 across states and years. The variable age15to24 lies between 0.13 and 0.20. It is therefore a share of the population, although the original file labels it as a percent. Sales and the retail price are complete, with 1,209 observations each, but lnincome has 1,014 observations, age15to24 has 819, and beer only 546. These counts already suggest that some predictor averages will rest on only part of their windows, a point that Section 3.2 checks.

3.2 Panel structure and coverage

A synthetic control needs a balanced panel, in which every state appears in every year. It also needs predictors that are observed in the years over which we average them. The next block checks the panel structure and records the first and the last year with data for each variable.

TREATED = "California"
STATES = df["state"].drop_duplicates().tolist() # Stata order: California is third
DONORS = [s for s in STATES if s != TREATED]
rows_per_state = df.groupby("state").size()
print(f"States: {len(STATES)}; donor states: {len(DONORS)}")
print(f"Years: {df['year'].nunique()} ({df['year'].min()} to {df['year'].max()}); "
f"rows per state: {rows_per_state.min()} to {rows_per_state.max()}")
coverage = pd.DataFrame([{
"variable": c,
"n_obs": int(df[c].notna().sum()),
"first_year": int(df.loc[df[c].notna(), "year"].min()),
"last_year": int(df.loc[df[c].notna(), "year"].max())} for c in NUMERIC])
print(coverage.to_string(index=False))

States: 39; donor states: 38
Years: 31 (1970 to 2000); rows per state: 31 to 31
variable n_obs first_year last_year
cigsale 1209 1970 2000
lnincome 1014 1972 1997
beer 546 1984 1997
age15to24 819 1970 1990
retprice 1209 1970 2000

The panel is balanced, with 39 states, 31 years from 1970 to 2000, and 31 rows per state. California is the treated state, so the other 38 states form the donor pool. Coverage differs sharply across variables, since beer consumption starts only in 1984 and the age share ends in 1990. The beer average over 1980–1988 therefore rests on 1984–1988 only, a fact that will later limit the in-time placebo.

Average sales also differ widely across states. This spread matters for the method. A weighted average of donors can reach California only if some donors sell less than California and others sell more. The block below lists the three lowest and the three highest averages over 1970–2000, the full period of the data. It then ranks California over 1970–1988 alone, because the later years already contain the effect of the program.

state_means = df.groupby("state")["cigsale"].mean().sort_values()
print("Lowest average sales, 1970 to 2000 (packs per capita):")
print(state_means.head(3).round(2).to_string(header=False))
print("Highest average sales, 1970 to 2000 (packs per capita):")
print(state_means.tail(3).round(2).to_string(header=False))
pre_means = df[df["year"] <= 1988].groupby("state")["cigsale"].mean().sort_values()
print(f"California, 1970 to 1988: {pre_means['California']:.2f} packs, rank "
f"{pre_means.index.get_loc('California') + 1} of {len(pre_means)} from the lowest")

Lowest average sales, 1970 to 2000 (packs per capita):
Utah 63.84
New Mexico 84.26
California 94.59
Highest average sales, 1970 to 2000 (packs per capita):
North Carolina 164.29
Kentucky 187.94
New Hampshire 213.06
California, 1970 to 1988: 116.21 packs, rank 14 of 39 from the lowest

Utah has the lowest average, 63.84 packs per capita, and New Hampshire the highest, 213.06 packs. California ranks third lowest over 1970–2000, at 94.59 packs, but that average includes the program years from 1989 onward, when sales in California fell steeply. Over 1970–1988, the years that the fit uses, California averages 116.21 packs and ranks 14th of 39, so 13 states sell less and 25 sell more. A weighted average of donors can therefore reach the level of California. Utah, which has by far the lowest sales, is a natural ingredient for synthetic California.

The table below describes the seven variables. The name lnincome suggests income, but the variable measures log GDP per capita, and age15to24 is a share rather than a percent. The role column anticipates how each variable enters the synthetic control.

Variable	Description	Role
`state`	State name (39 states)	Panel unit
`year`	Year, 1970–2000	Time variable
`cigsale`	Cigarette sales per capita, in packs	Outcome
`lnincome`	Log of state GDP per capita	Predictor, averaged over 1980–1988
`age15to24`	Share of the population aged 15–24 (a fraction)	Predictor, averaged over 1980–1988
`retprice`	Average retail price of cigarettes	Predictor, averaged over 1980–1988
`beer`	Beer consumption per capita	Predictor, averaged over 1984–1988, the years with data

Stata benchmark. The summarize command in the Stata log reports the same counts: 1,209 observations of cigsale, 1,014 of lnincome, 546 of beer, and 819 of age15to24. Its mean of cigsale, 118.8932, matches the Python mean. The xtsum table lists a lowest state mean of 63.84 packs and a highest of 213.06 packs, the averages of Utah and New Hampshire. The two programs therefore start from identical data.

The data are clean and balanced, and their coverage limits are known. Before we build any synthetic control, we ask whether a simpler comparison would suffice. The answer shows why the synthetic control method needs weights.

4. Raw trends: California versus the average donor state

The simplest counterfactual for California is the average of the 38 donor states. If California had moved in step with that average before 1989, the average after 1989 would be a reasonable guess for California without the program. We test this idea by comparing the two series year by year.

Predict first

Before 1989, sales in California and in the other states rose until the mid-1970s and then fell. Did California follow the average of the 38 donor states closely enough for that average to serve as its counterfactual? Will the gap between California and the donor average in 1988 be smaller than, similar to, or larger than the gap in 1970? Commit to an answer before scrolling.

Reveal the answer

Answer. No, it did not, and the gap in 1988 is far larger than the gap in 1970. The two series were close in 1970, at 123.0 against 120.08 packs, a gap of 2.92. By 1988, however, California sold 90.1 packs against 113.82 for the average, a gap of −23.72. A comparison with the average would therefore attribute to Proposition 99 a relative decline that began long before 1989.

TREAT_YEAR, FIRST_YEAR, LAST_YEAR = 1989, 1970, 2000
T0 = TREAT_YEAR - FIRST_YEAR # 19 pre-treatment years, 1970–1988
# Wide table: one row per year (31) and one column per state (39), in Stata order
Y = df.pivot(index="year", columns="state", values="cigsale")[STATES]
YEARS = Y.index.to_numpy()
ca_sales = Y[TREATED].to_numpy()
donor_avg = Y[DONORS].mean(axis=1).to_numpy()
trend = pd.DataFrame({"year": YEARS, "california": ca_sales,
"donor_average": donor_avg,
"difference": ca_sales - donor_avg})
print(trend[trend["year"].isin([1970, 1975, 1980, 1985, 1988, 1989, 1995, 2000])]
.round(2).to_string(index=False))
sales_1988 = Y.loc[1988, DONORS]
print(f"Donor range in 1988: {sales_1988.min():.1f} ({sales_1988.idxmin()}) to "
f"{sales_1988.max():.1f} ({sales_1988.idxmax()})")

 year california donor_average difference
1970 123.0 120.08 2.92
1975 127.1 136.93 -9.83
1980 120.2 138.09 -17.89
1985 102.8 123.12 -20.32
1988 90.1 113.82 -23.72
1989 82.4 109.66 -27.26
1995 56.4 103.16 -46.76
2000 41.6 92.13 -50.53
Donor range in 1988: 55.0 (Utah) to 180.4 (New Hampshire)

California started 2.92 packs above the donor average in 1970 but sat 23.72 packs below it in 1988. The difference kept growing after 1989 and reached −50.53 packs in 2000. The simple average therefore drifts away from California long before the program.

The figure below shows every donor state, their average, and California. The thin gray lines reveal how different the donor states are from one another. The dashed blue line is the average that the naive comparison would use.

Show the plotting code

fig, ax = plt.subplots(figsize=(11, 6))
fig.patch.set_linewidth(0)
for s in DONORS:
ax.plot(YEARS, Y[s], color=GREY_DONOR, linewidth=0.8, alpha=0.6, zorder=1)
ax.plot([], [], color=GREY_DONOR, linewidth=1.2, label="Individual donor states")
ax.plot(YEARS, donor_avg, color=STEEL_BLUE, linewidth=2.6, linestyle="--",
label="Average of the 38 donor states", zorder=3)
ax.plot(YEARS, ca_sales, color=WARM_ORANGE, linewidth=3.0, label="California",
zorder=4)
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5, zorder=2)
ax.text(TREAT_YEAR + 0.4, 285, "Proposition 99 (1989)", color=LIGHT_TEXT,
fontsize=11, va="top")
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_ylim(0, 300)
ax.set_xlabel("Year", fontsize=12)
ax.set_ylabel("Cigarette sales (packs per capita)", fontsize=12)
ax.set_title("Cigarette sales in California and the 38 donor states, 1970–2000",
fontsize=14, fontweight="bold", pad=12)
ax.legend(loc="lower left", fontsize=11)
plt.tight_layout()
plt.show()

Figure 1. Cigarette sales in California and the 38 donor states, 1970–2000. California falls faster than the average donor long before Proposition 99.

The figure adds a view that the table only hints at: the donor states differ widely from one another in every year. In 1988, for example, their sales range from 55.0 packs in Utah to 180.4 packs in New Hampshire, and many of them sell far more than California. The simple average gives every donor the same voice, and 25 of the 38 donors sell more than California, on average, over 1970–1988. The average therefore sits well above California after 1970 and cannot reproduce the path of California before 1989.

Two naive estimates are common in policy debates. The first compares sales in California before and after 1989, and the second subtracts the change of the average donor from the change of California. We compute both, because each will serve as a benchmark for the synthetic control. In the code, the slice [:T0] keeps the 19 years from 1970 to 1988, and [T0:] keeps the 12 years from 1989 to 2000.

pre_ca, post_ca = ca_sales[:T0].mean(), ca_sales[T0:].mean()
pre_avg, post_avg = donor_avg[:T0].mean(), donor_avg[T0:].mean()
print(f"California: {pre_ca:.2f} before 1989, {post_ca:.2f} after "
f"(change {post_ca - pre_ca:.2f})")
print(f"Donor average: {pre_avg:.2f} before 1989, {post_avg:.2f} after "
f"(change {post_avg - pre_avg:.2f})")
print(f"Difference in the two changes: {(post_ca - pre_ca) - (post_avg - pre_avg):.2f}")

California: 116.21 before 1989, 60.35 after (change -55.86)
Donor average: 130.57 before 1989, 102.06 after (change -28.51)
Difference in the two changes: -27.35

Sales in California fell by 55.86 packs per capita between the two periods, but the average donor also fell, by 28.51 packs. The before-and-after change therefore mixes the program with a national decline that started long before 1989. The difference in the two changes, −27.35 packs, is the difference-in-differences (DiD) estimate, and it removes the common decline. It assumes, however, that California and the average donor would have moved in parallel without the program. The data for 1970–1988 contradict that assumption, since the gap between them grew by more than 26 packs.

Stata benchmark. The Stata edition draws the same comparison with a graph of California and the donor average. Its do-file also prints a note on the same pattern: sales in California started near the donor average in 1970. By 1988, the note adds, sales stood about 24 packs per capita below that average, a gap that the Python table puts at −23.72 packs.

The average donor is a poor twin for California. We need a counterfactual that is built to track California before 1989, and the synthetic control method provides exactly that. The next section explains how the method chooses its weights.

5. The synthetic control method

5.1 The core idea

The synthetic control method replaces the equal weights of the simple average with chosen weights. It searches for a combination of donor states whose weighted sales and characteristics match those of California before 1989. If the match holds for 19 years, the same combination after 1989 is a credible estimate of sales in California without Proposition 99.

The method has a natural limit: it interpolates between donors rather than extrapolating beyond them. Because the weights are nonnegative and sum to one, the synthetic control is a convex combination of the donors. In any year, synthetic California can therefore sell neither more than the highest donor nor less than the lowest donor. In 1988, California sold 90.1 packs, well inside the donor range from 55.0 to 180.4 packs that Section 4 printed, so a convex combination can reach it.

5.2 The weight problem

Formally, let unit 1 be California and units 2 to $J+1$ the $J = 38$ donors. Each unit has $k = 7$ predictors, and $X_{jm}$ denotes predictor $m$ of unit $j$. The method chooses the donor weights $w_j$ by solving the following problem, which we call Equation 1. Its first line defines the synthetic value of each predictor, its second line states the objective, and its third line states the constraints on the weights:

$$\hat{X}_{1m} = \sum_{j=2}^{J+1} w_j X_{jm}$$

$$\min_{w_2, \ldots, w_{J+1}} \sum_{m=1}^{k} v_m \left( X_{1m} - \hat{X}_{1m} \right)^2$$

$$w_j \geq 0 \quad \text{and} \quad \sum_{j=2}^{J+1} w_j = 1$$

In words, the problem chooses nonnegative donor weights that sum to one. The synthetic value $\hat{X}_{1m}$ is the value of predictor $m$ for the weighted average of the donors. The weights make these synthetic values resemble California on each predictor, and the predictor weight $v_m$ sets how much a mismatch on predictor $m$ counts.

The predictors come in different units, such as log GDP per capita, shares, prices, and packs. Before the fit, mlsynth therefore divides each predictor by its standard deviation across the 39 states, and the $v_m$ apply to these scaled predictors. The fit thus compares California and the donors on a common scale.

Stata and mlsynth collect the donor weights $w_j$ in a vector W. They place the predictor weights $v_m$ on the diagonal of a matrix V, whose other entries are zero. The predictor weights are not fixed in advance. The nested method chooses them so that the resulting synthetic control also reproduces the sales of California from 1970 to 1988. The proof card below states this nested problem and explains why many different V can lead to the same donor weights.

Proof The nested problem behind V and W, and why V is not unique

First, divide each predictor by its standard deviation across the 39 states. Then stack the scaled predictors of California in a vector $\mathbf{X}_1$ and those of the donors in a matrix $\mathbf{X}_0$, with one column per donor. For a vector $\mathbf{W}$ of donor weights, the vector $\mathbf{e}(\mathbf{W})$ collects the predictor mismatches between California and the weighted donors. Given a diagonal matrix $\mathbf{V}$ whose diagonal entries are the predictor weights $v_m$, the inner problem picks the donor weights. The set $\Delta$, called the simplex, contains all weight vectors with nonnegative entries that sum to one.

$$\mathbf{e}(\mathbf{W}) = \mathbf{X}_1 - \mathbf{X}_0 \mathbf{W}$$

$$Q_{\mathbf{V}}(\mathbf{W}) = \mathbf{e}(\mathbf{W})^\top \mathbf{V} \mathbf{e}(\mathbf{W})$$

$$\mathbf{W}(\mathbf{V}) = \arg\min_{\mathbf{W} \in \Delta} Q_{\mathbf{V}}(\mathbf{W})$$

In words, the inner problem restates Equation 1 in matrix form with scaled predictors. The transpose $\top$ turns the column of mismatches into a row, so $Q_{\mathbf{V}}(\mathbf{W})$ adds up the weighted squared mismatches. This sum is the weighted distance between California and the weighted donors. The operator arg min returns the weights that make this distance smallest. The solution is therefore the convex combination of donors that is closest to California under the distance that $\mathbf{V}$ defines. The solution $\mathbf{W}(\mathbf{V})$ can change when $\mathbf{V}$ changes the relative cost of the mismatches. As the last paragraph of this card shows, however, it often does not. In Exercise 6, $\mathbf{X}_1$ is the array x1, and $\mathbf{X}_0$ is the transpose of the array x0, which stores one donor per row. The diagonal of $\mathbf{V}$ is the vector v_stata, and the function inner_loss computes $Q_{\mathbf{V}}(\mathbf{W})$.

The outer problem then picks the predictor weights. It keeps the $\mathbf{V}$ whose donor weights best reproduce sales in California over the 19 pre-treatment years. The backend mscmt of mlsynth searches over $\mathbf{V}$ with differential evolution, a global search method, following Becker and Klößner (2018).

$$\hat{Y}_{1t}(\mathbf{V}) = \sum_{j=2}^{J+1} w_j(\mathbf{V}) Y_{jt}$$

$$u_t(\mathbf{V}) = Y_{1t} - \hat{Y}_{1t}(\mathbf{V})$$

$$L(\mathbf{V}) = \sum_{t=1970}^{1988} u_t(\mathbf{V})^2$$

$$\mathbf{V}^{\ast} = \arg\min_{\mathbf{V}} L(\mathbf{V})$$

In words, $u_t(\mathbf{V})$ is the yearly miss of the pre-treatment sales path $\hat{Y}_{1t}(\mathbf{V})$ that the donor weights of $\mathbf{V}$ produce. The outer loss $L(\mathbf{V})$ adds up the squares of these yearly misses over 1970–1988. The outer problem keeps the $\mathbf{V}$ that makes this loss smallest, so it judges each candidate only by that path. The final donor weights are therefore $w_j^{\ast} = w_j(\mathbf{V}^{\ast})$. In the code, $Y_{1t}$ is the array ca_sales and $Y_{jt}$ is the column of donor $j$ in the matrix Y. The search returns the predictor weights in res.weights.summary_stats["predictor_weights"].

The outer problem cares about $\mathbf{V}$ only through $\mathbf{W}(\mathbf{V})$. When a few predictors can be matched almost exactly, many different $\mathbf{V}$ select the same $\mathbf{W}$, and the outer objective cannot tell them apart. The predictor weights are then not identified: the data cannot single out one set of predictor weights. Two programs can therefore report very different $\mathbf{V}$ with the same donor weights. Section 7.4 compares the $\mathbf{V}$ of mlsynth with the $\mathbf{V}$ of Stata, and Exercise 6 tests this claim by feeding the Stata $\mathbf{V}$ into the inner problem.

For later reference, Section 7 reads these quantities from the result object res of mlsynth. The donor weights $w_j$ are res.donor_weights, and the predictor weights $v_m$ are res.weights.summary_stats["predictor_weights"]. The treated entry of res.additional_outputs["covariate_balance"] holds the predictors $X_{1m}$ of California before scaling.

5.3 The synthetic outcome, the gap, and the ATT

Once the weights are known, building the counterfactual is simple arithmetic. Synthetic California in year $t$ is the weighted average of donor sales in that year, and the gap is the difference between actual and synthetic sales. Equation 2 states the two definitions:

$$\hat{Y}_{1t}^{N} = \sum_{j=2}^{J+1} w_j^{\ast} Y_{jt}$$

$$\hat{\tau}_t = Y_{1t} - \hat{Y}_{1t}^{N}$$

In words, the synthetic outcome $\hat{Y}_{1t}^{N}$ applies the optimal weights $w_j^{\ast}$ to the sales of the donors in year $t$. These weights solve Equation 1 at the predictor weights that the nested search selects. The superscript $N$ marks the outcome without the program, and the gap $\hat{\tau}_t$ subtracts this synthetic outcome from actual sales $Y_{1t}$. In the code, the two series are res.counterfactual and res.gap, short names for res.time_series.counterfactual_outcome and res.time_series.estimated_gap.

A single number summarizes the effect. It averages the gaps over the post-treatment years, from 1989 to 2000. With $T_0 = 19$ pre-treatment years and $T_1 = 12$ post-treatment years, Equation 3 defines the estimate:

$$\widehat{\mathrm{ATT}} = \frac{1}{T_1} \sum_{t=1989}^{2000} \hat{\tau}_t$$

In words, the ATT is the mean of the 12 yearly gaps from 1989 to 2000. A negative value means that California sold fewer packs than synthetic California, on average, after the program began. The code returns this average as res.att, and res.effects.att_percent expresses it as a percent of mean synthetic sales.

5.4 Estimand and assumptions

Estimand: the ATT. An estimand is the quantity that an analysis targets. Here it is the effect of Proposition 99 on California, the only treated state. The synthetic control method does not estimate what the same program would do in Nevada or Texas. The distinction matters, because the response of California may differ from the response of states with other smoking habits and other prices.

The estimate has a causal reading only under several assumptions, which Abadie (2021) discusses in detail. None of them can be fully tested, but the robustness checks of Sections 8 to 10 probe some of them. The list below states each assumption in the context of Proposition 99.

No interference. The program must not change sales in the donor states. Cross-border purchases could violate this assumption if a neighboring state enters synthetic California, and Section 7.3 shows whether one does.
No anticipation. Sales in California must not respond to the program before 1989. Voters approved the measure in November 1988, so any anticipation should be concentrated near the end of the pre-treatment period.
No donor contamination. The donor states must not adopt similar programs during the study period. Abadie, Diamond, and Hainmueller (2010) enforced this assumption when they built the data, by dropping states with large tobacco programs or large tax increases.
Interpolation. California must lie inside the range of the donors on the outcome and the predictors, so that a convex combination can match it.
A long, well-fitted pre-treatment period. Nineteen years of close fit make it unlikely that the match is a coincidence, while a short or poorly fitted period would undermine the counterfactual.

The method is now defined. The next section shows how mlsynth expresses the same problem as one dictionary of settings. That dictionary is the only interface that a beginner needs to learn.

6. Meet mlsynth: one dictionary, one fit

The mlsynth library organizes every estimator around the same pattern. We describe the data and the settings in a Python dictionary, pass it to an estimator class, and call fit(). For the classic synthetic control of Section 5, the estimator class is VanillaSC, whose name marks the plain, unmodified method. Section 12 reuses the same pattern with two other classes, SDID and CLUSTERSC. This section prepares the data in the format that mlsynth expects and then writes the dictionary for the baseline model.

6.1 Prepare the columns

The mlsynth library needs two kinds of columns that the raw data do not contain. The first is a treatment indicator that equals 1 for California from 1989 onward. The second is a separate column for each lagged outcome that serves as a predictor, because every predictor must be a column name.

LAG_YEARS = (1988, 1980, 1975) # cigsale(1988), cigsale(1980), cigsale(1975) in Stata
def prepare_panel(df):
"""Add the treatment indicator and the three lagged-outcome columns."""
out = df.copy()
out["treated"] = ((out["state"] == TREATED)
& (out["year"] >= TREAT_YEAR)).astype(int)
for y in LAG_YEARS:
sales = out.loc[out["year"] == y].set_index("state")["cigsale"]
out[f"cigsale_{y}"] = out["state"].map(sales)
return out
panel = prepare_panel(df)
print(f"Treated rows (California, 1989 to 2000): {panel['treated'].sum()}")
show = panel["state"].isin([TREATED, "Utah"]) & panel["year"].isin([1988, 1989])
cols = ["state", "year", "cigsale", "treated", "cigsale_1988", "cigsale_1980", "cigsale_1975"]
print(panel.loc[show, cols].round(1).to_string(index=False))

Treated rows (California, 1989 to 2000): 12
state year cigsale treated cigsale_1988 cigsale_1980 cigsale_1975
California 1988 90.1 0 90.1 120.2 127.1
California 1989 82.4 1 90.1 120.2 127.1
Utah 1988 55.0 0 55.0 74.8 75.8
Utah 1989 57.0 0 55.0 74.8 75.8

The indicator treated equals 1 in 12 rows, the years 1989–2000 of California. The lag columns repeat one number within each state: California has 90.1 packs in cigsale_1988, 120.2 in cigsale_1980, and 127.1 in cigsale_1975 in every year. Each lag column therefore works like a fixed characteristic of the state, which is exactly how the Stata syntax cigsale(1988) treats it.

6.2 Choose the predictors and their windows

Each predictor is averaged over a window of years. The four covariates use 1980–1988, as in the Stata edition, and each lag column uses its own single year. The helper predictor_means() computes these averages for all 39 states, skipping the years with missing values. The mlsynth library computes the same averages itself from the windows in the configuration, so the helper only lets us inspect them and check them against Stata.

BASE_COVARIATES = ["lnincome", "age15to24", "retprice", "beer"]
COVARIATES = BASE_COVARIATES + [f"cigsale_{y}" for y in LAG_YEARS]
# The operator ** unpacks each dictionary, so the two merge into one
WINDOWS = {**{c: (1980, 1988) for c in BASE_COVARIATES}, # averages over 1980–1988
**{f"cigsale_{y}": (y, y) for y in LAG_YEARS}} # one year each
LABELS = {"lnincome": "Log GDP per capita", "age15to24": "Share aged 15–24",
"retprice": "Retail price", "beer": "Beer per capita",
"cigsale_1988": "Sales in 1988", "cigsale_1980": "Sales in 1980",
"cigsale_1975": "Sales in 1975"}
def predictor_means(panel, covariates=COVARIATES, windows=WINDOWS):
"""Average each predictor over its own window, skipping missing years."""
cols = {}
for c in covariates:
lo, hi = windows[c]
sub = panel[(panel["year"] >= lo) & (panel["year"] <= hi)]
cols[c] = sub.groupby("state", sort=False)[c].mean()
return pd.DataFrame(cols).reindex(panel["state"].unique())
X = predictor_means(panel) # 39 states x 7 predictors
means = pd.DataFrame({
"window": [f"{lo}" if lo == hi else f"{lo}–{hi}" for lo, hi in WINDOWS.values()],
"california": X.loc[TREATED],
"donor_average": X.loc[DONORS].mean()})
print(means.round(4).to_string())

 window california donor_average
lnincome 1980–1988 10.0766 9.8292
age15to24 1980–1988 0.1735 0.1725
retprice 1980–1988 89.4222 87.2661
beer 1980–1988 24.2800 23.6553
cigsale_1988 1988 90.1000 113.8237
cigsale_1980 1980 120.2000 138.0895
cigsale_1975 1975 127.1000 136.9316

California differs from the average donor most visibly on lagged sales, with 90.1 against 113.82 packs in 1988 and 120.2 against 138.09 in 1980. The covariates look much closer, and the age share is nearly identical. For income, however, the closeness is deceptive, because the table reports it in logarithms. The donor average is 0.25 log points poorer, that is, 0.25 lower in natural logarithms, which amounts to a large gap in levels. Section 7.4 returns to this gap. Every number in this table equals its counterpart in the Treated and Average Control columns of the Stata balance table. The two programs therefore start from the same predictors.

In mlsynth 1.0.0, the windows of the predictors must not all span the same years. If they do, that version first drops every year in which any predictor is missing, and only then averages. This deletion changes the predictors and the estimate. Separate one-year windows for the lagged sales avoid the problem, and the helper sc_config() of Section 8 guards against it.

6.3 The configuration dictionary

The dictionary below describes the whole baseline model. Five keys describe the data, two keys describe the predictors, and the remaining keys choose the algorithm and switch off extras. The comments explain each setting, and the table after the code maps each setting to its Stata counterpart, where one exists.

config = {
"df": panel, # long panel: one row per state and year
"outcome": "cigsale", # the outcome Y, packs per capita
"treat": "treated", # 1 for California from 1989 on, 0 otherwise
"unitid": "state", # the unit identifier
"time": "year", # the time identifier
"covariates": COVARIATES, # the seven predictors
"covariate_windows": WINDOWS, # the years averaged for each predictor
"backend": "mscmt", # nested search over V and W, like nested allopt
"canonical_v": "min.loss.w", # request a canonical V, with the optimizer V as fallback
"seed": RANDOM_SEED, # seed of the global search over V
"inference": False, # skip the placebo test for now (the default is True)
"display_graphs": False, # no automatic figures
}

Two settings deserve a comment. The backend, the numerical routine that searches for the predictor weights, is mscmt. It runs the nested search of Section 5.2, in which an outer global search over V wraps the inner problem for the donor weights. In synth2, the option nested requests the same nested problem, but Stata solves it with a local, derivative-based search. The option allopt runs that search from three starting points and keeps the best result. The key inference defaults to True, which would run 38 extra placebo fits, so we switch it off until Section 8.

Two further keys concern the search. The key canonical_v replaces the V of the optimizer with one standard choice among the many V that give the same donor weights. If that standard choice fails an internal check, mlsynth keeps the V of its optimizer instead, as Section 7.4 shows. The key affects only the reported V, so the donor weights, the gaps, and the ATT are the same with or without it. With or without the key, mlsynth also reports v_agreement, which measures how far two such standard choices disagree. Finally, the key seed fixes the random numbers of the global search, so every run on the same software stack returns the same weights.

Stata `synth2`	mlsynth `VanillaSC`
`xtset state year`	`"unitid": "state"` and `"time": "year"`
Outcome `cigsale`	`"outcome": "cigsale"`
`trunit(3) trperiod(1989)`	`"treat": "treated"`, a column that equals 1 for California from 1989
`lnincome age15to24 retprice beer` with `xperiod(1980(1)1988)`	`"covariates"` and `"covariate_windows"` with the window (1980, 1988)
`cigsale(1988) cigsale(1980) cigsale(1975)`	The columns `cigsale_1988`, `cigsale_1980`, and `cigsale_1975`, each with a one-year window
`nested allopt`	`"backend": "mscmt"`, a global search over V
`set seed 42`	`"seed": 42`
`placebo(unit cut(2))`	`"inference": True` gives the test without a cutoff (p = 0.026); cut(2) needs the loop of Section 8 (p = 0.050)
`placebo(period(1985))`	A loop with a fake start year (Section 9)
`loo`	A loop that drops one donor at a time (Section 10)

A dictionary invites typing mistakes. The mlsynth library validates every configuration with pydantic, a data-validation library, before it runs any computation. The block below misspells one key on purpose and catches the resulting error with try and except.

bad_config = {k: v for k, v in config.items() if k != "covariate_windows"}
bad_config["covariate_window"] = WINDOWS # a typo: the final s is missing
try:
VanillaSC(bad_config)
except MlsynthConfigError as err:
lines = str(err).splitlines()
print(type(err).__name__)
# keep the first three lines and drop the bracketed pydantic details
print("\n".join(line.split(" [type")[0] for line in lines[:3]))

MlsynthConfigError
1 validation error for VanillaSCConfig
covariate_window
Extra inputs are not permitted

The error names the unknown key covariate_window and explains that extra inputs are not permitted. Without this check, mlsynth would ignore the misspelled windows and give every predictor the same default window, the full pre-treatment period. The trap of Section 6.2 would then drop every year before 1984 without any warning, so every average would rest on 1984–1988 alone. Strict validation thus turns a silent modeling error into a loud and immediate one.

6.4 Anatomy of the result object

Every mlsynth estimator returns a result object with the same structure. Its fields group the estimate, the fit, the time series, the weights, and the inference results. The table below lists the main fields that this tutorial uses, so readers can return to it whenever a later block reads one of them.

Field	Content
`res.method_details.method_name`	The estimator and its backend
`res.att`	The ATT, the mean gap over 1989–2000
`res.effects.att_percent`	The ATT as a percent of mean synthetic sales after 1988
`res.fit_diagnostics.rmse_pre`	The RMSE of the fit before 1989, also available as `res.pre_rmse`
`res.fit_diagnostics.r_squared_pre`	The R-squared of the fit before 1989
`res.time_series.observed_outcome`	Sales in California, 1970–2000
`res.time_series.counterfactual_outcome`	Synthetic California, also available as `res.counterfactual`
`res.time_series.estimated_gap`	Actual minus synthetic sales, also available as `res.gap`
`res.time_series.intervention_time`	The treatment date, when the estimator stores one (Section 7.5)
`res.donor_weights`	A dictionary of the donors with positive weight
`res.weights.summary_stats`	Counts, the sum of weights, the constraint, the predictor weights, and `v_agreement`
`res.additional_outputs["covariate_balance"]`	Predictor values of California, synthetic California, and the donor average
`res.additional_outputs["solver_diagnostics"]`	Details of the search, such as `v_method`, the source of the reported V
`res.additional_outputs["treated_name"]`, `["donor_names"]`, and `["pre_periods"]`	The design of the fit: the treated unit, the names of the donors, and the number of pre-treatment years
`res.inference`	Placebo results when `inference` is on, and `None` otherwise
`res.plot(kind=...)`	Quick figures of the paths (`"counterfactual"`) and of the gap (`"gap"`)

Stata benchmark. The Stata edition stores its results in e(), which plays the role of the mlsynth result object. The ereturn list in the log shows e(att), e(rmse), and e(r2), the counterparts of res.att, res.fit_diagnostics.rmse_pre, and res.fit_diagnostics.r_squared_pre. Its entry e(T0) matches res.additional_outputs["pre_periods"], and Section 7 compares their values one by one.

The dictionary is ready, and the result object is mapped. The next section runs the fit and reads the result field by field. It starts with the quality of the fit before 1989.

7. Baseline synthetic California

7.1 Fit the model

We now fit the baseline model with a single call. The fit() method solves the nested weight problem of Section 5 and returns the result object. The call takes about one second on a laptop, because only one synthetic control is built.

res = VanillaSC(config).fit()
print(f"Estimator: {res.method_details.method_name}")
print(f"Treated unit: {res.additional_outputs['treated_name']}")
print(f"Donor pool: {len(res.additional_outputs['donor_names'])} states")
print(f"Pre-treatment years: {res.additional_outputs['pre_periods']}")

Estimator: VanillaSC[mscmt]
Treated unit: California
Donor pool: 38 states
Pre-treatment years: 19

The estimator reports its name as VanillaSC[mscmt], which confirms the nested backend. California is the treated unit, the donor pool has 38 states, and 19 years precede the treatment. These numbers match the design of Section 1.2, so the data entered the estimator as intended.

7.2 Pre-treatment fit

Before we look at any effect, we check whether synthetic California tracks actual California before 1989. A poor fit would make every later number meaningless, because the counterfactual would already be wrong before the program. The block reads the fit statistics and compares the RMSE with the level of sales.

gap = np.asarray(res.gap, dtype=float) # observed minus synthetic
pre_rmse = res.fit_diagnostics.rmse_pre
worst = np.argmax(np.abs(gap[:T0]))
print(f"Pre-period RMSE: {pre_rmse:.3f} packs per capita")
print(f"Pre-period R-squared: {res.fit_diagnostics.r_squared_pre:.3f}")
print(f"Mean sales in California, 1970 to 1988: {ca_sales[:T0].mean():.2f}")
print(f"RMSE as a share of mean sales: {100 * pre_rmse / ca_sales[:T0].mean():.1f} percent")
print(f"Largest pre-period miss: {gap[worst]:.2f} packs in {YEARS[worst]}")

Pre-period RMSE: 1.754 packs per capita
Pre-period R-squared: 0.976
Mean sales in California, 1970 to 1988: 116.21
RMSE as a share of mean sales: 1.5 percent
Largest pre-period miss: 5.90 packs in 1970

The pre-treatment RMSE is 1.754 packs per capita, about 1.5 percent of the mean sales of 116.21 packs over 1970–1988. The R-squared of 0.976 says that synthetic California reproduces almost all the variation of sales in California before 1989. The largest miss, 5.90 packs, occurs in 1970 at the start of the series, and the misses in later years are much smaller. The fit is therefore close enough to support a credible counterfactual.

Stata benchmark. The synth2 header reports a root mean squared error of 1.756 and an R-squared of 0.974. The RMSE differs from the Python value by less than 0.002 packs. The R-squared differs in the third decimal because synth2 uses another definition, a point that Section 11 resolves.

7.3 Donor weights

The donor weights reveal which states form synthetic California. Any of the 38 donors could receive weight, and the weights must be nonnegative and sum to one. We sort the positive weights from largest to smallest and place the Stata weights next to them.

Predict first

The abstract reports that five of the 38 donors receive weight, led by Utah. Section 3.2 showed that Utah has by far the lowest average sales. Will Utah receive more or less than half of the weight, and will the other four donors receive similar shares or very different ones? Commit to an answer before scrolling.

Reveal the answer

Answer. Utah receives less than half of the weight: its 0.335 is about one third of the recipe. The other four shares differ widely, from 0.236 for Nevada and 0.202 for Montana down to 0.160 for Colorado and 0.068 for Connecticut. The other 33 states receive exactly zero, and such sparse weights are typical, because the nonnegativity constraint pushes most weights to zero.

STATA_W = {"Utah": 0.334, "Nevada": 0.235, "Montana": 0.202,
"Colorado": 0.161, "Connecticut": 0.068} # synth2, Stata log
w_pos = dict(sorted(res.donor_weights.items(), key=lambda kv: -kv[1])) # largest first
for state, w in w_pos.items():
print(f"{state:<12} mlsynth {w:.4f} Stata {STATA_W.get(state, 0.0):.4f}")
stats = res.weights.summary_stats
print(f"Positive donors: {stats['n_nonzero']} of {len(DONORS)}; "
f"sum of weights: {stats['sum_of_weights']:.4f}")
print(f"Constraint: {stats['constraint']}")

Utah mlsynth 0.3351 Stata 0.3340
Nevada mlsynth 0.2356 Stata 0.2350
Montana mlsynth 0.2019 Stata 0.2020
Colorado mlsynth 0.1595 Stata 0.1610
Connecticut mlsynth 0.0679 Stata 0.0680
Positive donors: 5 of 38; sum of weights: 1.0000
Constraint: simplex (non-negative, sum to 1)

Five states form synthetic California: Utah (0.335), Nevada (0.236), Montana (0.202), Colorado (0.160), and Connecticut (0.068). The output calls the constraint the simplex, which requires nonnegative weights that sum to one, and these weights satisfy it. Utah earns the largest share, because its low sales pull the weighted average down toward the level of California. Nevada, however, is the only donor that borders California, so cross-border purchases could inflate its sales, a risk that Section 15.3 weighs.

The bar chart compares the two sets of weights visually. Each donor has a blue bar for mlsynth and a gold bar for Stata. Matching bars indicate that the two programs found the same recipe.

Show the plotting code

order = list(w_pos)
h = 0.38
fig, ax = plt.subplots(figsize=(10, 5.5))
fig.patch.set_linewidth(0)
yy = np.arange(len(order))
ax.set_axisbelow(True)
ml_vals = [res.donor_weights[s] for s in order]
st_vals = [STATA_W[s] for s in order]
ax.barh(yy - h / 2, ml_vals, height=h, color=STEEL_BLUE, label="mlsynth (VanillaSC)")
ax.barh(yy + h / 2, st_vals, height=h, color=GOLD, label="Stata (synth2)")
for i, (a, b) in enumerate(zip(ml_vals, st_vals)):
ax.text(a + 0.004, i - h / 2, f"{a:.3f}", va="center", fontsize=11, color=WHITE_TEXT)
ax.text(b + 0.004, i + h / 2, f"{b:.3f}", va="center", fontsize=11, color=WHITE_TEXT)
ax.set_yticks(yy)
ax.set_yticklabels(order, fontsize=12)
ax.invert_yaxis()
ax.set_xlim(0, 0.42)
ax.set_xlabel("Weight in synthetic California", fontsize=12)
ax.set_title("Donor weights: mlsynth and Stata", fontsize=14,
fontweight="bold", pad=12)
ax.legend(loc="lower right", fontsize=11)
ax.text(0.0, -0.16, "The other 33 donor states receive zero weight in both fits.",
transform=ax.transAxes, fontsize=11, color=LIGHT_TEXT)
plt.tight_layout()
plt.show()

Figure 2. Donor weights of mlsynth and Stata. The other 33 donor states receive zero weight in both fits.

The two bars nearly coincide for every donor. The largest difference, for Colorado, stays below 0.002, which reflects the two optimizers and the rounding of the Stata weights to three decimals. The recipe of synthetic California is therefore essentially the same in both programs, and Section 11 traces the small differences that remain.

Stata benchmark. The table of optimal unit weights in the Stata log lists Utah 0.334, Nevada 0.235, Montana 0.202, Colorado 0.161, and Connecticut 0.068. A note below that table confirms that the other 33 donors receive a weight of zero. The Python weights reproduce these values closely, and Section 11 explains the small differences.

7.4 Predictor balance and predictor weights

Predictor balance asks whether synthetic California resembles California on the seven predictors, not only on sales. The predictor weights V show how heavily the nested search penalized a mismatch on each predictor when it chose the donors. We print both, together with two diagnostics of mlsynth, and compare V with the values that Stata reports.

Predict first

The two programs agree on the donor weights within 0.002. Stata puts 0.546 of V on the age share and 0.422 on sales in 1975. Will mlsynth report similar predictor weights? Commit to an answer before scrolling.

Reveal the answer

Answer. No, it will not. The mlsynth fit puts about one third of V on each of the age share, the retail price, and sales in 1975. Both V vectors lead to almost the same donor weights, so V is not identified: the data cannot single out one set of predictor weights. Its entries are tuning parameters, not measures of importance.

STATA_V = [0.00004916, 0.54587094, 0.01741005, 0.0031354, 0.00490342,
0.00655685, 0.42207419] # diagonal of V, Stata log
v_ml = np.array([stats["predictor_weights"][c] for c in COVARIATES])
print(pd.DataFrame({"v_mlsynth": v_ml, "v_stata": STATA_V}, index=COVARIATES)
.round(3).to_string())
print(f"v_agreement: {stats['v_agreement']}")
print(f"v_method: {res.additional_outputs['solver_diagnostics']['v_method']}")
bal = res.additional_outputs["covariate_balance"]
balance = pd.DataFrame({"california": bal["treated"], "synthetic": bal["synthetic"],
"donor_average": bal["donor_average"]}, index=COVARIATES)
balance["pct_gap_synthetic"] = 100 * (balance["synthetic"] / balance["california"] - 1)
balance["pct_gap_average"] = 100 * (balance["donor_average"] / balance["california"] - 1)
print(balance.round(4).to_string())

 v_mlsynth v_stata
lnincome 0.000 0.000
age15to24 0.332 0.546
retprice 0.334 0.017
beer 0.000 0.003
cigsale_1988 0.000 0.005
cigsale_1980 0.000 0.007
cigsale_1975 0.334 0.422
v_agreement: 0.99999999
v_method: optimizer-fallback
california synthetic donor_average pct_gap_synthetic pct_gap_average
lnincome 10.0766 9.8585 9.8292 -2.1643 -2.4548
age15to24 0.1735 0.1735 0.1725 0.0000 -0.5891
retprice 89.4222 89.4222 87.2661 -0.0000 -2.4112
beer 24.2800 24.2226 23.6553 -0.2364 -2.5731
cigsale_1988 90.1000 91.6539 113.8237 1.7246 26.3304
cigsale_1980 120.2000 120.4721 138.0895 0.2264 14.8831
cigsale_1975 127.1000 127.1000 136.9316 -0.0000 7.7353

The printed table confirms the reveal and adds one clue. In mlsynth, three predictors receive about one third of V each: the age share (0.332), the retail price (0.334), and sales in 1975 (0.334). These are exactly the three predictors that synthetic California matches with a gap of zero in the balance table. This is the case that the proof card of Section 5.2 describes: when a few predictors are matched exactly, many V select the same donor weights. V is then not identified, which means that the data cannot single out one set of predictor weights. This numerical sense of the word differs from the identification of a causal effect.

The two diagnostics of mlsynth need a careful reading. Neither of them changes the donor weights or the ATT. The field v_method reads optimizer-fallback: the canonical V failed an internal check of mlsynth, so the library reports the V of its optimizer instead. The diagnostic v_agreement, despite its name, measures disagreement, namely the largest difference between two canonical choices of the predictor weights. Here both choices failed the same check, so its value of 0.99999999 says little about the predictor weights. We therefore draw the lesson about V from the comparison with Stata and from Exercise 6, not from this number.

Synthetic California matches six of the seven predictors closely. It overshoots sales in 1988 by 1.72 percent and matches the other five within 0.3 percent. Income is the exception, and its gap of −2.16 percent looks small only because it compares logarithms. Synthetic California falls short by 0.22 log points, a difference in natural logarithms (9.8585 against 10.0766), which means a GDP per capita about 20 percent lower. Since income receives almost no weight in V, the donor weights barely improve its balance: the donor average misses by 0.25 log points. On sales in 1988, in contrast, the donor average misses by 26.33 percent, which shows how much the donor weights improve on a simple average.

The two-panel figure summarizes both results. Panel (a) shows the percent gap of each predictor for synthetic California and for the donor average. Panel (b) shows the predictor weights of the two programs side by side.

Show the plotting code

h = 0.38
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6.2), sharey=True)
fig.patch.set_linewidth(0)
yy = np.arange(len(COVARIATES))
ax1.set_axisbelow(True)
ax2.set_axisbelow(True)
ax1.barh(yy - h / 2, balance["pct_gap_synthetic"], height=h, color=STEEL_BLUE,
label="Synthetic California")
ax1.barh(yy + h / 2, balance["pct_gap_average"], height=h, color=GREY_DONOR,
label="Average of the 38 donors")
ax1.axvline(0, color=LIGHT_TEXT, linewidth=0.9)
for i, (a, b) in enumerate(zip(balance["pct_gap_synthetic"], balance["pct_gap_average"])):
ax1.text(max(a, 0) + 0.5, i - h / 2, f"{signed(a, 1)}%", va="center",
fontsize=10, color=WHITE_TEXT)
ax1.text(max(b, 0) + 0.5, i + h / 2, f"{signed(b, 1)}%", va="center",
fontsize=10, color=WHITE_TEXT)
ax1.set_yticks(yy)
ax1.set_yticklabels([LABELS[c] for c in COVARIATES], fontsize=12)
ax1.invert_yaxis()
ax1.set_xlim(-5, 32)
ax1.set_xlabel("Difference from California (percent)", fontsize=12)
ax1.set_title("(a) Predictor balance", fontsize=13, fontweight="bold", pad=10)
ax1.legend(loc="lower right", fontsize=10)
ax2.barh(yy - h / 2, v_ml, height=h, color=STEEL_BLUE, label="mlsynth")
ax2.barh(yy + h / 2, STATA_V, height=h, color=GOLD, label="Stata")
for i, (a, b) in enumerate(zip(v_ml, STATA_V)):
ax2.text(a + 0.008, i - h / 2, f"{a:.3f}", va="center", fontsize=10, color=WHITE_TEXT)
ax2.text(b + 0.008, i + h / 2, f"{b:.3f}", va="center", fontsize=10, color=WHITE_TEXT)
ax2.set_xlim(0, 0.7)
ax2.set_xlabel("Weight in the diagonal of V", fontsize=12)
ax2.set_title("(b) Predictor weights V", fontsize=13, fontweight="bold", pad=10)
ax2.legend(loc="lower right", fontsize=10)
fig.suptitle("Predictor balance of synthetic California and the predictor weights V",
fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 3. Predictor balance and the predictor weights V. Different V produce almost the same synthetic California.

Panel (a) shows that the donor weights shrink the large gaps of the donor average on lagged sales almost to zero, while the gap on income barely changes. Panel (b) shows that the two programs reach this balance with very different predictor weights. The lesson for applied work is simple: report V for transparency, but never rank predictors by it.

Stata benchmark. The balance table of synth2 reports V weights of 0.546 for the age share, 0.422 for sales in 1975, and 0.017 for the retail price. Its synthetic predictor values differ from the Python values by at most 0.03, a consequence of the small differences in the donor weights. The Stata edition draws the same lesson in a note that its do-file prints: “V is poorly identified, so its values are not measures of importance.”

7.5 Actual and synthetic California

With the weights in hand, the synthetic path follows from Equation 2. We print the actual sales, the synthetic sales, and the gap for each year after 1988, next to the gap that Stata reports. This table already contains the main result of the tutorial.

synth = np.asarray(res.counterfactual, dtype=float) # synthetic California
att = float(res.att) # the ATT, 1989–2000
STATA_GAPS = [-7.5945, -9.7039, -13.4751, -14.1075, -17.7897, -22.1295,
-22.1023, -22.9827, -23.9123, -22.0976, -26.3711, -25.7550] # Stata log
paths = pd.DataFrame({"year": YEARS[T0:], "actual": ca_sales[T0:],
"synthetic": synth[T0:], "gap": gap[T0:],
"stata_gap": STATA_GAPS})
print(paths.round(2).to_string(index=False))

 year actual synthetic gap stata_gap
1989 82.4 89.99 -7.59 -7.59
1990 77.8 87.50 -9.70 -9.70
1991 68.7 82.15 -13.45 -13.48
1992 67.5 81.59 -14.09 -14.11
1993 63.4 81.17 -17.77 -17.79
1994 58.6 80.70 -22.10 -22.13
1995 56.4 78.48 -22.08 -22.10
1996 54.5 77.46 -22.96 -22.98
1997 53.8 77.69 -23.89 -23.91
1998 52.3 74.37 -22.07 -22.10
1999 47.2 73.55 -26.35 -26.37
2000 41.6 67.33 -25.73 -25.76

Synthetic California keeps declining after 1989, but actual California declines much faster. The gap is −7.59 packs in 1989, passes −20 packs in 1994, and reaches −25.73 packs in 2000. The Stata gaps differ from the Python gaps by at most 0.03 packs in any year. Both programs therefore show a gap that opens in 1989 and widens through the 1990s.

The mlsynth library can draw these paths itself. The calls res.plot(kind="counterfactual") and res.plot(kind="gap") draw the two native figures, and the code below restyles both for the dark theme of the site. In version 1.0.0, VanillaSC stores no intervention time, so the native figures draw no line at 1989, and we add one by hand.

Show the plotting code

fig, axes = plt.subplots(1, 2, figsize=(14, 5.4))
fig.patch.set_linewidth(0)
res.plot(kind="counterfactual", ax=axes[0], theme=MLSYNTH_DARK_THEME, display=False,
observed_color=WARM_ORANGE, observed_linewidth=2.4,
counterfactual_colors=[STEEL_BLUE], counterfactual_linewidth=2.2,
xlabel="Year", ylabel="Cigarette sales (packs per capita)")
res.plot(kind="gap", ax=axes[1], theme=MLSYNTH_DARK_THEME, display=False,
counterfactual_colors=[TEAL], counterfactual_linewidth=2.2,
xlabel="Year", ylabel="Gap (packs per capita)")
axes[1].axhline(0, color=LIGHT_TEXT, linewidth=0.9) # the native zero line is black
for ax in axes:
# res.plot draws no intervention line for VanillaSC, so we add one
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.6,
label="1989 (added by hand)")
ax.legend(loc="lower left", fontsize=10, frameon=False)
fig.suptitle("The native res.plot() figures of mlsynth on a dark background",
fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 4. The native res.plot() figures of mlsynth, restyled for a dark background. The dotted line at 1989 is added by hand.

The native figures are a fast way to inspect any fit. They show the close overlap before 1989 and the widening gap that the table reports for 1989–2000. For a report, we redraw both with matplotlib. The first figure marks the gap in 2000 with an arrow, the second adds the ATT as a dashed line, and both shade the years after Proposition 99.

Show the plotting code

fig, ax = plt.subplots(figsize=(11, 6))
fig.patch.set_linewidth(0)
ax.axvspan(TREAT_YEAR, LAST_YEAR + 0.5, color=GRID_LINE, alpha=0.45, zorder=0)
ax.plot(YEARS, ca_sales, color=WARM_ORANGE, linewidth=3.0,
label="California (observed)", zorder=3)
ax.plot(YEARS, synth, color=STEEL_BLUE, linewidth=2.6, linestyle="--",
label="Synthetic California", zorder=4)
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5)
ax.annotate("", xy=(LAST_YEAR, ca_sales[-1]), xytext=(LAST_YEAR, synth[-1]),
arrowprops=dict(arrowstyle="<->", color=TEAL, lw=2))
ax.text(LAST_YEAR + 0.3, ca_sales[-1] - 3.5, f"Gap in 2000: {signed(gap[-1])} packs",
color=TEAL, fontsize=11, ha="right", va="top", fontweight="bold")
ax.text(TREAT_YEAR + 0.4, 135, "Proposition 99 (1989)", color=LIGHT_TEXT,
fontsize=11)
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_ylim(30, 140)
ax.set_xlabel("Year", fontsize=12)
ax.set_ylabel("Cigarette sales (packs per capita)", fontsize=12)
ax.set_title("Observed and synthetic California, 1970–2000",
fontsize=14, fontweight="bold", pad=12)
ax.legend(loc="lower left", fontsize=11)
plt.tight_layout()
plt.show()

Figure 5. Observed and synthetic California, 1970–2000. The shaded area marks the years after Proposition 99.

The two paths nearly overlap from 1971 to 1988, after a larger miss in 1970. After 1989, synthetic California declines gently, while actual California falls steeply. The vertical distance between the two lines is the gap, which the next figure plots directly.

Show the plotting code

fig, ax = plt.subplots(figsize=(11, 6))
fig.patch.set_linewidth(0)
ax.axvspan(TREAT_YEAR, LAST_YEAR + 0.5, color=GRID_LINE, alpha=0.45, zorder=0)
ax.axhline(0, color=LIGHT_TEXT, linewidth=0.9)
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5)
ax.plot(YEARS, gap, color=TEAL, linewidth=2.6, marker="o", markersize=4,
label="Gap: observed minus synthetic California", zorder=3)
ax.hlines(att, TREAT_YEAR, LAST_YEAR, color=WARM_ORANGE, linestyle="--",
linewidth=2.2, label=f"ATT over 1989–2000: {signed(att)} packs", zorder=4)
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_ylim(-32, 8)
ax.set_xlabel("Year", fontsize=12)
ax.set_ylabel("Gap (packs per capita)", fontsize=12)
ax.set_title("Gap between observed and synthetic California",
fontsize=14, fontweight="bold", pad=12)
ax.legend(loc="lower left", fontsize=11)
plt.tight_layout()
plt.show()

Figure 6. The gap between observed and synthetic California, with the ATT over 1989–2000.

From 1971 to 1988, the gap stays small, and its largest miss is −2.23 packs, in 1987. It then drops from −1.55 packs in 1988 to −7.59 in 1989 and keeps widening through the 1990s, with a few small reversals, as in 1998. A gap that opens when the program starts is the visual signature of a policy effect, and Section 9 asks whether it really opens only then.

7.6 The ATT in perspective

The ATT condenses the 12 yearly gaps into one number. Raw packs are hard to judge, so we also express the effect as a percent of synthetic sales. The block below computes the ATT in both ways and checks it against a hand calculation.

print(f"ATT (res.att): {att:.2f} packs per capita per year")
print(f"Mean of the 12 yearly gaps: {gap[T0:].mean():.2f}")
print(f"ATT in percent (res.effects.att_percent): {res.effects.att_percent:.1f}")
print(f"Mean sales 1989 to 2000: actual {ca_sales[T0:].mean():.2f}, "
f"synthetic {synth[T0:].mean():.2f}")
print(f"Year 2000: actual {ca_sales[-1]:.1f}, synthetic {synth[-1]:.2f}, "
f"gap {gap[-1]:.2f} ({100 * gap[-1] / synth[-1]:.1f} percent)")

ATT (res.att): -18.98 packs per capita per year
Mean of the 12 yearly gaps: -18.98
ATT in percent (res.effects.att_percent): -23.9
Mean sales 1989 to 2000: actual 60.35, synthetic 79.33
Year 2000: actual 41.6, synthetic 67.33, gap -25.73 (-38.2 percent)

The ATT is −18.98 packs per capita per year, identical to the mean of the 12 yearly gaps. Over 1989–2000, California sold 60.35 packs per capita per year on average, 18.98 packs fewer than synthetic California, a reduction of 23.9 percent. By 2000, the reduction reached 38.2 percent, so the effect of the program grew over time rather than fading.

One caveat concerns the last two years. In January 1999, Proposition 10 raised the state cigarette tax by a further 50 cents per pack. The gaps of 1999 and 2000 therefore cannot be credited to Proposition 99 alone. Even before that second increase, however, the gap stayed below −22 packs in every year from 1994 to 1998. The growth of the effect thus does not rest on the last two years.

Stata benchmark. The prediction table of synth2 reports an ATT of −19.00 packs, with gaps of −7.59 in 1989 and −25.76 in 2000. The Python ATT of −18.98 reproduces it closely. Stata computes its ATT from weights rounded to three decimals, and Exercise 2 recovers −19.0018 by applying those weights by hand.

An average gap of 19 packs looks large, but it needs a yardstick. No synthetic control predicts perfectly, so some gap would appear even without any program. The next section measures how large such gaps are in states that never adopted a comparable program.

8. In-space placebo test

8.1 The logic of the test

The in-space placebo test asks how unusual the gap of California is. It reassigns the treatment to each donor state in turn, builds a synthetic control for that state, and records the resulting gaps. If Proposition 99 had an effect, the gap of California should stand out from these placebo gaps.

A raw gap is not a fair statistic, because some states are hard to fit even before 1989. A state with a poor fit can show large gaps without any treatment. The standard statistic, Equation 4, therefore divides the post-treatment misses by the pre-treatment misses in three steps:

$$\mathrm{MSPE}_{j}^{\mathrm{pre}} = \frac{1}{T_0} \sum_{t=1970}^{1988} \hat{\tau}_{jt}^{2}$$

$$\mathrm{MSPE}_{j}^{\mathrm{post}} = \frac{1}{T_1} \sum_{t=1989}^{2000} \hat{\tau}_{jt}^{2}$$

$$r_j = \frac{\mathrm{MSPE}_{j}^{\mathrm{post}}}{\mathrm{MSPE}_{j}^{\mathrm{pre}}}$$

In words, the mean squared prediction error (MSPE) averages the squared gaps of state $j$, separately before and after 1989. The ratio $r_j$ then compares the two averages. Here $\hat{\tau}_{jt}$ is the gap of state $j$ in year $t$. It comes from a fit that treats state $j$ as the treated unit and builds its synthetic control from the other states. The index $j$ now runs over all 39 states, with California as unit 1. In the code, mspe(gap) returns the two averages, and the column ratio of the table placebo holds $r_j$.

The ratio rewards a close fit before 1989 and a large gap afterward. A state with a real effect and a good pre-treatment fit has a large ratio. A poorly fitted state has a large denominator, so its ratio tends to be small. The square root of the pre-treatment MSPE is the RMSE of Section 7.2, so the pre-treatment MSPE of California is 3.08, the square of 1.754. California thus enters the test with a small denominator.

The p-value then measures the share of states whose ratio is at least as large as that of California. California is always one of these states, because its own ratio meets the condition trivially. This p-value is called a permutation p-value, because the test behind it reassigns the treatment to every state in turn. With $J + 1 = 39$ states, Equation 5 defines the p-value:

$$p = \frac{1}{J+1} \sum_{j=1}^{J+1} \mathbf{1}\left[ r_j \geq r_1 \right]$$

In words, the p-value is the share of the 39 states whose ratio is at least as large as the ratio of California (unit 1). The indicator $\mathbf{1}[\cdot]$ equals one when the condition in brackets holds and zero otherwise. The p-value thus restates the rank of California as a share, and it is not the probability that the policy had no effect. The code computes it with placebo_pvalue(placebo).

Two details of the test deserve a closer look. The built-in test of mlsynth ranks the square root of the ratio instead of the ratio itself. In addition, because California always counts itself, the number of states sets a lower bound on the p-value. The proof card below shows that the first detail leaves the ranking unchanged and that the second sets a floor of 1/39.

Proof Why the MSPE ratio and its square root rank states the same way, and why p cannot fall below 1/39

Ratios of mean squared errors are never negative. The square root is strictly increasing on nonnegative numbers, so the order of any two ratios equals the order of their square roots. The following equivalence states this fact for the comparison inside Equation 5.

$$r_j \geq r_1 \iff \sqrt{r_j} \geq \sqrt{r_1}$$

In words, the RMSPE ratio, which divides root mean squared prediction errors, is the square root of the MSPE ratio. A state therefore ties or beats California on one ratio exactly when it does so on the other. Each indicator in Equation 5 thus stays the same, and so does the p-value. In the code, $r_j$ is the column ratio of the table placebo, and $\sqrt{r_1}$ for California is inf.details["treated_rmspe_ratio"] of the built-in test.

The equivalence holds within one set of placebo fits, however. The built-in test of mlsynth and the loop of this section use different donor pools for the placebo states, so their placebo ratios differ. Only the ratio of California, whose fit is the same in both designs, links the two exactly.

Because California always counts itself, the sum in Equation 5 is a whole number $q$ of at least one. The p-value can therefore take only the values $q/39$, so it cannot fall below $1/39$. After a filter keeps $N$ states, the values become $q/N$, with a floor of $1/N$.

This floor is also the right benchmark for the test. Suppose that the program had no effect in any state, which is the sharp null hypothesis. Suppose also that every state were equally likely to be the treated one. Each of the 39 states is then equally likely to have the largest ratio. California holds that rank with probability $1/39$, so under the null the smallest p-value arises with exactly the probability that it reports.

8.2 Two ways to run the test in mlsynth

The mlsynth library offers the test as a built-in option. Setting inference to True refits the model 38 times, once for each donor treated as if it had adopted the program. The built-in test leaves California out of every placebo donor pool, so the effect of the program cannot leak into the placebo counterfactuals. The expression {**config, "inference": True} builds a copy of config in which only the key inference changes, a pattern that later sections reuse.

Predict first

The abstract and the study design already report that California ranks first among the 39 states. The open questions concern the details of the test. Will the built-in test and the loop that follows it agree on that rank, even though their placebo fits differ? Will the states that rank just below California be well fitted or poorly fitted before 1989? Commit to an answer before scrolling.

Reveal the answer

Answer. The two designs agree: the built-in test reports rank 1 of 39, and the loop gives California an MSPE ratio of 129.0, ahead of Georgia at 97.0. California therefore attains the smallest possible p-value, 1/39 = 0.026. The states just below it are well fitted, since Georgia, Virginia, and Missouri, ranked second to fourth in the loop, all fit better than California before 1989. A poor fit inflates the denominator of the ratio, so poorly fitted states rarely rank near the top.

res_builtin = VanillaSC({**config, "inference": True}).fit() # 38 placebo fits
inf = res_builtin.inference
assert inf is not None, "an unknown inference mode is ignored silently"
print(f"Method: {inf.method}")
print(f"RMSPE ratio of California: {inf.details['treated_rmspe_ratio']:.2f}")
print(f"Rank of California: {inf.details['rank']} of {inf.details['n_placebos'] + 1}")
print(f"p-value: {inf.p_value:.3f}")

Method: in-space placebo (RMSPE ratio)
RMSPE ratio of California: 11.36
Rank of California: 1 of 39
p-value: 0.026

The built-in test ranks California first of 39 states, with a p-value of 0.026. Its statistic is the ratio of root mean squared prediction errors (RMSPE ratio), the post-treatment RMSE divided by the pre-treatment RMSE. For California, it equals 19.925 divided by 1.754, or 11.36. The assertion in the code also matters. The library ignores an unknown value of inference without any warning, so the check confirms that the test actually ran.

The Stata command and the original study keep California in the donor pool of every placebo fit. This choice lets the treated state enter some placebo counterfactuals, but it matches the design of the benchmark. The built-in test also has no cutoff option, so it cannot apply the cut(2) filter of Section 8.4. We therefore write our own loop in two steps, and the loop also returns the gap paths for the next figures. The first block defines three small helpers that make every later refit a one-line call, and its last line checks that they reproduce the baseline.

def base_config(panel, treated=TREATED, start=TREAT_YEAR):
"""Return the keys that every mlsynth estimator needs."""
data = panel.assign(treated=((panel["state"] == treated)
& (panel["year"] >= start)).astype(int))
return {"df": data, "outcome": "cigsale", "treat": "treated",
"unitid": "state", "time": "year", "display_graphs": False}
def sc_config(panel, treated=TREATED, start=TREAT_YEAR, covariates=COVARIATES,
windows=WINDOWS, inference=False, **extra):
"""Build the VanillaSC configuration used in every covariate fit."""
# Guard: mlsynth 1.0.0 drops whole years when all predictors cover the same
# years before the start, so we compare the years that each window covers
pre = range(FIRST_YEAR, start)
spans = {tuple(t for t in pre if windows[c][0] <= t <= windows[c][1]) or tuple(pre)
for c in covariates}
assert len(covariates) < 2 or len(spans) > 1, "give the predictors different windows"
return {**base_config(panel, treated, start),
"covariates": list(covariates),
"covariate_windows": {c: windows[c] for c in covariates},
"backend": "mscmt", "canonical_v": "min.loss.w",
"seed": RANDOM_SEED, "inference": inference, **extra}
def fit_sc(panel, **kwargs):
"""Fit VanillaSC with the specification of this post."""
return VanillaSC(sc_config(panel, **kwargs)).fit()
res_check = fit_sc(panel)
print(f"ATT from fit_sc(): {res_check.att:.4f}; ATT of the baseline: {res.att:.4f}")

ATT from fit_sc(): -18.9816; ATT of the baseline: -18.9816

The helper fit_sc() returns exactly the baseline ATT of −18.98, so it encodes the same specification. Its companion sc_config() accepts another treated state, another start year, or other predictors. That flexibility is all that the robustness checks need.

The guard inside sc_config() protects against the trap of Section 6.2 in three steps. First, the range pre lists the years before the start. Second, each element of the set spans lists the pre-treatment years that one window covers, or all of them when the window covers none. Third, when every predictor covers the same years, the set has a single element. The assertion then stops any configuration with two or more predictors before mlsynth can drop a year. Predictors that merely share a window, such as the four covariates of the baseline, pass the check, because the lag columns cover other years.

The loop below treats each of the 39 states in turn, including California itself. For every fit, it records the pre-treatment and the post-treatment MSPE, the ratio, and the pre-treatment MSPE relative to that of California. The 39 fits take about one minute.

def mspe(gap, t0=T0):
"""Mean squared prediction error before and after the treatment date."""
gap = np.asarray(gap, dtype=float)
return float(np.mean(gap[:t0] ** 2)), float(np.mean(gap[t0:] ** 2))
def placebo_in_space(panel, states):
"""Refit the model with each state as the treated unit (synth2 design).
California stays in the donor pool of every placebo fit.
"""
fits = {s: fit_sc(panel, treated=s) for s in states}
rows = []
for s, r in fits.items():
pre, post = mspe(r.gap)
rows.append({"unit": s, "pre_mspe": pre, "post_mspe": post})
tab = pd.DataFrame(rows).set_index("unit")
tab["ratio"] = tab["post_mspe"] / tab["pre_mspe"]
tab["pre_rel"] = tab["pre_mspe"] / tab.loc[TREATED, "pre_mspe"]
tab = tab.sort_values("ratio", ascending=False, kind="mergesort")
tab["rank"] = np.arange(1, len(tab) + 1)
return tab, fits
placebo, placebo_fits = placebo_in_space(panel, STATES) # 39 fits
gaps = {s: np.asarray(r.gap, dtype=float) for s, r in placebo_fits.items()}
ca_ratio = placebo.loc[TREATED, "ratio"]
print(f"Placebo fits: {len(placebo_fits)}")
print(f"MSPE ratio of California: {ca_ratio:.2f}; its square root: {np.sqrt(ca_ratio):.2f}")

Placebo fits: 39
MSPE ratio of California: 129.04; its square root: 11.36

The loop produced 39 fits, one for California and 38 placebo fits. The MSPE ratio of California is 129.0, and its square root is 11.36. This root equals the statistic of the built-in test, because both designs use the same fit for California. The placebo ratios of the two designs differ, however, because their donor pools differ.

8.3 Ranking and the permutation p-value

We now rank the 39 ratios and compute the permutation p-value of Equation 5. The table shows the six largest ratios with their two MSPE components. The helper placebo_pvalue() takes the mean of a column of True and False values, which equals the share of states that tie or beat California. The p-value uses all 39 states, without any filter.

def placebo_pvalue(tab, cutoff=None):
"""Share of retained units whose MSPE ratio is at least that of California."""
keep = tab if cutoff is None else tab[tab["pre_rel"] <= cutoff]
return keep, float((keep["ratio"] >= tab.loc[TREATED, "ratio"]).mean())
print(placebo[["pre_mspe", "post_mspe", "ratio", "rank"]].head(6).round(2).to_string())
_, p_all = placebo_pvalue(placebo)
print(f"Permutation p-value over 39 states: {p_all:.3f}")

 pre_mspe post_mspe ratio rank
unit
California 3.08 397.02 129.04 1
Georgia 1.41 136.79 96.96 2
Virginia 2.74 234.40 85.42 3
Missouri 1.09 66.03 60.86 4
Oklahoma 4.65 270.13 58.09 5
Texas 4.00 205.69 51.39 6
Permutation p-value over 39 states: 0.026

California has the largest ratio, 129.0, ahead of Georgia at 97.0 and Virginia at 85.4. Its post-treatment MSPE of 397.02 is large, and its pre-treatment MSPE of 3.08 is small, which is the pattern of a real effect. The permutation p-value is 0.026, the smallest value that 39 states allow.

8.4 The cut(2) filter

Some placebo states fit poorly even before 1989, and their large gaps after 1989 may reflect that poor fit rather than any shock. The option cut(2) of synth2 drops every placebo whose pre-treatment MSPE exceeds twice that of California. We apply the same rule with the column pre_rel and recompute the p-value among the states that remain.

CUTOFF = 2 # synth2 option cut(2)
placebo["kept_cut2"] = placebo["pre_rel"] <= CUTOFF
keep, p_cut = placebo_pvalue(placebo, CUTOFF)
n_kept = len(keep)
print(f"Threshold: pre-period MSPE at most {CUTOFF * placebo.loc[TREATED, 'pre_mspe']:.2f}")
print(f"States kept: {n_kept} of 39; rank of California among them: "
f"{int((keep['ratio'] >= ca_ratio).sum())}; p = {p_cut:.3f}")
print("Removed:", ", ".join(s for s in STATES if s not in keep.index))
print(placebo.loc[["Illinois", "South Dakota"], ["pre_rel", "kept_cut2"]].round(3).to_string())

Threshold: pre-period MSPE at most 6.15
States kept: 20 of 39; rank of California among them: 1; p = 0.050
Removed: Colorado, Connecticut, Delaware, Indiana, Iowa, Kansas, Kentucky, Maine, Minnesota, Nevada, New Hampshire, North Carolina, North Dakota, Rhode Island, South Dakota, Utah, Vermont, West Virginia, Wyoming
pre_rel kept_cut2
unit
Illinois 1.867 True
South Dakota 2.026 False

The filter keeps 20 of the 39 states, and California still ranks first, so the p-value becomes 1/20 = 0.050. The removed states include Utah and New Hampshire, whose extreme sales no weighted average of the other states can reach. They also include Rhode Island, whose pre-treatment MSPE is about 20 times that of California. The two printed rows show the states closest to the cutoff: Illinois, kept at 1.867 times the pre-treatment MSPE of California, and South Dakota, removed at 2.026. Another seed or software stack could therefore keep South Dakota and raise the number of states to 21.

The filter seldom helps California climb the ranking. A poorly fitted state has a large pre-treatment MSPE, which sits in the denominator of its ratio, so its ratio tends to be small already. What the filter buys is comparability of the gaps themselves, which matters for the gap figure and the pointwise p-values below. What it costs is resolution, because the smallest attainable p-value rises from 0.026 to 0.050.

The bar chart below shows all 39 ratios at once. California appears in orange at the top, states kept by cut(2) in blue, and removed states in gray. The figure makes the rank and the effect of the filter visible together.

Show the plotting code

fig, ax = plt.subplots(figsize=(10, 11))
fig.patch.set_linewidth(0)
units = placebo.index.tolist()
colors = [WARM_ORANGE if u == TREATED else (STEEL_BLUE if placebo.loc[u, "kept_cut2"]
else GREY_DONOR) for u in units]
yy = np.arange(len(units))
ax.set_axisbelow(True)
ax.barh(yy, placebo["ratio"], color=colors, height=0.72)
for i, u in enumerate(units):
ax.text(placebo.loc[u, "ratio"] + 1.2, i, f"{placebo.loc[u, 'ratio']:.1f}",
va="center", fontsize=9, color=WHITE_TEXT if u == TREATED else LIGHT_TEXT)
ax.set_yticks(yy)
ax.set_yticklabels(units, fontsize=10)
ax.invert_yaxis()
ax.set_xlim(0, 145)
ax.set_xlabel("Post-period MSPE divided by pre-period MSPE", fontsize=12)
ax.set_title("In-space placebo test: MSPE ratios of the 39 states",
fontsize=14, fontweight="bold", pad=12)
ax.legend(handles=[mpatches.Patch(color=WARM_ORANGE, label="California"),
mpatches.Patch(color=STEEL_BLUE, label=f"Placebo states kept by cut({CUTOFF})"),
mpatches.Patch(color=GREY_DONOR, label=f"Placebo states removed by cut({CUTOFF})")],
loc="lower right", fontsize=11)
plt.tight_layout()
plt.show()

Figure 7. MSPE ratios of the 39 states. California has the largest ratio, and the dimmed bars mark the states removed by cut(2).

Most removed states sit in the lower half of the ranking, as the logic of the ratio predicts. Some removed states, such as Indiana and West Virginia, still have sizable ratios, but none comes close to California. The rank of California, and therefore the conclusion, would be the same with or without the filter.

8.5 Placebo gaps

The ratio summarizes each state with one number, but the gaps themselves tell a richer story. The next figure draws the gap of California and the gaps of the 19 placebo states that cut(2) retains. The states removed by the filter are left out, because their gaps reflect poor fits.

Show the plotting code

fig, ax = plt.subplots(figsize=(11, 6))
fig.patch.set_linewidth(0)
for s in keep.index:
if s != TREATED:
ax.plot(YEARS, gaps[s], color=GREY_DONOR, linewidth=1.2, alpha=0.95, zorder=1)
ax.plot([], [], color=GREY_DONOR, linewidth=1.4,
label=f"Placebo states kept by cut({CUTOFF}) ({n_kept - 1})")
ax.plot(YEARS, gaps[TREATED], color=WARM_ORANGE, linewidth=3.0,
label="California", zorder=3)
ax.axhline(0, color=LIGHT_TEXT, linewidth=0.9)
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5)
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_ylim(-40, 40)
ax.set_xlabel("Year", fontsize=12)
ax.set_ylabel("Gap (packs per capita)", fontsize=12)
ax.set_title(f"Gaps of California and the {n_kept - 1} retained placebo states",
fontsize=14, fontweight="bold", pad=12)
ax.legend(loc="lower left", fontsize=11)
plt.tight_layout()
plt.show()

Figure 8. Gaps of California and the 19 retained placebo states. California is the most negative line in 9 of the 12 years after 1988.

Before 1989, the gap of California is as small as the placebo gaps, which confirms the good fit. From 1989 onward, it lies below every retained placebo in 9 of the 12 years. It ends at −25.73 packs in 2000, against −18.77 for Georgia, the lowest placebo in that year. Placebo gaps also widen after 1989, a reminder that prediction errors grow with the distance from the fitting period.

8.6 Pointwise p-values

The pointwise test repeats the ranking year by year, using the gaps instead of the ratio. In each year, it counts the retained states whose gap is at least as extreme as that of California, in one of three senses. The two-sided version uses absolute gaps, the right-sided version looks for gaps at least as large, and the left-sided version looks for gaps at least as small.

def pointwise_pvalues(keep_units, gaps, years, t0=T0):
"""Year-by-year placebo p-values with the synth2 tie rules.
California counts in the numerator and the denominator, so the smallest
attainable p-value is one over the number of retained units.
"""
G = np.array([gaps[u] for u in keep_units])[:, t0:] # kept units x 12 years
g1 = np.asarray(gaps[TREATED])[t0:]
return pd.DataFrame({"year": np.asarray(years)[t0:], "gap": g1,
"p_two": (np.abs(G) >= np.abs(g1)).mean(axis=0),
"p_right": (G >= g1).mean(axis=0),
"p_left": (G <= g1).mean(axis=0)})
pw = pointwise_pvalues(keep.index, gaps, YEARS)
print(pw.round(3).to_string(index=False))
# years in which California lies below every kept placebo (p_left equals 1/n_kept)
n_left_min = int((np.rint(pw["p_left"] * n_kept) == 1).sum())
print(f"Years with the left-sided p at its floor of 1/{n_kept}: {n_left_min} of 12")

 year gap p_two p_right p_left
1989 -7.589 0.05 1.00 0.05
1990 -9.698 0.10 0.95 0.10
1991 -13.452 0.10 0.95 0.10
1992 -14.086 0.05 1.00 0.05
1993 -17.768 0.05 1.00 0.05
1994 -22.105 0.05 1.00 0.05
1995 -22.076 0.05 1.00 0.05
1996 -22.961 0.05 1.00 0.05
1997 -23.895 0.05 1.00 0.05
1998 -22.069 0.10 0.95 0.10
1999 -26.348 0.10 1.00 0.05
2000 -25.733 0.05 1.00 0.05
Years with the left-sided p at its floor of 1/20: 9 of 12

Proposition 99 was designed to lower sales, so the test looks for a negative effect, and the left-sided p-value is the relevant one. It sits at its floor of 1/20 = 0.050 in 9 of the 12 years and never exceeds 0.100. The right-sided p-values, between 0.950 and 1.000, only confirm that the gap of California is not unusually positive. In 1999 alone, the two-sided p-value of 0.100 exceeds the left-sided one. The reason is Oklahoma, whose positive gap of 26.37 packs is larger in absolute value than the gap of California.

The figure plots the three pointwise p-values by year. Filled markers show the Python values, and hollow gold markers show the Stata values. Dashed and dotted lines mark the conventional thresholds of 0.050 and 0.100.

Show the plotting code

# Pointwise p-values that synth2 prints with cut(2) in the Stata log
pw["stata_p_two"] = [0.05, 0.10, 0.15, 0.10, 0.05, 0.05, 0.05, 0.05, 0.05, 0.10, 0.05, 0.05]
pw["stata_p_right"] = [1.00, 0.95, 0.90, 0.95, 1.00, 1.00, 1.00, 1.00, 1.00, 0.95, 1.00, 1.00]
pw["stata_p_left"] = [0.05, 0.10, 0.15, 0.10, 0.05, 0.05, 0.05, 0.05, 0.05, 0.10, 0.05, 0.05]
fig, axes = plt.subplots(1, 3, figsize=(15, 5.2), sharey=True)
fig.patch.set_linewidth(0)
panels = [("p_two", "stata_p_two", "Two-sided", STEEL_BLUE),
("p_right", "stata_p_right", "Right-sided", LIGHT_ORANGE),
("p_left", "stata_p_left", "Left-sided", TEAL)]
for ax, (col, scol, title, color) in zip(axes, panels):
ax.axhline(0.05, color=WARM_ORANGE, linestyle="--", linewidth=1.3, label="p = 0.05")
ax.axhline(0.10, color=LIGHT_TEXT, linestyle=":", linewidth=1.3, label="p = 0.10")
ax.plot(pw["year"], pw[col], color=color, marker="o", markersize=7,
linewidth=2, label="mlsynth", zorder=3)
ax.plot(pw["year"], pw[scol], linestyle="none", marker="o", markersize=12,
markerfacecolor="none", markeredgecolor=GOLD, markeredgewidth=1.6,
label="Stata", zorder=4)
ax.set_title(title, fontsize=13, fontweight="bold", pad=10)
ax.set_xlabel("Year", fontsize=12)
ax.set_xticks([1990, 1993, 1996, 1999])
ax.set_ylim(-0.03, 1.05)
axes[0].set_ylabel("Placebo p-value", fontsize=12)
axes[1].legend(handles=[
Line2D([], [], color=WARM_ORANGE, linestyle="--", linewidth=1.3, label="p = 0.05"),
Line2D([], [], color=LIGHT_TEXT, linestyle=":", linewidth=1.3, label="p = 0.10"),
Line2D([], [], color=LIGHT_TEXT, marker="o", markersize=7, linewidth=2,
label="mlsynth (filled markers)"),
Line2D([], [], linestyle="none", marker="o", markersize=12, markerfacecolor="none",
markeredgecolor=GOLD, markeredgewidth=1.6, label="Stata (hollow markers)")],
loc="center", fontsize=11)
fig.suptitle(f"Pointwise placebo p-values, 1989–2000, with cut({CUTOFF}) "
f"({n_kept} units)", fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 9. Pointwise placebo p-values with cut(2). The left-sided panel is the relevant one for a program designed to reduce sales.

The figure places the three tests side by side. The left-sided panel stays at or near the dashed line at 0.050, while the right-sided panel stays near 1.000, the mirror image that a negative effect produces. The hollow gold markers sit on the filled ones except in 1991, 1992, and 1999, and even there they differ by only one step of 0.050. Both programs therefore support a negative effect in almost every year.

Stata benchmark. The placebo table of synth2 reports an MSPE ratio of 123.5 for California, the largest of the 39 ratios. Its p-value is 0.026 with all states and 0.050 after cut(2). Its list of 19 excluded states matches the Python list exactly, so both programs keep the same 20 states. The pointwise p-values differ in 1991, 1992, and 1999, and the left-sided value of Stata reaches 0.150 once, in 1991. Stata refits California inside the placebo command and solves each placebo fit with its own optimizer. The Stata ratio for California therefore differs from 129.0. The left-sided p-value of Stata also reaches its floor in 8 of the 12 years rather than 9, as Section 11 explains.

California stands out among the placebo states. The next check moves the date instead of the state. A method that finds an effect at a date when nothing happened would cast doubt on the result for 1989, and Section 9 runs that test.

9. In-time placebo test

9.1 Design

The in-time placebo pretends that Proposition 99 began earlier than it did. Abadie, Diamond, and Hainmueller (2015) apply this test to German reunification. We adopt the fake date of the Stata edition, 1985, four years before the real program. Every predictor must then be measured before 1985, so the covariate windows end in 1984, and sales in 1988 leave the list of predictors.

An earlier fake date is tempting, because it would leave a longer fake post-treatment period. The helpers below build the specification for any fake start, and the last lines try a fake start in 1984. That attempt asks for a beer average over 1980–1983, a period without any beer data.

def in_time_spec(fake_year):
"""Predictors for a fake start year: windows end before it, lags precede it."""
lags = [y for y in LAG_YEARS if y < fake_year]
covariates = BASE_COVARIATES + [f"cigsale_{y}" for y in lags]
windows = {**{c: (1980, fake_year - 1) for c in BASE_COVARIATES},
**{f"cigsale_{y}": (y, y) for y in lags}}
return covariates, windows
def fit_in_time(panel, fake_year):
"""Fit the model as if Proposition 99 had started in fake_year."""
covariates, windows = in_time_spec(fake_year)
return fit_sc(panel, start=fake_year, covariates=covariates, windows=windows)
print("Predictors for a fake start in 1985:", in_time_spec(1985)[0])
try:
fit_in_time(panel, 1984) # the beer window would end in 1983
except MlsynthDataError as err:
print(f"Fake start in 1984: {type(err).__name__}: {err}")

Predictors for a fake start in 1985: ['lnincome', 'age15to24', 'retprice', 'beer', 'cigsale_1980', 'cigsale_1975']
Fake start in 1984: MlsynthDataError: Covariate means contain NaN (check windows/coverage).

The fit with a fake start in 1984 stops with MlsynthDataError, because no state has beer data in 1980–1983, so the beer average is undefined. When only some years of a window are missing, mlsynth instead averages the available years without any warning, as it does for beer over 1980–1988. The earliest feasible fake date that keeps all four covariates is therefore 1985, the date of the Stata edition. Even at that date, the beer average rests on 1984 alone, so the in-time fit matches beer on a single year.

9.2 Results with a fake start in 1985

We now fit the model with the fake start in 1985. The fit uses 1970–1984 to choose the weights and treats 1985–2000 as the post-treatment period. The gaps for 1985–1988 are fake effects, because the program did not exist yet.

FAKE_YEAR = 1985
res85 = fit_in_time(panel, FAKE_YEAR)
synth85 = np.asarray(res85.counterfactual, dtype=float)
gap85 = np.asarray(res85.gap, dtype=float)
k = FAKE_YEAR - FIRST_YEAR # 15 fitting years, 1970 to 1984
w85 = dict(sorted(res85.donor_weights.items(), key=lambda kv: -kv[1]))
print("Weights:", ", ".join(f"{s} {w:.3f}" for s, w in w85.items()))
print(f"Pre-period RMSE, 1970 to 1984: {res85.pre_rmse:.3f}")
fake = pd.DataFrame({"year": YEARS[k:T0], "actual": ca_sales[k:T0],
"synthetic": synth85[k:T0], "gap": gap85[k:T0]})
print(fake.round(2).to_string(index=False))
print(f"Mean fake gap, 1985 to 1988: {gap85[k:T0].mean():.2f}")
print(f"Gap in 1989: {gap85[T0]:.2f}; mean gap 1989 to 2000: {gap85[T0:].mean():.2f}")
print(f"Mean fake gap divided by the baseline ATT: {gap85[k:T0].mean() / att:.2f}")

Weights: Utah 0.351, Connecticut 0.348, Nevada 0.301
Pre-period RMSE, 1970 to 1984: 0.907
year actual synthetic gap
1985 102.8 106.11 -3.31
1986 99.7 103.27 -3.57
1987 97.5 106.14 -8.64
1988 90.1 98.47 -8.37
Mean fake gap, 1985 to 1988: -5.97
Gap in 1989: -14.11; mean gap 1989 to 2000: -18.75
Mean fake gap divided by the baseline ATT: 0.31

The fake-date fit uses three donors, Utah (0.351), Connecticut (0.348), and Nevada (0.301), and it fits 1970–1984 closely, with an RMSE of 0.907. The gaps for 1985–1988 are not zero: they range from −3.31 to −8.64 packs and average −5.97, or 0.31 times the baseline ATT. In 1989, the gap jumps to −14.11 packs, and over 1989–2000 it averages −18.75, close to the baseline ATT of −18.98.

The fake gaps call for a cautious reading of the timing. During 1985–1988, California was already falling faster than a synthetic California fitted to the years before 1985. Part of this pattern may be ordinary prediction error, since the model predicts four years ahead from a shorter fitting period. Part may reflect other changes in California before 1989, which these data cannot identify. Anticipation is a less likely source, because voters approved the measure only in November 1988 (Section 5.4), whereas the fake gap already opens in 1985. The gap steps down by 5.74 packs in 1989, but steps of similar size already occur in 1985 and 1987. The test therefore supports the timing of the effect only in part.

The figure shows the fake-date fit in two panels. Panel (a) plots actual and synthetic sales, and panel (b) plots the gap with lines at the fake date and at 1989. The shaded band marks the fake post-treatment years, 1985–1988.

Show the plotting code

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5.6))
fig.patch.set_linewidth(0)
for ax in (ax1, ax2):
ax.axvspan(FAKE_YEAR, TREAT_YEAR, color=GRID_LINE, alpha=0.55, zorder=0)
ax.axvline(FAKE_YEAR, color=GOLD, linestyle="--", linewidth=1.6,
label=f"Fake start ({FAKE_YEAR})")
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.6,
label="Proposition 99 (1989)")
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_xlabel("Year", fontsize=12)
ax1.plot(YEARS, ca_sales, color=WARM_ORANGE, linewidth=3.0, label="California")
ax1.plot(YEARS, synth85, color=STEEL_BLUE, linewidth=2.6, linestyle="--",
label=f"Synthetic California (fit to {FAKE_YEAR - 1})")
ax1.set_ylim(30, 140)
ax1.set_ylabel("Cigarette sales (packs per capita)", fontsize=12)
ax1.set_title("(a) Observed and synthetic paths", fontsize=13, fontweight="bold", pad=10)
ax1.legend(loc="lower left", fontsize=10)
ax2.axhline(0, color=LIGHT_TEXT, linewidth=0.9)
ax2.plot(YEARS, gap85, color=TEAL, linewidth=2.6, marker="o", markersize=4,
label="Gap with the fake start")
ax2.set_ylim(-30, 8)
ax2.set_ylabel("Gap (packs per capita)", fontsize=12)
ax2.set_title("(b) Gap", fontsize=13, fontweight="bold", pad=10)
ax2.legend(loc="lower left", fontsize=10)
fig.suptitle(f"In-time placebo test with a fake start in {FAKE_YEAR}",
fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 10. In-time placebo with a fake start in 1985. The fake gaps are about one third of the real effect.

The figure shows a divergence in steps rather than a single break at 1989. In panel (a), synthetic California, fitted to 1970–1984, stays above actual sales in every year from 1985 to 2000. In panel (b), the gap steps down in 1985, 1987, and 1989, from 1.27 packs in 1984 to −14.11 in 1989. After 1989, it widens more slowly, to −25.58 packs in 2000. Even without the program, the drift of 1985–1988 and the usual growth of prediction errors could therefore have produced part of the larger gaps after 1989.

Stata benchmark. The in-time table of synth2 reports fake gaps of −3.33, −3.59, −8.65, and −8.39 packs for 1985–1988, within 0.02 of the Python values. The same command also prints an RMSE of 2.205 and an R-squared of 0.953. These statistics belong to a fit that uses the six predictors of the fake-date design with the real start year, 1989. They do not describe the fake-date fit, as the companion script of this post confirms.

The in-time test therefore gives only partial support to the timing of the effect. The last robustness check asks a different question: does the result depend on any single donor state? Section 10 removes the donors one at a time.

10. Leave-one-out robustness

Synthetic California rests on five donor states, and Utah alone supplies a third of the recipe. If one donor drove the result, removing it would change the estimate sharply. The leave-one-out check refits the model five times, each time without one of the five positive-weight donors, as Abadie, Diamond, and Hainmueller (2015) recommend.

Predict first

The abstract already reports the range of the leave-one-out estimates. Suppose that we drop Utah, the largest donor, and refit the model with the remaining 37 donors. Which state will take over its role, and will the pre-treatment fit improve or worsen? Commit to an answer before scrolling.

Reveal the answer

Answer. New Mexico, which receives no weight in the baseline, becomes the leading donor with 0.614, almost twice the 0.335 that Utah held. Montana drops out of the recipe, and the pre-treatment RMSE worsens from 1.754 to 2.584. The ATT nevertheless stays close to the baseline, at −17.52 packs, the smallest reduction among the five removals.

def leave_one_out(panel, donors):
"""Refit the baseline once without each donor in turn."""
return {d: fit_sc(panel[panel["state"] != d]) for d in donors}
loo_donors = list(w_pos) # the five positive-weight donors
loo_fits = leave_one_out(panel, loo_donors)
loo = {}
for d, r in loo_fits.items():
g = np.asarray(r.gap, dtype=float)
loo[d] = {"synthetic": np.asarray(r.counterfactual, dtype=float), "gap": g,
"att": float(r.att)}
new_w = dict(sorted(r.donor_weights.items(), key=lambda kv: -kv[1]))
lead = next(iter(new_w))
print(f"Without {d:<12} ATT {r.att:7.2f} pre RMSE {r.pre_rmse:.3f} "
f"gap 2000 {g[-1]:7.2f} largest weight: {lead} {new_w[lead]:.3f}")
atts = [loo[d]["att"] for d in loo_donors]
print(f"ATT range: {min(atts):.2f} to {max(atts):.2f} (baseline {att:.2f})")

Without Utah ATT -17.52 pre RMSE 2.584 gap 2000 -23.48 largest weight: New Mexico 0.614
Without Nevada ATT -19.29 pre RMSE 2.285 gap 2000 -27.15 largest weight: Utah 0.551
Without Montana ATT -17.56 pre RMSE 1.924 gap 2000 -24.01 largest weight: Utah 0.384
Without Colorado ATT -19.26 pre RMSE 1.780 gap 2000 -26.12 largest weight: Utah 0.342
Without Connecticut ATT -18.89 pre RMSE 1.940 gap 2000 -25.64 largest weight: Utah 0.339
ATT range: -19.29 to -17.52 (baseline -18.98)

The five refits keep the ATT between −19.29 (without Nevada) and −17.52 (without Utah), within 1.46 packs of the baseline of −18.98. Dropping Utah worsens the fit most, because no other donor sells as little as Utah. New Mexico, the state with the next lowest sales, takes over as the leading donor with a weight of 0.614, but it replaces Utah only imperfectly. The gap in 2000 stays between −27.15 and −23.48 packs in every refit, so the effect remains large and negative whichever donor is removed.

The figure overlays the five refits on the baseline. Panel (a) shows the synthetic paths, and panel (b) shows the corresponding gaps. Each color marks one dropped donor.

Show the plotting code

LOO_COLORS = dict(zip(loo_donors, [GOLD, TEAL, LIGHT_ORANGE, LAVENDER, SAGE]))
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5.8))
fig.patch.set_linewidth(0)
for ax in (ax1, ax2):
ax.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5)
ax.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax.set_xlabel("Year", fontsize=12)
ax1.plot(YEARS, ca_sales, color=WARM_ORANGE, linewidth=3.0, label="California", zorder=4)
ax1.plot(YEARS, synth, color=STEEL_BLUE, linewidth=2.6, linestyle="--",
label="Baseline synthetic", zorder=3)
for d in loo_donors:
ax1.plot(YEARS, loo[d]["synthetic"], color=LOO_COLORS[d], linewidth=1.4,
label=f"Without {d}", zorder=2)
ax1.set_ylim(30, 140)
ax1.set_ylabel("Cigarette sales (packs per capita)", fontsize=12)
ax1.set_title("(a) Synthetic California without each donor", fontsize=13,
fontweight="bold", pad=10)
ax1.legend(loc="lower left", fontsize=10)
ax2.axhline(0, color=LIGHT_TEXT, linewidth=0.9)
ax2.plot(YEARS, gap, color=STEEL_BLUE, linewidth=2.6, label="Baseline gap", zorder=3)
for d in loo_donors:
ax2.plot(YEARS, loo[d]["gap"], color=LOO_COLORS[d], linewidth=1.4,
label=f"Without {d}", zorder=2)
ax2.set_ylim(-34, 8)
ax2.set_ylabel("Gap (packs per capita)", fontsize=12)
ax2.set_title("(b) Gaps", fontsize=13, fontweight="bold", pad=10)
ax2.legend(loc="lower left", fontsize=10)
fig.suptitle("Leave-one-out check: dropping each of the five donors in turn",
fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 11. Leave-one-out refits that drop each positive-weight donor in turn. Every refit keeps the gap negative after 1988, and below −17 packs from 1994 onward.

All six synthetic paths stay close to each other before 1989 and well above California afterward. The refit without Nevada produces the largest gaps in the late 1990s, with −30.62 packs in 1997. Its recipe leans on Utah (0.551) and New Hampshire (0.185). Sales in New Hampshire rose from 143.7 packs in 1992 to 174.4 in 1997, which lifts the synthetic path of this refit. Single years can move by several packs across refits, but the average effect barely changes.

Stata benchmark. The loo option of synth2 reports a range of gaps from −28.35 to −23.49 packs in 2000 and from −30.62 to −17.99 packs in 1997. The 1997 range agrees with the Python range within 0.01 packs at both ends. The 2000 range agrees at its upper end, but its lower end differs by 1.20 packs. The most likely cause is that the Stata edition runs its leave-one-out refits without allopt, which can stop the search at another optimum (Section 11). Stata prints only the smallest and the largest gaps, not the individual refits, so the log cannot confirm this cause.

The three robustness checks are complete. Before we explore other estimators, we pause to take stock of the replication. Readers who do not need the comparison with Stata can skip to Section 12, and readers interested only in the policy result can go directly to Section 15.

11. Replication scorecard

This section compares the Python results with their counterparts in the Stata log. The table lists the quantities in the order of the tutorial and labels each agreement as exact, close, or different. The paragraphs after the table explain the source of every difference.

Step	Quantity	Stata `synth2`	mlsynth `VanillaSC`	Agreement
Data	Rows, states, years	1,209; 39; 1970–2000	1,209; 39; 1970–2000	Exact
Predictors	Means for California and the donor average	10.0766 … 136.9316	The same 14 values	Exact to the printed decimals
Donor weights	Utah, Nevada, Montana, Colorado, Connecticut	0.3340, 0.2350, 0.2020, 0.1610, 0.0680	0.3351, 0.2356, 0.2019, 0.1595, 0.0679	Close, within 0.002
Predictor weights V	Age share, retail price, sales in 1975	0.5459, 0.0174, 0.4221	0.3316, 0.3342, 0.3343	Different, V is not identified
Fit	Pre-treatment RMSE	1.756	1.754	Close
Fit	R-squared	0.974	0.976, or 0.974 with the Stata definition	Different definitions
Effect	ATT	−19.00	−18.98, or −19.00 with the Stata weights	Close, exact with the Stata weights
Effect	Gaps in 1989 and 2000	−7.59 and −25.76	−7.59 and −25.73	Close, within 0.03
In-space placebo	Rank and p-value	1 of 39; 0.026	1 of 39; 0.026	Exact
In-space placebo	MSPE ratio of California	123.5	129.0	Different, Stata refits
cut(2)	States kept and p-value	20; 0.050	20; 0.050	Exact, the same 20 states
Pointwise test	Years with the left-sided p at its floor	8 of 12	9 of 12	Close
In-time placebo	Gaps for 1985–1988	−3.33, −3.59, −8.65, −8.39	−3.31, −3.57, −8.64, −8.37	Close, within 0.02
Leave-one-out	Range of gaps in 2000	−28.35 to −23.49	−27.15 to −23.48	Upper end close, lower end different
Leave-one-out	Range of gaps in 1997	−30.62 to −17.99	−30.62 to −17.98	Close

Weights in the third decimal. The two programs solve the same nested problem with different optimizers, which stop at slightly different points of a flat objective. The resulting weights differ by less than 0.002 for every donor, and the 12 gaps differ by at most 0.03 packs. These differences are numerical, not substantive, and they never change a conclusion.

Rounded weights in Stata. The synth routine that synth2 calls rounds the donor weights to three decimals before it predicts the synthetic path. Applying those rounded weights to the same data by hand reproduces the Stata ATT of −19.0018 and all 12 Stata gaps. The Stata ATT therefore rests on rounded weights. The two weight vectors also differ beyond rounding: Colorado differs by 0.0015, three times the largest change that rounding to three decimals can produce. The ATT difference of 0.02 packs thus combines this rounding with the optimizer differences described above. The printed Stata RMSE of 1.756, by contrast, comes from the unrounded weights, because synth computes it before it rounds them.

Predictor weights. The predictor weights differ sharply, yet the donor weights agree, because V is not identified (see the proof card of Section 5.2). Many V produce the same donor weights, and each program reports the one that its own search finds. Exercise 6 confirms the point from the Stata side, since the Stata V, fed into the inner problem in Python, returns the Stata donor weights. The difference in V therefore needs no further reconciliation.

Two ratios and two fits. The built-in test of mlsynth ranks the RMSPE ratio, while synth2 ranks the MSPE ratio, which is its square. Within one set of placebo fits, the ranking is the same either way (see the proof card of Section 8.1). The MSPE ratio of California still differs, 129.0 in Python against 123.5 in Stata. The reason is that the Stata placebo command refits California without allopt and with the looser convergence criterion sigf(6), which changes both components of the ratio. The pre-treatment MSPE rises from 3.08 to 3.17, and the post-treatment MSPE falls from 397.02 to 391.25. The difference between the two ratios therefore reflects a different fit, not a different statistic.

Two definitions of R-squared. The mlsynth library subtracts from one the ratio of the squared pre-treatment misses to the variation of actual sales in California, the textbook definition. The synth2 command puts the variation of synthetic California in that denominator instead. The block below computes both versions from the same gaps.

ssr = np.sum(gap[:T0] ** 2) # squared misses before 1989
r2_mlsynth = 1 - ssr / np.sum((ca_sales[:T0] - ca_sales[:T0].mean()) ** 2)
r2_synth2 = 1 - ssr / np.sum((synth[:T0] - synth[:T0].mean()) ** 2)
print(f"R-squared over the variation of California (mlsynth): {r2_mlsynth:.5f}")
print(f"R-squared over the variation of synthetic California (synth2): {r2_synth2:.5f}")

R-squared over the variation of California (mlsynth): 0.97621
R-squared over the variation of synthetic California (synth2): 0.97435

The textbook definition gives 0.976, and the Stata definition gives 0.974, the value in the Stata log. With the same definition, the two programs agree to four decimals. The difference in the R-squared is therefore a matter of definition, not of fit.

Refits inside Stata commands. The Stata edition runs its placebo and leave-one-out commands without allopt. A note in its do-file attributes this omission in the placebo run to computation time. Each command therefore uses the weaker search for its refit of the baseline and for every placebo or leave-one-out fit. The placebo run also sets sigf(6), a looser convergence criterion than the default. The help files of synth and synth2 give this default as seven significant figures, but the code of synth sets it to 12. The sigf option is the only setting that differs between the two refits of the baseline, which explains why their results differ. The placebo run reports an ATT of −18.83 with an RMSE of 1.780, and the leave-one-out run reports −18.87 with an RMSE of 1.783. The weaker search most likely also explains the differences in the leave-one-out ranges, although Stata does not print the individual refits that would confirm it.

Floor years of the pointwise test. Both the refit of California and the Stata placebo fits move single years across the floor of the left-sided p-value. Against the Python placebo gaps, the refit of California alone already lowers the count from 9 years to 8. The reason is that its gap of −7.42 in 1989 lies above the gap of Idaho, −7.51. The Stata placebo fits then change which years reach the floor, 1989 instead of 1992, but not their number. The difference between 9 and 8 floor years thus traces back to the refit of California.

Seeds and placebo fits. The search over V uses a random seed, and the placebo fits of other states depend on it more than the fit of California does. With seed 42 and the pinned stack, the Python loop keeps exactly the 20 states that Stata keeps. South Dakota sits at 2.026, just above the cutoff of 2, so another seed or library version could add it and change the cut(2) p-value to 1/21.

Taken together, the scorecard shows a close reproduction. The donor weights, the RMSE, the ATT, and the gaps of the shared fit all agree closely. The ATT agrees exactly once the rounded Stata weights are applied. Every remaining difference has a documented or probable source: small optimizer differences, rounding, the non-identified V, a definition, or a refit inside a Stata command. With the benchmark settled, the next section leaves Stata behind and asks how other estimators in mlsynth answer the same question.

12. A short tour of other mlsynth estimators

The mlsynth library offers many estimators behind the same dictionary interface. This section compares the baseline with three of them, each with a different idea of what a good counterfactual should match. All four estimators target the same estimand, the ATT for California over 1989–2000.

12.1 Three more estimators in a few lines

The simplest alternative keeps VanillaSC but drops the predictors. The helper base_config() of Section 8.2 returns the keys that every estimator needs, without any predictors. With these keys alone, VanillaSC matches the 19 pre-treatment outcomes directly, which yields the outcome-only synthetic control. The baseline, by contrast, matches the seven predictors of Abadie, Diamond, and Hainmueller (2010), which the code labels as the ADH predictors.

Synthetic difference-in-differences, available as SDID, combines unit weights with time weights and a constant level shift (Arkhangelsky et al. 2021). Its time weights favor the pre-treatment years that best predict the post-treatment sales of the donors. Its key B sets the number of placebo draws behind a standard error. The code passes 500, the default of mlsynth 1.0.0. The tour, however, reports only the point estimate, which does not depend on B. In mlsynth 1.0.0, the result object exposes neither set of weights, so the tour cannot show which donors SDID uses. Its counterfactual already includes the level shift, so we plot it as it is.

The class CLUSTERSC with method="pcr" takes a different route. It first selects a cluster of donors similar to California (Rho et al. 2025). It then applies principal component regression (Amjad, Shah, and Shen 2018; Agarwal et al. 2021). This regression summarizes donor sales by a few common patterns and regresses California on them. Because it places no sign constraint on the weights, some of them can be negative. Its donor weights sit in the attribute pcr of the result object.

base = base_config(panel) # data keys only, without predictors
res_outcome = VanillaSC({**base, "inference": True}).fit() # outcome-only SC
res_sdid = SDID({**base, "B": 500, "seed": RANDOM_SEED}).fit() # SDID, 500 placebo draws
res_clus = CLUSTERSC({**base, "method": "pcr"}).fit() # PCR with clustering
res_clus_nc = CLUSTERSC({**base, "method": "pcr", "clustering": False}).fit()
tour = [
{"key": "vanillasc_adh", "estimator": "VanillaSC, ADH predictors",
"counterfactual": synth, "att": att},
{"key": "vanillasc_outcome", "estimator": "VanillaSC, outcome only",
"counterfactual": np.asarray(res_outcome.counterfactual, dtype=float),
"att": float(res_outcome.att)},
{"key": "sdid", "estimator": "Synthetic DiD (SDID)",
"counterfactual": np.asarray(res_sdid.counterfactual, dtype=float),
"att": float(res_sdid.att)},
{"key": "clustersc_pcr", "estimator": "CLUSTERSC, PCR",
"counterfactual": np.asarray(res_clus.counterfactual, dtype=float),
"att": float(res_clus.att)},
]
for t in tour:
miss = ca_sales - t["counterfactual"]
print(f"{t['estimator']:<26} ATT {t['att']:7.2f} "
f"pre RMSE {np.sqrt(np.mean(miss[:T0] ** 2)):.3f} gap 2000 {miss[-1]:7.2f}")
w_out = {s: w for s, w in res_outcome.donor_weights.items() if w > 1e-6}
print(f"Outcome only: {len(w_out)} positive donors; placebo rank "
f"{res_outcome.inference.details['rank']} of 39, p = {res_outcome.inference.p_value:.3f}")
w_clus = res_clus.pcr.donor_weights
print(f"CLUSTERSC: {len(w_clus)} donors, {sum(v < 0 for v in w_clus.values())} negative "
f"weights, weights sum to {sum(w_clus.values()):.3f}")
print(f"CLUSTERSC without clustering: ATT {res_clus_nc.att:.2f}")

VanillaSC, ADH predictors ATT -18.98 pre RMSE 1.754 gap 2000 -25.73
VanillaSC, outcome only ATT -19.51 pre RMSE 1.656 gap 2000 -26.60
Synthetic DiD (SDID) ATT -15.61 pre RMSE 1.799 gap 2000 -24.50
CLUSTERSC, PCR ATT -21.39 pre RMSE 1.503 gap 2000 -32.88
Outcome only: 6 positive donors; placebo rank 3 of 39, p = 0.077
CLUSTERSC: 34 donors, 11 negative weights, weights sum to 0.918
CLUSTERSC without clustering: ATT -19.37

All four estimates are large and negative, although they range from −21.39 for CLUSTERSC to −15.61 for SDID. The outcome-only fit uses six donors and fits better than the baseline, with a pre-treatment RMSE of 1.656 against 1.754. CLUSTERSC spreads its weight over 34 donors and gives 11 of them negative weights. Its weights also sum to 0.918, because its default regression imposes neither constraint of the simplex. Its estimate moves from −21.39 to −19.37 when the clustering step is switched off, a reminder that default settings matter.

12.2 Comparison

The table collects the four estimators. Each row states what the estimator matches, how it constrains the weights, and the three summary numbers of this tutorial. The ATT column answers the same question for every row, so the differences come from the design and the settings of each estimator.

Estimator	mlsynth call	What it matches	Weight rule	Donors with weight	Pre-treatment RMSE	ATT	Gap in 2000
SC with predictors (baseline)	`VanillaSC(config)`	Seven predictors	Nonnegative, sum to one	5 of 38	1.754	−18.98	−25.73
SC, outcome only	`VanillaSC({**base, "inference": True})`	The 19 pre-treatment outcomes	Nonnegative, sum to one	6 of 38	1.656	−19.51	−26.60
Synthetic DiD	`SDID({**base, "B": 500, "seed": 42})`	Outcomes up to a constant, with time weights	Unit and time weights plus an intercept	Not exposed in 1.0.0	1.799	−15.61	−24.50
PCR with clustering	`CLUSTERSC({**base, "method": "pcr"})`	Principal components of the donors in the cluster of California	Unconstrained, negative weights allowed	34, of which 11 negative	1.503	−21.39	−32.88

The figure compares the four counterfactual paths in panel (a) and the four ATTs in panel (b). The hollow marker at −27.35 is a two-way fixed effects (TWFE) estimate of difference-in-differences. It comes from a regression that gives every state and every year its own intercept. This estimate weights all 38 donors equally, and Exercise 5 reproduces it with statsmodels. The plotting code computes that reference value by removing state means and year means from each series of the panel, a step called two-way demeaning.

Show the plotting code

def two_way_demean(s, data):
"""Remove state and year means from a series of a balanced panel."""
return (s - s.groupby(data["state"]).transform("mean")
- s.groupby(data["year"]).transform("mean") + s.mean())
# TWFE DiD reference, computed by two-way demeaning (see Exercise 5)
d_tilde = two_way_demean(panel["treated"].astype(float), panel)
y_tilde = two_way_demean(panel["cigsale"], panel)
twfe_att = float((d_tilde * y_tilde).sum() / (d_tilde ** 2).sum())
tour_plot = tour + [{"key": "twfe_did", "estimator": "TWFE DiD (reference)",
"att": twfe_att}]
TOUR_COLORS = {"vanillasc_adh": STEEL_BLUE, "vanillasc_outcome": TEAL,
"sdid": GOLD, "clustersc_pcr": LAVENDER, "twfe_did": GREY_DONOR}
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5.8),
gridspec_kw={"width_ratios": [1.35, 1]})
fig.patch.set_linewidth(0)
ax1.axvline(TREAT_YEAR, color=LIGHT_TEXT, linestyle=":", linewidth=1.5)
ax1.plot(YEARS, ca_sales, color=WARM_ORANGE, linewidth=3.0, label="California", zorder=5)
for t in tour_plot[:4]:
ax1.plot(YEARS, t["counterfactual"], color=TOUR_COLORS[t["key"]], linewidth=2.0,
linestyle="--", label=t["estimator"], zorder=3)
ax1.set_xlim(FIRST_YEAR - 0.5, LAST_YEAR + 0.5)
ax1.set_ylim(30, 140)
ax1.set_xlabel("Year", fontsize=12)
ax1.set_ylabel("Cigarette sales (packs per capita)", fontsize=12)
ax1.set_title("(a) Observed and counterfactual paths", fontsize=13,
fontweight="bold", pad=10)
ax1.legend(loc="lower left", fontsize=10)
yy = np.arange(len(tour_plot))
for i, t in enumerate(tour_plot):
hollow = t["key"] == "twfe_did"
ax2.plot(t["att"], i, marker="o", markersize=12, linestyle="none",
color=TOUR_COLORS[t["key"]],
markerfacecolor="none" if hollow else TOUR_COLORS[t["key"]],
markeredgewidth=2)
ax2.text(t["att"] + 1.0, i, signed(t["att"]), fontsize=11, color=WHITE_TEXT,
va="center")
ax2.axvline(0, color=LIGHT_TEXT, linewidth=0.9)
ax2.set_yticks(yy)
ax2.set_yticklabels([t["estimator"] for t in tour_plot], fontsize=11)
ax2.set_ylim(len(tour_plot) - 0.5, -0.5)
ax2.set_xlim(-31, 2)
ax2.set_xlabel("ATT, 1989–2000 (packs per capita)", fontsize=12)
ax2.set_title("(b) Average effect on California", fontsize=13, fontweight="bold", pad=10)
fig.suptitle("One case, four synthetic control estimators, and a DiD reference",
fontsize=14, fontweight="bold", color=WHITE_TEXT)
plt.tight_layout()
plt.show()

Figure 12. Four synthetic control estimators and a TWFE reference. All four estimators find a large reduction.

The weighting rule of each estimator helps to read panel (b). TWFE weights all 38 donors equally, so it inherits the poor pre-1989 match of the simple average from Section 4. Its estimate is therefore far more negative than the four synthetic control estimates. The two VanillaSC fits use sparse simplex weights, and their estimates nearly coincide, at −18.98 and −19.51. CLUSTERSC uses dense, signed weights, which let its counterfactual extrapolate beyond the donors. Its estimate of −21.39 is the most negative of the four. SDID adds time weights and a level shift, and its estimate of −15.61 is the least negative of the four. Version 1.0.0, however, does not expose the SDID weights. On the same data, estimators with different weighting rules thus give answers several packs apart.

The tour also teaches a lesson about fit. CLUSTERSC has the lowest pre-treatment RMSE of all four, 1.503. It reaches that fit, however, with negative weights, which let its counterfactual extrapolate beyond the donors. The outcome-only fit beats the baseline on fit by construction, because it matches the 19 pre-treatment outcomes directly. Its own placebo test, however, ranks California only third (p = 0.077), so a closer fit need not yield stronger evidence. A lower pre-treatment RMSE therefore does not, by itself, make an estimator more credible or its inference more decisive.

Stata benchmark. The Stata edition has no counterpart to this tour, because synth2 implements only the classic estimator of Section 5. Its ATT of −19.00 lies between the baseline estimate and the outcome-only estimate of mlsynth. The other two estimators move the answer to −15.61 and −21.39, so the choice of estimator matters more than the choice of software.

The tour shows that the answer depends on how an estimator weights the donors. The next section lets readers test this lesson directly. Its interactive lab shows how the estimates respond as each choice of the walkthrough changes.

13. Try it yourself: an interactive synthetic control lab

The lab below runs on the 1,209 observations of this post. Each tab isolates one decision that the walkthrough made: the donor recipe, the placebo cutoff, the fake treatment date, and the donor that we remove. At its default settings, the lab reproduces the numbers above. These include the ATT of −18.98 packs, the cut(2) p-value of 0.050, and the leave-one-out range from −19.29 to −17.52 packs.

The nine experiments below guide a first visit to the lab. Each one names the tab, the control to change, and the numbers to expect. Together, they revisit the main choices of Sections 7 to 12.

One donor is not a counterfactual. In the weight mixer, click “Utah only”. The pre-treatment RMSE jumps to 45.328, and the ATT turns positive, at 8.62 packs. Then raise the Montana slider and set Utah to zero. Montana alone fits better, with an RMSE of 4.475, but its ATT of −25.36 lies far from the fitted estimate of −18.98.
The right donors with the wrong recipe. Click “Equal fifths”, which gives each of the five donors a weight of 0.200. The RMSE rises to 4.302, and the ATT moves to −22.05, so the same five states need the fitted shares.
The naive comparison. Click “mlsynth fit”, and check the box for the unweighted average of the 38 donors. A second block of readouts now shows the average next to the fitted recipe. Its RMSE of 16.044 dwarfs the fitted value of 1.754, and its ATT of −41.71 lies far from the estimate of −18.98.
Two optimizers, one recipe. Click “Stata weights”. The RMSE reads 1.754, as in the mlsynth fit, and the ATT moves from −18.98 to −19.00. The Stata weights differ slightly from the mlsynth weights, yet the estimate barely moves. Stata itself prints an RMSE of 1.756, because it computes the RMSE from its unrounded weights (Section 11).
A closer fit, a weaker test. Click “Outcome-only fit”. The sixth slider now holds New Hampshire at 0.045, the RMSE falls to 1.656, and the ATT reads −19.51, the outcome-only estimate of Section 12. Section 12 showed, however, that its placebo test is less decisive.
Delete versus refit. Click “mlsynth fit” and drag the Utah slider to zero, which removes Utah without a refit. The RMSE climbs to 22.756, and the ATT reads −32.89. In the leave-one-out tab, select Utah instead: the refit gives New Mexico a weight of 0.614, an RMSE of 2.584, and an ATT of −17.52.
Tighten the placebo filter. In the placebo cutoff tab, set the cutoff to 1. Only 9 states remain, and the p-value becomes 0.111. California still ranks first, but the p-value can no longer reach 0.050.
Loosen the placebo filter. Set the cutoff to 5: 31 states remain, with p = 0.032. At the “No cutoff” stop, all 39 states remain, with p = 0.026. Watch the count of years at the smallest left-sided p-value. It falls from 9 at the default cutoff to 0 without a cutoff. The badly fitted placebo of Rhode Island explains the drop, because its gap lies below that of California in every year after 1988.
Move the fake date. In the in-time placebo tab, select each fake start from 1985 to 1988 in turn. The mean fake gap stays between −6.96 and −3.45 packs. The mean gap over 1989–2000 stays between −18.75 and −17.72, much larger in absolute value in every case.

The lab turns the main choices of the walkthrough into questions with visible answers. First, the recipe matters more than the list of ingredients. The same five states give −22.05 with equal shares and −18.98 with the fitted weights, and Utah alone even reverses the sign. Second, the p-value depends on the comparison set, because it never falls below one over the number of retained states. Third, the effect of Proposition 99 survives every check that refits the model. Every fake start, however, already yields a mean fake gap between −6.96 and −3.45 packs before 1989. Results like these are easy to misread, and the next section collects the most common misreadings.

14. Common misconceptions

Synthetic control results are easy to state and easy to misread. Each card below states a tempting claim about this analysis. It then shows what the numbers of this tutorial imply instead.

Misconception "Synthetic California is built from its neighbors."

What is actually true. Of the five donors, only Nevada borders California. Arizona and Oregon, the other neighbors, are absent from the data, because they ran their own large tobacco programs. Utah (0.335), Montana (0.202), Colorado (0.160), and Connecticut (0.068) enter for another reason. Together with Nevada, their weighted combination tracks the sales of California before 1989 and matches most of its predictors. A donor need not resemble California on its own, since Utah, the largest donor, sold 55.0 packs in 1988 against 90.1 in California. The recipe is chosen for its fit, not for its geography.

Misconception "p = 0.026 is the probability that Proposition 99 had no effect."

What is actually true. The p-value is the share of states whose MSPE ratio is at least as large as that of California. It cannot fall below 1/39 = 0.026, and it equals 1/20 = 0.050 after cut(2). It describes the rank of California in a placebo distribution, not the probability of any hypothesis.

Misconception "A sound in-time placebo must show gaps of exactly zero."

What is actually true. Even without any policy, prediction errors grow as the model forecasts beyond its fitting period, so fake gaps are rarely exactly zero. A sound in-time placebo shows fake gaps that are small relative to the real effect and no clear break at the fake date. Here the fake gaps for 1985–1988 average −5.97 packs, about one third of the mean of −18.75 over 1989–2000. Moreover, the gap already steps down at the fake date. This test therefore meets that standard only in part.

Misconception "The estimator with the smallest pre-treatment error is the most credible."

What is actually true. CLUSTERSC has the lowest pre-treatment RMSE (1.503), but it uses 34 donors, 11 of them with negative weights. These weights let the counterfactual extrapolate beyond the donors. The outcome-only fit beats the baseline on fit by construction (1.656 against 1.754), because it matches the 19 pre-treatment outcomes directly. Its placebo test, however, is less decisive (rank 3, p = 0.077). A close fit is necessary for credibility, but it does not certify the counterfactual.

Misconception "Dropping badly fitted placebos with cut(2) can only strengthen the evidence."

What is actually true. The filter raises the smallest attainable p-value from 0.026 to 0.050, because the comparison set shrinks from 39 to 20 states. With a cutoff of 1, only 9 states remain, so the p-value cannot fall below 0.111, even though California ranks first. The filter buys comparability of the gaps at the cost of resolution.

These misconceptions share one root: each reads a result without the design that produced it. The discussion below returns to the question of the tutorial with this caution in mind. It weighs the evidence as a whole and states what that evidence implies for policy.

15. Discussion

15.1 Answering the question

The question was how much Proposition 99 reduced cigarette sales in California. The synthetic control estimate is a reduction of 18.98 packs per capita per year over 1989–2000, or 23.9 percent of the sales that synthetic California predicts. The effect grew over time and reached 25.73 packs, or 38.2 percent, in 2000. The gaps of 1999 and 2000, however, may also reflect Proposition 10, a further tax increase of 50 cents per pack in January 1999.

The evidence for this answer is strong but not perfect. California has the largest MSPE ratio of all 39 states, the leave-one-out estimates stay between −19.29 and −17.52 packs, and four estimators agree on a large reduction. The in-time placebo is the weak spot, since a fake start in 1985 yields gaps about one third as large as the real effect. On balance, the evidence supports a large reduction, while the exact timing of its onset remains less certain.

15.2 Policy implications

The estimates carry a clear message for tobacco control policy. California combined a tax increase of 25 cents per pack with anti-smoking education. A large and persistent fall in taxed cigarette sales followed, well beyond the decline of synthetic California. For policymakers, the result suggests that such programs can reduce cigarette sales at scale. Their effect also builds over time rather than fading, as the gaps up to 1998 already show. Because the estimate covers the whole program, it speaks to the combination of a tax and education rather than to either measure alone.

The estimate is an ATT for California, however, not a forecast for every state. Other states differ in smoking habits, prices, and enforcement, so the same program could have a smaller or a larger effect elsewhere. A state that considers a similar program should treat the figure of 23.9 percent as a benchmark from one successful case, not as a guarantee.

15.3 Limitations

Several limitations qualify these results. Some come from the data, some from the method, and some from the software. The list below takes them in that order.

The data end in 2000. The tutorial cannot say whether the effect persisted, faded, or grew after 2000.
Sales are not smoking. The outcome counts taxed packs sold, so cross-border purchases can make sales fall faster than consumption.
A second tax increase. Proposition 10 raised the state cigarette tax by a further 50 cents per pack in January 1999, so the gaps of 1999 and 2000 cannot be attributed to Proposition 99 alone.
Spillovers. Residents of California could buy cigarettes in Nevada, which receives a weight of 0.236, and such purchases would inflate the synthetic path. If they did, dropping Nevada would shrink the estimated reduction. The refit without Nevada instead gives a slightly larger reduction (−19.29 against −18.98). That refit, however, shifts 0.185 of the weight to New Hampshire, whose own sales rose sharply after 1992, so this check cannot rule out spillovers.
Early divergence. The fake gaps of 1985–1988 show that California was already pulling away from a synthetic California fitted before 1985.
Coarse p-values. With 39 states, the placebo p-value cannot fall below 0.026, and after cut(2) the pointwise p-values move in steps of 0.050.
Predictor choice. Dropping beer and the age share moves the ATT to −17.56 (Exercise 3), so the estimate depends modestly on the specification.
Software details. The third decimal of the weights depends on the optimizer. The cut(2) set may also change with the seed, because South Dakota sits just above the cutoff.

None of these limitations overturns the main result. Together, they confine it to taxed sales in California over 1989–2000, and they call for caution about its timing. A careful reading therefore treats the estimate as strong evidence for one state, not as a general law.

16. Summary and takeaways

The six points below condense the tutorial into its main results and lessons. Each pairs a finding with the number that supports it. Read together, they also show what a complete synthetic control study reports: the recipe, the effect, the placebo evidence, the robustness checks, the benchmark, and the limits of inference.

The counterfactual is a five-state recipe. Synthetic California combines Utah (0.335), Nevada (0.236), Montana (0.202), Colorado (0.160), and Connecticut (0.068), and it tracks California before 1989 with an RMSE of 1.754 packs.
The effect is large and grows over time. The ATT is −18.98 packs per capita per year, a reduction of 23.9 percent relative to mean synthetic sales. The gap reaches −25.73 packs (38.2 percent) in 2000, although Proposition 10, a second tax increase in January 1999, may contribute to the gaps of 1999 and 2000.
California is the most extreme state in the placebo test. Its MSPE ratio of 129.0 ranks first of 39 (p = 0.026), and it stays first after cut(2), with p = 0.050.
The result survives most robustness checks. Leave-one-out estimates range from −19.29 to −17.52, but a fake start in 1985 produces gaps about one third as large as the real effect.
The mlsynth library reproduces the Stata benchmark closely. The weights agree within 0.002, and the rounded Stata weights reproduce the Stata ATT exactly. The predictor weights V differ, because they are not identified.
Permutation inference has hard limits, and complementary tools exist. With 39 states, no in-space placebo p-value can fall below 0.026. One next step is the use of prediction intervals, which mlsynth offers through inference="scpi" and which the prediction interval tutorial teaches with the scpi_pkg package. Another is the leave-two-out refinement of the placebo test, available as inference="lto", whose p-values move on a finer grid (Lei and Sudijono 2025). Further steps are conformal inference (Chernozhukov, Wüthrich, and Zhu 2021) and models of spillovers, such as the Bayesian spatial synthetic control.

17. Exercises

The exercises below move from reading the result object to rebuilding parts of the method by hand. Each exercise states a task and the numbers to report, and each solution card holds code that was run with the stack of Section 2. Exercises 3, 4, and 5 mirror the three exercises of the Stata edition, so they can be solved in both languages. Try every exercise before you open its solution, because the attempt itself is where most of the learning happens.

17.1 Warm-up

Exercise 1: Read the result object. The result object res stores far more than the ATT. Report the ATT in packs and in percent, the number of positive donors, the sum of the weights, and the weight constraint. Add the gap in 2000. Then print res.time_series.intervention_time, and explain its value in light of Section 7.5. Compare the count of positive donors with the number of names in res.additional_outputs["donor_names"], and explain the difference.

Solution Show the code and the numbers

stats = res.weights.summary_stats
print(f"ATT: {res.att:.2f} packs ({res.effects.att_percent:.1f} percent)")
print(f"Positive donors: {stats['n_nonzero']} (n_donors = {stats['n_donors']}); "
f"donors in the pool: {len(res.additional_outputs['donor_names'])}")
print(f"Sum of weights: {stats['sum_of_weights']:.6f}; constraint: {stats['constraint']}")
print(f"Gap in 2000: {res.gap[-1]:.2f}")
print(f"intervention_time: {res.time_series.intervention_time}")

ATT: -18.98 packs (-23.9 percent)
Positive donors: 5 (n_donors = 5); donors in the pool: 38
Sum of weights: 1.000000; constraint: simplex (non-negative, sum to 1)
Gap in 2000: -25.73
intervention_time: None

The ATT is −18.98 packs, or −23.9 percent of mean synthetic sales after 1988, and the gap in 2000 is −25.73 packs. The weights sum to one under the simplex constraint. The field n_donors counts only the five positive donors, while donor_names lists all 38 states of the pool. The intervention time is None in VanillaSC of mlsynth 1.0.0, which is why the native plot needs a line at 1989 drawn by hand. Field names can therefore mislead, and printing a field is the safest way to learn what it holds.

Exercise 2: Rebuild synthetic California by hand. Equation 2 says that synthetic California is a weighted average of donor sales. Build the 31 by 39 matrix of sales, multiply it by the vector of mlsynth weights, and compare the result with res.counterfactual. The weight vector needs one entry per column of Y, with zero for California and for every state outside the recipe. The call res.donor_weights.get(state, 0.0) supplies these zeros. Then repeat the calculation with the rounded Stata weights and report both ATTs.

Solution Show the code and the numbers

Y_MAT = Y.to_numpy() # 31 years x 39 states
w_ml = np.array([res.donor_weights.get(s, 0.0) for s in STATES])
w_stata = np.array([STATA_W.get(s, 0.0) for s in STATES])
synth_hand = Y_MAT @ w_ml
print(f"Largest difference from res.counterfactual: {np.max(np.abs(synth_hand - synth)):.1e}")
print(f"ATT with the mlsynth weights: {(ca_sales - synth_hand)[T0:].mean():.4f}")
print(f"ATT with the Stata weights: {(ca_sales - Y_MAT @ w_stata)[T0:].mean():.4f}")

Largest difference from res.counterfactual: 1.4e-14
ATT with the mlsynth weights: -18.9816
ATT with the Stata weights: -19.0018

The hand-built path matches res.counterfactual up to floating-point error, a largest difference of 1.4e-14 packs. The counterfactual is therefore nothing more than the weighted average of Equation 2. With the rounded Stata weights, the same arithmetic gives −19.0018, exactly the ATT in the Stata log. The Stata ATT thus rests on rounded weights, and the difference of 0.02 packs reflects both this rounding and small differences between the two optimizers.

17.2 Core

Exercise 3: Change the predictor set. The choice of predictors is a modeling decision, and the Stata edition asks how much it matters. Refit the baseline without beer and age15to24, keeping the other five predictors and their windows. Report the new donor weights, the ATT, the pre-treatment RMSE, and the predictor with the largest weight in V, and compare them with the baseline.

Solution Show the code and the numbers

covs_ex3 = [c for c in COVARIATES if c not in ("beer", "age15to24")]
res_ex3 = fit_sc(panel, covariates=covs_ex3, windows={c: WINDOWS[c] for c in covs_ex3})
w_ex3 = dict(sorted(res_ex3.donor_weights.items(), key=lambda kv: -kv[1]))
v_ex3 = res_ex3.weights.summary_stats["predictor_weights"]
print("Weights:", ", ".join(f"{s} {w:.3f}" for s, w in w_ex3.items()))
print(f"ATT: {res_ex3.att:.2f}; pre-period RMSE: {res_ex3.pre_rmse:.3f}")
print(f"Largest predictor weight: {max(v_ex3, key=v_ex3.get)} ({max(v_ex3.values()):.3f})")
loo_m = loo_fits["Montana"] # the refit without Montana, Section 10
w_loo = dict(sorted(loo_m.donor_weights.items(), key=lambda kv: -kv[1]))
v_loo = loo_m.weights.summary_stats["predictor_weights"]
print("Weights without Montana:", ", ".join(f"{s} {w:.3f}" for s, w in w_loo.items()))
print(f"Largest predictor weight without Montana: {max(v_loo, key=v_loo.get)} "
f"({max(v_loo.values()):.3f})")

Weights: Utah 0.384, Colorado 0.273, Nevada 0.255, Connecticut 0.088
ATT: -17.56; pre-period RMSE: 1.924
Largest predictor weight: cigsale_1980 (1.000)
Weights without Montana: Utah 0.384, Colorado 0.273, Nevada 0.255, Connecticut 0.088
Largest predictor weight without Montana: cigsale_1980 (1.000)

Without beer and the age share, Montana leaves the recipe, and Utah (0.384), Colorado (0.273), Nevada (0.255), and Connecticut (0.088) remain. The fit worsens slightly, to an RMSE of 1.924, and the ATT shrinks in size to −17.56. The estimate thus depends modestly on the predictor set, as Section 15.3 notes.

These weights and this ATT also coincide with the leave-one-out refit without Montana, whose ATT Section 10 reports as −17.56. The last two lines of the output show the weights of that refit and its largest predictor weight. Both searches put essentially all of V on sales in 1980, so beer and the age share barely count in the refit either. Montana, moreover, receives zero weight here although it remains in the pool, so dropping it removes nothing from this recipe. The coincidence therefore rests on two facts specific to these data: the weight of V on sales in 1980 and the zero weight of Montana. It does not reflect a general rule.

The weight of 1.000 on sales in 1980 needs a careful reading. It says that the recipe matches that predictor exactly, yet many donor mixes match it equally well. The other predictors keep tiny weights, because the search of mlsynth never lets a predictor weight fall to exactly zero. These tiny weights then decide among the mixes that match sales in 1980 exactly. The reported V therefore marks the predictor that the recipe matches exactly, not the only characteristic that matters.

Exercise 4: Loosen the placebo filter. The cutoff of cut(2) is a convention, not a law. Apply a cutoff of 5 to the table placebo, report the number of states kept and the p-value, and list the removed states. Explain what a looser filter gains and what it loses.

Solution Show the code and the numbers

keep5, p5 = placebo_pvalue(placebo, 5)
print(f"cut(5) keeps {len(keep5)} of 39 states; p = {p5:.3f}")
print("Removed:", ", ".join(s for s in STATES if s not in keep5.index))
print(f"For comparison, cut(2) keeps {n_kept} states (p = {p_cut:.3f}), "
f"and no cutoff keeps 39 (p = {p_all:.3f})")

cut(5) keeps 31 of 39 states; p = 0.032
Removed: Delaware, Kentucky, Nevada, New Hampshire, North Carolina, Rhode Island, Utah, Wyoming
For comparison, cut(2) keeps 20 states (p = 0.050), and no cutoff keeps 39 (p = 0.026)

A cutoff of 5 keeps 31 states and gives p = 0.032, between the value of cut(2), 0.050, and the value without a filter, 0.026. The eight removed states, including Utah, New Hampshire, and Kentucky, have a pre-treatment MSPE more than five times that of California. A looser filter gains resolution, but it admits placebos whose gaps partly reflect poor fits.

17.3 Stretch

Exercise 5: Compare with difference-in-differences. Difference-in-differences is the most common alternative to synthetic control. Estimate a two-way fixed effects regression of cigsale on treated with state and year fixed effects, using smf.ols from statsmodels. Use standard errors clustered by state, which allow the errors of a state to be correlated across years. Compare the estimate with the synthetic control ATT, and explain which estimate is more credible for this single-state evaluation.

Solution Show the code and the numbers

twfe = smf.ols("cigsale ~ treated + C(state) + C(year)", data=panel).fit(
cov_type="cluster", cov_kwds={"groups": pd.factorize(panel["state"])[0]})
print(f"TWFE estimate: {twfe.params['treated']:.2f} "
f"(standard error clustered by state: {twfe.bse['treated']:.2f})")
print(f"Synthetic control ATT: {res.att:.2f}")

TWFE estimate: -27.35 (standard error clustered by state: 2.85)
Synthetic control ATT: -18.98

The TWFE estimate is −27.35 packs, with a clustered standard error of 2.85, against −18.98 for the synthetic control. The TWFE estimate equals the difference in the two changes of Section 4. The two coincide because the panel is balanced and California is the only treated state, with one start date. Like that comparison, the TWFE estimate weights all 38 donors equally, and its causal reading rests on parallel trends. The data for 1970–1988 contradict that assumption, and a standard error that rests on a single treated state is unreliable. The synthetic control estimate is therefore the more credible one here.

Exercise 6: Feed the Stata predictor weights into the inner problem. The proof card of Section 5.2 claims that the donor weights follow from V through the inner problem alone. Build the seven predictors with predictor_means() and divide each by its standard deviation across the 39 states. Then solve the inner problem for the Stata V with scipy.optimize.minimize and the SLSQP method, a scipy routine for minimization under bounds and constraints. Compare the recovered weights with the Stata weights.

Solution Show the code and the numbers

x_sd = X.std(ddof=1) # SD of each predictor across the 39 states
x1 = (X.loc[TREATED] / x_sd).to_numpy() # California, scaled
x0 = (X.loc[DONORS] / x_sd).to_numpy() # 38 donors x 7 predictors, scaled
v_stata = np.array(STATA_V)
def inner_loss(w):
"""V-weighted distance between California and the synthetic predictors."""
diff = x1 - x0.T @ w
return float(diff @ (v_stata * diff))
sol = minimize(inner_loss, np.full(len(DONORS), 1 / len(DONORS)), method="SLSQP",
bounds=[(0, 1)] * len(DONORS),
constraints=[{"type": "eq", "fun": lambda w: w.sum() - 1}],
options={"ftol": 1e-14, "maxiter": 1000})
w_ex6 = {s: w for s, w in zip(DONORS, sol.x) if w > 1e-4}
print(sol.message)
for s, w in sorted(w_ex6.items(), key=lambda kv: -kv[1]):
print(f"{s:<12} recovered {w:.3f} Stata {STATA_W.get(s, 0.0):.3f}")

Optimization terminated successfully
Utah recovered 0.334 Stata 0.334
Nevada recovered 0.235 Stata 0.235
Montana recovered 0.202 Stata 0.202
Colorado recovered 0.161 Stata 0.161
Connecticut recovered 0.068 Stata 0.068

The solver recovers 0.334 for Utah, 0.235 for Nevada, 0.202 for Montana, 0.161 for Colorado, and 0.068 for Connecticut, the Stata weights to three decimals. The inner problem therefore turns the Stata V into the Stata donor weights, while the mlsynth V, which looks completely different, yields almost the same donor weights. This is the non-identification of V in action: the predictor weights are a means to the donor weights, not a finding in themselves.

References

Introduction to Panel Data Methods in Python

Fri, 02 Oct 2026 00:00:00 +0000

Abstract

Estimating the wage return to union membership is complicated by selection. Workers who join unions differ systematically from workers who do not. A naive cross-sectional regression therefore confounds the union effect with unobserved traits such as ability and motivation. Panel data, which observe the same workers repeatedly, offer several ways to remove such time-invariant confounders. However, beginners often struggle to see how the standard estimators relate to one another. This tutorial clarifies these relationships by applying seven panel-data methods to a single dataset: pooled OLS, between, first differences, the within (fixed effects) estimator, two-way fixed effects, random effects, and the correlated random effects (CRE) model of Mundlak. The data are an NLSY-style wage panel restricted to 2010 and 2012. The balanced sample contains 2,199 prime-age US workers and 4,398 worker-year observations. Only 16.3% of the observations are unionized, and only 73 workers (3.3%) change union status. Consequently, just 6.1% of the variance in union status occurs within workers. The methods are implemented in Python with pyfixest and linearmodels, and the Hausman test and the Mundlak term serve as specification checks. The cross-sectional estimators report a union premium of 7 to 11 log points (POLS 0.0750, Between 0.0662, RE 0.1092). The within estimators report about 21 log points (FDFE 0.2113, FE 0.2103, TWFE 0.2113, CRE 0.2103), which is nearly three times as large. With two periods, first differences with an intercept reproduce two-way fixed effects exactly. The textbook Hausman test rejects random effects at the 5% level (H = 5.62, p = 0.018), but it assumes classical errors. The robust Mundlak tests are only borderline (p = 0.072 with random effects and p = 0.106 with clustered pooled OLS). Together, the gap between the two groups of estimates and the negative Mundlak term (−0.1441) are consistent with negative selection into unions. The within estimate is also fragile, because it rests on 73 switchers and falls to 0.040 when all five survey waves are used. We conclude that the CRE (Mundlak) specification is a sensible default. It reproduces the fixed-effects coefficient, retains time-invariant covariates, and provides a built-in specification test.

1. Overview

1.1 Does joining a union raise wages?

Suppose we observe the same workers in 2010 and in 2012, and we want to know whether union membership raises wages. A simple regression on the pooled data suggests that it does, by about 7.5 log points. A log point is a difference of 0.01 in log wages, which is roughly a 1% difference in wages. However, this headline number hides a problem that has occupied econometricians for decades. Workers who join unions are not a random subset of the workforce. They may have different schooling, work in different industries, or differ in unobserved ability and motivation. If any of these unobserved differences also affect wages, the 7.5 estimate mixes the union effect with everything else that accompanies union status.

This problem is known as omitted-variable bias, and panel data offer several ways to address it. Panel data contain repeated observations on the same units over time. By comparing each worker with the same worker in another year, we can remove every trait that is constant within a person, such as innate ability, gender, schooling, or family background. The cost is a much smaller effective sample, because only the workers who change union status between 2010 and 2012 contribute to the estimate. In this dataset, only 73 of 2,199 workers do so. The benefit is a coefficient that is much harder to dismiss as confounded by fixed worker traits.

This tutorial applies the seven standard panel estimators to a real two-period wage panel. The estimators are pooled OLS, between, first differences, the within (fixed effects) estimator, two-way fixed effects, random effects, and the correlated random effects model of Mundlak. Along the way, we run the Hausman test, prove two short identities, and visualize what the within transformation does to the data. The central result may surprise some readers. Once we account for unobserved worker traits, the estimated union premium nearly triples, from about 7.5 to about 21 log points.

1.2 Learning objectives

After completing this tutorial, readers will be able to:

Decompose the variance of a panel variable into between and within parts, and explain why a small within share limits the precision of fixed effects.
Implement seven panel-data estimators in Python with pyfixest and linearmodels, using one short code block per method.
Prove that first differences and fixed effects coincide when T = 2, and identify the role of the intercept in that identity.
Visualize the within transformation and identify which workers identify the fixed-effects coefficient.
Compare fixed and random effects with the Hausman test and the Mundlak term, and judge when each test is valid and how much power it has when within variation is thin.
Interpret the gap between cross-sectional and within estimates as evidence of selection on unobservables.
Experiment with selection strength and the share of switchers in an interactive lab, and predict how each estimator responds.

1.3 The road ahead

The tutorial proceeds in seven stages. The same 2,199 workers carry the entire analysis. Each stage either produces an estimate, explains it, or tests it.

graph TD
A("<b>Describe the panel</b><br/>2,199 workers, T = 2<br/>only 6.1% of union variance is within") --> B("<b>Cross-sectional estimators</b><br/>POLS 0.075, Between 0.066")
B --> C("<b>Within estimators</b><br/>FD, FE, TWFE about 0.21<br/>identified by 73 switchers")
C --> D("<b>Random effects and tests</b><br/>RE 0.109, Hausman p = 0.018<br/>CRE recovers FE exactly")
D --> E("<b>Controls</b><br/>schooling and gender absorbed<br/>the age coefficient flips sign")
E --> F("<b>Interactive lab</b><br/>vary selection and switchers")
F --> G("<b>Consolidate</b><br/>misconceptions, discussion,<br/>graded exercises")
classDef data fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef within fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef tests fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef practice fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class A,B data
class C within
class D,E tests
class F,G practice

The border colors of the boxes mark the logic of the argument. The boxes with blue borders describe the data and the cross-sectional benchmark. The box with an orange border introduces the within estimators, which remove fixed worker traits. The boxes with teal borders test the choice between fixed and random effects and then add controls. Finally, the boxes with gray borders turn the analysis over to the reader through a lab, a set of misconceptions, and graded exercises.

1.4 How the estimators relate

The diagram below summarizes the estimator family. It classifies each estimator by the variation that it uses. It also shows how the two specification tests, the Hausman test and the Mundlak term, guide the choice between fixed and random effects. Readers can return to this map whenever the relationship between two estimators becomes unclear.

flowchart TD
A("<b>Panel data</b><br/>outcome y and regressor x<br/>for workers i in periods t") --> Q("<b>Which variation does<br/>the estimator use?</b>")
Q -->|"all variation,<br/>panel ignored"| POLS("Pooled OLS")
Q -->|"between<br/>workers only"| BETW("Between")
Q -->|"within<br/>workers only"| WITHIN("FE, FDFE,<br/>DVFE, TWFE")
Q -->|"weighted between<br/>and within"| RE("Random effects")
WITHIN --> TEST("<b>Specification test</b><br/>Hausman test or<br/>Mundlak term")
RE --> TEST
WITHIN --> CRE("<b>CRE (Mundlak)</b><br/>FE coefficient inside<br/>an RE model")
RE --> CRE
TEST -->|"reject H0:<br/>RE inconsistent"| USE_FE("Use FE<br/>(consistent)")
TEST -->|"fail to reject:<br/>RE plausible"| USE_RE("Use RE<br/>(efficient)")
classDef data fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef question fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef within fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef tests fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef bridge fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
class A,POLS,BETW data
class Q,TEST question
class WITHIN,USE_FE within
class RE,USE_RE tests
class CRE bridge

The diagram makes the central trade-off of panel methods visible. Pooled OLS and the between estimator rely on cross-sectional variation, so they ask how union and non-union workers compare. Random effects also leans heavily on this comparison, because it combines between and within variation. By contrast, the within estimators (FE, FDFE, DVFE, and TWFE) rely only on changes within workers, so they ask what happens when the same worker changes union status. The CRE (Mundlak) model connects the two groups, because it recovers the within coefficient inside a random-effects framework. The Hausman test and the Mundlak term are formal tools for choosing between fixed and random effects, and we run both in sections 13 and 14.

2. Key concepts at a glance

This tutorial relies repeatedly on a small vocabulary. The later sections assume that readers can move between these terms quickly. Each concept below has three parts. The definition is always visible, while the example and the analogy sit behind clickable cards. Readers can open the cards when needed or leave them collapsed for a quick scan. This section is the natural place to return to whenever a later term, such as “within transformation” or “Hausman test,” feels unclear.

1. Pooled OLS (POLS). Run ordinary OLS on the entire panel as if the rows were independent observations. Ignores that some rows come from the same worker. Naive baseline. Useful as the worst-case benchmark that every panel estimator should beat.

Example

POLS on this dataset returns a union coefficient of 0.0750 (SE 0.0231). Statistically significant, but well below the within-worker estimate. The gap is consistent with selection on time-invariant worker traits that also affect wages.

Analogy

Lump every observation together and ignore which rows belong to whom. Like averaging the grades of a class without noticing that some students took the exam twice. The repeats inflate the sample and obscure the right comparison.

2. Between vs within variation. The variance of any panel variable splits into a between part (across units) and a within part (over time, inside one unit). The decomposition shows what kind of variation each estimator can use.

Example

For union in this dataset, 93.9% of the variance is between workers (some are unionized, others are not), and only 6.1% is within workers (73 workers change union status across the two periods). FE relies on the 6.1%.

Analogy

Comparing different people versus comparing the same person over time. The 93.9% is “Alice versus Bob”; the 6.1% is “Alice in 2010 versus Alice in 2012.” Different questions; different answers.

3. Within transformation $\tilde{y}_{it} = y_{it} - \bar{y}_i$. Subtract the time-series mean of each unit from each of its observations. The unit-specific intercept $\alpha_i$ vanishes by construction. What remains is within-unit variation: the part of $y$ that moves over time inside one worker.

Example

After demeaning, union for Alice (always in a union, mean 1) becomes 0 in both periods, so she contributes nothing to FE. Bob (who changed status) keeps his signal. FE is identified entirely by workers like Bob.

Analogy

Subtracting the watermark from every page. The text underneath is what we came for. Before demeaning, every page is dominated by the watermark.

4. First differences (FD/FDFE) $\Delta y_{it}$. Subtract last period from this period. Removes $\alpha_i$ by differencing rather than demeaning. With $T = 2$, FD without an intercept and the within estimator give identical slopes, and FD with an intercept equals two-way FE. With $T > 2$, FD and FE generally differ.

Example

FDFE (with an intercept) returns 0.2113 (SE 0.0792), and FE returns 0.2103. Dropping the intercept from the FD regression gives exactly 0.2103. The 0.001 gap arises because the intercept absorbs the common 2010 to 2012 wage growth of 0.0727.

Analogy

Subtracting yesterday from today versus subtracting the average day from today. With only two days, the two operations carry the same information. With ten days, they weight the days differently, but both measure change within a unit.

5. Fixed effects (FE). The cleanest within comparison. Estimate each $\alpha_i$ explicitly (or absorb them) and let only the within-unit variation in $x$ identify $\beta$. Equivalent to running OLS on the demeaned data.

Example

FE returns a union coefficient of 0.2103, almost three times POLS. The within-worker union premium is about 21 log points (about 23% in levels). Selection hidden in POLS pulls the cross-sectional estimate toward zero.

Analogy

Comparing Bob in 2010 with Bob in 2012. Same person, same schooling, same gender. The only recorded change is the union status of Bob. That comparison is what FE buys.

6. Two-way FE (TWFE). FE that absorbs both unit effects $\alpha_i$ and time effects $\delta_t$. Removes calendar-year shocks (a recession, a policy change) along with unit-specific shifts. Standard for short panels with common macro shocks.

Example

TWFE returns 0.2113 (SE 0.0792), identical to FDFE, because with $T = 2$ the year effect plays the role of the FD intercept. One-way FE (0.2103) differs only by that common trend.

Analogy

Cleaning the negative twice. The first pass removes the watermark printed on every page (unit FE). The second pass removes the smudge that a particular printing run left on the entire stack (time FE).

7. Random effects (RE). Treats $\alpha_i$ as a random draw uncorrelated with the regressors. Uses GLS to combine within and between variation efficiently. More efficient than FE if the no-correlation assumption holds; inconsistent if it does not.

Example

RE returns 0.1092 (SE 0.0299), between POLS (0.0750) and FE (0.2103). The Hausman test below asks whether the no-correlation assumption of RE is tenable here.

Analogy

Trusting that worker-specific traits are random noise. If that is true, both kinds of variation can be used efficiently. If it is false (motivation correlates with union choice), the estimator treats a systematic difference as noise and becomes biased.

8. Mundlak / CRE. Add the unit mean of every time-varying regressor as an extra control. The coefficient on the original variable is then identified exactly as FE identifies it. The coefficient on the unit mean tests for correlation between $\alpha_i$ and $x$, the same question that the Hausman test asks.

Example

The CRE specification returns a within coefficient of 0.2103 (matching FE exactly). The Mundlak term union_bar has a coefficient of −0.1441 with p = 0.0717, which is exactly the between estimate minus the within estimate (0.0662 − 0.2103). The textbook Hausman test, by contrast, rejects RE (H = 5.6209, p = 0.0177). Sections 13 and 14 explain why the two tests disagree.

Analogy

A peace treaty between FE and RE. CRE delivers the robustness of FE and the framework of RE in one regression. The Mundlak term is the diplomatic clause: it absorbs whatever correlation between unit traits and treatment status would otherwise push the two estimators apart.

3. Setup and imports

We use pyfixest for OLS and absorbed fixed effects. We use linearmodels for the random-effects GLS estimator and scipy.stats.chi2 for the p-value of the Hausman test. The standard pandas, numpy, and matplotlib stack handles data and figures.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pyfixest as pf
import statsmodels.api as sm
from linearmodels.panel import RandomEffects
from scipy.stats import chi2
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
rng = np.random.default_rng(RANDOM_SEED)

The figures in this post use the dark-navy palette of the site. The corresponding plt.rcParams block appears in script.py. We omit it here because it does not affect any estimate.

4. Data loading

We load a wage panel from a Stata .dta file. The file contains data on US workers modeled on the National Longitudinal Survey of Youth (NLSY), observed in 2010, 2012, 2014, 2016, and 2018. For pedagogical clarity, we restrict the analysis to 2010 and 2012 only, so that T = 2. This choice gives the cleanest illustration of the textbook result that first differences and the within estimator are closely linked. With T = 2, every worker contributes exactly two observations, so the panel is automatically balanced.

DATA_URL = "https://github.com/quarcs-lab/data-open/raw/master/isds/wage_panel_bob4.dta"
df_full = pd.read_stata(DATA_URL)
# Keep two periods so the FD = Within identity is visible.
df = df_full[df_full["year"].isin([2010, 2012])].copy()
df = df.sort_values(["ID", "year"]).reset_index(drop=True)
# Convert union "Yes/No" to 1/0; build a female dummy.
df["union"] = df["union"].astype(str).map({"Yes": 1, "No": 0}).astype(float)
df["female"] = (df["gender"].astype(str).str.strip().str.lower() == "female").astype(float)
# Drop rows with missing values in the variables we use.
df = df.dropna(subset=["lwage", "union", "age", "schooling"]).reset_index(drop=True)

The next block prints the panel structure and descriptive statistics. The balance check confirms that every worker has exactly two observations. The descriptive table then shows how dispersed the key variables are.

print(f"Individuals (N): {df['ID'].nunique()}")
print(f"Time periods (T): {df['year'].nunique()}")
print(f"Observations (N×T): {len(df)}")
print(f"Balanced: {(df.groupby('ID')['year'].count() == df['year'].nunique()).all()}")
print(df[["lwage", "union", "age", "schooling"]].describe().round(4))

Individuals (N): 2199
Time periods (T): 2
Observations (N×T): 4398
Balanced: True
lwage union age schooling
count 4398.0000 4398.0000 4398.0000 4398.0000
mean 3.1061 0.1626 35.6794 14.5020
std 0.5982 0.3690 6.2576 2.1825
min -1.7325 0.0000 25.0000 3.0000
25% 2.7434 0.0000 30.0000 12.0000
50% 3.0958 0.0000 35.0000 15.0000
75% 3.4671 0.0000 41.0000 16.0000
max 6.0635 1.0000 49.0000 17.0000

Interpretation. The analysis sample is a perfectly balanced panel of 2,199 prime-age workers observed in 2010 and 2012. It contains 4,398 worker-year observations, and the workers are 35.7 years old on average (range 25 to 49). Only 16.3% of the observations are unionized (mean union = 0.1626), so the sample is dominated by non-union workers. Mean log wage is 3.11 with a standard deviation of 0.60, and average schooling is 14.5 years. Because the panel is balanced with T = 2, each worker contributes exactly one change per variable, which makes the within and first-difference transformations especially transparent.

5. Between vs within variance: how much do panel methods have to work with?

Before estimating anything, we ask a diagnostic question about each variable. How much of its variation comes from differences between workers, and how much from changes within workers over time? Fixed-effects estimators use only the within part. If that part is tiny, FE will be imprecise unless the sample is very large.

The decomposition splits the variance of each variable into two pieces. The between part is the variance of the two-year mean of each worker, $\mathrm{Var}(\bar{x}_i)$. The within part is the variance of each observation around the mean of its own worker, $\mathrm{Var}(x_{it} - \bar{x}_i)$. Their sum approximately equals the total variance, so each part can be expressed as a share of that sum.

Predict first

Union status has an overall standard deviation of 0.369, and the mean union rate is 16.3%. Roughly what share of the variance of union comes from workers who change status between 2010 and 2012: about 50%, about 25%, or less than 10%? Commit to an answer before scrolling.

Reveal the answer

Answer. Less than 10%: the within share is only 6.1%. Just 73 of 2,199 workers switch status, so almost all the variation in union membership comes from comparing different workers. Note that the within standard deviation (0.0911) is not a share. The share is its square divided by the sum of the squared between and within standard deviations.

for var in ["lwage", "union", "age", "schooling"]:
overall_sd = df[var].std()
between_sd = df.groupby("ID")[var].mean().std()
within_sd = (df[var] - df.groupby("ID")[var].transform("mean")).std()
between_pct = between_sd**2 / (between_sd**2 + within_sd**2) * 100
print(f"{var:<10} overall {overall_sd:.4f} between {between_sd:.4f}"
f" within {within_sd:.4f} between% {between_pct:.1f}"
f" within% {100 - between_pct:.1f}")

lwage overall 0.5982 between 0.5570 within 0.2184 between% 86.7 within% 13.3
union overall 0.3690 between 0.3576 within 0.0911 between% 93.9 within% 6.1
age overall 6.2576 between 6.1755 within 1.0147 between% 97.4 within% 2.6
schooling overall 2.1825 between 2.1827 within 0.0000 between% 100.0 within% 0.0

Interpretation. Almost all the variation in these variables is between workers rather than over time within a worker. Union status is 93.9% between and only 6.1% within, so fixed-effects estimators work with a thin slice of the total union variance. Schooling has no within variation at all, because no worker in the sample reports a change in education between 2010 and 2012. FE will therefore drop schooling mechanically. The main methodological consequence is that FE standard errors will be much larger than POLS standard errors. Consequently, the choice between FE and RE involves precision as well as bias.

6. Visualizing the panel: who actually changes union status?

The variance decomposition shows that the within share is small. A spaghetti plot of individual log-wage trajectories makes the same point visually. We sample 30 random workers and color each line by the union pattern of the worker. Orange marks workers who are always in a union, blue marks workers who are never in a union, and teal marks workers whose union status changed between 2010 and 2012.

sample_ids = rng.choice(df["ID"].unique(), size=30, replace=False)
fig, ax = plt.subplots(figsize=(10, 6))
for pid in sample_ids:
person = df[df["ID"] == pid].sort_values("year")
if person["union"].nunique() > 1:
ax.plot(person["year"], person["lwage"], "o-", color="#00d4c8", lw=2) # changer
else:
c = "#d97757" if person["union"].iloc[0] == 1 else "#6a9bcc"
ax.plot(person["year"], person["lwage"], "o-", color=c, alpha=0.35)
plt.savefig("panel_intro_trajectories.png", dpi=300, bbox_inches="tight")

Interpretation. Most of the 30 lines are blue or orange, because they belong to workers who are never or always in a union over the two-year window. Only 2 of the 30 lines are teal, and only these switchers provide identifying information for fixed effects, first differences, and the CRE model. The blue and orange lines can only be compared with one another, which is the comparison that a cross-sectional estimator makes. Conversely, the teal lines are the only ones that one-way fixed effects reads, because a line whose union status never changes carries no within-worker contrast. The central tension of this tutorial between cross-sectional and within methods is therefore a question of which lines we choose to read.

7. Pooled OLS: the naive baseline

We start with the simplest possible estimator. We regress log wages on union membership and treat every worker-year as an independent observation. This estimator is pooled OLS (POLS), and it ignores the panel structure entirely.

A note on standard errors before we start. Pooled OLS, the between estimator, first differences, and one-way FE report heteroskedasticity-robust (HC1) standard errors. Two-way FE clusters by worker, which also allows the errors of the same worker to be correlated across years. Random effects and CRE report White standard errors computed on the quasi-demeaned data, which linearmodels calls "robust". These choices follow script.py, and they explain why, for example, FE (SE 0.0812) and TWFE (SE 0.0792) do not share a standard error.

# Stata: reg lwage union, robust
fit_pols = pf.feols("lwage ~ union", data=df, vcov="HC1")
pols_coef = fit_pols.coef()["union"]
pols_se = fit_pols.se()["union"]
print(f"Union coefficient: {pols_coef:.4f} (SE {pols_se:.4f})")

Union coefficient: 0.0750 (SE 0.0231)

Interpretation. Pooled OLS reports a union wage premium of 7.5 log points (SE 2.3 log points), which is highly significant by conventional standards (t ≈ 3.25). This is the textbook cross-sectional answer and the number that a naive analyst would report. However, it is likely biased. For example, if workers with higher unobserved earning ability are less likely to hold union jobs, POLS confounds the union effect with the effect of ability. The rest of the post evaluates this hypothesis by removing the fixed component of ability in several different ways.

8. Between estimator: the cross-sectional benchmark

The between estimator takes POLS to its logical extreme. It collapses each worker to the two-year mean of each variable and then runs OLS across workers. This estimator uses only between-worker variation, which makes it the mirror image of fixed effects. It therefore provides a clean reference point for a purely cross-sectional answer.

# Stata: collapse (mean) lwage union, by(ID); regress lwage union, vce(robust)
df_between = df.groupby("ID")[["lwage", "union"]].mean().reset_index()
fit_between = pf.feols("lwage ~ union", data=df_between, vcov="HC1")
between_coef = fit_between.coef()["union"]
between_se = fit_between.se()["union"]
print(f"Union coefficient: {between_coef:.4f} (SE {between_se:.4f})")

Union coefficient: 0.0662 (SE 0.0311)

Interpretation. Collapsing the panel to 2,199 worker averages gives a premium of 6.6 log points (SE 3.1). This is the cross-sectional union effect with all within-worker variation removed. The estimate is close to POLS (0.066 versus 0.075), as expected, because 93.9% of the union variance is between workers. POLS and the between estimator therefore view almost the same comparison from slightly different angles. Both share the same identification problem, and both serve as cross-sectional benchmarks for the within estimators that follow.

9. First differences: subtracting the past from the present

The first within estimator is first differences, which this post abbreviates as FDFE (first-difference fixed effects). The idea is to subtract the 2010 values of each worker from the 2012 values. Any time-invariant trait, such as ability, schooling, or family background, cancels out in the subtraction. We are left with a regression of $\Delta\mathrm{lwage}$ on $\Delta\mathrm{union}$, which is identified entirely by the workers who changed union status.

Formally, we write the panel model as

$$y_{it} = \alpha_i + \beta x_{it} + u_{it}$$

where $\alpha_i$ is the unobserved worker-specific effect. Differencing across the two periods gives

$$\Delta y_i = \beta \Delta x_i + \Delta u_i$$

where $\Delta y_i = y_{i,2012} - y_{i,2010}$, and $\Delta x_i$ and $\Delta u_i$ are defined in the same way.

In words, the change in wages between 2010 and 2012 equals $\beta$ times the change in union status plus a noise term. The worker-specific effect $\alpha_i$ has vanished. In the code, $y$ is the lwage column, $x$ is union, $\alpha_i$ captures whatever is unique to each ID, and $\beta$ is the parameter of interest. We also add an intercept, which captures the wage growth that all workers share between the two years.

Predict first

Pooled OLS gave 0.0750. Will the first-difference estimate be smaller, about the same, or larger? And will its standard error be smaller or larger than the 0.0231 of POLS? Commit to both answers before scrolling.

Reveal the answer

Answer. Larger on both counts. The FD estimate is 0.2113, almost three times POLS, and its standard error is 0.0792, about 3.4 times larger. Differencing removes the fixed traits that depress the cross-sectional comparison, but it also discards every worker who never changes status, which leaves only 73 informative workers.

# Stata: xtset ID year, delta(2); regress D.lwage D.union, vce(robust)
df_diff = (df.sort_values(["ID", "year"])
.groupby("ID")[["lwage", "union"]].diff().dropna())
df_diff.columns = ["d_lwage", "d_union"]
fit_fdfe = pf.feols("d_lwage ~ d_union", data=df_diff, vcov="HC1")
fdfe_coef = fit_fdfe.coef()["d_union"]
fdfe_se = fit_fdfe.se()["d_union"]
print(f"Union coefficient: {fdfe_coef:.4f} (SE {fdfe_se:.4f})")
print(f"Intercept (common wage growth): {fit_fdfe.coef()['Intercept']:.4f}")
print(f"Differenced sample: {len(df_diff)} rows (one per worker since T=2).")
print(f"Workers who changed union status: {(df_diff['d_union'] != 0).sum()}")

Union coefficient: 0.2113 (SE 0.0792)
Intercept (common wage growth): 0.0727
Differenced sample: 2199 rows (one per worker since T=2).
Workers who changed union status: 73

Interpretation. The first-difference estimator returns a premium of 21.1 log points (SE 7.9), with a 95% confidence interval of roughly [0.06, 0.37]. The point estimate is almost three times the POLS estimate (0.211 versus 0.075), and the standard error is about 3.4 times larger. This combination is the classic signature of moving from a cross-sectional design to a design that uses only switchers. Here, only 73 workers change union status, and 36 of them join while 37 leave. The interval is wide. It excludes zero, so the within effect is statistically different from zero, but it also contains the POLS estimate of 0.075. The upward revision is therefore large in size but not, on its own, statistically decisive. The within-worker effect is therefore a different, and arguably cleaner, parameter than the cross-sectional comparison.

10. Within / Fixed effects: the same idea, run differently

The within estimator (also called fixed effects, FE) pursues the same goal as first differences through a different transformation. It subtracts the mean of each worker from every observation of that worker, so every variable becomes $\tilde{x}_{it} = x_{it} - \bar{x}_i$. OLS on the demeaned data then delivers the FE coefficient. Modern software, such as pyfixest in Python and reghdfe in Stata, hides the demeaning step. In pyfixest, the formula lwage ~ union | ID absorbs the worker fixed effects.

Predict first

With only two periods, demeaning and differencing use the same 73 switchers. Will the FE coefficient be exactly equal to the FD estimate of 0.2113, or slightly different? If it differs, which feature of the FD regression could explain the gap? Commit to an answer before scrolling.

Reveal the answer

Answer. Slightly different: FE gives 0.2103, a gap of 0.001. The gap comes from the intercept in the FD regression, which absorbs the common wage growth of 0.0727. Dropping that intercept makes FD return exactly 0.2103, the FE value.

# Manual demeaning — pedagogical, makes the within transformation visible.
df["lwage_demean"] = df["lwage"] - df.groupby("ID")["lwage"].transform("mean")
df["union_demean"] = df["union"] - df.groupby("ID")["union"].transform("mean")
# Stata: areg lwage union, absorb(ID) vce(robust) (HC1, as here)
fit_fe = pf.feols("lwage ~ union | ID", data=df, vcov="HC1")
fe_coef = fit_fe.coef()["union"]
fe_se = fit_fe.se()["union"]
print(f"Union coefficient: {fe_coef:.4f} (SE {fe_se:.4f})")
# FD without an intercept reproduces FE exactly when T = 2.
fit_fd0 = pf.feols("d_lwage ~ d_union - 1", data=df_diff)
print(f"FD slope without intercept: {fit_fd0.coef()['d_union']:.4f}")

Union coefficient: 0.2103 (SE 0.0812)
FD slope without intercept: 0.2103

The figure below visualizes the effect of demeaning. The left panel shows the raw data, with union status (jittered for visibility) on the horizontal axis, log wage on the vertical axis, and the POLS regression line. The right panel shows the same observations after subtracting the mean of each worker from both variables. In that panel, the FE regression line passes through the demeaned cloud and through the origin.

Interpretation. The two panels look like different datasets, but they contain the same observations. In the raw data on the left, the POLS slope is 0.075, because the mean wages of union and non-union workers are close to each other. In the demeaned data on the right, the FE slope is 0.210, and it is identified only by the 73 workers who changed union status. These are the only points that move away from zero on the horizontal axis. The figure therefore shows geometrically what the variance decomposition showed numerically. The within slope is steeper because the comparison is no longer across workers, where ability and other fixed traits confound the picture, but within the same worker over time.

The FE coefficient (0.2103) is almost identical to the FD coefficient (0.2113). The gap of 0.001 arises because the FD regression includes an intercept, which absorbs the common wage trend, whereas one-way FE does not. Without that intercept, FD reproduces FE exactly, as the second line of the output confirms. The proof below shows why this identity holds whenever T = 2.

Proof Why FD and FE coincide when T = 2

Write $\Delta x_i = x_{i2} - x_{i1}$ and $\Delta y_i = y_{i2} - y_{i1}$. With two periods, the mean of each worker is $\bar x_i = (x_{i1} + x_{i2})/2$, so the demeaned values are exact halves of the difference:

$$\tilde x_{i1} = -\frac{\Delta x_i}{2}, \qquad \tilde x_{i2} = \frac{\Delta x_i}{2}$$

The same holds for $y$. Line 1. The FE slope is OLS on the demeaned data, summed over both periods:

$$\hat\beta_{FE} = \frac{\sum_i \sum_t \tilde x_{it} \tilde y_{it}}{\sum_i \sum_t \tilde x_{it}^2} = \frac{\sum_i 2 \cdot \frac{\Delta x_i}{2} \cdot \frac{\Delta y_i}{2}}{\sum_i 2 \cdot \left(\frac{\Delta x_i}{2}\right)^2}$$

and the factors of 2 cancel:

$$\hat\beta_{FE} = \frac{\sum_i \Delta x_i \Delta y_i}{\sum_i \Delta x_i^2}$$

Line 2. The right-hand side is the OLS slope of $\Delta y_i$ on $\Delta x_i$ without an intercept. So one-way FE equals FD without an intercept: 0.2103 in both cases.

Line 3. Now add year effects. In a balanced panel, two-way demeaning subtracts the worker mean and the year mean and adds back the grand mean. For $T = 2$ this gives $\pm (\Delta x_i - \overline{\Delta x})/2$, where $\overline{\Delta x}$ is the average change across workers. Repeating Line 1 with these values yields

$$\hat\beta_{TWFE} = \frac{\sum_i (\Delta x_i - \overline{\Delta x})(\Delta y_i - \overline{\Delta y})}{\sum_i (\Delta x_i - \overline{\Delta x})^2}$$

which is the OLS slope of $\Delta y_i$ on $\Delta x_i$ with an intercept. So TWFE equals FD with an intercept: 0.2113 in both cases. The intercept, 0.0727, is the common wage growth that one-way FE leaves in its error term. With $T > 2$, the two transformations weight the periods differently and the identity breaks (Exercise 6).

A dummy-variable version of FE (DVFE) gives the same answer. Instead of demeaning, it adds one indicator for each worker to the regression. This version makes the worker effects explicit, because each dummy estimates one $\alpha_i$.

df["ID_str"] = df["ID"].astype(str)
fit_dvfe = pf.feols("lwage ~ union + C(ID_str)", data=df, vcov="HC1")
print(f"DVFE coefficient: {fit_dvfe.coef()['union']:.4f}")

DVFE coefficient: 0.2103

Interpretation. Including a dummy for every worker (N − 1 = 2,198 dummies in this sample) recovers the FE coefficient exactly: 0.2103. The within transformation, first differences without an intercept, and dummy-variable FE are therefore three recipes for the same estimate. Modern software prefers absorption (| ID) over explicit dummies for purely computational reasons. With about 2,200 dummies the regression still runs quickly, but with 100,000 workers the dummy specification becomes prohibitive, whereas absorbed fixed effects remain inexpensive.

11. Two-way fixed effects: closing the FD–FE gap

Two-way fixed effects (TWFE) absorb both worker effects and year effects. We let pyfixest handle both with the formula | ID + year. This specification is the workhorse of applied microeconomics, and it is the starting point of most difference-in-differences research.

Predict first

FE gave 0.2103 and FD gave 0.2113. Adding year effects to FE allows a common wage trend. Will TWFE equal FE, equal FD exactly, or land somewhere else? Commit to an answer before scrolling.

Reveal the answer

Answer. TWFE equals FD exactly: 0.2113, with a difference of zero to six decimals. With $T = 2$, the year effect plays exactly the role of the intercept in the FD regression, as the proof in section 10 shows.

# Stata: reghdfe lwage union, absorb(ID year) vce(cluster ID)
fit_twfe = pf.feols("lwage ~ union | ID + year", data=df, vcov={"CRV1": "ID"})
twfe_coef = fit_twfe.coef()["union"]
twfe_se = fit_twfe.se()["union"]
print(f"Union coefficient: {twfe_coef:.4f} (SE {twfe_se:.4f})")
print(f"TWFE minus FD: {twfe_coef - fdfe_coef:+.6f}")

Union coefficient: 0.2113 (SE 0.0792)
TWFE minus FD: +0.000000

Interpretation. TWFE returns a premium of 21.1 log points (SE 7.9), identical to the first-difference estimate. The year effect absorbs the common wage growth that the FD intercept captured, so the small gap between FD and one-way FE closes exactly. Schooling, gender, and any other time-invariant regressor would be absorbed by the worker fixed effects, because a variable that never changes within a worker cannot identify a within effect. This limitation is a structural feature of within methods rather than a coding error. It is also one of the main reasons that applied researchers turn to the CRE (Mundlak) model when they want within identification and coefficients on time-invariant variables.

12. Random effects: betting on the no-correlation assumption

The random-effects (RE) estimator takes a different stance. It treats the worker effect $\alpha_i$ as a random draw from a population, uncorrelated with the regressors. If that assumption holds, RE is more efficient than FE, because it uses both within and between variation. If the assumption fails, RE is inconsistent.

Two technical terms are central to this section. First, RE is fitted by generalized least squares (GLS), a weighted regression that combines between and within variation according to their relative noise. Second, an estimator is consistent if it converges to the true parameter as the number of workers grows, while an inconsistent estimator stays away from the true value however much data we collect. RE is consistent only under the no-correlation assumption, whereas FE is consistent under weaker assumptions. Therefore, FE is the safer default whenever the no-correlation assumption is in doubt.

# Stata: xtreg lwage union, re (same coefficient; its vce(robust) clusters by ID)
df_re = df.set_index(["ID", "year"])
exog = sm.add_constant(df_re[["union"]])
fit_re = RandomEffects(df_re["lwage"], exog).fit(cov_type="robust")
re_coef = fit_re.params["union"]
re_se = fit_re.std_errors["union"]
print(f"Union coefficient: {re_coef:.4f} (SE {re_se:.4f})")

Union coefficient: 0.1092 (SE 0.0299)

Interpretation. RE returns a premium of 10.9 log points (SE 3.0), which lies between POLS (0.075) and FE (0.210). Conceptually, RE is a weighted average of the between and within estimators, with weights determined by their relative precision. Because only 6.1% of the union variance is within workers, RE leans heavily toward the between comparison and lands much closer to POLS than to FE. The RE standard error (0.030) is 2.7 times smaller than the FE standard error (0.081). However, this gain in precision is genuine only if the worker effects are uncorrelated with union membership. If selection into unions depends on unobserved ability, as the gap between FE and POLS suggests, the extra precision comes at the cost of bias.

13. The Hausman test: FE or RE?

The classic specification test for choosing between FE and RE is due to Hausman (1978). Its logic is simple. If the RE assumption holds, both estimators are consistent and should give similar answers. If they differ substantially, the RE assumption is suspect and FE is preferred. Formally,

$$H = \hat{d}^\top [V_{\mathrm{FE}} - V_{\mathrm{RE}}]^{-1} \hat{d} \sim \chi^2(k)$$

where $\hat{d} = \hat{\beta}_{\mathrm{FE}} - \hat{\beta}_{\mathrm{RE}}$ is the difference between the two coefficient vectors.

In words, the statistic takes the difference between the two coefficient vectors and weights it by the inverse of the difference between their variance matrices. The resulting quadratic form is compared with a chi-square distribution whose degrees of freedom equal the number of regressors. A large $H$, and hence a small p-value, rejects the null hypothesis that RE is consistent. In the code, $\hat{d}$ is b_diff, $\hat{\beta}_{\mathrm{FE}}$ is fe_coef, $\hat{\beta}_{\mathrm{RE}}$ is re_coef, and $V_{\mathrm{FE}}$ and $V_{\mathrm{RE}}$ are the squared standard errors, because the model contains a single regressor.

One detail matters a great deal. The difference $V_{\mathrm{FE}} - V_{\mathrm{RE}}$ equals the variance of $\hat{d}$ only when RE is fully efficient under the null, which requires classical errors: homoskedastic and uncorrelated within a worker. The textbook test, which Stata runs with hausman fe re, therefore uses the classical (non-robust) variances of both estimators. We refit both models with classical standard errors for the test and keep the robust fits for everything else.

Predict first

The classical standard errors are 0.0509 for FE and 0.0278 for RE. The robust ones that we reported earlier are larger, 0.0812 and 0.0299. If we plugged the robust standard errors into the same formula, would $H$ rise or fall, and could the verdict at the 5% level change? Commit to an answer before scrolling.

Reveal the answer

Answer. $H$ falls from 5.62 to 1.79, and the p-value rises from 0.018 to 0.180, so the verdict flips from rejecting RE to not rejecting it. The robust FE standard error grows much more than the robust RE one, which inflates the denominator. However, the plug-in number is not a valid test, because with robust errors $V_{\mathrm{FE}} - V_{\mathrm{RE}}$ is no longer the variance of the difference.

# Stata: xtreg lwage union, fe; estimates store fe
# xtreg lwage union, re; estimates store re; hausman fe re
fit_fe_iid = pf.feols("lwage ~ union | ID", data=df, vcov="iid")
fit_re_iid = RandomEffects(df_re["lwage"], exog).fit() # classical variance
b_diff = np.array([fe_coef - re_coef])
v_diff = np.array([[fit_fe_iid.se()["union"] ** 2 - fit_re_iid.std_errors["union"] ** 2]])
H = float(b_diff @ np.linalg.pinv(v_diff) @ b_diff)
p_h = chi2.sf(H, df=1)
print(f"Classical SEs: FE {fit_fe_iid.se()['union']:.4f}, RE {fit_re_iid.std_errors['union']:.4f}")
print(f"H statistic: {H:.4f} p-value = {p_h:.4f}")
print(f"β_FE − β_RE = {b_diff[0]:+.4f}")
# For comparison only: robust SEs plugged into the same formula (not a valid test).
H_plugin = (fe_coef - re_coef) ** 2 / (fe_se ** 2 - re_se ** 2)
print(f"Robust SEs plugged in: H = {H_plugin:.4f} p = {chi2.sf(H_plugin, df=1):.4f}")

Classical SEs: FE 0.0509, RE 0.0278
H statistic: 5.6209 p-value = 0.0177
β_FE − β_RE = +0.1011
Robust SEs plugged in: H = 1.7941 p = 0.1804

Interpretation. The two estimators differ by about 0.101. With classical variances, the test statistic is 5.62 with one degree of freedom, giving a p-value of 0.018. Because 0.018 is below 0.05, we reject the null hypothesis that RE is consistent, and the textbook conclusion is to prefer FE. This verdict comes with two caveats. First, it relies on classical errors, and the robust standard errors are noticeably larger than the classical ones, especially for FE (0.0812 versus 0.0509), which suggests that the assumption is doubtful here. Second, the test cannot be made robust simply by plugging in the robust standard errors. Doing so gives H = 1.79 and p = 0.180, which reverses the verdict, but that number has no valid chi-square distribution. When the standard errors are robust, the appropriate check is the Mundlak test, to which the next section turns.

14. Correlated random effects (CRE / Mundlak): the modern bridge

Mundlak (1978) proposed a specification that bridges FE and RE. The idea is to add the mean of every time-varying regressor for each worker as an additional control and then estimate RE. The resulting model is the following.

$$y_{it} = \alpha + \beta x_{it} + \gamma \bar{x}_i + a_i + u_{it}$$

where $a_i$ is the random worker effect that RE assumes to be uncorrelated with the regressors. In words, wages depend on current union status and on the average union exposure of the worker across the panel. The coefficient $\beta$ on the time-varying $x_{it}$ captures the within effect, and it is numerically identical to the FE coefficient in a balanced panel. The coefficient $\gamma$ on the worker mean $\bar{x}_i$ captures the difference between the between and within relationships, which reflects selection. In a balanced panel, $\gamma$ equals the between estimate minus the within estimate exactly. If $\gamma \neq 0$, the worker effects are correlated with union status, and FE is preferred over RE. In the code, $\beta$ is cre_coef, $\gamma$ is mundlak_coef, and $\bar{x}_i$ is the union_bar column built with df.groupby("ID")["union"].transform("mean").

Proof Why the Mundlak coefficient on x equals the FE coefficient

The argument uses the Frisch-Waugh-Lovell (FWL) theorem from the FWL tutorial. Consider first the pooled regression of $y_{it}$ on $[1, x_{it}, \bar x_i]$.

Line 1. By FWL, $\hat\beta$ equals the slope from regressing $y_{it}$ on the residual of $x_{it}$ after regressing it on the other columns, $[1, \bar x_i]$.

Line 2. In a balanced panel, $\sum_t (x_{it} - \bar x_i) = 0$ for every worker. Hence $x_{it} - \bar x_i$ is orthogonal to any variable that is constant within a worker, including $1$ and $\bar x_i$. The decomposition

$$x_{it} = 0 + 1 \cdot \bar x_i + (x_{it} - \bar x_i)$$

is therefore exactly the OLS fit: the fitted value is $\bar x_i$, and the residual is $\tilde x_{it} = x_{it} - \bar x_i$.

Line 3. Regressing $y_{it}$ on $\tilde x_{it}$ gives $\sum \tilde x_{it} y_{it} / \sum \tilde x_{it}^2$. Because $\sum_t \tilde x_{it} \bar y_i = 0$, we can replace $y_{it}$ by $\tilde y_{it}$, and the ratio becomes the within estimator:

$$\hat\beta = \frac{\sum_i \sum_t \tilde x_{it} \tilde y_{it}}{\sum_i \sum_t \tilde x_{it}^2} = \hat\beta_{FE}$$

Line 4. The RE version quasi-demeans every column by the RE weight $\theta$, which lies between 0 (pooled OLS) and 1 (FE) and equals 0.6091 here: each variable $z_{it}$ becomes $z_{it} - \theta \bar z_i$. The transformed $x$ equals $\tilde x_{it} + (1 - \theta) \bar x_i$, and the transformed constant and $\bar x_i$ remain constant within workers. Lines 2 and 3 therefore apply unchanged, and the CRE coefficient equals $\hat\beta_{FE}$ for any $\theta$. Here both are 0.2103.

Predict first

The CRE model is estimated by random-effects GLS, which gave 0.1092 in section 12. After adding union_bar, will the coefficient on union stay near 0.11, move to somewhere between 0.11 and 0.21, or equal the FE value of 0.2103? Commit to an answer before scrolling.

Reveal the answer

Answer. It equals the FE value exactly: 0.2103. Once the worker mean of union status is controlled for, the only variation left in union is the within variation, so RE has nothing else to use. The proof above shows why this holds for any RE weight.

# Stata: bysort ID: egen union_bar = mean(union); xtreg lwage union union_bar, re
df["union_bar"] = df.groupby("ID")["union"].transform("mean")
df_cre = df.set_index(["ID", "year"])
exog_cre = sm.add_constant(df_cre[["union", "union_bar"]])
fit_cre = RandomEffects(df_cre["lwage"], exog_cre).fit(cov_type="robust")
cre_coef = fit_cre.params["union"]
cre_se = fit_cre.std_errors["union"]
mundlak_coef = fit_cre.params["union_bar"]
mundlak_p = fit_cre.pvalues["union_bar"]
print(f"Union (within) coefficient: {cre_coef:.4f} (SE {cre_se:.4f})")
print(f"Mundlak term (union_bar): {mundlak_coef:+.4f} (p = {mundlak_p:.4f})")

Union (within) coefficient: 0.2103 (SE 0.0703)
Mundlak term (union_bar): -0.1441 (p = 0.0717)

Interpretation. The CRE union coefficient is 0.2103, which matches the FE estimate to four decimal places, as the result of Mundlak predicts. The Mundlak term is −0.1441, which is exactly the between estimate minus the within estimate (0.0662 − 0.2103). Its p-value of 0.072 is not significant at the 5% level, although it is suggestive. Workers with higher average union exposure tend to earn less than their within-worker union effect would imply. This pattern is consistent with negative selection into unions, in which workers with lower earning potential are more likely to hold union jobs. The Mundlak term therefore points in the same direction as the FE–RE gap, but on its own it provides only borderline evidence against RE. Unlike the plug-in Hausman statistic, this test remains valid with robust standard errors. The standard errors here are robust to heteroskedasticity but not to correlation within a worker, and Exercise 4 adds clustering (p = 0.106).

15. Putting it all together: the method comparison

The figure below displays six of the basic estimators on a single chart with 95% confidence intervals. The table adds the two-way FE estimate, which coincides with FDFE. Together, they summarize the main results of the tutorial in one place.

Method	Coef	SE	What variation does it use?
POLS	0.0750	0.0231	All; ignores the panel structure
Between	0.0662	0.0311	Cross-sectional means only
FDFE	0.2113	0.0792	Within-worker differences
FE	0.2103	0.0812	Within-worker deviations from the mean
TWFE	0.2113	0.0792	Within-worker deviations, net of year effects
RE	0.1092	0.0299	GLS-weighted between and within
CRE	0.2103	0.0703	RE with Mundlak terms (equals FE within)

Interpretation. The estimators fall into two clear groups. The cross-sectional methods (POLS 0.075, Between 0.066, RE 0.109) report a union premium of 7 to 11 log points, whereas the within methods (FDFE 0.211, FE 0.210, TWFE 0.211, CRE 0.210) report about 21 log points. This near-tripling is the central finding of the tutorial. It is consistent with negative selection, in which workers with higher unobserved earning ability are less likely to be union members, so that cross-sectional comparisons understate the within-worker union premium. The standard errors move in the opposite direction, because the standard errors of the cross-sectional methods are 2.6 to 3.5 times smaller. Thus, the cross-sectional methods are precise but likely biased, while the within methods are noisier but rely on weaker assumptions.

16. Adding controls: the extended models

Applied research usually includes control variables. We therefore re-estimate POLS, TWFE, RE, and CRE with age, schooling, a female indicator, and year effects on the right-hand side. The next code block builds the four specifications, and the table below reports the union, age, schooling, and female coefficients.

Predict first

In POLS, each additional year of age raises log wages by about 0.02. Under TWFE, which absorbs both worker and year effects, will the age coefficient stay near +0.02, shrink toward zero, or change sign? Commit to an answer before scrolling.

Reveal the answer

Answer. It changes sign: −0.0576 under TWFE. Between 2010 and 2012, age rises by exactly two years for 1,885 of the 2,199 workers, which the year effect absorbs completely. The age coefficient is therefore identified only by the 314 workers whose age rose by one or three years, mostly because of differences in interview timing.

# POLS with controls
fit_pols_x = pf.feols(
"lwage ~ union + age + schooling + female + C(year)",
data=df, vcov="HC1")
# TWFE: schooling and female are time-invariant → absorbed by ID FE
fit_twfe_x = pf.feols("lwage ~ union + age | ID + year",
data=df, vcov={"CRV1": "ID"})
# RE + controls, with a 2012 dummy for the year effect
df["y2012"] = (df["year"] == 2012).astype(float)
df_rx = df.set_index(["ID", "year"])
exog_rx = sm.add_constant(df_rx[["union", "age", "schooling", "female", "y2012"]])
fit_re_x = RandomEffects(df_rx["lwage"], exog_rx).fit(cov_type="robust")
# CRE + controls: adds the worker mean of every time-varying regressor.
# The mean of y2012 is 0.5 for every worker, so it needs no Mundlak term.
df["age_bar"] = df.groupby("ID")["age"].transform("mean")
df_rx = df.set_index(["ID", "year"])
exog_cx = sm.add_constant(df_rx[["union", "union_bar", "age", "age_bar",
"schooling", "female", "y2012"]])
fit_cre_x = RandomEffects(df_rx["lwage"], exog_cx).fit(cov_type="robust")
fits = {"POLS": (fit_pols_x.coef(), fit_pols_x.se()),
"TWFE": (fit_twfe_x.coef(), fit_twfe_x.se()),
"RE": (fit_re_x.params, fit_re_x.std_errors),
"CRE": (fit_cre_x.params, fit_cre_x.std_errors)}
print(f"{'Variable':<12}" + "".join(f"{m:<18}" for m in fits))
for var in ["union", "age", "schooling", "female"]:
cells = [f"{b[var]:7.4f} ({se[var]:.4f})" if var in b.index else "absorbed"
for b, se in fits.values()]
print(f"{var:<12}" + "".join(f"{c:<18}" for c in cells))

Variable POLS TWFE RE CRE
union 0.0571 (0.0204) 0.2129 (0.0793) 0.0875 (0.0258) 0.2129 (0.0681)
age 0.0209 (0.0013) -0.0576 (0.0238) 0.0205 (0.0017) -0.0576 (0.0240)
schooling 0.1108 (0.0037) absorbed 0.1106 (0.0048) 0.1108 (0.0047)
female -0.2731 (0.0160) absorbed -0.2731 (0.0206) -0.2731 (0.0206)

The age result in the TWFE column needs one more piece of evidence. The following block counts how much each worker aged between the two survey waves. This distribution determines how much within-worker variation in age survives once the year effect is removed.

age_change = df.groupby("ID")["age"].diff().dropna().astype(int)
print(age_change.value_counts().sort_index().to_string())

Interpretation. Adding controls lowers the POLS union coefficient to 0.057, because the controls absorb part of the cross-sectional confounding. Nevertheless, TWFE and CRE both report a within-worker premium of 0.213, so the gap relative to POLS (0.057) and RE (0.088) remains intact. CRE reproduces the TWFE coefficients on union and age exactly, because the Mundlak terms do for the time-varying regressors what the worker fixed effects do in TWFE. The schooling premium of 11.1 log points per year and the female penalty of 27.3 log points are stable across POLS, RE, and CRE, and both are absorbed by the worker fixed effects in TWFE. The age coefficient is the exception. It is positive in the cross-sectional models, POLS (+0.021) and RE (+0.021), but negative in the within models, TWFE and CRE (−0.058). Because 1,885 workers age by exactly two years, the year effect absorbs almost all within-worker variation in age. The within age coefficient therefore rests on only 314 workers with irregular interview spacing, and it should not be read as an age profile of wages. The year effect matters for CRE as well: without the 2012 dummy, the CRE age coefficient would be +0.033, because age would then absorb the common wage growth of about 3.6 log points per year.

17. An interactive panel lab

Every result so far comes from one fixed sample. The lab below has two tabs that let readers change the data and observe how the estimators respond. The Selection lab simulates 2,199 workers with a true union effect of 0.21. Its defaults are calibrated to this dataset: 73 workers switch status, and the simulated estimates are POLS 0.075, Between 0.066, RE 0.112, and two-way FE 0.209. The Demeaning lab uses eight workers, three of whom switch status. In the raw view, their log wages can be moved by dragging a point, or by selecting it and pressing the arrow keys. At the defaults, POLS is 0.078 and FE is 0.260.

Interactive panel-data lab

Simulated panel calibrated to the post (true effect 0.21)

A simulated two-period panel of 2,199 workers in which the true union effect is 0.21 log points. Change how union membership is selected on the worker effect α_i, how many workers switch status, and how noisy wages are, then compare four estimators.

Selection on the worker effect ρ−0.15

Correlation between α_i and the propensity to be a union member. Negative values mean that lower-wage workers are more likely to be union members, as in the post. Default −0.15.

Share of workers who switch status3.3%

Idiosyncratic noise σ_ε0.30

Standard deviation of the period-specific wage shock ε_it. More noise widens every interval, the FE interval most of all. Default 0.30.

Four estimates of the union effect, with 95% confidence intervals

Pooled OLSall 4,398 worker-years
Betweenmean of each worker, 2,199 rows
Random effectsSwamy–Arora, partial demeaning
FE (two-way)= first differences with an intercept
FE 95% confidence intervalfrom the first-difference standard error
Switchers

Switch selection off. In the Selection lab, set the selection slider to zero. POLS (0.216), Between (0.217), and RE (0.214) all move to the true effect of 0.21, while FE stays at 0.209. FE does not move at all, because selection operates through the worker effect, which demeaning removes.
Reverse the selection. Move the selection slider to +0.4, so that high-wage workers select into unions. POLS (0.565) and RE (0.469) now overstate the effect by a wide margin, while FE remains at 0.209.
Add switchers. Reset the lab and raise the share of switchers from 3.3% to 30%. The FE confidence interval narrows from 0.111 to 0.306 to 0.190 to 0.254, and RE (0.216) moves close to FE (now 0.222), because the within variation is no longer thin.
Move a stayer. In the Demeaning lab, move a worker who never changes union status. The POLS line moves, but the FE slope does not change at all.
Move a switcher. Now move one of the three switchers. The FE slope responds immediately, because switchers are the only workers that identify it.

The lab turns the main lessons of this tutorial into experiments. Bias in the cross-sectional estimators requires selection, that is, a correlation between the worker effect and union status. The precision of FE depends on the number of switchers rather than on the total sample size. Finally, demeaning makes stayers irrelevant for the within slope, which is exactly why FE is robust to their unobserved traits. The lab reports classical standard errors, so its FE interval is narrower than the robust interval in section 9. For a full-page companion with a Monte Carlo experiment and a Hausman explorer, see the web app.

18. Common misconceptions

Panel estimators are easy to run and easy to over-interpret. Each card below states a common belief and then checks it against the numbers in this post. Together, the cards summarize what the estimates can and cannot support.

Misconception "Plugging robust standard errors into the Hausman formula gives a robust Hausman test."

What is actually true. The Hausman formula requires RE to be fully efficient under the null, so it needs classical variances. With classical variances, H = 5.62 (p = 0.018) and the test rejects RE (section 13). With the robust standard errors plugged in, H falls to 1.79 (p = 0.180) and the verdict flips, but that number has no valid chi-square distribution. The robust alternative is the Mundlak test, which gives p = 0.072 in the RE version and p = 0.106 with clustering (Exercise 4). Neither rejects at 5%, and failing to reject would not prove the RE assumption anyway. The honest summary is that the evidence against RE is real but not decisive.

Misconception "The FE estimate is the union premium for all workers."

What is actually true. FE is identified only by the 73 workers who change union status, which is 3.3% of the sample. The 1,805 never-members and 321 always-members contribute nothing to the slope. Exercise 5 shows that joiners and leavers alone give very different estimates (0.345 and 0.081), so the 0.21 is an average over a small and possibly unusual group.

Misconception "First differences and fixed effects answer different questions."

What is actually true. Both remove the worker effect and both use only within-worker change. With T = 2 they are algebraically linked: FD without an intercept equals FE (0.2103), and FD with an intercept equals TWFE (0.2113), as the proof in section 10 shows. They diverge only when T > 2, and then only through how they weight the periods (Exercise 6).

Misconception "Fixed effects remove all confounding."

What is actually true. Fixed effects remove only confounders that are constant within a worker over the sample period. A shock that changes both union status and wages, such as a job change between 2010 and 2012, remains in the error term. The estimate also depends on the window: with all five waves, two-way FE falls from 0.211 to 0.040 (Exercise 6).

Misconception "The estimator with the smallest standard error is the most credible."

What is actually true. Precision and credibility are different properties. RE has a standard error of 0.0299, against 0.0812 for FE, but RE is consistent only if the worker effects are uncorrelated with union status. A precise estimate of the wrong quantity is not an improvement, and the gap between RE (0.109) and FE (0.210) suggests that the RE assumption is doubtful here.

19. Discussion: what does our case study tell us?

We started with a deceptively simple question: does union membership raise wages, and if so, by how much? Seven estimators applied to the same dataset produced answers ranging from 0.066 to 0.211. This spread of more than a factor of three is not noise. Instead, it reflects how each method identifies the parameter.

The cross-sectional group (POLS, Between, and RE) asks how union and non-union workers compare. Its answer of 7 to 11 log points is appropriate only if union members resemble non-members on every relevant unobservable. The within group (FDFE, FE, TWFE, and CRE) asks what happens when the same worker changes union status. Its answer of about 21 log points is appropriate only if nothing else that affects wages changes systematically for switchers between 2010 and 2012. Both questions are legitimate, and the gap between the answers is the empirical signature of selection on unobservables.

The textbook Hausman test rejects the random-effects assumption (p = 0.018), which, by the textbook rule, favors FE. However, that test assumes classical errors, and the robust standard errors suggest that this assumption is doubtful. The robust alternative, the Mundlak term, reached p = 0.072 in the RE version and p = 0.106 in a cluster-robust pooled version, so it does not reject at the 5% level. Part of the reason is low power, because the within share of union status is only 6.1%. The Mundlak coefficient of −0.144, which equals the between estimate minus the within estimate, nevertheless suggests that workers with more union exposure earn less on average than their within-worker premium implies. The most accurate summary is therefore that the evidence against RE is suggestive rather than conclusive, and that the two tests disagree mainly because they make different assumptions about the errors.

For a practitioner facing this kind of dataset, the practical implication is that the CRE (Mundlak) model is usually the right specification to lead with. It delivers the FE coefficient on the time-varying treatment and retains the RE structure, which keeps schooling and gender in the regression. It also provides a built-in specification test, the t-statistic on the Mundlak term, which can be made robust to heteroskedasticity and clustering. The cost is one extra regressor for each time-varying covariate, which is negligible in modern software.

In causal-inference terms, the within estimators target an average union effect among switchers, the 73 workers who changed union status between 2010 and 2012. This interpretation requires strict exogeneity conditional on the worker fixed effect, which means that the wage shocks of a worker in any year are unrelated to the union status of that worker in every year. Exercise 5 shows that joiners and leavers yield very different estimates, which signals that this average may hide heterogeneity. POLS and the between estimator target a population-wide association between union status and log wages, and they have no causal interpretation without an unconfoundedness assumption. Reporting both kinds of estimates side by side, as we do here, is more informative than reporting only one.

20. Summary and next steps

Takeaways.

Method insight. With T = 2, the within transformation, dummy-variable FE, and first differences without an intercept all produce the same union coefficient (0.2103). First differences with an intercept and two-way FE both give 0.2113. The gap of 0.001 arises from the common wage trend, which the intercept and the year effect absorb. Understanding why these identities hold is one of the most useful intuitions in panel econometrics.
Data insight. Almost all the variation in the data is between workers (union 93.9%, age 97.4%, schooling 100%). Only 6.1% of the union variance is within workers, and only 73 workers switch status. This thin slice explains why the FE standard error (0.081) is 2.7 times larger than the RE standard error (0.030).
Limitation. With T = 2 and 73 switchers, the FE estimate is imprecise and fragile. The textbook Hausman test rejects RE (p = 0.018), but it assumes classical errors, and the robust Mundlak tests are only borderline (p = 0.072 and p = 0.106). Moreover, when all five waves are used, two-way FE falls to 0.040 (Exercise 6), so the 0.21 estimate is specific to the 2010 to 2012 window.
Next step. A natural extension uses all five waves of the panel (2010 to 2018), which gives T = 5 and a within share of 16.1% for union status. With T > 2, the choice between FD and FE becomes a substantive decision, because FD is more efficient when the errors follow a random walk and FE is more efficient when they are serially uncorrelated. Event-study designs also become possible.

21. Exercises

The exercises below reuse the objects built in the post, mainly df and df_full, so they should be run after the main code. Each exercise comes with a collapsible solution that contains the code, its output, and a short interpretation. Readers should attempt each exercise first and then open the card to compare.

21.1 Warm-up

Exercise 1: Compute the within share by hand. Using variances rather than standard deviations, compute the between and within variance of union and the within share. Confirm that the within standard deviation, 0.0911, is not the within share. Then explain why the two numbers differ.

Solution Show the code and the numbers

worker_mean = df.groupby("ID")["union"].transform("mean")
between_var = df.groupby("ID")["union"].mean().var()
within_var = (df["union"] - worker_mean).var()
within_share = within_var / (between_var + within_var)
print(f"Between variance: {between_var:.4f}")
print(f"Within variance: {within_var:.4f}")
print(f"Within share: {100 * within_share:.1f}%")
print(f"Within SD: {within_var ** 0.5:.4f}")

Between variance: 0.1279
Within variance: 0.0083
Within share: 6.1%
Within SD: 0.0911

The within variance (0.0083) is only 6.1% of the sum. The within standard deviation (0.0911) is the square root of that variance, so reading it as a percentage overstates the within share by half.

Exercise 2: Who identifies the within estimators? Classify each worker as never a member, always a member, a joiner (0 in 2010, 1 in 2012), or a leaver (1 in 2010, 0 in 2012). How many workers switch? Compare the number of joiners with the number of leavers.

Solution Show the code and the numbers

wide = df.pivot(index="ID", columns="year", values="union")
pattern = np.select(
[(wide[2010] == 0) & (wide[2012] == 0),
(wide[2010] == 1) & (wide[2012] == 1),
(wide[2010] == 0) & (wide[2012] == 1)],
["never", "always", "joiner"], default="leaver")
counts = pd.Series(pattern).value_counts().reindex(["never", "always", "joiner", "leaver"])
print(counts.to_string())
print(f"Switchers: {counts['joiner'] + counts['leaver']} "
f"({100 * (counts['joiner'] + counts['leaver']) / len(wide):.1f}% of workers)")

never 1805
always 321
joiner 36
leaver 37
Switchers: 73 (3.3% of workers)

Only 73 workers (3.3%) switch, and they split almost evenly between joiners and leavers. Every within estimator in this post is identified by these 73 workers alone.

21.2 Core

Exercise 3: Verify both T = 2 identities. The proof in section 10 makes two predictions that can be checked numerically. Estimate FD with and without an intercept, one-way FE, and TWFE without standard errors. Print the four slopes to six decimals and match them in pairs.

Solution Show the code and the numbers

d = df.sort_values(["ID", "year"]).groupby("ID")[["lwage", "union"]].diff().dropna()
d.columns = ["d_lwage", "d_union"]
fd_noint = pf.feols("d_lwage ~ d_union - 1", data=d).coef()["d_union"]
fd_int = pf.feols("d_lwage ~ d_union", data=d).coef()["d_union"]
fe = pf.feols("lwage ~ union | ID", data=df).coef()["union"]
twfe = pf.feols("lwage ~ union | ID + year", data=df).coef()["union"]
print(f"FD without intercept: {fd_noint:.6f} FE: {fe:.6f}")
print(f"FD with intercept: {fd_int:.6f} TWFE: {twfe:.6f}")

FD without intercept: 0.210318 FE: 0.210318
FD with intercept: 0.211314 TWFE: 0.211314

The two pairs agree to six decimals, as the proof in section 10 predicts. The intercept in the FD regression and the year effect in TWFE do the same job: both remove the common wage trend.

Exercise 4: A cluster-robust Mundlak test. Estimate the Mundlak model by pooled OLS, lwage ~ union + union_bar, with standard errors clustered by worker. Compare the union coefficient with FE and the p-value of union_bar with the Hausman test. Which test is more appropriate when the standard errors are cluster-robust?

Solution Show the code and the numbers

df["union_bar"] = df.groupby("ID")["union"].transform("mean")
mundlak = pf.feols("lwage ~ union + union_bar", data=df, vcov={"CRV1": "ID"})
print(f"Union (within): {mundlak.coef()['union']:.4f} (SE {mundlak.se()['union']:.4f})")
print(f"Mundlak union_bar: {mundlak.coef()['union_bar']:+.4f} "
f"(SE {mundlak.se()['union_bar']:.4f}, p = {mundlak.pvalue()['union_bar']:.4f})")

Union (within): 0.2103 (SE 0.0812)
Mundlak union_bar: -0.1441 (SE 0.0891, p = 0.1059)

Pooled OLS reproduces the FE coefficient (0.2103) and the RE-based Mundlak coefficient (−0.1441), as the proof in section 14 implies. With cluster-robust standard errors, the test on union_bar gives p = 0.106, which is weaker than the RE version (0.072) and far from the rejection of the textbook Hausman test (0.018). The clustered Mundlak test is the more appropriate one here, because it remains valid when the errors are heteroskedastic or correlated within a worker, whereas the Hausman test requires classical errors.

Exercise 5: Joiners versus leavers. Re-estimate TWFE twice: once on the stayers plus the joiners, and once on the stayers plus the leavers. Is the union premium symmetric? Interpret any difference between the two estimates.

Solution Show the code and the numbers

wide = df.pivot(index="ID", columns="year", values="union")
joiner_ids = wide.index[(wide[2010] == 0) & (wide[2012] == 1)]
leaver_ids = wide.index[(wide[2010] == 1) & (wide[2012] == 0)]
for label, drop_ids in [("Joiners + stayers", leaver_ids), ("Leavers + stayers", joiner_ids)]:
sub = df[~df["ID"].isin(drop_ids)]
fit = pf.feols("lwage ~ union | ID + year", data=sub, vcov={"CRV1": "ID"})
print(f"{label}: {fit.coef()['union']:.4f} (SE {fit.se()['union']:.4f})")

Joiners + stayers: 0.3447 (SE 0.1382)
Leavers + stayers: 0.0814 (SE 0.0750)

Joining is associated with a gain of 34 log points, while leaving is associated with a loss of only 8 log points. The pooled estimate of 0.21 averages two very different responses, each based on fewer than 40 workers, so the symmetry built into the within model deserves scrutiny.

21.3 Stretch

Exercise 6: All five waves. Use df_full (2010 to 2018) to estimate two-way FE and first differences with year effects, both clustered by worker. Report the within share of union variance. Do FD and FE still agree, and does the 0.21 premium survive?

Solution Show the code and the numbers

df5 = df_full.copy()
df5["union"] = df5["union"].astype(str).map({"Yes": 1, "No": 0}).astype(float)
df5 = df5.dropna(subset=["lwage", "union"]).sort_values(["ID", "year"])
d5 = df5.groupby("ID")[["lwage", "union"]].diff()
d5.columns = ["d_lwage", "d_union"]
d5[["ID", "year"]] = df5[["ID", "year"]]
d5 = d5.dropna()
fe5 = pf.feols("lwage ~ union | ID + year", data=df5, vcov={"CRV1": "ID"})
fd5 = pf.feols("d_lwage ~ d_union | year", data=d5, vcov={"CRV1": "ID"})
worker_mean = df5.groupby("ID")["union"].transform("mean")
within_var = (df5["union"] - worker_mean).var()
between_var = df5.groupby("ID")["union"].mean().var()
print(f"Observations: {len(df5)} Workers: {df5['ID'].nunique()} Waves: {df5['year'].nunique()}")
print(f"Within share of union variance: {100 * within_var / (within_var + between_var):.1f}%")
print(f"Two-way FE (demeaning): {fe5.coef()['union']:.4f} (SE {fe5.se()['union']:.4f})")
print(f"First differences + year FE: {fd5.coef()['d_union']:.4f} (SE {fd5.se()['d_union']:.4f})")

Observations: 11045 Workers: 2209 Waves: 5
Within share of union variance: 16.1%
Two-way FE (demeaning): 0.0396 (SE 0.0255)
First differences + year FE: 0.0566 (SE 0.0322)

With five waves, the within share rises to 16.1%, but the estimates fall to 0.040 (FE) and 0.057 (FD), and neither is significant at the 5% level. FD and FE no longer coincide, because with T > 2 they weight the periods differently. The 0.21 premium from 2010 to 2012 therefore does not generalize to the longer panel.

22. References

The FWL Theorem: Making Multivariate Regressions Intuitive

Mon, 28 Sep 2026 00:00:00 +0000

Abstract

Including multiple variables in a regression raises a deceptively simple question: what does it actually mean to “control for” a confounder, and how can that adjustment be visualized when a multivariate fit cannot be drawn on a two-dimensional scatter plot? This tutorial answers that question through the Frisch-Waugh-Lovell (FWL) theorem, which shows that any coefficient from a multivariate regression can be recovered from a univariate regression after partialling-out the other variables. Inspired by Courthoud (2022), the analysis uses a simulated fast-food chain of 50 restaurants, one per neighborhood, each of which hands out 100 discount coupons in a single day; neighborhood income confounds the effect of the coupon redemption rate on monthly sales, with a known true causal effect (ATE) of exactly +0.2. Using Ordinary Least Squares (OLS) in statsmodels, the study estimates naive, full, and residualized (FWL) regressions, then visualizes the conditional relationship with seaborn and matplotlib. The naive regression of sales on coupons yields a misleading slope of −0.1059 (p = 0.365). Controlling for income reverses it to +0.2673 (p = 0.031), with income itself at +0.3836 (p < 0.001). The gap decomposes exactly as 0.3836 × (−0.9730) = −0.3732, the omitted-variable-bias identity. The FWL residualize-both procedure reproduces the coefficient +0.2673 exactly, and its standard error (0.118) matches the full regression’s 0.120 up to a degrees-of-freedom correction; extending to two controls gives +0.2706 both ways. This sign reversal — a textbook Simpson’s paradox — demonstrates that omitted-variable bias can flip an effect’s direction, and that FWL both verifies what multivariate regression does under the hood and provides the linear prototype for Double Machine Learning. An appendix repeats the promotion in June and applies FWL to the resulting two-period panel with analyze_fwl_plot from expdpy: restaurant fixed effects, two-way fixed effects, and first differences all reduce to residual-on-residual regressions, and a persistent restaurant trait that pushes pooled OLS to +0.4496 is removed by the fixed effects (+0.1394, clustered SE 0.069).

1. Overview

1.1 What does it mean to “control for” a variable?

Including multiple variables in a regression raises a natural question: what does it actually mean to “control for” a confounder? The output is a coefficient, but a multivariate regression cannot be plotted on a simple two-dimensional scatter plot. This makes it hard to build intuition about what the regression is doing behind the scenes.

The Frisch-Waugh-Lovell (FWL) theorem answers this question. It shows that any coefficient from a multivariate regression can be recovered from a simple univariate regression — after removing the influence of all other variables through a procedure called partialling-out (also known as residualization or orthogonalization). Think of it as removing the part of each variable that the controls can explain, so only the variation the controls cannot account for remains.

This tutorial is inspired by Courthoud (2022), and applies the FWL theorem to a simulated fast-food promotion. A chain runs 50 restaurants, each in its own neighborhood. In January, every restaurant hands out 100 discount coupons on a single day and then counts how many of them come back during the month. That count is the coupon redemption rate: 37 coupons redeemed out of 100 is a rate of 37%. The chain wants to know whether a higher redemption rate raises the restaurant’s monthly sales. The catch: neighborhood income affects both how many coupons are redeemed and how much people spend, creating a confounding relationship that makes the naive analysis misleading. The analysis uses FWL to untangle these effects, verifies the theorem step by step, and visualizes the conditional relationship that multivariate regression captures but hides from view.

1.2 Learning objectives

By the end of this post you will be able to:

Diagnose omitted-variable bias in a confounded dataset and explain why the naive coupon slope (−0.1059) has the wrong sign.
Decompose the gap between the naive and controlled coefficients exactly, using the omitted-variable-bias identity.
State the Frisch-Waugh-Lovell theorem and prove it with the residual-maker matrix.
Implement FWL three ways (a full Ordinary Least Squares, or OLS, regression; a residual-on-residual regression; and plain NumPy) and confirm that all three return 0.2673.
Explain why residualizing only the treatment reproduces the coefficient but not its standard error, and trace the gap to its three sources: the missing intercept, income’s variation left in sales, and the degrees of freedom.
Visualize a conditional relationship with residualized, mean-rescaled scatter plots.
Evaluate when linear partialling-out identifies a causal effect and when it fails, and connect it to Double Machine Learning.
Extend FWL to panel data: show that restaurant fixed effects, two-way fixed effects, and first differences are all residual-on-residual regressions, using analyze_fwl_plot from expdpy (appendix).

1.3 The road ahead

The tutorial runs in eight stages. The same 50 restaurants carry the whole story, and the true coupon effect of 0.2 never changes. Each stage either finds a number, explains it, or tries to break it.

graph TD
A("<b>Simulate 50 restaurants</b><br/>true coupon effect = 0.2") --> B("<b>Naive regression</b><br/>slope −0.106,<br/>the wrong sign")
B --> C("<b>Control for income</b><br/>slope +0.267, and the<br/>OVB identity explains the gap")
C --> D("<b>The FWL theorem</b><br/>a short proof,<br/>then the same number in NumPy")
D --> E("<b>Verify step by step</b><br/>same coefficient,<br/>different SEs")
E --> F("<b>See it</b><br/>residual plots, rescaled axes,<br/>a second control")
F --> G("<b>Break it yourself</b><br/>interactive lab")
G --> H("<b>Beyond OLS</b><br/>misconceptions, DML,<br/>panel data appendix")
classDef puzzle fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef solve fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef check fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef practice fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class A,B puzzle
class C,D solve
class E,F check
class G,H practice

The border colors of the boxes mark the logic of the argument. The boxes with blue borders set up the puzzle, a known truth and a naive slope with the wrong sign. The boxes with orange borders solve it twice, first with the omitted-variable-bias identity and then with the FWL theorem. The boxes with teal borders check the answer, first with numbers and then with pictures. Finally, the boxes with gray borders turn the analysis over to the reader through a lab where the result can be broken on purpose, a set of misconceptions, and the bridge to Double Machine Learning. The last box also includes the appendix, which runs the same promotion again in June and applies FWL to the resulting two-month panel.

2. Key concepts at a glance

The rest of the post leans on a small vocabulary. Each concept has three parts. The definition is always visible; the example and analogy sit behind clickable cards. Open them when a term feels slippery — “partialling-out” and “omitted-variable bias” are the two that later sections lean on hardest.

1. Frisch-Waugh-Lovell theorem Two routes, one coefficient. The coefficient on $X_1$ in the full regression equals the slope from a simple regression of $\tilde Y$ on $\tilde X_1$. The tildes mark residuals from regressing each variable on the other controls.

Example

In this post, regressing sales on coupons and income jointly gives a coupon coefficient of +0.2673. Residualizing both variables on income and re-regressing gives +0.2673 too. It is the same number to machine precision, not an approximation.

Analogy

Weigh a patient in a winter coat and subtract the coat, or take the coat off first and weigh again: the scale shows the same person. The full regression subtracts income in place; FWL takes income off first.

2. Confounding A third variable pulling on both ends. A confounder $X_2$ affects both the treatment $X_1$ and the outcome $Y$. It creates an association between them that is not causal.

Example

In this post income confounds coupons → sales. Restaurants in high-income neighborhoods redeem fewer coupons but sell more, so the raw coupon slope is negative (−0.1059, p = 0.365) even though the true effect is +0.2.

Analogy

Ice-cream sales and drownings rise and fall together. Ice cream does not cause drowning; hot weather drives both. Here income plays the weather.

3. Partialling-out / residualization $\tilde y = y - \hat y$. Replace each variable with the part the controls cannot explain. That leftover is what FWL works with.

Example

Regress coupons on income and keep the residuals. Do the same for sales. Plot one set of residuals against the other, and the fitted line has slope +0.2673 — the controlled coefficient, now visible.

Analogy

Judging a runner on a windy day. Subtract what the wind alone would have done to the time, then compare what is left. The leftover time belongs to the runner, not the weather.

4. Omitted-variable bias $\hat\beta_1^{\mathrm{naive}} = \hat\beta_1 + \hat\gamma \cdot \hat\delta$. Leave a relevant variable out and the slope absorbs part of its effect. Here $\hat\gamma$ is the omitted variable’s coefficient in the full regression, and $\hat\delta$ is the slope from regressing the omitted variable on $X_1$. The identity is exact in the sample. It links the naive slope to the full-regression coefficient $\hat\beta_1$, not to the true 0.2.

Example

Income’s coefficient is $\hat\gamma = 0.3836$ and the slope of income on coupons is $\hat\delta = -0.9730$. Their product, 0.3836 × (−0.9730) = −0.3732, equals the gap −0.1059 − 0.2673 exactly. In the population $\delta = -1.0$, so the naive slope converges to 0.2 + 0.3 × (−1.0) = −0.10.

Analogy

A bathroom scale that reads 3 kg light for everyone. Once you know the error, you add it back exactly. The OVB identity is that correction for a regression: it says precisely how far, and in which direction, the naive slope was pushed.

5. Conditional vs. marginal effect $E[Y \mid X_1, X_2]$ vs. $E[Y \mid X_1]$. The conditional slope holds the other variables fixed. The marginal slope averages over them. The two coincide when the omitted variable does not affect $Y$ or is uncorrelated with $X_1$. In a sample they coincide exactly when $\hat\gamma \hat\delta = 0$.

Example

The marginal coupon slope is −0.1059. The conditional slope, holding income fixed, is +0.2673. They differ because $\hat\gamma \hat\delta = -0.3732$ is far from zero.

Analogy

Across a whole city, houses with more bedrooms can look cheaper, because big houses sit far from the center where land is cheap. Within one neighborhood, each extra bedroom raises the price. The city-wide comparison is marginal; the within-neighborhood comparison is conditional.

6. Backdoor path A non-causal route through a confounder. In the DAG coupons ← income → sales, the backdoor path runs from coupons back through income to sales. Conditioning on income blocks it and identifies the causal effect.

Example

Including income in the regression closes the backdoor. The resulting +0.2673 is an estimate of the causal effect, whose true value is 0.2. Closing the backdoor removes the bias, not the sampling noise: with 50 restaurants the 95% confidence interval runs from 0.025 to 0.509.

Analogy

A concert hall with a side door open to a noisy street. Shut the door and the music is clean. You don’t have to stop the traffic — you only have to block the path it takes into the hall.

7. DML bridge — FWL → Double Machine Learning Residualize $Y$ and $D$ with ML, then regress. FWL is the linear-OLS prototype of DML, where $D$ is the treatment (our $X_1$). Replace the OLS partialling-out with flexible machine-learning models and the same residual-on-residual logic handles high-dimensional or nonlinear controls. DML adds cross-fitting: each observation’s residuals come from models trained on the other folds of the data, so a flexible model cannot fit that observation’s own noise into them.

Example

The residualize-then-regress logic that recovers +0.2673 here is what doubleml runs at scale, with cross-fitting and a random forest or lasso in place of OLS.

Analogy

FWL is a carpenter’s straightedge. DML swaps it for a flexible curve ruler. The job stays the same: trace the part of each variable the controls explain, and keep what is left.

3. The causal structure

Before looking at data, it helps to understand the causal relationships among the variables. A Directed Acyclic Graph (DAG) is a diagram in which each arrow represents a direct causal effect of one variable on another. Drawing the graph first makes the assumptions of the analysis explicit, so the reader can see which variables must be controlled for and why.

In this fast-food scenario, three variables interact:

graph LR
I("<b>Income</b><br/>confounder") -.->|"fewer<br/>redemptions"| C("<b>Coupons</b><br/>treatment:<br/>redemption rate")
I -.->|"more<br/>spending"| S("<b>Sales</b><br/>outcome:<br/>monthly sales")
C ===>|"causal effect<br/>+0.2"| S
classDef confounder fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef treatment fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef outcome fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class I confounder
class C treatment
class S outcome
linkStyle 0,1 stroke:#d97757,stroke-width:2.5px,stroke-dasharray:7 5
linkStyle 2 stroke:#00d4c8,stroke-width:3px

Income acts as a confounder, which is a variable that influences both the treatment (the coupon redemption rate) and the outcome (monthly sales). The two dashed orange arrows trace this influence, because wealthier neighborhoods redeem fewer coupons but also spend more. Together, these arrows form a backdoor path from coupons to sales that runs through income. If the analysis ignores income, this backdoor path produces a spurious negative association between coupons and sales, and that association hides the true positive effect shown by the thick teal arrow.

To estimate the causal effect without this bias, the analysis must block the backdoor path by conditioning on income. Once income is held fixed, the only remaining link between coupons and sales is the thick teal arrow. The FWL theorem provides an elegant way to do this and also to visualize the result.

4. Setup and imports

The following code loads all necessary libraries. The analysis relies on statsmodels for OLS regression, seaborn for regression plots, and matplotlib for figure customization. The RANDOM_SEED ensures that every reader gets identical results.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.formula.api as smf
# Reproducibility
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
# Site color palette
STEEL_BLUE = "#6a9bcc"
WARM_ORANGE = "#d97757"
NEAR_BLACK = "#141413"
TEAL = "#00d4c8"

Note on figure styling: The published figures in this post come from the companion script.py, which adds a dark theme for visual consistency with the site and labels each fitted line with its equation and R². The plotting code below keeps matplotlib’s defaults and uses colors that read well on both light and dark backgrounds. To reproduce the dark-themed figures, add the following to your setup:
Dark theme settings (click to expand)
DARK_NAVY = "#0f1729"
GRID_LINE = "#1f2b5e"
LIGHT_TEXT = "#c8d0e0"
WHITE_TEXT = "#e8ecf2"
plt.rcParams.update({
"figure.facecolor": DARK_NAVY, "axes.facecolor": DARK_NAVY,
"axes.edgecolor": DARK_NAVY, "axes.linewidth": 0,
"axes.labelcolor": LIGHT_TEXT, "axes.titlecolor": WHITE_TEXT,
"axes.spines.top": False, "axes.spines.right": False,
"axes.spines.left": False, "axes.spines.bottom": False,
"axes.grid": True, "grid.color": GRID_LINE,
"grid.linewidth": 0.6, "grid.alpha": 0.8,
"xtick.color": LIGHT_TEXT, "ytick.color": LIGHT_TEXT,
"text.color": WHITE_TEXT, "font.size": 12,
"legend.frameon": False, "legend.labelcolor": LIGHT_TEXT,
"savefig.facecolor": DARK_NAVY, "savefig.edgecolor": DARK_NAVY,
})

5. Data simulation

Rather than importing data from an external source, this section builds a transparent data generating process (DGP) so that the true causal effect is known in advance and the methods can be verified against it. Think of it as running a controlled experiment in a computer: set the rules, generate the data, and then check whether the statistical tools find the right answer.

The DGP encodes the causal structure from the DAG above:

income is drawn from a normal distribution centered at \$50K
coupons is the redemption rate: the percentage of the restaurant’s 100 coupons that were redeemed during the month. It depends negatively on income (wealthier customers redeem fewer coupons) plus random noise. The simulation keeps two decimals, so read 36.93 as “about 37 of the 100 coupons came back.”
sales is the restaurant’s monthly sales in thousands of dollars. It depends positively on both coupons (+0.2) and income (+0.3), plus a day-of-week effect and random noise

The true causal effect of coupons on sales is exactly +0.2 — this is the Average Treatment Effect (ATE), the average impact of coupons on sales across all restaurants. In concrete terms, every 1 percentage point increase in the redemption rate (one more of the 100 coupons redeemed) causes a \$200 increase in monthly sales (measured in thousands).

def simulate_store_data(n=50, seed=42):
"""Simulate one month of the 50 restaurants, confounded by income."""
rng = np.random.default_rng(seed)
income = rng.normal(50, 10, n)
dayofweek = rng.integers(1, 8, n)
coupons = 60 - 0.5 * income + rng.normal(0, 5, n)
sales = (10 + 0.2 * coupons + 0.3 * income
+ 0.5 * dayofweek + rng.normal(0, 3, n))
return pd.DataFrame({
"sales": np.round(sales, 2),
"coupons": np.round(coupons, 2),
"income": np.round(income, 2),
"dayofweek": dayofweek,
})
N = 50
df = simulate_store_data(n=N, seed=RANDOM_SEED)
print("Dataset shape:", df.shape)
print()
print(df.head())
print()
print(df.describe().round(2))

Dataset shape: (50, 4)
sales coupons income dayofweek
0 37.37 36.93 53.05 6
1 36.88 38.06 39.60 6
2 33.09 32.04 57.50 6
3 35.09 33.43 59.41 5
4 27.01 43.21 30.49 4
sales coupons income dayofweek
count 50.00 50.00 50.00 50.00
mean 33.61 33.84 50.91 3.92
std 3.96 4.89 7.68 1.88
min 25.76 23.26 30.49 1.00
25% 31.30 31.53 45.78 2.00
50% 33.24 33.25 51.74 4.00
75% 36.00 36.89 56.42 5.75
max 44.38 43.79 71.42 7.00

The dataset contains 50 restaurants with average monthly sales of \$33,610, an average redemption rate of 33.84% (about 34 of the 100 coupons), and average neighborhood income of \$50,910. Monthly sales range from \$25,760 to \$44,380, reflecting meaningful variation across restaurants. The redemption rate spans from 23% to 44%, and income ranges from \$30,490 to \$71,420. This variation provides enough signal to estimate the relationships of interest.

6. The naive relationship

The simplest approach is to regress sales directly on the redemption rate, ignoring income entirely. This is what a rushed analyst might do — just look at whether restaurants that redeemed more coupons have higher or lower sales.

Predict first

Before running the naive regression: what sign will the slope of sales on coupons have, and which variable is to blame? Commit to a sign and a reason before scrolling.

Reveal the answer

Answer. Negative: the slope is −0.1059, even though the true effect is +0.2. Income is to blame. Wealthier neighborhoods redeem fewer coupons and spend more, so the restaurants with many redemptions tend to sit in poorer neighborhoods with lower sales. The naive regression credits that income gap to coupons.

sns.regplot(x="coupons", y="sales", data=df, ci=None,
scatter_kws={"color": STEEL_BLUE, "alpha": 0.7, "edgecolors": "gray",
"linewidths": 0.5, "s": 60},
line_kws={"color": WARM_ORANGE, "linewidth": 2, "label": "Linear fit"})
plt.legend()
plt.xlabel("Coupon redemption rate (%)")
plt.ylabel("Monthly sales (thousands $)")
plt.title("Naive relationship: Sales vs. coupon redemption")
plt.savefig("fwl_naive_regression.png", dpi=300, bbox_inches="tight")
plt.show()

Naive regression: the downward slope suggests coupons reduce sales, but this is driven by confounding from income.

The plot suggests a negative slope; the regression table puts a number and a p-value on it.

naive_model = smf.ols("sales ~ coupons", df).fit()
print(naive_model.summary().tables[1])

==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 37.1906 3.960 9.390 0.000 29.228 45.154
coupons -0.1059 0.116 -0.914 0.365 -0.339 0.127
==============================================================================

The naive regression suggests that coupons have a negative effect on sales: each additional percentage point of redemption is associated with \$106 less in monthly sales. However, this coefficient is not statistically significant (p = 0.365), and the 95% confidence interval [−0.339, 0.127] spans both negative and positive values. More importantly, the true effect is +0.2, so this estimate is not just imprecise — it points in the wrong direction. The confounder (income) is pulling the estimate downward because wealthier neighborhoods redeem fewer coupons but spend more.

7. Controlling for income

7.1 The full regression

To block the backdoor path through income, the next step includes it as a control variable in the regression. This is the standard approach in applied work: add the confounder to the right-hand side of the regression equation.

full_model = smf.ols("sales ~ coupons + income", df).fit()
print(full_model.summary().tables[1])

==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 5.0278 7.181 0.700 0.487 -9.418 19.474
coupons 0.2673 0.120 2.222 0.031 0.025 0.509
income 0.3836 0.076 5.015 0.000 0.230 0.537
==============================================================================

Controlling for income reverses the picture entirely. The coefficient on coupons is now +0.2673 (p = 0.031): an estimated \$267 more in monthly sales for each additional percentage point of redemption. The true effect is +0.2 (\$200), and the 95% confidence interval [0.025, 0.509] contains it while no longer including zero. Income’s own coefficient, +0.3836 (p < 0.001), confirms that wealthier neighborhoods spend more. By conditioning on income, the backdoor path is blocked and the estimate moves much closer to the true causal effect.

7.2 Where the bias comes from: the OVB identity

The naive slope is −0.1059 and the controlled slope is +0.2673. The gap between them is not a mystery. It can be computed exactly, before any theorem about residuals.

Predict first

The gap between the naive and full coefficients is −0.1059 − 0.2673 = −0.3732, and income’s coefficient in the full regression is $\hat\gamma = 0.3836$. If the gap equals $\hat\gamma \cdot \hat\delta$, what must $\hat\delta$ be, and what is its sign? Commit to an answer before scrolling.

Reveal the answer

Answer. $\hat\delta = -0.3732 / 0.3836 \approx -0.973$. It is negative: restaurants with more redemptions sit in poorer neighborhoods. The code below prints −0.9730. The population value, implied by the DGP, is −1.0.

The omitted-variable-bias (OVB) identity links the two regressions:

$$\hat\beta_1^{\mathrm{naive}} = \hat\beta_1 + \hat\gamma \cdot \hat\delta$$

In words, this says that the naive slope equals the controlled slope plus a bias term. The bias term is the omitted variable’s effect on the outcome times how strongly the omitted variable moves with the treatment. Here $\hat\beta_1^{\mathrm{naive}}$ is the coupon coefficient in naive_model, $\hat\beta_1$ is the coupon coefficient in full_model, $\hat\gamma$ is the income coefficient in full_model, and $\hat\delta$ is the slope from regressing income on coupons. The identity is exact in any sample, not only on average.

gamma_hat = full_model.params["income"]
delta_hat = smf.ols("income ~ coupons", df).fit().params["coupons"]
naive_hat, full_hat = naive_model.params["coupons"], full_model.params["coupons"]
print(f"gamma_hat x delta_hat = {gamma_hat:.4f} x {delta_hat:.4f} = {gamma_hat * delta_hat:.4f}")
print(f"naive - full = {naive_hat:.4f} - {full_hat:.4f} = {naive_hat - full_hat:.4f}")

gamma_hat x delta_hat = 0.3836 x -0.9730 = -0.3732
naive - full = -0.1059 - 0.2673 = -0.3732

The two lines agree to the last digit: 0.3836 × (−0.9730) = −0.3732 = −0.1059 − 0.2673. Income raises sales ($\hat\gamma > 0$) and moves against coupons ($\hat\delta < 0$), so the bias is negative and large enough to flip the sign. The population version follows from the DGP. There, Cov(income, coupons) = −0.5 × 100 = −50 and Var(coupons) = 0.25 × 100 + 25 = 50, so $\delta = -1.0$ and the naive slope converges to 0.2 + 0.3 × (−1.0) = −0.10. The naive regression omits dayofweek too, but dayofweek is independent of coupons in the DGP, so it adds nothing to the population bias.

The direction of the auxiliary regression matters. Regressing coupons on income instead (slope −0.3935) gives a bias term of 0.3836 × (−0.3935) = −0.151. That would imply a naive slope of 0.2673 − 0.151 = +0.116, not the −0.106 we observe, so it reconciles nothing. The omitted variable goes on the left-hand side; the treatment goes on the right.

But what is the regression actually doing when it “controls for” income? This is where the FWL theorem provides a clear answer.

8. The FWL theorem

8.1 The statement

Ragnar Frisch and Frederick Waugh first published the result in 1933, for the case of detrending time series. Michael Lovell generalized it in 1963 in the Journal of the American Statistical Association, in a paper on seasonal adjustment. Decades later he published a short proof for teaching, “A Simple Proof of the FWL Theorem” (Lovell, 2008). The theorem gives a precise algebraic decomposition of what multivariate regression does under the hood.

Consider a linear model with two sets of regressors:

$$y_i = \beta_1 x_{i,1} + \beta_2 x_{i,2} + \varepsilon_i$$

In words, this equation says that the outcome $y$ (sales) equals the effect $\beta_1$ of the variable of interest $x_1$ (coupons), plus the effect $\beta_2$ of the control variable $x_2$ (income), plus an error term $\varepsilon$. In this analysis, $y$ corresponds to the sales column, $x_1$ to coupons, and $x_2$ to income; the intercept counts as one more control. The coefficient $\beta_2$ here is the $\gamma$ of section 7.2, and the lowercase $y$ and $x_1$ are the $Y$ and $X_1$ of the concept cards in section 2.

The FWL theorem states that the Ordinary Least Squares (OLS) estimator — the standard method for fitting a regression line by minimizing squared prediction errors — $\hat{\beta}_1$ from this multivariate regression is identical to the estimator obtained from a simpler procedure:

$$\hat{\beta}_1^{\mathrm{FWL}} = \frac{\text{Cov}(\tilde{y}, \, \tilde{x}_1)}{\text{Var}(\tilde{x}_1)}$$

where $\tilde{x}_1$ is the residual from regressing $x_1$ on $x_2$, and $\tilde{y}$ is the residual from regressing $y$ on $x_2$.

In words, this says: to estimate the effect of coupons while controlling for income, we can (1) remove income’s influence from coupons, (2) remove income’s influence from sales, and (3) regress the cleaned sales on the cleaned coupons. The resulting coefficient is exactly the same as the one from the full multivariate regression.

This procedure is called partialling-out because it removes the variation explained by the control variables, keeping only the residual variation: the part that is orthogonal to — that is, uncorrelated with — income in the sample (not necessarily independent of it). The three equivalent estimators are:

Full OLS: Regress $y$ on $x_1$ and $x_2$ jointly
Partial FWL: Regress $y$ on $\tilde{x}_1$ (residuals of $x_1$ on $x_2$)
Full FWL: Regress $\tilde{y}$ on $\tilde{x}_1$ without an intercept (residuals of both variables on $x_2$)

All three produce the same $\hat{\beta}_1$. The full FWL (option 3) also reproduces the full regression’s residuals exactly, so its standard error matches up to a degrees-of-freedom correction; section 10.3 shows how much.

8.2 Why it works

The theorem is not a coincidence of this dataset; it follows from a few lines of matrix algebra, and section 9 checks each line numerically.

Proof Why FWL holds, in five lines of matrix algebra

Stack the $n$ restaurants. Let $y$ be the outcome vector, $X_1$ the treatment column, and $X_2$ the matrix of controls, including the column of ones. Define the residual-maker matrix

$$M_2 = I_n - X_2 (X_2^\top X_2)^{-1} X_2^\top$$

Multiplying any vector by $M_2$ returns the residuals from regressing that vector on $X_2$. Three facts follow directly from the definition:

$$M_2 X_2 = 0, \qquad M_2^\top = M_2, \qquad M_2 M_2 = M_2$$

Line 1. Write the full OLS fit with its residual vector $e$. The normal equations, the first-order conditions for minimizing the sum of squared residuals, make $e$ orthogonal to every regressor:

$$y = X_1 \hat\beta_1 + X_2 \hat\beta_2 + e$$

$$X_1^\top e = 0, \qquad X_2^\top e = 0$$

Line 2. Premultiply by $M_2$. The control term vanishes because $M_2 X_2 = 0$, and $M_2 e = e$ because $e$ is already orthogonal to $X_2$:

$$M_2 y = M_2 X_1 \hat\beta_1 + e$$

Line 3. Premultiply by $X_1^\top$. The last term drops because $X_1^\top e = 0$:

$$X_1^\top M_2 y = X_1^\top M_2 X_1 \hat\beta_1$$

Line 4. Solve for the coefficient:

$$\hat\beta_1 = (X_1^\top M_2 X_1)^{-1} X_1^\top M_2 y$$

Line 5. Write $\tilde x_1 = M_2 X_1$ and $\tilde y = M_2 y$. Because $M_2$ is symmetric and idempotent, $X_1^\top M_2 X_1 = \tilde x_1^\top \tilde x_1$ and $X_1^\top M_2 y = \tilde x_1^\top \tilde y$. So

$$\hat\beta_1 = (\tilde x_1^\top \tilde x_1)^{-1} \tilde x_1^\top \tilde y$$

That is the regression of the residualized outcome on the residualized treatment — the FWL theorem. Three corollaries fall out.

(a) Step 1 works. $\tilde x_1^\top \tilde y = \tilde x_1^\top y$, because $M_2$ is symmetric and idempotent. Regressing the raw outcome on $\tilde x_1$ gives the same slope. That is Step 1 in section 10.1.

(b) Only the degrees of freedom differ. Line 2 says $\tilde y = \tilde x_1 \hat\beta_1 + e$, so the residual-on-residual regression leaves exactly the full model’s residuals $e$. The sum of squared residuals is identical. By the same algebra (the partitioned-inverse formula), the diagonal element of the full regression’s $(X^\top X)^{-1}$ that belongs to $X_1$ equals $(\tilde x_1^\top \tilde x_1)^{-1}$, so the variance factor is shared too. Only the divisor changes, 49 instead of 47, which is why Step 2’s standard error is $\sqrt{47/49} = 0.9794$ times the full model’s.

(c) The covariance formula. With a single treatment column, $(\tilde x_1^\top \tilde x_1)^{-1} \tilde x_1^\top \tilde y$ is a ratio of two numbers. The residuals have mean zero because $X_2$ contains the constant, so dividing both by $n - 1$ turns the ratio into the Cov/Var formula of section 8.1.

9. FWL by hand in NumPy

9.1 The covariance formula

statsmodels hides the arithmetic. The formula in section 8.1 needs only two residual vectors, one covariance, and one variance, so it fits in a few lines of NumPy. np.linalg.lstsq solves each least-squares problem. The design matrix X2 holds a column of ones next to income, so the intercept is partialled out along with income.

y = df["sales"].to_numpy()
x1 = df["coupons"].to_numpy()
X2 = np.column_stack([np.ones(N), df["income"]])
c_tilde = x1 - X2 @ np.linalg.lstsq(X2, x1, rcond=None)[0]
s_tilde = y - X2 @ np.linalg.lstsq(X2, y, rcond=None)[0]
cov_sc = np.cov(s_tilde, c_tilde)[0, 1] # np.cov divides by n - 1
var_c = np.var(c_tilde, ddof=1) # so the variance must too
print(f"Cov(s_tilde, c_tilde) = {cov_sc:.4f}")
print(f"Var(c_tilde) = {var_c:.4f}")
print(f"beta_1 = Cov / Var = {cov_sc / var_c:.4f}")
print(f"Mixed divisors (wrong) = {cov_sc / np.var(c_tilde):.4f}")

Cov(s_tilde, c_tilde) = 3.9380
Var(c_tilde) = 14.7320
beta_1 = Cov / Var = 0.2673
Mixed divisors (wrong) = 0.2728

Covariance over variance returns 0.2673, the same coefficient as the full regression, with no regression library involved. One trap is worth a warning. np.cov divides by $n - 1$ by default, while np.var divides by $n$ unless you pass ddof=1. The numerator and denominator must use the same divisor — then it cancels. Mixing them inflates the slope by 50/49, to 0.2728.

9.2 The matrix form

The proof in section 8.2 runs through the residual-maker matrix $M_2$. For 50 restaurants it is only a 50 × 50 matrix, so we can build it explicitly and check its defining properties.

M2 = np.eye(N) - X2 @ np.linalg.inv(X2.T @ X2) @ X2.T
print("M2 symmetric: ", np.allclose(M2, M2.T))
print("M2 idempotent:", np.allclose(M2 @ M2, M2))
print("M2 @ X2 is zero:", np.abs(M2 @ X2).max() < 1e-10) # tiny float noise varies by machine
beta_matrix = (x1 @ M2 @ y) / (x1 @ M2 @ x1)
print(f"(x1' M2 y) / (x1' M2 x1) = {beta_matrix:.4f}")

M2 symmetric: True
M2 idempotent: True
M2 @ X2 is zero: True
(x1' M2 y) / (x1' M2 x1) = 0.2673

$M_2$ is symmetric and idempotent, and it annihilates the controls: every entry of $M_2 X_2$ is zero up to floating-point error. The code tests against a tolerance of 1e-10 because the leftover noise, of order 1e-14, differs from one machine to the next. The last line is Line 4 of the proof with a single treatment column, and it returns 0.2673 once more. Notice that the code multiplies the raw x1 and y by $M_2$; the matrix does the residualizing inside the product. That is Line 5 at work: because $M_2$ is symmetric and idempotent, $X_1^\top M_2 y$ is the same number as $\tilde x_1^\top \tilde y$.

10. Verifying FWL step by step

Let us verify each step of the theorem using the simulated data and statsmodels, and watch what happens to the standard errors along the way.

10.1 Step 1: Residualize coupons only

First, we regress coupons on income and extract the residuals $\tilde{x}_1$. These residuals represent the variation in the redemption rate that cannot be explained by income — the “purified” coupon signal. Then we regress raw sales on these residuals, without an intercept.

Predict first

This regression uses residualized coupons but raw sales, with no intercept. (a) Will the coefficient still be 0.2673? (b) Will the standard error still be about 0.12? Commit to an answer before scrolling.

Reveal the answer

Answer. (a) Yes, exactly 0.2673 — corollary (a) of the proof guarantees it. (b) No: the standard error jumps to 1.2715, more than ten times larger. Most of that jump comes from the missing intercept; section 10.2 takes it apart.

# Residualize coupons with respect to income
df["coupons_tilde"] = smf.ols("coupons ~ income", df).fit().resid
# Regress sales on residualized coupons (no intercept)
fwl_step1 = smf.ols("sales ~ coupons_tilde - 1", df).fit()
print(fwl_step1.summary().tables[1])

=================================================================================
coef std err t P>|t| [0.025 0.975]
---------------------------------------------------------------------------------
coupons_tilde 0.2673 1.271 0.210 0.834 -2.288 2.822
=================================================================================

The coefficient is exactly 0.2673 — identical to the full regression. However, the standard error has exploded from 0.120 to 1.271, making the estimate appear insignificant (p = 0.834). A reader who stopped here would conclude that coupons do nothing.

Why does one number survive and the other not? The coefficient is unaffected by dropping the intercept, because coupons_tilde is orthogonal both to the constant and to income. But raw sales is not mean-zero (its mean is 33.61), so dropping the intercept matters a great deal for the standard error.

10.2 Why Step 1’s standard error explodes

Four regressions separate the causes. All of them return the same coefficient; only the standard error, the sum of squared residuals (SSR), and the residual degrees of freedom change.

step1_int = smf.ols("sales ~ coupons_tilde", df).fit()
step1_dm = smf.ols("I(sales - sales.mean()) ~ coupons_tilde - 1", df).fit()
rows = [("Step 1, no intercept", fwl_step1, "coupons_tilde"),
("Step 1 + intercept", step1_int, "coupons_tilde"),
("Step 1, demeaned sales", step1_dm, "coupons_tilde"),
("Full regression", full_model, "coupons")]
for label, m, term in rows:
print(f"{label:<23} coef {m.params[term]:.4f} SE {m.bse[term]:.4f} "
f"SSR {m.ssr:>9,.1f} df {m.df_resid:.0f}")

Step 1, no intercept coef 0.2673 SE 1.2715 SSR 57,181.2 df 49
Step 1 + intercept coef 0.2673 SE 0.1437 SSR 715.0 df 48
Step 1, demeaned sales coef 0.2673 SE 0.1422 SSR 715.0 df 49
Full regression coef 0.2673 SE 0.1203 SSR 490.8 df 47

Forced through the origin, the Step 1 line must also explain the level of sales, and it cannot: the sales mean of 33.6 stays in the residuals. The SSR is 57,181 without an intercept and 715 with one. The difference equals $n$ times the squared mean of sales, exactly:

share = ((fwl_step1.bse["coupons_tilde"] - step1_int.bse["coupons_tilde"])
/ (fwl_step1.bse["coupons_tilde"] - full_model.bse["coupons"]))
print(f"SSR gap from the intercept: {fwl_step1.ssr - step1_int.ssr:,.4f}")
print(f"n x mean(sales)^2: {N * df['sales'].mean() ** 2:,.4f}")
print(f"Share of SE gap closed: {share:.2f}")
print(f"Demeaned SE / intercept SE: {step1_dm.bse['coupons_tilde'] / step1_int.bse['coupons_tilde']:.4f}")
print(f"sqrt(48 / 49): {np.sqrt(48 / 49):.4f}")

SSR gap from the intercept: 56,466.1455
n x mean(sales)^2: 56,466.1455
Share of SE gap closed: 0.98
Demeaned SE / intercept SE: 0.9897
sqrt(48 / 49): 0.9897

Adding the intercept closes about 98% of the gap between the Step 1 and full-model standard errors. In SE levels, (1.2715 − 0.1437) / (1.2715 − 0.1203) = 0.98. Demeaning sales instead of adding an intercept gives 0.1422 rather than 0.1437. The two differ only by the degrees-of-freedom factor $\sqrt{48/49}$: the same SSR of 715 is divided by 49 in one case and by 48 in the other. The remaining step, 0.1437 → 0.1203, is income’s variation still left in sales. Residualizing sales on income as well cuts the SSR from 715 to 491. The degrees-of-freedom change works slightly the other way — the full model divides by 47, not 48 — so the SSR effect is a little larger than the SE step suggests.

10.3 Step 2: Residualize both variables

To fix the standard errors, we also residualize sales with respect to income. Now both variables have had income’s influence removed.

Predict first

Step 2 regresses residualized sales on residualized coupons, with no intercept, and it leaves exactly the same residuals as the full model. Starting from the full model’s standard error of 0.1203, predict Step 2’s standard error to three decimals. Hint: count the parameters each regression estimates. Commit to a number before scrolling.

Reveal the answer

Answer. 0.118. The sum of squared residuals is the same, but statsmodels divides it by 50 − 1 = 49 residual degrees of freedom in Step 2 (one coefficient) and by 50 − 3 = 47 in the full model (intercept, coupons, and income). So Step 2’s standard error is the full model’s times $\sqrt{47/49} = 0.9794$: 0.1203 × 0.9794 = 0.1178.

# Residualize sales with respect to income
df["sales_tilde"] = smf.ols("sales ~ income", df).fit().resid
# Regress residualized sales on residualized coupons (no intercept)
fwl_step2 = smf.ols("sales_tilde ~ coupons_tilde - 1", df).fit()
print(fwl_step2.summary().tables[1])

=================================================================================
coef std err t P>|t| [0.025 0.975]
---------------------------------------------------------------------------------
coupons_tilde 0.2673 0.118 2.269 0.028 0.031 0.504
=================================================================================

The coefficient remains exactly 0.2673, and now the standard error (0.118) and p-value (0.028) are nearly identical to the full regression (SE = 0.120, p = 0.031). The slight difference comes from a degrees-of-freedom adjustment. The full regression estimates two extra parameters — the intercept and the income coefficient — so it has two fewer residual degrees of freedom (47 instead of 49). The check below confirms that this is the whole story.

print(f"Step 2 SE / full SE = {fwl_step2.bse.iloc[0] / full_model.bse['coupons']:.4f}")
print(f"sqrt(47 / 49) = {np.sqrt(47 / 49):.4f}")
print("Same residuals:", np.allclose(fwl_step2.resid, full_model.resid))

Step 2 SE / full SE = 0.9794
sqrt(47 / 49) = 0.9794
Same residuals: True

The residuals are identical, and the SE ratio equals $\sqrt{47/49}$ to four decimals. So which standard error should you report? The full-model 0.1203 is correct. Step 2’s software SE of 0.1178 slightly overstates precision, because the software does not know that two parameters were already estimated in the partialling-out step. The substantive conclusion is the same either way: after partialling out income, the coupon coefficient is positive and statistically significant.

11. Visualizing partialling-out

What does partialling-out actually look like? Regressing coupons on income produces fitted values that form a line through the data. The residuals — the vertical distances between each point and this line — represent the coupon variation that income cannot explain.

df["coupons_hat"] = smf.ols("coupons ~ income", df).fit().predict()
fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(df["income"], df["coupons"], color=STEEL_BLUE, alpha=0.7,
edgecolors="gray", linewidths=0.5, s=60, label="Restaurants")
sns.regplot(x="income", y="coupons", data=df, ci=None, scatter=False,
line_kws={"color": WARM_ORANGE, "linewidth": 2, "label": "Linear fit"}, ax=ax)
ax.vlines(df["income"],
np.minimum(df["coupons"], df["coupons_hat"]),
np.maximum(df["coupons"], df["coupons_hat"]),
linestyle="--", color="gray", alpha=0.6, linewidth=1,
label="Residuals")
ax.set_xlabel("Neighborhood income (thousands $)")
ax.set_ylabel("Coupon redemption rate (%)")
ax.set_title("Partialling-out: removing income's effect on coupon redemption")
ax.legend()
plt.savefig("fwl_residuals_income.png", dpi=300, bbox_inches="tight")
plt.show()

Partialling-out: the dashed lines are the residuals — the coupon variation that income cannot explain.

The downward-sloping fitted line confirms that higher-income neighborhoods redeem fewer coupons. The vertical dashed lines are the residuals — the part of the redemption rate that income does not predict. Some restaurants got back more of their 100 coupons than their neighborhood income would suggest (positive residuals), and others got back fewer (negative residuals). Partialling out income keeps only these residuals, effectively asking: “Among restaurants in neighborhoods with similar income, which ones had unusually high or low redemption?”

12. The conditional relationship revealed

It is now possible to plot the relationship that the multivariate regression captures but cannot directly display: residualized sales against residualized coupons. Both variables have had income’s influence removed, so any remaining relationship is the conditional effect of coupons on sales — the effect after accounting for income differences.

fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(df["coupons_tilde"], df["sales_tilde"], color=STEEL_BLUE,
alpha=0.7, edgecolors="gray", linewidths=0.5, s=60,
label="Restaurants (residualized)")
sns.regplot(x="coupons_tilde", y="sales_tilde", data=df, ci=None, scatter=False,
line_kws={"color": WARM_ORANGE, "linewidth": 2, "label": "Linear fit"}, ax=ax)
ax.set_xlabel("Residual coupon redemption rate")
ax.set_ylabel("Residual monthly sales")
ax.set_title("Conditional relationship after partialling-out income")
ax.legend()
plt.savefig("fwl_partialled_out.png", dpi=300, bbox_inches="tight")
plt.show()

After removing income’s influence from both variables, a positive conditional relationship between coupons and sales emerges.

The positive slope is now clearly visible. Stripping away the confounding influence of income reveals that restaurants where redemption is higher than expected (given their neighborhood income) tend to also have monthly sales that are higher than expected. The slope of this line is exactly 0.2673 — the same coefficient produced by the full multivariate regression.

13. Scaling for interpretability

One drawback of the partialled-out plot is that both axes show residuals centered around zero, which makes the magnitudes hard to interpret. A coupon value of −5 does not mean the restaurant redeemed −5% of its coupons — it means its redemption rate is 5 percentage points (5 of the 100 coupons) below what income alone would predict.

Adding the sample mean back to each residualized variable fixes this.

Predict first

Adding the sample means back shifts every point up and to the right. Will it change the slope? Commit to an answer before scrolling.

Reveal the answer

Answer. No: the slope is 0.2673 again. Adding a constant to either axis moves the cloud, not its tilt; the intercept absorbs the shift. The standard error does move slightly, 0.119 against Step 2’s 0.118. That is degrees of freedom again: this regression estimates an intercept, so it has 48 residual degrees of freedom instead of 49.

df["coupons_tilde_scaled"] = df["coupons_tilde"] + df["coupons"].mean()
df["sales_tilde_scaled"] = df["sales_tilde"] + df["sales"].mean()
# Verify the coefficient is unchanged
scaled_model = smf.ols("sales_tilde_scaled ~ coupons_tilde_scaled", df).fit()
print(scaled_model.summary().tables[1])

========================================================================================
coef std err t P>|t| [0.025 0.975]
----------------------------------------------------------------------------------------
Intercept 24.5585 4.053 6.059 0.000 16.409 32.708
coupons_tilde_scaled 0.2673 0.119 2.246 0.029 0.028 0.507
========================================================================================

The slope is still exactly 0.2673 (p = 0.029); the intercept absorbs the shift. Plotting the rescaled residuals shows what the shift buys: axes in the original units.

fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(df["coupons_tilde_scaled"], df["sales_tilde_scaled"],
color=STEEL_BLUE, alpha=0.7, edgecolors="gray", linewidths=0.5, s=60,
label="Restaurants (residualized + scaled)")
sns.regplot(x="coupons_tilde_scaled", y="sales_tilde_scaled", data=df,
ci=None, scatter=False,
line_kws={"color": WARM_ORANGE, "linewidth": 2, "label": "Linear fit"}, ax=ax)
ax.set_xlabel("Coupon redemption rate (%, residualized + mean)")
ax.set_ylabel("Monthly sales (thousands $, residualized + mean)")
ax.set_title("Scaled residuals: interpretable magnitudes")
ax.legend()
plt.savefig("fwl_scaled_residuals.png", dpi=300, bbox_inches="tight")
plt.show()

Adding the sample means back to the residuals restores interpretable units without changing the slope.

Adding the means back moves the axes without changing the slope. The standard error of 0.119 differs from Step 2’s 0.118 only through degrees of freedom: the scaled regression estimates an intercept, so it has 48 residual degrees of freedom instead of 49, and its SE is larger by the factor $\sqrt{49/48}$. Now the axes are in interpretable units: a redemption rate around 34% and monthly sales around \$33,600. This scaled plot is a display device, ideal for presentations where the audience needs to see both the direction and the magnitude of the conditional relationship at a glance. For inference, report the full-model standard error of 0.1203.

14. Extending to multiple controls

The FWL theorem works with any number of control variables, not just one. To demonstrate, the next step adds dayofweek as a second control alongside income. The theorem says both controls can be partialled out simultaneously, and the full regression and the residual-on-residual regression will return the same coupon coefficient.

Predict first

The DGP makes dayofweek independent of coupons and of income. Will adding it as a second control change the coupon coefficient? If so, by how much, and why? Commit to an answer before scrolling.

Reveal the answer

Answer. Only slightly: from 0.2673 to 0.2706. The shift runs through the partial association between dayofweek and coupons once income is held fixed. In this sample that association is tiny but not zero: the partial correlation, the correlation between the two variables after income is partialled out of each, is −0.021. So the coefficient moves by about 0.003.

# Full regression with both controls
full_model_2 = smf.ols("sales ~ coupons + income + dayofweek", df).fit()
print(full_model_2.summary().tables[1])

==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 3.9825 7.172 0.555 0.581 -10.454 18.419
coupons 0.2706 0.119 2.266 0.028 0.030 0.511
income 0.3774 0.076 4.961 0.000 0.224 0.531
dayofweek 0.3195 0.245 1.306 0.198 -0.173 0.812
==============================================================================

The full regression puts the coupon coefficient at 0.2706. The FWL route should match it: residualize both sales and coupons on income and dayofweek together, then regress residual on residual.

# FWL: partial out both income and dayofweek
df["coupons_tilde_2"] = smf.ols("coupons ~ income + dayofweek", df).fit().resid
df["sales_tilde_2"] = smf.ols("sales ~ income + dayofweek", df).fit().resid
fwl_multi = smf.ols("sales_tilde_2 ~ coupons_tilde_2 - 1", df).fit()
print(fwl_multi.summary().tables[1])

===================================================================================
coef std err t P>|t| [0.025 0.975]
-----------------------------------------------------------------------------------
coupons_tilde_2 0.2706 0.116 2.338 0.023 0.038 0.503
===================================================================================

With both controls, the full regression gives a coupon coefficient of 0.2706 (p = 0.028). The FWL procedure — partialling out income and day of week from both sales and coupons — yields the identical coefficient of 0.2706 (p = 0.023). The day-of-week coefficient itself (0.3195, p = 0.198) is not statistically significant in this sample, even though its true value in the DGP is 0.5: with 50 restaurants, the test lacks the power to detect it. The matching coefficients confirm that FWL scales to any number of controls.

Adding dayofweek moves the coupon coefficient from 0.2673 to 0.2706, a shift of only 0.003. The OVB identity describes it exactly: 0.2673 = 0.2706 + 0.3195 × (−0.0101), where 0.3195 is dayofweek’s coefficient in the three-variable model and −0.0101 is the slope of dayofweek on coupons after controlling for income (partial correlation −0.021). The shift reflects a tiny chance association between dayofweek and coupons once income is held fixed — not the raw correlation of −0.076, and not a precision gain (precision shows up in the SE, which moves separately from 0.1203 to 0.1194). Because dayofweek is independent of coupons and income in the DGP, the shift would vanish in large samples. Exercise 2 verifies this decomposition in code.

15. Naive vs. conditional: the full picture

To appreciate how much the FWL procedure changes the conclusions, the next figure places the naive and conditional relationships side by side. The left panel shows the raw data; the right panel shows the same data after partialling out income.

fig, axes = plt.subplots(1, 2, figsize=(14, 6))
# Left: naive relationship
axes[0].scatter(df["coupons"], df["sales"], color=STEEL_BLUE, alpha=0.7,
edgecolors="gray", linewidths=0.5, s=60)
sns.regplot(x="coupons", y="sales", data=df, ci=None, scatter=False,
line_kws={"color": WARM_ORANGE, "linewidth": 2}, ax=axes[0])
axes[0].set_xlabel("Coupon redemption rate (%)")
axes[0].set_ylabel("Monthly sales (thousands $)")
axes[0].set_title("Naive (no controls)")
# Right: after partialling-out income
axes[1].scatter(df["coupons_tilde_scaled"], df["sales_tilde_scaled"],
color=TEAL, alpha=0.7, edgecolors="gray", linewidths=0.5, s=60)
sns.regplot(x="coupons_tilde_scaled", y="sales_tilde_scaled", data=df,
ci=None, scatter=False,
line_kws={"color": WARM_ORANGE, "linewidth": 2}, ax=axes[1])
axes[1].set_xlabel("Coupon redemption rate (%, after partialling-out)")
axes[1].set_ylabel("Monthly sales (thousands $, after partialling-out)")
axes[1].set_title("After partialling-out income (FWL)")
plt.suptitle("Simpson's paradox resolved: the FWL theorem reveals the conditional relationship",
fontsize=14, fontweight="bold", y=1.02)
plt.tight_layout()
plt.savefig("fwl_comparison.png", dpi=300, bbox_inches="tight")
plt.show()

Simpson’s paradox resolved: the naive negative slope (left) reverses to a positive slope (right) after partialling out income.

The contrast is striking. On the left, the naive analysis suggests a negative relationship (slope = −0.106) — coupons appear to hurt sales. On the right, after removing income’s confounding influence, a positive conditional relationship emerges (slope = +0.267, an estimate of the true effect of 0.2). This is a textbook example of Simpson’s paradox: a trend that appears in aggregate data reverses when the data is properly conditioned on a relevant variable.

16. Try it yourself: an interactive FWL lab

Everything so far used one fixed sample. The lab below simulates the same DGP in your browser and lets you change it. Its defaults are the post’s 50 restaurants, so it reproduces the naive and controlled slopes of sections 6–7 exactly before you touch anything.

Interactive FWL lab

Showing the post’s 50 restaurants

Change how strongly income drives coupons and sales, then compare the naive slope in the raw data with the FWL slope after income is partialled out of both variables.

Income → coupons slope $\pi$−0.50

Coupons per extra unit of income. Default −0.50: richer neighborhoods redeem fewer coupons.

Income → sales effect $\beta_2$0.30

Direct effect of income on sales. Default 0.30. Set it to 0 and income no longer confounds the naive slope.

True coupon effect $\beta_1$0.20

Extra sales from one more coupon, the causal effect a good estimator should recover. Default 0.20.

Restaurants per sample50

20, 50, 100, 200, 500 or 1000 restaurants. A new size draws a simulated sample from the same process.

Raw data: the naive slope

Income partialled out: the FWL slope

All three slopes on one scale

Naive OLS slope (raw data)
FWL slope (income partialled out)
True coupon effect $\beta_1$
Restaurants by income third: low, middle, high

True coupon effect $\beta_1$: 0.2000
Naive slopesales on coupons only
FWL slope= coupon coefficient in sales ~ coupons + income
Naive slope in a very large sample$\beta_1 + \beta_2 \cdot 100\pi / (100\pi^2 + 25)$
$\hat\gamma$ income coefficientin sales ~ coupons + income
$\hat\delta$ slope of income on couponsauxiliary regression income ~ coupons
Omitted-variable gap $\hat\gamma\,\hat\delta$= naive − FWL, exactly, in this sample

Sign flip in this sample: yes · in a very large sample: yes

Switch off the income → sales arrow. Set income → sales to 0. The naive slope jumps toward the FWL slope. With no path from income to sales, income stops being a confounder, and the gap that remains is sampling noise — still exactly $\hat\gamma \hat\delta$ (Exercise 3 does the same experiment in code).
Switch off the income → coupons arrow. Reset, then set income → coupons to 0. Income still drives sales, but it no longer moves with coupons, so leaving it out biases nothing on average. At n = 50 the two slopes can still differ by chance; that gap is again exactly $\hat\gamma \hat\delta$, and it shrinks as n grows.
Find the sign-flip window. Reset, then slide income → coupons from 0 toward −1, and watch the “Naive slope in a very large sample” row. In the population the naive slope turns negative once the slope passes about −0.19, and it bottoms out at −0.10 exactly at −0.5, the post’s own value — the worst case. It would turn positive again only past −1.31. The main readout for the post’s own 50 restaurants traces a noisier version of this curve: it flips later and bottoms out at a different slider value.
Grow the sample. Reset, then set n = 1000. The FWL slope tightens around 0.2, while the naive slope settles near −0.10. More data makes the naive answer more precise, not less wrong.
Redraw the restaurants. Return to n = 50 and press “Draw a new sample” a few times. Both slopes jump around. The FWL slope stays centered on 0.2; the naive slope stays centered on −0.10. The post’s −0.1059 and +0.2673 are one draw among many.

The lab turns the OVB identity into something you can feel. Bias needs both arrows: income must move sales and move with coupons. Cut either one and the naive and controlled slopes agree on average. Sampling noise, by contrast, never goes away at n = 50 — it only shrinks as the sample grows. For a companion lab on a full page, open the web app: it draws fresh random samples instead of the post’s 50 restaurants, repeats the experiment 100 times in a Monte Carlo tab, and lines up the post’s estimators in a forest plot.

17. Summary of results

Method	Coupons coefficient	Std. error	p-value	Residual df
Naive OLS (no controls)	−0.1059	0.1158	0.365	48
Full OLS (+ income)	+0.2673	0.1203	0.031	47
FWL Step 1 (residualize coupons only)	+0.2673	1.2715	0.834	49
FWL Step 1 + intercept	+0.2673	0.1437	0.069	48
FWL Step 2 (residualize both)	+0.2673	0.1178	0.028	49
FWL by hand (NumPy)	+0.2673	—	—	—
Full OLS (+ income + day)	+0.2706	0.1194	0.028	46
FWL (+ income + day)	+0.2706	0.1157	0.023	49

All FWL variants produce the same coefficient as the corresponding full regression, confirming the theorem. Within each family (the one-control rows and the two-control rows), the standard errors differ for only two reasons: how much variation is left in the residuals (Step 1 leaves the sales mean and income’s share of sales in them) and the residual degrees of freedom in the last column. Across families, and for the naive row, a third factor enters: the variation left in coupons itself after the controls (the $\tilde x_1^\top \tilde x_1$ of the proof), which sits in the denominator of the standard error. The coefficient of +0.267 is close to the true DGP value of +0.200, with the difference attributable to finite-sample noise in 50 observations.

18. Common misconceptions

FWL is simple enough to state in one sentence, and that makes it easy to over-read. Each card below states a common belief, then checks it against the numbers in this post.

Misconception "FWL only works with one control."

What is actually true. FWL works with any number of controls: partial them all out at once. In section 14, two controls give 0.2706 from the full regression and 0.2706 from the residual-on-residual regression. Fixed-effects software such as reghdfe in Stata and pyfixest in Python applies the same theorem to hundreds or thousands of fixed-effect dummies. Exercise 6 shows the within-group demeaning version on this data.

Misconception "Residualizing only the treatment gives the right standard error."

What is actually true. It gives the right coefficient and the wrong standard error. Step 1 reports 1.2715 instead of 0.1203, which makes the coupon effect look like noise (p = 0.834) — a badly misleading conclusion. Most of the gap is the missing intercept; the rest is income’s variation left in sales (section 10.2). Even after residualizing both variables, a degrees-of-freedom correction of $\sqrt{49/47}$ remains.

Misconception "Residuals are independent of the controls."

What is actually true. OLS residuals are only uncorrelated with the controls, by construction. In this sample the correlation between coupons_tilde and income is numerically zero — floating-point noise far below 1e-10. But a residual can still depend on a control nonlinearly. Exercise 1 builds a curved coupon equation where the linear residuals have zero correlation with income but a correlation of 0.78 with (income − 50)².

Misconception "FWL fixes nonlinear confounding."

What is actually true. Linear partialling-out removes only the linear part of the controls. That is enough when only the treatment equation is nonlinear in income: the outcome equation is still correctly specified, and the coupon coefficient stays unbiased. Bias arises when the outcome equation has a nonlinear term in the controls that the linear control misses, and that term is correlated with the treatment beyond linear income. Exercise 7 shows both cases: 0.200 when only coupons curve in income, 0.127 when sales curves too. The fix is to add the matching terms (here, income² as a control, which restores 0.200) or to let flexible learners do the partialling-out — Double Machine Learning.

Misconception "The controlled coefficient is the true causal effect."

What is actually true. +0.2673 is an estimate of the causal effect. Its 95% confidence interval runs from 0.025 to 0.509, and the true value of 0.2 sits inside it. The estimate has a causal interpretation only because the simulation guarantees no unmeasured confounding: income is the only confounder, and we observe it. In real data that guarantee must be argued, not assumed. FWL is algebra, not identification.

19. Applications of the FWL theorem

The FWL theorem is not just a mathematical curiosity — it has practical applications across several domains.

19.1 Data visualization

As shown above, FWL makes it possible to plot the conditional relationship between two variables after controlling for confounders. This is invaluable when presenting regression results to non-technical audiences who understand scatter plots but not regression tables with multiple coefficients.

19.2 Computational efficiency

When a regression includes high-dimensional fixed effects — for example, year, industry, and country dummies that could add hundreds of columns — computing the full regression becomes expensive. The FWL theorem allows software to partial out these fixed effects first, reducing the problem to a much smaller regression. Widely used packages that exploit this strategy include:

reghdfe in Stata
fixest in R
pyfixest in Python — a fast, user-friendly package for fixed-effects regression (including multi-way clustering and interaction effects), inspired by fixest’s R API
pyhdfe in Python

19.3 Machine learning and causal inference

Perhaps the most impactful modern application is Double Machine Learning (DML), developed by Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey, and Robins (2018). DML keeps the FWL recipe and changes two things. First, it replaces both OLS partialling-out regressions with flexible machine-learning predictions (random forests, lasso, neural networks) of the outcome and of the treatment from the controls. Second, it uses cross-fitting: it splits the data into folds and computes each restaurant’s residuals from models trained on the other folds, so a flexible learner cannot fit a restaurant’s own noise into that restaurant’s residuals. It then runs the same residual-on-residual regression. The target is $\theta$ in the partially linear model $Y = \theta D + g(X_2) + \varepsilon$, where the controls $X_2$ (here, income) can affect the outcome through any function $g$ and can drive the treatment in complex, nonlinear ways. DML recovers $\theta$ when there is no unmeasured confounding, the same assumption OLS needs, and when the learners predict the outcome and the treatment well enough.

DML grew out of earlier work on many controls. Belloni, Chernozhukov, and Hansen (2014) use the lasso twice, once to pick the controls that predict the outcome and once to pick those that predict the treatment, and then run OLS with the union of both sets. Their post-double-selection estimator is partialling-out with a selection step in front, and DML extends the same double-residualization idea to any learner.

If you want to see DML in action, check out the companion tutorial on Introduction to Causal Inference: Double Machine Learning, which applies the partialling-out estimator to a real randomized experiment.

20. Discussion

This tutorial set out to answer a simple question: what does it mean to “control for” a variable in regression, and how can the result be visualized? The FWL theorem provides a definitive answer. Controlling for income in a regression of sales on coupons is equivalent to removing income’s influence from both variables and then regressing the residuals.

In the simulated fast-food scenario, failing to control for income produced a misleading negative coefficient of −0.106, suggesting coupons reduce sales. After partialling out income, the coefficient reversed to +0.267 (p = 0.031), an estimated \$267 in monthly sales per percentage point of redemption. The sign is right, and the true effect, 0.2 (\$200), lies well inside the 95% confidence interval [0.025, 0.509]; the gap between the two is sampling variability in just 50 restaurants. The omitted-variable-bias identity leaves nothing unexplained: income’s coefficient (0.3836) times the slope of income on coupons (−0.9730) equals the −0.3732 gap between the two estimates, to the last digit.

For a practitioner — say, the marketing director of the fast-food chain — the takeaway is clear. An analysis that ignored neighborhood income would conclude the coupon program was counterproductive. The FWL-based analysis shows it works, and provides a plot that makes this case visually compelling. The theorem bridges the gap between the numbers in a regression table and the intuitive two-variable scatter plot.

21. Summary and next steps

Key takeaways:

Sign reversal. The naive coupon coefficient was −0.1059 (negative, not significant). After controlling for income it became +0.2673 (positive, p = 0.031). Ignoring a confounder can reverse the direction of an estimated effect, not just its size.
Exact equivalence. FWL reproduces the full-regression coefficient exactly: 0.2673 with one control, 0.2706 with two. statsmodels, residual-on-residual regressions, and plain NumPy all agree. The theorem is an algebraic identity, not an approximation.
The bias is measurable. The OVB identity accounts for the whole gap between the naive and controlled slopes: −0.3732 = 0.3836 × (−0.9730). It holds exactly in the sample, and its population version (a bias of −0.30, so a naive slope of −0.10) explains why more data would not rescue the naive estimate.
Standard errors need care: the intercept, the leftover income variation, and the df. The coefficient survives every FWL shortcut; the standard error does not. Dropping the intercept inflates it to 1.2715. Adding the intercept back gives 0.1437, still above 0.1203 because income’s variation remains in sales. Residualizing both variables leaves only a degrees-of-freedom gap (0.1178 against 0.1203). Report the full-model SE.
Visualization. FWL reduces any multivariate regression to a scatter plot with one slope. That makes conditional relationships visible to audiences who read plots, not regression tables.
Foundation for DML. Double Machine Learning keeps the residual-on-residual logic but predicts the outcome and the treatment with flexible learners, using cross-fitting so that no restaurant’s residual comes from a model trained on that restaurant. That matters when the outcome depends on the controls in ways a linear control misses, and those terms move with the treatment. A curve in the treatment equation alone is harmless for linear FWL.

Limitations:

The data is simulated from a known linear DGP with a single, observed confounder. In real data the DGP is unknown, and “no unmeasured confounding” must be defended rather than assumed.
Linear partialling-out removes only the linear part of the controls. It stays unbiased when only the treatment equation is nonlinear in the controls. It fails when the outcome equation contains a nonlinear function of the controls that the linear control misses, and that function is correlated with the treatment beyond linear income (Exercise 7). Adding the matching terms or using flexible learners (DML) fixes it.
With only 50 observations, the confidence interval is wide (0.025 to 0.509). Larger samples sharpen the estimate (Exercise 4).

Next steps and related tutorials.

Visualizing Regression with the FWL Theorem in R — the fwlplot package draws a residualized scatter in one line and extends the idea to fixed effects with real data. It uses a different simulated sample of 200 stores, so its coefficients differ from this post’s.
Visualizing Regression with the FWL Theorem in Stata — the same recipe with scatterfit and reghdfe, on the same 200-store sample as the R edition.
What Does TWFE Actually Do? Manual Demeaning and the FWL Theorem — two-way fixed effects as FWL on demeaned data, the natural sequel to Exercise 6.
High-Dimensional Fixed Effects Regression: An Introduction in Python — pyfixest, the Python software that applies FWL to thousands of fixed effects.
Introduction to Causal Inference: Double Machine Learning — FWL with machine-learning residualization and cross-fitting, the fix for the nonlinear case in Exercise 7.
Introduction to Causal Inference: The DoWhy Approach with the Lalonde Dataset — the full identify-estimate-refute workflow on real data, where the no-unmeasured-confounding assumption has to be argued.

22. Exercises

The exercises below reuse the objects built in the post (df, naive_model, full_model, fwl_step2, and so on), so run them after the main code. Each one comes with a collapsible solution: try it first, then open the card to compare code and numbers. The solutions never modify df in place.

22.1 Warm-up

Exercise 1 — Uncorrelated is not independent. Check that coupons_tilde has mean zero and zero correlation with income in the post’s data. Then simulate 500 restaurants (seed 42) with a curved coupon equation, coupons = 60 - 0.5 * income + 0.05 * (income - 50)**2 + noise, residualize coupons on income linearly, and correlate the residuals with income and with (income − 50)². What do you conclude?

Solution Show the code and the numbers

# Floating-point noise differs by machine, so "zero" means below a tolerance
tol = 1e-10
# (a) The post's residuals: mean zero and uncorrelated with income
print("mean(coupons_tilde) is zero: ", abs(df["coupons_tilde"].mean()) < tol)
print("corr(coupons_tilde, income) is zero:", abs(np.corrcoef(df["coupons_tilde"], df["income"])[0, 1]) < tol)
# (b) A curved coupon equation: linear residuals are uncorrelated, not independent
rng = np.random.default_rng(42)
inc = rng.normal(50, 10, 500)
cpn = 60 - 0.5 * inc + 0.05 * (inc - 50) ** 2 + rng.normal(0, 5, 500)
res = smf.ols("coupons ~ income", pd.DataFrame({"coupons": cpn, "income": inc})).fit().resid
print("corr(resid, income) is zero: ", abs(np.corrcoef(res, inc)[0, 1]) < tol)
print(f"corr(resid, (income - 50)^2) = {np.corrcoef(res, (inc - 50) ** 2)[0, 1]:.4f}")

mean(coupons_tilde) is zero: True
corr(coupons_tilde, income) is zero: True
corr(resid, income) is zero: True
corr(resid, (income - 50)^2) = 0.7818

The post’s residuals have mean zero and zero correlation with income, up to floating-point error (below 1e-14 here; the exact digits vary by machine). That is what OLS guarantees. In the curved design, the linear residuals are still exactly uncorrelated with income, yet their correlation with (income − 50)² is 0.7818. The residuals clearly depend on income; they just do not depend on it linearly. Uncorrelated is not independent.

Exercise 2 — Decompose the dayofweek shift. Adding dayofweek moved the coupon coefficient from 0.2673 to 0.2706. Show that 0.2673 = 0.2706 + $\hat\gamma_{dow} \cdot \hat\delta$, where $\hat\gamma_{dow}$ is dayofweek’s coefficient in full_model_2 and $\hat\delta$ is the coupons coefficient in dayofweek ~ coupons + income. Report the partial correlation of dayofweek and coupons given income.

Solution Show the code and the numbers

gamma_dow = full_model_2.params["dayofweek"]
delta_dow = smf.ols("dayofweek ~ coupons + income", df).fit().params["coupons"]
b_one, b_two = full_model.params["coupons"], full_model_2.params["coupons"]
print(f"one control: {b_one:.4f} two controls: {b_two:.4f} shift: {b_two - b_one:+.4f}")
print(f"gamma_dow x delta = {gamma_dow:.4f} x {delta_dow:.4f} = {gamma_dow * delta_dow:+.4f}")
print("identity holds: ", abs(b_one - (b_two + gamma_dow * delta_dow)) < 1e-10)
dow_tilde = smf.ols("dayofweek ~ income", df).fit().resid
print(f"partial corr(dayofweek, coupons | income) = {np.corrcoef(dow_tilde, df['coupons_tilde'])[0, 1]:.3f}")
print(f"raw corr(dayofweek, coupons) = {np.corrcoef(df['dayofweek'], df['coupons'])[0, 1]:.3f}")

one control: 0.2673 two controls: 0.2706 shift: +0.0032
gamma_dow x delta = 0.3195 x -0.0101 = -0.0032
identity holds: True
partial corr(dayofweek, coupons | income) = -0.021
raw corr(dayofweek, coupons) = -0.076

The identity closes to floating-point precision. Holding income fixed, dayofweek has a tiny chance association with coupons ($\hat\delta = -0.0101$, partial correlation −0.021) and a coefficient of 0.3195 on sales. Their product, −0.0032, is exactly the shift. The raw correlation of −0.076 is the wrong quantity: what matters is the association left over after income is partialled out.

Exercise 3 — Switch off the confounding. Rewrite the simulator so the two income arrows are arguments. Then (a) set income → sales to 0, and separately (b) set income → coupons to 0. Compare the naive and FWL slopes with those of the default design, at n = 50 and at n = 10,000. Is the gap between them still equal to $\hat\gamma \hat\delta$?

Solution Show the code and the numbers

def simulate_switch(n=50, seed=42, income_to_coupons=-0.5, income_to_sales=0.3):
"""The post's DGP with the two income arrows exposed as arguments."""
rng = np.random.default_rng(seed)
income = rng.normal(50, 10, n)
dayofweek = rng.integers(1, 8, n)
coupons = 60 + income_to_coupons * income + rng.normal(0, 5, n)
sales = (10 + 0.2 * coupons + income_to_sales * income
+ 0.5 * dayofweek + rng.normal(0, 3, n))
return pd.DataFrame({"sales": sales, "coupons": coupons, "income": income})
designs = [("both arrows on", {}),
("(a) income -> sales = 0", {"income_to_sales": 0.0}),
("(b) income -> coupons = 0", {"income_to_coupons": 0.0})]
for n in [50, 10000]:
for label, switch in designs:
d = simulate_switch(n=n, **switch)
naive = smf.ols("sales ~ coupons", d).fit().params["coupons"]
full = smf.ols("sales ~ coupons + income", d).fit()
gamma = full.params["income"]
delta = smf.ols("income ~ coupons", d).fit().params["coupons"]
exact = np.isclose(naive - full.params["coupons"], gamma * delta)
print(f"n = {n:<6} {label:<26} naive {naive:+.3f} FWL {full.params['coupons']:.4f} "
f"gamma {gamma:+.3f} delta {delta:+.3f} gap = gamma x delta: {exact}")

n = 50 both arrows on naive -0.106 FWL 0.2673 gamma +0.384 delta -0.973 gap = gamma x delta: True
n = 50 (a) income -> sales = 0 naive +0.186 FWL 0.2673 gamma +0.084 delta -0.973 gap = gamma x delta: True
n = 50 (b) income -> coupons = 0 naive +0.410 FWL 0.2673 gamma +0.350 delta +0.408 gap = gamma x delta: True
n = 10000 both arrows on naive -0.102 FWL 0.2021 gamma +0.302 delta -1.007 gap = gamma x delta: True
n = 10000 (a) income -> sales = 0 naive +0.200 FWL 0.2021 gamma +0.002 delta -1.007 gap = gamma x delta: True
n = 10000 (b) income -> coupons = 0 naive +0.202 FWL 0.2021 gamma +0.301 delta +0.001 gap = gamma x delta: True

The FWL slope does not move at all: 0.2673 in all three designs at n = 50, and 0.2021 at n = 10,000. Switching off a linear income arrow changes sales or coupons only by a linear function of income, which partialling-out removes exactly. (The simulator skips the post’s rounding to two decimals; with rounding, the three FWL values would differ in the fourth decimal.) The naive slope moves instead, from −0.106 to 0.186 in (a) and 0.410 in (b). The gap is still exactly $\hat\gamma \hat\delta$, but at n = 50 it is sampling noise, not confounding: in (a) $\hat\gamma$ is a chance 0.084 rather than 0, and in (b) $\hat\delta$ is a chance 0.408. At n = 10,000 the noise is gone. One of the two factors collapses to about zero, and the naive slope lands at 0.200 or 0.202, next to the FWL slope. With both arrows on, the naive slope stays at −0.102, near its population value of −0.10. Bias needs both arrows; noise needs only a small sample.

22.2 Core

Exercise 4 — Sample size sensitivity. Re-run the naive and controlled regressions with simulate_store_data() at n = 50, 500, 5,000, and 50,000. How do the two coefficients change? How fast do the standard errors shrink? Is the naive estimate still misleading with a large sample?

Solution Show the code and the numbers

for n in [50, 500, 5000, 50000]:
d = simulate_store_data(n=n, seed=RANDOM_SEED)
nv = smf.ols("sales ~ coupons", d).fit()
fl = smf.ols("sales ~ coupons + income", d).fit()
print(f"n = {n:>6,}: naive {nv.params['coupons']:+.4f} (SE {nv.bse['coupons']:.4f}) "
f"FWL {fl.params['coupons']:+.4f} (SE {fl.bse['coupons']:.4f})")

n = 50: naive -0.1059 (SE 0.1158) FWL +0.2673 (SE 0.1203)
n = 500: naive -0.0725 (SE 0.0258) FWL +0.2073 (SE 0.0296)
n = 5,000: naive -0.0876 (SE 0.0075) FWL +0.2178 (SE 0.0090)
n = 50,000: naive -0.0973 (SE 0.0024) FWL +0.1999 (SE 0.0028)

The controlled slope converges on the truth, from +0.2673 at n = 50 to +0.1999 at n = 50,000. The naive slope converges too — to the wrong number. It settles near its population value of −0.10 (−0.0973 at n = 50,000) with ever-smaller standard errors. From n = 500 onward, each tenfold increase shrinks the SEs by about $\sqrt{10} \approx 3.2$ (0.0296 → 0.0090 → 0.0028), the familiar $1/\sqrt{n}$ rate. At n = 5,000 both estimates still sit up to two standard errors from their population values; one sample is one draw. More data buys precision, not validity: at n = 50,000 the naive estimate is precisely wrong.

Exercise 5 — Standard errors that survive FWL. Rescale Step 2’s standard error by $\sqrt{49/47}$ and compare it with the full model’s. Then compute heteroskedasticity-robust standard errors (cov_type="HC0" through "HC3") for the full regression and for Step 2. Which ones match, which differ by the same $\sqrt{49/47}$ factor, and which differ for another reason?

Solution Show the code and the numbers

print(f"Step 2 SE x sqrt(49/47) = {fwl_step2.bse.iloc[0] * np.sqrt(49 / 47):.4f}")
print(f"Full-model SE = {full_model.bse['coupons']:.4f}")
print(f"sqrt(49/47) = {np.sqrt(49 / 47):.4f}")
for hc in ["HC0", "HC1", "HC2", "HC3"]:
se_full = smf.ols("sales ~ coupons + income", df).fit(cov_type=hc).bse["coupons"]
se_fwl = smf.ols("sales_tilde ~ coupons_tilde - 1", df).fit(cov_type=hc).bse["coupons_tilde"]
print(f"{hc}: full {se_full:.4f} Step 2 {se_fwl:.4f} ratio {se_full / se_fwl:.4f}")

Step 2 SE x sqrt(49/47) = 0.1203
Full-model SE = 0.1203
sqrt(49/47) = 1.0211
HC0: full 0.0955 Step 2 0.0955 ratio 1.0000
HC1: full 0.0985 Step 2 0.0965 ratio 1.0211
HC2: full 0.0999 Step 2 0.0979 ratio 1.0199
HC3: full 0.1045 Step 2 0.1005 ratio 1.0405

Rescaling by $\sqrt{49/47} = 1.0211$ recovers the full-model 0.1203 exactly. The robust SEs split three ways. HC0 matches exactly (0.0955 both ways), because it uses the raw residuals, which FWL reproduces, with no degrees-of-freedom factor. HC1 rescales HC0 by $\sqrt{n/(n-k)}$, so it differs by the same $\sqrt{49/47}$ (0.0985 against 0.0965). HC2 and HC3 do not match by any fixed factor (ratios 1.0199 and 1.0405). They reweight each residual by its leverage $h_{ii}$, a number between 0 and 1 that measures how unusual a restaurant’s regressor values are, and the leverages of the full regression, which include income and the intercept, differ from those of the one-regressor Step 2 regression. If you need HC2 or HC3, compute them from the full model.

Exercise 6 — Fixed effects are FWL too. Treat dayofweek as a categorical control with C(dayofweek), which adds a separate intercept for each day. Then reproduce the coupon coefficient without any dummies: subtract each day’s mean from sales, coupons, and income, and regress the demeaned sales on the demeaned coupons and income with no intercept.

Solution Show the code and the numbers

fe = smf.ols("sales ~ coupons + income + C(dayofweek)", df).fit()
cols = ["sales", "coupons", "income"]
within = df[cols] - df.groupby("dayofweek")[cols].transform("mean")
fe_fwl = smf.ols("sales ~ coupons + income - 1", within).fit()
print(f"Day dummies, C(dayofweek): {fe.params['coupons']:.4f}")
print(f"Within-day demeaning: {fe_fwl.params['coupons']:.4f}")
print("Identical: ", abs(fe.params["coupons"] - fe_fwl.params["coupons"]) < 1e-10)

Day dummies, C(dayofweek): 0.2326
Within-day demeaning: 0.2326
Identical: True

The two coefficients agree to floating-point precision (0.2326 both ways). A regression with one intercept per day is FWL with the day dummies as the partialled-out controls, and residualizing on a full set of group dummies is the same as subtracting group means. The number differs from section 14’s 0.2706 because C(dayofweek) gives each day its own level instead of forcing a straight line in dayofweek. This is exactly how reghdfe, fixest, and pyfixest handle thousands of fixed effects: they demean instead of building dummies.

22.3 Stretch

Exercise 7 — Nonlinear confounding. Make income enter the coupon equation through a square: coupons = 60 - 0.01 * income**2 + noise. (a) Keep sales linear in income and estimate the coupon effect with the post’s linear control. (b) Add 0.01 * income**2 to the sales equation as well and estimate again. (c) Add I(income**2) as a control. Use n = 5,000 and average over 20 fixed seeds so sampling noise does not blur the answer.

Solution Show the code and the numbers

def simulate_nonlinear(n=5000, seed=42, curve_in_sales=0.0):
"""Income enters coupons through income**2, and optionally sales too."""
rng = np.random.default_rng(seed)
income = rng.normal(50, 10, n)
dayofweek = rng.integers(1, 8, n)
coupons = 60 - 0.01 * income**2 + rng.normal(0, 5, n)
sales = (10 + 0.2 * coupons + 0.3 * income + curve_in_sales * income**2
+ 0.5 * dayofweek + rng.normal(0, 3, n))
return pd.DataFrame({"sales": sales, "coupons": coupons, "income": income})
designs = [("(a) curve in coupons only, linear control", 0.00, "sales ~ coupons + income"),
("(b) curve in both, linear control", 0.01, "sales ~ coupons + income"),
("(c) curve in both, + income^2 control", 0.01,
"sales ~ coupons + income + I(income**2)")]
for label, curve, formula in designs:
est = [smf.ols(formula, simulate_nonlinear(seed=s, curve_in_sales=curve)).fit()
.params["coupons"] for s in range(20)]
print(f"{label:<42} mean {np.mean(est):.3f} SD {np.std(est, ddof=1):.3f}")

(a) curve in coupons only, linear control mean 0.200 SD 0.008
(b) curve in both, linear control mean 0.127 SD 0.010
(c) curve in both, + income^2 control mean 0.200 SD 0.008

(a) With the square only in the coupon equation, the linear control still recovers the truth: a mean of 0.200 across 20 samples. The outcome equation is linear in coupons and income, so it is correctly specified, and a curve in how coupons are assigned does no harm. (b) Once the square also enters sales, the linear control misses a term that moves with coupons beyond linear income, and the estimate drops to 0.127. That bias of about −0.07 is many times the spread across seeds (SD 0.010), so it is not noise, and more data would not fix it. (c) Adding I(income**2) as a control removes the bias: 0.200 again. When you do not know which terms matter, Double Machine Learning lets flexible learners find them.

Exercise 8 — Real data. The classic wage-education-ability example: does education’s return shrink once you control for ability? Load wage2 from the wooldridge package (install it first with pip install wooldridge if it is missing; the Colab notebook’s setup cell does this for you). Regress lwage on educ, then on educ and IQ, verify the FWL identity by residualizing on IQ, and plot the naive and conditional relationships side by side.

Solution Show the code and the numbers

import wooldridge
wage = wooldridge.data("wage2")[["lwage", "educ", "IQ"]].dropna()
naive_w = smf.ols("lwage ~ educ", wage).fit()
full_w = smf.ols("lwage ~ educ + IQ", wage).fit()
wage["educ_tilde"] = smf.ols("educ ~ IQ", wage).fit().resid
wage["lwage_tilde"] = smf.ols("lwage ~ IQ", wage).fit().resid
fwl_w = smf.ols("lwage_tilde ~ educ_tilde - 1", wage).fit()
delta_w = smf.ols("IQ ~ educ", wage).fit().params["educ"]
print(f"n = {len(wage)} workers")
print(f"Naive, lwage ~ educ: {naive_w.params['educ']:.4f}")
print(f"Full, lwage ~ educ + IQ: {full_w.params['educ']:.4f}")
print(f"FWL, residual on residual: {fwl_w.params['educ_tilde']:.4f}")
print(f"OVB: {full_w.params['IQ']:.5f} x {delta_w:.4f} = {full_w.params['IQ'] * delta_w:.4f}"
f" vs naive - full = {naive_w.params['educ'] - full_w.params['educ']:.4f}")
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
sns.regplot(x="educ", y="lwage", data=wage, ci=None, ax=axes[0],
scatter_kws={"color": STEEL_BLUE, "alpha": 0.4, "s": 20},
line_kws={"color": WARM_ORANGE, "linewidth": 2})
axes[0].set_title(f"Naive: slope {naive_w.params['educ']:.3f}")
sns.regplot(x="educ_tilde", y="lwage_tilde", data=wage, ci=None, ax=axes[1],
scatter_kws={"color": TEAL, "alpha": 0.4, "s": 20},
line_kws={"color": WARM_ORANGE, "linewidth": 2})
axes[1].set_title(f"After partialling-out IQ: slope {fwl_w.params['educ_tilde']:.3f}")
plt.tight_layout()
plt.show()

n = 935 workers
Naive, lwage ~ educ: 0.0598
Full, lwage ~ educ + IQ: 0.0391
FWL, residual on residual: 0.0391
OVB: 0.00586 x 3.5338 = 0.0207 vs naive - full = 0.0207

Controlling for IQ cuts the estimated return to a year of education from 0.0598 to 0.0391 log points, about a third lower, and the residual-on-residual regression reproduces 0.0391 exactly. The OVB identity accounts for the difference: IQ’s coefficient times the slope of IQ on education, 0.00586 × 3.5338 = 0.0207, equals the gap. More-educated workers have higher measured ability, and ability raises wages, so the naive slope overstates the return to schooling. The right panel shows what “controlling for IQ” looks like, just as the restaurant plots did for income.

23. Appendix: FWL with panel data

Everything so far used one month of data. The chain did not stop there: in June it ran the same promotion again. Each of the 50 restaurants handed out another 100 coupons on a single day and counted how many came back during the month. Stacking the two months gives a panel: the same 50 restaurants observed twice, 100 rows in all. This appendix shows that the workhorse panel estimators — restaurant fixed effects, two-way fixed effects, and first differences — are all FWL in disguise. It uses analyze_fwl_plot from the expdpy package to partial out the fixed effects and draw the residual-on-residual plot in one call.

23.1 January and June

January is the main-body sample, value for value. June adds three things, each chosen so that a different panel tool has a job to do:

A summer lift. Every restaurant sells \$3,000 more per month in June, whatever its coupons. A month effect should absorb it.
A small income change. Neighborhood income moves a little between the two months ($\text{income}_{\text{June}} = \text{income}_{\text{Jan}} + N(0, 2)$), so income is still worth controlling for.
A persistent restaurant trait. Some restaurants simply sit on busier corners. The trait raises sales in both months by the same amount, and nobody records it. In January the coupons were mailed door to door, so the trait had nothing to do with redemptions. In June they were handed out at the counter, so busier restaurants put them into the hands of customers who come back anyway: the trait now raises the redemption rate too. That makes it a confounder that no data column captures.

The true coupon effect is still exactly +0.2 in both months. The June equations are

$$\text{coupons}_{i,\text{Jun}} = 60 - 0.5\,\text{income}_{i,\text{Jun}} + \alpha_i + u_i, \qquad u_i \sim N(0, 4^2)$$

$$\text{sales}_{i,\text{Jun}} = 13 + 0.2\,\text{coupons}_{i,\text{Jun}} + 0.3\,\text{income}_{i,\text{Jun}} + 0.5\,\text{dayofweek}_{i,\text{Jun}} + \alpha_i + e_i, \qquad e_i \sim N(0, 1.5^2)$$

where $\alpha_i \sim N(0, 3^2)$ is the restaurant trait. In words: June redemption depends on income and on the trait, and June sales carry the summer lift (13 instead of 10) plus the same trait. The trick that keeps January untouched is that $\alpha_i$ is January’s sales noise: the function below draws January exactly as simulate_store_data() does, keeps that sales noise as the trait, and then continues the same random stream for June.

def simulate_restaurant_panel(n=50, seed=42, season=3.0, income_shock_sd=2.0,
trait_to_coupons=1.0):
"""January (period 1) replays simulate_store_data(); June (period 2) follows."""
rng = np.random.default_rng(seed)
# January: the same draws, in the same order, as simulate_store_data()
income = rng.normal(50, 10, n)
dayofweek = rng.integers(1, 8, n)
coupons = 60 - 0.5 * income + rng.normal(0, 5, n)
trait = rng.normal(0, 3, n) # January's sales noise, now a lasting trait
sales = 10 + 0.2 * coupons + 0.3 * income + 0.5 * dayofweek + trait
# June: the same random stream continues
income_j = income + rng.normal(0, income_shock_sd, n)
dayofweek_j = rng.integers(1, 8, n)
coupons_j = (60 - 0.5 * income_j + trait_to_coupons * trait
+ rng.normal(0, 4, n))
sales_j = (10 + season + 0.2 * coupons_j + 0.3 * income_j
+ 0.5 * dayofweek_j + trait + rng.normal(0, 1.5, n))
def month(period, s, c, i, d):
return pd.DataFrame({"restaurant_id": np.arange(1, n + 1), "period": period,
"sales": np.round(s, 2), "coupons": np.round(c, 2),
"income": np.round(i, 2), "dayofweek": d})
return pd.concat([month(1, sales, coupons, income, dayofweek),
month(2, sales_j, coupons_j, income_j, dayofweek_j)],
ignore_index=True)
panel = simulate_restaurant_panel(n=N, seed=RANDOM_SEED)
cols4 = ["sales", "coupons", "income", "dayofweek"]
january = panel.loc[panel["period"] == 1, cols4].reset_index(drop=True)
print("Panel shape:", panel.shape)
print("January equals the main-body data:", january.equals(df[cols4]))
print()
print(panel.sort_values(["restaurant_id", "period"]).head(6).to_string(index=False))
print()
print(panel.groupby("period")[["sales", "coupons", "income"]].mean().round(2))

Panel shape: (100, 6)
January equals the main-body data: True
restaurant_id period sales coupons income dayofweek
1 1 37.37 36.93 53.05 6
1 2 40.87 41.72 53.39 6
2 1 36.88 38.06 39.60 6
2 2 40.01 54.63 42.76 3
3 1 33.09 32.04 57.50 6
3 2 32.34 22.84 57.82 1
sales coupons income
period
1 33.61 33.84 50.91
2 37.07 34.50 51.05

The first check matters most: the January half of the panel is the main-body data, so nothing in sections 5–22 changes. Individual restaurants moved a lot — restaurant 2’s redemption rate jumped from 38% to 55% — but on average, June sales are \$3,460 higher, while redemption and income barely moved.

23.2 Pooled OLS is fooled by the trait

The simplest panel regression ignores the panel structure: stack the 100 rows and run OLS with income and a June dummy as controls. The standard errors are clustered by restaurant, because the two rows of the same restaurant are not independent draws.

Predict first

Pooled OLS controls for income and for the month. The restaurant trait is left out, it raises sales, and in June it raises redemption. Will the pooled coupon coefficient land near 0.2, above it, or below it? Commit to an answer before scrolling.

Reveal the answer

Answer. Well above it: +0.4496. The omitted trait raises sales and moves with coupons, so by the OVB logic of section 7.2 the bias is positive. This time the confounder is not a column you can add to the regression.

CLUSTER = {"cov_type": "cluster", "cov_kwds": {"groups": panel["restaurant_id"]}}
pooled = smf.ols("sales ~ coupons + income + C(period)", panel).fit(**CLUSTER)
print(pooled.summary().tables[1])

==================================================================================
coef std err z P>|z| [0.025 0.975]
----------------------------------------------------------------------------------
Intercept -7.2932 6.540 -1.115 0.265 -20.112 5.525
C(period)[T.2] 3.1031 0.384 8.083 0.000 2.351 3.856
coupons 0.4496 0.081 5.525 0.000 0.290 0.609
income 0.5044 0.085 5.953 0.000 0.338 0.671
==================================================================================

The pooled coupon coefficient is +0.4496 (clustered SE 0.081), more than twice the true 0.2, and its 95% confidence interval [0.290, 0.609] excludes the truth. The June dummy (+3.10) finds the summer lift, but income’s coefficient (+0.50 against a true 0.3) is contaminated too. Controlling for everything you observe is not enough when the confounder is unobserved. What saves the analysis is that the trait does not change between January and June.

23.3 Restaurant fixed effects are FWL

A restaurant fixed effect gives every restaurant its own intercept, which soaks up anything about the restaurant that does not change over time — the busy corner included. There are three ways to compute it, and FWL says they must agree:

Dummies (LSDV). Add 49 restaurant dummies to the regression.
Demeaning. Subtract each restaurant’s own two-month mean from sales, coupons, and income, then regress. Residualizing on a full set of group dummies is subtracting group means (Exercise 6 showed this for days of the week).
analyze_fwl_plot. Partial the restaurant effects and income out of both sales and coupons, regress residual on residual, and plot the result.

Install the package first if you do not have it: pip install expdpy.

import expdpy as ex
# Route 1: one dummy per restaurant (least-squares dummy variables, LSDV)
fe_lsdv = smf.ols("sales ~ coupons + income + C(restaurant_id)", panel).fit(**CLUSTER)
# Route 2: subtract each restaurant's own two-month mean, then regress
cols = ["sales", "coupons", "income"]
within = panel[cols] - panel.groupby("restaurant_id")[cols].transform("mean")
fe_demeaned = smf.ols("sales ~ coupons + income - 1", within).fit()
# Route 3: expdpy partials out income and the restaurant effects, then plots
fwl_fe = ex.analyze_fwl_plot(panel, dv="sales", var="coupons", controls="income",
feffects="restaurant_id", clusters="restaurant_id",
n_sample=None)
print(f"LSDV, C(restaurant_id): {fe_lsdv.params['coupons']:.4f}")
print(f"Demeaned within restaurant: {fe_demeaned.params['coupons']:.4f}")
print(f"analyze_fwl_plot: {fwl_fe.slope:.4f} (clustered SE {fwl_fe.se:.4f}, "
f"n = {fwl_fe.n_obs}, within R2 = {fwl_fe.r2_within:.3f})")

LSDV, C(restaurant_id): 0.2138
Demeaned within restaurant: 0.2138
analyze_fwl_plot: 0.2138 (clustered SE 0.0807, n = 100, within R2 = 0.133)

All three routes return 0.2138 — the same coefficient to machine precision, exactly as the theorem promises. The persistent trait is gone: a restaurant-level constant disappears the moment each restaurant is compared with itself. analyze_fwl_plot also returns the residual frame (fwl_fe.df) and a Plotly figure (fwl_fe.fig.show()). The figure below is that figure, restyled to match the post. Hover over a point to see which restaurant and month it is.

Restaurant fixed effects: after partialling out income and each restaurant’s own average, the slope of residual sales on residual redemption is 0.2138. Blue points are January, teal points are June.

Look at the plot’s structure: with two months per restaurant, each restaurant contributes a mirror-image pair of points, one on each side of zero. After demeaning, a restaurant’s January and June residuals are equal and opposite. The only thing left to explain is how each restaurant changed between the months.

23.4 Two-way fixed effects: restaurant and month

One-way restaurant effects still leave out the summer lift. Every restaurant sold about \$3,000 more in June, and the average redemption rate also edged up (33.84% to 34.50%). Without a month effect, part of the summer lift gets credited to that small June rise in redemptions. Two-way fixed effects (TWFE) add a month effect alongside the restaurant effects. For FWL that is just one more set of dummies to partial out.

twfe_lsdv = smf.ols("sales ~ coupons + income + C(restaurant_id) + C(period)",
panel).fit(**CLUSTER)
fwl_twfe = ex.analyze_fwl_plot(panel, dv="sales", var="coupons", controls="income",
feffects=["restaurant_id", "period"],
clusters="restaurant_id", n_sample=None)
print(f"LSDV, C(restaurant_id) + C(period): {twfe_lsdv.params['coupons']:.4f}")
print(f"analyze_fwl_plot: {fwl_twfe.slope:.4f} (clustered SE {fwl_twfe.se:.4f}, "
f"within R2 = {fwl_twfe.r2_within:.3f})")

LSDV, C(restaurant_id) + C(period): 0.1394
analyze_fwl_plot: 0.1394 (clustered SE 0.0691, within R2 = 0.202)

The dummy regression and analyze_fwl_plot agree again: 0.1394. Partialling out the restaurant and month effects, plus income, from both variables and regressing residual on residual reproduces the coefficient from the full dummy regression, which is FWL with two sets of fixed effects. The estimate moved from 0.2138 to 0.1394 because the month effect no longer lets the summer lift ride on the small June rise in redemptions.

Two-way fixed effects: the residual-on-residual slope is 0.1394, identical to the regression with restaurant and month dummies.

Is 0.1394 a problem, when the truth is 0.2? A normal-approximation 95% confidence interval from the clustered SE (0.1394 ± 1.96 × 0.0691) runs from about 0.004 to 0.275 and contains 0.2. Section 23.7 checks the question properly with many simulated panels.

23.5 First differences: the T = 2 shortcut

With exactly two periods there is an even simpler route. Subtract each restaurant’s January row from its June row. The trait, which is the same in both months, cancels. What remains is one row per restaurant: the change in sales, the change in redemption, and the change in income. The intercept of that regression picks up the change common to all restaurants — the summer lift.

wide = panel.pivot(index="restaurant_id", columns="period", values=cols)
fd = pd.DataFrame({f"d_{c}": wide[(c, 2)] - wide[(c, 1)] for c in cols})
fd_model = smf.ols("d_sales ~ d_coupons + d_income", fd).fit(cov_type="HC1")
print(fd_model.summary().tables[1])
print("FD slope equals two-way FE:", abs(fd_model.params["d_coupons"] - fwl_twfe.slope) < 1e-10)

==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 3.3165 0.313 10.601 0.000 2.703 3.930
d_coupons 0.1394 0.070 2.006 0.045 0.003 0.276
d_income 0.4218 0.192 2.195 0.028 0.045 0.798
==============================================================================
FD slope equals two-way FE: True

The first-difference slope is 0.1394, identical to two-way fixed effects. This is not a coincidence. With two periods, demeaning within a restaurant turns each row into plus or minus half the June–January difference, so the demeaned regression and the differenced regression are the same regression up to a factor of ½ on both sides, which cancels in the slope. The period dummy in TWFE plays the role of the intercept in first differences: 3.32 here (SE 0.31), an estimate of the true summer lift of 3. First differences are FWL too: differencing is one more way of partialling out the restaurant effects.

23.6 Which standard error?

The coefficients agree across every route; the standard errors, as in section 10, need care. Four versions of the TWFE standard error are on the table:

import pyfixest as pf
twfe_raw = pf.feols("sales ~ coupons + income | restaurant_id + period", data=panel,
vcov={"CRV1": "restaurant_id"},
ssc=pf.ssc(k_adj=False, G_adj=False)) # no small-sample factors
fd_hc0 = smf.ols("d_sales ~ d_coupons + d_income", fd).fit(cov_type="HC0")
print(f"Two-way FE, cluster SE with no small-sample factors: {twfe_raw.se()['coupons']:.4f}")
print(f"First differences, HC0 robust SE: {fd_hc0.bse['d_coupons']:.4f}")
print(f"analyze_fwl_plot (pyfixest defaults): {fwl_twfe.se:.4f}")
print(f"statsmodels LSDV, cluster SE counting every dummy: {twfe_lsdv.bse['coupons']:.4f}")

Two-way FE, cluster SE with no small-sample factors: 0.0674
First differences, HC0 robust SE: 0.0674
analyze_fwl_plot (pyfixest defaults): 0.0691
statsmodels LSDV, cluster SE counting every dummy: 0.0988

Stripped of small-sample corrections, the clustered TWFE standard error and the robust first-difference standard error are identical (0.0674). That is the same algebra as the coefficients: with T = 2, each restaurant’s cluster is its first difference. The other two numbers apply different small-sample factors to that core:

analyze_fwl_plot uses pyfixest’s defaults (the fixest conventions): a factor $G/(G-1)$ for the 50 clusters and $(N-1)/(N-K)$ for the parameters, where $K$ does not count the restaurant effects, because they are nested inside the restaurant clusters. Result: 0.0691, and the robust first-difference SE with HC1 is almost the same (0.0695).
statsmodels does not know that the 49 restaurant dummies are fixed effects nested in the clusters. It counts all 53 parameters in $K$, which divides by 100 − 53 = 47 instead of about 96 and inflates the SE to 0.0988.

This mirrors the Step 2 lesson of section 10.3: when parameters are partialled out, the software must count them correctly. With fixed effects nested in the clusters, the fixest convention is standard, so report 0.0691 (or the equivalent first-difference SE).

23.7 Is 0.1394 bad luck?

One panel is one draw. Re-simulating the whole design 500 times shows what each estimator does on average:

def pooled_and_twfe(seed):
"""Pooled OLS slope, two-way FE slope (via first differences), and its SE."""
p = simulate_restaurant_panel(n=N, seed=seed)
b_pooled = smf.ols("sales ~ coupons + income + C(period)", p).fit().params["coupons"]
w = p.pivot(index="restaurant_id", columns="period", values=cols)
d = pd.DataFrame({f"d_{c}": w[(c, 2)] - w[(c, 1)] for c in cols})
m = smf.ols("d_sales ~ d_coupons + d_income", d).fit(cov_type="HC1")
return b_pooled, m.params["d_coupons"], m.bse["d_coupons"]
draws = np.array([pooled_and_twfe(seed) for seed in range(1000, 1500)])
print(f"500 panels pooled OLS: mean {draws[:, 0].mean():.3f} SD {draws[:, 0].std(ddof=1):.3f}")
print(f" two-way FE: mean {draws[:, 1].mean():.3f} SD {draws[:, 1].std(ddof=1):.3f}"
f" average SE {draws[:, 2].mean():.3f}")
print(f"Share of two-way FE draws at or below 0.1394: {(draws[:, 1] <= 0.1394).mean():.2f}")
print(f"SD of d_coupons here: {fd['d_coupons'].std():.2f} (population: {np.sqrt(0.25 * 4 + 9 + 16 + 25):.2f})")

500 panels pooled OLS: mean 0.376 SD 0.072
two-way FE: mean 0.203 SD 0.043 average SE 0.042
Share of two-way FE draws at or below 0.1394: 0.07
SD of d_coupons here: 5.59 (population: 7.14)

Across 500 panels, pooled OLS centers on 0.376 — biased by the trait every time, not just in our sample. Two-way fixed effects center on 0.203, the truth up to simulation noise, with a spread of 0.043 that matches the average reported standard error (0.042). Our panel’s 0.1394 sits in the lower tail: 7% of panels land that low. This panel also has less within-restaurant movement in redemption than usual (the SD of the change is 5.59, against 7.14 in the population), which is why its own standard error, 0.069, is larger than the typical 0.042.

23.8 Takeaways

Estimator	Coupons coefficient	Clustered SE	What it partials out
Pooled OLS (+ income + month)	+0.4496	0.0814	income, month — not the trait
Restaurant FE (LSDV = demeaning = `analyze_fwl_plot`)	+0.2138	0.0807	income, restaurant
Two-way FE (LSDV = `analyze_fwl_plot`)	+0.1394	0.0691	income, restaurant, month
First differences (+ intercept)	+0.1394	0.0695 (HC1)	the restaurant, by differencing; month via the intercept

Fixed effects are FWL. A regression with one dummy per restaurant (and per month) gives the same coupon coefficient as partialling those dummies out of both variables — by demeaning, by analyze_fwl_plot, or, with two periods, by first differencing.
Panels can defeat unobserved confounders — if they do not change. The restaurant trait was never recorded, yet the fixed effects removed it because it was constant over time. Pooled OLS could not, however many observed controls it included.
With T = 2, first differences and two-way FE are the same estimator. The same coefficient, and, before small-sample factors, the same clustered standard error.
Count the absorbed parameters correctly. The same lesson as Step 2 of section 10: FWL reproduces the coefficient exactly, but the standard error depends on how the software counts what was partialled out.

For more on fixed effects as FWL, see What Does TWFE Actually Do? Manual Demeaning and the FWL Theorem and High-Dimensional Fixed Effects Regression: An Introduction in Python.

24. References

Acknowledgements

AI tools (Claude Code, Gemini, NotebookLM) were used to make the contents of this post more accessible to students. Nevertheless, the content in this post may still have errors. Caution is needed when applying the contents of this post to true research projects.

Evaluating the Impact of Infrastructure: A Beginner's Guide to Difference-in-Differences with the Jamuna Bridge

Wed, 05 Aug 2026 00:00:00 +0000

Abstract

Transport megaprojects absorb a large share of development finance, yet economists still disagree about whether connecting a poor region to a rich one revives the periphery or hollows it out. This tutorial works that question through one case, replicating in Python the evaluation of the Jamuna Bridge — a 4.8-kilometre crossing that opened in June 1998, cost about US\$985 million, and linked 26 million isolated Bangladeshis to Dhaka while cutting freight costs roughly in half. The analysis compares 123 treated upazilas in the Jamuna hinterland with 125 comparison upazilas in the Padma hinterland, a symmetric region whose own bridge was not begun until 2015. Four panels carry the evidence: satellite nighttime lights for 359 upazilas over seven three-year periods from 1992 to 2013, three population censuses, district rice yields back to 1988, and DHS and HIES village records. Two-way fixed-effects difference-in-differences is estimated with the diff-diff library and cross-checked in pyfixest, and the paper’s two doubly robust estimators are rebuilt by hand in NumPy. The bridge raised nighttime lights 10.9 percent, rice yields 6.3 percent and the services employment share 2.3 percentage points, while the manufacturing share fell 1.0 percentage point; population density fell 2.5 percent in the short run and rose 5.9 percent in the long run. Because density rose rather than fell, the loss of manufacturing here is the signature of comparative advantage rather than of backwash — a region can lose its factories and still be better off.

1. Overview

1.1 A bridge, two rivers, and three predictions

Bangladesh is a delta sliced into three by two of the largest rivers on earth. The Jamuna — the local name for the Brahmaputra, ninth in the world by discharge — cut the poor northwest off from Dhaka. The Ganges, locally the Padma, cut off the south. Before 1998, crossing the Jamuna meant a ferry that took more than three hours on a good day and, during Eid, could mean waiting thirty-six. A truck from Bogra to Dhaka took twenty hours.

Then the bridge opened, and that truck took six.

The question is what a shock like that does to the region on the far side. There are three answers in the literature, and they are not variations on a theme — they point in opposite directions.

The big push view is the one that gets megaprojects funded. Integrating a segmented market raises competition and allocational efficiency, and the lagging region revives.

The backwash view, which runs from Myrdal in 1957 through Krugman in 1991, says the opposite. With increasing returns and mobile factors, lowering trade costs lets the core capture the gains. Manufacturing concentrates where the market already is, and the newly connected periphery is hollowed out — deindustrialised, drained of people, worse off than before the road arrived.

The third view is the one this paper contributes, and it is the reason the case is worth studying carefully. Suppose the hinterland has a comparative advantage in agriculture. Then even with no increasing returns and no spatial sorting at all, a fall in trade costs pulls labour out of manufacturing and into the things the region is relatively good at. Manufacturing declines — but as specialisation, not as decay.

Here is the trap. Backwash and comparative advantage make the same prediction about factories. A study that measured only manufacturing would see the share fall, write “deindustrialisation”, and conclude backwash. The two stories only separate on a second outcome, and getting that second outcome right is the whole methodological lesson of this tutorial.

1.2 Learning objectives

By the end of this post you will be able to:

Compute a difference-in-differences estimate by hand from four group means, and explain what each subtraction removes.
State the estimand you are targeting — the ATT — and say why it is not the ATE.
Estimate two-way fixed-effects DiD with the diff-diff library, and cross-check it in pyfixest.
Build an event study and read its pre-treatment coefficients as a test rather than a result.
Construct two doubly robust estimators from scratch: propensity-odds weights and Kline’s Oaxaca-Blinder reweighting.
Assess how badly parallel trends would have to fail before your conclusion changes, using HonestDiD bounds and randomisation inference.
Audit a published paper against its own replication package, and read a regression footer for the tell that something has gone wrong.

1.3 The road ahead

The tutorial runs in eight stages. Each is a section below, and each either adds an assumption or tests one. Nighttime lights are the running example — they are the richest panel, with seven periods and 247 upazilas — and once the machinery is built, the other three outcomes go through it quickly.

graph TD
A("<b>Four data families</b><br/>night lights, census,<br/>rice yields, DHS and HIES") --> B("<b>The design</b><br/>Jamuna hinterland treated<br/>Padma hinterland comparison<br/>Dhaka core excluded")
B --> C("<b>Baseline</b><br/>A 2x2 by hand, then<br/>two-way fixed effects")
C --> D("<b>Dynamics</b><br/>short run versus long run,<br/>and a full event study")
D --> E("<b>Doubly robust</b><br/>LWDR logit-odds weights<br/>KOBDR Oaxaca-Blinder weights")
E --> F("<b>Two engines, one answer</b><br/>diff-diff with SurveyDesign<br/>and pyfixest with weights")
F --> G("<b>Robustness</b><br/>placebos, HonestDiD,<br/>public-goods placebo")
G --> H("<b>Verdict</b><br/>density rises, so backwash fails.<br/>comparative advantage survives")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class A,B blue
class C,D orange
class E,F teal
class G,H anchor

Notice that the estimator gets more sophisticated as you move down, but the question never changes. Stages three through five all estimate the same thing; they differ only in how hard they work to make the Padma hinterland a fair stand-in for the Jamuna hinterland. Stages six and seven then try to break the answer.

2. Key concepts at a glance

1. Difference-in-differences Two subtractions, one estimate. Compare how much the treated group changed with how much an untreated group changed over the same period. Whatever moved both groups equally cancels out.

Example

The Jamuna hinterland’s mean log luminosity rose 0.0719 between the pre- and post-bridge periods. The Padma hinterland’s rose 0.0078. The difference, 0.0641, is the raw estimate.

Analogy

Two bakeries raise prices the same week. One also changed its recipe. Subtract the other’s price rise to isolate the recipe.

2. Parallel trends The assumption that carries everything. Absent the treatment, the treated and comparison groups would have moved by the same amount. Not to the same level — by the same amount.

Example

The pre-bridge trend difference in nightlights is 0.008. The conventional test cannot reject a difference — its standard error is 0.092, wide enough to hide almost anything — but the equivalence test, which is the sharper instrument, rejects a difference larger than the margin at $p = 0.001$.

Analogy

Two hikers on parallel ridges a hundred metres apart in height. As long as the terrain runs parallel, you can measure what a helicopter lift did to one of them. The permanent height gap cancels; only a divergence in the slope would break it.

3. ATT (average treatment effect on the treated) The estimand here. The average effect among the units that actually got treated — not among a randomly chosen unit.

Example

Every estimate in this post answers “what did this bridge do to the 123 upazilas behind it”, not “what would a bridge do to a randomly chosen upazila in Bangladesh”.

Analogy

Asking how much a specific medicine helped the patients who took it, rather than how much it would help the general population.

4. Two-way fixed effects Grading on two curves at once. Remove each unit’s permanent level and each period’s common shock; whatever is left is unit-and-period specific.

Example

absorb=["geocode", "year"] removes 247 upazila levels and 7 period shocks. The satellite recalibration that dimmed every pixel in 2005 is soaked up by the year effect.

Analogy

A teacher compares each student to their own past average, then compares each exam to the class average on that exam. What survives both is the textbook effect.

5. Event study Every period gets its own coefficient. Instead of one post-treatment dummy, estimate a separate effect for each period relative to a baseline. The pre-treatment ones are a test; the post-treatment ones are the answer.

Example

The nightlights event study gives $-0.008$ before the bridge, then $0.007 \rightarrow 0.033 \rightarrow 0.050 \rightarrow 0.083 \rightarrow 0.128$ across the five post-bridge periods.

Analogy

Instead of asking “was the patient better after treatment”, chart the temperature every day and look at whether it was already falling before the pill.

6. Propensity score The probability of being treated, given what you can observe. Used to reweight the comparison group so that it looks like the treated group on measured characteristics.

Example

A logit of treatment on 1991 log population and log distance to the bridge foot. The two hinterlands overlap almost completely, with a median score near 0.50.

Analogy

Before comparing two schools’ exam results, work out how likely each pupil was to have enrolled at the better-funded one, and weight accordingly.

7. Doubly robust Two parachutes. Combine a model of who got treated with a model of the outcome. The estimator is consistent if either model is right.

Example

LWDR and KOBDR both weight the comparison group and also include the same covariates in the regression. They land at 0.106 and 0.109 against the unweighted 0.088.

Analogy

A skydiver carries a main chute and a reserve. Only if both fail does the jump end badly. But two chutes do not help if you jumped over the wrong country — double robustness protects against getting a model’s shape wrong, never against a confounder you never measured.

8. HonestDiD sensitivity How wrong can the assumption be? Rather than testing parallel trends and declaring victory, bound how large a post-treatment violation the conclusion can survive.

Example

The nightlights result survives until $M \approx 1$ — the post-bridge violation would have to be as large as the largest violation seen before the bridge.

Analogy

Instead of asking whether the bridge cable is fraying, ask how many strands could snap before the bridge falls down.

3. The research design

3.1 Why the Padma hinterland is the comparison

Everything below rests on one choice: which places stand in for the Jamuna hinterland’s missing counterfactual. Four things make the Padma hinterland unusually well suited.

It has the same problem and no solution. The Padma hinterland is cut off from Dhaka by the other great river. A Padma bridge had been discussed since before independence, but the government could not afford two at once. Construction began only in 2015, and it was still incomplete when the paper was written. Since the data end in 2013, the comparison region stayed isolated for the entire study window.

The choice of which river to bridge first was political, not economic. President Ershad’s political base was in Rangpur and Prime Minister Khaleda Zia’s in Bogra — both in the Jamuna hinterland. That is a threat only if bridge priority tracked economic shocks in the northwest, and the historical record says it tracked personal geography instead.

They are agro-climatically almost the same place. The northernmost point of the Jamuna hinterland sits at latitude 26.62, the southernmost point of the comparison at 23.81 — under three degrees apart. Florida spans more than five.

The imbalances that do exist are measurable and correctable. We will see in section 7.5 that the two regions differ significantly in the pre-bridge services and agriculture shares, and that conditioning on 1991 population, distance to the bridge foot, and rainfall removes those differences entirely.

A third region — the Dhaka and Chittagong core — appears in the data but is excluded from every regression. It is neither treated nor a credible comparison, and the authors dropped it from the design after a referee pointed out exactly that.

3.2 The estimand: what number are we actually after?

Before touching an estimator, state the target. This tutorial estimates the average treatment effect on the treated (ATT): the average effect of the Jamuna Bridge on the 123 upazilas that actually sit in its hinterland. It is not the ATE — it does not tell you what a bridge would do to a randomly chosen upazila in Bangladesh.

$$\tau_{ATT} = E\left[ Y_{it}(1) - Y_{it}(0) \mid D_J = 1, \, t > 1998 \right]$$

In words, this says: take the upazilas behind the Jamuna Bridge, and only the years after it opened; compare the luminosity they actually recorded against the luminosity they would have recorded had the bridge never been built; average the difference. The second quantity is never observed for anyone, which is the entire problem.

Symbol	Meaning	Code
$Y_{it}(1)$	outcome with the bridge	observed `lmn` where `treat == 1` and `post == 1`
$Y_{it}(0)$	outcome without the bridge	never observed; DiD reconstructs its change from the comparison group
$D_J$	Jamuna hinterland indicator	`treat`
$\tau_{ATT}$	the estimand	the coefficient on `treat_post`

The distinction matters for policy. The ATT answers “was this bridge worth building?”. The ATE would answer “should we build bridges generally?” — a different and much harder question, and not one this design can address.

This is observational data, not a randomised experiment. Bridge placement was not assigned by a coin flip, and the covariates below are not there to improve precision — they are there to address confounding. If bridge priority had been driven by pre-existing economic shocks in the northwest, and those shocks had persistent effects, the design would fail regardless of how many controls we add.

3.3 What difference-in-differences assumes

The estimator itself is four numbers and two subtractions.

$$\widehat{\tau}_{2 \times 2} = \left( \bar{Y}_{J,post} - \bar{Y}_{J,pre} \right) - \left( \bar{Y}_{P,post} - \bar{Y}_{P,pre} \right)$$

In words, this says: take how much the Jamuna hinterland changed, take how much the Padma hinterland changed over the same years, and subtract the second from the first. A national fertiliser subsidy, a monsoon, a change in how the satellite was calibrated — anything that moved both regions equally drops out of the subtraction.

The assumption that licenses it:

$$E\left[ Y_{it}(0) - Y_{i,t-1}(0) \mid D_J = 1 \right] = E\left[ Y_{it}(0) - Y_{i,t-1}(0) \mid D_J = 0 \right]$$

In words: had the bridge never been built, luminosity in the Jamuna hinterland would have changed period to period by exactly the same amount as luminosity in the Padma hinterland. Note what it does not say. The two regions need not sit at the same level — only move at the same rate. And because it is a statement about a world that never happened, it can never be proven. Everything in sections 10.3 and 16 is an attempt to make it more or less plausible.

graph LR
J0("<b>Jamuna hinterland</b><br/>1992-1997<br/>pre-bridge mean") -->|"observed change<br/>in the treated"| J1("<b>Jamuna hinterland</b><br/>1998-2013<br/>post-bridge mean")
P0("<b>Padma hinterland</b><br/>1992-1997<br/>pre-bridge mean") -->|"observed change<br/>in the comparison"| P1("<b>Padma hinterland</b><br/>1998-2013<br/>post-bridge mean")
P0 -.->|"parallel trends<br/>assumption"| CF("<b>Counterfactual Jamuna</b><br/>where Jamuna would have<br/>landed with no bridge")
P1 -.-> CF
J1 --> ATT("<b>ATT</b><br/>treated change minus<br/>comparison change")
CF --> ATT
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class J0,J1 blue
class P0,P1 orange
class CF anchor
class ATT teal

The two solid arrows are things we measure. The two dashed arrows are the assumption. Every robustness check later in the post is an attempt to make those dashed arrows more credible, and none of them can make the assumption disappear.

3.4 Big push, backwash, or comparative advantage

Now put the three theories into the same picture and find where they can be told apart.

graph TD
Q("<b>Trade costs to the core fall by half</b><br/>what happens to the hinterland?")
Q --> T1("<b>Big push</b><br/>integration raises efficiency<br/>and revives the lagging region")
Q --> T2("<b>Backwash</b><br/>Myrdal 1957, Krugman 1991<br/>The core captures the<br/>increasing returns")
Q --> T3("<b>Comparative advantage</b><br/>No increasing returns needed.<br/>The hinterland specialises in what<br/>it is relatively good at")
T1 --> P1("Predicts<br/>manufacturing share up,<br/>or at worst flat")
T2 --> P2("Predicts<br/>manufacturing share DOWN<br/><b>and</b> population density DOWN")
T3 --> P3("Predicts<br/>manufacturing share DOWN<br/><b>and</b> population density UP or flat")
P1 --> E1{"Did the manufacturing<br/>share fall?"}
P2 --> E1
P3 --> E1
E1 -->|"Yes, minus 1.2 pp"| OUT1("<b>Big push rejected</b>")
E1 -->|"Yes, minus 1.2 pp"| E2{"<b>The discriminating test</b><br/>what did population<br/>density do?"}
E2 -->|"Fell"| OUT2("Backwash supported")
E2 -->|"Rose, plus 5.9 percent<br/>in the long run"| OUT3("<b>Backwash rejected</b><br/>comparative advantage survives")
classDef sty_E1 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class E1 sty_E1
classDef sty_E2 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class E2 sty_E2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class Q anchor
class T1,P1 blue
class T2,P2,OUT1,OUT2 orange
class T3,P3,OUT3 teal

Formally, the discriminating test is a joint sign restriction:

$$\text{backwash} \implies \theta_1^{ind} < 0 \quad \text{and} \quad \theta_1^{dens} < 0$$

$$\text{comparative advantage} \implies \theta_1^{ind} < 0 \quad \text{and} \quad \theta_1^{dens} \geq 0$$

In words: both stories predict the factories leave, so the manufacturing coefficient alone is useless for telling them apart. They disagree about people. Backwash means the periphery is being emptied — capital and labour move to the core. Comparative advantage means the periphery is specialising, not emptying. So the sign of the population-density effect decides the case.

The key insight lives in that second diamond. A study that measured only factories would have declared backwash and stopped. Adding population density — one extra outcome, from a census that was already sitting there — turns an ambiguous finding into a decisive one. The lesson generalises well beyond bridges: when two theories predict the same sign on your headline outcome, go looking for the outcome where they disagree.

4. Setup and imports

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import pyfixest as pf
from diff_diff import (
DifferenceInDifferences,
MultiPeriodDiD,
SurveyDesign,
check_parallel_trends,
compute_honest_did,
equivalence_test_trends,
placebo_timing_test,
)
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
# Both libraries warn about the same harmless thing all the way through: `post` is
# collinear with the year fixed effects, so it gets dropped. Section 9.1 explains why
# that is expected. `analysis.py` silences them for the same reason.
import warnings
warnings.filterwarnings("ignore")

Two libraries do the estimation. diff-diff is a difference-in-differences toolkit with a scikit-learn-style API — you instantiate an estimator, call .fit(), and get a results object — plus a full diagnostic suite for parallel trends, placebo tests and sensitivity bounds. pyfixest is a fast high-dimensional fixed-effects regression package modelled on R’s fixest; it appears here as an independent second opinion. Install both with pip install diff-diff pyfixest.

A handful of code blocks below call helper functions rather than repeating boilerplate. stata_fe is a weighted unit fixed-effects regression with Stata’s exact cluster-robust degrees-of-freedom correction, which is what makes the reproduction audit in section 17 possible; stata_ols is its pooled sibling; event_study and diffdiff_mean are thin wrappers around diff-diff; add_common is written out in full in section 6.1, and build_weights is simply the twelve lines of section 11.2 to 11.4 packaged as a function so the other datasets can reuse them. Every one of them is defined in analysis.py, and apart from those definitions the code blocks below run in the order they appear.

The figures use the site’s dark palette, set once:

DARK_NAVY, GRID_LINE = "#0f1729", "#1f2b5e"
LIGHT_TEXT, WHITE_TEXT = "#c8d0e0", "#e8ecf2"
STEEL_BLUE, WARM_ORANGE, TEAL = "#6a9bcc", "#d97757", "#00d4c8"
plt.rcParams.update({
"figure.facecolor": DARK_NAVY, "axes.facecolor": DARK_NAVY,
"axes.labelcolor": LIGHT_TEXT, "axes.titlecolor": WHITE_TEXT,
"axes.grid": True, "grid.color": GRID_LINE, "grid.alpha": 0.8,
"xtick.color": LIGHT_TEXT, "ytick.color": LIGHT_TEXT,
"text.color": WHITE_TEXT, "font.size": 12, "legend.frameon": False,
"savefig.facecolor": DARK_NAVY,
})

5. Loading the data

5.1 Four data families

The evaluation uses five files covering four kinds of outcome. All are committed as tidy CSVs alongside this post, so the notebook runs without the original Stata package.

BASE = ("https://raw.githubusercontent.com/cmg777/starter-academic-v501/"
"master/content/tutorials/python_bridge_impact/data/")
nl_raw = pd.read_csv(BASE + "bridge_nightlights.csv") # satellite luminosity
emp_raw = pd.read_csv(BASE + "bridge_employment.csv") # population censuses
yld_raw = pd.read_csv(BASE + "bridge_yield.csv") # Boro rice yield
hh_raw = pd.read_csv(BASE + "bridge_dhs_household.csv") # DHS/HIES households
vill_raw = pd.read_csv(BASE + "bridge_dhs_village.csv") # DHS village questionnaire
for nm, d, unit in [("employment", emp_raw, "geocode"), ("nightlights", nl_raw, "geocode"),
("yield", yld_raw, "dist"), ("dhs household", hh_raw, "District"),
("dhs village", vill_raw, "District")]:
ins = d[d["smp1"].notna()]
print(f" {nm:15s} rows={len(d):5d} units={d[unit].nunique():4d}"
f" periods={d['year'].nunique()}"
f" treated units={ins.loc[ins.treat == 1, unit].nunique():4d}"
f" comparison units={ins.loc[ins.treat == 0, unit].nunique():4d}")

 employment rows= 1053 units= 351 periods=3 treated units= 123 comparison units= 125
nightlights rows= 2513 units= 359 periods=7 treated units= 127 comparison units= 125
yield rows= 128 units= 16 periods=8 treated units= 5 comparison units= 6
dhs household rows= 1543 units= 37 periods=7 treated units= 16 comparison units= 21
dhs village rows= 1455 units= 41 periods=7 treated units= 20 comparison units= 21

Four things to notice. The nightlights panel is by far the richest — 359 upazilas observed in seven periods — which is why it carries the tutorial. The yield panel is the poorest: sixteen former districts, of which only eleven enter the estimation. That is not a typo; agricultural statistics in Bangladesh are published at the old district level, so the entire rice-yield result rests on nine to eleven clusters. Third, smp1 is the sample filter: it is missing for the Dhaka-Chittagong core, which is neither treated nor a valid comparison. And fourth, the treated and comparison groups are almost exactly balanced in size — 123 against 125 upazilas in the census panel.

5.2 What one row means, and why `year` is not a year

This is the single detail most likely to break a replication of this paper.

NL_YEARS = {1: "1992-94", 2: "1995-97", 3: "1998-00", 4: "2001-04",
5: "2005-07", 6: "2008-10", 7: "2011-13"}
YLD_YEARS = {1: "1988-91", 2: "1992-94", 3: "1995-97", 4: "1998-00",
5: "2001-04", 6: "2005-07", 7: "2008-10", 8: "2011-13"}
EMP_YEARS = {1: "1991", 2: "2001", 3: "2011"}
peek = nl_raw[["geocode", "year", "mn", "treat", "smp1"]].head(4)
print(peek.astype({"geocode": int, "year": int, "treat": int}).to_string(index=False))

 geocode year mn treat smp1
10409 1 1.016667 0 0.0
10409 2 1.053333 0 0.0
10409 3 1.045000 0 0.0
10409 4 1.083333 0 0.0

One row is one upazila in one three-year window. The year column holds the integers 1 through 7, not calendar years: annual nightlights and yields are averaged into three-year blocks to smooth out transitory shocks. The bridge opened in June 1998, which falls inside block 3, so blocks 1 and 2 are the pre-period.

This matters beyond labelling, because the controls interact baseline characteristics with year. If you substitute calendar years there, every coefficient changes.

The .astype(int) in that snippet is cosmetic — the CSVs store every numeric column as a float — but notice which column is not cast. smp1 has to stay floating point because it holds missing values for the Dhaka-Chittagong core, and missingness is the whole point of it.

5.3 The outcome variables

Family	Outcome	Built from	Unit	Periods
Nighttime lights	`lmn` = $\ln(mn + 1)$, and its growth `D_lmn`	DMSP-OLS satellite, 1 km pixels averaged to upazila	upazila	7
Rice yield	`lyld` = $\ln(yld)$, and `D_lyld`	Bangladesh Bureau of Statistics yearbooks	former district	8
Population and employment	`ldensity`, and the shares `sagr`, `sind`, `sserv`	Population censuses 1991, 2001, 2011 (IPUMS)	upazila	3
Public goods	electricity access, distances to schools, clinics, banks	DHS 1993-2014 and HIES 1995/96	village-year	7

The +1 inside the nightlights log is not cosmetic. Luminosity is bottom-coded at 1.0 in this dataset, and most rural upazilas sit near the floor, so the transformation keeps the many near-dark observations from dominating.

6. Data preparation

Nothing below is exotic, but three of these steps decide whether the replication lands on the published numbers or somewhere nearby, so they are worth doing slowly.

6.1 From raw files to working frames

Every panel dataset needs the same five derived variables, so they go in one function. The first line of it is the one the whole design rests on.

def add_common(d, rain_plus_one=False, dist_scale=1.0):
"""The five derived variables every panel dataset needs."""
d = d.copy()
# Distance to whichever crossing is the relevant one: the Jamuna bridge for a
# treated upazila, the Padma site for a comparison one. Taking the minimum of
# the two is what makes the two hinterlands mirror images of each other.
d["mdist"] = np.minimum(d["jamuna_m"], d["padma_m"]) / 1000.0 # metres to km
d["lmdist"] = np.log(d["mdist"] + 1) / dist_scale
d["lpop91"] = np.log(d["pop91"]) # 1991 population, fixed pre-bridge
# Stata's ln() returns missing for zero; NumPy returns -inf and the row survives.
# The replace() is what keeps 24 zero-rainfall rows out of the estimation sample.
d["lrainm"] = (np.log(d["rainm"] + 1) if rain_plus_one
else np.log(d["rainm"].replace(0, np.nan)))
d["lrainsd"] = np.log(d["rainsd"].replace(0, np.nan))
return d
nl = add_common(nl_raw, rain_plus_one=True) # nite_2021.do uses log(rainm + 1)
emp = add_common(emp_raw) # employment_2021.do uses plain log(rainm)
yld = add_common(yld_raw, dist_scale=10.0) # Yield_2021.do rescales lmdist by 10

Two of those three calls carry a keyword, and both keywords come from reading the original do-files rather than from any statistical principle. nite_2021.do writes gen lrainm = ln(rainm+1) while employment_2021.do writes gen lrainm = ln(rainm); the yield do-file divides its distance log by ten. None of the three changes a treatment coefficient by much, but all three change it in the third decimal, which is the difference between reproducing a table and almost reproducing it.

The replace(0, np.nan) deserves its own sentence, because it is the first of the three traps in section 17. Twenty-four employment rows record zero rainfall. Stata’s ln(0) is missing and the row drops out; NumPy’s np.log(0) is -inf and the row stays, quietly corrupting every sample size downstream. It fails silently in exactly the way that is hardest to notice.

6.2 Outcomes, periods and the post indicator

nl = nl.sort_values(["geocode", "year"])
nl["lmn"] = np.log(nl["mn"] + 1)
nl["D_lmn"] = nl.groupby("geocode")["lmn"].diff() # growth: a triple difference
emp["ldensity"] = np.log(emp["density"])
for share, num in [("sagr", "pop_agr"), ("sind", "pop_ind"), ("sserv", "pop_serv")]:
emp[share] = emp[num] / emp["emp"] # sector shares of employment
yld = yld.sort_values(["dist", "year"])
yld["lyld"] = np.log(yld["yld"])
yld["D_lyld"] = yld.groupby("dist")["lyld"].diff()
# Each file has its own period grid, so each gets its own pre/post cut.
nl["post"], nl["sr"], nl["lr"] = nl.year.gt(2), nl.year.between(3, 4), nl.year.gt(4)
emp["post"], emp["sr"], emp["lr"] = emp.year.gt(1), emp.year.eq(2), emp.year.eq(3)
yld["post"], yld["sr"], yld["lr"] = yld.year.ge(4), yld.year.between(4, 5), yld.year.gt(5)
for d in (nl, emp, yld):
for h in ("post", "sr", "lr"):
d[h] = d[h].astype(int)
d[f"treat_{h}"] = d["treat"] * d[h]

Because three of the four outcomes are logarithms, their coefficients read as proportional changes. The exact conversion is

$$g(\widehat{\theta}_1) = 100 \left( \exp(\widehat{\theta}_1) - 1 \right)$$

In words: a coefficient of 0.109 on log nightlights means luminosity is $100(e^{0.109} - 1) = 11.5$ percent higher. Below about 0.05 the exact and approximate readings differ by under half a percentage point, so the coefficient can simply be read as a percentage. At 0.109 the gap has grown to 0.6 points, so when this post and the original paper both call that estimate “10.9 percent” they are quoting the approximation, not the exact figure. The employment shares are not logged, so those coefficients are already in percentage points and need no conversion at all.

6.3 Controls: initial conditions on a trend

for d in (nl, emp, yld):
d["lpop91_t"] = d["lpop91"] * d["year"] # initial size, interacted with the trend
d["lmdist_t"] = d["lmdist"] * d["year"] # initial remoteness, likewise
CONTROLS = ["lpop91_t", "lrainm", "lrainsd", "lmdist_t"]

These two lines are the subtle ones and they are worth dwelling on. lpop91 and lmdist are fixed characteristics — they never change over the panel — so a unit fixed effect already absorbs them completely. Including them alone would do nothing. Interacting them with the time index is different: it allows an upazila that was large or remote in 1991 to be on a permanently different trajectory thereafter.

That relaxes the identifying assumption in a useful way. Instead of “all upazilas would have trended alike”, we now need only “upazilas that started at the same size and remoteness would have trended alike”. Since size and remoteness are the two things most obviously correlated with bridge placement, this is exactly the relaxation the design needs.

Note that year here is the integer period index from section 5.2, not a calendar year. Substituting calendar years into these two lines is the single most common way to fail to reproduce this paper.

6.4 The estimation samples

# `smp1` is missing for the Dhaka-Chittagong core, which is neither treated nor a
# credible comparison. These three lines are section 3.1's design decision, in code.
NL = nl[nl["smp1"].notna()].dropna(subset=CONTROLS).copy()
EMP = emp[emp["smp1"].notna()].dropna(subset=CONTROLS).copy()
YLD = yld[yld["smp1"].notna()].dropna(subset=CONTROLS).copy()
for d, unit in [(NL, "geocode"), (EMP, "geocode"), (YLD, "dist")]:
d[["year", unit, "treat"]] = d[["year", unit, "treat"]].astype(int)
# The two DHS files never use smp1 -- they are already restricted to the two hinterlands.
HH = hh_raw.copy()
HH["mdist"] = np.minimum(HH["jamuna_m"], HH["padma_m"]) / 1000.0
HH["lmdist_t"] = np.log(HH["mdist"] + 1) * HH["year"]
HH["treat_sr"] = HH["treat"] * HH["year"].eq(4)
HH["treat_lr"] = HH["treat"] * HH["year"].ge(5)
VILL = vill_raw.copy()
VILL["mdist"] = np.minimum(VILL["jamuna_m"], VILL["padma_m"]) / 1000.0
VILL["lmdist_t"] = np.log(VILL["mdist"] + 1) * VILL["year"]
VILL["treat_sr"] = VILL["treat"] * VILL["year"].eq(4)
VILL["treat_lr"] = VILL["treat"] * VILL["year"].ge(5)
for nm, d, unit in [("NL", NL, "geocode"), ("EMP", EMP, "geocode"), ("YLD", YLD, "dist")]:
print(f" {nm:4s} rows={len(d):5d} units={d[unit].nunique():4d}")

 NL rows= 1729 units= 247
EMP rows= 738 units= 246
YLD rows= 88 units= 11

One convention for the rest of the post: lower case is the full frame, upper case is the estimation sample. nl still holds all 359 upazilas including the core; NL holds the 247 that the regressions actually see. The distinction matters more than it looks, and section 15 turns on it — the distance terciles have to be cut on nl, before rows are lost to missing controls, not on NL.

Those three sample sizes are worth checking against the paper before going any further. 1729 and 247 are the N and the upazila count in the first column of the published Table 1; 738 and 246 are the census panel’s; 88 and 11 are the yield panel’s. If a replication is going to go wrong, it usually goes wrong here, and it is far cheaper to find out now than after the coefficients disagree.

7. Exploratory analysis

7.1 The identification geometry

You do not need a shapefile to see the design. Plot every upazila by its distance to each of the two river crossings, and the map draws itself.

geo = emp_raw.drop_duplicates("geocode").copy()
geo["grp"] = np.where(geo["treatd"] == 1, "core (excluded)",
np.where(geo["treat"] == 1, "Jamuna hinterland (treated)",
"Padma hinterland (comparison)"))
fig, ax = plt.subplots(figsize=(9.5, 7.5))
for lab, color, mk in [("Jamuna hinterland (treated)", WARM_ORANGE, "o"),
("Padma hinterland (comparison)", STEEL_BLUE, "o"),
("core (excluded)", "#7a8399", "x")]:
s = geo[geo["grp"] == lab]
ax.scatter(s["jamuna_m"] / 1000, s["padma_m"] / 1000,
s=np.sqrt(s["pop91"]) / 6, color=color, marker=mk, alpha=0.75,
edgecolors="none", label=f"{lab} (n={len(s)})")
ax.plot([0, 400], [0, 400], color=WHITE_TEXT, ls=":", lw=1.4)
ax.set_xlabel("Distance to the Jamuna bridge (km)")
ax.set_ylabel("Distance to the Padma crossing (km)")

Figure 1. Every upazila plotted by its distance to the Jamuna bridge against its distance to the Padma crossing. Marker area is proportional to 1991 population; the dotted line is the equidistance diagonal. The two hinterlands separate on either side of it, and the excluded Dhaka-Chittagong core sits away from both.

The two hinterlands separate cleanly on either side of the equidistance diagonal, and the excluded core sits away from both. Treated upazilas run from 8.4 km to 269.5 km from the bridge foot; the comparison upazilas span the same range on their own river. That symmetry is what makes the comparison credible — the Padma hinterland is not a generic control group, it is the same kind of place with the same kind of river problem and no bridge.

7.2 Luminosity paths

fig, ax = plt.subplots(figsize=(10, 6))
for grp, color, lab in [(1, WARM_ORANGE, "Jamuna hinterland (treated)"),
(0, STEEL_BLUE, "Padma hinterland (comparison)")]:
m = NL[NL["treat"] == grp].groupby("year")["lmn"].mean()
ax.plot(m.index, m.to_numpy(), marker="o", ms=5, lw=2, color=color, label=lab)
ax.axvline(2.5, color=TEAL, ls="--", lw=1.5) # the bridge opens inside period 3

Figure 2. Mean log nighttime luminosity by three-year period, Jamuna hinterland in orange against the Padma hinterland in blue. The teal dashed line marks period 3, inside which the bridge opened in June 1998.

Before 1998 the two lines track each other closely; afterwards the Jamuna line pulls away. The gap in levels is small — luminosity is bottom-coded and most of these upazilas are dark — but the divergence is monotone across all five post-bridge periods. A one-off shock would produce a step; this produces a ramp.

7.3 The other three outcomes

Figure 3. Mean log Boro rice yield by period, 1988–2013. Both series climb with the late Green Revolution; only the gap between them is evidence.

Rice yields rise everywhere in Bangladesh over this period — the long tail of the Green Revolution — so the treated series climbing is not evidence of anything on its own. What matters is that the two series climb together until the late 1990s and then separate. That common component is exactly what the subtraction removes.

Figure 4. Log population density and the agriculture, industry and services employment shares, by census year. Only 1991 is pre-bridge, which is why these four outcomes support a level-balance test but no pre-trend test.

The census panel is the thinnest of the four, and only one of its three years is pre-bridge. That single pre-period is a real limitation: no pre-trend test is possible for population density or the employment shares, only a level-balance test. It is worth flagging early because it is the outcome that later decides the theoretical question.

Figure 5. Employment composition of the two hinterlands, 1991–2011. Agriculture’s share falls and services rises in both — the question is whether it happened faster on the bridged side.

Structural transformation is visible in both regions: agriculture’s share falls, services rises. Both hinterlands are developing. The question is whether it happened faster on the bridged side, and by how much — which is a question about the difference between two slopes, not about either slope.

7.4 Distance gradients before the bridge

This next figure is the one that explains the paper’s most surprising result before we get to it.

pre = EMP[(EMP["year"] == 1) & (EMP["treat"] == 1)] # 1991, treated upazilas only
for col in ["sagr", "sind", "sserv", "ldensity"]:
slope = np.polyfit(pre["mdist"], pre[col], 1)[0]
print(f" {col:10s} {slope:+.6f} per km")

 sagr +0.000692 per km
sind -0.000278 per km
sserv -0.000414 per km
ldensity -0.001864 per km

Figure 6. 1991 employment shares against distance to the bridge foot, treated upazilas only. Remote upazilas were more agricultural and less dense before the bridge existed — which is why they later had the most to gain.

Before the bridge existed, the agriculture share rose by about 0.07 percentage points per kilometre of distance from the bridge foot, while manufacturing and services both fell. Remote upazilas were more agricultural and less dense. Hold on to that: it means the places furthest from the bridge had the most agricultural output to ship, and therefore the most to gain from a fall in the cost of shipping it. This is why the effects turn out to be largest at the far end of the line rather than next to the bridge — a result that looks backwards until you have seen this figure.

7.5 Balance and pre-trends

Do the two hinterlands actually look alike before 1998? Partly.

pre_emp = EMP[EMP["year"] == 1] # 1991 cross-section
for y in ["ldensity", "sind", "sserv", "sagr"]:
naive = stata_ols(pre_emp, y, ["treat"], cluster="geocode")
cond = stata_ols(pre_emp, y, ["treat", "lpop91", "lrainm", "lrainsd", "lmdist"],
cluster="geocode")
print(f" {y:9s} naive {naive['coef']['treat']:+.4f} (p={naive['p']['treat']:.3f})"
f" conditional {cond['coef']['treat']:+.4f} (p={cond['p']['treat']:.3f})")

 ldensity naive -0.0236 (p=0.730) conditional +0.1741 (p=0.106)
sind naive +0.0002 (p=0.967) conditional +0.0065 (p=0.294)
sserv naive -0.0879 (p=0.000) conditional -0.0179 (p=0.334)
sagr naive +0.0877 (p=0.000) conditional +0.0114 (p=0.581)

Figure 7. Pre-bridge treated-comparison differences in levels and in trends, under all four estimators. The services and agriculture level gaps are large unconditionally and vanish once the four controls enter; the one significant trend difference does not survive reweighting.

The raw pre-bridge differences in the services and agriculture shares are large and unambiguous: the Jamuna hinterland was 8.8 percentage points more agricultural and 8.8 points less service-oriented than the Padma hinterland in 1991. Conditioning on 1991 population, distance to the bridge foot and the two rainfall variables collapses both to statistical insignificance — services to $-0.018$ ($p = 0.33$), agriculture to $+0.011$ ($p = 0.58$).

That result is the reason those four controls appear in every specification for the rest of the post. The imbalance is real, and it is entirely explained by observable initial conditions.

One pre-trend difference is significant, and it should be said plainly rather than waved away: the unweighted nightlights trend, at $+0.043$ with a standard error of 0.019 ($p = 0.022$). Taken alone it says the Jamuna hinterland was already brightening slightly faster than the Padma hinterland before the bridge, which is exactly the kind of thing that invalidates a DiD.

It does not survive the reweighting that every headline specification uses. Under LWDR the same trend difference falls to $+0.027$ ($p = 0.16$), and under KOBDR to $+0.027$ ($p = 0.16$). No trend difference is significant at 5 percent under either doubly robust estimator, in any panel. That is a better argument for the weights than anything in section 11 — they were introduced to fix a level imbalance, and they turn out to fix the trend imbalance too.

8. The 2x2: four numbers and no library

Before any estimator, do it by hand. This is the whole idea, and it fits in three lines.

cell = NL.groupby(["treat", "post"])["lmn"].mean().unstack()
d_treated = cell.loc[1, 1] - cell.loc[1, 0]
d_control = cell.loc[0, 1] - cell.loc[0, 0]
print(f" treated pre {cell.loc[1,0]:.4f} post {cell.loc[1,1]:.4f} "
f"change {d_treated:+.4f}")
print(f" comparison pre {cell.loc[0,0]:.4f} post {cell.loc[0,1]:.4f} "
f"change {d_control:+.4f}")
print(f" difference-in-differences = {d_treated - d_control:+.4f}")

 treated pre 1.2551 post 1.3270 change +0.0719
comparison pre 1.2113 post 1.2191 change +0.0078
difference-in-differences = +0.0641

Figure 8. The 2×2 in one picture. Solid lines are the two observed changes; the teal dashed line is where the Jamuna hinterland would have landed at the comparison group’s growth rate. The bracket is the ATT.

The Jamuna hinterland brightened by 0.072 log points across the bridge opening; the Padma hinterland by 0.008. The difference, 0.064, is the estimate. The teal dashed line in the figure is the counterfactual: where the treated group would have landed if it had grown at the comparison group’s rate. The gap between that line and where it actually landed is the ATT.

Everything the rest of this post does is a refinement of these four numbers. It is worth registering now that the fully specified doubly robust estimate will come out at 0.109 — larger than the naive 2x2, not smaller. Adjustment does not always shrink an effect.

9. Baseline: two-way fixed effects

9.1 The specification

The 2x2 uses two group means and two period means. With 247 upazilas and 7 periods we can do much better: give every upazila its own level and every period its own shock.

$$Y_{it} = \theta_0 + \mu_i + \mu_t + \theta_1 \left( D_J \times D_{post} \right) + \sum_q \beta_q X_{qit} + \sum_m \pi_m \left( Z_{mi0} \times t \right) + \varepsilon_{it}$$

In words: each upazila carries a permanent level $\mu_i$, each period carries a national shock $\mu_t$, and after removing both we ask whether the Jamuna upazilas moved differently once the bridge opened. The final sum is the trend-interacted initial conditions from section 6.3.

Symbol	Meaning	Code
$Y_{it}$	outcome	`lmn`
$\mu_i$, $\mu_t$	upazila and period effects	`absorb=["geocode", "year"]`
$D_J \times D_{post}$	the treatment indicator	`treat_post`
$X_{qit}$	time-varying controls	`lrainm`, `lrainsd`
$Z_{mi0} \times t$	initial conditions on a trend	`lpop91_t`, `lmdist_t`
$\theta_1$	the ATT	the coefficient we report
$\varepsilon_{it}$	error	clustered on `geocode`

Note that $D_J$ and $D_{post}$ do not appear on their own. They cannot: $D_J$ never varies within an upazila, so the upazila effects absorb it, and $D_{post}$ is a function of the period, so the period effects absorb that. Only the interaction survives, which is a useful reminder that DiD identification lives entirely in the interaction.

9.2 Estimating it with diff-diff

res = DifferenceInDifferences(cluster="geocode").fit(
NL, outcome="lmn", treatment="treat", time="post",
covariates=["lpop91_t", "lrainm", "lrainsd", "lmdist_t"],
absorb=["geocode", "year"], unit="geocode")
print(res)
res.print_summary()

DiDResults(ATT=0.0881***, SE=0.0220, p=0.0001)

The bridge raised nighttime luminosity by 0.088 log points — about 9.2 percent — with a standard error of 0.022. That is a t-statistic near 4, and it is a substantially larger estimate than the raw 2x2 of 0.064, which tells us the fixed effects and controls were doing real work.

Three API details are worth learning here rather than discovering the hard way.

treatment and time are the group and period indicators, and diff-diff forms the interaction itself. Here time="post" is the binary pre/post switch, not the seven-period index. Passing time="year" to DifferenceInDifferences would ask it to treat the period index as the post indicator, which is not the model we want.

Use absorb= rather than fixed_effects=. Both return the same coefficient, but absorb partials the fixed effects out before the degrees-of-freedom correction, which is what Stata’s xtreg, fe does; fixed_effects= builds explicit dummies and counts them in $K$, giving 0.0238 here instead of 0.0220. The published standard error is 0.022, so absorb is the one that reproduces it.

The sibling estimator TwoWayFixedEffects is not a drop-in substitute in this design. Calling it with time="year" returns 0.0184 (0.0040) — a different model entirely, because it interprets the period index rather than a pre/post switch. Calling it with time="post" returns 0.1339 (0.0200), which differs from the 0.0881 above because it does not absorb the seven period effects, only a single post dummy. Neither is a bug; both are the answer to a different question. When a library offers several routes to “the DiD estimate”, check which one reproduces a number you already know.

9.3 The same regression in pyfixest

fit = pf.feols("lmn ~ treat_post + post + lpop91_t + lrainm + lrainsd + lmdist_t"
" | geocode + year", data=NL, vcov={"CRV1": "geocode"})
print(f'{fit.coef()["treat_post"]:.10f} {fit.se()["treat_post"]:.10f}')

0.0880677516 0.0220101168

A second library, a completely different implementation, and the same number to seven decimal places. This is worth doing routinely. A DiD estimate is a small number extracted from a large panel through several layers of transformation, and the cheapest insurance against a coding error is to reproduce it in software that shares none of your code.

pyfixest also prints a warning on that call — 1 variables dropped due to multicollinearity. The following variables are dropped: ['post']. That is not a problem, it is the library confirming the point made at the end of section 9.1: post is a function of the period, so the year fixed effects have already absorbed it. Leaving it in the formula costs nothing and makes the redundancy visible.

10. Dynamics: short run, long run, and the event study

10.1 One post-bridge dummy is not enough

Migration, credit and supply chains all take time. Splitting the post-bridge window in two lets the data report a path instead of an average.

$$Y_{it} = \delta_0 + \mu_i + \mu_t + \delta_1 \left( D_J \times D_{SR} \right) + \delta_2 \left( D_J \times D_{LR} \right) + \sum_q \beta_q X_{qit} + \sum_m \pi_m \left( Z_{mi0} \times t \right) + \varsigma_{it}$$

In words: replace the single post-bridge switch with two — one for the years just after opening, one for the years well after — and estimate two effects instead of one.

Symbol	Code	Nightlights	Census
$D_{SR}$	`sr`	periods 3-4, 1998-2004	2001
$D_{LR}$	`lr`	periods 5-7, 2005-2013	2011
$\delta_1$	`treat_sr`	short-run ATT	short-run ATT
$\delta_2$	`treat_lr`	long-run ATT	long-run ATT

10.2 The event study

Better still: give every period its own coefficient, measured relative to the last pre-bridge one.

$$Y_{it} = \mu_i + \mu_t + \sum_{k \neq 2} \gamma_k \, D_J \cdot \mathbf{1}[t = k] + \sum_q \beta_q X_{qit} + u_{it}$$

In words: $\gamma_k$ is the treated-comparison gap in period $k$, normalised to zero in period 2, the last pre-bridge window. The coefficients for $k < 3$ are a test — if parallel trends is reasonable they should be indistinguishable from zero. The coefficients for $k \geq 3$ are the answer — they trace the whole path of the effect.

ev = MultiPeriodDiD(cluster="geocode").fit(
NL, outcome="lmn", treatment="treat", time="year",
post_periods=[3, 4, 5, 6, 7], covariates=CONTROLS,
absorb=["geocode"], reference_period=2, unit="geocode")
for p in sorted(ev.period_effects):
e = ev.get_effect(p)
print(f" period {p} ({NL_YEARS[p]}): {e.effect:+.4f} (se {e.se:.4f})")

 period 1 (1992-94): -0.0082 (se 0.0167)
period 3 (1998-00): +0.0068 (se 0.0179)
period 4 (2001-04): +0.0328 (se 0.0217)
period 5 (2005-07): +0.0501 (se 0.0216)
period 6 (2008-10): +0.0831 (se 0.0238)
period 7 (2011-13): +0.1279 (se 0.0271)

Figure 9. Event study for log nighttime lights, normalised to period 2 with 95 percent confidence intervals. The single pre-bridge coefficient sits on zero; the five post-bridge ones climb monotonically. The original paper never drew this figure.

This is the most persuasive figure in the analysis, and the original paper never drew it. The one available pre-bridge coefficient is $-0.008$ against a standard error of 0.017 — sitting on zero, exactly what parallel trends requires. Then the effect climbs monotonically across all five post-bridge periods, from $+0.7$ percent in 1998-2000 to $+12.8$ percent in 2011-13.

Think about what a confounder would have to look like to generate this. It would need to be absent before June 1998, appear at the right moment, and then grow steadily for fifteen years without ever reversing. Such things exist, but the list is short, and every candidate on it is easier to argue about once you have seen this picture than once you have seen a single pooled coefficient.

10.3 Testing parallel trends formally

pt = check_parallel_trends(NL, outcome="lmn", time="year",
treatment_group="treat", pre_periods=[1, 2])
eq = equivalence_test_trends(NL, outcome="lmn", time="year",
treatment_group="treat", unit="geocode",
pre_periods=[1, 2])
def show(title, d, keys):
print(f" {title}")
for k in keys:
v = d[k]
print(f" {k:28s} {v:.5f}" if isinstance(v, float) else f" {k:28s} {v}")
show("check_parallel_trends (nightlights, 1992-97):", pt,
["trend_difference", "trend_difference_se", "p_value", "parallel_trends_plausible"])
print()
show("equivalence_test_trends:", eq,
["equivalence_margin", "tost_p_value", "equivalent"])

 check_parallel_trends (nightlights, 1992-97):
trend_difference 0.00844
trend_difference_se 0.09242
p_value 0.92725
parallel_trends_plausible True
equivalence_test_trends:
equivalence_margin 0.04019
tost_p_value 0.00107
equivalent True

The two tests do different jobs and the second is the more useful one. check_parallel_trends fails to reject a difference in trends, with a p-value of 0.93. That is reassuring but weak: failing to reject is not evidence of similarity, especially when the standard error is 0.092 and could hide almost anything.

The equivalence test flips the null around. It asks whether the trend difference is smaller than a pre-specified margin, and rejects the hypothesis that it is larger at $p = 0.001$. That is a positive finding rather than an absence of one. When you have the option, report both — and treat an insignificant pre-trend on its own as the weakest form of the evidence, not the strongest.

11. Two doubly robust estimators, built by hand

11.1 Why doubly robust

So far the comparison group has been used as-is. But we know the two hinterlands differ on measured characteristics — section 7.5 showed the imbalance. There are two classical ways to fix that.

Reweight the comparison group so that its covariate distribution matches the treated group’s. This needs a model of who got treated.

Regression-adjust, putting the covariates on the right-hand side. This needs a model of how the outcome depends on covariates.

A doubly robust estimator does both, and is consistent if either model is correct. That is a much weaker requirement than getting both right.

graph TD
Z("<b>Pre-bridge covariates</b><br/>log population in 1991<br/>log distance to bridge foot") --> L("<b>Logit model</b><br/>probability of being in the<br/>Jamuna hinterland, given Z")
L --> P("<b>Fitted propensity p</b><br/>one number per upazila")
P --> TR("<b>Trim</b><br/>drop comparison upazilas in the<br/>bottom 5 percent of p")
TR --> W1("<b>LWDR weight</b><br/>odds of p, rescaled.<br/>treated units weighted 1")
Z --> W2("<b>KOBDR weight</b><br/>Oaxaca-Blinder projection of the<br/>treated covariate mean onto controls")
TR --> W2
W2 --> NEG("Drop comparison units with<br/>negative KOBDR weight")
W1 --> REG("<b>Weighted two-way fixed effects</b><br/>the same covariates enter again<br/>as regression adjustment")
NEG --> REG
REG --> DR("<b>Doubly robust ATT</b><br/>consistent if EITHER the weight model<br/>OR the outcome model is right")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Z,L,P blue
class TR,W1,NEG orange
class W2,DR teal
class REG anchor

The two branches out of the covariate box are the two protections. The left branch models who got treated; the right branch models what the outcome would have been. Note also that trimming and negative-weight dropping happen before the regression: both are sample decisions, and both must be reported.

11.2 The propensity model and the 5 percent trim

$$p_i = \Pr\left( D_J = 1 \mid Z_i \right) = \frac{\exp\left( \alpha_0 + \alpha_1 \ln P_i + \alpha_2 \ln d_i \right)}{1 + \exp\left( \alpha_0 + \alpha_1 \ln P_i + \alpha_2 \ln d_i \right)}$$

In words: estimate how likely each upazila was to end up on the Jamuna side, using only two things that were fixed before the bridge existed — how many people lived there in 1991, and how far it sits from the nearest bridge foot.

s = nl[nl["smp1"].notna()].dropna(subset=["lpop91", "lmdist"]).copy()
X = sm.add_constant(s[["lpop91", "lmdist"]].astype(float))
D = s["treat"].to_numpy(float)
logit = sm.Logit(D, X).fit(disp=0)
s["p"] = logit.predict(X)
cut = np.percentile(s["p"], 5)
print(f" logit N={len(s)} coefs={logit.params.to_numpy().round(6)}")
print(f" 5% propensity trim at p={cut:.7f}")

 logit N=1743 coefs=[-7.206172 0.313503 0.741378]
5% propensity trim at p=0.2918769

Then the trimming rule:

$$w_i = 0 \quad \text{whenever} \quad D_{J,i} = 0 \quad \text{and} \quad p_i < q_{0.05}(p)$$

Discard the comparison upazilas whose estimated likelihood of treatment sits in the bottom 5 percent. Why? Because a comparison unit with a propensity of 0.03 has to be multiplied by roughly thirty to stand in for a treated one, and it then speaks with the voice of thirty upazilas. It is like settling a national budget at an exchange rate of thirty to one: a mis-measured rainfall figure or one enumerator’s mistake arrives in your final accounts multiplied thirtyfold. Trimming is the rule that you will not trade at rates above a certain level.

There is a cost, and it is worth being explicit about it. Trimming shifts the estimand slightly: you are now estimating the ATT on the region of common support, not on all 123 treated upazilas.

11.3 LWDR: propensity odds as ATT weights

$$w_i^{LW} = \frac{p_i}{1 - p_i} \cdot \frac{1 - \pi}{\pi}, \qquad \pi = \frac{1}{N} \sum_j D_{J,j}$$

In words: weight each comparison upazila by its odds of having been treated, rescaled so the weights average about one. A comparison unit that looks almost exactly like a Jamuna upazila gets a large weight; one that looks nothing like a Jamuna upazila gets a small one.

pi = D.mean()
s["ipw1"] = np.where(D == 1, 1.0, s["p"] / (1 - s["p"]) * (1 - pi) / pi)
s["ipw3"] = np.where((s["p"] < cut) & (D == 0), np.nan, s["ipw1"]) # trimmed

The line np.where(D == 1, 1.0, ...) is not a formatting convenience. Treated units get a weight of exactly one because we want the effect on the treated: the treated distribution is the target, and only the comparison group is reshaped to match it. Weighting the treated units by $1/p$ instead would target the ATE. That single line is what makes these ATT weights.

11.4 KOBDR: Kline’s Oaxaca-Blinder reweighting

$$w_i^{KOB} = \frac{1 - D_{J,i}}{N_1} \left( \sum_{j : D_{J,j} = 1} X_j \right)^{\prime} \left( \sum_{j : D_{J,j} = 0} X_j X_j^{\prime} \right)^{-1} X_i$$

In words: run the outcome regression on the comparison upazilas only, then evaluate it at the average treated covariate profile. Kline (2011) showed that this two-step Oaxaca-Blinder procedure is algebraically identical to taking a weighted average of the comparison outcomes, and this is the weight it implies.

n1, Xm, nD = D.sum(), X.to_numpy(float), 1.0 - D
ob = ((D @ Xm) @ np.linalg.inv(Xm.T @ (Xm * nD[:, None])) @ Xm.T / n1) * nD * n1
s["ipw2"] = np.where(D == 1, 1.0, np.where(ob < 0, np.nan, ob)) # negatives dropped
s["ipw4"] = np.where((s["p"] < cut) & (D == 0), np.nan, s["ipw2"]) # trimmed
print(f" [nightlights] negative Oaxaca-Blinder weights: {int((ob < 0).sum())} obs dropped")
# Merge the four weight columns back onto the estimation sample. `analysis.py`
# wraps the twelve lines above into build_weights() and runs it once per family:
# a 5 percent trim for nightlights and the census, none for the yield panel,
# where sixteen districts leave nothing to trim.
NL = NL.merge(s[["geocode", "year", "p", "ipw1", "ipw2", "ipw3", "ipw4"]],
on=["geocode", "year"], how="left")
EMP = EMP.merge(build_weights(emp[emp["smp1"].notna()], trim_pct=5)
.groupby("geocode")[["p", "ipw1", "ipw2", "ipw3", "ipw4"]]
.first().reset_index(), on="geocode", how="left")
YLD = YLD.merge(build_weights(yld[yld["smp1"].notna()], trim_pct=None)
.groupby(["dist", "year"])[["p", "ipw1", "ipw2"]]
.first().reset_index(), on=["dist", "year"], how="left")

 [nightlights] negative Oaxaca-Blinder weights: 35 obs dropped

Think of it as casting understudies. A director needs to stage the Jamuna hinterland but may only use Padma actors. She writes down the profile of the average Jamuna upazila — this much 1991 population, this far from a bridge foot — and asks what blend of Padma actors reproduces that profile exactly. The blend proportions are the Oaxaca-Blinder weights.

Two things follow immediately. The blend is chosen to match the treated average, which is again what makes this an ATT estimator. And nothing in the algebra forces the proportions to be positive: sometimes the best way to hit the target is to weight one actor at 1.4 and another at $-0.4$. Negative casting is not interpretable, so those units are dismissed — 35 upazila-periods here.

11.5 Does the reweighting actually work?

nlp = NL.drop_duplicates("geocode") # one row per upazila; weights are time-invariant
for var in ["lpop91", "lmdist"]:
t = nlp.loc[nlp.treat == 1, var]
for lab, wcol in [("unweighted", None), ("LWDR", "ipw3"), ("KOBDR", "ipw4")]:
c = nlp[nlp.treat == 0].dropna(subset=[var] + ([wcol] if wcol else []))
w = np.ones(len(c)) if wcol is None else c[wcol].to_numpy(float)
cm = np.average(c[var], weights=w)
sd = np.sqrt((t.var() + c[var].var()) / 2)
print(f" {var:8s} {lab:11s} std. diff {(t.mean() - cm) / sd:+.4f}")

 lpop91 unweighted std. diff +0.0969
lpop91 LWDR std. diff -0.0476
lpop91 KOBDR std. diff -0.0127
lmdist unweighted std. diff +0.4041
lmdist LWDR std. diff +0.0793
lmdist KOBDR std. diff +0.0541

Figure 10. Standardised treated-comparison differences in the two covariates, unweighted and under each weighting scheme. Distance starts at 0.404, well above the conventional 0.10 threshold, and KOBDR cuts it to 0.054.

This is the clearest evidence that the reweighting does what it claims. The distance covariate starts badly imbalanced at 0.404 — well above the conventional 0.10 threshold — because Jamuna upazilas sit systematically farther from their bridge foot than Padma upazilas do from theirs. KOBDR cuts that to 0.054, and the population imbalance to 0.013. Both move from clearly imbalanced to clearly balanced.

Figure 11. Left: propensity-score overlap between the two hinterlands, with the 5 percent trim line. Right: the Oaxaca-Blinder weight against the logit-odds weight for each comparison upazila; the two schemes correlate at 0.971.

The left panel is the overlap check, and it is unusually healthy: the two propensity distributions sit almost on top of each other with a median near 0.50. This is the quantitative version of the claim that the two hinterlands are hard to tell apart. The right panel shows the two weighting schemes agree closely — they correlate at 0.971 — which is why the LWDR and KOBDR columns land within 0.003 of each other on the nightlights and census panels. On the nine-cluster yield panel, where every estimate is noisier, the two drift as far apart as 0.009.

11.6 Running the weights through diff-diff

diff-diff accepts external weights through a SurveyDesign object. Setting weight_type="aweight" reproduces Stata’s analytic weights, and psu sets the clustering unit.

e4 = NL[NL["ipw4"].notna()].copy()
dums = pd.get_dummies(e4["year"], prefix="yd", drop_first=True).astype(float)
for c in dums.columns:
e4[c] = dums[c].to_numpy()
res_dr = DifferenceInDifferences(cluster="geocode").fit(
e4, outcome="lmn", treatment="treat", time="post",
covariates=CONTROLS + list(dums.columns), absorb=["geocode"], unit="geocode",
survey_design=SurveyDesign(weights="ipw4", weight_type="aweight", psu="geocode"))
print(res_dr)

DiDResults(ATT=0.1088***, SE=0.0230, p=0.0000)

One implementation detail is worth knowing before you hit it. diff-diff refuses to absorb two fixed-effect dimensions at once when survey weights are supplied — weighted sequential demeaning is not the same operation as unweighted sequential demeaning, and the library declines to pretend otherwise. The workaround is to absorb the unit and pass explicit year dummies as covariates, which is what the code above does.

11.7 Three engines, one answer

Since we now have three ways to run the same regression, run all three.

# All 24 mean-effect specifications: four outcome families x three estimators.
# The yield panel has no propensity trim, so its weights are ipw1 / ipw2.
SPECS = [(NL, "lmn", "geocode", "ipw3", "ipw4"), (NL, "D_lmn", "geocode", "ipw3", "ipw4"),
(EMP, "ldensity", "geocode", "ipw3", "ipw4"), (EMP, "sind", "geocode", "ipw3", "ipw4"),
(EMP, "sserv", "geocode", "ipw3", "ipw4"), (EMP, "sagr", "geocode", "ipw3", "ipw4"),
(YLD, "lyld", "dist", "ipw1", "ipw2"), (YLD, "D_lyld", "dist", "ipw1", "ipw2")]
comparison = []
for data, y, unit, w_lw, w_kob in SPECS:
for est, wcol in [("OLS", None), ("LWDR", w_lw), ("KOBDR", w_kob)]:
sub = data if wcol is None else data[data[wcol].notna()]
manual = stata_fe(data, y, ["treat_post", "post"] + CONTROLS,
unit=unit, time="year", weight=wcol)
fx = pf.feols(f"{y} ~ treat_post + post + lpop91_t + lrainm + lrainsd + lmdist_t"
f" | {unit} + year", data=sub, weights=wcol, vcov={"CRV1": unit})
dd = diffdiff_mean(data, y, unit, wcol)
comparison.append((est,
(manual["coef"]["treat_post"], manual["se"]["treat_post"]),
(float(fx.coef()["treat_post"]), float(fx.se()["treat_post"])),
(dd[0], dd[1])))
print(f" {len(comparison)} specifications compared across three engines.")
def spread(triple, i):
a, b, c = (t[i] for t in triple)
return max(abs(a - b), abs(a - c), abs(b - c))
print(" Largest coefficient disagreement across the three engines: "
f"{max(spread(row[1:], 0) for row in comparison):.6f}")
print(" Largest standard-error disagreement: "
f"{max(spread(row[1:], 1) for row in comparison):.6f}")

 24 specifications compared across three engines.
Largest coefficient disagreement across the three engines: 0.000000
Largest standard-error disagreement: 0.024989

Figure 12. Coefficients and standard errors from diff-diff and pyfixest against the Stata-identical estimator, all 24 mean-effect specifications. The coefficients agree to nine decimals; the standard errors separate on the weighted rows, where diff-diff switches to a design-based variance.

All three engines return the same point estimate to nine decimal places across all 24 mean-effect specifications. The standard errors are a different and more interesting story. pyfixest matches the Stata recipe to within 0.0007 everywhere. diff-diff matches on the unweighted specifications but diverges on the weighted ones, because supplying a SurveyDesign switches it to a design-based Taylor-linearisation variance rather than the classical cluster-robust sandwich.

The gap is usually small — the median ratio is 0.999 — but the extreme case is instructive. On the rice-yield panel with nine clusters, diff-diff reports 0.034 where the Stata recipe reports 0.023, a standard error 50 percent larger. Neither is a bug. They are two defensible variance conventions disagreeing precisely where the asymptotics are thinnest, and when a design has nine clusters you should probably prefer the more conservative one.

12. Results I: nighttime lights

for est, wcol in [("OLS", None), ("LWDR", "ipw3"), ("KOBDR", "ipw4")]:
r = stata_fe(NL, "lmn", ["treat_post", "post"] + CONTROLS,
unit="geocode", time="year", weight=wcol)
print(f" {est:6s} {r['coef']['treat_post']:+.4f} ({r['se']['treat_post']:.4f})"
f" N={r['n']} upazilas={r['g']}")

 OLS +0.0881 (0.0220) N=1729 upazilas=247
LWDR +0.1059 (0.0222) N=1673 upazilas=239
KOBDR +0.1088 (0.0223) N=1673 upazilas=239

Averaged over the whole post-bridge period, the bridge raised nighttime luminosity by 10.9 percent under the preferred doubly robust estimator. Both reweighted estimates are larger than the unweighted one, by about two percentage points — reweighting toward comparison upazilas that resemble the treated ones raises the estimated effect rather than deflating it, which is the opposite of what people often expect adjustment to do.

The sample falls from 247 to 239 upazilas when the weights are applied: eight comparison upazilas are lost to the propensity trim and to negative Oaxaca-Blinder weights. That is a 3 percent reduction, small enough not to worry about and large enough to report.

13. Results II: population density and employment shares

13.1 Mean effects

Figure 13. Mean post-bridge effect for every outcome under all three estimators, with 95 percent confidence intervals.

Outcome	OLS	LWDR	KOBDR	N	Upazilas
Log nightlights	0.088 (0.022)	0.106 (0.022)	0.109 (0.022)	1673	239
Nightlights growth	0.016 (0.016)	0.032 (0.016)	0.033 (0.016)	1434	239
Log population density	0.032 (0.015)	0.025 (0.015)	0.025 (0.015)	714	238
Industry empl. share	−0.010 (0.004)	−0.009 (0.004)	−0.010 (0.004)	714	238
Services empl. share	0.017 (0.006)	0.022 (0.005)	0.023 (0.005)	714	238
Agriculture empl. share	−0.008 (0.007)	−0.013 (0.007)	−0.013 (0.007)	714	238
Log rice yield	0.049 (0.031)	0.059 (0.026)	0.063 (0.023)	72	9
Rice yield growth	−0.042 (0.085)	0.049 (0.049)	0.053 (0.049)	63	9

The services share rose 2.3 percentage points and the industry share fell 1.0. That industry number looks negligible until you check the base: the 1991 manufacturing share in the treated hinterland was 2.8 percent, so a 1.0 point fall removes roughly a third of the sector. Deindustrialisation is real here, and it is not small.

Population density comes out at $+2.5$ percent and is not distinguishable from zero ($p = 0.10$). Hold that thought, because the null is an artefact of averaging.

13.2 The discriminating test

for y in ["ldensity", "sind", "sserv", "sagr"]:
r = stata_fe(EMP, y, ["treat_sr", "treat_lr"] + CONTROLS,
unit="geocode", time="year", weight="ipw4")
print(f" {y:9s} SR {r['coef']['treat_sr']:+.4f} ({r['se']['treat_sr']:.4f})"
f" LR {r['coef']['treat_lr']:+.4f} ({r['se']['treat_lr']:.4f})")

 ldensity SR -0.0248 (0.0142) LR +0.0590 (0.0164)
sind SR -0.0060 (0.0049) LR -0.0120 (0.0047)
sserv SR +0.0204 (0.0052) LR +0.0242 (0.0076)
sagr SR -0.0144 (0.0061) LR -0.0122 (0.0089)

Figure 14. Short-run against long-run effects under KOBDR. Population density is the row to read: negative in the short run, positive and significant in the long run.

Here is the answer to the question the post opened with. Population density falls 2.5 percent in the short run and rises 5.9 percent in the long run — a genuine sign reversal that the pooled mean effect averaged into an uninformative $+2.5$ percent. In the years right after the bridge, people left; over the following decade, more came than had left.

Now apply the discriminating test. The manufacturing share falls 1.2 percentage points in the long run, which both backwash and comparative advantage predict. But backwash requires the region to be emptying, and the density coefficient is $+0.059$ with a standard error of $0.016$ — positive and significant at the 0.1 percent level. The region gained people while losing factories.

That combination is what the core-periphery model cannot produce and the comparative-advantage story predicts directly. The Jamuna hinterland did not decline; it specialised.

The services result adds the mechanism. The share rises 2.0 points in the short run and 2.4 in the long run, and services here means trading, transport and processing — the activities that a region takes on when it starts shipping its agricultural output somewhere. The short-run agriculture decline of 1.4 points, which partly reverses later, looks like overshooting: labour left farming faster than was sustainable when migration was still costly.

14. Results III: rice yields

es_yld, es_yld_res = event_study(YLD, "lyld", "dist", [4, 5, 6, 7, 8], 3,
"Rice yield", YLD_YEARS)

 period label effect se is_post
1 1988-91 -0.0605 0.0954 False
2 1992-94 0.0622 0.0670 False
3 1995-97 0.0000 0.0000 False
4 1998-00 -0.0422 0.0523 True
5 2001-04 -0.0613 0.0722 True
6 2005-07 0.0329 0.0648 True
7 2008-10 0.0544 0.0716 True
8 2011-13 0.0733 0.0688 True

Figure 15. Event studies for rice yield, population density and the services employment share. The yield effect is negative for two post-bridge windows before turning; its wide pre-period bands are a warning about how little a nine-cluster panel can rule out.

Rice yields are $+1.2$ percent ($p = 0.64$) in the short run and $+7.9$ percent ($p = 0.001$) in the long run. The event study shows why: the effect is actually negative for the first two post-bridge windows and only turns positive from 2005-07.

The delay has a documented mechanism. Average fertiliser prices in the Jamuna hinterland were 9 percent below the Padma hinterland during 2006-2009 and 13 percent below during 2010-2013 — the input distribution networks took years to reorganise around the new road. Add credit constraints that only relax as crop prices improve, and the short-run labour outflow reducing land productivity, and a decade-long lag is unsurprising.

This event study is also the honest counterweight to the nightlights one. Its two pre-bridge coefficients are $-0.061$ (0.095) and $+0.062$ (0.067). Both are statistically insignificant, but with standard errors that wide the test has very little power to detect a pre-trend even if one existed. With nine clusters there is not much this panel can rule out, and it would be wrong to present its clean pre-period as strong evidence.

15. Results IV: distance from the bridge

The average effect turns out to hide almost everything interesting. Split the sample into distance terciles — recomputed within the estimation sample, exactly as the original code does.

Where the cut happens is not a detail. nite_2021.do runs its xtile on the whole smp1 sample — that is, on nl, before any row is lost to a missing control — while employment_2021.do drops first and cuts afterwards. Two upazilas sit close enough to a boundary that the order decides which band they land in, which is enough to move every nightlights coefficient in this section in the third decimal. So the nightlights bands are cut on nl and mapped onto NL; the other two are cut in place.

# Nightlights: cut on the full frame, BEFORE rows are lost to missing controls.
bands = pd.qcut(nl.loc[nl["smp1"].notna(), "lmdist"], 3, labels=["near", "mid", "far"])
band_map = (nl.loc[nl["smp1"].notna(), ["geocode"]].assign(band=bands.to_numpy())
.drop_duplicates("geocode").set_index("geocode")["band"])
NL["band"] = NL["geocode"].map(band_map)
# Census and yield: their do-files drop first, so cut on the estimation sample.
for D_ in (EMP, YLD):
D_["band"] = pd.qcut(D_["lmdist"], 3, labels=["near", "mid", "far"])
for D_ in (NL, EMP, YLD):
for b in ["near", "mid", "far"]:
D_[f"tsr_{b}"] = D_["treat"] * D_["sr"] * (D_["band"] == b)
D_[f"tlr_{b}"] = D_["treat"] * D_["lr"] * (D_["band"] == b)
print(EMP.groupby("band", observed=True)["mdist"].agg(["min", "max", "count"]).round(2))

 min max count
band
near 8.44 83.42 246
mid 83.73 128.15 246
far 128.33 269.47 246

Figure 16. Short-run and long-run effects by distance tercile, all outcomes. Read the agriculture and services rows across: the average effect reverses sign between the nearest and the farthest band.

Long-run effects by band:

Outcome	Nearest (<84 km)	Middle (84-128 km)	Farthest (128-270 km)
Log rice yield	0.049 (0.023)	0.065 (0.022)	0.265 (0.025)
Rice yield growth	0.025 (0.036)	0.011 (0.034)	0.344 (0.091)
Log nightlights	0.026 (0.034)	0.149 (0.040)	0.102 (0.038)
Log population density	0.069 (0.024)	0.005 (0.029)	0.093 (0.024)
Industry empl. share	−0.006 (0.010)	−0.025 (0.007)	−0.001 (0.008)
Services empl. share	−0.026 (0.013)	0.017 (0.012)	0.059 (0.016)
Agriculture empl. share	0.032 (0.015)	0.008 (0.014)	−0.057 (0.017)

Read the bottom two rows across, and the average effect reverses sign. In the nearest band labour moves into agriculture ($+3.2$ points) and out of services ($-2.6$). In the farthest band it moves the other way and three times harder: agriculture $-5.7$, services $+5.9$. Rice yields in the farthest band rise 26.5 percent, four times the middle band and five times the nearest. The manufacturing decline is not spread evenly either — it sits almost entirely in the middle band at $-2.5$ points.

This looks backwards at first. The upazilas nearest the bridge got the largest proportional cut in travel time, roughly 40 percent, against about 17 percent at the far end. Why do the distant ones gain more?

Think about two discounts. A 40 percent cut on a ten-dollar taxi saves four dollars; a 17 percent cut on a five-hundred-dollar flight saves eighty-five. The percentage is smaller, the base is enormous, and the saving is much larger. Upazilas near the bridge foot were already reasonably connected — the ferry was an inconvenience, not a wall. Upazilas 250 kilometres out were close to autarky, where fertiliser rarely arrived and rice rarely left, and where a modest proportional cut on a very high delivered cost is still a very large absolute cut. Trade responds to the level of the barrier, not to the percentage change in it.

This is also why the nearest band moves labour into farming. Those upazilas are close to Dhaka with good onward links, so what they gained access to was the high-value fruit, flower and vegetable market — agriculture, but not the kind that shows up as subsistence rice.

The practical lesson is blunt. An evaluation reporting only the average would tell a minister to build near the demand centre. The heterogeneity says the payoff was at the end of the line.

16. Validation and robustness

16.1 Placebo timing and randomisation inference

pl = placebo_timing_test(NL, outcome="lmn", treatment="treat", time="year",
fake_treatment_period=2, post_periods=[3, 4, 5, 6, 7],
cluster="geocode")
print(f" placebo effect = {pl.placebo_effect:+.5f} (se {pl.se:.5f}), "
f"p = {pl.p_value:.4f}, significant = {pl.is_significant}")
print(f" for comparison, the real effect is {pl.original_effect:+.5f} "
f"(se {pl.original_se:.5f})")

 placebo effect = +0.00844 (se 0.01025), p = 0.4106, significant = False
for comparison, the real effect is +0.06409 (se 0.01850)

Move the bridge one period earlier, into a window when it did not exist, and the estimated effect collapses from $+0.064$ to $+0.008$ and loses significance. If some slow-moving regional divergence were driving the result, this test would find it.

rng = np.random.default_rng(RANDOM_SEED)
units = NL.drop_duplicates("geocode")[["geocode", "treat"]]
null = []
for _ in range(500):
perm = units.assign(ptreat=rng.permutation(units["treat"].to_numpy()))
tmp = NL.merge(perm[["geocode", "ptreat"]], on="geocode")
tmp["treat_post"] = tmp["ptreat"] * tmp["post"]
null.append(stata_fe(tmp, "lmn", ["treat_post", "post"] + CONTROLS,
unit="geocode", time="year")["coef"]["treat_post"])
true = 0.0881
share = np.mean(np.abs(null) >= abs(true))
print(" Randomisation inference over 500 placebo assignments:")
print(f" true estimate {true:+.4f}; placebo |effect| >= |true| in {share:.1%} of draws")

 Randomisation inference over 500 placebo assignments:
true estimate +0.0881; placebo |effect| >= |true| in 0.0% of draws

Figure 17. The null distribution from 500 random reassignments of treatment, with the actual estimate marked. Not one placebo draw reaches it.

Randomly reassign which upazilas count as “treated”, re-estimate, and repeat 500 times. The resulting null distribution is centred on zero and not one draw reaches the magnitude of the real estimate, giving a randomisation p-value below 1/500. This inference makes no asymptotic assumptions at all, which is worth having alongside the cluster-robust standard errors.

16.2 HonestDiD: how much violation can it survive?

Passing the pre-trend test is a low bar. The more useful question is how badly parallel trends would have to fail before the conclusion changes.

for M in [0.0, 0.25, 0.5, 1.0, 1.5, 2.0]:
h = compute_honest_did(ev, method="relative_magnitude", M=M)
verdict = "includes zero" if h.ci_lb <= 0 <= h.ci_ub else "excludes zero"
print(f" M={M:<4} CI = [{h.ci_lb:+.4f}, {h.ci_ub:+.4f}] {verdict}")

 M=0.0 CI = [+0.0236, +0.0967] excludes zero
M=0.25 CI = [+0.0174, +0.1028] excludes zero
M=0.5 CI = [+0.0113, +0.1090] excludes zero
M=1.0 CI = [-0.0010, +0.1212] includes zero
M=1.5 CI = [-0.0132, +0.1335] includes zero
M=2.0 CI = [-0.0255, +0.1458] includes zero

Figure 18. Rambachan-Roth relative-magnitude bounds for the nightlights effect. The confidence set widens as the allowed post-treatment violation M grows; the breakdown value sits just under M = 1.

The Rambachan-Roth relative-magnitude bounds allow the post-treatment violation of parallel trends to be up to $M$ times the largest violation observed before treatment, and report the widest confidence set consistent with that. The breakdown value here sits just under $M = 1$.

In plain terms: the post-bridge violation would have to be as large as the largest pre-bridge violation for the nightlights result to become inconclusive. That is a moderate robustness margin, not a spectacular one, and it is better to say so than to dress it up. A result that survived to $M = 3$ would be much stronger; a result that broke at $M = 0.3$ would be fragile. This one sits in between.

16.3 The public-goods placebo

The most serious alternative explanation is political rather than economic. A prime minister with roots in the Jamuna hinterland might simply have sent more schools, clinics and electricity there, and the “bridge effect” could be a public-spending effect wearing a disguise.

VILL_VARS = ["dist_Thana", "dist_district", "dist_satellite_clinic", "dist_hos",
"primary_school", "high_school", "madrassa_school",
"grameen_bank", "cinema", "post_office", "co_operative_soc", "NGO"]
hits = n = 0
for data, unit, outcomes in [(HH, "District", ["Electricity"]),
(VILL, "District", VILL_VARS)]:
for v in outcomes:
r = stata_fe(data, v, ["treat_sr", "treat_lr", "lmdist_t"],
unit=unit, time="year")
for h in ("treat_sr", "treat_lr"):
if h in r["p"] and np.isfinite(r["p"][h]):
n += 1
hits += r["p"][h] < 0.05
print(f" Public-goods outcomes significant at 5%: {hits} of {n} estimates")

 Public-goods outcomes significant at 5%: 0 of 21 estimates

Figure 19. t-statistics for all 21 public-goods estimates, with the 5 percent critical values marked. None crosses. If the bridge effect were really a public-spending effect, its fingerprints would be here.

Twenty-one estimates across household electricity access and eleven village infrastructure measures, and not one is significant at 5 percent. The closest is the long-run distance to a high school at $+0.535$ ($0.290$, $p = 0.065$) — and it has the wrong sign for the story, since it says schools got farther away in treated villages.

This is a well-designed placebo. It is not a test of the outcome we care about; it is a test of a specific rival mechanism that would produce the same headline result for a different reason. Good robustness checks name the alternative explanation and go looking for its fingerprints.

17. Reproduction audit

Every headline coefficient in the paper’s four main tables was re-estimated and compared with the authors’ own Stata output. analysis.py writes that comparison to a CSV as it goes; here it is, summarised.

audit = pd.read_csv("python_bridge_impact_audit_reproduction.csv")
n_tot = len(audit)
n_coef = int(audit["match_coef"].sum()) # coefficient matches
n_exact = int((audit["match_flag"] == "exact").sum()) # coefficient AND se match
print(f" {n_coef} of {n_tot} coefficients reproduce to the printed precision "
f"({n_coef / n_tot:.1%})")
print(f" {n_exact} of {n_tot} reproduce both the coefficient and the standard error "
f"({n_exact / n_tot:.1%})")

 122 of 122 coefficients reproduce to the printed precision (100.0%)
113 of 122 reproduce both the coefficient and the standard error (92.6%)

Figure 20. All 122 published coefficients and standard errors against the replication, with the 45-degree line. Every coefficient lands on it; nine standard errors sit fractionally off, all in the thinnest panels.

All 122 coefficients across Tables 1, 2, 3 and 4 reproduce to the printed three decimals, with a maximum absolute deviation of 0.0005 — inside the tolerance implied by three-decimal rounding. The nine cells that match on the coefficient but not the standard error differ in the third decimal by between 0.0006 and 0.0013, all in the nine-cluster yield panel and the two nightlights growth specifications.

Getting to 122 out of 122 took care in three places worth naming, because each of them is a trap that any replication of a Stata paper can fall into:

ln(0) must become missing, not negative infinity. Twenty-four employment rows have zero recorded rainfall. Stata’s ln() returns missing and the row drops; NumPy returns -inf and the row survives, corrupting the sample.
The degrees-of-freedom correction counts only the non-absorbed regressors. Stata’s xtreg, fe uses $\frac{G}{G-1} \cdot \frac{N-1}{N-K}$ with $K$ excluding the fixed effects. Counting them inflates every standard error by roughly 20 percent.
Distance terciles are computed at different points in different do-files. nite_2021.do builds them before dropping rows with missing controls; employment_2021.do drops first. Getting that order wrong shifts every nightlights heterogeneity coefficient in the third decimal — it was the last discrepancy resolved here.

18. Notes from inside the replication package

Replication is not only about confirming numbers. Working through someone else’s code teaches you things that reading their paper cannot, and the Jamuna package has one lesson in it that is worth more than the rest of this section combined.

The authors deserve credit before any of this: they published a complete package — four do-files, five datasets, the logs, and every intermediate table. Almost nothing below would be knowable otherwise. That is the point.

18.1 The macro that was never defined

employment_2021.do opens with global trimL 5. nite_2021.do does not — but it still contains the line gen cut11 = r(p$trimL).

graph LR
A("<code>global trimL</code> never defined<br/>in nite_2021.do") --> B("<code>gen cut11 = r(p$trimL)</code><br/>expands to <code>r(p)</code>,<br/>which does not exist")
B --> C("cut11 is missing for<br/>all 1,743 observations")
C --> D("<code>replace ipw4 = . if p &lt; cut11</code><br/>in Stata any number is less<br/>than missing, so this is TRUE<br/>for every comparison unit")
D --> E("Every comparison upazila<br/>loses its weight")
E --> F("The regression runs on<br/>treated units only<br/>N = 868, 124 upazilas")
F --> G("<b>treat_yr = 1.064, se 0.710</b><br/>an unidentified number<br/>that still prints")
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class A,B,C,D orange
class E,F,G anchor

We can reproduce both branches exactly:

# The bug: an undefined macro means the cutoff is missing, and in Stata
# every real number is smaller than a missing value.
NL_BUG = NL.copy()
NL_BUG["ipw4_bug"] = np.where(NL_BUG["treat"] == 1, 1.0, np.nan)
for label, data, wcol in [("published (trim = 5th pctile)", NL, "ipw4"),
("as shipped (trimL undefined)", NL_BUG, "ipw4_bug")]:
for y in ("lmn", "D_lmn"):
r = stata_fe(data, y, ["treat_post", "post"] + CONTROLS,
unit="geocode", time="year", weight=wcol)
print(f" {label:32s} {y:7s} {r['coef']['treat_post']:+.4f} "
f"({r['se']['treat_post']:.4f}) N={r['n']:5d} upazilas={r['g']}")

 published (trim = 5th pctile) lmn +0.1088 (0.0223) N= 1673 upazilas=239
published (trim = 5th pctile) D_lmn +0.0326 (0.0163) N= 1434 upazilas=239
as shipped (trimL undefined) lmn +1.0636 (0.7097) N= 868 upazilas=124
as shipped (trimL undefined) D_lmn -0.5193 (0.2861) N= 744 upazilas=124

Figure 21. The published nightlights estimate against the one the shipped do-file actually produces. The coefficient is ten times larger and the standard error thirty times larger — but the tell is the sample: 124 upazilas where there should be 239.

The reproduction lands on 1.0636 (0.7097) against the archived nlite_mean.txt value of 1.064 (0.710), on 868 observations and 124 upazilas — an exact match to the degenerate output sitting in the package. The published paper carries the correct numbers, which the package also contains in a parallel set of files named nlite2_*. So the bug never reached print; it survives only in the shipped code.

Every step in that chain is legal Stata. Nothing warns. The comparison group does not vanish with an error — it dissolves into missing values, and the regression cheerfully estimates a within-treated-group time contrast and calls it a treatment effect.

The tell is not in the coefficient, which is merely large. It is in the footer: 124 upazilas where there should be 239. The first thing to read in any regression output is the sample size. If you take one habit from this post, take that one.

18.2 A one-row shift in published Table 3

Comparing the published Table 3 against results/did_vill.txt shows the coefficient column slipping one row down from “Hospitals” onward. The paper dropped two rows — satellite clinics and madrassa schools — from the printed table but did not drop their coefficients.

The printed “Cooperatives” short-run estimate of 0.420 (0.263) is in fact post_office. The true co_operative_soc short-run estimate is 0.090 (0.108). The N column follows the correct labels while the coefficients follow the original positions, so the misalignment is visible by cross-checking the two.

No conclusion changes, because every estimate in the block is insignificant either way. But it is a good reminder that a replication which reproduced the printed table rather than the underlying output would have concluded, wrongly, that it had failed.

18.3 Where the text and the table disagree

Two smaller inconsistencies, both in the article rather than the code.

Section 8.1.2 states that long-run agricultural productivity gains are “strongest in the intermediate distance from the bridge”. Table 4 shows the farthest band at 0.265 against the middle band’s 0.065 — the farthest band dominates by a factor of four, and our replication confirms it.

Section 7.3 describes the long-run effect on total agricultural labour as “a numerically small and statistically significant impact”. The estimate is $-0.017$ with a standard error near $0.021$, and the surrounding sentence — which says agricultural labour “gained back most of its lost ground” — only makes sense if the word should be insignificant.

18.4 What replication is for

None of the four items above changes a single conclusion of the paper. The bridge still raised luminosity, yields and services employment; density still rose; backwash is still rejected. That is the honest summary.

But notice what made each of them findable. The $trimL bug is visible because the authors shipped both the buggy output and the corrected output. The Table 3 shift is visible because they shipped did_vill.txt. The text-table inconsistencies are visible because the tables are reproducible from the data.

A paper that published only its conclusions would be opaque on all four counts, and a reader would have no way to tell an honest slip from a substantive error. The correct reaction to this section is not “the paper is unreliable”; it is that this paper is unusually checkable, and that checkability is what made a 122-of-122 reproduction possible at all.

19. Discussion

The bridge worked, and it worked in a way that neither of the two textbook predictions anticipated.

Nighttime lights rose 10.9 percent on average and 11.2 percent in the long run. Rice yields rose 7.9 percent in the long run. The services employment share rose 2.4 points. Population density fell 2.5 percent in the short run and then rose 5.9. Manufacturing’s share fell 1.2 points — a third of a small sector.

Read the manufacturing number alone and you would write the backwash story. Read it alongside population density and you cannot: a region being hollowed out by its metropolitan neighbour does not gain residents. The pattern is what you get when a place stops making things it was never especially good at and starts doing more of what it was — growing rice, and moving, processing and trading what it grows.

The spatial results carry the sharper policy lesson. Almost everything interesting happens away from the bridge. Yields in the farthest tercile rise 26.5 percent against 4.9 percent nearest; services employment rises 5.9 points farthest and falls 2.6 points nearest. An evaluation that stopped at the average effect — or worse, that studied only the districts adjacent to the bridge, which is the intuitive place to look — would have produced a materially misleading answer.

Three limitations deserve to be stated plainly.

Displacement. If the long-run density and luminosity gains partly reflect people leaving the still-isolated Padma hinterland, the comparison group is contaminated downward and these estimates are upper bounds. The authors say so themselves. It does not rescue the backwash story, which requires the treated region to lose people, but it does mean the national welfare gain is smaller than the regional one — one region’s gain is partly another’s loss, even as regional inequality is magnified.

Thin clusters. The entire rice-yield result rests on nine to eleven former districts. Cluster-robust inference with nine clusters is fragile, and this is precisely where the two libraries’ standard errors diverged most (0.023 against 0.034). Treat the yield magnitudes as indicative.

One pre-period for the census outcomes. Population density and the employment shares — including the density variable that settles the theoretical question — have exactly one pre-bridge observation. No pre-trend test is possible for them, only a level-balance test. The nightlights and yield panels support pre-trend testing and pass it, which is reassuring by association but is not the same as testing the outcome that does the work.

20. Summary and next steps

Difference-in-differences is two subtractions. Everything else — fixed effects, controls, reweighting — is a refinement of four group means, and it is worth computing those four numbers by hand before running any estimator.
The assumption is about trends, not levels. Parallel trends permits the treated group to start anywhere; it requires only that it would have moved the same way. It is untestable in principle, which is why the post spends more effort bounding violations than testing for them.
An event study is a test and a result at once. The pre-treatment coefficients check the assumption; the post-treatment ones trace the effect. The nightlights event study — flat before, monotone climb after — carries more conviction than any single pooled coefficient.
Doubly robust means two chances, not immunity. Weighting protects you if the treatment model is right; regression adjustment protects you if the outcome model is right. Neither protects against a confounder you never measured.
Averages hide reversals. Population density was insignificant on average because it was negative then positive. Services employment was positive on average because a large gain far from the bridge outweighed a loss near it. Split by time and by space before believing a null.
Read the sample size first. The most instructive thing in the replication package is a bug that changed a coefficient from 0.109 to 1.064 without producing a single warning, and whose only visible symptom was 124 upazilas in a table that should have shown 239.

Where to go next. The design here is a clean two-group, single-date DiD, so heterogeneity-robust staggered estimators — Callaway and Sant’Anna, Sun and Abraham, and the imputation approaches — are not needed. They become essential the moment treatment timing varies across units, and diff-diff implements all of them: CallawaySantAnna, SunAbraham, ImputationDiD, StackedDiD. A natural extension of this analysis is synthetic control (SyntheticControl, SyntheticDiD), which would build a weighted combination of Padma upazilas to match each Jamuna upazila’s pre-bridge luminosity path rather than reweighting on two covariates. And the spatial dimension invites a spillover-aware design: with SpilloverDiD and the spatial HAC variance in diff_diff.conley, one could ask whether the comparison hinterland was affected at all — the displacement concern from section 19, tested rather than assumed.

21. Exercises

Change the clustering level. Re-run the mean-effect nightlights DiD clustering on dist rather than geocode. Does the standard error on treat_post rise or fall from 0.022? Which level is defensible, and what does the answer imply about the significance stars in Table 1?
Interrogate the plus one. The outcome is $\ln(mn + 1)$. Recompute the KOBDR mean effect with $\ln(mn + 0.01)$ and $\ln(mn + 5)$. How much of the 10.9 percent headline depends on that constant, and which upazilas drive the difference?
Reproduce the bug on purpose. Set the trimming cutoff so that every comparison unit fails it, and confirm you recover 1.064 (0.710) on 868 observations and 124 upazilas. Then write one sentence saying what that 1.064 is actually estimating.
Trim sensitivity. Re-estimate the nightlights mean effect trimming at 1, 5, 10 and 20 percent. Plot the KOBDR coefficient and its confidence interval against the trim fraction. At what point, if any, does the effect stop being significant, and how many comparison upazilas remain?
Two roads to the same number. Fit MultiPeriodDiD on ldensity with the three census years and 1991 as reference. Show that the two period effects equal the short-run and long-run coefficients of $-0.025$ and $+0.059$. Then try the same on the nightlights panel and explain why they do not coincide there.
Redefine the bands. The terciles pool treated and comparison upazilas on distance to the nearer bridge foot. Recut them using only the treated upazilas’ distance to the Jamuna foot, assigning each comparison unit to its nearest treated neighbour’s band. Does the farthest-band long-run yield effect of 26.5 percent survive?
Stress the “doubly”. Rebuild both weight vectors with log mean rainfall added as a third covariate, and report how far the mean effects move. Then break the outcome model by dropping lmdist_t while keeping correct weights, and separately break the weights while keeping the correct outcome model. Which failure does the estimator survive, and does that match the promise of double robustness?
How much pre-trend can it take? Run compute_honest_did with method="smoothness" instead of "relative_magnitude". Does the breakdown value move? Translate the answer into a sentence a minister could act on.
Swap the treatment. Pretend the Padma hinterland was treated in 1998 and the Jamuna hinterland was the comparison, holding everything else fixed. What sign should the estimate take, and what would you conclude if the placebo came back significant with the same sign as the real estimate?
Rebuild Table 3 correctly. Using bridge_dhs_village.csv, reproduce all twelve village public-goods regressions. Show that the published “Cooperatives” short-run coefficient of 0.420 is in fact post_office, that co_operative_soc is 0.090 (0.108), and produce the corrected table. Does the paper’s conclusion change?

22. References

Blankespoor, B., Emran, M. S., Shilpi, F., & Xu, L. (2021). Bridge to bigpush or backwash? Market integration, reallocation and productivity effects of Jamuna Bridge in Bangladesh. Journal of Economic Geography. Accepted 11 May 2021.
Blankespoor, B., Emran, M. S., Shilpi, F., & Xu, L. (2018). Bridge to bigpush or backwash? Policy Research Working Paper 8508, The World Bank. https://doi.org/10.1596/1813-9450-8508
Myrdal, G. (1957). Economic Theory and Underdeveloped Regions. New York: Harper and Row.
Krugman, P. (1991). Increasing returns and economic geography. Journal of Political Economy, 99(3), 483-499. https://doi.org/10.1086/261763
Fujita, M., & Thisse, J.-F. (2002). Economics of Agglomeration: Cities, Industrial Location, and Regional Growth. Cambridge University Press.
Baldwin, R., Forslid, R., Martin, P., Ottaviano, G., & Robert-Nicoud, F. (2005). Economic Geography and Public Policy. Princeton University Press.
Kline, P. (2011). Oaxaca-Blinder as a reweighting estimator. American Economic Review, 101(3), 532-537. https://doi.org/10.1257/aer.101.3.532
Kline, P., & Moretti, E. (2014). Local economic development, agglomeration economies, and the big push: 100 years of evidence from the Tennessee Valley Authority. Quarterly Journal of Economics, 129(1), 275-331. https://doi.org/10.1093/qje/qjt034
Busso, M., Gregory, J., & Kline, P. (2013). Assessing the incidence and efficiency of a prominent place based policy. American Economic Review, 103(2), 897-947. https://doi.org/10.1257/aer.103.2.897
Robins, J. M., Rotnitzky, A., & Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427), 846-866. https://doi.org/10.1080/01621459.1994.10476818
Wooldridge, J. M. (2007). Inverse probability weighted estimation for general missing data problems. Journal of Econometrics, 141(2), 1281-1301. https://doi.org/10.1016/j.jeconom.2007.02.002
Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200-230. https://doi.org/10.1016/j.jeconom.2020.12.001
Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555-2591. https://doi.org/10.1093/restud/rdad018
Roth, J. (2022). Pretest with caution: event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305-322. https://doi.org/10.1257/aeri.20210236
Donaldson, D. (2018). Railroads of the Raj: estimating the impact of transportation infrastructure. American Economic Review, 108(4-5), 899-934. https://doi.org/10.1257/aer.20101199
Faber, B. (2014). Trade integration, market size, and industrialization: evidence from China’s National Trunk Highway System. Review of Economic Studies, 81(3), 1046-1070. https://doi.org/10.1093/restud/rdu010
Storeygard, A. (2016). Farther on down the road: transport costs, trade and urban growth in sub-Saharan Africa. Review of Economic Studies, 83(3), 1263-1295. https://doi.org/10.1093/restud/rdw020
Ahsan, R., et al. (2008). Assessment of the economic impact of the Jamuna Multipurpose Bridge. Bangladesh Bridge Authority.
World Bank (1994). Staff Appraisal Report: Bangladesh — Jamuna Bridge Project. Washington, DC: The World Bank.
DMSP-OLS Nighttime Lights Time Series, Version 4. NOAA National Centers for Environmental Information, Earth Observation Group. https://www.ncei.noaa.gov/products/dmsp-operational-linescan-system
IPUMS International, Minnesota Population Center. Bangladesh population censuses 1991, 2001 and 2011. https://international.ipums.org/international/
The DHS Program. Bangladesh Demographic and Health Surveys 1993, 1997, 2003, 2007, 2011, 2014; and Bangladesh Household Income and Expenditure Survey 1995/96, Bangladesh Bureau of Statistics. https://dhsprogram.com/
NOAA Precipitation Reconstruction over Land (PREC/L). NOAA Physical Sciences Laboratory. https://psl.noaa.gov/data/gridded/data.precl.html
diff-diff: Difference-in-Differences causal inference in Python. Documentation: https://diff-diff.readthedocs.io. Source: https://github.com/igerber/diff-diff
pyfixest: Fast high-dimensional fixed effects regression in Python. https://py-econometrics.github.io/pyfixest/

Acknowledgements

This tutorial replicates work by Brian Blankespoor, M. Shahe Emran, Forhad Shilpi and Lu Xu, whose complete and well-documented replication package made a coefficient-by-coefficient audit possible. All errors in the Python port are mine.

Bayesian Spatial Synthetic Control in Python: California's Proposition 99 with scspill and mlsynth

Mon, 03 Aug 2026 00:00:00 +0000

Abstract

Cigarette taxes leak across state lines, which means the most replicated result in applied causal inference — the effect of California’s 1988 Proposition 99 on per-capita cigarette sales — rests on an assumption the data can test and reject. Classical synthetic control requires donor weights to lie on the simplex, and requires that no donor absorb any part of the treatment; if Californians drive to Nevada for cheaper cigarettes, both requirements fail and the counterfactual is built partly out of contaminated donors. This tutorial estimates the effect three ways on one panel and asks how much of the answer each assumption was carrying. The data are the bundled Proposition 99 panel: 39 US states from 1970 to 2000, 1,209 observations, per-capita cigarette sales and real retail price, with rook-contiguity spatial weights in which Nevada is California’s only neighbour inside the donor pool. Estimation uses mlsynth.VanillaSC for the simplex baseline, mlsynth.BSCM and scspill.SCSPILL at zero spillover intensity for the Bayesian stage, and scspill.SCSPILL with method="sar" for the Bayesian spatial stage. The average treatment effect on the treated is −18.43 packs per capita per year under the simplex, −15.68 under a no-intercept horseshoe prior on unconstrained weights (−18.85 when the same prior is fitted with an intercept) and −16.87 once spillovers are modelled, with an estimated spillover intensity of 0.316 (95% credible interval 0.231 to 0.403) and a Nevada spillover of −5.50 packs per capita, 11 times the next-largest donor. The effect on California survives every relaxation; the assumption that the donor pool was clean does not.

1. Overview

In November 1988 California voters passed Proposition 99, a 25-cent-per-pack cigarette excise tax with the revenue earmarked for anti-smoking programmes. The evaluation of that policy by Abadie, Diamond and Hainmueller (2010) introduced the synthetic control method to a generation of applied economists, and it is now the single most replicated causal-inference result in the discipline. It is in every textbook. It is in every software package’s documentation. It is, in this post, in three of them.

That ubiquity makes it the right place to ask an uncomfortable question. The headline number rests on two assumptions that are rarely stated as assumptions at all:

The simplex. The synthetic California is a weighted average of other states, and the weights are required to be non-negative and to sum to one. This is a modelling choice, not a fact about the world, and it has a price that can be measured.
SUTVA on the donor pool. Every donor state is assumed to be untouched by California’s policy. Its observed cigarette sales are taken to be its no-treatment outcome.

The second assumption is the interesting one here, because Proposition 99 raised the price of a pack in California by 25 cents and did nothing to the price in Nevada. Cross-border purchasing is the obvious behavioural response, and if it happened at any scale then Nevada’s observed sales rose because of a policy Nevada never passed. Nevada would then not be a clean control but a second, oppositely-treated unit that we had mistakenly enrolled as a donor — and in the classical fit it carries a weight of about 0.24.

That is the hypothesis. The data say the leak runs the other way. Nevada’s estimated spillover is −5.50 packs per capita per year: its sales came in below what the model reconstructs as its no-treatment path, not above. Whatever net effect Proposition 99 had on Nevada, it looks like the anti-smoking campaign travelling across the border rather than the tax arbitrage travelling back. Section 12 puts the sign to a test the estimator could have failed and did not.

The direction matters for the headline number, and not in the way most readers expect. A donor whose observed series sits below its no-treatment path drags the synthetic California down with it, so the classical comparison gives the policy less credit than it deserves — the contaminated estimate understates the effect. Section 9.1 derives that as an identity; section 12 confirms it numerically.

The spatial weights shipped with the data make the channel unusually concrete. Under rook contiguity — the chess convention in which two states count as neighbours only if they share a stretch of border, not merely a corner — Nevada is the only state in the Abadie–Diamond–Hainmueller donor pool that shares a border with California. Oregon and Arizona also border California, but neither is in the pool: both are excluded for having run their own tobacco-control programmes. So there is exactly one leak, in exactly one direction, and we can name it.

This post estimates the same effect three ways, relaxing one assumption at a time, using two Python libraries:

mlsynth (Jared Greathouse), which puts 46 modern synthetic-control estimators behind a single configuration-dictionary interface.
scspill, which implements the Bayesian spatial spillover model of Sakaguchi and Tagawa (2026) and returns two estimands: the effect on the treated unit, purged of contamination, and the spillover effect received by each donor.

The argument of the post, stated up front so you can check it as you go: the effect on California survives every relaxation, and the claim that the donor pool was clean does not.

There is an R edition of this post that runs the same three stages using the authors’ own R and C++ replication code. It reports different numbers. Section 10 is about why, and it turns out to be the most instructive section here.

1.1 Learning objectives

By the end of this tutorial you will be able to:

Distinguish the two estimands a spillover-aware synthetic control reports — the effect on the treated unit, and the spillover received by each donor — and explain why classical synthetic control cannot express the second one at all.
Implement three nested estimators on one panel: the simplex baseline with mlsynth.VanillaSC, the Bayesian horseshoe with mlsynth.BSCM and with scspill at zero spillover intensity, and the Bayesian spatial model with scspill.SCSPILL(method="sar").
Derive the spillover-bias decomposition by hand on a three-donor example, and predict from it the direction in which a SUTVA failure moves a classical estimate.
Diagnose a Bayesian spatial sampler with a prior predictive check, a Geweke joint distribution test and a prior-sensitivity grid, and read an effective sample size as the reason to distrust one credible interval while trusting another.
Reconcile two implementations of the same paper by walking the six documented differences between them, and decide which of those differences changes an answer.

1.2 The road ahead

The three stages are nested. Each one keeps everything the previous stage assumed except a single restriction, which it replaces with something weaker. That structure is worth holding in mind, because it is what makes the comparison at the end meaningful: when the number moves, we know exactly which assumption moved it.

graph LR
A("<b>Difference-in-differences</b><br/>every donor weighted 1/N<br/>parallel trends") --> B("<b>Stage 1: classical SC</b><br/>weights chosen to fit<br/>simplex constraint")
B --> C("<b>Stage 2: Bayesian SC</b><br/>simplex replaced by<br/>a horseshoe prior")
C --> D("<b>Stage 3: Bayesian spatial SC</b><br/>SUTVA on donors dropped<br/>SAR layer, intensity rho")
D --> E("<b>Two estimands</b><br/>effect on California<br/>+ spillover on each donor")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class A anchor
class B blue
class C key
class D teal
class E orange

Read the arrows as successive relaxations. Difference-in-differences fixes the donor weights at $1/N$ and asks parallel trends to do all the work. Classical synthetic control lets the data choose the weights, but confines them to the simplex. The Bayesian stage replaces that hard constraint with a prior that prefers zero without forbidding anything else. The spatial stage keeps the Bayesian weights and drops the last assumption — that the donors were bystanders.

Only the final stage can answer the question “who else was treated?”, because only the final stage has a parameter that represents the leak. Sections 13 and 16 return to the full comparison.

2. Key concepts

The rest of this tutorial leans on a small vocabulary. The definition of each concept below is always visible — open the example and analogy cards when you need them, and leave them collapsed for a quick scan. Two terms are slipperier than the rest and worth re-reading later: spillover bias, which is the thing the third stage removes, and effective sample size, which is the thing that decides whether a credible interval means anything.

1. Potential outcomes under interference $Y_{it}(d_1, d_2, \ldots, d_N)$. The outcome unit $i$ would take at time $t$ under a whole vector of treatment assignments. Standard causal inference writes $Y_{it}(d_i)$ and drops everyone else’s assignment from the notation. Here we keep it, because dropping it is exactly SUTVA, and SUTVA is what this post tests.

Example

Nevada in 1990 has two potential outcomes that matter. $Y_{\mathrm{NV},1990}(\mathbf{0})$ is its cigarette sales in a world where California never passed Proposition 99. $Y_{\mathrm{NV},1990}(1, \mathbf{0})$ is its sales in the world we actually observe, where California is treated and Nevada is not. We see the second. The first is what the spatial stage reconstructs, and the gap between them is result.spillover_panel["Nevada"].

Analogy

A pharmacy runs a flu-shot campaign in one town. To measure it you compare against the neighbouring town — but if people drove across to get the free shot, the neighbouring town’s flu rate also fell. Its observed rate is no longer its untreated rate, and using it as a control understates the campaign.

2. Average treatment effect on the treated (ATT) $\mathrm{ATT} = \frac{1}{T_1}\sum_{t > T_0} \big(Y_{1t} - Y_{1t}(0)\big)$. The causal effect averaged over the periods after treatment, for the unit that actually received it. With a single treated unit there is no population to average over — the ATT is the gap between what California did and what a reconstructed untreated California would have done.

Example

In this post the ATT is California’s per-capita cigarette sales from 1988 to 2000 minus a synthetic California’s, averaged over those 13 years. Every stage targets the same ATT. They differ only in how the synthetic California is built.

Analogy

A patient takes a new drug; we never see that same patient untreated. So we build an imagined twin from similar untreated patients and take the difference. The ATT is the patient’s actual outcome minus the twin’s.

3. Donor pool. The set of untreated units from which the counterfactual is built. Here it is the 38 US states that had no large tobacco-control programme of their own between 1970 and 2000. Which states are excluded is a modelling decision made before any estimation happens, and it turns out to matter a great deal for this particular question.

Example

Eleven states are absent from the panel: Alaska, Arizona, Florida, Hawaii, Maryland, Massachusetts, Michigan, New Jersey, New York, Oregon and Washington. Two of those — Oregon and Arizona — border California. Their exclusion is why Nevada ends up as California’s only contiguous donor, and why the leak in this application has exactly one channel.

Analogy

Choosing a control group is like choosing which of your neighbours to ask about a normal electricity bill. You exclude the one who installed solar panels — but if you also exclude everyone except the neighbour who shares a wall with you, your comparison inherits whatever passes through that wall.

4. Simplex constraint $\alpha_j \geq 0$ and $\sum_j \alpha_j = 1$. The requirement that donor weights be non-negative and sum to one. It guarantees the synthetic unit is an interpolation of the donors rather than an extrapolation, which is what makes classical synthetic control feel safe. It also forces sparsity: a constrained least-squares problem in 38 variables typically puts all its weight on a handful of them.

Example

In Stage 1 the simplex puts 98.6% of the weight on four states — Utah 0.343, Montana 0.254, Nevada 0.242, Connecticut 0.146 — and exactly zero on 33 others. Section 4.2 constructs a case where that constraint has a measurable cost, and section 8 shows what the same data say when it is lifted.

Analogy

Mixing paint. You can combine tins in any proportions that add to one, but you cannot add a negative amount of blue to make something oranger. Sometimes that restriction is exactly what you want. Sometimes the colour you are matching lies outside anything you can mix.

5. Horseshoe prior $\alpha_j \mid \lambda_j \sim \mathcal{N}(0, \lambda_j^2)$ with $\lambda_j \mid \tau \sim \mathcal{C}^{+}(0, \tau)$. A continuous shrinkage prior with an infinite spike at zero and heavy Cauchy tails. It makes “this donor gets no weight” overwhelmingly likely a priori, while leaving any individual donor free to escape to a large value if the data insist. It is the Bayesian answer to the sparsity that the simplex imposes by decree.

Example

Under the horseshoe, 25 of 38 donors carry a posterior weight above 0.01 in absolute value, against the simplex’s five. But only Nevada’s 95% credible interval excludes zero. The pool looks much broader and is, in the end, no more informative about which states resemble California.

Analogy

A hiring policy that says “assume nobody is qualified” versus one that says “only four people may ever be hired”. The first can still hire twenty if twenty candidates are outstanding. The second cannot, no matter what the evidence says.

6. SUTVA (stable unit treatment value assumption). The assumption that one unit’s treatment does not affect another unit’s outcome. Under SUTVA, $Y_{jt}(d_1, \ldots, d_N) = Y_{jt}(d_j)$, and every donor’s observed outcome is its no-treatment outcome. Classical synthetic control does not merely assume this — it has no way to express its failure.

Example

Nevada carries weight 0.24 in synthetic California, and its estimated spillover is −5.50 packs per capita — its observed series sits below its no-treatment path. The counterfactual is therefore built partly from a contaminated donor. Section 9.1 shows the resulting bias has a closed form; section 12 evaluates that formula across all 38 donors and recovers 1.13 of the 1.19 packs separating the contaminated and purged estimates, of which Nevada alone supplies 1.10.

Analogy

Measuring whether a new streetlight reduces crime by comparing with the next street over — while the criminals simply move to the next street over. The comparison street is not a control: it absorbed part of the policy’s effect. Note that the sign can run either way. Displacement pushes the control street’s crime up and makes the light look better than it is; Nevada’s case runs the other way, and makes Proposition 99 look worse than it was.

7. Spatial autoregressive (SAR) model $\mathbf{y} = \rho W \mathbf{y} + X\beta + \varepsilon$. A regression in which each unit’s outcome depends on a weighted average of its neighbours’ outcomes. The scalar $\rho$ measures how strongly, and $W$ encodes who is a neighbour. Setting $\rho = 0$ removes the spatial channel entirely and returns an ordinary regression.

Example

Here the SAR layer sits on the donor outcomes, and $\rho$ is estimated at 0.316 with a 95% credible interval from 0.231 to 0.403. Because that interval excludes zero, the data reject the restriction that would collapse Stage 3 back to Stage 2.

Analogy

House prices. Your home is worth more when the houses around it are worth more, and theirs are worth more because yours is. Everything is determined at once rather than in sequence, which is why the model has to be solved rather than simply evaluated.

8. Spillover effect $\xi^{c}_{t} = \mathbf{Y}^{c}_{t} - \mathbf{Y}^{c}_{t}(\mathbf{0})$. The difference between a donor’s observed outcome and the outcome it would have had if the treated unit had never been treated. This is the second estimand, and it exists only in the spatial stage. It is reported per donor and per year.

Example

Nevada’s mean post-1988 spillover is −5.50 packs per capita per year, against −0.49 for Idaho and −0.49 for Utah. Every other donor’s posterior mean is below 0.06 packs in absolute value. The policy’s geographic footprint is essentially one state wide.

Analogy

The splash radius of a stone dropped in a pond. The stone is the policy, the treated unit is where it lands, and the spillover is how far the ripple reaches before it is lost in the noise.

9. Effective sample size (ESS). The number of independent draws an autocorrelated MCMC chain is worth. A chain of 250,000 highly correlated draws can carry the information of a few dozen independent ones. A credible interval computed from a chain with a small ESS is not a posterior summary; it is an artefact of where the chain happened to wander.

Example

The R edition of this post reported a 95% interval for the ATT that was 0.38 packs wide, from a chain whose ESS for $\rho$ this post recomputes as 2.93. The corrected run here reports an interval 12.71 packs wide from an ESS of 137. The policy did not change. The interval was wrong.

Analogy

Asking a thousand people their opinion — but they were all in the same room and heard each other answer. You have a thousand responses and perhaps five opinions. Reporting a margin of error based on a thousand would be dishonest.

3. The estimand, and two ways it goes wrong

Everything below is in service of a single number: how many packs per capita did Proposition 99 cost California each year? Getting that number requires two things, and each can fail independently.

We need a counterfactual — some construction of what California would have done untreated — and we need uncontaminated donors to build it from. Stage 1 and Stage 2 are two answers to the first requirement. Stage 3 is the only one of the three that addresses the second.

3.1 Potential outcomes when the treatment leaks

Write $D_i \in \{0, 1\}$ for whether unit $i$ is treated, and let $\mathbf{D} = (D_1, \ldots, D_N)$ be the whole assignment vector. Indexing potential outcomes by the full vector rather than by $D_i$ alone is the notational commitment that lets us even state the problem:

$$Y_{it} = Y_{it}(\mathbf{D}), \qquad i = 1, \ldots, N, \qquad t = 1, \ldots, T$$

In words, this says: what unit $i$ does at time $t$ may depend on who else got treated, not only on whether $i$ did. SUTVA is the restriction $Y_{it}(\mathbf{D}) = Y_{it}(D_i)$, which throws away every argument but one.

Let unit 1 be California, treated from period $T_0 + 1$ onward, and let $\mathbf{e}_1$ be the assignment vector in which only California is treated. Two estimands follow, and the second is invisible to classical synthetic control:

$$\xi_{0t} = Y_{1t}(\mathbf{e}_1) - Y_{1t}(\mathbf{0}), \qquad \xi^{c}_{jt} = Y_{jt}(\mathbf{e}_1) - Y_{jt}(\mathbf{0})$$

In words: $\xi_{0t}$ is the effect on California, the thing everybody reports. And $\xi^{c}_{jt}$ is the effect on donor $j$ of a policy donor $j$ never passed. Under SUTVA the second is defined to be zero, which is why a classical synthetic control cannot report it, cannot test it, and cannot be wrong about it in a way you would notice.

Symbol	Meaning	In the code
$Y_{1t}$	California’s observed sales	`panel.df.query("state == 'California'").cigsale`
$Y_{1t}(\mathbf{0})$	California’s no-treatment sales	`result.counterfactual`
$\xi_{0t}$	effect on California in year $t$	`result.gap`
$\xi^{c}_{jt}$	spillover onto donor $j$ in year $t$	`result.spillover_panel[j][t]`
$T_0$	last pre-treatment period (1987)	`result.inputs.T0`

3.2 Why difference-in-differences will not do

The simplest counterfactual is the average of the donors, shifted to match California’s pre-treatment level. That is difference-in-differences, and its estimand is

$$\widehat{\mathrm{ATT}}_{\mathrm{DiD}} = \Big(\bar{Y}_{1,\mathrm{post}} - \bar{Y}_{1,\mathrm{pre}}\Big) - \frac{1}{N-1}\sum_{j \neq 1} \Big(\bar{Y}_{j,\mathrm{post}} - \bar{Y}_{j,\mathrm{pre}}\Big)$$

In words: California’s before-and-after change, minus the average donor’s before-and-after change. This is unbiased only under parallel trends — the assumption that absent the policy, California’s sales would have moved by exactly the average donor’s amount.

Look at the data and that assumption is visibly false. California is not a typical state: it starts below the donor average and falls faster throughout the 1970s and 1980s, long before Proposition 99 exists.

Figure 1. Annual per-capita cigarette sales, 39 US states, 1970–2000. California in orange, the 38 donor states in grey.

California is already declining relative to the pack well before the dashed line. Any method that assumes California would otherwise have tracked the average donor will attribute a pre-existing trend to the policy. Synthetic control exists precisely because of this picture: rather than assuming California resembles the average donor, it goes looking for the combination of donors that California actually does resemble.

3.3 The donor pool as a weighted average

The fix is to replace the equal weights $1/(N-1)$ with weights chosen so that the blend tracks California before the treatment. Write $\mathbf{Y}^{c}_{t}$ for the vector of donor outcomes in year $t$ and $\alpha$ for a vector of weights. The counterfactual becomes

$$\widehat{Y}_{1t}(\mathbf{0}) = \sum_{j=2}^{N} \alpha_j Y_{jt} = \alpha^{\top} \mathbf{Y}^{c}_{t}$$

and the weights are chosen to make that blend match California over the pre-treatment window:

$$\widehat{\alpha} = \arg\min_{\alpha \in \Delta} \sum_{t=1}^{T_0} \Big(Y_{1t} - \alpha^{\top}\mathbf{Y}^{c}_{t}\Big)^2, \qquad \Delta = \Big\{\alpha : \alpha_j \geq 0, \, \sum_j \alpha_j = 1\Big\}$$

In words: find the mix of donor states whose weighted average tracks California most closely over 1970–1987, subject to the mix being a genuine average — no negative weights, and the weights adding to one. The set $\Delta$ is the simplex, and everything that distinguishes the three stages of this post is a statement about $\Delta$.

Symbol	Meaning	In the code
$\alpha$	vector of 38 donor weights	`result.donor_weights`
$\Delta$	the simplex	`VanillaSC`’s default constraint
$T_0$	18 pre-treatment years (1970–1987)	`result.inputs.T0`
$\mathbf{Y}^{c}_{t}$	donor outcomes in year $t$	`result.inputs.Yc`

Stage 1 solves this problem as written. Stage 2 replaces $\alpha \in \Delta$ with a prior over all of $\mathbb{R}^{38}$. Stage 3 keeps Stage 2’s weights and changes what $\mathbf{Y}^{c}_{t}$ is assumed to be.

4. Three donors, four years, no computer

Before handing 38 donors to an optimiser, it is worth solving a version small enough to check by hand. Everything that happens in the next ten sections happens here first, in arithmetic you can do on paper.

Three donor states, four pre-treatment years:

Year	Donor A	Donor B	Donor C
1	10	20	30
2	12	18	30
3	14	16	30
4	16	14	30

Notice the one structural fact that makes this tractable: A and B sum to 30 in every year. A rises, B falls, and they cross. C is flat at 30.

4.1 An exact blend on the simplex

Suppose our treated unit sits at 15 in all four pre-treatment years. Can a simplex-constrained blend match it exactly?

Take $\alpha = (0.5,\, 0.5,\, 0)$. Year 1 gives $0.5 \times 10 + 0.5 \times 20 = 15$. Year 2 gives $0.5 \times 12 + 0.5 \times 18 = 15$. Years 3 and 4 give 15 as well, because $A_t + B_t = 30$ for every $t$. The fit is exact, the weights are non-negative and they sum to one.

import numpy as np
A = np.array([10.0, 12.0, 14.0, 16.0])
B = np.array([20.0, 18.0, 16.0, 14.0])
C = np.array([30.0, 30.0, 30.0, 30.0])
Z = np.array([15.0, 15.0, 15.0, 15.0]) # the treated unit, pre-treatment
blend = 0.5 * A + 0.5 * B + 0.0 * C
print("blend :", blend)
print("exact fit :", np.allclose(blend, Z))
print("weights sum:", 0.5 + 0.5 + 0.0)

blend : [15. 15. 15. 15.]
exact fit : True
weights sum: 1.0

The point worth extracting is about donor C. It receives a weight of exactly zero — not because a constraint forbade it, but because it is useless: a flat series at 30 cannot help match a flat series at 15 when two other donors already do it perfectly. Sparsity here came from the data. In section 7 the simplex will also produce sparsity, and it will be much harder to tell which source it came from.

4.2 The treated unit outside the hull

Now move the treated unit to 35 in all four years, and keep the same three donors. Every simplex blend is a weighted average of numbers no larger than 30, so no simplex blend can ever exceed 30. The treated unit lies outside the convex hull of the donors, and the constraint now costs something we can measure.

The best a simplex can do is put all the weight on C, giving 30 every year and a gap of 5:

best_simplex = 0.0 * A + 0.0 * B + 1.0 * C # all weight on the highest donor
gap = np.array([35.0, 35.0, 35.0, 35.0]) - best_simplex
print("best simplex blend :", best_simplex)
print("pre-treatment RMSE :", np.sqrt((gap ** 2).mean()))
# Drop the sum-to-one requirement and the fit becomes exact.
alpha_unconstrained = np.array([0.0, 0.0, 7 / 6])
print("unconstrained blend:", alpha_unconstrained[2] * C)
print("weights sum :", alpha_unconstrained.sum().round(4))

best simplex blend : [30. 30. 30. 30.]
pre-treatment RMSE : 5.0
unconstrained blend: [35. 35. 35. 35.]
weights sum : 1.1667

A weight of $7/6 \approx 1.167$ fits perfectly and is not a convex combination — it is an extrapolation, scaling C up by 17%. Whether you find that acceptable is a genuine modelling judgement, and it is exactly the judgement Stage 2 puts in the hands of a prior rather than a constraint. What is not a judgement is the accounting: the simplex bought interpretability at a cost of 5 units of pre-treatment misfit, and that misfit does not disappear after treatment. It walks straight into the post-period as bias.

This is why pre-treatment RMSE is the first diagnostic to read in any synthetic control table. It is the visible part of the price.

4.3 What one leaky donor does

Now the third assumption, and the one this post is really about. Give the toy a post-treatment period — extend the table one year, so A reaches 18 and B falls to 12, and $A_t + B_t$ is still 30, which means the same 50-50 blend still lands on 15. Suppose we know the truth:

The treatment lowers the treated unit by 20 units.
Donor B is not a bystander. It absorbs a spillover of −8 — its observed post-treatment value is 8 below what it would have been.
Our weights are the exact-fit ones from section 4.1: $\alpha = (0.5, 0.5, 0)$.

What does a classical synthetic control report? It builds the counterfactual from observed donor values, which for B are already 8 too low:

alpha_toy = np.array([0.5, 0.5, 0.0])
Y_no_treatment = np.array([18.0, 12.0, 30.0]) # A, B, C in year 5, absent any treatment
xi = np.array([0.0, -8.0, 0.0]) # the spillover each donor absorbs
Y_observed = Y_no_treatment + xi # what we actually see
Y_treated_true = alpha_toy @ Y_no_treatment - 20.0 # the true post-treatment outcome
att_true = Y_treated_true - alpha_toy @ Y_no_treatment
att_naive = Y_treated_true - alpha_toy @ Y_observed
print(f"true ATT : {att_true:+.1f}")
print(f"naive (SUTVA) ATT : {att_naive:+.1f}")
print(f"bias : {att_naive - att_true:+.1f}")
print(f"-sum(alpha_j * xi_j) : {-(alpha_toy * xi).sum():+.1f}")

true ATT : -20.0
naive (SUTVA) ATT : -16.0
bias : +4.0
-sum(alpha_j * xi_j) : +4.0

The naive estimate is −16 when the truth is −20. It understates the effect by 4, and that 4 is exactly $-\sum_j \alpha_j \xi_j = -(0.5 \times -8) = +4$.

Three things follow, and all three recur in the real data:

The bias has a closed form. It is the weighted sum of the spillovers, with the same weights used to build the counterfactual. Section 9.1 states it in general.
The sign is determined by the sign of the spillover. Negative spillovers on positively-weighted donors push the estimate upward — which, when the true effect is negative as it is here, means toward zero. That is the case for Nevada, and it is why purging the contamination in section 12 makes the estimated effect larger, not smaller.
A donor’s damage is the product of two things, its weight and its spillover. A heavily contaminated donor with zero weight is harmless. A lightly contaminated donor carrying half the counterfactual is not.

Hold onto the number $-\sum_j \alpha_j \xi_j$. In section 12 we compute it on the real panel and find it accounts for 1.13 of the 1.19 packs separating the contaminated and purged estimates.

5. Setup: two libraries, two pins

Both packages are young and under active development, so both are pinned. scspill was at version 0.2.1 when this post was written; mlsynth is pinned to a commit rather than a release, because its PyPI release lags its main branch by weeks at the same version string — a trap documented in the companion post on the synthetic control ladder.

The numbers in this post are reproducible only under these two pins. The [numba] extra is optional — it makes the samplers about five times faster and returns results identical to the pure-numpy backend on this panel, so the fallback after || is there for readers whose Python has no llvmlite wheel.

pip install "scspill[numba]==0.2.1" || pip install "scspill==0.2.1"
pip install "mlsynth[bayes] @ git+https://github.com/jgreathouse9/mlsynth.git@15f168bb90487098a7324be00b6663fcab0139ef"

The [bayes] extra on mlsynth matters: four of the estimators in section 13 import numpyro, which is an optional dependency. Without it they fail with a ModuleNotFoundError on an otherwise clean install.

import os
# BLAS reduction order changes the last digits of every matrix product, and at
# N = 38 single-threaded BLAS is also faster than multi-threaded. Both are
# reasons to pin it, and it has to happen before numpy is imported.
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
os.environ.setdefault(v, "1")
os.environ.setdefault("JAX_ENABLE_X64", "1") # numpyro is float32 by default
import numpy as np
import pandas as pd
import scipy.sparse.csgraph
import mlsynth
import scspill
from scspill import SCSPILL
from scspill.data import load_california
SEED = 20251022 # the R edition's seed, so the two are comparable
TREAT_YEAR = 1988
M_ITER, BURN = 500_000, 250_000
print(f"scspill {scspill.__version__} mlsynth {mlsynth.__version__}")

scspill 0.2.1 mlsynth 1.0.0

Half a million iterations for a 13-year effect looks excessive. Section 14 is the evidence that it is not: the ATT is stable from about 100,000 draws onward, but the spatial parameter needs roughly five times that before its effective sample size reaches anything reportable.

6. The data

scspill ships the Proposition 99 panel and both of the spatial objects the third stage needs, so there is nothing to download and nothing to merge.

6.1 The panel

scspill ships the Proposition 99 panel, so nothing has to be downloaded or reshaped. load_california() returns a panel object that already knows which columns are the unit, the time index and the outcome, and that carries the two spatial objects alongside them.

panel = load_california()
df = panel.df.copy()
donors = list(panel.spatial_W.index)
print(panel.description)
print(df.head())
print(f"\nshape {df.shape} states {df['state'].nunique()} "
f"years {df['year'].min()}-{df['year'].max()}")

California Proposition 99 tobacco panel (Abadie, Diamond & Hainmueller 2010): 39
states, 1970-2000, per-capita cigarette sales, treatment in 1988. Spatial weights
are rook contiguity from the 2024 TIGER/Line state shapefile, stored
unnormalized. Source: scspill replication package, nonproprietary export.
state state_id year cigsale retprice treated
0 Alabama 1 1970 89.80 39.6 0
1 Alabama 1 1971 95.40 42.7 0
2 Alabama 1 1972 101.10 42.3 0
3 Alabama 1 1973 102.90 42.1 0
4 Alabama 1 1974 108.20 43.1 0
shape (1209, 6) states 39 years 1970-2000

A balanced panel: 39 states $\times$ 31 years $=$ 1,209 rows, no missing values. The outcome cigsale is annual per-capita cigarette sales in packs; the single covariate retprice is the real retail price per pack. There are 18 pre-treatment years and 13 post-treatment years.

This is a leaner predictor set than the original Abadie–Diamond–Hainmueller specification, which also matched on beer consumption, income, the share of the population aged 15–24, and three individual lags of the outcome. That is deliberate: the spatial model in Stage 3 is identified off the outcome dynamics, and the comparison across stages is cleaner when all three see the same variables.

6.2 The spatial weights

Two objects, and the distinction between them is the one thing to get right in this section.

# Pin the donor ordering once; every later section indexes off it.
W = panel.spatial_W.loc[donors, donors] # donor-to-donor contiguity
w = panel.spatial_w.reindex(donors) # each donor's exposure to California
# spatial_w: how exposed is each DONOR to the TREATED unit?
print(w[w > 0])
# spatial_W: donor-to-donor contiguity. Row-normalised inside the estimator.
print(W.iloc[:4, :4])
print(f"\nW is symmetric: {np.allclose(W.values, W.values.T)}")
deg = W.sum(axis=1)
print(f"degree: min {deg.min():.0f} max {deg.max():.0f} mean {deg.mean():.2f}")

Nevada 1.0
Name: spatial_w, dtype: float64
Alabama Arkansas Colorado Connecticut
Alabama 0.0 0.0 0.0 0.0
Arkansas 0.0 0.0 0.0 0.0
Colorado 0.0 0.0 0.0 0.0
Connecticut 0.0 0.0 0.0 0.0
W is symmetric: True
degree: min 1 max 8 mean 3.95

spatial_w has exactly one non-zero entry. Nevada is the only donor that borders California. Oregon and Arizona border California too, but neither is in the donor pool — both were excluded by Abadie, Diamond and Hainmueller for having run their own tobacco-control programmes. Eleven states are absent from the panel for similar reasons: Alaska, Arizona, Florida, Hawaii, Maryland, Massachusetts, Michigan, New Jersey, New York, Oregon and Washington.

That exclusion is doing quiet work. It means the spatial model has a single channel to estimate, which is both a gift (the parameter is easy to interpret) and a limitation (a single channel is thin evidence for a scalar).

Figure 2. Left: the 38 × 38 donor contiguity matrix, states ordered by degree. Centre: how many neighbours each donor has, with Nevada in orange. Right: the eigenvalues of the row-normalised W, and the shaded stability region the sampler confines ρ to.

The right-hand panel matters for section 9.4. A spatial autoregressive model is only well-defined when $I - \rho W$ is invertible, which bounds $\rho$ by the reciprocal of the largest eigenvalue of the row-normalised weights. For row-normalised contiguity that eigenvalue is exactly 1, so the mathematical bound is $|\rho| < 1$; scspill shrinks it to $|\rho| < 0.95$ as a numerical safety margin, and the sampler will not propose outside it. Section 11.3 shows why that 5% margin is itself a prior choice worth reporting.

6.3 Treatment in 1988, not 1989

One detail that will bite anyone comparing this post against other sources. Proposition 99 passed in November 1988 and the tax took effect on 1 January 1989. Abadie, Diamond and Hainmueller treat 1989 as the first treated year, and so does mlsynth’s own Proposition 99 example. scspill’s load_california() uses 1988, following the R replication package that accompanies the Sakaguchi–Tagawa paper.

Neither is wrong, but they cannot be mixed. This post pins 1988 everywhere, so all three stages and the whole benchmark in section 13 condition on the same 18 pre-treatment years:

# Rebuild the treatment dummy from the stated rule and assert it matches, so a
# future change in the shipped column cannot silently move the post-period.
rebuilt = ((df["state"] == "California") & (df["year"] >= TREAT_YEAR)).astype(int)
assert (rebuilt.to_numpy() == df["treated"].to_numpy()).all()
years = np.sort(df["year"].unique())
wide = df.pivot(index="year", columns="state", values="cigsale")
y_treated = wide["California"].to_numpy()
post = years >= TREAT_YEAR
print(f"T0 = {(~post).sum()} T1 = {post.sum()} donors = {len(donors)}")

T0 = 18 T1 = 13 donors = 38

To use the 1989 convention instead, overwrite df["treated"] before passing the frame to any estimator. The effect on the headline number is small — one fewer post-treatment year, and 1988 was a partial year in any case — but the comparison across libraries stops being apples-to-apples the moment two of them disagree about $T_0$.

7. Stage 1 — classical simplex synthetic control

The first stage solves exactly the problem written down in section 3.3. In mlsynth every estimator takes the same configuration dictionary — a long data frame plus four column names — and the class chosen decides the estimator.

common = dict(df=df, outcome="cigsale", treat="treated",
unitid="state", time="year", display_graphs=False)
sc = mlsynth.VanillaSC(dict(common)).fit()
w_sc = pd.Series(sc.donor_weights).reindex(donors).fillna(0.0)
print(f"ATT : {sc.att:.4f} packs per capita per year")
print(f"pre-treatment RMSE : {sc.pre_rmse:.4f}")
print(f"weights sum : {w_sc.sum():.6f}")
print(f"active donors : {(w_sc > 1e-4).sum()} of {len(donors)}")
print(w_sc[w_sc > 1e-4].sort_values(ascending=False).round(4).to_string())

ATT : -18.4277 packs per capita per year
pre-treatment RMSE : 1.5998
weights sum : 1.000000
active donors : 5 of 38
Utah 0.3430
Montana 0.2545
Nevada 0.2423
Connecticut 0.1457
New Hampshire 0.0144

Five of 38 donors carry the entire counterfactual, and four of them carry 98.6% of it. The pre-treatment RMSE of 1.60 is small against an outcome averaging 117.7 packs over the pre-period — a relative error of 1.4%, and a pre-treatment $R^2$ of 0.973.

Two things are worth pausing on. First, this reproduces the R edition to within 0.04 packs: that post reports −18.46 using the tidysynth package and a different optimiser, with weights of Utah 0.327, Nevada 0.255, Montana 0.245 and Connecticut 0.148. Two independent implementations landing this close is the strongest evidence either one gets that the estimator is correctly coded.

Second, and less comfortably: Nevada is in the synthetic California, with a weight of 0.24. The one state we have a specific reason to suspect of contamination is carrying nearly a quarter of the counterfactual.

Figure 3. Top: California and its simplex-weighted synthetic. Bottom: the gap between them, shaded after 1988.

The two series track closely until 1988 and separate steadily thereafter, reaching −26.7 packs in 2000. The pre-treatment gap is not exactly zero — it wanders between −3.5 and +5.0 packs, which is the visible form of the misfit that section 4.2 priced. An RMSE of 1.60 is an average over that wandering, not a promise that any single year fits well.

Figure 4. The simplex assigns weight to five donors and exactly zero to the other 33.

That wall of zeros is the question Stage 2 exists to ask. Are 33 states genuinely irrelevant to reconstructing California, or is that just what a constrained least-squares problem in 38 variables does? Section 4.1 showed sparsity can come from the data. It can also come from the constraint, and from the outside the two look identical.

Stage 1 leaves two questions on the table, and the remaining stages take one each:

Is the sparsity real? Stage 2 replaces the constraint with a prior and looks again.
Is Nevada a donor or a victim? Stage 3 gives the model a way to answer.

8. Stage 2 — Bayesian synthetic control

The simplex is a hard constraint: it declares certain weight vectors impossible. A prior is softer. It declares them unlikely, and lets the data overrule it if the evidence is strong enough. Stage 2 makes that substitution and changes nothing else.

8.1 The horseshoe hierarchy

The prior we want has two properties that pull in opposite directions. It should put enormous mass near zero, so that a donor with nothing to contribute gets nothing. And it should have tails heavy enough that a donor with a great deal to contribute is not shrunk into irrelevance. The horseshoe prior of Carvalho, Polson and Scott (2010) does both:

$$\alpha_j \mid \lambda_j \sim \mathcal{N}\big(0, \, \lambda_j^2\big), \qquad \lambda_j \mid \tau \sim \mathcal{C}^{+}(0, \tau), \qquad \tau \sim \mathcal{C}^{+}(0, \sigma), \qquad \sigma \sim \mathcal{C}^{+}(0, 10)$$

In words, this says: each donor weight is normal around zero, but with its own variance, and that variance is drawn from a half-Cauchy. The half-Cauchy has infinite density at zero and a tail that decays only polynomially, so most $\lambda_j$ come out tiny — shrinking that donor to nothing — while any individual $\lambda_j$ can be enormous if the likelihood demands it. The global scale $\tau$ decides how sparse the whole vector is; the local scales $\lambda_j$ decide which donors get to escape.

The name comes from the shrinkage factor $\kappa_j = 1/(1 + \lambda_j^2)$, whose implied prior is a $\mathrm{Beta}(1/2, 1/2)$ — a U shape with peaks at 0 and 1, like a horseshoe. Weights are pushed either to “shrunk to nothing” or to “left alone”, and rarely in between.

Sampling from this directly is awkward because the half-Cauchy is not conjugate to anything — there is no closed-form conditional distribution to draw from, so a sampler cannot simply take a value and move on. Makalic and Schmidt (2015) supply the trick that makes it a Gibbs sampler: every half-Cauchy is a scale mixture of inverse gammas, so introducing one auxiliary variable per scale gives closed-form conditionals throughout:

$$\lambda_j^2 \mid \nu_j \sim \mathcal{IG}\Big(1, \, \tfrac{1}{\nu_j}\Big), \qquad \nu_j \sim \mathcal{IG}\Big(\tfrac{1}{2}, \, 1\Big) \, \Longrightarrow \, \lambda_j \sim \mathcal{C}^{+}(0, 1)$$

In words: an inverse-gamma whose own scale is inverse-gamma distributed is a half-Cauchy. Nothing is approximated — this is an exact reparameterisation, and it is why both libraries can run hundreds of thousands of iterations in seconds rather than hours.

Symbol	Meaning	In the code
$\alpha_j$	weight on donor $j$	`result.alpha_hat`
$\lambda_j$	local shrinkage scale for donor $j$	internal to the sampler
$\tau$	global shrinkage scale	internal to the sampler
$\nu_j$	Makalic–Schmidt auxiliary variable	internal to the sampler
$\kappa_j$	shrinkage factor, $1/(1+\lambda_j^2)$	not exposed

8.2 Fitting it with mlsynth.BSCM

mlsynth.BSCM takes the same configuration dictionary as VanillaSC, plus the prior family and the MCMC settings. Four chains of 20,000 draws is generous for a 38-donor regression; the horseshoe’s funnel geometry is the reason not to be stingy with either.

bscm = mlsynth.BSCM({**common, "prior": "horseshoe", "n_iter": 20_000,
"burn_in": 10_000, "chains": 4, "seed": SEED}).fit()
w_bscm = pd.Series(bscm.donor_weights).reindex(donors).fillna(0.0)
beta0 = float(np.mean(np.asarray(bscm.posterior.beta0)))
print(f"ATT : {bscm.att:.4f}")
print(f"95% CrI : [{bscm.att_ci[0]:.4f}, {bscm.att_ci[1]:.4f}]")
print(f"intercept : {beta0:.4f}")
print(f"weights sum : {w_bscm.sum():.4f}")
print(f"active donors : {(w_bscm.abs() > 0.01).sum()} of {len(donors)}")

ATT : -18.8469
95% CrI : [-26.4568, -9.9884]
intercept : 16.8619
weights sum : 0.7576
active donors : 26 of 38

The donor pool has gone from 5 active states to 26, and the weights no longer sum to one. Both are consequences of dropping the simplex, and neither is a defect. This is also the first output carrying a 95% credible interval, which is the Bayesian counterpart of a confidence interval: the range holding 95% of the posterior’s mass, which is to say the range the model assigns 95% probability to after seeing the data — a statement about the parameter, not about repeated sampling.

Figure 5. Posterior mean weight with 95% credible intervals for the 24 donors with the largest magnitude, from mlsynth.BSCM.

Notice what the credible intervals do. Most of them straddle zero comfortably: the model is willing to entertain a role for these donors but the data do not insist on one. The horseshoe has done what it promised — it made zero the default without making it compulsory.

There is one number in that output which deserves more attention than it usually gets, and it explains a discrepancy the next subsection would otherwise leave hanging: the intercept of 16.86.

8.3 The same model in scspill, at zero spillover intensity

scspill has no rho argument. The spatial intensity is a parameter to be estimated, not a setting. But the model collapses exactly to a Bayesian horseshoe synthetic control when $\rho = 0$, and that special case is computed and exposed as part of every fit — so Stage 2 costs no extra MCMC at all.

One ordering note if you are running these blocks in sequence: result is the Stage 3 fit built in section 9.5. The narrative needs its $\rho = 0$ case here, two sections earlier than the fit that produces it, so run 9.5 first and come back.

# Fitted once in section 9.5; both quantities come out of that single fit.
from scspill.utils.scspill_helpers.sar.effects import treated_counterfactual
att_rho0 = result.effects_detail.att_scm # the rho = 0 ATT
cf_rho0 = treated_counterfactual(result.inputs.Y0, result.inputs.Yc,
result.inputs.Wn, result.inputs.wn,
result.alpha_hat, rho=0.0) # the whole path
print(f"scspill at rho = 0 : {att_rho0:.4f}")
print(f"mlsynth.BSCM : {bscm.att:.4f}")
print(f"R edition Stage 2 : -15.8400")
print(f"rho=0 counterfactual, last 3 years: {cf_rho0[-3:].round(1)}")

scspill at rho = 0 : -15.6816
mlsynth.BSCM : -18.8469
R edition Stage 2 : -15.8400
rho=0 counterfactual, last 3 years: [73.4 71.2 67.2]

Two Bayesian synthetic controls with the same prior family on the same data, 3.17 packs apart. That gap is not noise, and it is not a bug in either library. It is an intercept.

mlsynth.BSCM implements Kim, Lee and Gupta (2020), which fits an explicit intercept $\beta_0$ and leaves the donor series on their original scale. scspill’s Step 1 has no intercept and standardises the donors first. The consequences are visible in the output above: BSCM’s weights sum to 0.758 rather than 1, because the intercept of 16.86 packs is absorbing the level difference that the weights would otherwise have to carry.

$$\text{BSCM:} \quad Y_{1t} = \beta_0 + \sum_j \alpha_j Y_{jt} + \varepsilon_t \qquad\text{versus}\qquad \text{scspill:} \quad Y_{1t} = \sum_j \alpha_j Y_{jt} + \varepsilon_t$$

In words: one model is allowed to say “California is like this blend of states, shifted up by 17 packs”; the other must say “California is this blend of states”. The direction matters. Because the weights sum to only 0.758, BSCM’s raw blend sits well below California — roughly $0.758 \times 131.5 \approx 100$ packs against California’s pre-period mean of 117.7 — and the positive intercept lifts it back onto the treated series. Neither is more correct in the abstract. But they answer different questions, and averaging them would be meaningless.

The tiebreaker available here is external. scspill’s $\rho = 0$ case lands within 0.16 packs of the R edition’s Stage 2 estimate of −15.84, which was produced by entirely separate C++ code. Two independent implementations of the no-intercept horseshoe agree; the intercept model is doing something else, deliberately.

Figure 6. Left: the counterfactual path implied by each estimator. Right: point estimates with intervals; hollow diamonds are the R edition’s published values.

The lesson generalises well beyond this post. When two packages disagree about a model they both claim to implement, the first thing to check is not the sampler. It is whether they are fitting the same equation.

9. Stage 3 — Bayesian spatial synthetic control

Everything so far has assumed the donors were bystanders. Stage 3 drops that.

9.1 Where the contamination enters

Start from the estimator itself and ask what it actually computes when SUTVA fails. The synthetic control is built from observed donor outcomes, and if those observations already contain a spillover, the counterfactual inherits it:

$$Y_{1t} - \sum_j \alpha_j Y_{jt} \, = \, \xi_{0t} \, - \, \sum_j \alpha_j \, \xi^{c}_{jt}$$

The two terms on the right are doing very different jobs:

Term	What it is	Do we want it?
$\xi_{0t}$	the causal effect on California	yes — this is the estimand
$-\sum_j \alpha_j \, \xi^{c}_{jt}$	a weighted sum of the spillovers that landed on the donors	no — this is bias, and we get it for free

In words: the number a classical synthetic control reports is the true effect on California minus a weighted sum of the spillovers that landed on the donors — with the same weights used to build the counterfactual. The derivation is one line of adding and subtracting $\sum_j \alpha_j Y_{jt}(\mathbf{0})$, using $Y_{jt} = Y_{jt}(\mathbf{e}_1)$ and $\xi^{c}_{jt} = Y_{jt}(\mathbf{e}_1) - Y_{jt}(\mathbf{0})$.

So the bias term is

$$\mathrm{bias}_t = -\sum_j \alpha_j \, \xi^{c}_{jt}$$

This is exactly the identity the toy example in section 4.3 verified with $-\left(0.5 \times -8\right) = +4$. Three consequences are worth stating plainly:

If every $\xi^{c}_{jt} = 0$, the bias vanishes and classical synthetic control is right. SUTVA is not a technicality — it is the whole justification.
The bias is a product of weight and spillover, so contamination in a zero-weight donor is harmless.
The sign of the bias is the opposite of the sign of the weighted spillover. Negative spillovers on positively-weighted donors push the estimate toward zero.

graph TD
P("<b>Proposition 99</b><br/>California, 1988") --> CA("California's sales fall")
P -.->|"leak, intensity rho"| NV("<b>Nevada</b>'s sales also fall<br/>spillover = -5.50")
NV --> SYN("Synthetic California<br/>alpha_NV = 0.20")
CA --> ATT("Measured gap")
SYN --> ATT
ATT --> BIAS("<b>Bias</b> = -sum(alpha_j * xi_j)<br/>= +1.13 packs<br/>the effect looks SMALLER")
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
class P orange
class CA,ATT anchor
class NV teal
class SYN blue
class BIAS key

The dashed arrow is the one classical synthetic control cannot draw. To estimate its strength we need a model of how outcomes travel between neighbouring states — which is what a spatial autoregression is for.

9.2 A spatial process on the donor outcomes

Sakaguchi and Tagawa’s proposal is to put a spatial autoregressive structure on the donor block. Each donor’s outcome depends on its neighbours’ outcomes, and on the treated unit’s outcome, with a single intensity parameter $\rho$ governing both:

$$\mathbf{Y}^{c}_{t} = \rho \big( \mathbf{w} \, Y_{1t} + W \mathbf{Y}^{c}_{t} \big) + X_t \beta + \mathbf{u}_t, \qquad \mathbf{u}_t = \eta \gamma_t + \mathbf{e}_t$$

In words: a donor’s cigarette sales are a weighted average of its neighbours’ sales, plus a term for how exposed it is to California, plus covariates, plus a common latent factor and idiosyncratic noise. The vector $\mathbf{w}$ is exposure to the treated unit — one non-zero entry, Nevada — and $W$ is donor-to-donor contiguity.

The error is not white noise. It carries a latent factor $\gamma_t$ following an AR(1) process with loadings $\eta$, which soaks up the common national trends visible in Figure 1 — the surgeon-general reports, federal tax changes, and the general secular decline in smoking:

$$\gamma_t = \phi_\gamma \gamma_{t-1} + \epsilon_t, \qquad \epsilon_t \sim \mathcal{N}(0, \sigma^2_\gamma), \qquad \eta_{jk} \sim \mathcal{N}(0, \sigma^2_\eta \omega_k), \qquad \omega_k \sim \mathcal{C}^{+}(0, 10)$$

The critical structural point is why $\mathbf{w}$ multiplies $Y_{1t}$, California’s observed outcome. Nevada’s residents respond to what California actually does — the actual prices, the actual advertising, the actual sales — not to some counterfactual California. That is what makes the system solvable: everything on the right-hand side is observed.

Setting $\rho = 0$ removes both spatial terms at once and returns Stage 2 exactly. This is the sense in which the three stages are nested, and it is why section 8.3 could read Stage 2 off a Stage 3 fit.

Symbol	Meaning	In the code
$\rho$	spatial intensity, the leak	`result.rho_hat`
$\mathbf{w}$	donor exposure to California	`panel.spatial_w`
$W$	donor-to-donor contiguity	`panel.spatial_W`
$W_n$	the same matrix, row-normalised	`result.inputs.Wn`
$\beta$	covariate coefficients	`result.sar_posterior.beta`
$\gamma_t$, $\eta$	latent AR(1) factor and loadings	`p_factors=1`
$\sigma^2$	idiosyncratic variance	`result.sar_posterior.sigma2`

9.3 Identification in closed form

Here is where the model earns its keep. The simplex is gone, and something has to replace it as the identifying assumption. What replaces it is a perfect pre-treatment fit with unconstrained weights:

$$\exists \, \alpha \in \mathbb{R}^{N} \, : \, Y_{1t}(\mathbf{0}) = \sum_j \alpha_j Y_{jt}(\mathbf{0}) \quad \text{for all } t$$

In words: some fixed combination of the donors’ no-treatment outcomes reproduces California’s no-treatment outcome exactly, in every period. The weights may be negative and need not sum to one. This is a stronger assumption than approximate fit and a weaker one than convexity, and it is worth being explicit that it is an assumption rather than a result.

Given that, define $A = W + \mathbf{w}\alpha^{\top}$ — the contiguity graph plus the indirect path from each donor back to itself through the synthetic California. Solving the simultaneous system for the donors’ no-treatment outcomes gives

$$\mathbf{Y}^{c}_{t}(\mathbf{0}) = \big(I_N - \rho A\big)^{-1}\Big[\big(I_N - \rho W\big)\mathbf{Y}^{c}_{t} - \rho \, \mathbf{w} \, Y_{1t}\Big]$$

and both estimands follow immediately:

$$\xi_{0t} = Y_{1t} - \alpha^{\top}\mathbf{Y}^{c}_{t}(\mathbf{0}), \qquad \boldsymbol{\xi}^{c}_{t} = \mathbf{Y}^{c}_{t} - \mathbf{Y}^{c}_{t}(\mathbf{0})$$

Look at what is not in those expressions. No $\beta$. No $\gamma_t$, no $\eta$, no $\sigma^2$. Only $(\alpha, \rho, \mathbf{w}, W)$ and the observed outcomes. The covariate coefficients, the latent factors and the error variances all cancel out of the effects.

That is not an aesthetic nicety. It is the reason the whole approach is usable. A model this rich has many parameters that are poorly identified from 31 years of data on 39 states — and section 9.5 will show that one of them, $\rho$ itself, mixes badly enough to need half a million draws. If the effects depended on the entire nuisance block, that weak identification would poison everything. Because they depend on four objects only, it does not. It is confined to $\rho$, where we can see it, measure it, and report it honestly.

9.4 The two-step sampler

The product $\rho \, \mathbf{w} \, \alpha^{\top}$ appears inside $\rho A$, so the exposure channel identifies $\rho$ and $\alpha$ only jointly. Sampling them together mixes very badly. The paper’s answer is to factorise the posterior into two steps — a cut posterior, meaning a deliberate refusal to let the second step feed information back into the first — estimating $\alpha$ first from the pre-treatment fit and then holding it fixed while $\rho$ is drawn.

graph LR
S1("<b>Step 1</b><br/>horseshoe Gibbs on the<br/>pre-treatment regression<br/>-> alpha") --> FIX("alpha fixed at its<br/>posterior mean")
FIX --> S2("<b>Step 2</b> Gibbs sweep")
S2 --> F("latent factors<br/>forward-filter<br/>backward-sample")
S2 --> B("beta<br/>horseshoe")
S2 --> SIG("sigma^2<br/>inverse gamma")
S2 --> RHO("<b>rho</b><br/>adaptive random-walk<br/>Metropolis")
RHO --> EFF("Effects, in closed form<br/>eigendecomposition +<br/>Sherman-Morrison")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class S1 blue
class FIX,F,B,SIG anchor
class S2 key
class RHO teal
class EFF orange

Read the split as the price of the identification problem: Step 1 never sees $\rho$, and Step 2 never re-litigates $\alpha$. The $\rho$ step is a random-walk Metropolis move — propose a small jump, accept it with a probability that depends on how much better the new value fits, and otherwise stay put — which is the only part of this sampler that is not a draw from a closed-form conditional, and the reason section 9.6 has an effective-sample-size problem to report.

Three implementation details explain both the speed and the one weakness.

The support for $\rho$. The system is only invertible when $I_N - \rho A$ is non-singular, which bounds $\rho$ by the spectral radius of the row-normalised weights:

$$|\rho| < \frac{0.95}{\max\big(1, \, \max_i |\mu_i(W_n)|\big)}$$

For row-normalised contiguity the largest eigenvalue is exactly 1, so the mathematical bound is $|\rho| < 1$ and the 0.95 in the numerator is a numerical safety margin the package imposes, not a consequence of invertibility. Section 11.3 shows this bound is the only prior setting in the model that meaningfully moves the answer.

The Jacobian. Each Metropolis proposal needs $\log|I_N - \rho A|$, which is an $O(N^3)$ determinant if computed naively, at every one of 500,000 iterations. Pre-computing the eigenvalues $\mu_i$ of $A$ once turns it into a sum — written $\mu$ rather than $\lambda$ because $\lambda_j$ is already the horseshoe’s local shrinkage scale in section 8.1:

$$\log\big|I_N - \rho A\big| = \sum_{i=1}^{N} \log\big(1 - \rho \mu_i\big)$$

$O(N)$ per iteration instead of $O(N^3)$. This single substitution is what makes a half-million-draw chain take two minutes instead of two days.

Adaptive step size. The Metropolis proposal standard deviation is tuned during burn-in by Robbins–Monro stochastic approximation, targeting a 44% acceptance rate — the optimum for a one-dimensional random walk (Gelman, Roberts and Gilks, 1996; the 0.234 asymptotic result for high dimensions is Roberts, Gelman and Gilks, 1997):

$$\log s_{m+1} = \log s_m + (m+1)^{-0.6}\big(a_m - 0.44\big)$$

where $a_m$ is the acceptance probability of the proposal made at iteration $m$. When a proposal is readily accepted the step grows; when proposals keep being rejected it shrinks. The exponent $-0.6$ makes the corrections vanish fast enough for the chain to settle, and adaptation stops entirely at the end of burn-in so the sampled portion is a genuine Markov chain. In the run below this lands at an acceptance rate of 0.444 against the 0.44 target, from a starting step of 0.05.

Section 10 shows what happens when this is switched off — which is what the R replication code does.

9.5 Fitting it

The configuration is the panel’s own config_kwargs() plus five settings: chain length, burn-in, the seed, a switch for the package’s automatic plotting, and a cap on how many posterior draws the effects sweep uses. Everything spatial — $\mathbf{w}$, $W$, the covariates — comes from the bundled panel, so there is nothing to align by hand.

result = SCSPILL({
**panel.config_kwargs(), # df, columns, spatial_w, spatial_W, covariates
"m_iter": M_ITER,
"burn": BURN,
"seed": SEED,
"display_graphs": False,
"max_effect_draws": 5_000, # thin the effects sweep; the ATT is unaffected
}).fit()
print(f"ATT : {result.att:.4f} 95% CrI "
f"[{result.att_ci[0]:.4f}, {result.att_ci[1]:.4f}]")
print(f"ATT at rho=0: {result.effects_detail.att_scm:.4f}")
print(f"rho : {result.rho_hat:.4f} 95% CrI "
f"[{result.rho_ci[0]:.4f}, {result.rho_ci[1]:.4f}]")
print(f"ESS(rho) : {result.rho_ess:.1f} acceptance {result.acc_rho:.3f}")

ATT : -16.8680 95% CrI [-23.0450, -10.3316]
ATT at rho=0: -15.6816
rho : 0.3161 95% CrI [0.2312, 0.4032]
ESS(rho) : 136.8 acceptance 0.444

The credible interval for $\rho$ excludes zero. That is the formal statement that the data reject the restriction collapsing Stage 3 back to Stage 2 — SUTVA on the donor pool is not merely doubtful here, it is rejected by the model that nests it.

The ATT moves from −15.68 to −16.87 once the leak is modelled: purging the contamination makes the estimated effect larger, by 1.19 packs. Section 4.3 predicted the direction from the sign of the spillover, and section 12 checks the magnitude against the identity.

scspill ships its own diagnostics table, and it is worth reading in full because it shows precisely where the weak identification lives:

print(result.diagnostics(top_n_alpha=6).round(4))

 mean sd q025 q50 q975 ess rhat_split mcse geweke_z
parameter
rho 0.3161 0.0430 0.2312 0.3162 0.4032 136.7806 1.0154 0.0037 -0.5704
sigma2 61.5905 3.4887 55.1382 61.4554 68.8232 204094.9179 1.0000 0.0077 -0.2965
alpha[Tennessee] -0.2505 0.1675 -0.5833 -0.2543 0.0121 10194.7170 1.0000 0.0017 -0.4424
alpha[Connecticut] 0.2296 0.1672 -0.0273 0.2319 0.5575 11798.6429 1.0000 0.0015 0.0571
alpha[Nevada] 0.1997 0.0370 0.1191 0.2016 0.2693 25855.2663 1.0000 0.0002 1.2922
alpha[Montana] 0.1289 0.1299 -0.0305 0.1015 0.4114 11047.2550 1.0001 0.0012 1.2323
alpha[West Virginia] 0.1253 0.0954 -0.0189 0.1261 0.3072 12238.6311 1.0000 0.0009 -0.7574
alpha[Illinois] 0.1049 0.1159 -0.0356 0.0755 0.3810 14970.6186 1.0000 0.0009 -0.6747
beta[retprice] 0.3475 0.0256 0.2979 0.3472 0.3988 388.4297 1.0060 0.0013 0.6091

Read the ess column top to bottom. $\sigma^2$ has an effective sample size of 204,000. The donor weights are in the 10,000–26,000 range. $\rho$ has 137, and the price coefficient 388. From the same chain. The pattern is not that one scalar is hard and everything else is easy — it is that the two quantities leaning on the single contiguity channel are hard, and $\rho$, the scalar the whole third stage exists to estimate, is the harder of the two.

That is not a defect in the software, and it is not that the data are silent about $\rho$ — the posterior is tight, with a standard deviation of 0.043 on a support 1.9 wide. It is that the sampler has to move one scalar through a strongly correlated conditional: $\rho$ is the only parameter drawn by random-walk Metropolis rather than from a closed-form conditional, and each draw is highly correlated with the one before. Section 14 shows what it costs to pin it down, and section 10 shows what happens if you do not try.

Two further observations from the table. Nevada’s weight is the most precisely estimated of all the donors (sd 0.037 against 0.10–0.17 for the other five shown) — the model is confident about the one state it also assigns nearly all the spillover to. And rhat_split for $\rho$ is 1.0154: above the 1.01 threshold Vehtari et al. (2021) recommend and inside the conventional 1.01–1.05 warning band, though well short of its upper end. Consistent with a chain that has mixed adequately but not comfortably — and one more reason to report the effective sample size beside the interval rather than in place of it.

Figure 7. result.plot(kind="panel") — observed against counterfactual, the treatment effect over time, and the eight largest spillover paths, drawn by the library itself.

Figure 8. result.plot(kind="weights") — the horseshoe posterior against the simplex weights for the same donors.

Figure 9. result.plot(kind="rho") and result.plot(kind="trace"). The trace is visibly slower-moving than a well-mixed chain, which is what an effective sample size of 137 out of 250,000 draws looks like.

9.6 The spillover received by each donor

The second estimand is a full panel — one spillover per donor per year — rather than a single number.

spill = result.spillover_panel.loc[TREAT_YEAR:] # post-treatment rows only
means = spill.mean()
ranked = means.reindex(means.abs().sort_values(ascending=False).index)
print(ranked.head(6).round(4).to_string())

Nevada -5.4995
Idaho -0.4929
Utah -0.4917
Wyoming -0.0590
Montana -0.0466
Colorado -0.0311

Nevada absorbs 11.2 times the next-largest effect. Idaho and Utah — the two states that border Nevada, one step further out on the contiguity graph — pick up about half a pack each. Everything beyond that second ring is two orders of magnitude smaller again.

One caveat on reading this panel: the pre-treatment rows are fit residuals, not causal spillovers. Nothing had leaked yet in 1975. Slice from TREAT_YEAR onward, as above, or you will average signal with noise.

Figure 10. Mean post-1988 spillover by state on a linear colour scale. California is the treated unit; dark tiles are states outside the donor pool.

The concentration in that map is the finding, not a rendering artefact. On a linear scale almost every tile sits at the pale end because one state absorbs an order of magnitude more than any other.

Figure 11. The eight donors with the largest estimated spillover, by posterior mean. These are point estimates: scspill returns the effects panel as posterior means rather than per-draw, so no interval is available for an individual donor’s spillover.

That last sentence is a real limitation and worth stating plainly rather than burying. The 95% credible interval reported for $\rho$ covers the spatial parameter, and the one reported for the ATT covers the treated unit — but nothing in this pipeline puts an interval around Nevada’s −5.50. Statements below about which donors are “distinguishable from zero” are therefore statements about relative magnitude, not about posterior tail probability.

With that caveat, the verdict on SUTVA is still unambiguous, because it rests on $\rho$ rather than on any individual donor: the interval for $\rho$ excludes zero, so the spatial channel is real, and the point estimates say it is concentrated almost entirely in one state. That is a better outcome than diffuse contamination would have been — a single identifiable leak can be modelled, and has been.

10. Why these numbers differ from the R edition

There is an R edition of this post on this site. It runs the same three stages on the same panel using the authors’ own R and C++ replication code, and it reports ATT −16.59 with a 95% credible interval of [−16.78, −16.39], against this post’s −16.87 with [−23.05, −10.33].

The point estimates are close. The intervals are not remotely close — one is 0.38 packs wide, the other 12.71. Something has to explain a factor of 33, and “different language” is not it.

scspill departs from the R replication code in six documented ways. Three have escape hatches, so we can put the Python code back into the R specification and see whether it reproduces the R numbers. Three do not, so the reproduction will be close rather than exact.

10.1 Reproducing the R specification

Three of the six departures have escape hatches, so the Python code can be put back into the R specification and run. R_SPEC below is that specification: the ridge prior, weights held at their posterior mean, and a fixed Metropolis step. If the gap between the two editions were a porting error, this configuration would not reproduce the R edition’s numbers.

R_SPEC = dict(beta_prior="ridge", # departure 2: flat-plus-ridge, not horseshoe
propagate_alpha=False, # departure 3: alpha fixed at its posterior mean
adapt_rho=False, # departure 4: fixed Metropolis step
step_rho=0.01)
rspec = SCSPILL({**panel.config_kwargs(), "m_iter": 5_000, "burn": 2_500,
"seed": SEED, "display_graphs": False, **R_SPEC}).fit()
print(f"ATT {rspec.att:.4f} CrI [{rspec.att_ci[0]:.4f}, {rspec.att_ci[1]:.4f}] "
f"width {rspec.att_ci[1] - rspec.att_ci[0]:.3f} rho {rspec.rho_hat:.4f}")

ATT -16.2858 CrI [-16.5914, -16.1093] width 0.482 rho 0.2282

Run at the R edition’s own budget of 5,000 iterations, the agreement is close enough to settle the question:

Quantity	R edition	scspill, R spec, R budget	Difference
ATT	−16.590	−16.286	0.304
95% CrI width	0.384	0.482	0.098
$\hat\rho$	0.2226	0.2282	0.0056
ESS($\rho$)	2.93	3.27	0.34
Nevada spillover	−3.750	−3.778	0.028

Independent code in a different language, reproducing $\hat\rho$ to three decimal places and the Nevada spillover to 0.03 packs — including the pathology. An effective sample size of 3.27 is not a coincidence to be explained away; it is the R sampler’s behaviour, faithfully reproduced.

Now change one thing at a time.

Specification	Iterations	ATT	95% CrI	Width	$\hat\rho$	ESS($\rho$)	Acceptance
R edition (published)	5,000	−16.590	[−16.78, −16.39]	0.384	0.2226	2.9	—
scspill, R spec	5,000	−16.286	[−16.59, −16.11]	0.482	0.2282	3.3	0.264
scspill, R spec	500,000	−16.796	[−17.16, −16.46]	0.702	0.3134	66.9	0.254
scspill, corrected	500,000	−16.868	[−23.05, −10.33]	12.713	0.3161	136.8	0.444
mlsynth `SPILLSYNTH(sar)`	500,000	−16.525	[−16.93, −16.18]	0.757	0.2476	135.2	—

Read the third row carefully, because it rules out the explanation most people reach for first. Running the R specification for a hundred times as many iterations does not widen the interval. It goes from 0.482 to 0.702 — still an order of magnitude too narrow — while the effective sample size climbs from 3.3 to 66.9. Chain length was never the problem.

The width comes from departure 3: propagate_alpha. The R code varies $\rho$ across draws while holding the donor weights $\alpha$ fixed at their posterior mean. The reported interval therefore reflects uncertainty about the spatial parameter and none at all about which states make up synthetic California — even though section 8 showed those weights have credible intervals several times wider than the weights themselves. Switching on paired $(\alpha^{(m)}, \rho^{(m)})$ draws restores the missing uncertainty, and the interval grows by a factor of 18.

Effective sample size and interval width are answering different questions here, and it is worth keeping them apart:

ESS asks whether the interval is reliable — whether the chain visited enough of the posterior for its quantiles to mean anything. At ESS 3 they do not.
propagate_alpha asks whether the interval is complete — whether it accounts for everything the model is uncertain about. With $\alpha$ pinned, it does not.

The R edition’s interval failed both tests, which is why it is 33 times too narrow.

Figure 12. Left: the ρ chain under three specifications. Right: effective sample size, with the conventional floor of 100 marked.

The fifth row is the cross-check that matters most. mlsynth.SPILLSYNTH(method="sar") is an independent port of the same paper by a different author, sharing no code with scspill. At the same budget it reports an ATT of −16.525 against scspill’s −16.868 — 0.34 packs apart — and an ESS of 135 against 137. Its interval is narrow because it follows the R convention on $\alpha$. Two independent implementations agreeing on the point estimate and on the diagnostics, while differing exactly where their documented conventions differ, is about as much reassurance as this kind of comparison can offer.

10.2 The six departures

scspill ships the full list as a dataframe, which is the honest way to publish a claim of this kind — the departures are enumerated in the package, not asserted in a blog post.

# scspill's own documentation of where it parts company with the R code.
print(pd.read_csv("scspill_departures.csv").to_string(index=False))

#	Area	R replication code	scspill	Escape hatch	Changes the answer?
1	Covariates	scrambled by a $(T,N,K)$ versus $(N,T,K)$ memory-layout mismatch	a proper $(T, N, K)$ array throughout	drop covariates	yes — the big one
2	Prior on $\beta$	flat-plus-ridge conditional	the paper’s horseshoe	`beta_prior="ridge"`	modestly
3	ATT bands	vary $\rho$ only, $\alpha$ at its posterior mean	paired $(\alpha, \rho)$ draws	`propagate_alpha=False`	yes — the interval
4	$\rho$ sampler	fixed Metropolis step	Robbins–Monro toward 44% acceptance	`adapt_rho=False`	the interval, not the point
5	Factor scales	inconsistent $\omega_k$ conditionals; $\mathcal{C}^{+}(0,1)$ hyperprior	$\mathcal{N}(0, \sigma^2_\eta \omega_k)$ with $\mathcal{C}^{+}(0,10)$	none	little, but the sampler was invalid
6	FFBS initialisation	$\gamma_1$ inconsistent with its own conditionals	coherent $\gamma_0 = 0$	none	little, but the sampler was invalid

Departure 1 deserves a sentence of its own, because it is the most instructive kind of bug. The covariate array in the R code was indexed as though it were $(N, T, K)$ when it was laid out as $(T, N, K)$, which silently shuffles which state’s price goes with which state’s sales. Nothing crashes. Nothing looks wrong. The estimate simply answers a slightly different question than the one asked. It is also the departure with no escape hatch worth using: dropping the covariates entirely is the only way to sidestep it, which trades one specification error for another. A memory-layout mistake was doing a substantial share of the modelling.

10.3 What a joint distribution test catches

Departures 5 and 6 were not found by staring at output. They were found by a Geweke joint distribution test, and they are the reason to run one.

The idea is simple enough to state in two sentences. Draw parameters from the prior and simulate data from them: that gives you samples from the joint distribution of parameters and data, the marginal-conditional route. Alternatively, simulate data once and then run one sweep of your Gibbs sampler, repeatedly: if every conditional distribution in the sampler is correct, this successive-conditional route targets the same joint distribution. Any statistic’s mean must then agree between the two routes, and a systematic disagreement means at least one conditional is wrong.

That is how an $\omega_k$ conditional treating $\omega$ as a variance while its neighbours treated it as a precision came to light, and how an FFBS initialisation inconsistent with its own conditionals came to light. Neither changes the California answer much. Both mean the R sampler was not converging to any posterior at all — it was converging to something, and that something had no interpretation.

This is the argument for the test in general. A sampler with an incoherent conditional does not announce itself. It produces plausible numbers, converges, passes trace-plot inspection, and is wrong.

11. Diagnostics

Three checks, all shipped as first-class functions in scspill, run before any of the numbers above should be believed. They all take the model’s inputs directly rather than the fitted result, so pull those off the fit once:

# result.inputs carries everything the sampler saw, already aligned.
Y0 = np.asarray(result.inputs.Y0, dtype=float).ravel() # treated outcome, (T,)
Yc = np.asarray(result.inputs.Yc, dtype=float) # donor outcomes, (T, N)
if Yc.shape[0] != Y0.size:
Yc = Yc.T
X = None if result.inputs.X is None else np.asarray(result.inputs.X, dtype=float)
T0_idx = result.inputs.T0
Y0_pre, Yc_pre = Y0[:T0_idx], Yc[:T0_idx]
X_pre = None if X is None else X[:T0_idx]
print(f"Y0_pre {Y0_pre.shape} Yc_pre {Yc_pre.shape} "
f"X_pre {None if X_pre is None else X_pre.shape}")

Y0_pre (18,) Yc_pre (18, 38) X_pre (18, 38, 1)

Note the shape of X_pre: a proper $(T_0, N, K)$ array. That third dimension is departure 1 from section 10.2 — the R code indexed the same block as though it were $(N, T_0, K)$.

11.1 Prior predictive check

Does the prior generate data that look anything like the data we have? If not, the posterior is a fight between a badly-specified prior and the likelihood, and the winner is not always the likelihood.

from scspill.validation import prior_predictive, plot_prior_predictive
W_raw, w_raw = result.inputs.W_raw, result.inputs.w_raw
ppc = prior_predictive(Y0_pre, W_raw, w_raw, result.alpha_hat,
Yc_obs=Yc_pre, X=X_pre, p=0,
a0=3.0, b0=1.0, n_draws=2000, seed=SEED)
# PriorPredictiveResult carries `observed`, `stats` and `p_values` rather than
# a ready-made table; assemble the two that matter.
ppc_tab = pd.DataFrame({"statistic": list(ppc.p_values.keys()),
"observed": [ppc.observed[k] for k in ppc.p_values],
"p_value": list(ppc.p_values.values())})
print(ppc_tab.round(4).to_string(index=False))

 statistic observed p_value
yc_mean 131.5000 0.9195
log_yc_var 6.9840 0.7045
spatial_quadratic 17844.7200 0.8200
corr_y0_wyc 0.9450 0.6105
ac1 0.8938 0.9960
ac2 0.8105 0.9985
pve_pc1 0.6248 0.0285
avg_skewness -0.3964 0.4950
avg_kurtosis -0.9427 0.7320

Each row asks whether the observed value of a statistic falls in a plausible region of its prior predictive distribution. Read the column as a two-sided tail probability: values near 0.5 are unremarkable, and values near either 0 or 1 are not. Six of nine land comfortably inside. Three do not, and they tell the same story.

pve_pc1 at 0.028 is the share of variance explained by the first principal component of the donor block. The observed value is 0.625: nearly two-thirds of the joint movement of 38 state cigarette markets is one common factor, and the prior did not expect that much. ac1 at 0.996 and ac2 at 0.9985 are the mirror image, and are in fact further into their tails — the observed lag-1 and lag-2 autocorrelations of 0.894 and 0.810 sit above almost every prior draw. A prior that under-predicts persistence and under-predicts common variance is under-predicting the same thing twice: the national secular decline visible in Figure 1.

This is a mild warning rather than a failure, and it points at a specific, fixable thing: the model has one latent factor (p_factors=1), and the data may want more. The R edition runs a coarser four-statistic visual check at 1,000 draws and reads all four as compatible with the prior; the finer nine-statistic check here is what surfaces the conflict.

Figure 13. Observed statistics against their prior predictive distributions, drawn by plot_prior_predictive.

11.2 The Geweke joint distribution test

Section 10.3 explained what the test does. Running it requires care, and the function’s own documentation says why: the successive-conditional simulator mixes slowly, so an under-resolved run flags spurious failures. Two rules follow from that, and both are in the docs:

Keep the test panel small. A large $T_0 \times N$ makes the $\rho$ chain diffuse slowly, so $T_0 = 4$, $N = 4$ is what the function prescribes.
Do not test the production kernel. Its half-Cauchy scale hierarchies are funnel-shaped and, in the documentation’s own words, “effectively untestable at feasible chain lengths” — which is why the replication package only ever tested the simplified kernel, and then at two million draws.

The documentation also gives the way to tell a real problem from a mixing artifact: a genuine incoherence shows up as a stable, sign-consistent z across seeds and scales; mixing artifacts flip sign and shrink as the chain grows. So run it twice, at two chain lengths.

from scspill.validation import geweke_test
for m in (20_000, 200_000):
rep = geweke_test(kernel="simple", T0=4, N=4, K=0, p=1,
m_iid=m, m_mcmc=m, burn=5_000, seed=SEED)
tab = pd.DataFrame(rep.table)
# GewekeReport does the Bonferroni bookkeeping itself.
print(f"m = {m:>7,} max |z| = {tab['z'].abs().max():.2f} "
f"flagged {rep.n_flagged} of {len(tab)} at |z| > {rep.z_crit:.2f} "
f"passed = {rep.passed}")

m = 20,000 max |z| = 3.48 flagged 1 of 8 at |z| > 2.73 passed = False
m = 200,000 max |z| = 2.50 flagged 0 of 8 at |z| > 2.73 passed = True

At 20,000 draws one statistic (log_yc_var, $z = 3.48$) crosses the Bonferroni threshold of 2.73 and the report comes back passed = False. At 200,000 nothing does, and the maximum score falls from 3.48 to 2.50. Across the eight statistics, $|z|$ shrinks for six.

That is the documented signature of a mixing artefact, not of an incoherent conditional. Had the sampler contained a real error, the score would have held its position or grown as the standard errors tightened around a genuinely wrong mean.

Figure 14. Each statistic’s |z| at 20,000 and 200,000 draws. Arrows show the direction of change; scores that fall are slow mixing, not incoherence.

The pedagogical point is worth more than the result. A single Geweke run that flags a failure tells you almost nothing on its own. Two runs at different scales tell you which kind of failure you have.

11.3 Prior sensitivity

The last check varies the priors and asks whether the answer follows.

from scspill.validation import prior_sensitivity
grid = pd.DataFrame([
dict(a0=1.0, b0=1.0, rho_lo=-0.99, rho_hi=0.99, step_rho=0.05),
dict(a0=3.0, b0=1.0, rho_lo=-0.99, rho_hi=0.99, step_rho=0.05),
dict(a0=0.1, b0=0.1, rho_lo=-0.99, rho_hi=0.99, step_rho=0.05),
dict(a0=1.0, b0=1.0, rho_lo=-0.50, rho_hi=0.50, step_rho=0.05), # truncated
dict(a0=1.0, b0=1.0, rho_lo=-0.99, rho_hi=0.99, step_rho=0.01),
dict(a0=5.0, b0=2.0, rho_lo=-0.99, rho_hi=0.99, step_rho=0.05),
])
sens = prior_sensitivity(Yc, W_raw, w_raw, result.alpha_hat, grid, X=X, p=1,
m_burn=5_000, m_keep=20_000, base_seed=SEED)
# prior_sensitivity returns a PriorSensitivityResult, not a dataframe. The long
# table -- 18 rows over six parameter labels -- is on `.table`. Only rho here.
print(sens.table.query("parameter == 'rho'")
.drop(columns="parameter").round(4).to_string(index=False))

 grid_row step_rho rho_hi rho_lo b0 a0 mean sd q025 q975
0 0.05 0.99 -0.99 1.0 1.0 0.8526 0.0073 0.8387 0.8672
1 0.05 0.99 -0.99 1.0 3.0 0.7825 0.0103 0.7620 0.8025
2 0.05 0.99 -0.99 0.1 0.1 0.7827 0.0107 0.7619 0.8040
3 0.05 0.50 -0.50 1.0 1.0 0.4990 0.0010 0.4964 0.5000
4 0.01 0.99 -0.99 1.0 1.0 0.8425 0.0232 0.7740 0.8638
5 0.05 0.99 -0.99 2.0 5.0 0.8441 0.0239 0.7749 0.8662

A caveat before reading these. prior_sensitivity runs the simplified Step-2 kernel (kernel="simple"), not the production sampler, so the level of $\rho$ here is not the headline 0.316 and should not be compared with it. What is informative is the variation across rows, and there the finding is sharp.

Across the five rows with an unrestricted support, sweeping $a_0$ from 0.1 to 5, $b_0$ from 0.1 to 2 and the Metropolis step size by a factor of five moves the posterior mean of $\rho$ by 0.070. Row 3 truncates the support to $[-0.5, 0.5]$ and the posterior lands at 0.4990 — pinned against the bound, with a standard deviation of 0.001. That is a shift of 0.32, about five times what every conventional prior setting managed put together.

The lesson is not that the inverse-gamma hyperparameters are irrelevant. It is that the support constraint is a prior too, and it is the one nobody thinks to report. In the headline configuration the bound is $|\rho| < 0.95$ and the posterior sits at 0.316, nowhere near it — so this model is not being squeezed. Had the analyst chosen a “conservative-looking” $[-0.5, 0.5]$ range, the answer would have been determined by that choice and by nothing else.

Figure 15. Posterior mean of ρ across six prior settings, with 95% credible intervals. Orange marks the row where the support constraint binds.

12. What the leak actually cost

Section 4.3 derived the bias of a SUTVA-imposing estimator as $-\sum_j \alpha_j \xi^{c}_{j}$ and verified it on three donors. The real panel supplies all the pieces, so it can be checked directly.

alpha = pd.read_csv("stage2_alpha_posterior.csv").set_index("state")["alpha_hat"]
xi = pd.read_csv("stage3_spillover_effects.csv").set_index("state")["avg_spillover"]
contrib = (alpha * xi).sort_values()
print(contrib.head(4).round(4).to_string())
print(f"\nsum_j alpha_j * xi_j : {contrib.sum():+.4f}")
print(f"att (purged) - att_scm (contam.) : {result.att - result.effects_detail.att_scm:+.4f}")

Nevada -1.0984
Utah -0.0178
Idaho -0.0060
Montana -0.0060
sum_j alpha_j * xi_j : -1.1295
att (purged) - att_scm (contam.) : -1.1863

The identity holds to 0.057 packs — the small residual is because the purged ATT averages over paired $(\alpha, \rho)$ draws while the plug-in decomposition uses $\hat\alpha$ alone.

Three things to take from those four numbers:

Nevada is 97% of the story. Its contribution of −1.098 out of −1.130 comes from a weight of 0.200 multiplied by a spillover of −5.50. Utah has a comparable spillover per capita to Idaho but contributes three times as much bias, because Utah carries more weight. This is the “damage is a product of two things” point from section 4.3, visible in a real table.

The direction is the one section 4.3 predicted. The spillovers are negative and the weights are positive, so the bias is positive — the contaminated estimate is closer to zero than the truth. The horseshoe estimate understates Proposition 99’s effect on California by 1.19 packs per capita per year, roughly 7% of the effect. Applying the same plug-in to the simplex weights, where Nevada carries 0.242 rather than 0.200, gives 1.51 packs, about 8%. The classical estimate is the more contaminated of the two, precisely because the constraint pushed more weight onto the one leaking donor.

It is small. After all of this — a spatial model, half a million MCMC draws, a rejected SUTVA assumption — the correction to the headline number is 1.2 packs out of 17. That is the honest summary, and it is worth stating plainly rather than burying: the spillover was real, statistically clear, and substantively modest for California. It was not modest for Nevada, which is a different question and the one classical synthetic control could not have asked.

13. The rest of the catalogue

mlsynth ships 46 estimators. Eight more classes will run on this panel — nine configurations, because SPILLSYNTH has two methods worth separating — and it is worth seeing what they say — with one column that keeps the table from lying.

# Two of the eight need something the bare panel does not supply.
# SpSyDiD wants the 39x39 UNIT-INCLUSIVE matrix, not the 38x38 donor block:
units = ["California"] + donors
W39 = pd.DataFrame(0.0, index=units, columns=units)
W39.loc[donors, donors] = W.values
W39.loc["California", donors] = w.to_numpy()
W39.loc[donors, "California"] = w.to_numpy()
# BPSCS wants point coordinates. Rather than hard-coding state centroids -- a
# second source of truth that could disagree with W -- embed the rook graph
# itself in 2-D by classical MDS on its shortest-path distances. These are NOT
# geographic coordinates, and the results table says so.
D_graph = scipy.sparse.csgraph.shortest_path(W39.to_numpy(), unweighted=True)
D_graph[~np.isfinite(D_graph)] = np.nanmax(D_graph[np.isfinite(D_graph)]) + 1.0
n = len(units)
J = np.eye(n) - np.ones((n, n)) / n # the centring matrix
B = -0.5 * J @ (D_graph ** 2) @ J # double-centred squared distances
ev, evec = np.linalg.eigh(B)
top2 = np.argsort(ev)[::-1][:2]
coords = pd.DataFrame(evec[:, top2] * np.sqrt(np.clip(ev[top2], 0, None)),
index=units, columns=["mds_1", "mds_2"])
df_coords = df.merge(coords.rename_axis("state").reset_index(), on="state", how="left")
print(coords.loc[["California", "Nevada", "Maine"]].round(3).to_string())

 mds_1 mds_2
California -0.039 4.791
Nevada 0.511 3.811
Maine -7.975 -0.182

The sanity check worth running: California and Nevada land 1.12 apart while California and Maine land 9.36 apart, so the embedding has recovered the graph’s coarse geography without ever being shown a map. df_coords is what the BPSCS row of the table below is fitted on.

Estimator	Family	Comparable?	ATT	95% interval	Seconds
`BVSS` — soft simplex, spike-and-slab	Bayesian	yes	−16.32	[−33.83, −6.55]	191.0
`MVBBSC` — Martinez & Vives-i-Bastida	Bayesian	yes	−23.13	[−29.14, −17.10]	13.1
`BFSC` — Bayesian factor SC	Bayesian	yes	−18.10	[−34.96, −0.65]	173.6
`BPSCS` — penalised SC under spillovers	spillover	yes	−17.19	[−32.04, +1.80]	77.4
`SPILLSYNTH(sar)` — the same paper, ported	spillover	yes	−16.52	[−16.93, −16.18]	994.4
`SPOTSYNTH` — spillover-detecting SC	spillover	yes	−26.32	[−29.22, −23.87]	6.6
`SPILLSYNTH(cd)` — Cao & Dowd	spillover	no	−2.77	—	0.3
`SpSyDiD` — spatial synthetic DiD	spillover	no	−17.11	—	0.02
`ISCM` — imperfect/inclusive SC	spillover	no	−37.76	[−136.28, +60.76]	0.2

All nine ran; none errored. Four observations.

The comparable column is doing real work. Those three “no” rows are not failures — they are estimators answering different questions, and reading them against the ladder would produce nonsense:

SPILLSYNTH(cd) measures against its own no-spillover baseline of −10.52, not the simplex’s −18.43, so its −2.77 is a difference from a different starting point. Its Nevada spillover comes out at +12.77, opposite in sign to everything else here, because the Cao–Dowd construction demeans and leaves one out.
SpSyDiD reports three effects at once: a direct effect of −17.11, a total effect of −20.16, and an average indirect effect of +14.88. Quoting the first without the other two would be a choice, not a reading.
ISCM returns −37.76 with an interval spanning [−136, +61]. It is not wrong; it is uninformative on 18 pre-treatment periods, and it says so.

Among the comparable rows, the spread is real but bounded. Six estimators, six prior structures, spanning −16.3 to −26.3. MVBBSC and SPOTSYNTH sit furthest out — the first imposes a hard simplex with a Bernstein–von Mises interval, the second removes contaminated donors from the pool entirely rather than modelling them. Both are defensible designs, and neither changes the sign or the order of magnitude.

Runtime spans nearly five orders of magnitude — 0.02 seconds for SpSyDiD against 994 seconds for SPILLSYNTH(sar) at the headline budget. That is not a quality ranking. It is the difference between a closed-form estimator and a half-million-draw MCMC, and it is worth knowing before you put one in a bootstrap loop.

Two of the eight needed input the panel does not carry. SpSyDiD raised MlsynthDataError: Spatial matrix W has shape (38, 38); expected (39, 39) until given the unit-inclusive matrix, and BPSCS raised MlsynthConfigError: BPSCS coordinate column(s) not in the panel: ['lon','lat'] until given coordinates. Both are good errors — loud, specific, and naming the fix.

Figure 16. Teal markers target the same estimand as Stages 1 and 3 and can be read against the dashed reference lines. Grey markers cannot.

14. How long must the chain be?

Section 9.5 used 500,000 iterations to estimate a 13-year effect. That needs justifying, and the justification is not “more is better.”

for m in (5_000, 20_000, 50_000, 100_000, 250_000, 500_000):
r = SCSPILL({**panel.config_kwargs(), "m_iter": m, "burn": m // 2,
"seed": SEED, "display_graphs": False,
"max_effect_draws": 5_000}).fit()
print(f"{m:>7,} ATT {r.att:+.4f} width {r.att_ci[1] - r.att_ci[0]:6.3f} "
f"rho {r.rho_hat:.4f} ESS {r.rho_ess:7.2f} acc {r.acc_rho:.3f}")

 5,000 ATT -16.6683 width 11.819 rho 0.3183 ESS 2.75 acc 0.403
20,000 ATT -16.9258 width 12.520 rho 0.3486 ESS 5.82 acc 0.435
50,000 ATT -17.0050 width 12.770 rho 0.3454 ESS 13.30 acc 0.435
100,000 ATT -16.8962 width 12.417 rho 0.3361 ESS 24.22 acc 0.443
250,000 ATT -16.8458 width 12.632 rho 0.3224 ESS 74.60 acc 0.444
500,000 ATT -16.8680 width 12.713 rho 0.3161 ESS 136.78 acc 0.444

Iterations	ATT	CrI width	$\hat\rho$	ESS($\rho$)	Acceptance
5,000	−16.668	11.82	0.3183	2.8	0.403
20,000	−16.926	12.52	0.3486	5.8	0.435
50,000	−17.005	12.77	0.3454	13.3	0.435
100,000	−16.896	12.42	0.3361	24.2	0.444
250,000	−16.846	12.63	0.3224	74.6	0.444
500,000	−16.868	12.71	0.3161	136.8	0.444

Two columns move at completely different rates. The ATT spans 0.337 packs across the whole ladder — a hundredfold increase in compute buys a third of a pack, which is nothing next to the 3.17-pack spread across estimators in section 16. ESS($\rho$) spans 2.8 to 136.8, and does not cross the conventional floor of 100 until half a million draws.

That is the whole argument for the budget. The estimand you came for converges early; the nuisance parameter that makes the third stage possible does not.

It is also worth reading the first row against section 10. At only 5,000 iterations, the corrected configuration already reports an interval 11.82 packs wide — against the R edition’s 0.384 at the same budget. The width was never about the chain length. It was about propagating $\alpha$.

Figure 17. Teal: the ATT and its credible interval. Gold: the effective sample size for ρ, against the conventional floor of 100.

Read the last column as a rate rather than a level. From 20,000 iterations onward the table delivers a near-constant 0.00055 effective draws per kept draw — 5,816 / 10,000, then 13,296 / 25,000, 24,215 / 50,000, 74,598 / 125,000, 136,781 / 250,000. Effective sample size is growing roughly linearly in chain length, not tailing off. There is no diminishing-returns cliff here to stop at; there is a fixed, poor exchange rate. At that rate an ESS of 400 costs about 1.5 million iterations, or a few more minutes of sampling. Whether that is a good use of a laptop is the question exercise 4 asks.

15. Monte Carlo: does modelling the leak pay?

Everything so far has been one panel where the truth is unknown. scspill ships a simulation module, so the same question can be asked where the truth is planted.

from scspill.simulate import mc_grid
mc = mc_grid(Ns=(16,), T0s=(20,), T1=10,
rhos=(-0.6, -0.3, -0.1, 0.0, 0.1, 0.3, 0.6),
sims_per=60, K=1, beta=(1.0,), sigma2=0.1,
treated=(0, 1, 2, 3), m_iter=3_000, burn=1_000,
step_rho=0.05, seed=SEED)
print(mc.round(4).to_string(index=False))

 N T0 T1 rho method bias_point rmse_point cover95_point
16 20 10 -0.6 SCM -0.0848 0.4111 NaN
16 20 10 -0.6 BSCM -0.1356 0.1896 0.0017
16 20 10 -0.6 SCSPILL -0.0003 0.0041 0.9833
16 20 10 -0.3 SCM -0.0877 0.3690 NaN
16 20 10 -0.3 BSCM -0.0705 0.0995 0.0033
16 20 10 -0.3 SCSPILL -0.0012 0.0069 0.9833
16 20 10 -0.1 SCM -0.0080 0.3353 NaN
16 20 10 -0.1 BSCM -0.0260 0.0363 0.0050
16 20 10 -0.1 SCSPILL -0.0029 0.0093 0.9500
16 20 10 0.0 SCM 0.0069 0.3268 NaN
16 20 10 0.0 BSCM -0.0000 0.0000 1.0000
16 20 10 0.0 SCSPILL 0.0005 0.0083 1.0000
16 20 10 0.1 SCM 0.0147 0.3057 NaN
16 20 10 0.1 BSCM 0.0285 0.0403 0.0067
16 20 10 0.1 SCSPILL -0.0000 0.0113 0.9517
16 20 10 0.3 SCM 0.0787 0.3121 NaN
16 20 10 0.3 BSCM 0.0978 0.1443 0.0017
16 20 10 0.3 SCSPILL -0.0015 0.0170 0.9033
16 20 10 0.6 SCM 0.2392 0.4709 NaN
16 20 10 0.6 BSCM 0.2880 0.4041 0.0000
16 20 10 0.6 SCSPILL 0.0033 0.0263 0.9667

A 4×4 rook lattice, 20 pre-periods, 10 post-periods, 60 replications per cell, and a known spillover intensity. Three estimators see each dataset.

True $\rho$	Bias: SCM	Bias: BSCM	Bias: SCSPILL	Coverage: BSCM	Coverage: SCSPILL
−0.6	−0.085	−0.136	−0.0003	0.002	0.983
−0.3	−0.088	−0.071	−0.0012	0.003	0.983
−0.1	−0.008	−0.026	−0.0029	0.005	0.950
0.0	+0.007	−0.000	+0.0005	1.000	1.000
+0.1	+0.015	+0.029	−0.0000	0.007	0.952
+0.3	+0.079	+0.098	−0.0015	0.002	0.903
+0.6	+0.239	+0.288	+0.0033	0.000	0.967

The bias column behaves exactly as section 9.1’s identity predicts. SCM and BSCM are unbiased at $\rho = 0$ and nowhere else, and the bias grows with $|\rho|$ in the direction of the spillover — monotonically for BSCM, and monotonically for SCM up to Monte Carlo error. SCSPILL sits within 0.004 of zero at every value, including $\rho = \pm 0.6$.

The coverage column is more striking and needs care in reading. SCSPILL’s intervals cover at 0.90–0.98 against a nominal 0.95 — good, though the 0.903 at $\rho = 0.3$ is a real dip. BSCM’s cover at essentially zero everywhere except $\rho = 0$. That is not because BSCM is broken. Its posterior intervals in this design are far narrower than its own sampling spread. At $\rho = -0.6$ its root-mean-square error across replications is 0.19, of which 0.136 is bias — yet its credible intervals are narrow enough that a bias of that size falls outside them in essentially every replication. The interval is describing the posterior’s precision, not the estimator’s accuracy, and under a misspecified model those are different things. A narrow interval plus a small bias equals no coverage, which is precisely the failure mode section 10 diagnosed on the real data.

One detail worth noting because it is easy to over-read: at $\rho = 0$ exactly, BSCM’s bias and RMSE are both 0.0000 while classical SCM’s RMSE is 0.327. With no spillover, the perfect-fit assumption holds exactly and an unconstrained estimator recovers the truth exactly. The simplex still cannot — a third of a unit of error at $\rho = 0$ is the price of the constraint, showing up in simulation just as section 4.2 priced it by hand.

Two honest caveats. Sixty replications per cell gives a Monte Carlo standard error on a coverage of 0.95 of about 0.028, so differences of a percentage point or two are noise; the paper’s own design uses 1,000. And mc_grid’s inner sampler fixes adapt_rho=False to mirror the reference study, so this section cannot demonstrate the adaptation fix from section 10 — it isolates the modelling question, not the sampling one.

Figure 18. Left: bias against the planted ρ. Right: coverage of the nominal 95% interval, with the estimated ρ̂ for California marked.

At the $\rho$ this post estimates for California — 0.316 — the simulation says a SUTVA-imposing estimator carries a bias of around 0.08 to 0.10 in the units of that design. The simulation design is on a different scale, so the two numbers are not convertible — but the direction and the relative size agree: 1.19 packs against an effect of 16.87.

16. The whole ladder

Every number above, in one table. The r_reference column is the same panel estimated by the authors’ own R and C++ code, and r_gap is the honest measure of how much the implementation choice mattered.

print(pd.read_csv("att_ladder.csv").round(4).to_string(index=False))

Stage	Engine	ATT	95% interval	Active donors	R edition	Gap
1. Classical SC	mlsynth	−18.428	[−22.08, −14.30]	5	−18.46	+0.032
2a. Bayesian SC (BSCM)	mlsynth	−18.847	[−26.46, −9.99]	26	—	—
2b. Bayesian SC ($\rho = 0$)	scspill	−15.682	—	25	−15.84	+0.158
3. Bayesian spatial SC	scspill	−16.868	[−23.05, −10.33]	25	−16.59	−0.278
3′. Bayesian spatial SC	mlsynth	−16.525	[−16.93, −16.18]	—	−16.59	+0.065

Three observations close the argument.

Every stage agrees on the sign, and the spread is 3.17 packs. From −18.85 to −15.68 across two libraries, four prior structures and one spatial layer. Proposition 99 reduced per-capita cigarette sales in California by somewhere between 15 and 19 packs a year, and no defensible modelling choice in this post moves it outside that band.

The maximum disagreement with the R edition is 0.278 packs, across four comparable stages. Two implementations, two languages, two authors, one of them using hand-written C++ and the other a pip-installable package — agreeing to within about 1.6% on every stage. That is the strongest evidence either edition offers that its estimator is correctly coded.

The active donor count is the unstable quantity, not the effect. Five under the simplex, 25–26 under every prior. If your conclusion is “the ATT is about −17”, the modelling choices barely matter. If your conclusion is “synthetic California is mostly Utah, Nevada, Montana and Connecticut”, it rests entirely on a constraint you chose.

Figure 19. Point estimates and intervals across the ladder. Hollow diamonds are the R edition’s published values.

17. Which estimator should you choose?

Nothing in this post argues that the spatial model is always the right one. It argues that the spatial model answers a question the others cannot, and that you should know which question you are asking before you pick.

graph TD
Q1{"Could the treatment<br/>have reached any<br/>donor unit?"}
Q1 -->|"No, and you can defend it"| Q2{"Is the treated unit<br/>inside the donors'<br/>convex hull?"}
Q1 -->|"Yes, or you cannot rule it out"| Q3{"Do you have a credible<br/>exposure structure<br/>(w and W)?"}
Q2 -->|Yes| SC("<b>Classical SC</b><br/>VanillaSC<br/>interpretable, sparse")
Q2 -->|"No, or the pre-fit is poor"| BSC("<b>Bayesian SC</b><br/>BSCM or scspill at rho=0<br/>extrapolation allowed")
Q3 -->|Yes| SAR("<b>Bayesian spatial SC</b><br/>SCSPILL method='sar'<br/>two estimands")
Q3 -->|"No, but you can name<br/>the affected units"| ALT("<b>Screen or net out</b><br/>SPOTSYNTH, ISCM<br/>SPILLSYNTH method='cd'")
classDef sty_Q1 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q1 sty_Q1
classDef sty_Q2 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q2 sty_Q2
classDef sty_Q3 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q3 sty_Q3
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class SC blue
class BSC key
class SAR teal
class ALT orange

The first question is the one that gets skipped, and it is the only one with no statistical answer. Whether Proposition 99 could plausibly have reached Nevada is a question about cigarettes, borders and advertising, not about panels. The data can tell you how large the leak was given that you allowed for one; they cannot tell you to look.

The second question is a diagnostic you already have. A classical fit with a poor pre-treatment RMSE relative to the outcome’s scale is telling you the treated unit is hard to reach from inside the hull — exactly the situation section 4.2 constructed. Here the RMSE was 1.60 against a mean of 117.7, so the simplex was not obviously straining.

The third question is the practical constraint. The spatial model needs someone to supply $\mathbf{w}$ and $W$, and those are modelling choices carrying real content. Contiguity is the natural default for a tax, but trade flows, migration or commuting intensity might be better for other policies — the Sudan application shipped with scspill uses bilateral trade for exactly that reason.

18. Discussion

Section 1 asked how much of the Proposition 99 answer each assumption was carrying. Three answers.

The effect on California is robust. Across two libraries, four prior structures, one spatial layer and an independent third implementation, every estimator on the ladder reports between −15.7 and −18.8 packs per capita per year, and every interval on the ladder excludes zero. Widening to the six comparable estimators of section 13 stretches the range to −16.3 and −26.3 without ever changing the sign, and only BPSCS, whose interval spans [−32.04, +1.80], leaves zero admissible. Proposition 99 worked, and it worked at a magnitude that no reasonable modelling choice moves by more than about 20%. If your interest is the headline number, the classical estimate was fine.

The composition of the donor pool is not robust at all. The same data support five active donors or 25, depending entirely on whether sparsity is imposed by a constraint or expressed as a prior. Sentences of the form “synthetic California is mostly Utah, Nevada, Montana and Connecticut” read like findings and are closer to artefacts of $\Delta$. The gap between what is stable (the ATT) and what is not (the weights) is worth internalising, because the weights are the part that gets narrated.

SUTVA is false here, and the direction is the surprise. Nevada absorbed a spillover of −5.50 packs per capita per year, an order of magnitude more than any other state, with a credible interval for $\rho$ that excludes zero. The prior expectation was cross-border shopping raising Nevada’s sales; the estimate says they came in below their no-treatment path. Whatever mechanism dominates — advertising, media, social norms crossing a border that tax arbitrage also crosses — the net effect on Nevada ran the same way as the effect on California, not against it.

That has a consequence for how the policy should be described. Reporting only California’s number understates Proposition 99’s total public-health footprint, because a neighbouring state that never voted on it also smoked less. It also means the classical estimate was biased toward zero: the honest version of “the classical estimate was fine” is “the classical estimate was fine, and slightly conservative, for a reason it could not have told you about.”

What this post does not establish. The SAR layer does not make anything causal that was not causal before. It is a model of how outcomes co-move across a fixed, researcher-supplied graph, and swapping contiguity for a different graph would produce different spillovers. The identifying assumption in section 9.3 — that some fixed unconstrained combination of donors reproduces California exactly — is strong and untestable. And $\rho$ remains the weakly identified parameter of the model even at half a million draws: an effective sample size of 137 is reportable, not comfortable. The right deliverable from this analysis is the interval together with its ESS, not the point estimate on its own.

19. Summary and next steps

Method. Three nested estimators on one panel. The simplex constrains weights to a convex combination; the horseshoe replaces that constraint with a prior that prefers zero without forbidding anything; the SAR layer drops SUTVA on the donor pool and adds a second estimand. Each stage keeps everything the previous one assumed but one thing.
Data. The Abadie–Diamond–Hainmueller Proposition 99 panel, 39 states over 1970–2000, bundled with rook-contiguity weights in which Nevada is California’s only donor-pool neighbour. Treatment pinned at 1988 throughout so all three stages share a post-period.
Result. ATT of −18.43 (simplex), −15.68 (horseshoe), −16.87 (spatial), with $\hat\rho = 0.316$ excluding zero and a Nevada spillover of −5.50 packs, 11 times the next-largest donor. Modelling the leak makes the estimated effect larger, because the contaminated donor was biased in the same direction as the treated unit.
Inferential lesson. The R edition of this post reported a 95% interval 0.38 packs wide from a chain whose ESS for $\rho$ this post recomputes as 2.93. The corrected run reports 12.71 packs from an ESS of 137. Nothing about the policy changed. Report the effective sample size beside every credible interval, or the interval is decoration.
Limitation. $\rho$ is the slowest-mixing parameter in the model rather than the least identified one. Its posterior is tight — a standard deviation of 0.043 on a support 1.9 wide — but its chain is heavily autocorrelated, because the two-step sampler moves it by random-walk Metropolis conditional on a strongly correlated factor block. Half a million draws buys an ESS of 137. Report that number beside the interval; do not read it as a statement about how much the data know.
Next step. Everything here conditions on a contiguity graph nobody estimated. The natural follow-up is Bayesian estimation of the spatial weight matrix itself, which asks the data who the neighbours are rather than assuming a border tells you.

The R edition of this post runs the same three stages with the authors’ own R and C++ code, and the synthetic control ladder in Python climbs a different set of stages — difference-in-differences through synthetic difference-in-differences — on the Brexit referendum.

20. Exercises

Move the treatment year. Rebuild df["treated"] at 1989 rather than 1988, following the Abadie–Diamond–Hainmueller convention, and rerun all three stages. How much does the ATT move, and is the change larger or smaller than the gap between the simplex and the horseshoe? Explain why one post-treatment year matters as much or as little as it does.
Break the graph on purpose. Zero out Nevada’s entry in panel.spatial_w so the model believes no donor borders California, and refit. What happens to $\hat\rho$, to the ATT, and to the spillovers assigned to Idaho and Utah? This is the cleanest way to see how much of section 9’s story is carried by a single entry in a single vector.
Change what “neighbour” means. Replace rook contiguity with an inverse-distance or a $k$-nearest-neighbours weight matrix built from state centroids, row-normalise it, and refit. Does the Nevada result survive? Section 17 argues the choice of $W$ carries real content; this exercise measures how much.
Buy a better $\rho$. Section 14 shows effective sample size growing almost linearly in chain length, at about 0.00055 effective draws per kept draw. Extrapolate: how many iterations would an ESS of 400 take? Run it, check whether the linear rate actually holds that far out, and decide whether the answer changes anything you would report.
A second case study. scspill.data.load_sudan() ships the other application from the paper — 34 African countries, 2000–2015, South Sudan’s 2011 secession, with weights built from bilateral trade rather than borders. Run the same three stages. The trade-based exposure structure is dense where contiguity was sparse, so $\rho$ should be much better identified. Check whether it is.

21. References and further reading

Abadie, A., Diamond, A. and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746
Abadie, A. and Gardeazabal, J. (2003). The economic costs of conflict: a case study of the Basque Country. American Economic Review, 93(1), 113–132. https://doi.org/10.1257/000282803321455188
Abadie, A. (2021). Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425. https://doi.org/10.1257/jel.20191450
Sakaguchi, S. and Tagawa, H. (2026). Identification and Bayesian inference for synthetic control methods with spillover effects. The Econometrics Journal. https://doi.org/10.1093/ectj/utag006. Working paper: arXiv:2408.00291. Replication package: Zenodo 10.5281/zenodo.19066186
Carvalho, C. M., Polson, N. G. and Scott, J. G. (2010). The horseshoe estimator for sparse signals. Biometrika, 97(2), 465–480. https://doi.org/10.1093/biomet/asq017
Makalic, E. and Schmidt, D. F. (2015). A simple sampler for the horseshoe estimator. IEEE Signal Processing Letters, 23(1), 179–182. https://doi.org/10.1109/LSP.2015.2503725
Kim, S., Lee, C. and Gupta, S. (2020). Bayesian synthetic control methods. Journal of Marketing Research, 57(5), 831–852. https://doi.org/10.1177/0022243720936230
LeSage, J. and Pace, R. K. (2009). Introduction to Spatial Econometrics. Chapman and Hall/CRC. https://doi.org/10.1201/9781420064254
Geweke, J. (2004). Getting it right: joint distribution tests of posterior simulators. Journal of the American Statistical Association, 99(467), 799–804. https://doi.org/10.1198/016214504000001132
Robbins, H. and Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586
Roberts, G. O., Gelman, A. and Gilks, W. R. (1997). Weak convergence and optimal scaling of random walk Metropolis algorithms. The Annals of Applied Probability, 7(1), 110–120. https://doi.org/10.1214/aoap/1034625254
Carter, C. K. and Kohn, R. (1994). On Gibbs sampling for state space models. Biometrika, 81(3), 541–553. https://doi.org/10.1093/biomet/81.3.541
Vehtari, A., Gelman, A., Simpson, D., Carpenter, B. and Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: an improved $\widehat{R}$ for assessing convergence of MCMC. Bayesian Analysis, 16(2), 667–718. https://doi.org/10.1214/20-BA1221
Cao, J. and Dowd, C. (2019). Estimation and inference for synthetic control methods with spillover effects. arXiv:1902.07343
Di Stefano, R. and Mellace, G. (2024). The inclusive synthetic control method. arXiv:2403.17624
Grossi, G., Mariani, M., Mattei, A., Lattarulo, P. and Öner, Ö. (2025). Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. Journal of the Royal Statistical Society Series A, 188(1), 223–240. https://doi.org/10.1093/jrsssa/qnae052
Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W. and Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159
Mendez, C. (2026). scspill: synthetic control models with spillover effects. Python package version 0.2.1. https://quarcs-lab.github.io/scspill/
Greathouse, J. (2026). mlsynth: a Python library for synthetic control and related methods. https://mlsynth.readthedocs.io/
Cunningham, S. (2021). Causal Inference: The Mixtape. Yale University Press. https://mixtape.scunning.com/
Mendez, C. (2026). Bayesian spatial synthetic control: California’s Proposition 99 in R. https://carlos-mendez.org/tutorials/r_sc_bayes_spatial/

Acknowledgements

This tutorial was prepared with the assistance of AI tools for code generation, drafting and editing. All analytical choices, interpretations and any remaining errors are the author’s own.

The Synthetic Control Ladder in Python: A Guided Tour of mlsynth on the Brexit Referendum

Sun, 02 Aug 2026 00:00:00 +0000

Abstract

Estimating what a policy cost requires building a version of the world in which it never happened, and the software for doing that is scattered across a dozen packages with a dozen different interfaces. This tutorial is a guided tour of mlsynth, a Python library that puts ninety-two panel-data causal estimators behind a single configuration dictionary and a single result object, and it uses that library to climb the whole ladder of single-treated-unit estimators — difference-in-differences, synthetic control, demeaned synthetic control, synthetic difference-in-differences in three flavours, matching-and-synthetic-control and augmented synthetic control — one class per stage. The data are quarterly log real GDP for 24 OECD economies from 1995Q1 to 2020Q4, leaving the United Kingdom as the treated unit, 23 donors and 86 pre-treatment quarters. Dating the referendum at 2016Q3 and matching on outcomes alone, the estimated shortfall in UK GDP at the end of 2018 is 3.04% under synthetic control, 2.99% under demeaned SC, 2.80% under SDID, 2.73% under MASC and 3.04% under augmented SC, widening to between 3.83% and 4.19% a year later — every one above the 2.4% previously published for this dataset. An in-sample placebo tournament over twenty artificial treatment dates ranks the SDID family first at 0.0066 log points of root mean squared error against 0.0086 for plain synthetic control. Three findings emerge that only a package-level reading produces: three of the library’s defaults each change the answer by more than the spread across the entire ladder, one estimator silently rounds the number you are most likely to quote, and the three routes mlsynth offers for “controlling for a covariate” disagree by 1.8 percentage points.

1. Overview

On 23 June 2016 the United Kingdom voted to leave the European Union. Three and a half years later UK GDP was some number of percentage points below where it would otherwise have been. The trouble is “otherwise have been”: there is one United Kingdom, it took the treatment, and the version that stayed in the EU exists nowhere in the data.

The standard move is to build that missing country out of the countries we do observe — weight the other OECD economies, add them up, and require the blend to track the real UK quarter by quarter through the two decades before the referendum. That is the synthetic control method, and it is one stage of a ladder that starts at difference-in-differences and climbs through several increasingly flexible estimators.

The R edition of this post climbs that ladder the hard way: every estimator is hand-coded in twenty lines before its package is called, and four separate R packages are needed to cover the six stages. This post makes a different argument. Every stage here is a single class from one library, mlsynth, and the interesting work is not in deriving the estimators but in driving the package: which configuration field selects which estimator, what the result object actually contains, and — the part that turns out to matter most — where a default will quietly hand you a different estimator than the one you meant to fit.

That last point is not a minor caveat. By the end of this post you will have seen three separate defaults that change the headline estimate materially, two of them by more than the spread across the entire six-stage ladder. Knowing the econometrics is not enough. You have to know the software.

1.1 Learning objectives

By the end of this tutorial you will be able to:

Install mlsynth and read its one-config-dict, one-result-object interface, including the Pydantic validation that turns a typo into an exception instead of a silently wrong answer.
Map each stage of the synthetic-control ladder onto a specific mlsynth class and configuration field, and explain why mlsynth.DSC is not the DSC on this ladder.
Extract the average treatment effect on the treated, the donor weights, the time weights, the counterfactual path and the event study from a fitted result.
Identify the three defaults — zeta, set_f and the covariate method — that change the answer materially, and set each one deliberately.
Compare estimators on a common in-sample placebo tournament and read the resulting ranking with appropriate scepticism.
Choose among the wider mlsynth catalogue when your design is not the canonical one-treated-unit case.

1.2 The road ahead

Each stage of this ladder exists because the stage below it gets something wrong. The diagram traces that sequence of complaints, and names the mlsynth class that answers each one.

flowchart TD
D("<b>Panel data</b><br/>24 countries, 104 quarters<br/>one treated unit") --> Q0{"Which donors<br/>count, and by<br/>how much?"}
Q0 -->|"all of them, equally"| DID("<b>Stage 0 — DiD</b><br/>FDID(...).fit().did")
DID --> Q1{"But the donors do<br/>not look like the UK"}
Q1 --> SC("<b>Stage 1 — SC</b><br/>VanillaSC")
SC --> Q2{"But the blend must<br/>match the LEVEL, not<br/>just the shape"}
Q2 --> DSC("<b>Stage 2 — DSC</b><br/>TSSC(method='MSCa')")
DSC --> Q3{"But every pre-period<br/>counts the same"}
Q3 --> SDID("<b>Stage 3 — SDID</b><br/>SDID(zeta=0.0)")
SDID --> PIVOT("<b>The pivot</b><br/>extrapolation bias vs<br/>interpolation bias")
PIVOT --> MASC("<b>Stage 4 — MASC</b><br/>MASC(set_f=...)")
PIVOT --> ASCM("<b>Stage 5 — ASCM</b><br/>VanillaSC(augment='ridge')")
MASC --> SEL("<b>Which stage?</b><br/>in-sample placebo<br/>tournament")
ASCM --> SEL
SEL --> INF("<b>Inference</b><br/>six methods, one flag")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
class D,PIVOT anchor
class Q0,Q1,Q2,Q3 gray
class DID,SC,MASC,ASCM blue
class DSC teal
class SDID orange
class SEL,INF key

Read the diagram top to bottom as a conversation. Every arrow labelled “but” is an objection to the stage above it, and every box below an objection is the estimator that answers it. The two boxes hanging off the pivot are not a further step up but two different reactions to the same discovery, which is why the ladder branches there rather than continuing.

2. Key concepts

Eight ideas carry the whole post. Two repay slow reading: the distinction between unit weights and time weights, and the difference between an estimator and the solver that fits it.

The missing counterfactual and the donor pool. There is one United Kingdom and it took the treatment. The path it would have followed without Brexit is in no dataset. Synthetic control builds that path from countries that were not treated, and those countries are the donor pool.

Example

After dropping the twelve OECD countries with incomplete records, 24 remain. The UK is the treated unit; the other 23 — from Australia to the United States — are the donor pool. In mlsynth you never name them: the library reads the donor pool off the treat column, which is 1 for the treated unit in post-treatment periods and 0 everywhere else.

Analogy

The master tape of a song is lost and the band has broken up. You hire session musicians and rehearse them against a bootleg until they are indistinguishable from the original, then have them play a song the original band never recorded. The donor pool is the pool of session musicians; the pre-treatment period is the rehearsal.

The simplex. Synthetic control weights must be non-negative and sum to one. That set of allowed weight vectors is the simplex, and the blends it can reach form the convex hull of the donors.

Example

mlsynth reports the constraint it imposed. VanillaSC returns weights.summary_stats["constraint"] as "simplex (non-negative, sum to 1)", and only nine of the 23 donors come back with any weight at all: Hungary 0.2231, the United States 0.1926, Japan 0.1826, Canada 0.1751, Norway 0.1350, and four smaller ones.

Analogy

Hammer a pin into a corkboard for every donor country, stretch a rubber band around all the pins and let it snap tight. Everything inside is reachable by some blend; nothing outside is. Reaching outside would need a negative amount of some country, like a recipe calling for minus two eggs.

Unit weights and time weights. Unit weights say how much each donor country counts. Time weights say how much each pre-treatment quarter counts. Both are chosen by the same kind of optimisation, run in two different directions.

Example

mlsynth keeps them in separate places, and finding the second one is the single most common stumbling block. Unit weights are result.donor_weights; SDID’s time weights are result.cohorts[a].time_weights. Ours put 0.9585 on 2016Q2 and roughly nothing on the other 85 quarters.

Analogy

A mixing desk has two banks of faders. The first sets how loud each instrument is; the second sets which seconds of the rehearsal tape you play back when you check the mix. Difference-in-differences leaves both banks flat. Synthetic control moves the first. Synthetic difference-in-differences moves both.

The intercept, which is a unit fixed effect in disguise. Sometimes the blend moves in near-perfect parallel with the treated unit but sits at a slightly different level. The intercept is the average pre-treatment gap, subtracted off, and adding it is exactly the same as putting a unit fixed effect in the regression.

Example

For the UK, TSSC’s MSCa variant estimates an intercept of $+0.00241$ log points, about a quarter of one per cent of GDP. That is why demeaned SC lands at 2.99% and plain SC at 3.04%: a small intercept is evidence that the SC fit was already level-balanced.

Analogy

Your bathroom scale reads two kilograms heavy. You do not throw it out; you subtract two. Plain synthetic control insists on a scale that is already exactly right and will reject a perfectly consistent one. Demeaned synthetic control just calibrates the offset.

Extrapolation bias and interpolation bias. Two different ways a weighted counterfactual goes wrong. Extrapolation bias: the blend’s characteristics do not match the treated unit’s. Interpolation bias: the characteristics match, but the outcome is a curved function of them, so averaging outcomes is not the same as the outcome at the average. Neither name means quite what you would guess.

Example

The unit weights $\omega$ attack extrapolation bias; matching attacks interpolation bias; SDID’s time weights $\lambda$ are what let one estimator attack both. That claim is the theoretical contribution of the paper this post replicates, and section 16 is where we test it.

Analogy

Extrapolation bias is guessing a stranger’s weight from a photograph of someone else. Interpolation bias is averaging the heights of a five-year-old and a fifty-year-old and calling the result the height of a typical twenty-seven-year-old — both inputs are real people, but growth is not linear in age.

The configuration object. Every mlsynth estimator takes one dictionary, validated by Pydantic. Five fields are always the same: df, outcome, treat, unitid, time. Everything else is estimator-specific.

Example

Swapping VanillaSC for SDID in a script means changing one word. The five data fields, the DataFrame, the treatment indicator and the call pattern Estimator(config).fit() are all identical. The configs set extra="forbid", so writing backendd= instead of backend= raises MlsynthConfigError rather than being ignored.

Analogy

A camera body with interchangeable lenses. The grip, the shutter button and the memory card do not change when you swap a wide angle for a macro; only the glass does, and only the glass has its own settings.

The solver’s fingerprint. An estimator is a mathematical object. Fitting it requires an optimiser, and on a badly conditioned problem two correct optimisers stop in different places.

Example

mlsynth hands the synthetic-control problem to a convex solver and returns 3.039%. R’s synthdid walks the same objective with Frank-Wolfe on a capped iteration budget and stops at 3.06%. Neither is buggy. Section 15 shows why the difference is the interesting part.

Analogy

Two hikers are told to find the lowest point of a wide, almost flat valley in fog. They both walk downhill and they both stop when the ground stops obviously falling away. They end up two hundred metres apart, and both are following the instructions correctly.

A three-letter trap: DSC. mlsynth ships a class named DSC. It is not the estimator on this ladder, and importing the wrong one raises no error at all.

Example

mlsynth.DSC is Distributional Synthetic Control (Gunsilius 2023): it matches whole outcome distributions in Wasserstein space and needs micro-level data with many observations per unit-period. This post’s DSC is Demeaned Synthetic Control, which in mlsynth is TSSC(method="MSCa"). Section 6 is devoted to this.

Analogy

Two colleagues in the same building are both called J. Smith. Sending the quarterly report to the wrong one does not bounce. It just arrives somewhere useless, and you find out weeks later.

With the vocabulary in place, we can install the library.

3. Setup: installing mlsynth

3.1 Install and version pinning

mlsynth is on PyPI, but both the README and the documentation still recommend installing from GitHub — and there is a concrete reason to follow that advice rather than reach for pip install mlsynth.

The PyPI release numbered 1.0.0 is behind git main at the same version number. Installing from PyPI gives you a package that reports mlsynth.__version__ == "1.0.0" but is missing VanillaSCConfig.w_constr, which sections 9.2 and 19 use. Nothing in the version string warns you. So install from git, and pin the commit:

# The moving target:
# pip install -U "git+https://github.com/jgreathouse9/mlsynth.git"
#
# The exact commit this post was verified against:
# pip install -U "git+https://github.com/jgreathouse9/mlsynth.git@15f168b"
#
# Optional extras:
# "mlsynth[design] @ git+..." SCIP solver, for SYNDES / MAREX designs
# "mlsynth[bayes] @ git+..." NumPyro, for SPOTSYNTH's Bayesian mode
# "mlsynth[all] @ git+..." everything
import warnings
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import mlsynth
from mlsynth import FDID, MASC, SDID, TSSC, VanillaSC
from mlsynth.exceptions import MlsynthConfigError, MlsynthDataError
print("mlsynth", mlsynth.__version__)
print("estimators exported:", len(mlsynth.__all__))

mlsynth 1.0.0
estimators exported: 92

Everything below was produced with mlsynth 1.0.0 at commit 15f168b, Python 3.13.11, NumPy 2.3.5, pandas 3.0.1, Matplotlib 3.10.8 and CVXPY 1.8.1. mlsynth requires Python 3.10 or later — the README’s claim of 3.9 is out of date, and pyproject.toml is the authority. The core dependencies are pandas, NumPy, Matplotlib, SciPy, scikit-learn, statsmodels, CVXPY, ECOS, Pydantic and PyArrow; both optional solver backends are lazily imported, so a base install can always import mlsynth and construct any estimator class.

3.2 The library at a glance

Ninety-two exported names is a lot, and the temptation is to reach for whichever class name looks closest to what you want. Resist it: several of the names are near-homonyms of each other. The library ships a machine-readable index of the whole catalogue, which is the fastest way to find out what a class actually does.

from mlsynth._guides_api import get_llm_guide
guide = get_llm_guide() # "concise" | "full" | "practitioner"
print(guide[:420])

# mlsynth
> mlsynth is a strongly-typed Python library of synthetic-control and
> difference-in-differences estimators for causal inference with panel data.
> Every estimator exposes a Pydantic config and a `.fit()` that returns a
> standardized results object. Most are validated against the source paper's
> empirical result, Monte Carlo, or an authoritative reference implementation
> (see the Replications page).

That guide is worth reading in full before you pick an estimator, and section 20 tabulates the part of it relevant to this ladder. For now the important sentence is the second one: every estimator exposes a Pydantic config and a .fit() that returns a standardized result. That uniformity is the whole reason this post can cover six estimators without six separate interfaces to learn.

4. The mlsynth data contract

4.1 Five fields, one long panel

Every estimator in the library wants the same five things: a long-format DataFrame with one row per unit-period, and the names of the outcome, treatment, unit and time columns. The treat column is a 0/1 indicator that is 1 for the treated unit in post-treatment periods and 0 everywhere else — including for the treated unit before treatment.

That convention is worth dwelling on, because it is where most first attempts go wrong. treat is not “this unit is ever treated”; it is “this unit is under treatment right now”. mlsynth reads both the donor pool and the treatment date off that single column.

panel = pd.read_csv("brexit_analysis.csv")
print(f"{panel.shape[0]} rows x {panel.shape[1]} columns")
print(f"{panel.country.nunique()} countries, quarters t = {panel.t.min()}..{panel.t.max()}")
print(panel[["country", "quarter_label", "t", "log_rgdp", "treated"]].head(3).to_string(index=False))

2496 rows x 16 columns
24 countries, quarters t = 1..104
country quarter_label t log_rgdp treated
Australia 1995Q1 1 -0.351514 0
Australia 1995Q2 2 -0.345316 0
Australia 1995Q3 3 -0.339262 0

The panel is quarterly log real GDP for 24 OECD economies over 1995Q1–2020Q4, assembled by Born, Müller, Schularick and Sedláček [2] and redistributed in the replication package of de Brabander, Juodis and Miyazato Szini [1]. There are no missing values, 23 donors, and 86 pre-treatment quarters if we date the treatment at 2016Q3.

4.2 Pydantic configs, and what happens when you get it wrong

The configs are Pydantic v2 models with extra = "forbid". In practice this means the library refuses to accept a field it does not recognise, which is a much better failure mode than silently ignoring it.

base = dict(df=window(T_2018Q4), outcome="log_rgdp", treat="treat",
unitid="country", time="tt", display_graphs=False)
for bad, why in [
(dict(base, backendd="outcome-only"), "misspelled keyword"),
(dict(base, outcome="gdp_log"), "column not in the DataFrame"),
]:
try:
VanillaSC(bad).fit()
except (MlsynthConfigError, MlsynthDataError) as exc:
print(f"{why:<28s} -> {type(exc).__name__}: {str(exc).splitlines()[0][:70]}")

misspelled keyword -> MlsynthConfigError: 1 validation error for VanillaSCConfig
column not in the DataFrame -> MlsynthDataError: Missing required columns in DataFrame 'df': gdp_log

There are four exception types — MlsynthConfigError, MlsynthDataError, MlsynthEstimationError and MlsynthPlottingError — and they tell you which phase failed. A MlsynthConfigError means you wrote the config wrong; a MlsynthDataError means the DataFrame does not satisfy the contract (empty, missing columns, duplicate unit-period pairs); a MlsynthEstimationError means the optimiser gave up. Catching the first two separately from the third is a good habit in any loop over specifications.

4.3 What the data look like before we assume anything

Before fitting anything, look at the twenty-four series.

fig, ax = plt.subplots(figsize=(9.5, 6))
for c in DONORS:
ax.plot(QDEC, Y[c], color=GREY_DONOR, lw=0.8, alpha=0.75)
ax.plot(QDEC, Y[TREATED], color=ORANGE, lw=2.4, label="United Kingdom", zorder=5)
ax.axvline(QDEC[T0], color=LIGHT_TEXT, ls="--", lw=1.0)
ax.set_xlabel("year"); ax.set_ylabel("log real GDP")
plt.savefig("python_sc_dsc_sdid_01_gdp_paths.png", dpi=300, bbox_inches="tight")
plt.show()

The UK is one line among twenty-four, and nothing in the picture tells you what would have happened without the referendum. Notice also that the series are strongly trending and highly correlated: this is close to a set of random walks with drift, which will matter enormously in section 11 when we look at what the time weights do.

4.4 The one trick that makes everything else simple

Every mlsynth estimator reports an ATT averaged over all post-treatment periods. But the question here — and the question in the published tables — is the shortfall at two specific quarters, 2018Q4 and 2019Q4.

There is a neat way to get exactly that without any post-estimation arithmetic. Keep the 86 pre-treatment quarters plus the single quarter of interest, renumber time so it runs 1 to 87, and “the average over all post periods” becomes an average over one period. A bare .fit() then returns precisely the number you want.

T0 = 86 # pre-treatment quarters, through 2016Q2
EVAL = {"2018Q4": 96, "2019Q4": 100} # evaluation quarters, as values of `t`
TREATED = "United Kingdom"
def window(post, pre=T0):
"""`pre` pre-treatment quarters + the given post quarter(s), renumbered 1..pre+1."""
post = [post] if np.isscalar(post) else list(post)
sub = panel[(panel.t <= pre) | (panel.t.isin(post))].copy()
sub["tt"] = sub.groupby("country")["t"].rank(method="dense").astype(int)
sub["treat"] = ((sub.country == TREATED) & (sub.tt > pre)).astype(int)
return sub
def cfg(post, pre=T0, **extra):
"""The five fields every mlsynth estimator wants, plus estimator-specific ones."""
return dict(df=window(post, pre), outcome="log_rgdp", treat="treat",
unitid="country", time="tt", display_graphs=False, **extra)
def pct(att):
"""mlsynth reports treated minus counterfactual. Flip into a % GDP shortfall."""
return -100.0 * att

Those three helpers are the entire scaffolding of this post. cfg is the reason every subsequent code block is one line long, and pct is the reason every number is positive: mlsynth reports the ATT as treated minus counterfactual, which is negative when the treatment hurt, and the literature reports Brexit as a positive percentage loss.

In formal terms, the estimand throughout is the average treatment effect on the treated at a single period,

$$\tau_t = Y_{\text{UK},t}(1) - Y_{\text{UK},t}(0),$$

where $Y_{\text{UK},t}(0)$ is the unobserved no-Brexit path. Every stage is a different estimator of that same quantity, and we report $-100 \times \tau_t$ so that a larger number means a larger loss.

5. Anatomy of a fit

5.1 Config in, standardized result out

The call pattern never changes: build a config, construct the estimator, call .fit(), read the result.

res = VanillaSC(cfg(96, inference=False)).fit()
print(f"type(result) {type(res).__name__}")
print(f"result.att {res.att:+.6f}")
print(f"result.pre_rmse {res.pre_rmse:.6f}")
print(f"result.fit_diagnostics.r_squared_pre {res.fit_diagnostics.r_squared_pre:.6f}")
print(f"result.method_details.method_name {res.method_details.method_name}")
print(f"len(result.donor_weights) {len(res.donor_weights)}")
print(f"result.counterfactual.shape {np.shape(res.counterfactual)}")

type(result) BaseEstimatorResults
result.att -0.030388
result.pre_rmse 0.005589
result.fit_diagnostics.r_squared_pre 0.998060
result.method_details.method_name VanillaSC[outcome-only]
len(result.donor_weights) 9
result.counterfactual.shape (87,)

Two things are worth noticing. len(result.donor_weights) is 9, not 23: mlsynth returns only the donors with non-zero weight, so do not treat that dictionary as a dense vector over the donor pool. And method_details.method_name reports VanillaSC[outcome-only], which is the backend the "auto" setting resolved to — the library tells you which algorithm it actually ran, which is exactly the information you need when comparing against another implementation.

The result is a BaseEstimatorResults (aliased EffectResult), organised into six sub-models plus a set of flat convenience accessors:

Sub-model	Contains
`effects`	`att`, `att_percent`, `att_std_err`, `additional_effects`
`fit_diagnostics`	`rmse_pre`, `r_squared_pre`, `rmse_post`
`time_series`	`observed_outcome`, `counterfactual_outcome`, `estimated_gap`, `time_periods`
`weights`	`donor_weights`, `time_weights`, `unit_weights`, `summary_stats`
`inference`	`p_value`, `ci_lower`, `ci_upper`, `standard_error`, `method`, `details`
`method_details`	`method_name`, `is_recommended`, `parameters_used`

Flat accessor	Resolves to
`.att`	`effects.att`
`.att_ci`	`(inference.ci_lower, inference.ci_upper)`
`.counterfactual`	`time_series.counterfactual_outcome`
`.gap`	`time_series.estimated_gap`
`.donor_weights`	`weights.donor_weights`
`.weight_vector`	dense array of `donor_weights.values()`
`.pre_rmse`	`fit_diagnostics.rmse_pre`

Learn the seven flat accessors and you can read the output of any effect estimator in the library without opening its documentation.

5.2 Plotting: what the package gives you for free

Every effect result carries a .plot() method driven by a PlotConfig. Set display_graphs=True and you get a figure without writing any Matplotlib at all.

res = VanillaSC(cfg(96, inference=False)).fit()
fig, axes = plt.subplots(1, 2, figsize=(11, 4.2))
# display=False on every call. Without it .plot() ends in plt.show(), which in
# a notebook flushes the figure after the FIRST panel and leaves the second empty.
res.plot(kind="counterfactual", ax=axes[0], display=False)
res.plot(kind="gap", ax=axes[1], display=False)
plt.tight_layout()
plt.show()

That figure is deliberately unstyled — it is what the library draws with no help from us, and it is already publishable. kind takes "auto", "counterfactual" or "gap"; passing ax= lets you compose several results into one panel.

The display=False above is not cosmetic, and it is worth a paragraph because it is a fourth silent default. .plot() ends with if pc.display: plt.show(), and the PlotConfig it consults is self.plot_config or PlotConfig() — but the fitted result’s plot_config is None, so the display_graphs=False you set on the config never reaches it and the fallback PlotConfig() has display=True. In a script the stray plt.show() is a harmless no-op under the Agg backend. In a notebook it flushes and closes the figure after the first panel, so the second panel renders blank and a stray <Figure size 640x480 with 0 Axes> appears underneath. The per-call display=False override is the fix; the same applies any time you compose two .plot() calls into one figure.

Cosmetics are configured through the nested plot field rather than through Matplotlib, so a house style travels with the config:

from mlsynth.config_models import PlotConfig
styled = VanillaSC(cfg(96, inference=False, plot=PlotConfig(
observed_color="#d97757",
counterfactual_colors=["#6a9bcc"],
counterfactual_linestyle="--",
xlabel="quarter index (renumbered 1..87)",
ylabel="log real GDP",
title="VanillaSC via PlotConfig",
display=False,
))).fit()

The legacy flat fields treated_color and counterfactual_color still work — note that the latter takes a list, not a string — but PlotConfig is the maintained route and supports themes, save targets and per-call overrides.

Every remaining figure in this post is drawn by hand in the site’s dark palette, because a tutorial benefits from consistent colours. In your own work, display_graphs=True is usually enough.

5.3 Where estimators deviate from the standard result

Three of the six estimators return a richer object than BaseEstimatorResults, and each adds exactly what its method needs:

Estimator	Returns	The extra you need
`FDID`	`FDIDResults`	`.fdid` and `.did`, two `FDIDMethodFit` objects — the forward-selected fit and the plain two-way DiD benchmark
`TSSC`	`TSSCResults`	`.variants` (a dict of the four MSC variants), `.selection` (the Step-1 tests), `.recommended_method`
`SDID`	`SDIDResults`	`.inference_detail`, `.event_study`, `.cohorts[a]` (which is where the time weights live)

All three still populate the standard sub-models and flat accessors, so .att works everywhere. The extras are additive, not alternative.

6. A naming hazard, before we import anything

This section exists because getting it wrong costs nothing at runtime and everything in interpretation.

mlsynth.DSC is not the DSC on this ladder.

mlsynth.DSC is Distributional Synthetic Control (Gunsilius 2023, Econometrica). It reconstructs the treated unit’s whole outcome distribution as a Wasserstein-space average of donor distributions and returns quantile treatment effects. It requires micro-level data: many individual observations per unit-period. Handed a panel like ours, with one observation per country-quarter, it is answering a question the data cannot support.
The DSC on this ladder is Demeaned Synthetic Control (Doudchenko and Imbens [8]; Ferman and Pinto [9]) — synthetic control plus an intercept. In mlsynth it is TSSC(method="MSCa").

Same three letters, different estimators, and no error is raised. The mapping used throughout this post follows mlsynth issue #312, which is itself a reading of the paper we replicate.

The library has several more acronym neighbours worth knowing about before you reach for one by name:

Class	Is actually	Not to be confused with
`DSC`	Distributional SC (Gunsilius)	demeaned SC = `TSSC(method="MSCa")`
`DSCAR`	Dynamic SC for Auto-Regressive processes	either of the above
`SCD`	SC with Differencing	`DSC`
`DRSC`	Distribution-Regression SC	`DSC`
`MEDSC`	Mediation SC	demeaned SC

The general lesson is that in a ninety-two-class library, the class name is a mnemonic and not a definition. Check the docstring before you fit.

7. One regression, six sets of weights

Before the code, one unifying idea. Every stage on this ladder is the same weighted two-way regression,

$$(\hat\tau, \hat\mu, \hat\alpha, \hat\beta) = \arg\min \sum_{i,t} \omega_i \lambda_t \left( Y_{it} - \mu - \alpha_i - \beta_t - \tau D_{it} \right)^2,$$

with a different choice of the unit weights $\omega_i$ and the time weights $\lambda_t$. In words: fit a two-way fixed-effects regression, but let some units and some periods count more than others. The stages differ only in how those two weight vectors are chosen.

graph TD
R("<b>One weighted two-way regression</b><br/>min Σ ω<sub>i</sub> λ<sub>t</sub> (Y<sub>it</sub> − μ − α<sub>i</sub> − β<sub>t</sub> − τD<sub>it</sub>)²")
R --> A("ω uniform, λ uniform<br/><b>DiD</b>")
R --> B("ω fitted, λ uniform, no intercept<br/><b>SC</b>")
R --> C("ω fitted, λ uniform, intercept<br/><b>DSC</b>")
R --> D("ω fitted, λ fitted, intercept<br/><b>SDID</b>")
R --> E("<b>Change the feasible set instead</b>")
E --> F("blend with m-nearest-neighbour matching<br/><b>MASC</b>")
E --> G("allow negative weights, penalise them<br/><b>ASCM</b>")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class R,E anchor
class A,B,F,G blue
class C teal
class D orange

The diagram splits the ladder into two families. The first four stages change which weights the same regression uses. The last two change what weights are allowed at all — MASC by mixing in a different estimator, ASCM by relaxing the simplex. That distinction is why the ladder branches rather than continuing upward, and it is the reason section 16’s tournament cannot simply declare the top stage the winner.

Here is the whole ladder as mlsynth code, which is the table to bookmark:

Stage	$\omega$	$\lambda$	mlsynth call
DiD	uniform	uniform	`FDID(cfg).fit().did`
SC	fitted, simplex	uniform	`VanillaSC(cfg)`
DSC	fitted, simplex + intercept	uniform	`TSSC(cfg, method="MSCa")`
SDID	fitted, simplex + intercept	fitted, simplex	`SDID(cfg, zeta=0.0)`
MASC	convex blend of matching and SC	uniform	`MASC(cfg, set_f=...)`
ASCM	SC weights plus a ridge correction	uniform	`VanillaSC(cfg, augment="ridge")`

Now we climb it.

8. Stage 0 — Difference-in-differences

mlsynth has no standalone DiD class, and this trips people up. The plain two-way estimator comes free inside FDID (Forward DiD, Li 2024), which fits its own estimator and the textbook benchmark side by side and exposes them as .fdid and .did.

fdid_res = {k: FDID(cfg(e)).fit() for k, e in EVAL.items()}
did = {k: r.did for k, r in fdid_res.items()}
print(f"DiD 2018Q4 {pct(did['2018Q4'].att):.2f}% 2019Q4 {pct(did['2019Q4'].att):.2f}%")
print(f"SE (analytic) {100 * did['2018Q4'].att_se:.2f}")
wd = did["2018Q4"].donor_weights
print(f"donor weights: {len(wd)} donors, all equal to {next(iter(wd.values())):.6f} = 1/{len(wd)}")
print(f"pre-treatment RMSE {did['2018Q4'].pre_rmse:.5f}, R^2 {did['2018Q4'].r_squared:.4f}")

DiD 2018Q4 4.98% 2019Q4 6.18%
SE (analytic) 2.19
donor weights: 23 donors, all equal to 0.043478 = 1/23
pre-treatment RMSE 0.02170, R^2 0.9706

Difference-in-differences puts the Brexit cost at 4.98% of GDP by the end of 2018 — far above every other stage, and far above the 2.4% previously published. The uniform $1/23 = 0.043478$ weights are the giveaway that nothing was fitted: DiD assumes the average of all twenty-three OECD economies would have moved in parallel with the UK, and the pre-treatment RMSE of 0.0217 says it did not. That number is roughly four times the 0.0056 that synthetic control achieves, and it is the entire reason the rest of the ladder exists.

A free seventh stage. The same call also gives you Forward DiD, which selects a subset of donors greedily and then runs DiD on them. It is not on the paper’s ladder, but it costs nothing:

f18 = fdid_res["2018Q4"].fdid
print(f"Forward DiD selects {len(f18.selected_names)} donors: {', '.join(f18.selected_names)}")
print(f"Forward DiD 2018Q4 {pct(f18.att):.2f}% pre-RMSE {f18.pre_rmse:.5f}")

Forward DiD selects 4 donors: Norway, Hungary, Austria, United States
Forward DiD 2018Q4 2.42% pre-RMSE 0.00880

Forward DiD picks four donors, cuts the pre-treatment RMSE from 0.0217 to 0.0088, and lands at 2.42% — remarkably close to Born et al.’s published 2.4%, and the lowest estimate anywhere in this post. Three of its four donors (Norway, Hungary, the United States) are also among synthetic control’s five largest weights, which is reassuring: two quite different selection procedures are finding the same countries.

9. Stage 1 — Synthetic control

DiD’s complaint is that the donor average does not look like the UK. Synthetic control fixes that by fitting the unit weights, subject to the simplex constraint

$$\hat\omega = \arg\min_{\omega \in \mathbb{W}} \sum_{t=1}^{T_0} \left( Y_{\text{UK},t} - \sum_j \omega_j Y_{j,t} \right)^2, \qquad \mathbb{W} = \left\{ \omega : \omega_j \ge 0, \sum_j \omega_j = 1 \right\}.$$

In words, pick the non-negative weights summing to one that make the blend track the UK as closely as possible over the 86 pre-treatment quarters. In code, Y is the log_rgdp column, the sum over $j$ runs over DONORS, and $T_0$ is T0 = 86.

sc = {k: VanillaSC(cfg(e, inference=False)).fit() for k, e in EVAL.items()}
s18 = sc["2018Q4"]
print(f"SC 2018Q4 {pct(s18.att):.2f}% 2019Q4 {pct(sc['2019Q4'].att):.2f}%")
print(f"backend chosen by 'auto': {s18.method_details.method_name}")
print(f"pre-RMSE {s18.pre_rmse:.6f} R^2 {s18.fit_diagnostics.r_squared_pre:.5f}")
ss = s18.weights.summary_stats
print(f"weights: {ss['n_nonzero']} nonzero, sum {ss['sum_of_weights']:.6f}, {ss['constraint']}")
for c, w in sorted(s18.donor_weights.items(), key=lambda kv: -kv[1]):
if w > 0.01:
print(f" {c:<16s} {w:.4f}")

SC 2018Q4 3.04% 2019Q4 4.17%
backend chosen by 'auto': VanillaSC[outcome-only]
pre-RMSE 0.005589 R^2 0.99806
weights: 9 nonzero, sum 1.000000, simplex (non-negative, sum to 1)
Hungary 0.2231
United States 0.1926
Japan 0.1826
Canada 0.1751
Norway 0.1350
Ireland 0.0523
Italy 0.0196
Portugal 0.0124

Synthetic control puts the shortfall at 3.04% by the end of 2018 and 4.17% a year later, with a pre-treatment $R^2$ of 0.998. The synthetic UK is roughly a fifth Hungary, a fifth the United States, a fifth Japan, a sixth Canada and an eighth Norway. Fourteen of the twenty-three donors get nothing at all — this sparsity is a feature of the simplex constraint, not an accident, and it is why synthetic control estimates are usually easy to describe in a sentence.

Note that the United States carries about a fifth of the counterfactual. That will matter in section 19, when we ask whether the answer survives dropping it.

9.1 The five backends

VanillaSC has a backend field with five settings, and understanding when each applies saves a lot of confusion:

`backend`	What it does	When it applies
`"auto"` (default)	`"outcome-only"` without covariates, `"mscmt"` with them	always a safe default
`"outcome-only"`	convex simplex fit on pre-treatment outcomes	no covariates; unique solution
`"mscmt"`	global differential-evolution search over predictor weights $V$	covariates (Becker-Klössner)
`"malo"`	corner search over $V$ (Malo et al. 2024)	covariates
`"penalized"`	Abadie-L’Hour unique/sparse estimator	covariates, when uniqueness matters

The distinction that matters: with no covariates the problem is convex and has a unique solution; with covariates it becomes a bilevel program whose predictor weights are generically not identified. That is why VanillaSC reports a v_agreement diagnostic alongside covariate fits — small means the predictor weights are well identified, large means they are fragile. We come back to this in section 17.

9.2 One estimator, four solvers

Here is where the package-level reading earns its keep. Plain synthetic control is a single mathematical object, but mlsynth can reach it four different ways, and R’s synthdid reaches it a fifth.

(tssc_att below is a three-line helper that reads TSSC’s estimate off its gap series rather than its rounded .att field. Section 10.1 explains why it is needed; for now take it as “the unrounded ATT”.)

for label, klass, kw in [
("VanillaSC, backend='auto'", VanillaSC, dict(inference=False)),
("VanillaSC, backend='outcome-only'", VanillaSC, dict(backend="outcome-only", inference=False)),
("VanillaSC, w_constr='simplex'", VanillaSC, dict(w_constr="simplex", inference=False)),
("TSSC, method='SC'", TSSC, dict(method="SC", inference=False)),
]:
r = klass(cfg(96, **kw)).fit()
att = tssc_att(r, "SC") if klass is TSSC else r.effects.att
print(f"{label:<36s} 2018Q4 {pct(att):.3f}")
# Reference value from the R edition, not refitted here (see section 15).
print(f"{'R synthdid (Frank-Wolfe, published)':<36s} 2018Q4 3.060")

VanillaSC, backend='auto' 2018Q4 3.039
VanillaSC, backend='outcome-only' 2018Q4 3.039
VanillaSC, w_constr='simplex' 2018Q4 3.039
TSSC, method='SC' 2018Q4 3.039
R synthdid (Frank-Wolfe, published) 2018Q4 3.060

Four independent code paths inside mlsynth — two backends, an explicit constraint family, and a completely different estimator class — agree to three decimals at 3.039%. R’s synthdid stops at 3.06%. Section 15 explains why, and argues that this is the most interesting number in the post.

The w_constr field, incidentally, is a general escape hatch. It accepts "simplex", "ols", "lasso", "ridge" and "L1-L2", which lets you ask what the estimate would be under a different feasible set without changing estimator classes. Section 19 uses it.

9.3 The counterfactual path

The windowed fits above answer “how large is the shortfall at 2018Q4”. To draw the whole counterfactual path, fit once on the untruncated panel.

full = cfg(list(range(T0 + 1, 105))) # all 18 post-treatment quarters
sc_full = VanillaSC(dict(full, inference=False)).fit()
cf_sc = np.asarray(sc_full.counterfactual, float).ravel()
gap_sc = np.asarray(sc_full.gap, float).ravel()
print(f"pre-RMSE {sc_full.pre_rmse:.6f}, mean post gap {gap_sc[T0:].mean():+.5f} log points")

pre-RMSE 0.005589, mean post gap -0.02882 log points

The average post-treatment gap is $-0.0288$ log points, or about 2.9% of GDP averaged across all eighteen quarters after the referendum. The bottom panel is the one to look at: the gap oscillates around zero for eighty-six quarters and then goes negative and stays there. That persistence, rather than the size of any single quarter’s gap, is what makes the result credible.

10. Stage 2 — Demeaned synthetic control

Synthetic control insists the blend match the UK’s level. Sometimes a blend tracks the shape perfectly while sitting slightly above or below, and plain SC will reject it in favour of a worse-shaped blend at the right level. Demeaned SC adds one free parameter — a constant offset — so the blend has to match the shape but not the level.

In mlsynth this is the MSCa variant of the two-step estimator of Li and Shankar (2023).

dsc = {k: TSSC(cfg(e, method="MSCa", inference=False)).fit() for k, e in EVAL.items()}
d18 = dsc["2018Q4"].variants["MSCa"]
print(f"DSC 2018Q4 {pct(tssc_att(dsc['2018Q4'])):.2f}% 2019Q4 {pct(tssc_att(dsc['2019Q4'])):.2f}%")
print(f"MSCa intercept: {d18.intercept:+.6f} log points = {100 * d18.intercept:+.3f}% of GDP")

DSC 2018Q4 2.99% 2019Q4 4.12%
MSCa intercept: +0.002410 log points = +0.241% of GDP

The estimated offset is $+0.0024$ log points, about a quarter of one per cent of GDP. That is small, and its smallness is informative: it says the SC fit was already close to level-balanced, which is why DSC’s 2.99% sits so near SC’s 3.04%.

10.1 A precision trap worth knowing about

TSSC rounds every scalar it reports. The series it returns are full precision, but .att, .rmse_pre and the donor weights are rounded before they reach you.

print(f"variants['MSCa'].att {d18.att!r} -> {pct(d18.att):.4f}%")
print(f"gap[-1] (full precision) {np.asarray(d18.gap, float)[-1]!r}")
print(f" -> {pct(tssc_att(dsc['2018Q4'])):.4f}%")
print(f"variants['MSCa'].rmse_pre {d18.rmse_pre!r}")

variants['MSCa'].att -0.03 -> 3.0000%
gap[-1] (full precision) np.float64(-0.0298873295228968)
-> 2.9887%
variants['MSCa'].rmse_pre 0.006

Take the headline off .att and DSC reads exactly 3.00%. Take it off the gap series and it reads 2.99%, which is what mlsynth’s own replication table reports and what the published paper reports. A hundredth of a percentage point is not going to change anyone’s policy view, but it will make you think you have failed to replicate a table when you have not. The fix is three lines:

def tssc_att(res, variant="MSCa", n_post=1):
"""Full-precision ATT for a TSSC variant, read off the unrounded gap series."""
gap = np.asarray(res.variants[variant].gap, float).ravel()
return float(gap[-n_post:].mean())

The general lesson generalises past TSSC: when a package hands you both a scalar summary and the series it was computed from, and the two disagree, trust the series.

10.2 The four variants and the Step-1 selection

Left to itself, TSSC fits all four variants of the Li-Shankar estimator and runs a subsampling procedure to pick one. method= forces a single variant and skips the selection entirely.

t_all = TSSC(cfg(96, draws=500, seed=SEED)).fit() # no method= : fit all four
print(f"TSSC recommends: {t_all.recommended_method}")
for m in ("SC", "MSCa", "MSCb", "MSCc"):
v = t_all.variants[m]
ic = "none" if v.intercept is None else f"{v.intercept:+.5f}"
print(f" {m:<5s} loss {pct(tssc_att(t_all, m)):5.3f}% "
f"(.att reports {v.att:+.5f}) intercept {ic}")
for name, test in t_all.selection.tests.items():
print(f" test '{name}': stat {test.statistic:+.5f} "
f"CI [{test.ci_lower:+.5f}, {test.ci_upper:+.5f}] rejected {test.rejected}")

TSSC recommends: SC
SC loss 3.039% (.att reports -0.03000) intercept none
MSCa loss 2.989% (.att reports -0.03000) intercept +0.00241
MSCb loss 3.041% (.att reports -0.03000) intercept none
MSCc loss 3.021% (.att reports -0.03000) intercept +0.00282

 test 'joint': stat +2.21193 CI [+0.02894, +9.10199] rejected False

All four variants land between 2.99% and 3.04%, and all four report .att as exactly $-0.03$ — a compact demonstration of the rounding trap. The four differ in which constraints they impose: SC is the simplex with no intercept, MSCa adds an intercept, MSCb relaxes the sum-to-one constraint, and MSCc relaxes both. The Step-1 test does not reject the restriction, so TSSC recommends plain SC.

The cost of that convenience is real. Fitting four variants with 500 subsampling draws takes about 17 seconds; forcing one variant with inference=False takes 0.01 seconds — a factor of roughly 1,700. If you are running an estimator inside a loop, as we do in section 16, set method= and inference=False. Note that inference=False requires method to be set; asking for no inference without naming a variant raises, because there would be nothing to select with.

11. Stage 3 — Synthetic difference-in-differences

DSC still treats all eighty-six pre-treatment quarters as equally informative. SDID fits the time weights too, solving the transposed version of the same problem: which blend of quarters, judged across all donors, best predicts the treatment quarter.

sdid = {k: SDID(cfg(e, zeta=0.0, vce="placebo", B=500, seed=SEED)).fit()
for k, e in EVAL.items()}
s = sdid["2018Q4"]
print(f"SDID 2018Q4 {pct(s.effects.att):.2f}% 2019Q4 {pct(sdid['2019Q4'].effects.att):.2f}%")
inf = s.inference_detail
print(f"ATT {inf.att:+.5f}, SE {inf.se:.5f}, CI [{inf.ci[0]:+.5f}, {inf.ci[1]:+.5f}]")
print(f"p = {inf.p_value:.4f}, method '{inf.method}', n_placebo {inf.n_placebo}")

SDID 2018Q4 2.80% 2019Q4 3.94%
ATT -0.02801, SE 0.02636, CI [-0.07968, +0.02365]
p = 0.2016, method 'placebo', n_placebo 500

SDID puts the shortfall at 2.80% and 3.94%. Note the standard error: 0.0264 against a point estimate of 0.0280, giving a confidence interval that comfortably contains zero and a placebo p-value of 0.20. With one treated unit and twenty-three donors, that is the honest state of the evidence — section 18 returns to it.

11.1 The one setting that carries the result

default = SDID(cfg(96, vce="noinference")).fit()
print(f"zeta left at its default: {pct(default.effects.att):.2f}%")
print(f"zeta = 0.0: {pct(sdid['2018Q4'].effects.att):.2f}%")

zeta left at its default: 2.67%
zeta = 0.0: 2.80%

zeta is a ridge penalty on the unit weights, and it is on by default. Leave it alone and SDID reports 2.67%; set it to zero, which is what the paper solves, and it reports 2.80%. That 0.13-percentage-point gap is larger than the entire spread between SC, DSC and ASCM.

This is not an mlsynth quirk. Every implementation in every language penalises by default: R’s synthdid needs zeta.omega = 0, Stata’s sdid needs zeta_omega(0), and mlsynth needs zeta=0.0. Stata’s version is the nastiest, because its documented default of 1e-6 is a magic sentinel that requests the full penalty. The lesson survives translation: if you are replicating a published synthetic-DiD number, find out what the authors did with the penalty before you conclude anything.

One companion flag deserves a note because the documentation emphasises it:

ia = SDID(cfg(96, zeta=0.0, intercept_adjust=True, vce="noinference")).fit()
print(f"intercept_adjust=True: {pct(ia.effects.att):.4f}%")

intercept_adjust=True: 2.8012%

Identical to four decimals. intercept_adjust matters when there are several post-treatment periods to average over; under the truncate-and-renumber trick there is exactly one, so there is nothing to adjust. Worth setting anyway if you fit on an untruncated panel.

11.2 Time weights, cohorts and the event study

SDID’s time weights are the one output that is not where you would first look. They live on the cohort object, not on weights:

coh = list(sdid["2018Q4"].cohorts.values())[0]
lam = np.asarray(coh.time_weights, float)
print(f"cohorts: {list(sdid['2018Q4'].cohorts)} (n_treated={coh.n_treated}, n_post={coh.n_post})")
print(f"lambda: {len(lam)} weights summing to {lam.sum():.6f}")
for i in np.where(lam > 1e-4)[0]:
print(f" {QLAB[i]:<8s} {lam[i]:.4f}")

cohorts: [87] (n_treated=1, n_post=1)
lambda: 86 weights summing to 1.000000
2008Q4 0.0386
2014Q3 0.0029
2016Q2 0.9585

The time weights put 95.85% of their mass on 2016Q2, the last pre-treatment quarter, and essentially nothing on the other eighty-five. Difference-in-differences would place the dashed uniform weight, $1/86 \approx 0.0116$, on all of them.

Why? Because log real GDP behaves close to a random walk. If the outcome is a random walk, the best predictor of next quarter is this quarter, and the eighty-five quarters before it add noise rather than information. The source paper reports exactly this collapse and attributes it to the same cause. Do not read it as a bug; read it as SDID correctly discovering that most of the pre-treatment history is not informative about the level at the treatment date.

mlsynth also aggregates an event-study estimator alongside the headline ATT, which the R packages on this ladder do not:

full_sdid = SDID(dict(full, zeta=0.0, vce="placebo", B=200, seed=SEED)).fit()
es = full_sdid.event_study
et, tau = np.asarray(es.event_times, float), np.asarray(es.tau, float)
print(f"event times {et.min():.0f}..{et.max():.0f}")
print(f"post-treatment tau mean {tau[et > 0].mean():+.5f}")
print(f"pre-treatment tau mean {tau[(et < 0) & (et >= -20)].mean():+.5f}")

event times -86..17
post-treatment tau mean -0.02809
pre-treatment tau mean +0.00208

The pre-treatment effects average $+0.0021$ — essentially zero — while the post-treatment effects average $-0.0281$. That flat pre-treatment path is a falsification test the single ATT number cannot give you, and it is available from result.event_study for the cost of one extra line.

11.3 Three flavours of SDID

The time weights have to be fitted against something in the post-treatment period, and there are three natural choices. The published paper reports all three, and only the last falls out of a bare .fit().

Variant	$\lambda$ is fitted to predict	Evaluated at
(i)	the first treated quarter, 2016Q3	any horizon
(ii)	the average of 2016Q3 through the evaluation date	that date
(iii)	the evaluation quarter alone	that date

Variants (i) and (ii) need the weights the result already hands you, applied at a different quarter. This is a good demonstration of reading $\omega$ and $\lambda$ back out of a fitted result and using them yourself:

$$\hat\tau_t = \left( Y_{\text{UK},t} - \sum_j \hat\omega_j Y_{j,t} \right) - \sum_{s \le T_0} \hat\lambda_s \left( Y_{\text{UK},s} - \sum_j \hat\omega_j Y_{j,s} \right).$$

In words: take the gap at the evaluation quarter, then subtract the $\lambda$-weighted average gap over the pre-treatment period. The second term is the bias adjustment, and it is the only thing that differs between the three flavours.

def sdid_weights(post, pre=T0):
"""Fit SDID on a given post window; return (omega dict, lambda array)."""
res = SDID(cfg(post, pre, zeta=0.0, vce="noinference")).fit()
return res.donor_weights, np.asarray(list(res.cohorts.values())[0].time_weights, float)
def sdid_loss(w, lam, t, pre=T0):
"""Apply an (omega, lambda) pair at an arbitrary quarter."""
wv = np.array([w[c] for c in DONORS])
gap = lambda s: float(Y.loc[s, TREATED] - Y.loc[s, DONORS].to_numpy() @ wv)
bias = float(sum(lam[s - 1] * gap(s) for s in range(1, pre + 1)))
return -100.0 * (gap(t) - bias)
w_i, lam_i = sdid_weights(T0 + 1) # (i)
w_ii, lam_ii = sdid_weights(range(T0 + 1, 97)) # (ii)
w_iii, lam_iii = sdid_weights(96) # (iii)
print(f"SDID (i) 2018Q4 {sdid_loss(w_i, lam_i, 96):.3f}")
print(f"SDID (ii) 2018Q4 {sdid_loss(w_ii, lam_ii, 96):.3f}")
print(f"SDID (iii) 2018Q4 {sdid_loss(w_iii, lam_iii, 96):.3f}")

SDID (i) 2018Q4 2.771
SDID (ii) 2018Q4 2.801
SDID (iii) 2018Q4 2.801

The three variants land within 0.03 percentage points of each other, because all three put essentially all their time weight on the same last pre-treatment quarter. Whatever else is uncertain here, the choice among SDID flavours is not where the uncertainty lives — a conclusion the published paper’s placebo table appears to contradict, and section 16 shows why that appearance is an artefact.

11.4 Choosing an inference method

SDID’s vce field takes four values, and the right choice depends on what you are doing:

`vce`	Method	Use when
`"placebo"` (default)	refit treating each donor as pseudo-treated	one treated unit — the only valid choice here
`"jackknife"`	leave-one-treated-unit-out	several treated units
`"bootstrap"`	resample units with replacement	many treated units
`"noinference"`	skip it	inside a loop, where you only want the point estimate

With a single treated unit, jackknife and bootstrap have nothing to resample, so placebo is the only defensible option. "noinference" is what section 16’s tournament uses, and it is the difference between a two-minute loop and a two-hour one.

12. Stage 4 — MASC

Sections 9 to 11 improve the counterfactual by choosing better weights within the simplex. MASC (Kellogg, Mogstad, Pouliot and Torgovitsky [11]) does something different: it forms a convex combination of synthetic control and $m$-nearest-neighbour matching,

$$\hat{Y}^{\text{MASC}} = \phi \cdot \hat{Y}^{\text{match}}_m + (1 - \phi) \cdot \hat{Y}^{\text{SC}},$$

and chooses $m$ and $\phi$ jointly by rolling-origin cross-validation. The motivation is the bias decomposition: synthetic control attacks extrapolation bias, matching attacks interpolation bias, and the cross-validation buys whichever trade-off the data prefer.

M_GRID = list(range(1, 11))
SET_F = list(range(6, T0 + 1))
masc = {k: MASC(cfg(e, m_grid=M_GRID, set_f=SET_F)).fit() for k, e in EVAL.items()}
m18 = masc["2018Q4"]
print(f"MASC 2018Q4 {pct(m18.att):.2f}% 2019Q4 {pct(masc['2019Q4'].att):.2f}%")
print(f"weights.summary_stats: {m18.weights.summary_stats}")

MASC 2018Q4 2.73% 2019Q4 3.83%
weights.summary_stats: {'constraint': 'simplex (matching+SC blend)', 'phi_hat': 0.1576922233857826, 'm_hat': 10}

MASC gives 2.73%, the lowest estimate on the ladder. The two tuned dials are in weights.summary_stats, not in method_details.parameters_used — which is None for this estimator, so looking there first will send you away empty-handed. The cross-validation picked $m = 10$ neighbours and $\phi = 0.158$, so the estimate is roughly one-sixth matching and five-sixths synthetic control.

12.1 The argument that decides the answer

set_f and min_preperiods both control the cross-validation fold set, and they are mutually exclusive. The default is not the fold set the paper uses, and the difference is not subtle.

for label, kw in [
("set_f=range(6, 87) [the paper]", dict(m_grid=M_GRID, set_f=SET_F)),
("min_preperiods=None [default]", dict(m_grid=M_GRID)),
("min_preperiods=43 [ceil(T0/2)]", dict(m_grid=M_GRID, min_preperiods=43)),
]:
vals = [pct(MASC(cfg(e, **kw)).fit().att) for e in EVAL.values()]
print(f"{label:<34s} 2018Q4 {vals[0]:.3f} 2019Q4 {vals[1]:.3f}")

set_f=range(6, 87) [the paper] 2018Q4 2.726 2019Q4 3.828
min_preperiods=None [default] 2018Q4 3.191 2019Q4 4.325
min_preperiods=43 [ceil(T0/2)] 2018Q4 3.191 2019Q4 4.325

One argument moves the estimate by 0.47 percentage points, from 2.73% to 3.19%. That is bigger than the gap between the highest and lowest stages of the entire ladder excluding DiD. The default min_preperiods resolves to $\lceil T_0/2 \rceil = 43$, which uses only the second half of the pre-treatment period for cross-validation; passing set_f=range(6, 87) uses folds starting at quarter 6, which is what the paper does and what R’s masc reproduces to three decimals.

This is the second of the three defaults promised in the overview. Like zeta, it is not wrong — it is a defensible choice that happens not to be the one your reference used.

Tracing the cross-validation makes the trade-off visible. Refitting with a single-element m_grid forces each neighbour count in turn:

rows = []
for m in M_GRID:
r = MASC(cfg(96, m_grid=[m], set_f=SET_F)).fit()
st = r.weights.summary_stats
rows.append(dict(m=m, loss=pct(r.att), phi_hat=st["phi_hat"]))
print(pd.DataFrame(rows).to_string(index=False, float_format=lambda v: f"{v:.5f}"))

 m loss phi_hat
1 2.89028 0.03808
2 3.01677 0.00531
3 2.97685 0.06197
4 3.09494 0.08790
5 3.18100 0.09211
6 3.23477 0.08848
7 3.07342 0.04640
8 2.96191 0.11266
9 2.95586 0.12382
10 2.72551 0.15769

Two patterns. First, $\phi$ rises with $m$ — the more neighbours you average over, the more weight the cross-validation is willing to put on matching, because averaging more neighbours reduces the variance that makes matching unattractive. Second, the estimate is not monotone in $m$: it wanders between 2.73% and 3.23% with no obvious structure. That non-monotonicity is worth remembering when someone reports a single MASC number without saying what grid produced it.

13. Stage 5 — Augmented synthetic control

Every stage so far assumes the treated unit lies inside the convex hull of the donors, because the simplex cannot reach outside it. If the fit is imperfect — if there is pre-treatment imbalance that no non-negative blend can close — augmented SC (Ben-Michael, Feller and Rothstein [12]) fits a ridge regression to whatever imbalance is left and corrects for it. The correction is allowed to use negative weights.

ascm = {k: VanillaSC(cfg(e, augment="ridge", inference=False)).fit() for k, e in EVAL.items()}
a18 = ascm["2018Q4"]
aw = np.array(list(a18.donor_weights.values()))
print(f"ASCM 2018Q4 {pct(a18.att):.2f}% 2019Q4 {pct(ascm['2019Q4'].att):.2f}%")
print(f"weights: sum {aw.sum():.5f}, {int((aw < -1e-6).sum())} negative, "
f"min {aw.min():+.4f}, max {aw.max():+.4f}")
print(f"pre-RMSE {a18.pre_rmse:.6f} vs SC's {s18.pre_rmse:.6f}")

ASCM 2018Q4 3.04% 2019Q4 4.19%
weights: sum 1.00000, 8 negative, min -0.0090, max +0.2262
pre-RMSE 0.005428 vs SC's 0.005589

Augmented SC lands at 3.04%, indistinguishable from plain SC. The reason is visible in the weights: eight are negative, but the largest in magnitude is $-0.0090$, and the pre-treatment RMSE improves only from 0.005589 to 0.005428. There was very little imbalance left for the ridge correction to fix, because the UK sits comfortably inside the convex hull of twenty-three OECD economies. Augmentation earns its keep when the treated unit is extreme; here it has almost nothing to do.

The augment field is the whole interface — one keyword on the class you already used for plain SC. Two companions are worth knowing:

res_ascm = {k: VanillaSC(cfg(e, augment="ridge", residualize=True, inference=False)).fit()
for k, e in EVAL.items()}
print(f"residualize=True: 2018Q4 {pct(res_ascm['2018Q4'].att):.2f}% "
f"2019Q4 {pct(res_ascm['2019Q4'].att):.2f}%")

residualize=True: 2018Q4 3.04% 2019Q4 4.19%

residualize=True is the paper’s “ASCM res.” column — it residualises the outcome on covariates before augmenting. With no covariates supplied there is nothing to residualise on, so it returns the same numbers, which is the correct behaviour rather than a silent failure. ridge_lambda lets you fix the penalty by hand instead of letting the cross-validation choose it.

14. The whole ladder, side by side

Every section above appended one row to a running ledger; analysis.py writes it out as att_headline.csv.

ladder = pd.read_csv("att_headline.csv")
print(ladder[["method", "command", "loss_2018Q4", "loss_2019Q4",
"r_post_2018Q4", "published_2018Q4"]].to_string(index=False))

method command loss_2018Q4 loss_2019Q4 r_post_2018Q4 published_2018Q4
DiD FDID(...).fit().did 4.98 6.18 4.98 NaN
SC VanillaSC(...) 3.04 4.17 3.06 3.06
DSC TSSC(..., method="MSCa") 2.99 4.12 2.98 2.98
SDID SDID(..., zeta=0.0) 2.80 3.94 2.79 2.79
MASC MASC(..., set_f=range(6, 87)) 2.73 3.83 2.73 2.73
ASCM VanillaSC(..., augment="ridge") 3.04 4.19 3.04 3.04

Reading across the columns: r_post_2018Q4 is what the R edition of this post reports using synthdid, Synth, masc and augsynth; published_2018Q4 is de Brabander, Juodis and Miyazato Szini [1]. Three of the six stages — DiD, MASC and ASCM — agree with both to two decimals. DSC lands a hundredth above the published 2.98 only because 2.9887 sits on a rounding boundary; the R edition reports the same estimate as 2.99. The two real disagreements are SC and SDID, and both differ in the same direction and for the same reason.

Every stage puts the cost of the referendum above the 2.4% that Born, Müller, Schularick and Sedláček [2] published for this same dataset, and the excluding-DiD range is a fairly tight 2.73% to 3.04% at the end of 2018, widening to 3.83% to 4.19% a year later.

The weights tell a consistent story. Hungary, Canada, the United States, Japan and Norway carry the counterfactual under every method, and only ASCM ever goes negative — eight times, all of them tiny. Five estimators built on quite different principles are picking essentially the same five countries.

The dot plot makes the shape of the disagreement clear: DiD is off on its own, and the other seven estimates cluster within about a third of a percentage point of each other at 2018Q4. The width of that cluster, not any single point in it, is the honest answer.

14.1 Comparing counterfactuals with one call

Building that comparison by hand is instructive, but mlsynth ships utilities for it:

from mlsynth import compare_estimators, plot_counterfactual_comparison
comparison = compare_estimators(
{"SC": VanillaSC(cfg(96, inference=False)),
"SDID": SDID(cfg(96, zeta=0.0, vce="noinference")),
"MASC": MASC(cfg(96, m_grid=M_GRID, set_f=SET_F))},
show_bands=False,
)
ax = plot_counterfactual_comparison(comparison)

compare_estimators fits several estimators on one panel and lines their counterfactuals up on a common time axis; compare_counterfactuals does the same for already-fitted results; plot_counterfactual_comparison draws them with their prediction intervals. For an exploratory comparison this is one call instead of thirty lines, and it is the right starting point before you invest in custom figures.

15. The disagreement is the finding

Two cells in section 14’s table disagree with R: SC (3.04 against 3.06) and SDID (2.80 against 2.79). Neither is a bug in either library, and the explanation is the most useful thing in this post.

The synthetic-control objective on this panel has a condition number of roughly $7.5 \times 10^5$. In geometric terms that means a long, narrow, nearly flat valley of near-optimal weight vectors: many quite different $\omega$ give almost the same pre-treatment fit. Any optimiser has to decide when to stop walking down it.

R’s synthdid walks the valley with Frank-Wolfe on a capped iteration budget and stops at 3.06%.
mlsynth hands the identical problem to a convex solver which runs it to optimality and returns 3.039%.
Stata’s sdid inherits synthdid’s Frank-Wolfe and stops in the same place; tighten its convergence with max_iter(100000) min_dec(1e-9) and its SDID estimate drifts from 2.79% to 2.80%, which is where mlsynth already is.

All three legs ship with this post, so you can run the comparison yourself rather than take it on trust: cheatsheet_python.py, cheatsheet_R.R and cheatsheet_stata.do. Same data, same treatment date, same two evaluation quarters, same comparative table at the end — and each file hard-codes the others’ column, so a disagreement shows up the moment you run any one of them.

The Stata file is the one worth reading even if you never open Stata. Two things in it. Its ASCM row is a different estimator — allsynth implements the bias-corrected synthetic control of Abadie and L’Hour rather than the ridge-augmented version VanillaSC(augment="ridge") gives you — and its MASC row is empty, because MASC has no Stata implementation. Reporting those honestly rather than approximating them is the point.

Stata’s version of the penalty trap is also the nastiest of the three. Its documented default is zeta_omega(1e-6), which looks like a value but is a magic sentinel: sdid.ado reads if (EOmega==1e-6) EtaOmega = (yNtr*yTpost)^(1/4), so passing the documented default explicitly still requests the full penalty. Only a literal 0 switches it off.

So three implementations, written independently in three languages, sort themselves into exactly two camps — and the split is by solver, not by language or by author. The R edition of this post reached the same conclusion from a completely different direction, by running the Frank-Wolfe iteration ladder by hand and watching the estimate converge to 3.039 as the budget grew. Section 9.2’s figure is the same finding a third time: four independent code paths inside mlsynth, all convex, all landing on 3.039.

The practical lesson is not that one library is right. It is that a synthetic control estimate carries its solver’s fingerprint, and a second-decimal disagreement between implementations is the normal state of affairs rather than a cause for alarm. When you replicate a published synthetic-control number and land 0.02 away, the first hypothesis should be the optimiser, not the data.

16. Which stage should you choose?

The estimates cluster, but they do not coincide, and the ladder gives no reason to prefer the top stage. The source paper’s answer is an in-sample placebo tournament: advance the treatment date to a quarter when nothing happened, build the counterfactual on data up to that point only, and compare with what actually occurred. The true effect is zero, so every estimate is pure error.

flowchart LR
A("Pick a fake<br/>treatment date k<br/>(2010Q1 … 2014Q4)") --> B("Fit every stage<br/>on quarters 1..k")
B --> C("Predict quarter<br/>k + h")
C --> D("Compare with<br/>what happened.<br/>True effect = 0")
D --> E("Score:<br/>RMSE, MAB")
E --> A
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A blue
class B,C anchor
class D orange
class E teal

The loop runs twenty times, once for each last-pre-treatment quarter from 2010Q1 to 2014Q4, and each pass refits all seven estimators from scratch. Everything the earlier sections taught about defaults now pays off: vce="noinference", method="MSCa", inference=False and an explicit m_grid are what keep this to thirteen seconds rather than several hours.

def placebo_one(k, h):
"""Every stage refit as if the treatment had happened at quarter k+1."""
e = k + h
row = {"k": k, "last_pre": QLAB[k - 1], "horizon": h}
row["SC"] = VanillaSC(cfg(e, pre=k, inference=False)).fit().effects.att
row["DSC"] = tssc_att(TSSC(cfg(e, pre=k, method="MSCa", inference=False)).fit())
w1, l1 = sdid_weights(k + 1, pre=k) # variant (i)
w2, l2 = sdid_weights(range(k + 1, k + 5), pre=k) # variant (ii)
w3, l3 = sdid_weights(k + 4, pre=k) # variant (iii)
row["SDID (i)"] = -sdid_loss(w1, l1, e, pre=k) / 100.0
row["SDID (ii)"] = -sdid_loss(w2, l2, e, pre=k) / 100.0
row["SDID (iii)"] = -sdid_loss(w3, l3, e, pre=k) / 100.0
row["MASC"] = MASC(cfg(e, pre=k, m_grid=M_GRID, set_f=list(range(6, k + 1)))).fit().effects.att
row["ASCM"] = VanillaSC(cfg(e, pre=k, augment="ridge", inference=False)).fit().effects.att
return row
placebo = pd.DataFrame([placebo_one(k, h) for h in (1, 4) for k in range(61, 81)])

 Horizon h = 1 quarter
method RMSE MAB MedAB
SC 0.0086 0.0068 0.0051
DSC 0.0086 0.0068 0.0051
SDID (i) 0.0066 0.0037 0.0016
SDID (ii) 0.0066 0.0038 0.0017
SDID (iii) 0.0066 0.0039 0.0020
MASC 0.0080 0.0062 0.0045
ASCM 0.0086 0.0068 0.0051

At a one-quarter horizon the whole SDID family scores 0.0066 root mean squared error, against 0.0080 for MASC and 0.0086 for SC, DSC and ASCM alike. The gap is even larger in mean absolute bias: 0.0037 for SDID (i) against 0.0068 for SC, close to a factor of two. The time weights are doing real work, and this is the paper’s central theoretical claim surviving an empirical test.

16.1 The published table is not comparing like with like

Now look at how the published version of that table is produced. In the replication code, SC, DSC, SDID (i), MASC and ASCM are all graded one quarter ahead; SDID (ii) and (iii) are graded four quarters ahead. Forecasting a year out is a strictly harder task, so part of the reported gap is the exam, not the student.

Running every estimator at both horizons settles it:

Method	RMSE, $h = 1$	RMSE, $h = 4$	Published
SC	0.0086	0.0145	0.0089
DSC	0.0086	0.0146	0.0087
SDID (i)	0.0066	0.0132	0.0067
SDID (ii)	0.0066	0.0133	0.0134
SDID (iii)	0.0066	0.0133	0.0134
MASC	0.0080	0.0140	0.0080
ASCM	0.0086	0.0146	0.0086

Every cell reproduces the published value to within 0.0003 — except the two that were graded on a different exam. Graded on the same task, the three SDID variants are indistinguishable: 0.0066 at one quarter, 0.0132–0.0133 at four. The published conclusion that variants (ii) and (iii) “perform the worst” is an artefact of the horizon, not a property of the estimators. This is the same finding the R edition reports, arrived at with a different library, which is about as much corroboration as a result of this kind can get.

What survives is the finding that matters more: at either horizon the whole SDID family beats every other stage, and the ordering below it is stable — SDID, then MASC, then SC, ASCM and DSC, and those last three are indistinguishable at one quarter and separated by less than 0.0001 at four.

17. Do covariates help? Three meanings of “control for”

So far everything has matched on outcomes alone. The obvious next question is whether adding the covariates — consumption, investment, export and import shares, labour productivity growth and the employment-population ratio — improves the counterfactual.

The answer mlsynth gives is more interesting than yes or no: it asks which of three different things you mean. SDIDConfig.covariates is not a list. It is a dictionary keyed by method, and passing a bare list raises.

Key	Method	What it does
`"adjust"`	Kranz (2022) two-step	residualise the outcome on covariates first, then run SDID on the residuals
`"match"`	de Brabander et al. [1], eqs. 11–12	put the covariates inside the unit-weight problem
`"optimized"`	Arkhangelsky et al. [10], fn. 4	estimate weights and covariate coefficients jointly

These are three different estimators, and they are all defensible readings of “SDID with covariates” in the literature.

COVARIATES = ["cons_share", "inv_share", "exp_share", "imp_share",
"labprod_growth", "emp_pop"]
for meth in ("adjust", "match", "optimized"):
kw = dict(zeta=0.0, vce="noinference", covariates={meth: COVARIATES})
if meth == "match":
kw["match_pre_periods"] = "last"
vals = [pct(SDID(cfg(e, **kw)).fit().effects.att) for e in EVAL.values()]
print(f"covariates={{'{meth}': ...}} 2018Q4 {vals[0]:5.2f} 2019Q4 {vals[1]:5.2f}")

covariates={'adjust': ...} 2018Q4 3.61 2019Q4 4.85
covariates={'match': ...} 2018Q4 1.85 2019Q4 2.93
covariates={'optimized': ...} 2018Q4 3.11 2019Q4 4.56

The three routes disagree by 1.76 percentage points at 2018Q4 — from 1.85% to 3.61%, against 2.80% with no covariates at all. That spread is more than five times the spread across the entire outcomes-only ladder. Adding covariates does not refine the answer here; it replaces one well-identified number with three poorly-identified ones.

match_pre_periods is the companion setting for the "match" route, taking "all", "half", "last" or an integer. It controls which pre-treatment periods the covariate means are computed over, and mlsynth’s own benchmarks report that its agreement with R’s Synth degrades from a weight-vector correlation of 0.998 under "last" to 0.636 under "all" — precisely because the predictor weights become less identified as more periods enter.

17.1 The same problem in VanillaSC, and a clean demonstration of why

VanillaSC has its own covariate route: the Abadie-Diamond-Hainmueller bilevel program, where an outer loop searches over predictor weights $V$ and an inner loop solves for donor weights $\omega$. Section 9.1 noted that those predictor weights are generically not identified. That claim sounds abstract until you test it, and the test costs one keyword.

Fit the identical model twice, with the same seed and the same data, changing only how long the differential-evolution search is allowed to run. If $V$ were well identified, the budget would not matter.

for label, budget in [("default (maxiter=300, popsize=15)", {}),
("reduced (maxiter=120, popsize=12)",
dict(mscmt_maxiter=120, mscmt_popsize=12))]:
r = VanillaSC(cfg(96, covariates=COVARIATES, backend="mscmt",
canonical_v="min.loss.w", seed=SEED,
inference=False, **budget)).fit()
ss = r.weights.summary_stats
print(f"{label:<36s} {pct(r.effects.att):5.2f}% "
f"pre-RMSE {r.fit_diagnostics.rmse_pre:.6f} "
f"v_agreement {ss['v_agreement']:.5f}")
top = sorted(ss["predictor_weights"].items(), key=lambda kv: -abs(kv[1]))[:4]
print(" " + ", ".join(f"{k} {v:.3f}" for k, v in top))

default (maxiter=300, popsize=15) 1.32% pre-RMSE 0.009662 v_agreement 0.05530
imp_share 1.000, exp_share 0.000, labprod_growth 0.000, emp_pop 0.000
reduced (maxiter=120, popsize=12) 1.11% pre-RMSE 0.009928 v_agreement 0.08281
labprod_growth 0.419, emp_pop 0.396, cons_share 0.141, inv_share 0.045

Same estimator, same seed, same data. Shortening the search moves the estimate from 1.32% to 1.11% — and, far more strikingly, moves the predictor weights from a corner solution that puts all the weight on the import share to a spread across labour productivity, employment, and the consumption and investment shares. Those are not slightly different answers to the same question. They are different economic stories about what makes a country comparable to the UK.

Three diagnostics all point the same way. The pre-treatment RMSE rises from 0.005589 to about 0.0097 — adding six covariates makes the pre-treatment fit nearly twice as bad, because the optimiser now spends its effort matching predictor means instead of the outcome path. v_agreement, the gap between the two canonical choices of $V$, is 0.055 to 0.083 rather than near zero. And the answer moves with the optimiser budget, which is the definition of a non-identified problem.

This is why the R edition’s placebo tournament found covariates make the counterfactual worse rather than better, and why the headline specification here matches on outcomes alone. When you have eighty-six pre-treatment quarters of the outcome itself, six covariate means are not adding information — they are adding a poorly identified optimisation problem. The fact that mlsynth reports v_agreement at all is what let us see it.

18. Inference

VanillaSC exposes nine inference methods behind a single inference= field, which is unusually generous. Here is what each returns on this panel:

rows = []
for meth in ("placebo", "scpi", "lto", "conformal", "ttest", "jackknife_plus"):
try:
r = VanillaSC(cfg(96, inference=meth, alpha=0.05)).fit()
inf = r.inference
rows.append(dict(method=meth, att=r.att, p_value=inf.p_value,
ci_lower=inf.ci_lower, ci_upper=inf.ci_upper,
reported_as=inf.method))
except Exception as exc:
rows.append(dict(method=meth, att=np.nan,
reported_as=f"FAILED: {type(exc).__name__}"))
print(pd.DataFrame(rows).to_string(index=False))

 method att p_value ci_lower ci_upper reported_as
placebo -0.030388 0.041667 NaN NaN in-space placebo (RMSPE ratio)
scpi -0.030388 NaN -0.055333 -0.009374 scpi prediction intervals (Cattaneo-Feng-Titiunik 2021)
lto -0.030388 0.008333 NaN NaN leave-two-out refined placebo (Lei-Sudijono 2025)
conformal -0.030388 0.020000 -0.046603 -0.014292 conformal prediction intervals (Chernozhukov-Wuthrich-Zhu 2021)
ttest -0.030388 0.005788 -0.040042 -0.020228 debiased SC t-test (Chernozhukov-Wuthrich-Zhu 2025)
jackknife_plus NaN NaN NaN NaN FAILED: MlsynthEstimationError

Reported honestly: jackknife_plus raises MlsynthEstimationError on this configuration and is excluded. The other five all reject at the 5% level: four report p-values between 0.006 and 0.042, and scpi reports no p-value but an interval that stops short of zero.

That looks decisive, and it should be read with more caution than it invites. The five methods are not five independent tests — they all use the same point estimate and the same donor pool, and they differ in how they build a reference distribution from twenty-three donors. Note also that SDID’s own placebo inference in section 11 gave p = 0.20 with an interval containing zero. Different estimator, different variance estimator, very different verdict. Read these as orders of magnitude rather than as digits.

18.1 Placebo in space

The most interpretable of the five is worth doing explicitly: give every donor the treatment in turn and see where the UK ranks.

rows = []
for country in [TREATED] + DONORS:
sub = panel.copy()
sub["tt"] = sub.groupby("country")["t"].rank(method="dense").astype(int)
sub["treat"] = ((sub.country == country) & (sub.tt > T0)).astype(int)
r = VanillaSC(dict(df=sub, outcome="log_rgdp", treat="treat", unitid="country",
time="tt", display_graphs=False, inference=False)).fit()
gap = np.asarray(r.gap, float).ravel()
pre, post = np.sqrt(np.mean(gap[:T0] ** 2)), np.sqrt(np.mean(gap[T0:] ** 2))
rows.append(dict(country=country, rmspe_pre=pre, rmspe_post=post, ratio=post / pre))
placebo_space = (pd.DataFrame(rows)
.assign(rank=lambda d: d["ratio"].rank(ascending=False).astype(int))
.sort_values("ratio", ascending=False))
print(placebo_space.head(6).round(4).to_string(index=False))

The statistic is the ratio of post-treatment to pre-treatment root mean squared prediction error,

$$R_j = \frac{\text{RMSPE}_j^{\text{post}}}{\text{RMSPE}_j^{\text{pre}}},$$

which asks whether unit $j$’s gap grew after the treatment date relative to how well it was fitted beforehand. Dividing by the pre-treatment fit is what stops a badly fitted donor from looking treated.

 country rmspe_pre rmspe_post ratio rank
United Kingdom 0.0056 0.0327 5.8488 1
Belgium 0.0043 0.0151 3.5231 2
Finland 0.0202 0.0666 3.2905 3
New Zealand 0.0134 0.0397 2.9693 4
Hungary 0.0210 0.0593 2.8193 5
Austria 0.0042 0.0111 2.6324 6

The UK’s ratio of 5.85 is the largest of all twenty-four units, giving a permutation p-value of $1/24 = 0.042$. That is as small as this design can produce: with twenty-four units the smallest attainable p-value is 0.042, so the test is at its floor and could not have been more favourable.

18.2 What this can and cannot tell you

It can tell you that the UK’s post-2016 divergence is unusual relative to what these twenty-three donors do. It cannot tell you the effect is 3.04% rather than 2.73%, it cannot separate Brexit from anything else distinctive that happened to the UK after mid-2016, and with twenty-four units it has essentially no power to detect anything subtler. Every interval in this section is wide, and the SDID interval contains zero.

19. Robustness: the specification zoo

Four departures from the headline specification, each one line of config. The SDID column reports variant (i) for the date and donor-pool rows and variant (ii) — the post’s headline — for the zeta rows, because those are the variants each check was run on.

Departure	SC	DSC	SDID
treatment date 2016Q3 (headline)	3.04	2.99	2.77 (i)
treatment date 2016Q2	3.09	3.05	3.18 (i)
drop the United States	3.06	3.04	2.83 (i)
`zeta = 0` (the paper)	—	—	2.80 (ii)
`zeta` at its default	—	—	2.67 (ii)
`w_constr='simplex'`	3.04	—	—
`w_constr='ols'`	3.37	—	—
`w_constr='ridge'`	3.37	—	—
`w_constr='lasso'`	2.97	—	—

Four observations.

The treatment date matters most for SDID. Moving from 2016Q3 to 2016Q2 barely touches SC (3.04 to 3.09) but moves SDID from 2.77% to 3.18%. That is exactly what section 11.2 predicts: SDID puts 96% of its time weight on the last pre-treatment quarter, so changing which quarter that is changes the bias adjustment directly. The estimator that is best on the placebo tournament is also the one most sensitive to the treatment date — a trade-off worth stating out loud.

Dropping the United States barely moves anything. The US carries about a fifth of the counterfactual, and removing it entirely shifts SC from 3.04% to 3.06% and SDID from 2.77% to 2.83%. The result does not hinge on one donor.

The constraint set is worth more than the estimator choice. Relaxing the simplex to OLS or ridge moves SC from 3.04% to 3.37%, a third of a percentage point — larger than the gap between SC and MASC. w_constr is a research decision, not a tuning knob.

And zeta again. 2.80% against 2.67%, for a setting most users will never see.

20. Beyond the ladder: the mlsynth catalogue

Six of ninety-two classes have appeared in this post. The rest are worth knowing about, because the reason to invest in this library rather than four single-purpose ones is that the config you have already written works for all of them. Here is the map, organised by what makes your design non-canonical:

If your design has…	Reach for	Note
The canonical one treated unit	`VanillaSC`, `TSSC`, `FDID`	the workhorses; everything in this post
Staggered adoption, several cohorts	`SDID`, `SequentialSDID`, `SSC`, `PPSCM`, `CSCM`	`SDID` handles both cases; `dataprep` detects which
Many donors relative to periods	`CLUSTERSC`, `MLSC`, `FSCM`, `SparseSC`, `MSQRT`, `PDA`, `SCUL`, `RESCM`	regularisation or donor selection
Bayesian uncertainty	`BVSS`, `BSCM`, `BFSC`, `MVBBSC`, `MTGP`, `BPSCS`	posterior rather than placebo intervals
A treated unit outside the convex hull	`ISCM`, `NSC`	relax the hull rather than augment the fit
Spillovers onto donors	`SpSyDiD`, `SPILLSYNTH`, `SPOTSYNTH`, `RRSC`	`SPOTSYNTH` also detects which donors are contaminated
Missing cells in the panel	`MCNNM`, `SNN`, `RMSI`	matrix completion
Multiple outcomes	`SCMO`	joint rather than one-at-a-time
Micro-level distributions	`DSC`, `MicroSynth`	the distributional DSC, not this post’s
Endogenous treatment	`SIV`, `ORTHSC`, `PROXIMAL`	instruments and proximal inference
No untreated donors at all	`SHC`	synthetic historical control
A design to run before treating	`MAREX`, `SYNDES`, `PANGEO`, `SPCD`, `MUSC`	these return a `DesignResult`, not an `EffectResult`
A continuous treatment	`CTSC`	dose rather than on/off
Factor structure in the outcome	`FMA`, `CFM`, `CSCIPCA`, `TASC`, `DSCAR`	interactive fixed effects

The library sorts its outputs into exactly two families, which is worth internalising because it determines what .fit() gives you back. EffectResult is an observational report — measure an effect on data you already have. DesignResult is a research design — choose which units to treat before any intervention, and it resolves into an EffectResult once outcomes exist. Everything in this post is the first kind.

For the current list on your installed version, get_llm_guide() is authoritative; the counts quoted in the README and the documentation prose disagree with each other and with __all__.

21. Discussion

What Brexit cost. Taking the ladder as a whole, the referendum had cost the UK between 2.7% and 3.0% of GDP by the end of 2018, and between 3.8% and 4.2% by the end of 2019. That is above the 2.4% previously published for this dataset, and the reason is not exotic: the earlier figure came from a specification that matched on covariates, and section 17 shows covariates make the counterfactual worse here rather than better.

Three caveats belong with the number. It is a net gap between the UK and a blend of OECD economies, so anything else distinctive that happened to the UK after mid-2016 is inside it. The no-interference assumption is strong over four years when the United States carries a fifth of the weight. And the estimate is a point on a specification cloud rather than a parameter that has been pinned down — every interval in section 18 is wide, and SDID’s contains zero.

What the software taught. This is where a package-first reading pays off, and the findings are not econometric.

Three defaults change the answer materially, and two of them by more than the spread across the ladder. zeta moves SDID by 0.13 percentage points, set_f moves MASC by 0.47, and the choice among the three covariate methods moves SDID by 1.76. The ladder’s own spread, excluding DiD, is 0.31, so set_f and the covariate method each clear it on their own while zeta stays just inside. You can pick the wrong default and be further from the truth than if you had picked the wrong estimator.

One estimator silently rounds the number you are most likely to quote. TSSC returns .att as exactly $-0.03$ while its gap series carries the full $-0.029887$. Nothing warns you.

Names are mnemonics, not definitions. mlsynth.DSC is not this post’s DSC, and importing it raises no error.

A version number is not a version. The PyPI release numbered 1.0.0 is behind git main at the same version number, and is missing a config field this post uses. mlsynth.__version__ will tell you "1.0.0" in both cases. Pin the commit, not the release.

A solver leaves a fingerprint. Four convex code paths inside mlsynth agree at 3.039% while R’s Frank-Wolfe stops at 3.06%. Three languages, two camps, split by optimiser rather than by author.

So what should you actually do? Fit the ladder, not a stage, and publish the cloud rather than a point. mlsynth makes that cheap — six estimators, one config, thirteen seconds for a twenty-date placebo tournament — which removes the main practical excuse for reporting a single specification. And when you report, say which defaults you set. On this dataset that sentence carries more information than the choice of estimator.

22. Summary and next steps

The Brexit referendum cost the UK 2.7–3.0% of GDP by end-2018 and 3.8–4.2% by end-2019, at every stage of the ladder, against 2.4% previously published for the same data.
mlsynth puts all six stages behind one interface: Estimator({"df": ..., "outcome": ..., "treat": ..., "unitid": ..., "time": ...}).fit(), returning a result with seven flat accessors that work everywhere.
Three defaults matter, and two of them more than the estimator choice. SDID penalises unit weights unless you set zeta=0.0 (2.80% vs 2.67%); MASC cross-validates on a different fold set unless you set set_f (2.73% vs 3.19%); and SDID’s three covariate methods disagree by 1.76 percentage points. Against a ladder that spans 0.31, the last two each clear it alone.
The time weights earn their keep. SDID’s placebo RMSE is 0.0066 against 0.0086 for plain SC, a 23% reduction, and the advantage holds at both forecast horizons.
But the published ranking among SDID variants does not survive a matched horizon. Graded on the same task, the three flavours score 0.0066, 0.0066 and 0.0066.
Covariates hurt here. They raise the pre-treatment RMSE from 0.0056 to 0.0099 and produce a v_agreement of 0.083, both signs of a poorly identified predictor-weight problem.
A limitation to carry forward: with one treated unit and 23 donors, the smallest attainable permutation p-value is 0.042. The design is at its inferential floor, and no estimator choice changes that.
Next: the same panel with CLUSTERSC or PDA if you have many more donors; SequentialSDID if adoption is staggered; SPOTSYNTH if you suspect spillovers onto the donor pool. All three take the config you already wrote.

All three cheat sheets ship with this post, so the cross-language comparison in section 15 is reproducible without leaving the bundle: cheatsheet_python.py (about half a minute), cheatsheet_stata.do (twenty seconds with standard errors off, three minutes with them on) and cheatsheet_R.R (thirty seconds with SE <- FALSE, four minutes otherwise). Each prints the same ladder and the same comparative table on the same data, and each hard-codes the other languages’ column so you can check one against another directly. The R edition of this post hand-codes every estimator before calling its package and is the place to go for the derivations. On the wider site, the classic synthetic control on the Basque Country, augmented synthetic control on the Kansas tax cuts and synthetic difference-in-differences on Proposition 99 in Stata cover single stages in isolation.

23. Exercises

Move the treatment date. Re-run the headline table with pre=T0-1 (treatment materialising 2016Q2). Which stage moves most, and can you explain it from the time weights?
Break the rounding. Compute DSC’s estimate from variants["MSCa"].att and from the gap series across all twenty placebo windows in section 16. How large does the discrepancy get?
Time the inference default. VanillaSC’s inference defaults to True, which runs in-space placebo. Time a fit with inference=False against the default, and decide when the difference matters.
Read the simplex. Print weights.summary_stats for all six stages. Which report n_negative > 0, and which report a constraint other than the plain simplex?
Grade the horizon. Extend section 16’s tournament to $h = 8$. Does the SDID family’s advantage survive a two-year forecast?
Separate MASC’s two dials. Fix m=10 and vary set_f; then fix set_f and vary m_grid. Which of the two drives the 0.47-point swing?
Try a fourth covariate route. Fit SDID with covariates={"match": [...]} under each of match_pre_periods in {"all", "half", "last", 20}. How wide is the resulting range, and how does it compare with the range across methods?
Pick a different estimator entirely. Fit CLUSTERSC and PDA on this panel with the config you already have. Do they land inside the ladder’s range, and what does that tell you about the range?

24. References

Acknowledgements

From DiD to SDID: A Ladder of Synthetic Control Estimators, and What Brexit Cost the UK

Fri, 31 Jul 2026 00:00:00 +0000

Abstract

The United Kingdom voted to leave the European Union on 23 June 2016, and because there is only one United Kingdom, that decision can only be costed against a country that never existed. This tutorial builds that country seven times over, hand-coding every stage of the single-treated-unit ladder before running it with its R package: difference-in-differences, synthetic control, demeaned synthetic control, synthetic difference-in-differences in three flavours, matching-and-synthetic-control and augmented synthetic control. The data are quarterly log real GDP for 24 OECD economies from 1995Q1 to 2020Q4: the UK treated, 23 donors, 86 pre-treatment quarters. Dating the treatment at 2016Q3 and matching on outcomes alone, the estimated shortfall in UK GDP at the end of 2018 is 3.06% under synthetic control, 2.99% under demeaned SC, 2.76% under SDID, 2.73% under MASC and 3.04% under augmented SC, widening to between 3.83% and 4.20% a year later — every one larger than the 2.4% previously reported for this dataset. A placebo tournament over twenty artificial treatment dates ranks the SDID family first, at 0.0067 log points of root mean squared error against 0.0089 for plain SC, and shows covariates make the counterfactual worse rather than better. Two things the published tables hide: the headline synthetic-control number is partly an artefact of where the optimiser stopped, and the ranking among the three SDID variants dissolves once they are graded on the same forecast horizon. Porting the ladder to Stata and Python sharpens the first: three independent implementations split by solver, not by language.

1. Overview

On 23 June 2016 the United Kingdom voted to leave the European Union. Three and a half years later, at the end of 2019, UK GDP was some number of percentage points below where it would otherwise have been. The trouble is the phrase “otherwise have been.” There is one United Kingdom, it took the treatment, and the version of it that stayed in the EU does not exist anywhere in the data.

The standard move is to build that missing country out of the countries we do observe. Take the other OECD economies, give each one a weight, add them up, and require that the resulting blend tracks the real UK quarter by quarter through the two decades before the referendum. If the blend and the UK were indistinguishable for eighty-six quarters, the argument goes, the blend is a credible stand-in for the UK afterwards. Whatever gap opens up after 2016 is the estimated effect.

That is the synthetic control method, and it is the second stage of a ladder. This tutorial climbs the whole ladder. We start at the bottom, with difference-in-differences, which is the same construction with the weights frozen at one twenty-third. Then we let the data choose the weights (synthetic control). Then we allow the blend to sit at a different level from the UK, provided it moves in parallel (demeaned synthetic control). Then we let the data also choose which pre-treatment quarters to hold the blend accountable for (synthetic difference-in-differences). Then two more recent estimators that attack the problem from different directions: a cross-validated blend of matching and synthetic control (MASC), and a ridge-augmented synthetic control that is allowed to use negative weights (ASCM).

Every stage opens with the same question: what does the previous stage get wrong? That question has a precise answer, and it is the intellectual centre of this post. Following de Brabander, Juodis and Miyazato Szini [1], we will decompose the error of any weighted counterfactual into two pieces: an extrapolation bias and an interpolation bias. Synthetic control’s unit weights attack only the first. Nearest-neighbour matching attacks only the second. SDID’s time weights are what let a single estimator attack both. Once you have that picture, the ladder stops being a list of acronyms and becomes a sequence of answers to one question.

We do everything twice. Each estimator is first written from scratch, in ten or twenty lines of R, so you can see exactly which optimisation problem is being solved and what is being held fixed. Only then do we call the package — synthdid, Synth, masc, augsynth — and check that the two agree. When they do not agree, we find out why, and in one case the answer turns out to be more interesting than the agreement would have been.

The empirical stakes are real. Born, Müller, Schularick and Sedláček [2] estimated, using this same dataset, that the referendum had cost the UK about 2.4% of GDP by the end of 2018. Every stage of the ladder we build puts the number higher.

Three companion posts on this site cover pieces of this ground in isolation: the classic synthetic control on the Basque Country, the augmented synthetic control on the Kansas tax cuts, and synthetic difference-in-differences on Proposition 99 in Stata. This post is the one that puts them on a single ladder and asks which stage to choose.

1.1 Learning objectives

By the end of this tutorial you will be able to:

Derive each estimator as the same weighted two-way fixed-effects regression with a different choice of unit weights $\omega$ and time weights $\lambda$.
Implement DiD, SC, DSC, SDID, MASC and ASCM from scratch in base R, and reproduce each one with synthdid, Synth, masc and augsynth.
Decompose the bias of any weighted counterfactual into extrapolation and interpolation components, and say which weights target which.
Estimate the effect of the Brexit referendum on UK GDP under two treatment dates, with and without covariates.
Select among estimators using an in-sample placebo tournament — and recognise when such a tournament is not comparing like with like.

1.2 The road ahead

The roadmap below is the shape of the post. Read it as a sequence of complaints: each estimator exists because of something the one before it could not do. The three diamonds are the whole story; everything else is bookkeeping.

flowchart TD
D("<b>Data</b><br/>24 OECD economies<br/>1995Q1-2020Q4<br/>log real GDP") --> Q0{"How much should<br/>each donor country<br/>count?"}
Q0 -->|"all the same"| DID("<b>1. DiD</b><br/>omega = 1/J<br/>parallel trends")
Q0 -->|"let the data decide"| SC("<b>2. SC</b><br/>omega on the simplex<br/>match level AND trend")
SC --> Q1{"Must the blend sit at<br/>the same LEVEL<br/>as the UK?"}
Q1 -->|"no, absorb the gap"| DSC("<b>3. DSC</b><br/>demeaned omega<br/>+ constant adjustment")
DSC --> Q2{"Should every<br/>pre-treatment quarter<br/>count the same?"}
Q2 -->|"no, weight them too"| SDID("<b>4. SDID</b><br/>omega AND lambda<br/>three variants")
SDID --> BIAS("<b>The pivot</b><br/>extrapolation bias<br/>vs interpolation bias")
BIAS --> MASC("<b>5. MASC</b><br/>trade the two off<br/>by cross-validation")
BIAS --> ASCM("<b>6. ASCM</b><br/>de-bias imperfect fit<br/>with a ridge leash")
MASC --> R("<b>Results</b><br/>2.7 to 3.1 per cent<br/>at 2018Q4")
ASCM --> R
R --> SEL("<b>Which one?</b><br/>in-sample placebo<br/>over 20 fake dates")
SEL --> INF("<b>Inference</b><br/>beyond the paper")
classDef sty_Q0 fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class Q0 sty_Q0
classDef sty_Q1 fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class Q1 sty_Q1
classDef sty_Q2 fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class Q2 sty_Q2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class D,SC,DSC,R blue
class DID,MASC,ASCM orange
class SDID,SEL teal
class BIAS,INF anchor

Notice that the dark node in the middle is not an estimator. It is the section that explains why the last two branches exist at all, and it is the part of this material that transfers to problems that have nothing to do with Brexit.

2. Key concepts

Eight ideas carry the whole post. Two of them repay slow reading: the distinction between unit weights and time weights, and the distinction between extrapolation bias and interpolation bias. Neither of those bias names means what you would guess, which is exactly why they need a card.

The missing counterfactual and the donor pool. There is one United Kingdom and it took the treatment. The path it would have followed without Brexit is not in any dataset. Synthetic control builds that path from countries that were not treated. Those countries are the donor pool.

Example

After dropping the twelve OECD countries with incomplete records, 24 remain. The UK is the treated unit. The other 23 — from Australia to the United States — are the donor pool. None of them held a referendum on EU membership in 2016.

Analogy

The master tape of a song is lost and the original band has broken up. You hire session musicians and rehearse them against a bootleg recording until they are indistinguishable from the original. Then you have them play a song the original band never recorded. The donor pool is the pool of session musicians; the pre-treatment period is the rehearsal.

The simplex and the convex hull. Synthetic control weights must be non-negative and must sum to one. That set of allowed weight vectors is called the simplex. The blends it can produce form a region called the convex hull of the donors.

Example

With 23 donors, the simplex is the set of 23 non-negative numbers adding to one. In our fit only about eight donors get a weight above 0.01: Hungary near 0.22, the United States near 0.20, Japan near 0.18, Canada near 0.16, Norway near 0.13. The other fifteen get essentially nothing.

Analogy

Hammer a pin into a corkboard for every donor country, positioned by its economic characteristics. Stretch a rubber band around all the pins and let it snap tight. Everything inside the band is reachable by some blend; nothing outside it is. If the UK’s pin lands outside the band, no recipe of non-negative parts can reach it — you would need a negative amount of some country, like a recipe calling for minus two eggs.

Unit weights and time weights. Unit weights say how much each donor country counts. Time weights say how much each pre-treatment quarter counts. The two are chosen by the same kind of optimisation, run in two different directions.

Example

Our unit weights put about 0.22 on Hungary and 0.00 on France. Our time weights put 0.96 on 2016Q2 and roughly zero on the other 85 quarters. Both vectors are non-negative and both sum to one.

Analogy

A mixing desk has two banks of faders. The first bank sets how loud each instrument is; the second sets which seconds of the rehearsal tape you play back when you check the mix. Difference-in-differences leaves both banks flat. Synthetic control moves the first. Synthetic difference-in-differences moves both.

The bias-adjustment term, which is a unit fixed effect in disguise. Sometimes the blend moves in near-perfect parallel with the treated unit but sits at a slightly different level. The bias-adjustment term is the average pre-treatment gap, subtracted off. Adding it is exactly the same as putting a unit fixed effect in the regression.

Example

For the UK the demeaned synthetic control’s adjustment is $+0.0024$ log points, about a quarter of one per cent of GDP. That is why the DSC estimate of 2.99% at 2018Q4 sits so close to the SC estimate of 3.06%. A small adjustment is evidence that the SC fit was already level-balanced.

Analogy

Your bathroom scale reads two kilograms heavy. You do not throw it out; you subtract two. Synthetic control insists on a scale that is already exactly right and will reject a perfectly consistent one. Demeaned synthetic control just calibrates the offset.

Extrapolation bias and interpolation bias. Two different ways a weighted counterfactual can be wrong. Extrapolation bias: the blend’s characteristics do not match the treated unit’s. Interpolation bias: the characteristics match, but the outcome is a curved function of them, so averaging outcomes is not the same as the outcome at the average.

Example

Synthetic control chooses its weights to minimise pre-treatment prediction error, which is exactly minimising the first kind of error. It does nothing about the second unless log GDP happens to be a linear function of the underlying drivers. SDID’s time weights are what attack the second.

Analogy

You are roasting a 3.4 kilogram turkey and the chart lists only 3 kg and 4 kg. Extrapolation bias is misreading the scale and looking up 5 kg — right chart, wrong row. Interpolation bias is reading both rows correctly and averaging their times — right rows, but roasting time bends with weight, so the average of two times is not the time for the average bird. The two errors are independent, and fixing one does nothing for the other.

Rolling-origin cross-validation. A way to tune a parameter using only pre-treatment data. Walk forward through the pre-period; at each stopping point, fit on everything before it, forecast the next step, and score the forecast.

Example

MASC uses it to pick the number of matched neighbours $m$ and the blend weight $\phi$. With 86 pre-treatment quarters it produces eighty rolling origins, each giving a one-quarter-ahead forecast error for every candidate pair. Here it lands on $m = 10$ and $\phi = 0.158$.

Analogy

You do not grade a weather forecaster on how vividly they describe yesterday. You replay the archive, ask them to predict tomorrow from each day’s vantage point, and total up the misses.

The in-sample placebo. Pretend the treatment happened earlier, when nothing did. Estimate the counterfactual anyway and compare it to what actually occurred. The error you get is pure false alarm, and it measures the estimator’s precision.

Example

We repeat the whole exercise for twenty artificial treatment dates, with last pre-treatment quarters running from 2010Q1 to 2014Q4. SDID’s root mean squared error over those twenty is 0.0067 log points; plain synthetic control’s is 0.0089.

Analogy

A fire drill. There is no fire, so any alarm is a false one, and the detector that stays quietest is the one you install in the server room.

The ridge leash. Augmented synthetic control drops the non-negativity constraint so the weights can leave the convex hull. A quadratic penalty pulls them back toward the ordinary SC weights, and the penalty parameter sets how far they may roam.

Example

The ASCM weights in this application include Switzerland at $-0.0090$ and Slovak Republic at $-0.0085$ — impossible under the simplex — and the resulting estimate is 3.04% at 2018Q4, very close to the 3.06% from plain SC.

Analogy

An elastic leash on a dog. Slack leash: the dog goes wherever the scent leads, including off the lawn. Taut leash: it stays where you started. The stiffness of the leash is the ridge parameter.

3. Setup

Five packages do the estimation and four more do the plotting. Three of the five are not on CRAN, which is worth knowing before you start.

library(quadprog) # exact simplex least squares -- our hand-coded solver
library(synthdid) # SC, DSC and SDID all come out of one function (GitHub)
library(Synth) # only needed for the nested V-optimisation with covariates
library(masc) # matching and synthetic control (GitHub)
library(augsynth) # ridge-augmented synthetic control (GitHub)
library(ggplot2); library(tidyr); library(readr)
library(patchwork); library(scales); library(jsonlite)
# The three GitHub packages:
# remotes::install_github("synth-inference/synthdid")
# remotes::install_github("ebenmichael/augsynth")
#
# masc declares a hard dependency on Gurobi, a commercial solver it never
# actually needs on the code path we use. Drop the dependency first:
# git clone --depth 1 https://github.com/maxkllgg/masc /tmp/masc_src
# sed -i '' 's/^ gurobi,$//' /tmp/masc_src/DESCRIPTION
# R CMD INSTALL /tmp/masc_src
set.seed(20260801)

Everything in this post runs on R 4.5.2 with synthdid 0.0.9, augsynth 0.2.0, masc 0.1.1 and Synth 1.1-10. The full script is analysis.R, which also needs patchwork, scales and jsonlite for its figures and exports.

If you want the estimates without the derivations, there are three cheat sheets — one per language — each of which calls the packages directly and ends with the same comparative table: cheatsheet_R.R, cheatsheet_stata.do and cheatsheet_python.py. Section 19 compares what the three languages produce, and why two of them disagree in the second decimal.

4. The data

The dataset is the one assembled by Born and coauthors [2] from the OECD Economic Outlook: quarterly national accounts for 36 OECD economies. Twelve are dropped for incomplete records, leaving 24 countries observed over 104 quarters with no missing values at all.

url <- paste0("https://raw.githubusercontent.com/cmg777/starter-academic-v501/",
"master/content/tutorials/r_sc_dsc_sdid/brexit_analysis.csv")
panel <- if (file.exists("brexit_analysis.csv")) read.csv("brexit_analysis.csv") else read.csv(url)
COUNTRIES <- unique(panel$country[order(panel$unit_id)])
UK <- which(COUNTRIES == "United Kingdom")
DONORS <- setdiff(seq_along(COUNTRIES), UK)
# Y is [time x country]: 104 quarters by 24 countries.
Y <- matrix(panel$log_rgdp[order(panel$unit_id, panel$t)],
nrow = 104, dimnames = list(NULL, COUNTRIES))

 panel : 24 countries x 104 quarters (1995Q1 to 2020Q4)
treated : United Kingdom (unit_id 23)
donors (23) : Australia, Austria, Belgium, Canada, Finland, France, Germany,
Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg,
Netherlands, New Zealand, Norway, Portugal, Slovak Republic,
Spain, Sweden, Switzerland, United States
headline spec: treatment materialises 2016Q3, T0 = 86 pre-periods
evaluated at : 2018Q4 (t=96) and 2019Q4 (t=100)

Interpretation. The donor pool has 23 countries and the pre-treatment window has 86 quarters — a long panel by synthetic-control standards, which matters because the bias of these estimators shrinks with the number of pre-treatment periods. The outcome is the natural log of real GDP indexed so that each country’s 1995 average equals one, so all effects are in log points and a difference of 0.03 is about a 3% shortfall.

4.1 Two definitions of $T_0$, pinned down once

This is the single most common place to go wrong, so we fix it before writing any code. In the papers, $T_0$ is the treatment period. In the synthdid package, the T0 argument is the number of pre-treatment periods. Here they are 87 and 86.

Concept	In the papers	In `synthdid`	Quarter	`t`
First pre-treatment quarter	$t = 1$	—	1995Q1	1
Last pre-treatment quarter	$t = T_0 - 1$	period `T0`	2016Q2	86
Number of pre-treatment quarters	$T_0 - 1 = 86$	`T0 = 86`	—	—
Treatment quarter	$t = T_0$	period `T0 + 1`	2016Q3	87
First evaluation quarter	—	—	2018Q4	96
Second evaluation quarter	—	—	2019Q4	100

The referendum was held on 23 June 2016, right at the end of 2016Q2. Following the source paper we date the treatment by the quarter in which the effect materialises, which makes 2016Q3 the headline choice and 2016Q2 a robustness check. Section 17 shows the choice is not innocuous.

5. What the data look like before we assume anything

ggplot(donors, aes(date, y, group = country)) +
geom_line(colour = GREY_DONOR, linewidth = 0.35, alpha = 0.75) +
geom_line(data = uk, aes(date, y), colour = ORANGE, linewidth = 1.1) +
geom_vline(xintercept = 2016.50, linetype = "dashed", colour = TEAL)

Interpretation. The UK is one line in a crowd. No single donor tracks it, which is the whole reason we will be building a blend. But look at 2008–09: every line collapses at once. That synchronised movement is a common factor, and the estimators from stage three onward are built precisely to exploit it.

Now the same picture with the crowd averaged into a single line. This is the difference-in-differences counterfactual, drawn before we name it.

Interpretation. The two lines were already drifting apart from about 2013, three years before the referendum. That is parallel trends failing in plain sight. Difference-in-differences will happily produce a number anyway — we compute it in the next section — and the number will be nearly twice what every other method reports. This is not a subtle failure.

The six covariates that Born and coauthors match on tell their own story:

Interpretation. The six predictors live on scales that differ by orders of magnitude — a consumption share near 0.65, a quarterly productivity growth rate near 0.1. That is why Abadie’s method needs a predictor-importance matrix at all, and it is the first hint of why adding covariates might cost more than it buys.

6. One regression, seven sets of weights

Before the ladder, the frame. Everything below is the same regression run with different weights.

Write the observed outcome in potential-outcome form. Let $y_{j,t}$ be log real GDP in country $j$ and quarter $t$, and let $w_{j,t}$ be one when country $j$ is treated in quarter $t$:

$$y_{j,t} = w_{j,t}\, y^{1}_{j,t} + (1 - w_{j,t})\, y^{0}_{j,t}$$

In words, this says that for every country-quarter we observe exactly one of two numbers — the treated outcome if the referendum applies, the untreated outcome otherwise — and the other is permanently missing. In code, $y_{j,t}$ is Y[t, j] and $w_{j,t}$ is one only for the UK from t = 87 onward.

What we want is the gap between what the UK did and what it would have done:

$$\hat{\tau}_t = y_{1,t} - \hat{y}^{0}_{1,t}$$

In words, the effect in any post-referendum quarter is the observed UK outcome minus the estimated counterfactual. In code, $y_{1,t}$ is Y[96, UK] at 2018Q4 and $\hat{y}^{0}_{1,t}$ is Y[96, DONORS] %*% omega.

Now the frame. Every estimator in this post is a weighted two-way fixed-effects regression:

$$\left(\hat{\tau}, \hat{\alpha}, \hat{\beta}\right) = \underset{\tau, \alpha, \beta}{\arg\min} \sum_{j=1}^{J+1} \sum_{t=1}^{T_0} \left( y_{j,t} - \alpha_j - \beta_t - w_{j,t}\, \tau \right)^{2} \hat{\omega}_j\, \hat{\lambda}_t$$

In words, this says: regress the outcome on a country effect, a quarter effect and a treatment dummy, weighting each country by $\hat{\omega}_j$ and each quarter by $\hat{\lambda}_t$. In code, $\hat{\omega}_j$ is omega[j], $\hat{\lambda}_t$ is lambda[t], and $\alpha_j$ is the unit fixed effect that omega.intercept = TRUE switches on.

The entire ladder is a table of settings for that one expression:

Stage	Unit weights $\omega$	Time weights $\lambda$	Unit effect $\alpha$	Feasible set for $\omega$
DiD	fixed at $1/J$	fixed at $1/(T_0-1)$	yes	—
SC	optimised	none	no	simplex
SC(B)	optimised	none	no	simplex
DSC	optimised on demeaned data	fixed at $1/(T_0-1)$	yes	simplex
SDID	optimised on demeaned data	optimised	yes	simplex
MASC	$\phi \cdot$ matching $+\, (1-\phi) \cdot$ SC	none	no	simplex
ASCM	SC weights + ridge correction	none	no	sums to one, sign free

graph TD
OBJ("<b>One weighted two-way regression</b><br/>minimise the sum of<br/>(y - alpha - beta - w*tau)^2 * omega * lambda")
OBJ --> A("<b>omega uniform</b><br/><b>lambda uniform</b><br/>alpha included")
OBJ --> B("<b>omega optimised</b><br/><b>no lambda</b><br/>alpha SUPPRESSED")
OBJ --> C("<b>omega optimised</b><br/><b>lambda uniform</b><br/>alpha included")
OBJ --> E("<b>omega optimised</b><br/><b>lambda optimised</b><br/>alpha included")
A --> A1("DiD")
B --> B1("SC and SC(B)")
C --> C1("DSC")
E --> E1("SDID")
B1 --> F("<b>Change the feasible set<br/>instead of the weights</b>")
F --> F1("MASC<br/>cap omega at 1/m,<br/>then blend")
F --> F2("ASCM<br/>drop non-negativity,<br/>add a ridge pull")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class OBJ,F anchor
class A,F1,F2 orange
class B,C blue
class E teal
class A1,B1,C1,E1 gray

Two things are worth pausing on. First, synthetic control is the only stage that switches the unit fixed effect off, and that single omission is what forces it to match the UK’s level as well as its shape. Second, MASC and ASCM hang off a different branch: they do not re-weight the regression, they change what counts as an admissible weight vector.

6.1 One solver, used five times

Every stage reduces to the same problem — minimise a sum of squares over the simplex — with different inputs. So we write the solver once:

$$\underset{w}{\min} \, \lVert b - A w \rVert^{2} \quad \text{subject to} \quad w_k \geq 0 \, \text{for all } k, \quad \sum_k w_k = 1$$

In words, find the non-negative shares of the columns of $A$ that come closest to reproducing the target vector $b$. In code, $A$ is Z0 (the donors’ pre-treatment paths) when we want unit weights, and its transpose when we want time weights.

simplex_ls <- function(A, b, ridge = 1e-10) {
k <- ncol(A)
w <- solve.QP(Dmat = crossprod(A) + ridge * diag(k), # the quadratic term
dvec = crossprod(A, b), # the linear term
Amat = cbind(rep(1, k), diag(k)), # col 1: sum(w)=1; rest: w>=0
bvec = c(1, rep(0, k)),
meq = 1)$solution # meq=1: first constraint is =
w[w < 1e-10] <- 0
w / sum(w)
}

The 1e-10 on the diagonal is numerical hygiene so the Cholesky factorisation never fails. It is not the regularisation parameter of Arkhangelsky and coauthors, which is a modelling choice we look at in section 17.

7. Stage 1 — Difference-in-differences

DiD is the ladder’s ground floor: every donor counts the same, and a unit fixed effect absorbs whatever constant level gap remains.

$$\hat{\tau}^{did}_{t} = \left( y_{1,t} - \bar{y}_{1} \right) - \frac{1}{J} \sum_{j=2}^{J+1} \left( y_{j,t} - \bar{y}_{j} \right)$$

In words, the change for the UK from its own pre-referendum average, minus the average change across all 23 donors from theirs. In code, $\bar{y}_j$ is colMeans(Z0) and $1/J$ is rep(1/23, 23).

w_did <- rep(1 / 23, 23)
b_did <- mean(Z1 - Z0 %*% w_did) # the unit fixed effect
loss <- function(w, e, b = 0) -100 * (Y[e, UK] - drop(Y[e, DONORS] %*% w) - b)
c(loss(w_did, 96, b_did), loss(w_did, 100, b_did))

 DiD 2018Q4 4.981 2019Q4 6.182 uniform weights
pre-treatment RMSE vs the donor average : 0.02175

Interpretation. DiD says Brexit cost the UK 4.98% of GDP by the end of 2018 and 6.18% by the end of 2019 — roughly double what every other stage will report. It is not in the source paper’s tables, and it should not be trusted: its pre-treatment fit error is 0.0218 log points, four times what synthetic control will achieve. The estimator is fitting a trend divergence that began years before the referendum and calling it Brexit.

DiD’s error is that it gave Luxembourg and the United States the same vote. The next stage lets the data vote.

8. Stage 2 — Synthetic control

8.1 The optimisation problem

Instead of fixing the weights, choose them to make the blend track the treated unit as closely as possible over the pre-treatment window:

$$\hat{\boldsymbol{\omega}}^{sc} = \underset{\boldsymbol{\omega} \in \mathbb{W}}{\arg\min} \sum_{t=1}^{T_0-1} \left( y_{1,t} - \sum_{j=2}^{J+1} \omega_j\, y_{j,t} \right)^{2}$$

In words, pick the blend of donors whose path over the 86 pre-referendum quarters is as close as possible, in squared error, to the UK’s own. In code, the inner sum is Z0 %*% omega and the objective is sum((Z1 - Z0 %*% omega)^2).

The feasible set is the simplex:

$$\mathbb{W} = \left\{ \boldsymbol{\omega} \in \mathbb{R}^{J} : \omega_j \geq 0 \, \text{for all } j, \, \sum_{j=2}^{J+1} \omega_j = 1 \right\}$$

In words, the weights are shares — never negative, always adding to one. In code, that is the Amat/bvec/meq block of simplex_ls.

Once the weights are fixed, the counterfactual in any quarter is just the weighted sum of what the donors actually did:

$$\hat{y}^{0,sc}_{1,t} = \sum_{j=2}^{J+1} \hat{\omega}^{sc}_{j}\, y_{j,t}$$

8.2 What the simplex actually is

Two constraints, non-negativity and summing to one, sound innocuous. Geometrically they are not. They confine the synthetic UK to the convex hull of the donors — the region you can reach by averaging.

Interpretation. The shaded region is everything a non-negative blend can reach. The UK sits comfortably inside it, which is why synthetic control works well here. Had the orange point fallen outside the band — a very rich or very poor treated unit — no recipe of non-negative shares could have reached it, and we would need stage six.

That picture is in outcome space. The optimisation happens in weight space, which is a different object. Restrict attention to the three donors that end up carrying the most weight and you can draw the entire search:

Interpretation. Every point in the triangle is one set of weights summing to one; the corners are “put everything on one donor”. The minimum sits in the interior, meaning all three donors earn a positive share. The real problem is this same picture in 22 dimensions, and a weight of exactly zero means the optimum sat on an edge.

8.3 From scratch, then the package

Z1 <- Y[1:86, UK] # the UK's pre-treatment path
Z0 <- Y[1:86, DONORS] # the donors' pre-treatment paths
w_sc_qp <- simplex_ls(Z0, Z1) # exact, via quadprog
# The package. Note the layout: units x time, treated unit LAST, and T0 is the
# NUMBER of pre-treatment periods.
Y_sd <- t(cbind(Y[, DONORS], Y[, UK]))
sd_sc <- synthdid_estimate(Y_sd[, 1:87], N0 = 23, T0 = 86,
zeta.omega = 0, zeta.lambda = 0,
omega.intercept = FALSE, lambda.intercept = FALSE)
w_sc_pkg <- as.numeric(attr(sd_sc, "weights")$omega)

Why call synthdid rather than Synth here? Because with no covariates, the classic synthetic-control problem of the equation above is synthdid_estimate with both intercepts off and both penalties set to zero. Using one optimiser for the whole ladder means every difference we report between stages is the method, not the solver. Synth reappears in section 16, where its nested optimisation is genuinely needed.

There is a third way to solve the same problem, and we need it in a moment. synthdid does not call a quadratic programming solver at all — internally it runs Frank–Wolfe, an iterative method that walks toward the optimum one simplex vertex at a time. Porting it is twenty lines, and it is the only way to see what the package is actually doing:

# A line-for-line port of synthdid's internal optimiser (sc.weight.fw).
fw_step <- function(A, x, b, eta) {
Ax <- A %*% x
half <- t(Ax - b) %*% A + eta * x
i <- which.min(half) # the steepest simplex vertex
dx <- -x; dx[i] <- 1 - x[i] # the direction to move in
if (all(dx == 0)) return(x)
derr <- A[, i] - Ax
s <- -drop(half %*% dx) / (sum(derr^2) + eta * sum(dx^2))
x + min(1, max(0, s)) * dx # the optimal step length, clipped
}
simplex_fw <- function(A, b, intercept = FALSE, min.decrease = 1e-5,
max.iter = 10000) {
if (intercept) { A <- sweep(A, 2, colMeans(A)); b <- b - mean(b) }
run <- function(x, mi) {
vals <- rep(NA_real_, mi); it <- 0
# Stop on a small enough improvement -- OR on the iteration cap.
while (it < mi && (it < 2 || vals[it - 1] - vals[it] > min.decrease^2)) {
it <- it + 1
x <- fw_step(A, x, b, 0)
vals[it] <- sum((A %*% x - b)^2) / nrow(A)
}
x
}
x <- run(rep(1 / ncol(A), ncol(A)), 100) # short pre-round, then sparsify
x[x <= max(x) / 4] <- 0; x <- x / sum(x)
run(x, max.iter)
}
w_sc_fw <- simplex_fw(Z0, Z1, min.decrease = 1e-5 * sd(apply(t(Z0), 1, diff)))

Watch the while condition. It exits on either a small enough improvement or the iteration cap — and which of those fires turns out to matter.

 exact QP : SSR 2.686818e-03 nonzero 9 loss(2018Q4) 3.039
hand-coded FW: SSR 2.751662e-03 nonzero 13 loss(2018Q4) 3.056
synthdid : SSR 2.751662e-03 nonzero 13 loss(2018Q4) 3.056
|FW - package| max abs weight difference : 0.000e+00 <-- these agree
|QP - package| max abs weight difference : 1.564e-02 <-- these do NOT

Interpretation. Something has gone wrong — or rather, something instructive has gone right. Our hand-coded Frank–Wolfe loop matches the package exactly, to the last bit. But the exact quadratic-programming solution disagrees, giving 3.04% where the package gives 3.06%, and achieving a lower sum of squared residuals while doing it. The package is not finding the optimum.

8.4 Why the two solvers disagree

The reason is the shape of the objective. With 23 donors and 86 pre-treatment quarters and no regularisation, the problem is nearly degenerate:

 noise.level (sd of donor quarterly changes) : 0.012425
synthdid min.decrease : 1.242e-07
donor Gram: smallest eigenvalue 3.192e-04, condition number 7.487e+05

A condition number near a million means the objective has directions along which it is almost perfectly flat. Frank–Wolfe crawls along those directions and never triggers its stopping rule, so it halts on its iteration cap instead. Let it run longer and the answer keeps moving:

Frank–Wolfe iterations	Sum of squares	Loss at 2018Q4	Loss at 2019Q4
100	0.0034364	3.177	4.289
300	0.0031180	3.038	4.183
1,000	0.0029646	3.056	4.203
3,000	0.0028366	3.061	4.210
10,000 (the package default)	0.0027517	3.056	4.204
30,000	0.0027141	3.047	4.187
100,000	0.0026963	3.042	4.178
exact QP	0.0026868	3.039	4.172

Interpretation. The published synthetic-control estimate of 3.06% is the value Frank–Wolfe happens to be passing through at ten thousand iterations. The true minimiser of the stated objective gives 3.04%. The difference is 0.02 percentage points — economically nothing, and it does not change a single conclusion in this post. But it is worth knowing that it is there, because it tells you something general: when a synthetic-control objective is this flat, the donor weights are not identified to more than a couple of decimal places, even though the estimate they produce is stable. Two solvers disagreeing by 0.016 on individual weights agree to within 0.02 percentage points on the effect.

For the rest of the post we report the package answer, so the tables line up with the published ones.

8.5 The counterfactual and the gap

 country omega_fw omega_qp
Hungary 0.2186 0.2231
United States 0.1994 0.1926
Japan 0.1773 0.1826
Canada 0.1612 0.1751
Norway 0.1256 0.1350
Ireland 0.0543 0.0523
Italy 0.0353 0.0196
Portugal 0.0123 0.0124
pre-treatment RMSPE : 0.00566 (DiD: 0.02175)

Interpretation. Synthetic Britain is roughly one-fifth Hungary, one-fifth the United States, one-fifth Japan, one-sixth Canada and one-eighth Norway. That combination has no economic interpretation and is not supposed to have one — it is whatever reproduces the UK’s growth path. The pre-treatment fit error of 0.0057 log points is a quarter of what DiD managed, which is the entire argument for the method.

Interpretation. Twenty-one years of near-perfect tracking, then a gap that opens right at the referendum and keeps widening: 3.06% by 2018Q4 and 4.20% by 2019Q4. The flatness of the gap before 2016 is what licenses reading the gap after 2016 as an effect.

9. Stage 3 — Demeaned synthetic control

9.1 What SC gets wrong

Synthetic control has no intercept. Look again at what that means. Suppose some blend of donors moved in perfect parallel with the UK for twenty-one years but sat consistently 0.5% below it. SC’s objective would score that blend badly and reject it, even though it is exactly what we want for forecasting a counterfactual — a series with the right dynamics and a known, constant offset.

Ferman and Pinto [9] and Doudchenko and Imbens [8] both proposed the same fix: demean first, then add the offset back.

$$\hat{\boldsymbol{\omega}}^{dsc} = \underset{\boldsymbol{\omega} \in \mathbb{W}}{\arg\min} \sum_{t=1}^{T_0-1} \left( (y_{1,t} - \bar{y}_{1}) - \sum_{j=2}^{J+1} \omega_j\, (y_{j,t} - \bar{y}_{j}) \right)^{2}$$

In words, the same problem as before, but every country’s own pre-treatment average is stripped out first, so the fit is judged on shape rather than on level. In code, Z0_dm <- sweep(Z0, 2, colMeans(Z0)) and Z1_dm <- Z1 - mean(Z1).

The offset comes back as a constant:

$$b^{dsc} = \frac{1}{T_0-1} \sum_{t=1}^{T_0-1} \left( y_{1,t} - \sum_{j=2}^{J+1} \hat{\omega}^{dsc}_{j}\, y_{j,t} \right)$$

In words, the average distance over the 86 pre-referendum quarters between the UK and its blend. In code, b_dsc <- mean(Z1 - Z0 %*% w_dsc).

$$\hat{\tau}^{dsc}_{t} = y_{1,t} - \sum_{j=2}^{J+1} \hat{\omega}^{dsc}_{j}\, y_{j,t} - b^{dsc}$$

In words, the raw gap minus the offset we already knew about from before the referendum.

Adding $b^{dsc}$ is algebraically identical to putting the unit fixed effect $\alpha_j$ back into the master regression of section 6. And that gives us a tidy result: DiD is DSC with the weights frozen at $1/J$. The ground floor and the third stage are the same estimator with different $\omega$.

9.2 Two changed lines

Z0_dm <- sweep(Z0, 2, colMeans(Z0)) # <-- the only change, part 1
Z1_dm <- Z1 - mean(Z1) # <-- the only change, part 2
w_dsc_qp <- simplex_ls(Z0_dm, Z1_dm) # identical call to stage 2
# The package: one argument flips.
sd_dsc <- synthdid_estimate(Y_sd[, 1:87], N0 = 23, T0 = 86,
zeta.omega = 0, zeta.lambda = 0,
omega.intercept = TRUE, lambda.intercept = TRUE)
w_dsc <- as.numeric(attr(sd_dsc, "weights")$omega)
b_dsc <- mean(Z1 - Z0 %*% w_dsc)

 bias adjustment b_dsc : +0.00242 log points (0.242% of GDP)
|QP - package| max weight diff : 7.523e-03
correlation of SC and DSC weights: 0.9928
DSC 2018Q4 2.985 2019Q4 4.121

Interpretation. DSC is SC’s curve slid up by a single number, and here that number is +0.0024 log points — about a quarter of one percent. The estimate moves from 3.06% to 2.99%. This is an anticlimax, and the anticlimax is the finding: a small bias adjustment means the synthetic control was already level-balanced, so SC was not sacrificing shape to chase level. On a dataset where the treated unit sits awkwardly relative to the donor pool, this term would be doing real work. Notice also that the two weight vectors correlate at 0.993 — demeaning barely changed who gets picked, only how the result is read off.

10. Stage 4 — Synthetic difference-in-differences

10.1 What DSC gets wrong

DSC’s bias adjustment is a flat average over all 86 pre-treatment quarters. It gives 1995Q1 exactly as much say as 2016Q2 in deciding how far apart the UK and its blend sit. But 1995 resembles 2016 hardly at all, and if the gap between the UK and its blend has been drifting, a flat average is the wrong correction.

Arkhangelsky and coauthors [10] let the data choose which quarters to trust.

10.2 The time-weight problem is the unit-weight problem, transposed

This is the key structural insight of the whole post, so it gets its own sentence: the time-weight problem is the unit-weight problem run on the transpose.

$\omega$ asks: which countries, blended, reproduce the UK’s pre-treatment path?
$\lambda$ asks: which quarters, blended, reproduce the treatment quarter, judged across all the donors?

$$\hat{\boldsymbol{\lambda}}^{sdid} = \underset{\boldsymbol{\lambda} \in \mathbb{L}}{\arg\min} \sum_{j=2}^{J+1} \left( (y_{j,T_0} - \bar{y}_{T_0}) - \sum_{t=1}^{T_0-1} \lambda_t\, (y_{j,t} - \bar{y}_{t}) \right)^{2}$$

In words, find the blend of pre-referendum quarters that best predicts the treatment quarter, scored across all 23 donors, after removing the cross-country average at each date. In code, this is simplex_ls(t(Z0_centred), y_treatment_quarter_centred) — the same function, with a transposed argument.

$$\mathbb{L} = \left\{ \boldsymbol{\lambda} \in \mathbb{R}^{T_0-1} : \lambda_t \geq 0 \, \text{for all } t, \, \sum_{t=1}^{T_0-1} \lambda_t = 1 \right\}$$

A trap worth naming. The word “demeaned” means two different things in stage three and stage four. DSC removes each country’s own time-series mean: sweep(Z0, 2, colMeans(Z0)). The SDID time-weight problem removes each quarter’s cross-sectional mean: sweep(Z0, 1, rowMeans(Z0)). Same verb, orthogonal operations. If you take one thing away from this section, take that.

The bias adjustment then becomes a weighted average instead of a flat one:

$$b^{sdid} = \sum_{t=1}^{T_0-1} \hat{\lambda}^{sdid}_{t} \left( y_{1,t} - \sum_{j=2}^{J+1} \hat{\omega}^{dsc}_{j}\, y_{j,t} \right)$$

Compare that with $b^{dsc}$ two sections above. They are the same expression with $1/(T_0-1)$ replaced by $\hat{\lambda}_t$. That is the entire difference between stages three and four. SDID does not even estimate its own unit weights — it reuses DSC’s.

# A has donors as ROWS and quarters as COLUMNS -- the transpose of the omega
# problem. `intercept = TRUE` demeans each quarter across donors, rather than
# each country across quarters as stage 3 did.
#
# We use the Frank-Wolfe port here rather than simplex_ls, so that the result is
# comparable with the package to the last bit. The exact QP gives the same
# answer to six decimals and an identical treatment effect.
lambda <- simplex_fw(t(Z0), Y[87, DONORS], intercept = TRUE,
min.decrease = 1e-5 * sd(apply(t(Z0), 1, diff)))
b_dsc <- mean(Z1 - Z0 %*% w_dsc) # stage 3: flat average
b_sdid <- drop(lambda %*% (Z1 - Z0 %*% w_dsc)) # stage 4: weighted average

 |hand-coded lambda - synthdid lambda| : 0.000e+00
lambda(i): 3 nonzero weights; 0.958 on the last pre-period (2016Q2)
2016Q2 0.9585
2008Q4 0.0386
2014Q3 0.0029
DSC bias adjustment +0.00242 vs SDID(i) bias adjustment +0.00015
SDID (i) 2018Q4 2.758 2019Q4 3.894

Interpretation. Our hand-coded time weights match synthdid’s to zero — not to within a tolerance, exactly. And the answer is startling: 96% of the weight lands on a single quarter, 2016Q2, with a small 3.9% on 2008Q4, the trough of the financial crisis. Because the weighted average of pre-treatment gaps is so different from the flat one ($+0.00015$ against $+0.00242$), the estimate drops from 2.99% to 2.76%.

10.3 Where DiD and two-way fixed effects live inside SDID

$$\lambda^{did}_t = \mathbf{1}\{ t = T_0 - 1 \}, \qquad \lambda^{twfe}_t = \frac{1}{T_0 - 1}$$

In words, if all the time weight lands on the last pre-treatment quarter you have a difference-in-differences correction; if it is spread evenly you have the two-way fixed-effects correction that DSC uses. SDID is supposed to let the data pick a point between those two extremes.

Here it picks one of the extremes. That deserves an explanation, not a shrug.

Interpretation. Log GDP is very close to a random walk — the scatter of quarter $t$ against quarter $t-1$ sits almost exactly on the 45-degree line. If the best predictor of next quarter is simply this quarter, then the best weighted blend of past quarters for predicting the treatment quarter is the most recent quarter alone. The collapse is a property of the data, not a bug in the code. The source paper notes this too, and declines to switch to differenced data on the grounds that matching levels and trends is the entire point of a synthetic control. Exercise 4 asks you to try it anyway.

10.4 Three flavours of SDID

The time weights have to be fitted against something in the post-treatment period, and there are three natural choices:

Variant	$\lambda$ is fitted to predict	Evaluated at
(i)	the first treated quarter, 2016Q3	any horizon
(ii)	the average of all quarters from 2016Q3 to the evaluation date	that date
(iii)	the evaluation quarter alone	that date

 SDID (i) 2018Q4 2.758 2019Q4 3.894
SDID (ii) 2018Q4 2.787 2019Q4 3.923
SDID (iii) 2018Q4 2.787 2019Q4 3.923

Interpretation. The three variants land within 0.03 percentage points of each other, because all three put essentially all their time weight on the same last pre-treatment quarter. Section 15 asks whether the placebo evidence can tell them apart — and finds that the published answer to that question does not survive scrutiny.

11. The pivot: extrapolation bias and interpolation bias

We now have four stages and no principled reason to prefer any of them. This section supplies one, and it is the theoretical contribution of the source paper.

11.1 A warning about the names

Neither term means what you would guess.

Extrapolation bias is not about predicting outside the range of the data in time. It is about the blend having the wrong characteristics.
Interpolation bias is not about filling in missing quarters. It is about the response function being curved.

Hold those corrected definitions in mind, because the formal decomposition is short and it will go past quickly.

11.2 The decomposition

Write each country’s untreated outcome as a function of its characteristics, and write a generic weighted counterfactual:

$$\hat{y}^{0}_{1,T_0} = \sum_{j=2}^{J+1} w_j\, y^{0}_{j,T_0}\left[ \boldsymbol{x}_{j,T_0} \right]$$

In words, every estimator in this post is a choice of the weights $w_j$ in this one expression; the square brackets are a device for tracking where the response function is being evaluated.

The total bias is what we want and what we get:

$$\text{Bias}_{1,T_0} = y^{0}_{1,T_0}\left[ \boldsymbol{x}_{1,T_0} \right] - \sum_{j=2}^{J+1} w_j\, y^{0}_{j,T_0}\left[ \boldsymbol{x}_{j,T_0} \right]$$

Insert a middle term — the response function evaluated at the blend’s characteristics — and it splits in two. First piece:

$$B^{ext} = y^{0}_{1,T_0}\left[ \boldsymbol{x}_{1,T_0} \right] - y^{0}_{1,T_0}\left[ \sum_{j=2}^{J+1} w_j\, \boldsymbol{x}_{j,T_0} \right]$$

In words, the same function evaluated at two different places: the UK’s true characteristics, and the blend’s characteristics. If the blend matches the UK exactly, this term is zero.

Second piece:

$$B^{int} = y^{0}_{1,T_0}\left[ \sum_{j=2}^{J+1} w_j\, \boldsymbol{x}_{j,T_0} \right] - \sum_{j=2}^{J+1} w_j\, y^{0}_{j,T_0}\left[ \boldsymbol{x}_{j,T_0} \right]$$

In words, the outcome at the averaged characteristics minus the average of the outcomes. These coincide only if the function is a straight line, so this term is pure curvature.

$$\text{Bias}_{1,T_0} = B^{ext} + B^{int}$$

The middle term cancels, so the two pieces add up exactly. That is what makes it legitimate to ask, of any estimator, which of the two it is attacking.

11.3 A two-donor picture

With two donors and one characteristic, there are two obvious weighting rules and each kills exactly one term. Linear-interpolation weights place the blend’s characteristic exactly on the treated unit’s:

$$w^{li}_{3} = \frac{x_{1} - x_{2}}{x_{3} - x_{2}}, \qquad w^{li}_{2} = 1 - w^{li}_{3}, \qquad w^{nn}_{2} = 1, \qquad w^{nn}_{3} = 0$$

In words, one recipe puts the blend at the right place on the horizontal axis and reads off the chord; the other copies the nearest donor outright. Neither can do both.

Interpretation. On the left, the blend’s characteristic is exactly right, so there is no extrapolation bias — but the chord between the two donor outcomes sits below the curve, and that vertical distance is the interpolation bias. On the right, we copy the nearest donor, so there is no interpolation bias — but we are reading the curve at the wrong place. Two errors, two weighting rules, one killed each time. The question the ladder has been building toward is whether anything can kill both.

11.4 Where each estimator sits

graph LR
T("<b>Total bias</b><br/>true UK outcome minus<br/>weighted donor outcome") --> X("<b>Extrapolation bias</b><br/>same function,<br/>WRONG PLACE")
T --> I("<b>Interpolation bias</b><br/>right place,<br/>CURVED FUNCTION")
X --> XA("<b>omega weights</b><br/>minimise pre-treatment<br/>prediction error")
I --> IA("<b>lambda weights</b><br/>find pre-periods that<br/>resemble the treatment period")
XA --> SC2("SC: yes")
XA --> NN2("Matching: no")
IA --> SC3("SC: only if y is linear in x")
IA --> NN3("Matching: yes, by construction")
XA --> SD("<b>SDID: yes</b>")
IA --> SD
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class T anchor
class X,XA blue
class I,IA orange
class SC2,NN2,SC3,NN3 gray
class SD teal

Only one node has two arrows pointing into it. Synthetic control minimises extrapolation bias by construction, because it is the argmin of the pre-treatment fit. Matching minimises interpolation bias by construction, because it never blends. SDID’s unit weights do the first job and its time weights do the second, which is the source paper’s headline claim and the reason it recommends the method.

11.5 The honest caveat

Do not over-learn this. The decomposition assumes a common response function across countries and sets the idiosyncratic error aside entirely. SDID “targets” both biases, but it pays for the privilege by estimating 85 extra parameters, and the source paper’s own conclusion is that the gain over DSC is “marginal at best” for the kind of trend specification we have here. Section 20 returns to this.

12. Stage 5 — MASC

12.1 Buying the trade-off explicitly

If SC kills one bias and matching kills the other, why not buy some of each? Kellogg, Mogstad, Pouliot and Torgovitsky [11] do exactly that:

$$\hat{\boldsymbol{\omega}}^{masc}(m, \phi) = \phi\, \hat{\boldsymbol{\omega}}^{ma}(m) + (1 - \phi)\, \hat{\boldsymbol{\omega}}^{sc}$$

In words, a dial between pure matching and pure synthetic control, with the dial position chosen by out-of-sample forecast error rather than by taste. In code, phi * nn_weights(Z0, Z1, m) + (1 - phi) * w_sc.

The matching weights themselves are the solution to a linear program:

$$\hat{\boldsymbol{\omega}}^{ma}(m) = \underset{\boldsymbol{\omega} \in \mathbb{S}}{\arg\min} \sum_{j=2}^{J+1} \omega_j \lVert \boldsymbol{y}_j - \boldsymbol{y}_1 \rVert, \qquad \mathbb{S} = \left\{ \boldsymbol{\omega} : 0 \leq \omega_j \leq \tfrac{1}{m}, \, \sum_j \omega_j = 1 \right\}$$

In words, capping every weight at $1/m$ and minimising total distance forces the solution to spread $1/m$ across exactly the $m$ closest donors. Matching, written as an optimisation problem.

12.2 The cross-validation, written out

nn_weights <- function(Z0, Z1, m) {
d <- colSums((Z1 - Z0)^2)
sel <- d %in% sort(d)[1:m]
as.numeric(sel) / sum(sel)
}
# Rolling origin: for each stopping point k, refit BOTH estimators on quarters
# 1..k, forecast quarter k+1, and score. phi then has a closed form.
set_f <- 6:85
wt <- rep(1 / length(set_f), length(set_f))
ysc <- vapply(set_f, function(k) drop(Z0[k+1, ] %*% simplex_ls(Z0[1:k, ], Z1[1:k])), 0)
ytr <- Z1[set_f + 1]
cv <- do.call(rbind, lapply(1:10, function(m) {
ymt <- vapply(set_f, function(k) drop(Z0[k+1, ] %*% nn_weights(Z0[1:k, ], Z1[1:k], m)), 0)
phi <- min(1, max(0, drop((wt*(ytr-ysc)) %*% (ymt-ysc) / ((wt*(ymt-ysc)) %*% (ymt-ysc)))))
data.frame(m = m, phi = phi, cv = sum(wt * (ytr - phi*ymt - (1-phi)*ysc)^2))
}))

 hand-coded : m = 10, phi = 0.1577
masc package: m = 10, phi = 0.1577 |hand - package| = 0.000e+00
MASC 2018Q4 2.726 2019Q4 3.828 m = 10, phi = 0.158

Interpretation. The data buy 15.8% matching and 84.2% synthetic control, using the ten nearest donors. The estimate, 2.73%, is the lowest on the ladder. Note that $\phi$ is not a taste parameter: it is the minimiser of a forecast error computed entirely from pre-treatment data, and at $\phi = 0$ MASC collapses to plain SC, so on this criterion it can never do worse.

A trap in the package. The published replication code passes masc’s min_preperiods argument, but current package master reads that value as the fold start, producing only five folds. Five folds select $\phi = 1$ — pure matching — and an estimate of 2.36%, which does not match the published result. The authors’ numbers correspond to folds running from 6 to $T_0 - 1$, so we set set_f explicitly. Silent, plausible-looking, and wrong by a third of a percentage point.

13. Stage 6 — Augmented synthetic control

13.1 What every stage so far assumes

All six previous stages require the UK to lie inside the donors’ convex hull — inside the rubber band of section 8.2. When it does not, the pre-treatment fit is imperfect and Abadie’s own advice is to stop. Ben-Michael, Feller and Rothstein [12] instead de-bias:

$$\hat{\boldsymbol{\omega}}^{ascm} = \underset{\boldsymbol{\omega} : \sum_j \omega_j = 1}{\arg\min} \, \frac{1}{2 \lambda^{ridge}} \sum_{t=1}^{T_0-1} \left( y_{1,t} - \sum_{j=2}^{J+1} \omega_j\, y_{j,t} \right)^{2} + \frac{1}{2} \sum_{j=2}^{J+1} \left( \omega_j - \hat{\omega}^{sc}_{j} \right)^{2}$$

In words, keep improving the pre-treatment fit, but pay a quadratic price for every step away from the ordinary synthetic-control weights. Non-negativity is gone; only the sum-to-one constraint survives. In code, $\lambda^{ridge}$ is chosen by leave-one-period-out cross-validation inside augsynth.

The estimator has a closed form, so we can write it out: take the SC weights and add a ridge-predicted correction for whatever pre-treatment imbalance they left behind.

ascm_hand <- function(lambda_ridge) {
Xc <- sweep(t(Z0), 2, colMeans(t(Z0))) # donors x quarters, period-centred
x1c <- Z1 - colMeans(t(Z0)) # the treated path, same centring
# SC weights, plus a ridge regression of the residual imbalance on the donors
as.numeric(w_sc_qp + solve(tcrossprod(Xc) + lambda_ridge * diag(23),
Xc %*% (x1c - crossprod(Xc, w_sc_qp))))
}

One thing we cannot supply by hand is $\lambda^{ridge}$ itself — it is chosen by cross-validation, and inventing a value would be cheating. Fit the package, read its choice back out, and feed it in:

ad <- data.frame(unitnum = rep(1:24, each = 87), t = rep(1:87, 24),
value = as.vector(Y[1:87, c(UK, DONORS)]))
ad$treatment <- as.integer(ad$unitnum == 1 & ad$t == 87)
ascm <- augsynth(value ~ treatment, unitnum, t, ad, progfunc = "Ridge", scm = TRUE)
w_ascm_hand <- ascm_hand(ascm$lambda) # ascm$lambda is augsynth's CV choice
max(abs(w_ascm_hand - as.numeric(ascm$weights)))

 package : sum(w) = 1.0000, min(w) = -0.0090, 8 negative weights
hand : ridge lambda = 0.13858 (chosen by augsynth's own CV)
loss(2018Q4) = 3.045 |hand - package| max weight diff = 3.881e-06
negative weights (impossible under the simplex):
country omega
Switzerland -0.0090
Slovak Republic -0.0085
Belgium -0.0066
Spain -0.0056
Sweden -0.0030
Korea -0.0027
Austria -0.0013
Netherlands -0.0012
ASCM 2018Q4 3.045 2019Q4 4.187

Interpretation. Eight donors get negative weight — the synthetic UK now subtracts a little Switzerland and a little Belgium, which is flatly impossible under the simplex. And the estimate, 3.04%, is almost exactly plain SC’s 3.06%. That is the expected result when the pre-treatment fit was already excellent: the ridge correction has almost nothing to fix, so it barely moves. ASCM earns its keep on datasets where SC visibly fails, which this is not. For a case where it does matter, see the Kansas tax-cut tutorial.

14. The whole ladder, side by side

Everything is now in place. First the units. Because the outcome is in logs, the counterfactual-minus-actual difference is in log points:

$$L_t = 100 \times \left( \hat{y}^{0}_{1,t} - y_{1,t} \right)$$

In words, multiply the log gap by 100 to get an approximate percentage shortfall, positive when the UK underperformed its counterfactual. In code, the script stores tau = y1 - yhat0, so the reported loss is -100 * tau. At these magnitudes the log approximation is good to two decimal places.

Method	2018Q4	2019Q4	Published	Note
DiD	4.98	6.18	—	not in the paper; pre-trends fail
SC	3.06	4.20	3.06 / 4.20	Frank–Wolfe
SC (exact QP)	3.04	4.17	—	the true optimum
DSC	2.99	4.12	2.98 / 4.12
SDID (i)	2.76	3.89	2.76 / 3.89
SDID (ii)	2.79	3.92	2.79 / 3.92
SDID (iii)	2.79	3.92	2.79 / 3.92
MASC	2.73	3.83	2.73 / 3.83	$m = 10$, $\phi = 0.158$
ASCM	3.04	4.19	3.04 / 4.19	8 negative weights
Born et al. (2019)	2.40	3.60		with covariates

Interpretation. Every cell of that table reproduces the published one to within 0.01 percentage points. Three things stand out. First, the spread across methods at 2018Q4 is 2.73% to 3.06% — a range of a third of a percentage point, which is small next to the gap to Born et al.’s 2.40%. Second, the estimated damage grows over time, from roughly 2.9% to roughly 4.1%, which looks like a change in the growth rate rather than a one-off level shift. Third, and this is the paper’s empirical punchline: every stage of the ladder puts the cost of the referendum above the previously published figure, and the methods that adjust for level and timing put it lowest, not highest.

Interpretation. Before the referendum the six lines are on top of each other — they are all fitting the same 86 quarters, and all fitting them well. The disagreement is entirely a post-treatment phenomenon, which is a useful reminder that pre-treatment fit cannot arbitrate between methods that all achieve it.

Interpretation. Four of the five recipes are nearly the same blend, dominated by Hungary, the United States, Japan, Canada and Norway. The SDID and DSC panels are identical by construction, since SDID reuses DSC’s unit weights and changes only the bias adjustment. Only ASCM crosses zero.

15. Which stage should you choose?

15.1 The in-sample placebo tournament

Pre-treatment fit cannot choose between these methods, because they all fit. So the source paper does something better: it moves the treatment date back to a quarter when nothing happened, builds the counterfactual using only data up to that point, and compares it to what actually occurred. The true effect is zero, so every estimate is pure error.

$$\text{RMSE} = \sqrt{ \frac{1}{20} \sum_{k=1}^{20} \left( y_{1, T’_k + h} - \hat{y}^{0}_{1, T’_k + h} \right)^{2} }$$

In words, across twenty artificial treatment dates, how far off is the counterfactual from an outcome we can actually check. In code, the loop runs k over the last pre-treatment quarters 2010Q1 to 2014Q4, always starting the window at 1995Q1, with h the forecast horizon.

flowchart LR
A("<b>Pick a fake<br/>treatment date</b><br/>2010Q1 ... 2014Q4<br/>20 of them") --> B("<b>Fit on 1995Q1<br/>up to that date</b><br/>all seven estimators")
B --> C("<b>Predict h quarters<br/>ahead</b>")
C --> D("<b>Compare to what<br/>actually happened</b><br/>true effect is zero")
D --> E("<b>Score</b><br/>RMSE, mean and median<br/>absolute error")
E --> A
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A,B blue
class C,D orange
class E teal

15.2 The published table, reproduced

Method	RMSE	MAB	MedAB	Published RMSE
SC	0.0089	0.0072	0.0055	0.0089
DSC	0.0087	0.0070	0.0052	0.0087
SDID (i)	0.0067	0.0037	0.0016	0.0067
MASC	0.0080	0.0062	0.0045	0.0080
ASCM	0.0086	0.0068	0.0051	0.0086
SDID (ii)	0.0134	0.0111	0.0103	0.0134
SDID (iii)	0.0134	0.0111	0.0107	0.0134

Interpretation. Every number reproduces exactly. Read as published, the table says SDID (i) wins comfortably — its median absolute error of 0.0016 is a third of plain synthetic control’s 0.0055 — and that variants (ii) and (iii) are the worst of the lot, twice as bad as plain SC. That is the basis for the paper’s recommendation to use variant (i) in practice.

15.3 The table is not comparing like with like

Look closely at how those numbers are produced. In the replication code, SC, DSC, SDID (i), MASC and ASCM are all graded one quarter ahead. SDID (ii) and (iii) are graded four quarters ahead. Forecasting a year out is a strictly harder task than forecasting a quarter out, so part of the gap is the exam, not the student.

Running every estimator at both horizons settles it:

Method	RMSE, $h = 1$	RMSE, $h = 4$
SC	0.0089	0.0150
DSC	0.0087	0.0149
SDID (i)	0.0067	0.0134
SDID (ii)	0.0066	0.0134
SDID (iii)	0.0066	0.0134
MASC	0.0080	0.0140
ASCM	0.0086	0.0146

Interpretation. Graded on the same task, the three SDID variants are indistinguishable — 0.0067, 0.0066, 0.0066 at one quarter, and 0.0134 for all three at four quarters. The published conclusion that variants (ii) and (iii) “perform the worst” is an artefact of the horizon, not a property of the estimators.

What survives is the finding that matters more: at either horizon, the whole SDID family beats every other stage, and the ordering below it is stable — SDID, then MASC, then ASCM, then DSC, then SC. The time weights are doing real work. Which variant supplies them is not settled by this evidence, and variant (i) remains the sensible default simply because it is the cheapest and requires no choice of post-treatment window.

16. Do covariates help?

The Brexit dataset ships with six covariates, and Born and coauthors matched on them. The source paper’s most pointed conclusion is that you should not.

Matching on covariates requires Abadie’s nested optimisation: an inner problem that picks weights given a predictor-importance matrix $\boldsymbol{V}$, and an outer problem that picks $\boldsymbol{V}$.

$$\hat{\boldsymbol{\omega}}(\boldsymbol{V}) = \underset{\boldsymbol{\omega} \in \mathbb{W}}{\arg\min} \, (\boldsymbol{z}_1 - \boldsymbol{Z}\boldsymbol{\omega})’ \boldsymbol{V} (\boldsymbol{z}_1 - \boldsymbol{Z}\boldsymbol{\omega})$$

In words, for a fixed opinion about how important each predictor is, pick the blend that best balances the predictors. In code, Synth::synth(X1 = ..., X0 = ...).

$$\hat{\boldsymbol{V}}^{sc} = \underset{\boldsymbol{V} \in \mathbb{V}}{\arg\min} \sum_{t=1}^{T_0-1} \left( y_{1,t} - \sum_{j=2}^{J+1} \hat{\omega}_j(\boldsymbol{V})\, y_{j,t} \right)^{2}$$

In words, choose the predictor importances that make the resulting blend track the UK’s outcome best. The variant used by Born and coauthors — call it SC(B) — replaces the outcome in this outer objective with the full stacked predictor vector, so covariates are scored in both loops:

# SC with covariates: outer loop scores the OUTCOME
synth(X1 = X1, X0 = X0, Z1 = as.matrix(Z1), Z0 = Z0)
# SC(B): outer loop scores the stacked PREDICTORS
synth(X1 = X1, X0 = X0, Z1 = X1, Z0 = X0)

X1 <- as.matrix(c(Z1, cov_means[UK, ])) # 86 outcomes + 6 covariate means
X0 <- rbind(Z0, t(cov_means[DONORS, ]))
s_born <- synth(X1 = X1, X0 = X0, Z1 = X1, Z0 = X0) # SC(B)
s_sc <- synth(X1 = X1, X0 = X0, Z1 = as.matrix(Z1), Z0 = Z0) # SC cov.
s_dsc <- synth(X1 = X1d, X0 = X0d, Z1 = as.matrix(Z1_dm), Z0 = Z0_dm) # DSC cov.

 method loss_2018Q4 loss_2019Q4 published
SC(B) 2.428 3.606 2.43
SC cov. 3.028 4.170 3.11
DSC cov. 2.942 4.050 2.90
SDID cov. (i) 2.731 3.839 2.75
SC no cov. 3.056 4.204 3.06
DSC no cov. 2.985 4.121 2.98
SDID no cov. (i) 2.758 3.894 2.76

Interpretation. The first row is the headline. SC(B) with mean covariates gives 2.43% at 2018Q4 and 3.61% at 2019Q4 — that is Born et al.’s 2.4% and 3.6%, reproduced to the second decimal. So the gap between the earlier published figure and everything else in this post is not a data difference or a coding difference. It is entirely the choice of estimator, and specifically the choice to score covariates in both optimisation loops.

Note also that the four covariate rows sit below their no-covariate counterparts in every case, and that our Synth-based numbers drift from the published ones by up to 0.08 percentage points. That drift is honest and expected: the outer optimisation over the predictor-importance matrix is not convex, Synth runs a derivative-free search over 92 dimensions, and different starting values land in different local optima. When a specification’s answer depends on where the optimiser started, that is information about the specification.

But the decisive evidence is not in this table at all — it is in the placebo tournament. Rerunning section 15 with covariates in the matching set makes almost every estimator worse:

Method	RMSE without covariates	RMSE with mean covariates
SC	0.0089	0.0092
DSC	0.0087	0.0106
SDID (i)	0.0067	0.0063
MASC	0.0080	0.0048
ASCM	0.0086	0.0083

(the covariate column is Table 8 of the source paper; our no-covariate column reproduces its Table 7 exactly)

Interpretation. Adding six covariates degrades SC and DSC and barely helps SDID and ASCM. Only MASC clearly benefits. The source paper’s conclusion is blunt, and our replication supports it: if the object of interest is the GDP series, do not add covariates. The reason is not mysterious. Kaul and coauthors [13] showed that once all pre-treatment outcomes are in the matching set, covariates are redundant — the outcomes already encode whatever the covariates would have told you. With 86 pre-treatment outcomes in play, the six extra predictors add estimation noise and nothing else.

17. Robustness: the specification zoo

Four departures from the headline specification, each a short table and a single lesson.

(a) The other treatment date. The referendum fell at the very end of 2016Q2, so dating the treatment there rather than at 2016Q3 is entirely defensible.

 method 2016Q2_2018Q4 2016Q3_2018Q4
SC 3.123 3.056
DSC 3.036 2.985
SDID (i) 3.170 2.758
MASC 2.769 2.726

Interpretation. SC, DSC and MASC barely notice — they move by less than 0.07 percentage points. SDID moves by 0.41, from 2.76% to 3.17%, which is more than the entire spread across methods at a fixed date. The reason is visible in the time-weight figure: SDID fits its time weights against the treatment quarter, so changing which quarter that is changes the target of the fit. At 2016Q2 the weight splits 0.84/0.16 across the last two quarters instead of collapsing onto one. The choice of treatment date is not innocuous, and it bites hardest on precisely the estimator the tournament recommends.

(b) Dropping the United States. The US carries about a fifth of the weight, and if spillovers exist they should be concentrated in the highest-weighted donors [17].

 method with_US without_US
SC 3.056 3.067
DSC 2.985 3.032
SDID (i) 2.758 2.818

Interpretation. Every estimate moves by at most 0.06 percentage points and all three move up. Whatever the no-interference assumption is doing here, it is not driving the result.

(c) The ridge penalty. Arkhangelsky and coauthors propose an automatic regularisation of the unit weights, whose main theoretical benefit is that it makes the solution unique — which, given section 8.4, is not a trivial gain.

$$\zeta = \left( T_{post} \right)^{1/4} \sqrt{ \frac{1}{J (T_0 - 2)} \sum_{j=2}^{J+1} \sum_{t=1}^{T_0-2} \left( \Delta_{j,t} - \bar{\Delta} \right)^{2} }$$

In words, the penalty scale is the standard deviation of donors’ quarter-to-quarter GDP changes, inflated by the fourth root of the number of post-treatment quarters. In code, this is what synthdid uses when you omit zeta.omega = 0.

 method no_penalty with_penalty
SC 3.056 3.093
DSC 2.985 3.090
SDID (i) 2.758 2.642

Interpretation. Penalising moves SC up by 0.04, DSC up by 0.11 and SDID down by 0.12. All within the spread we have already seen. The penalty is worth switching on for the uniqueness it buys, not because it changes any conclusion.

(d) Mean covariates. Section 16 asks whether covariates help as a matter of estimator design; the same question belongs here as a specification cell, because it is the one departure that moves every stage in the same direction.

 method no_covariates with_covariates
SC 3.056 3.028
DSC 2.985 2.942
SDID (i) 2.758 2.731

Interpretation. All three fall, and none by more than 0.05 percentage points. The covariate route that genuinely moves the answer is SC(B) at 2.43%, and that is a different estimator — Synth’s nested optimisation over 92 predictors — rather than a different specification of the ones on this ladder. Section 16 has the detail.

Interpretation. Across every specification in this section, the estimated loss sits between roughly 2.6% and 3.2%. Nothing plotted here reaches Born et al.’s 2.4% — the closest is SDID under the ridge penalty, at 2.64. Only SC(B) gets there, and it sits in section 16 rather than in this figure. That is the whole robustness story in one picture, and it is the reason the source paper insists on reporting a cloud rather than a point.

18. Inference: what the paper does not do

This section goes beyond the paper. De Brabander, Juodis and Miyazato Szini state in their Remark 1 that they consider point estimates only and set inference aside entirely, on the grounds that inference for this class of problems is genuinely hard. That is a defensible position for a methods comparison, and a poor place for a first-time learner to stop. Everything that follows is our addition.

18.1 Placebo in space

The classic device, due to Abadie and coauthors: pretend each donor in turn was the treated country, run the whole procedure, and see whether the UK’s post-treatment gap is unusual against that reference distribution. The statistic is the ratio of post-treatment to pre-treatment fit error, which corrects for the fact that a country the method fits badly will show a large gap for uninteresting reasons.

$$R_j = \frac{ \sqrt{ \frac{1}{T - T_0 + 1} \sum_{t \geq T_0} \hat{\tau}_{j,t}^{2} } }{ \sqrt{ \frac{1}{T_0 - 1} \sum_{t < T_0} \hat{\tau}_{j,t}^{2} } }, \qquad p = \frac{1}{J+1} \sum_{j=1}^{J+1} \mathbf{1}\{ R_j \geq R_1 \}$$

In words, form the post-over-pre error ratio for every country and ask what fraction look at least as extreme as the UK. In code, ratio_of() applied to each placebo run, then the rank of the UK.

placebo_space <- lapply(DONORS, function(j) {
pool <- setdiff(DONORS, j) # the UK is excluded throughout
w <- simplex_fw(Y[1:86, pool], Y[1:86, j])
Y[, j] - as.vector(Y[, pool] %*% w) # this donor's placebo gap path
})

 UK post/pre RMSPE ratio : 5.82
rank among 24 countries : 1
permutation p-value : 0.042 (finest attainable: 0.042)
synthdid placebo standard error (SDID): 0.00948 log points

Interpretation. The UK’s post-treatment fit error is 5.8 times its pre-treatment fit error, and that ratio is the largest of all 24 countries. The permutation p-value is therefore 0.042, the smallest value this design can produce. Visually, the orange line leaves the grey band shortly after the referendum and never returns.

The synthdid placebo standard error tells a more sobering story: 0.0095 log points, which puts a conventional 95% interval around the SDID estimate at roughly 0.9% to 4.6%. The point estimate is much better determined than the interval, which is the normal state of affairs with one treated unit and 23 donors. Anyone quoting “Brexit cost 2.8% of GDP” without that interval is overstating what this design can deliver.

18.2 What this can and cannot tell you

Three caveats, all of which matter.

First, with 23 donors the finest attainable p-value is $1/24 \approx 0.042$. The test simply cannot reject at the 1% level no matter how extreme the UK looks. This is a property of the design, not of Brexit.

Second, this is randomisation inference: it asks how unusual is the United Kingdom among OECD countries, not what is the sampling error of this estimate. Those are different questions, and only the first one has a well-defined answer here.

Third, for a genuinely model-based alternative, the source paper itself points to the conformal-inference approach of Chernozhukov, Wüthrich and Zhu [16], which augsynth implements and which the Kansas tutorial works through in detail.

19. The same ladder in Stata and Python

Everything above is R. The ladder is not, and a reader who works in Stata or Python should not have to take the results on trust. This section ports the whole thing twice — cheatsheet_stata.do and cheatsheet_python.py, alongside cheatsheet_R.R — and the disagreements between the three turn out to be the most instructive part.

All three sheets share one device. Each of these packages reports an ATT averaged over all post-treatment periods, but we want the shortfall at two specific quarters. So keep the 86 pre-treatment quarters plus the single quarter of interest, renumber time, and the average over “all post periods” becomes an average over one period. The bare package call then returns exactly the number we want, and none of the three files contains any post-estimation arithmetic.

19.1 What maps onto what

Stage	R	Stata	Python (`mlsynth`)
DiD	`did_estimate()`	`sdid …, method(did)`	`FDID(…).fit().did`
SC	`sc_estimate()`	`sdid …, method(sc)`	`VanillaSC(…)`
DSC	`synthdid_estimate(lambda = uniform)`	`sdid …, method(sc)` on demeaned $y$	`TSSC(…, method = "MSCa")`
SDID	`synthdid_estimate()`	`sdid …, zeta_omega(0) zeta_lambda(0)`	`SDID(…, zeta = 0)`
MASC	`masc()`	— none —	`MASC(…, set_f = range(6, 87))`
ASCM	`augsynth(progfunc = "Ridge")`	`allsynth …, bcorrect(merge)`	`VanillaSC(…, augment = "ridge")`

Two entries need explaining before the numbers make sense.

DSC in Stata. There is no dsc command and none is needed. Demeaned SC is SC run on outcomes from which each country’s own pre-treatment mean has been subtracted: after demeaning, the pre-treatment gap averages to zero by construction, so the double difference collapses to the single one. Three lines of bysort and a method(sc) call reproduce it exactly.

ASCM in Stata is a different estimator. allsynth implements the bias-corrected synthetic control of Abadie and L’Hour and of Ben-Michael, Feller and Rothstein: fit SC, then regress the outcome on the predictors across the donor pool and subtract the predicted discrepancy. augsynth uses ridge-augmented SC. They are cousins, not the same estimator. Worse, the bias correction is an OLS fit across donors, so it needs more control units than predictors — with 23 donors we cannot hand it all 86 pre-treatment quarters the way a ridge penalty can. The path has to be summarised, and the summary matters enormously: with sparse individual lags the bias-corrected estimate swings between $-0.8$ and $5.1$ depending on which quarters you pick. Block means are far better conditioned, and the do-file fits a small grid of them and keeps the one with the lowest pre-treatment RMSPE — a rule fixed in advance that never looks at the post-treatment answer.

19.2 The three ports, side by side

Shortfall in UK real GDP (%), treatment dated 2016Q3, outcomes only.

Stage	R 2018Q4	Stata 2018Q4	Python 2018Q4	R 2019Q4	Stata 2019Q4	Python 2019Q4
DiD	4.98	4.98	4.98	6.18	6.18	6.18
SC	3.06	3.06	3.04	4.20	4.20	4.17
DSC	2.99	2.99	2.99	4.12	4.12	4.12
SDID (ii)	2.79	2.79	2.80	3.92	3.92	3.94
MASC	2.73	—	2.73	3.83	—	3.83
ASCM	3.04	3.10	3.04	4.19	4.22	4.19

The SDID row reports variant (ii) in all three languages, so that the comparison is like with like; section 14’s headline SDID is variant (i), at 2.76. Stata reproduces R to five decimal places at every stage it can fit. Python agrees on DiD, DSC, MASC and ASCM, and differs in the second decimal on SC and SDID.

19.3 The disagreement is the finding

The bolded cells are not a bug in any of the three libraries. They are section 8.4’s solver story arriving from a completely independent direction.

Recall the problem: the SC objective on this panel has a condition number around $7.5 \times 10^{5}$, which means a wide, nearly flat valley of near-optimal weight vectors. synthdid walks that valley with Frank–Wolfe on a capped iteration budget and stops at 3.06. mlsynth hands the identical problem to a convex solver, which runs it to optimality and returns 3.04 — which is precisely the “SC (exact QP)” row computed by hand in section 8.4, on both quarters. Stata’s sdid inherits synthdid’s Frank–Wolfe and stops in the same place; tighten its convergence with max_iter(100000) min_dec(1e-9) and the SDID estimate drifts from 2.79 to 2.80, which is where Python already is.

So three implementations, written independently in three languages, sort themselves into exactly two camps — and the split is by solver, not by language or by author. That is a much stronger piece of evidence for the section 8.4 claim than the iteration ladder in the original analysis, because nobody was trying to make this point when they wrote mlsynth.

The practical lesson is not that one library is right. It is that a synthetic control estimate carries its solver’s fingerprint, and that a second-decimal disagreement between implementations is the normal state of affairs rather than a cause for alarm.

19.4 Three traps the ports exposed

Each language has a default that quietly gives you the wrong estimator, and in all three cases it is the same default.

Every package penalises by default; the paper does not. R needs zeta.omega = 0, zeta.lambda = 0, Python needs zeta = 0, and Stata needs zeta_omega(0) zeta_lambda(0). Leave any of them alone and SDID reports 2.66–2.67 instead of 2.79. Stata’s version of this trap is the nastiest: the documented default is zeta_omega(1e-6), which looks like a value but is a magic sentinel — sdid.ado reads if (EOmega==1e-6) EtaOmega = (yNtr*yTpost)^(1/4), so passing the documented default explicitly still requests the full penalty. Only 0 switches it off.

MASC’s fold set has to be given explicitly in both languages that have MASC. R’s masc and Python’s mlsynth.MASC both cross-validate over a fold set that, left to its default, is not the one the paper uses. Pass set_f = 6:T0 in R and set_f=range(6, 87) in Python and the two agree to three decimals at 2.726. This is the same trap flagged in section 12, and it survives translation.

mlsynth.DSC is not this post’s DSC. mlsynth ships a class named DSC which implements Distributional Synthetic Control (Gunsilius) — matching whole outcome distributions. The DSC on this ladder is Demeaned Synthetic Control, which in mlsynth is TSSC(method = "MSCa"). Same three letters, different estimators, and importing the wrong one raises no error at all. It simply answers a different question. The mapping used here follows mlsynth issue #312, which is itself a reading of the paper this post replicates.

The Python column of these tables is only a cheat sheet. The Python edition of this post climbs the same ladder at full length with mlsynth alone — every config option, every result field, the three SDID flavours, the covariate routes this cheat sheet skips, and the wider catalogue of estimators the library ships. Read it if you work in Python; read this one for the derivations.

19.5 What each language cannot do

Reported plainly rather than papered over. MASC has no Stata implementation, so that row is empty rather than approximated. SC(B) and the other covariate specifications are in none of the three sheets, because they need Synth’s nested optimisation over 92 predictors and turn a thirty-second script into a coffee break — section 16 and analysis.R §14d cover them. And the standard errors each package reports are its own recommended method, not a common yardstick: R’s placebo SEs, Stata’s placebo SEs at a different replication count, augsynth’s jackknife and mlsynth’s analytic FDID error are not comparable digit for digit. Read them as orders of magnitude, and note that every one of them is wide enough to contain zero.

20. Discussion

What Brexit cost. Taking the ladder as a whole, the referendum had cost the UK somewhere between 2.7% and 3.1% of GDP by the end of 2018, and between 3.8% and 4.2% by the end of 2019. That is above the 2.4% previously published for this dataset, and the reason is not exotic: the earlier figure came from a specification that matched on covariates, and covariates make the counterfactual worse here rather than better.

Three caveats belong with that number. It is a net gap between the UK and a blend of OECD economies, not a Brexit-only effect — anything else distinctive that happened to the UK after mid-2016 is inside it. The no-interference assumption is strong over a four-year horizon when the United States carries a fifth of the weight in the counterfactual. And the estimate is a point on a specification cloud, not a parameter that has been pinned down.

What the ladder taught. The durable idea is the bias decomposition, not the leaderboard. Extrapolation bias and interpolation bias are separate failures with separate fixes, unit weights address the first, time weights address the second, and any weighted counterfactual you ever build can be interrogated on both counts. The ranking of estimators on this dataset is far more perishable — it depends on the outcome behaving like a random walk, on a long pre-period and on a treated unit that sits inside the convex hull.

It also cuts the other way, and the source paper says so plainly: SDID’s advantage over DSC is marginal once you count the 85 extra parameters it estimates, and neither MASC nor ASCM justifies its computational cost here. Our own placebo results agree — the whole SDID family clusters together, and the gap down to DSC is smaller than the gap between covariates and no covariates.

So what should you actually do? Fit the ladder, not a stage. Run an in-sample placebo tournament and check that the horizons are matched. Report the range. If one specification is going to be the headline, choose it before you see the estimates, and show the others anyway. Ferman, Pinto and Possebom [15] have documented how much room for cherry-picking this literature leaves; the honest response is to publish the cloud.

21. Summary and next steps

Every estimator here is one weighted two-way regression with a different choice of unit weights $\omega$, time weights $\lambda$, and whether the unit fixed effect is switched on. DiD, SC, DSC and SDID are four settings of the same expression; MASC and ASCM change the feasible set instead.
One solver does five jobs. The simplex least-squares problem solves the unit weights, the demeaned unit weights, the time weights (on the transpose) and both components of MASC’s cross-validation.
The Brexit cost is 2.7–3.1% at end-2018 and 3.8–4.2% at end-2019, above the previously published 2.4%, and the difference is driven mostly by the covariate specification.
The SDID family wins the placebo tournament at either forecast horizon, but the published ranking within that family does not survive matching the horizons.
Covariates hurt here. With 86 pre-treatment outcomes already in the matching set, six extra badly-scaled predictors add estimation noise without adding identification.
Two practical traps cost real accuracy: masc’s fold argument silently produces five folds instead of eighty, and a flat objective means synthdid’s optimiser stops on its iteration cap rather than at the optimum.
Both traps survive translation. Porting the ladder to Stata and Python (section 19) reproduces every estimate, and the places where it does not are the solver, not the language: mlsynth’s convex solver lands on the exact-QP answer while synthdid and Stata’s sdid stop where Frank–Wolfe stops. Three implementations, two camps, split by solver.

Where to go next: the three cheat sheets if you just want working code — cheatsheet_R.R, cheatsheet_stata.do, cheatsheet_python.py — then multi-country and staggered adoption with multisynth, the same SDID estimator in Stata on Proposition 99, or manual demeaning and the FWL theorem if the unit-fixed-effect story in stage three felt too quick. The Monte Carlo study in the source paper, which stress-tests this ranking on simulated data, is the subject of a future post.

22. Exercises

Move the treatment date. Re-run SC and SDID (i) dating the treatment at 2016Q1 rather than 2016Q2 or 2016Q3. Which estimator moves more, and can you explain why using the time-weight figure?
Read the simplex. Using simplex_ls, solve the SC problem restricted to just the United States, Hungary and Canada, and plot the objective over the triangle. Now add Japan as a fourth donor. By how much does the pre-treatment MSPE fall?
Break DSC on purpose. Add a constant of 0.05 log points to every UK observation, before and after the referendum. Which of SC, DSC and SDID change their estimated effect, and which do not? Explain the result using the unit fixed effect $\alpha_j$ in the master regression.
Force the time weights to spread out. Re-run SDID on first-differenced log GDP instead of levels. Does the spike on the final quarter survive? Then argue, using the source paper’s own reasoning, why the authors declined to make this switch in their headline results.
Finish the horizon audit. Section 15.3 matched the horizons for the placebo tournament. Extend it to $h = 2$ and $h = 8$. Is the SDID family’s advantage over SC stable in the horizon, or does it shrink?
Separate MASC’s two dials. Compute the 2018Q4 estimate for $\phi$ on a grid from 0 to 1 in steps of 0.05, holding $m = 10$. How much of the difference between MASC’s 2.73% and SC’s 3.06% is due to the chosen $\phi$ rather than to the choice of $m$?
Drop the biggest donor. The United States carries about a fifth of the weight in most specifications, and spillover risk is concentrated in the highest-weighted donors. Re-run the entire ladder without the United States. Does your conclusion about Brexit change? Then ask the harder question: does your conclusion about which estimator to use change?
Build your own stage. DSC and SDID differ only in how the bias adjustment is weighted across pre-periods — flat in one, optimised in the other. Propose a third weighting, for example exponentially decaying weights with a half-life you choose, implement it, and enter it in the placebo tournament. Does it beat SDID?

23. References

de Brabander, E., Juodis, A., & Miyazato Szini, G. (2025). On the use of synthetic difference-in-differences approach with (-out) covariates: The case study of Brexit referendum. Econometric Reviews, 44(10), 1617–1646.
Born, B., Müller, G. J., Schularick, M., & Sedláček, P. (2019). The costs of economic nationalism: Evidence from the Brexit experiment. The Economic Journal, 129(623), 2722–2744.
Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93(1), 113–132.
Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies. Journal of the American Statistical Association, 105(490), 493–505.
Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510.
Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
Doudchenko, N., & Imbens, G. W. (2016). Balancing, regression, difference-in-differences and synthetic control methods: A synthesis. NBER Working Paper 22791.
Ferman, B., & Pinto, C. (2021). Synthetic controls with imperfect pretreatment fit. Quantitative Economics, 12(4), 1197–1221.
Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118.
Kellogg, M., Mogstad, M., Pouliot, G. A., & Torgovitsky, A. (2021). Combining matching and synthetic control to trade off biases from extrapolation and interpolation. Journal of the American Statistical Association, 116(536), 1804–1816.
Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536), 1789–1803.
Kaul, A., Klößner, S., Pfeifer, G., & Schieler, M. (2022). Standard synthetic control methods: The case of using all preintervention outcomes together with covariates. Journal of Business & Economic Statistics, 40(3), 1362–1376.
Botosaru, I., & Ferman, B. (2019). On the role of covariates in the synthetic control method. The Econometrics Journal, 22(2), 117–130.
Ferman, B., Pinto, C., & Possebom, V. (2020). Cherry picking with synthetic controls. Journal of Policy Analysis and Management, 39(2), 510–532.
Chernozhukov, V., Wüthrich, K., & Zhu, Y. (2021). An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, 116(536), 1849–1864.
Di Stefano, R., & Mellace, G. (2024). The inclusive synthetic control method. arXiv:2403.17624.
Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16(4), 437–450.
Software, R: synthdid · Synth · masc · augsynth · quadprog
Software, Stata: sdid (Clarke, Pailañir, Athey & Imbens) · synth (Abadie, Diamond & Hainmueller) · allsynth (Wiltshire)
Software, Python: mlsynth (Greathouse). The estimator mapping used in section 19 follows issue #312, which reads the same source paper this post replicates.
Companion tutorials on this site: Synthetic control on the Basque Country · Augmented synthetic control and the Kansas tax cuts · Synthetic difference-in-differences on Proposition 99 · Manual demeaning and two-way fixed effects · Multi-country augmented synthetic control

Acknowledgements

Who Are My Neighbors? Bayesian Estimation of Spatial Weight Matrices

Thu, 30 Jul 2026 00:00:00 +0000

Abstract

Every spatial econometric result rests on a neighbourhood map that somebody chose before the analysis began, and if that map is wrong, every spillover estimate built on it is wrong too. This tutorial does not choose the map: it estimates it. Using the estimateW package and its built-in nuts1growth panel — 90 European NUTS-1 regions observed annually from 2001 to 2019, 1,710 observations, assembled from Eurostat regional accounts — it treats all 8,010 off-diagonal cells of the spatial adjacency matrix as unknown parameters and samples them jointly with the spatial autoregressive parameter, the slope coefficients and the error variance. Estimation uses a Gibbs sampler with element-wise Bernoulli updates for the links, a griddy-Gibbs step for the spatial parameter, and a beta-binomial sparsity prior anchored at seven expected neighbours per region. The replication reproduces all twelve published quantities of Krisztin and Piribauer’s Table 3 to five decimal places, recovering a spatial autoregressive parameter of 0.71322 with a posterior standard deviation of 0.01574. The indirect impact of initial productivity, −0.03972, is 2.11 times the direct impact of −0.01880, so most of the growth response to a region’s starting position is felt outside its own borders. The estimated network turns out to be organised by country rather than by shared borders. The practical implication is that reported spillovers are only as credible as the neighbourhood map behind them, and here the map the data choose disagrees with the one geography would have supplied.

1. Overview

Imagine you have the full transcript of a long conference dinner: who laughed at which joke, who fell silent when, who picked up whose thread. What you do not have is the seating chart. You could still work out who was sitting near whom, because people who sit together react together.

Spatial econometrics normally works the other way around. It hands you the seating chart first — drawn by a cartographer, from shared borders or straight-line distances — and asks you to trust it. Every conclusion about spillovers is then conditional on that chart being right.

This tutorial asks the obvious question: which European regions actually behave as each other’s neighbours for productivity growth — and does it matter whether we assume that map or estimate it?

The tool is estimateW, an R package by Tamás Krisztin and Philipp Piribauer that treats the neighbourhood structure as something to be inferred rather than assumed. We replicate their published European application exactly, then push past it: we check the sampler against a network we build ourselves, we tour the rest of the model family, and we set the estimated map side by side with the contiguity map we would otherwise have used.

This tutorial assumes no prior spatial econometrics and no prior Bayesian statistics. Both are built from zero. If you already know one of them, the relevant section will read quickly; nothing later depends on you having skipped it.

If you want the conventional treatment first — the same model family with the neighbourhood map fixed in advance — the companion post Spatial Dynamic Panel Data Modeling in R fits SAR, SDM and their dynamic extensions to US cigarette demand with an assumed contiguity matrix. Read side by side, the two posts are the same machinery with the map moved from the inputs to the outputs.

graph LR
A("<b>Data</b><br/>90 NUTS-1 regions<br/>2001-2019, T = 19") --> B("<b>Priors</b><br/>structure + sparsity<br/>k-bar = 7")
B --> C("<b>Answer key</b><br/>sim_dgp()<br/>recover a known W")
C --> D("<b>Estimate</b><br/>sarw()<br/>rho, beta, sigma2, W")
D --> E("<b>Read the network</b><br/>links, chords,<br/>multipliers")
E --> F("<b>Benchmark</b><br/>estimated W vs<br/>contiguity and 7-NN")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class A,E blue
class B orange
class C teal
class D key
class F anchor

Read the arrows as escalating trust. Step 2 makes an impossible-looking problem tractable, step 3 shows the machinery works on a case where we know the answer, and only then does step 4 spend that credibility on real data. The last box is the payoff: one model, three different neighbourhood maps, three different answers.

1.1 Learning objectives

Explain what a spatial weight matrix is, why row-standardization matters, and why fixing it in advance is a modelling assumption rather than a fact.
Derive the Bernoulli conditional posterior that lets a Gibbs sampler switch a single link on or off, and the griddy-Gibbs step that handles the spatial parameter.
Estimate a spatial autoregressive panel with an unknown neighbourhood structure using sarw(), reproducing the published spatial parameter of 0.713 and the full impact decomposition.
Assess whether the sampler works by recovering a network you built yourself with sim_dgp(), and by reading trace plots, effective sample sizes and multi-chain diagnostics.
Compare the estimated map against contiguity and nearest-neighbour maps built from real NUTS-1 geometry, and judge which conclusions survive the change.

1.2 Key concepts at a glance

This post leans on a small vocabulary repeatedly, and everything after Section 5 assumes you can move between these terms quickly. Each concept below has three parts. The definition is always visible. The example and analogy sit behind clickable cards: open them when you need them, leave them collapsed for a quick scan. If a later section mentions the spatial multiplier or the sparsity prior and the term feels slippery, this is the section to re-read.

1. Spatial weight matrix $W$. An $n \times n$ table of neighbour weights. Row $i$ says who influences region $i$. Row-standardization makes every row sum to one. So $W y$ is just each region’s neighbour average.

Example

Here $W$ is 90 by 90, one row and one column per European region. A region with 7 estimated neighbours gives each of them weight $1/7 \approx 0.143$. The other 82 entries in that row are exactly zero.

Analogy

A phone contact list where your daily call minutes are split evenly among everyone on it. Adding a name does not buy you more minutes. It just gives everyone a thinner slice. That even split is row-standardization.

2. Adjacency matrix $\Omega$. The raw yes-or-no version of $W$. Each entry $\omega_{ij}$ is 1 if $j$ is a neighbour of $i$, and 0 otherwise. The diagonal is fixed at zero. Conventional models fix every entry in advance. Here every entry is estimated.

Example

With $n = 90$ regions, $\Omega$ has $90^2 - 90 = 8{,}010$ off-diagonal cells, and each one is a separate unknown parameter. We have only $90 \times 19 = 1{,}710$ observations to pin them down.

Analogy

An old telephone switchboard with 8,010 unlabelled sockets. Each one is either patched or not. Somebody threw away the wiring diagram, and you have to reconstruct it from listening to the calls.

3. Spatial autoregressive parameter $\rho$. One number for the whole system. It scales how strongly neighbours’ outcomes feed into your own. At $\rho = 0$ space is irrelevant. Stability requires $|\rho| < 1$.

Example

We estimate $\rho = 0.713$ with a posterior standard deviation of 0.016. A one-point rise in a region’s neighbour-average growth rate moves its own growth by about 0.713 points, before any feedback.

Analogy

The gain knob on a public-address system. Every speaker feeds every microphone a little. Turn the knob up and the room gets louder and louder from the same original sound. Past one, it howls.

4. Spatial multiplier $(I_n - \rho W)^{-1}$. The device that turns one local shock into a system-wide outcome. Round one hits your neighbours. Round two hits their neighbours. Powers of $\rho$ shrink each later round.

Example

With $\rho = 0.713$ the second round still carries weight $0.713^2 = 0.508$ and the third $0.362$. That is why the multiplier network is far denser than $W$ itself: almost everyone eventually reaches almost everyone.

Analogy

A rumour retold office to office. Each retelling reaches more people and loses a little detail. It never quite stops, but it does get quieter every time it is passed on.

5. Direct, indirect and total impacts. Slope coefficients are not marginal effects in a spatial model. The direct impact is the own-region effect. The indirect impact is the spillover onto everyone else. Total is the two added.

Example

For initial productivity the direct impact is $-0.0188$ and the indirect is $-0.0397$. The spillover is 2.11 times the own-region effect, so most of the action happens outside the region’s own borders.

Analogy

Your shop cuts its prices. The direct effect is what happens at your own till. The indirect effect is what happens at every other till on the street. On a busy street the second number is the bigger one.

6. Prior, likelihood, posterior. The prior is what you believed before seeing the data. The likelihood says how well each candidate explains the data. The posterior is the updated belief. Bayesian output is always a posterior distribution, never a single number.

Example

Before seeing any data, each of the 8,010 possible links carries a prior probability of $7/89 = 0.079$. After the sampler runs, a handful of them sit far above that and most sit far below.

Analogy

A weather app that opens with the seasonal average for today’s date, then revises the forecast with every new radar sweep. The seasonal average is the prior. The radar is the likelihood. What you actually read on screen is the posterior.

7. Gibbs sampling. A way to sample from a huge joint distribution one piece at a time. Hold everything else fixed. Draw one unknown from its conditional distribution. Repeat, thousands of times.

Example

Each sweep of our sampler visits all 8,010 possible links in random row order. For each one it computes the probability that the link is on, given everything else, and flips a weighted coin.

Analogy

Tuning a piano string by string. You never solve the whole instrument at once. You fix one string given how the others currently sound, then move on, and go round again until nothing changes.

8. Sparsity prior. A prior on how many neighbours each region has, rather than on which ones. It pulls the network toward being sparse. The seemingly neutral alternative quietly expects a very dense network.

Example

We anchor the prior at $\underline{k} = 7$ expected neighbours, which sets $\underline{b}_\omega = (89 - 7)/7 = 11.71$. The flat alternative would instead expect 44.5 neighbours per region — half of Europe.

Analogy

A hiring budget that does not forbid any particular offer, but quietly assumes you will end up with about seven people. You can hire the eighth. You just have to justify it.

2. Setup

The analysis needs estimateW for the samplers and its built-in data, sf for the NUTS-1 geometry used in the benchmark section, coda for convergence diagnostics, circlize for the chord diagrams, and the usual ggplot2 and dplyr stack. Everything is on CRAN and pure R except sf, which needs GDAL. Section 13 is the only part that touches the network, and even that is cached after the first run.

cran_packages <- c("estimateW", "ggplot2", "dplyr", "tidyr", "readr", "tibble",
"purrr", "glue", "scales", "patchwork", "coda", "sf", "circlize")
missing <- cran_packages[!sapply(cran_packages, requireNamespace, quietly = TRUE)]
if (length(missing) > 0) install.packages(missing, repos = "https://cloud.r-project.org")
library(estimateW)
library(ggplot2); library(dplyr); library(tidyr); library(coda); library(sf)
packageVersion("estimateW")

[1] '0.2.0'

Pin that version. Every number in this post comes from estimateW 0.2.0 under R 4.5.2, and the exact-reproduction claim in Section 9 depends on both.

All figures use the site’s dark-navy palette, defined once and reused:

BG <- "#0f1729"; GRID <- "#1f2b5e"; TEXT <- "#c8d0e0"; WHITE <- "#e8ecf2"
STEEL <- "#6a9bcc"; ORANGE <- "#d97757"; TEAL <- "#00d4c8"
# theme_dark_site() and save_fig() are defined in analysis.R

3. The data: 90 European regions, 19 years

The package ships the panel used in the published application, so nothing has to be downloaded.

data(nuts1growth)
str(nuts1growth)

'data.frame': 1710 obs. of 6 variables:
$ NUTS1 : chr "AT1" "AT2" "AT3" "BE1" ...
$ year : int 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 ...
$ growth_gdp_pw: num 0.0264 0.021 0.0249 0.0223 0.0139 ...
$ init_gdp_pw : num 11 10.8 10.9 11.3 11 ...
$ loweduc : num 22.5 22.6 26.2 37.5 40.4 44.5 34.5 30.5 38.5 13.9 ...
$ higheduc : num 15.9 11.5 13.5 37 26.6 25 15.2 21.2 25.1 11.5 ...

Six columns and 1,710 rows. growth_gdp_pw is the annual growth rate of real gross value added per worker — labour productivity growth, our outcome. init_gdp_pw is the log level of productivity in the previous year, the conditional-convergence term. loweduc and higheduc are the shares of the working-age population with the lowest and highest ISCED education levels, both lagged one year; the omitted middle category is the benchmark. The rows are stacked year by year, and the region order repeats identically in each of the 19 blocks — a detail that matters, because the sampler indexes the weight matrix by row position, not by region code.

The model matrices follow the published specification exactly. There are no spatially lagged regressors here, so everything goes into Z:

Y <- as.matrix(nuts1growth$growth_gdp_pw)
Z <- cbind(1, nuts1growth$init_gdp_pw,
nuts1growth$loweduc,
nuts1growth$higheduc)
n <- 90
tt <- 19
dim(Y); dim(Z)

[1] 1710 1
[1] 1710 4

That is $n \times T = 90 \times 19 = 1{,}710$ rows, and four columns in Z: an intercept, initial productivity, and the two education shares. Note what is not in the call — no weight matrix. That absence is the whole point of the exercise.

The 90 regions come from 26 countries. Following the published grouping, we sort them into five supranational blocs, which will do a lot of work when we come to read the estimated network:

table(regions$country_group)

Southern Northern Western CEE Baltics
21 6 41 19 3

Western Europe supplies 41 of the 90 regions, largely because Germany alone contributes 16 and France 13. Eleven countries — Cyprus, Czechia, Denmark, Estonia, Ireland, Lithuania, Luxembourg, Latvia, Malta, Slovenia and Slovakia — are a single NUTS-1 region each. Keep that asymmetry in mind: a finding that “regions cluster within countries” is partly mechanical for Germany and impossible for Malta.

4. What the growth data look like before we assume anything spatial

Before introducing any neighbourhood structure, it is worth seeing what is in the data.

p1a <- ggplot(panel, aes(year, growth_gdp_pw, group = NUTS1)) +
geom_line(colour = TEXT, alpha = 0.16, linewidth = 0.3) +
geom_line(data = filter(panel, NUTS1 %in% hi),
aes(colour = country_group), linewidth = 0.9)
p1b <- ggplot(panel_summary, aes(init_gdp_pw_2001, mean_growth,
colour = country_group)) +
geom_point(size = 2.1) +
geom_smooth(method = "lm", se = FALSE, colour = ORANGE)
save_fig(p1a / p1b, "01_panel_overview", w = 10, h = 8)

Figure 1. Top: growth paths, with the 2009 crisis visible as a synchronised collapse across every region. Bottom: initially poorer regions grew faster over 2001–2019.

Two things stand out. First, the growth paths move together — the 2009 trough is a single downward spike shared by essentially all 90 regions. That common movement is exactly what a spatial model tries to explain, and also exactly what makes the exercise hard: co-movement can come from genuine spillovers or from shocks that hit everyone at once. Second, the scatter shows textbook beta-convergence, with the Central and Eastern European regions in the poor-and-fast corner and the Western European regions in the rich-and-slow one.

A pooled OLS regression makes the convergence pattern numerical, and gives us a baseline with no spatial structure whatsoever:

ols <- lm(growth_gdp_pw ~ init_gdp_pw + loweduc + higheduc, data = panel)
summary(ols)$coefficients

 Estimate Std. Error t value Pr(>|t|)
(Intercept) 4.199319e-01 0.0185948061 22.5832924 5.132094e-99
init_gdp_pw -3.677661e-02 0.0019464708 -18.8939955 1.900699e-72
loweduc -6.120117e-05 0.0000692265 -0.8840715 3.767822e-01
higheduc 2.638630e-04 0.0001544331 1.7085905 8.770872e-02

A one-log-point higher starting productivity is associated with 3.68 percentage points slower annual growth. Hold on to that number: when we add spatial structure, the coefficient on the same variable will fall to −1.69 percentage points. Nothing about the data will have changed. What changes is the bookkeeping — the non-spatial model credits each region alone with everything that happens near it, while the spatial model splits the same association into an own-region part and a spillover part.

Nothing above used the word neighbour. That word is about to do an enormous amount of work, so we define it next.

5. A spatial weight matrix, and why choosing one is a decision

5.1 From a yes-or-no adjacency matrix to weights

Spatial models need a single object that says who is next to whom. It starts as a yes-or-no table, the adjacency matrix $\Omega$, whose entry $\omega_{ij}$ is 1 when region $j$ counts as a neighbour of region $i$ and 0 otherwise. The diagonal is fixed at zero: you are not your own neighbour.

Raw counts of neighbours are awkward to compare, because a region with eight neighbours would mechanically get eight times the spatial pull of a region with one. Row-standardization fixes that:

$$w_{ij} = \frac{\omega_{ij}}{\sum_{j=1}^{n} \omega_{ij}}$$

In words, this says that each link’s weight is its share of that region’s total links, so every row of $W$ adds up to one. A region with 7 neighbours gives each of them $1/7$. A region with no neighbours at all keeps a row of zeros. In code, $\omega_{ij}$ is a cell of the sampled adjacency matrix and $w_{ij}$ is the corresponding cell of postw, the array of retained draws.

It is worth building one by hand once, on four regions arranged in a line:

Omega <- matrix(0, 4, 4)
Omega[1, 2] <- 1
Omega[2, c(1, 3)] <- 1
Omega[3, c(2, 4)] <- 1
Omega[4, 3] <- 1
W <- Omega / rowSums(Omega)
Omega
W

 [,1] [,2] [,3] [,4]
[1,] 0 1 0 0
[2,] 1 0 1 0
[3,] 0 1 0 1
[4,] 0 0 1 0
[,1] [,2] [,3] [,4]
[1,] 0.0 1.0 0.0 0.0
[2,] 0.5 0.0 0.5 0.0
[3,] 0.0 0.5 0.0 0.5
[4,] 0.0 0.0 1.0 0.0

Region 1 sits at the end of the line and has one neighbour, so that neighbour gets the full weight of 1. Region 2 sits in the middle with two neighbours, so each gets 0.5. Multiply any vector of regional outcomes by this $W$ and you get, for each region, the average outcome of its neighbours — nothing more exotic than that.

If the phrase spatial weights matrix is new, the post Exploratory Spatial Data Analysis in Python builds one from scratch with PySAL’s Queen contiguity and shows what Moran’s I does with it, and How to build a spatial weights matrix is the shorter, purely mechanical companion.

5.2 The model, and why coefficients are not effects

The general specification in the package is the spatial Durbin model, which nests the rest of the family:

$$y_t = \rho W y_t + X_t \beta_1 + W X_t \beta_2 + Z_t \beta_3 + \varepsilon_t$$

In words, a region’s outcome this year depends on its neighbours’ outcomes, on its own covariates, on its neighbours’ covariates, on covariates that never get spatially lagged, and on noise. Here $y_t$ is the year-$t$ slice of Y, $Z_t$ is the year-$t$ slice of Z, $\rho$ is stored in postr, the slopes in postb, and $W$ in postw.

The published application uses the simpler spatial autoregressive form, with no spatially lagged regressors:

$$y_t = \rho W y_t + Z_{t-1} \beta + \varepsilon_t$$

In words, labour-productivity growth in a region equals a fraction $\rho$ of its neighbours’ average growth, plus the effect of last year’s conditions, plus noise. The outcome $y_t$ is growth_gdp_pw; the lagged conditions in $Z_{t-1}$ are init_gdp_pw, loweduc, higheduc and an intercept; errors are normal with variance $\sigma^2$.

That $\rho W y_t$ term on the right-hand side is what makes the model interesting and what makes its coefficients hard to read. Solve for $y_t$ and you get the reduced form:

$$y_t = (I_n - \rho W)^{-1}(Z_{t-1} \beta + \varepsilon_t)$$

In words, once you untangle the simultaneity, every region’s outcome is a function of every region’s covariates and shocks, not just its own. The matrix $(I_n - \rho W)^{-1}$ that does the untangling is the spatial multiplier, and it has an illuminating expansion:

$$(I_n - \rho W)^{-1} = I_n + \rho W + \rho^2 W^2 + \rho^3 W^3 + \cdots$$

In words, the multiplier is the sum of all the bounces: yourself first, then your neighbours, then your neighbours’ neighbours, each round discounted by another power of $\rho$. With the value we will estimate, $\rho = 0.713$, the second round still carries weight 0.508 and the third 0.362. Nothing is local for long.

The immediate consequence is that $\beta$ is not a marginal effect. We will return to the right summaries — direct, indirect and total impacts — in Section 9, once we have a fitted model to attach them to.

5.3 Counting the unknowns

Now the uncomfortable arithmetic. If $W$ is known, the model has very few unknowns. If $W$ is not known, it has an enormous number:

$$\text{Number of unknowns} = 2 + (n^2 - n) + 2q_1 + q_2$$

In words, $\rho$ and $\sigma^2$ make two, every off-diagonal cell of the adjacency matrix is another, and then come the slope coefficients. With $n = 90$, no spatially lagged regressors ($q_1 = 0$) and four ordinary ones ($q_2 = 4$):

n <- 90; tt <- 19; q1 <- 0; q2 <- 4
c(unknown_links = n^2 - n,
total_unknowns = 2 + (n^2 - n) + 2 * q1 + q2,
observations = n * tt,
params_per_obs = (2 + (n^2 - n) + 2 * q1 + q2) / (n * tt))

 unknown_links total_unknowns observations params_per_obs
8010.000000 8016.000000 1710.000000 4.687719

	Assumed $W$	Estimated $W$
$\rho$ and $\sigma^2$	2	2
Slope coefficients	4	4
Off-diagonal cells of $\Omega$	0 (fixed)	8,010
Total unknowns	6	8,016
Observations ($nT$)	1,710	1,710
Observations per unknown	285	0.21

graph TD
subgraph CONV["Conventional spatial econometrics"]
C1("<b>W is ASSUMED</b><br/>contiguity, k-NN,<br/>or distance band")
C2("<b>Estimate</b><br/>rho, sigma2, 4 betas<br/><b>6 unknowns</b>")
end
subgraph EST["estimateW"]
E1("<b>Omega is UNKNOWN</b><br/>8,010 binary cells,<br/>diagonal fixed at zero")
E2("<b>Estimate jointly</b><br/>Omega, rho, sigma2, 4 betas<br/><b>8,016 unknowns</b>")
end
C1 --> C2
E1 --> E2
C2 --> Q("<b>Same data</b><br/>90 regions x 19 years<br/>= 1,710 observations")
E2 --> Q
style CONV fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style EST fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class C1,C2 blue
class E1,E2 orange
class Q anchor

Both columns end at the same 1,710 numbers. The left column asks those numbers to pin down 6 unknowns, which is comfortable. The right column asks them to pin down 8,016, which is impossible by likelihood alone.

A model with 4.7 parameters per observation cannot be estimated by likelihood alone. Everything below depends on the prior. That is not a caveat about the method; it is the method. The next two sections are about nothing else.

6. Bayesian machinery, from zero

6.1 Prior, likelihood, posterior

If you have never used a Bayesian method, here is the whole idea in one line:

$$p(\theta \mid \mathcal{D}) \propto p(\mathcal{D} \mid \theta) \, p(\theta)$$

In words, your belief after seeing the data is proportional to how well each candidate explains the data, times your belief before seeing it. The symbol $\theta$ stands for all the unknowns at once — here $\rho$, $\sigma^2$, the four slopes, and all 8,010 links — and $\mathcal{D}$ is the data.

Think of a weather app. It opens with the seasonal average for today’s date; that is $p(\theta)$, the prior. Then radar sweeps come in, and each one is more consistent with some forecasts than others; that is the likelihood $p(\mathcal{D} \mid \theta)$. What you actually read on the screen is the reconciliation of the two, the posterior. The app never shows you a single certain temperature, and neither will we: every output in this post is a distribution, summarised by its mean and standard deviation.

The reason a Bayesian approach is not optional here is Section 5.3. With 4.7 unknowns per observation, the likelihood alone does not have a unique peak to climb. The prior is what makes the problem well posed. Another Bayesian spatial MCMC application on this site, with a different purpose, is Bayesian Spatial Synthetic Control.

6.2 The likelihood of a spatial panel

The likelihood for the spatial autoregressive panel is the usual Gaussian fit term with one extra factor:

$$p(\mathcal{D} \mid \cdot) = \frac{|S|}{(2\pi\sigma^2)^{nT/2}} \exp\left[-\frac{1}{2\sigma^2}(SY - U\beta)’(SY - U\beta)\right]$$

In words, this is the familiar sum-of-squared-errors criterion, multiplied by a Jacobian $|S|$ that charges the model for the spatial feedback it uses. Here $S = I_T \otimes (I_n - \rho W)$ is the spatial filter applied to every time period, $Y$ is the stacked outcome, and $U$ collects the regressors — in our specification simply Z.

That determinant is not decoration. Without it, the model could raise $\rho$ for free: more spatial feedback would always fit better, and $\rho$ would run to its upper bound. The Jacobian is the price of borrowing strength from your neighbours, and it is also the reason the sampler needs a fast way to recompute a determinant, which we come to in Section 6.4.

6.3 Gibbs sampling, and the one-link Bernoulli step

Sampling 8,016 unknowns jointly is hopeless. Gibbs sampling makes it tractable by never doing anything jointly: it draws one block at a time, conditioning on the current values of everything else, and cycles.

For the links, the conditional distribution is remarkably simple. A link is either on or off, so once everything else is held fixed there are only two states to evaluate:

$$p(\omega_{ij} \mid \Omega_{-ij}, \cdot, \mathcal{D}) \sim \text{Bernoulli}\left(\frac{\bar{p}^{(1)}_{ij}}{\bar{p}^{(0)}_{ij} + \bar{p}^{(1)}_{ij}}\right)$$

In words: score the model with the link switched on, score it with the link switched off, normalise the two scores so they sum to one, and flip a coin weighted by the result. Here $\bar{p}^{(1)}_{ij}$ is the likelihood times the prior evaluated at $\omega_{ij} = 1$, and $\bar{p}^{(0)}_{ij}$ the same at $\omega_{ij} = 0$. Because this holds for any proper prior on a binary quantity, ordinary Gibbs sampling works for every one of the 8,010 cells.

That is genuinely all there is to it. The sampler is tuning a piano string by string: it never solves the whole instrument at once, it just fixes one string given how the others currently sound, and goes round again.

6.4 Griddy Gibbs for $\rho$, and why the sampler is fast

The slopes and the variance have textbook conditional distributions — Gaussian and inverse-Gamma respectively — so they are drawn directly. The spatial parameter $\rho$ does not:

$$p(\rho \mid \cdot, \mathcal{D}) \propto p(\rho) |S| \exp\left[-\frac{1}{2\sigma^2}(SY - U\beta)’(SY - U\beta)\right]$$

In words, this has no closed form you can sample from, because $\rho$ sits inside both the determinant and the quadratic form. The package therefore evaluates this density on a fine grid over the admissible range and samples from the grid as if it were a histogram — the griddy Gibbs step.

Griddy Gibbs. A trick for a parameter with no textbook conditional distribution. Score its density on a fine grid. Then sample from the grid in proportion to the scores. It is like measuring a hill’s height every ten metres and then picking a spot in proportion to the heights you measured. The consequence is visible in the trace plots later: $\rho$ takes visibly stepped values, because it can only ever land on one of the grid points.

The remaining problem is speed. Every one of the 8,010 link updates changes $W$, which changes $S$, which changes $|S|$ — and a determinant of a $90 \times 90$ matrix costs $O(n^3)$. At 200 iterations that would be 1.6 million such determinants. The package avoids nearly all of that cost by noticing that flipping one link is a rank-one change to $W$, so the Sherman-Morrison identity updates both $\log|S|$ and $(I_n - \rho W)^{-1}$ exactly, at trivial cost. This is why estimating 8,010 links takes minutes rather than weeks.

graph TD
S("<b>Initialize</b><br/>random Omega drawn<br/>from the prior, seed 571") --> A("<b>Step a</b><br/>sweep all 8,010 links<br/>in random row order,<br/>Bernoulli flip each")
A --> B("<b>Step b</b><br/>draw beta<br/>Gaussian conditional")
B --> C("<b>Step c</b><br/>draw sigma2<br/>inverse-Gamma conditional")
C --> D("<b>Step d</b><br/>draw rho<br/>griddy Gibbs on a grid")
D -->|"iterations 1 to 100:<br/>discard as burn-in"| A
D -->|"iterations 101 to 200:<br/>keep"| K("<b>Retained output</b><br/>postw, postb,<br/>posts, postr")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef key fill:#1f2b5e,stroke:#e8ecf2,stroke-width:3px,color:#e8ecf2
class S anchor
class A orange
class B,C blue
class D teal
class K key

Step (a) is where the 8,010 unknowns live, and it is the reason the Sherman-Morrison update matters. Steps (b) and (c) are ordinary Bayesian regression. Step (d) is the grid. Notice that the map changes between every draw of $\beta$ — which is exactly why Section 11 has to check that $\beta$ mixes at all.

7. The prior architecture

The prior on the links has two independent parts, and separating them is the design decision that makes the package usable:

$$\underline{p}_{ij} \propto \underline{\omega}_{ij} \, \underline{m}(k_i)$$

In words, the prior probability of a link is where you think links can be times how many links you think each region has. The first component $\underline{\omega}_{ij}$ is supplied through the W_prior argument, the second $\underline{m}(k_i)$ through nr_neighbors_prior. One says where, the other says how many.

7.1 The spatial component: where links can be

Each $\underline{\omega}_{ij}$ lives in $[0, 1]$. A value of 0.5 means “I have no idea, estimate it”. A value of 0 means “this link cannot exist”. A value of 1 means “this link definitely exists”. The package’s default is 0.5 everywhere off the diagonal.

The paper illustrates the possibilities on a stylised linear city of 30 regions, where region 1 borders region 2, region 2 borders 1 and 3, and so on. We reproduce all four cases:

NC <- 30
d_lin <- abs(outer(1:NC, 1:NC, "-"))
mk <- function(m) { diag(m) <- 0; m }
prior_cases <- list(
"A. All links unknown (the default)" = mk(matrix(0.5, NC, NC)),
"B. Known 10-nearest-neighbour W" = mk((d_lin <= 5) * 1),
"C. Distance band, unknown inside" = mk((d_lin <= 8) * 0.5),
"D. Mixed: forced, unknown, excluded" = mk(ifelse(d_lin == 1, 1,
ifelse(d_lin <= 8, 0.5, 0)))
)

Figure 2. Replication of the paper’s Figure 1. Dark navy = excluded, blue = unknown and estimated, orange = forced.

Case A is the default we will use: nothing is ruled in or out, and all 870 off-diagonal cells are sampled. Case C encodes partial knowledge — links beyond a distance band are ruled out, leaving 408 cells to estimate — which both injects information and cuts the computational cost, because cells fixed at 0 or 1 are never sampled at all.

Case B deserves a sentence of its own, because it is the whole conventional literature in one panel. Setting W_prior to a matrix of zeros and ones reduces sarw() to sar(). Standard spatial econometrics is not a different method from this one; it is this method with a degenerate prior, in which the neighbourhood map is asserted with probability one and never questioned. Everything that follows is about what happens when you relax that assertion.

7.2 The sparsity component: how many links there are

The second component governs how many neighbours each region has. Write $k_i = \sum_j \omega_{ij}$ for the number of neighbours of region $i$. The package’s default puts a beta-binomial prior on $k_i$, which in terms of the weight you actually supply is:

$$\underline{m}(k_i) = \Gamma(\underline{a}_\omega + k_i) \, \Gamma(n - 1 + \underline{b}_\omega - k_i)$$

In words, this is a flexible weight attached to each possible neighbour count, whose shape is controlled by two numbers $\underline{a}_\omega$ and $\underline{b}_\omega$. In code it is exactly bbinompdf(0:(n-1), nsize = n-1, a = a_pr, b = b_pr).

There are three natural choices, and comparing them is the most important thing in this section:

n <- 30; k <- 0:(n - 1); kbar <- 5
m_flat <- rep(1, n)
m_default <- bbinompdf(k, nsize = n - 1, a = 1, b = 1)
m_shrinkage <- bbinompdf(k, nsize = n - 1, a = 1, b = ((n - 1) - kbar) / kbar)
# the implied prior on k multiplies by the number of ways to choose k neighbours
p_k <- function(m) { p <- choose(n - 1, k) * m; p / sum(p) }
sapply(list(m_flat, m_default, m_shrinkage), function(m) sum(k * p_k(m)))

[1] 14.5 14.5 5.0

Figure 3. Replication of the paper’s Table 2 for n = 30. Top row: the weight you supply. Bottom row: the prior it implies. Orange line: prior mean degree.

The figure repays careful reading, because the two rows tell different stories. Look at the middle column: the weight $\underline{m}(k)$ you supply is violently U-shaped, with all its mass at 0 and 29 neighbours and essentially nothing in between — it looks like the most opinionated prior imaginable. But the bottom row shows the prior it actually implies on the neighbour count, and that is perfectly uniform across 0 to 29. The binomial coefficient $\binom{n-1}{k}$ does the reconciling: there are astronomically more ways to pick 15 neighbours out of 29 than to pick 0, and the U-shaped weight exactly cancels that combinatorial explosion. What you supply is not what you get, and only the bottom row is the prior you are actually asserting.

The left column carries the warning that matters most in practice. A flat weight, rep(1, n), is the natural thing to write if you want to “not impose anything”. But it implies a Binomial$(n-1, 0.5)$ prior on the neighbour count, centred at $(n-1)/2$. At our real sample size that is not 14.5 but 44.5 neighbours per region — a prior belief that every European region is directly connected to half of Europe.

Figure 4. At the real sample size, the two “non-informative” priors expect 44.5 neighbours per region. Only the shrinkage prior expects a sparse network.

Neither the flat nor the default prior is neutral at $n = 90$. Both put their mass on extremely dense networks, and with only 19 time periods that prior would dominate the likelihood rather than be updated by it. This is the concrete reason the published application uses shrinkage.

7.3 Anchoring the expected degree

To impose sparsity you fix $\underline{a}_\omega = 1$ and solve for the $\underline{b}_\omega$ that delivers a target expected number of neighbours $\underline{k}$:

$$\underline{b}_\omega = \frac{(n - 1) - \underline{k}}{\underline{k}}$$

In words, pick how many neighbours you think a region has, and this formula gives you the prior hyperparameter that encodes it. With $n = 90$ and $\underline{k} = 7$, that is $\underline{b}_\omega = (89 - 7)/7 = 11.714$, and a prior probability of $7/89 = 7.87\%$ for any individual link.

kbar <- 7
Wprior <- matrix(0.5, ncol = n, nrow = n)
diag(Wprior) <- 0
a_pr <- 1
b_pr <- ((n - 1) - kbar) / kbar
priorW_hierarchical <- bbinompdf(0:(n - 1), nsize = (n - 1), a = a_pr, b = b_pr)
AA <- W_priors(n = n, W_prior = Wprior,
nr_neighbors_prior = priorW_hierarchical)
c(b_prior = b_pr, prior_link_prob = kbar / (n - 1))

 b_prior prior_link_prob
11.71428571 0.07865169

If this construction feels familiar from a completely different setting, it should. It is the Ley and Steel model-size prior from Bayesian model averaging, applied to the number of links instead of the number of regressors. The post Bayesian Model Averaging, LASSO and WALS in R derives the same prior in a setting with no geography at all.

The value 7 is a researcher choice, and it should be reported as one. Section 15 returns to this; the honest practice is to report a sweep over $\underline{k}$, never a single value.

7.4 Priors for the remaining parameters

The other three blocks take conventional, deliberately diffuse priors, and the defaults are what the published application uses:

beta_priors(k = 4) # Gaussian, mean 0, variance 100 * I
sigma_priors() # inverse-Gamma, shape = rate = 0.001
str(rho_priors()) # 4-parameter Beta, plus the sampler's own settings

List of 11
$ rho_a_prior : num 1
$ rho_b_prior : num 1
$ griddy_n : num 60
$ rho_min : num 0
$ rho_max : num 1
$ init_rho_scale : num 1
$ use_griddy_gibbs: logi TRUE
$ use_pace_barry : logi TRUE
$ mh_tune_low : num 0.4
$ mh_tune_high : num 0.6
$ mh_tune_scale : num 0.1

rho_priors() carries more than a prior: griddy_n = 60 is the number of grid points the griddy-Gibbs step scores each sweep, which is why $\rho$ can only ever take 60 distinct values; use_pace_barry = TRUE selects the Barry-Pace approximation for the log-determinant grid; and setting use_griddy_gibbs = FALSE swaps in a self-tuning Metropolis-Hastings step instead.

Note the support of the prior on $\rho$: it is $(0, 1)$, not $(-1, 1)$. Positive spatial dependence is assumed, not tested. That is the one genuinely non-standard restriction in the whole setup, it exists to help identification when $W$ is unknown, and Section 14 spells out what it costs you.

8. Does the sampler actually work? A network we built ourselves

Section 5.3 left an uncomfortable arithmetic on the table: 8,016 unknowns against 1,710 observations. Section 7 explained how the prior makes that tractable. Neither of those is a demonstration that it works.

The only way to check an estimator is to run it on data whose answer you already know. Think of a driving test on a course where you placed the cones yourself: you can score the driver precisely because you know where everything was supposed to be. So before letting this loose on Europe, we build a network, generate data from it, hide it, and see what comes back.

The package supplies the generator:

set.seed(4242)
sim <- sim_dgp(n = 40, tt = 20, rho = 0.6,
beta3 = c(0.5, -1), sigma2 = 0.05,
n_neighbor = 4, intercept = TRUE)
W_true <- sim$W
Om_true <- (W_true > 0) * 1
c(n = 40, tt = 20, observations = 800, unknown_cells = 40^2 - 40,
true_rho = 0.6, neighbours_each = mean(rowSums(Om_true)))

Simulated panel: n = 40, T = 20, true rho = 0.6, true sigma2 = 0.05, 4 neighbours per unit by construction
Unknown off-diagonal cells: 1,560 against 800 observations

The ratio is deliberately unforgiving — 1,560 unknown links against 800 observations, nearly two parameters per data point, the same predicament as the European panel. We give the sampler the sparsity prior anchored at the truth ($\underline{k} = 4$), a proper budget of 2,000 iterations with 1,000 retained, and nothing else. In particular it never sees W_true.

prior_sim <- W_priors(n = 40,
W_prior = { m <- matrix(0.5, 40, 40); diag(m) <- 0; m },
nr_neighbors_prior = bbinompdf(0:39, nsize = 39, a = 1, b = (39 - 4) / 4))
res_sim <- sarw(Y = sim$Y, tt = 20, Z = sim$Z,
niter = 2000, nretain = 1000, W_prior = prior_sim)
pip_sim <- apply(res_sim$postw > 0, c(1, 2), mean)

Figure 18. Recovering a network we built ourselves. The estimated probabilities separate true links from non-links almost perfectly; the parameter posteriors are another story.

sim_recovery

 metric value truth q025 q975 covered
auc 0.97560 NA NA NA NA
precision_at_0.5 0.91430 NA NA NA NA
recall_at_0.5 0.60000 NA NA NA NA
accuracy_at_0.5 0.95320 NA NA NA NA
youden_cut 0.06300 NA NA NA NA
recall_at_youden 0.91250 NA NA NA NA
precision_at_youden 0.57030 NA NA NA NA
hamming_at_0.5 73.00000 NA NA NA NA
mean_degree 3.05100 4.00 NA NA NA
intercept 0.58420 0.50 0.5485 0.62180 FALSE
beta_x -1.03200 -1.00 -1.0530 -1.00900 FALSE
rho 0.52820 0.60 0.5085 0.54240 FALSE
sigma2 0.06368 0.05 0.0559 0.07251 FALSE

The network recovery is excellent. The area under the ROC curve is 0.976: rank all 1,560 candidate cells by their estimated probability and a randomly chosen true link outranks a randomly chosen non-link 98 times out of 100. At the natural 0.5 cutoff, 91.4 per cent of the links the model asserts are real ones, and overall 95.3 per cent of the 1,560 cells are classified correctly, for a Hamming distance of 73. The sampler genuinely finds the network.

It is, however, conservative: recall at the 0.5 cut is only 0.600, so it misses two of every five true links, and the estimated mean degree is 3.05 against a true 4. That is the sparsity prior doing exactly what we asked it to do — when the evidence for a link is weak, shrinkage wins. Lower the threshold to the value that best balances the two errors, 0.063, and recall jumps to 0.913 at the cost of precision falling to 0.570. There is no free lunch, only a dial, and where you set it should depend on whether false links or missing links are worse for your question.

Now the part that must not be glossed over: none of the four parameters’ 95 per cent credible intervals contain the truth. The spatial parameter is estimated at 0.528 with an interval of [0.509, 0.542] when the true value is 0.600. The error variance comes out at 0.064 against a true 0.050. The slopes are close — −1.032 against −1.000 — but still outside their intervals.

Two things are happening, and they compound. The sparsity prior recovers a slightly sparser network than the truth, and a model with fewer channels for spatial transmission compensates by attributing less to $\rho$ — that is a real bias, not noise, and it is the price of the regularisation that made the problem estimable at all. On top of that, the intervals are simply too narrow: an autocorrelated chain reports less uncertainty than it has earned.

The bottom-right panels of Figure 18 also show something worth recognising when you see it in your own output. The posterior of $\rho$ is not a smooth curve but a comb of narrow spikes, while the slopes and the variance are smooth. That is the griddy-Gibbs step from Section 6.4 made visible: $\rho$ is drawn from a 60-point grid, so it can only ever land on one of 60 values, and a kernel density of those draws looks like a picket fence. It is an artefact of the sampler’s design, not a sign that anything is broken — but if you need a finer resolution on $\rho$ than 1/60 of its range, raise griddy_n in rho_priors() or switch to the Metropolis-Hastings alternative.

So the honest summary is: this method locates the network well and should be trusted for questions about structure; its credible intervals for the structural parameters are overconfident at tutorial scale and should not be reported as though they were calibrated. One simulated dataset is not a Monte Carlo study — Krisztin and Piribauer (2023) run a proper one, across many designs, and find good recovery of both the network and the parameters. What our single draw establishes is the direction of the failure mode you should watch for, and it applies with at least equal force to the European results in the next section, which run on a tenth of this chain length with more than twice as many regions.

The cones were where we put them. Now the real course, where nobody knows where the cones are.

9. The headline: European regional growth with an estimated W

9.1 Running the sampler at the paper’s budget

Everything is now in place. The call is short, because all the modelling decisions live in the prior object:

set.seed(571)
res_sarw <- sarw(Y = Y, tt = tt, Z = Z,
niter = 200, nretain = 100,
W_prior = AA)

Two hundred iterations, of which the first hundred are discarded, is a deliberately tiny budget. The package authors chose it so their vignette runs in minutes, and we reproduce it exactly because reproducing the published table is the point of this section. It is not enough for careful inference, and Section 11 measures precisely how much it is not enough. Read the numbers below as a faithful replication, not as a final answer.

The run took 350 seconds on the machine used for this post — about 1.75 seconds per iteration, essentially all of it spent visiting the 8,010 candidate links. The returned object carries every retained draw of every unknown:

str(res_sarw[c("postb", "posts", "postr", "postw",
"post.direct", "post.indirect", "post.total")], max.level = 1)

 postb 4 x 100
posts 1 x 100
postr 1 x 100
postw 90 x 90 x 100
post.direct 4 x 100
post.indirect 4 x 100
post.total 4 x 100

Note the shape of postw: it is a $90 \times 90 \times 100$ array — a complete neighbourhood map for each of the 100 retained draws. We are not getting one estimated network, we are getting a posterior distribution over networks, and Section 10 is about reading it.

The package’s own quick overview is a plain summary():

summary(res_sarw)

 Z1 Z2 Z3 Z4
Min. :0.1533 Min. :-0.01972 Min. :-9.303e-05 Min. :0.0002104
1st Qu.:0.1688 1st Qu.:-0.01768 1st Qu.:-3.015e-06 1st Qu.:0.0003732
Median :0.1757 Median :-0.01687 Median : 3.122e-05 Median :0.0004329
Mean :0.1765 Mean :-0.01692 Mean : 3.570e-05 Mean :0.0004413
3rd Qu.:0.1839 3rd Qu.:-0.01603 3rd Qu.: 7.138e-05 3rd Qu.:0.0005137
Max. :0.2033 Max. :-0.01429 Max. : 1.619e-04 Max. :0.0007710
sigma2 rho
Min. :0.0004679 Min. :0.6780
1st Qu.:0.0005116 1st Qu.:0.6949
Median :0.0005260 Median :0.7119
Mean :0.0005285 Mean :0.7132
3rd Qu.:0.0005445 3rd Qu.:0.7288
Max. :0.0005862 Max. :0.7458

The columns are labelled Z1 to Z4 in the order the regressors were supplied — intercept, initial productivity, low education, high education. The posterior mean of $\rho$ is 0.7132, and every retained draw of $\rho$ lies between 0.678 and 0.746, so the data are quite sure that spatial dependence is strong. Note also that Z3, the low-education share, has a posterior that straddles zero: its minimum is negative and its maximum positive.

Figure 5. Everything the model reports, with credible intervals. The low-education interval crosses zero; nothing else does.

9.2 Table 3, reproduced and audited

Before the table, a word about what “reproduction” can even mean for an MCMC result. A Gibbs sampler compares likelihood ratios against uniform random draws, so a single flipped coin early in the chain sends the two runs down permanently different paths. Identical seed, identical package version, identical random-number generator and identical linear-algebra library reproduce a chain exactly; change any one of them and you should expect agreement only up to Monte Carlo noise. So the audit below reports not just the difference but how large that difference is relative to the Monte Carlo standard error of our own estimate.

table3_audit |>
transmute(quantity, paper_mean, our_mean = signif(our_mean, 5),
abs_diff = signif(abs_diff, 3), verdict)

Quantity	Paper	Ours	Difference	Verdict
intercept	0.17651	0.17651	1.0e-06	exact
log initial GVA per worker	−0.01692	−0.016922	2.0e-06	exact
share low education	0.00004	0.000036	4.3e-06	exact
share high education	0.00044	0.000441	1.3e-06	exact
$\rho$	0.71322	0.71322	3.4e-07	exact
$\sigma^2$	0.00053	0.000529	1.5e-06	exact
av. direct, log initial GVA	−0.01880	−0.018797	3.4e-06	exact
av. direct, share low education	0.00004	0.000040	3.9e-07	exact
av. direct, share high education	0.00049	0.000490	2.6e-07	exact
av. indirect, log initial GVA	−0.03972	−0.039723	3.1e-06	exact
av. indirect, share low education	0.00008	0.000084	4.3e-06	exact
av. indirect, share high education	0.00104	0.001036	3.5e-06	exact

All twelve published quantities reproduce to the full five decimal places the paper prints; every difference is below $10^{-5}$, which is the resolution of the printed table rather than a real discrepancy. That is the strongest form of replication available for a stochastic algorithm, and it happened because the seed, the package version (0.2.0), the random-number generator (Mersenne-Twister with inversion) and the reference BLAS all matched. If you run this on a machine with a different linear-algebra backend and get agreement only to two or three decimals, that is expected behaviour, not a bug — and it is why the script reports the difference in units of Monte Carlo standard error as well as in absolute terms.

Reading the estimates themselves: $\rho = 0.713$ with a posterior standard deviation of 0.016 says spatial dependence is both strong and precisely estimated. The coefficient on initial productivity is −0.0169, so conditional convergence survives once space is accounted for, though at less than half the magnitude the naive pooled regression in Section 4 reported (−0.0368). Tertiary education attainment enters positively at 0.00044, roughly four posterior standard deviations from zero. The low-education share is 0.000036 with a standard deviation of 0.000053 — indistinguishable from zero, and it should be reported that way rather than as a positive effect.

9.3 Impacts: direct, indirect and total

The slope coefficients above are not marginal effects, for the reason set out in Section 5.2: a change anywhere propagates everywhere through the multiplier. The right summaries collapse the full $n \times n$ effect matrix

$$\Pi_l = (I_n - \rho W)^{-1}(I_n \beta_{1,l} + W \beta_{2,l})$$

into two numbers. The average direct impact is the average of its diagonal, and the average indirect impact is the average of everything off the diagonal:

$$\text{Average direct impact} = \frac{\text{trace}(\Pi_l)}{n}$$

$$\text{Average indirect impact} = \frac{\mathbf{1}_n’(\Pi_l - \text{diag}(\Pi_l))\mathbf{1}_n}{n}$$

In words, the direct impact is what happens in the region where the change occurred, including feedback that leaves and comes back; the indirect impact is everything that happens in all the other regions; and the total is the sum. In code these are post.direct, post.indirect and post.total, each a matrix of retained draws with one row per variable.

Variable	Direct	Indirect	Total	Indirect ÷ direct
log initial GVA per worker	−0.01880	−0.03972	−0.05852	2.11
share low education	0.000040	0.000084	0.000124	2.13
share high education	0.00049	0.00104	0.00153	2.11

The spillover is roughly 2.1 times the own-region effect for every variable, and that repetition is not a coincidence. In a spatial autoregressive model with no spatially lagged regressors, $\Pi_l = (I_n - \rho W)^{-1}\beta_l$, so the split between diagonal and off-diagonal depends only on $\rho$ and $W$ — never on which variable you shock. Every explanatory variable in a SAR model is forced to share the same indirect-to-direct ratio. If you need those ratios to differ across variables, you need a Durbin specification, which is one of the models we fit in Section 12.

Substantively, the education result is the one with policy content. A region that raises its tertiary attainment share by one percentage point is associated with about 0.049 percentage points of extra annual productivity growth at home and 0.104 percentage points spread across the rest of the system — about twice as much growth outside its own borders as inside them. Whoever pays for the universities captures roughly a third of the measured return. That is the textbook case for funding higher education above the regional level, and it is invisible to a model that ignores spillovers: the pooled OLS in Section 4 reported a single education coefficient of 0.00026 and had nowhere to put the other two-thirds.

10. Reading the estimated network

Everything so far has treated the network as a nuisance to be integrated over. But postw holds 100 complete neighbourhood maps, and they are the most interesting output the model produces.

10.1 From draws to a posterior mean map

Two summaries do most of the work. The posterior mean weight averages $w_{ij}$ over draws. The posterior inclusion probability is the share of draws in which the link was switched on at all.

Posterior link probability $\Pr(\omega_{ij} = 1 \mid \mathcal{D})$. The share of retained draws in which region $i$ treated region $j$ as a neighbour. It lives between 0 and 1, and it is the evidence for that one specific connection. Think of re-drawing an office org chart a hundred times and counting how often two names end up side by side.

W_mean <- apply(res_sarw$postw, c(1, 2), mean) # posterior mean weights
W_pip <- apply(res_sarw$postw > 0, c(1, 2), mean) # inclusion probabilities
degree <- t(apply(res_sarw$postw > 0, 3,
function(m) rowSums(matrix(m, 90, 90)))) # draws x regions
c(mean_degree = mean(colMeans(degree)),
mean_pip = mean(W_pip[row(W_pip) != col(W_pip)]),
share_pip_above_half = mean(W_pip[row(W_pip) != col(W_pip)] > 0.5))

Estimated degree: mean 6.47 | median 6.9 | min 1 | max 16.11 (prior anchor k-bar = 7)
Posterior inclusion probability: mean 0.0727 | max 1 | share above 0.5: 0.41%

The average region ends up with 6.47 neighbours, slightly below the prior anchor of 7 — the data pulled the network a little sparser than the prior expected, which is reassuring: the prior is being updated, not merely obeyed. But the range is what matters. Some region gets a single neighbour and another gets 16, and that heterogeneity is information the prior did not contain, since the prior treats all 90 regions identically. The mean inclusion probability, 0.0727, is likewise just below the prior’s 0.0787.

Figure 9. Posterior probability that region i (row) treats region j (column) as a neighbour. Thin lines mark country borders.

The block structure along the diagonal is the headline. Regions of the same country light up together, and that is not something anyone told the model.

Figure 10. Estimated degree per region. Most regions sit near the prior anchor; a few sit far above it.

We can put a number on the clustering by comparing the share of estimated link mass that stays within a country against the share you would get if links were scattered at random, respecting only how many regions each country has:

share_same_country <- sum(edge$w_mean[edge$same_country]) / sum(edge$w_mean)
cs <- as.vector(table(regions$country))
rand_same_country <- sum(cs * (cs - 1)) / (90 * 89)

Link mass inside the same country: 35.6% (random benchmark 7.1%)
Link mass inside the same supranational group: 60.5% (random benchmark 30.4%)

Regions put five times more of their neighbourhood weight on compatriots than chance would predict, and twice as much on their supranational bloc. This is the paper’s qualitative claim — that regions of the same country are more strongly interconnected — turned into a testable ratio. It is worth pausing on how surprising it should be: nothing in the specification mentions countries. The model was given only growth rates, initial productivity and education shares, and it reconstructed national borders from co-movement alone.

10.2 The strongest links

Plotting all 8,010 possible links produces an unreadable hairball, so the paper shows only the strongest 10 per cent, and we follow that convention. The layout is deliberately not geographic: it is classical multidimensional scaling on the estimated link probabilities, so regions sit close together when the data say they are connected, not when they happen to be near each other.

top10 <- edge |> slice_max(w_mean, n = ceiling(0.10 * nrow(edge)))
head(arrange(top10, desc(w_mean)), 10)

 rank from to same_country w_mean pip
1 EL4 EL6 TRUE 1.0000 1.00
2 PL4 PL2 TRUE 1.0000 1.00
3 PL8 PL6 TRUE 1.0000 1.00
4 EL6 EL5 TRUE 0.9950 1.00
5 EL5 EL3 TRUE 0.9900 1.00
6 BG4 CZ0 FALSE 0.9833 1.00
7 PL2 PL4 TRUE 0.9550 1.00
8 HU2 HU3 TRUE 0.9508 1.00
9 HU1 HU3 TRUE 0.9135 0.99
10 PL7 PL8 TRUE 0.8900 0.92

Nine of the ten strongest links are within a country — Greek region to Greek region, Polish to Polish, Hungarian to Hungarian. A weight of 1.0000 with an inclusion probability of 1.00 means that region was given exactly one neighbour, the same one, in every single retained draw. The exception at rank 6 is instructive: BG4, south-western and south-central Bulgaria, links to CZ0, Czechia, with weight 0.98 in every retained draw — two countries that share no border and whose centroids sit 1,084 km apart. A contiguity matrix scores that link zero by construction, and a 7-nearest-neighbour matrix would not come close either.

Figure 11. The estimated network of direct links. Position reflects estimated connectedness, not location.

Figure 12. The multiplier network, on identical coordinates. Everything reaches everything, eventually.

10.3 Country-level chord diagrams

Aggregating the 90 × 90 matrix up to 26 × 26 by country makes the pattern legible at a glance. Following the paper, we keep the strongest 20 per cent of country-to-country flows and, importantly, we keep the diagonal: the claim being visualised is precisely that most of a country’s inflow comes from itself.

Figure 13. Country-aggregated W. The self-loops dominate.

Figure 14. Country-aggregated multiplier. Indirect reach spreads the flows out.

10.4 The multiplier network is not the W network

It is tempting to read the estimated $W$ as “the network”, but the object that actually governs how shocks travel is the multiplier from Section 5.2. They are very different:

c(density_W = mean(edge$w_mean > 0),
density_multiplier = mean(edge$mult > 1e-8),
rank_correlation = cor(rank(edge$w_mean), rank(edge$mult)))

Density: W 70.6% of off-diagonal cells non-zero, multiplier 92.9%; Spearman rank correlation of the two link rankings = 0.718

Two distinct things are going on here, and it is worth separating them carefully. The multiplier is denser than $W$ for a substantive reason: with $\rho = 0.713$, second-order neighbours still carry weight 0.508 and third-order 0.362, so almost every region eventually reaches almost every other one. That is the Neumann expansion made visible, and it is why the rumour analogy in the concept cards matters — the retelling never quite stops.

But the 70.6% figure for $W$ needs a caveat that is easy to miss. In any single draw a region has about 6.5 neighbours, which is 7.3 per cent of its possible partners. The 70.6 per cent counts cells that were switched on in at least one of the 100 retained draws. Averaging over an uncertain network makes the average look far denser than any network the model ever actually entertained. The posterior mean of $W$ is a summary of uncertainty, not a network you should interpret as though it were drawn once.

11. Convergence, and what 100 draws can and cannot support

The conditional posterior of the slope coefficients depends on the current state of $W$, which is being rewritten link by link inside every sweep. It is not obvious that anything mixes at all under that much churn, so this section checks.

Figure 6. Replication of the paper’s Figure 3. Note that rho moves in visible steps: griddy Gibbs samples it from a 60-point grid.

The chains fluctuate around a stable level with no trend or drift, which is the paper’s conclusion and ours. But “no visible trend” is a weak standard, so we ran two further independent chains of 400 iterations each, at different seeds:

res_long_a <- sarw(Y = Y, tt = tt, Z = Z, niter = 400, nretain = 200, W_prior = AA) # seed 20260731
res_long_b <- sarw(Y = Y, tt = tt, Z = Z, niter = 400, nretain = 200, W_prior = AA) # seed 20260732
coda::gelman.diag(coda::mcmc.list(scalar_mcmc(res_long_a), scalar_mcmc(res_long_b)),
autoburnin = FALSE, multivariate = FALSE)

Gelman-Rubin potential scale reduction factors (2 chains):
Point est. Upper C.I.
intercept 1.0168 1.0350
log initial GVA per worker 1.0228 1.0451
share low education 1.0073 1.0366
share high education 1.0136 1.0144
rho 0.9998 1.0047
sigma2 1.0114 1.0493

Every scale reduction factor is at or below 1.023, comfortably inside the conventional 1.05 threshold, and the upper confidence limits stay below 1.05 too. Two chains started from independent random networks agree on where the posterior is. That is genuine evidence that the sampler works despite the shifting topology.

Figure 7. Two chains, different seeds, same answers.

The point estimates barely move between budgets, which is the reassuring half of the story:

	Paper chain	Chain A	Chain B
iterations / retained	200 / 100	400 / 200	400 / 200
$\rho$ posterior mean	0.7132	0.7131	0.7140
$\rho$ posterior SD	0.01574	0.01678	0.01602
ESS($\rho$)	26.5	34.8	28.4
ESS(share high education)	122.2	84.3	130.9
runtime (seconds)	350	847	811

Now the uncomfortable half.

Figure 8. Running means, effective sample sizes and Geweke z-scores.

Three separate claims need to be kept apart, and conflating them is the most common way to over-read a result like this one.

Posterior means are trustworthy. They reproduce the published table exactly, they barely move when the chain is doubled, and the multi-chain diagnostic passes. If all you want is the point estimate of $\rho$ or of the indirect impact, 100 draws is enough.

Posterior standard deviations and intervals are not. The effective sample size for $\rho$ is 26.5 out of 100 retained draws — the chain is strongly autocorrelated, because $\rho$ is being resampled from a grid conditional on a network that only changes a little each sweep. An ESS of 26 supports a mean; it does not support a credible interval you would defend in a referee report. The Geweke z-score for $\rho$ in the paper chain is −3.74, outside the ±1.96 band, which formally rejects stationarity for that parameter at that budget. We report this rather than hide it: the paper’s own framing is that this is an illustration, and the diagnostic agrees.

Individual link rankings are the least reliable of all. With 100 retained draws, a posterior link probability can only take the values 0, 0.01, 0.02, … , 1. A link seen in 3 draws is indistinguishable from one seen in 1, and the boundary of the “strongest 10 per cent” in Figure 11 falls exactly in that noisy region. The links at the top of the ranking are solid — several appear in all 100 draws — but the marginal ones are close to arbitrary. If your research question is about a specific pair of regions rather than about the aggregate structure, run tens of thousands of iterations, not two hundred.

For contrast on how bad this can get, the Bayesian Spatial Synthetic Control post on this site reports an effective sample size of 3 for its spatial parameter. Small ESS is the normal state of affairs for tutorial-scale spatial MCMC, and the right response is to say so.

12. The rest of the family

Everything so far has used one model, the spatial autoregressive panel. The package covers the standard taxonomy, and the lattice below reads top to bottom as restrictions: each arrow switches off one channel of spatial dependence.

graph TD
SDM("<b>SDM</b><br/>rho W y and W X<br/>sdm() / sdmw()") -->|"beta2 = 0"| SAR("<b>SAR</b><br/>rho W y only<br/>sar() / sarw()")
SDM -->|"rho = 0"| SLX("<b>SLX</b><br/>W X only<br/>slx() / slxw()")
SDEM("<b>SDEM</b><br/>W X and spatial error<br/>sdem() / sdemw()") -->|"beta2 = 0"| SEM("<b>SEM</b><br/>spatial error only<br/>sem() / semw()")
SDEM -->|"error not spatial"| SLX
SAR -->|"rho = 0"| OLS("<b>Pooled OLS</b><br/>no spatial term at all")
SEM -->|"rho = 0"| OLS
SLX -->|"beta2 = 0"| OLS
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class SDM,SDEM orange
class SAR teal
class SEM,SLX blue
class OLS anchor

The box with the teal border is the model we estimated in Section 9. Every other box is one function call away, and every box has two doors — which is the package’s whole design:

Model	What carries the spatial lag	Fixed $W$	Estimated $W$
Spatial Durbin (SDM)	dependent variable and covariates	`sdm()`	`sdmw()`
Spatial Durbin error (SDEM)	covariates and errors	`sdem()`	`sdemw()`
Spatial autoregressive (SAR)	dependent variable only	`sar()`	`sarw()`
Spatial error (SEM)	errors only	`sem()`	`semw()`
Spatial lag of X (SLX)	covariates only	`slx()`	`slxw()`

The rule is simple: the plain name takes a neighbourhood map you supply; the w suffix estimates it. Everything else about the interface is identical. That doubling is why Section 7.1’s observation matters — the two columns are not two methods, they are the same method with different priors on $W$.

Three practical differences are worth knowing before you reach for them. slx() and slxw() have no $\rho$ at all, since nothing is spatially lagged on the left-hand side, so they return no postr. sem(), semw(), sdem() and sdemw() return no impact decomposition, because in an error-only specification a covariate has no spillover to decompose — the spatial structure is a nuisance parameter, not a transmission channel. And in the Durbin family X and Z must be disjoint: the package builds the regressor block as $[X, WX, Z]$, so a variable placed in both appears twice and the design becomes rank deficient. It does not warn you.

12.1 Fitting the family

Estimating $W$ is expensive and the cost grows fast with $n$, so the two halves of the tour use different data on purpose. The estimated-W models run on a small simulated panel where we know the answer; the exogenous-W models run on the real European panel, where they cost almost nothing because there is no network to sample.

For the estimated-W half we generate 25 units over 15 periods from a Durbin process — $\rho = 0.5$, covariates entering both directly and spatially lagged, three neighbours each — and fit all five specifications with the same sparsity prior:

sim_tax <- sim_dgp(n = 25, tt = 15, rho = 0.5, beta1 = 1, beta2 = -0.5,
beta3 = c(0.5, 1), sigma2 = 0.05, n_neighbor = 3, intercept = TRUE)
sarw(Y = Yt, tt = 15, Z = Zt, niter = 400, nretain = 200, W_prior = prior_tax)
sdmw(Y = Yt, tt = 15, X = Xt, Z = Zt, niter = 400, nretain = 200, W_prior = prior_tax)
# ... semw(), sdemw(), slxw() likewise

Model	Function	$\rho$ (true 0.5)	Mean degree (true 3)	Slopes	Impacts?	Runtime
SDEM	`sdemw()`	0.438	3.05	4	no	33 s
SDM	`sdmw()`	0.414	2.93	4	yes	32 s
SAR	`sarw()`	0.169	3.08	2	yes	29 s
SEM	`semw()`	0.112	2.87	2	no	31 s
SLX	`slxw()`	—	2.99	4	no	13 s

Every specification recovers the network density well — mean degree lands between 2.87 and 3.08 against a truth of 3 — but the spatial parameter is only recovered by the specifications that include the spatially lagged regressors. SDM and SDEM, which match the data-generating process, return 0.41 and 0.44 against a true 0.5. SAR and SEM, which omit the $WX$ terms, collapse to 0.17 and 0.11. That is classic omitted-variable bias wearing a spatial hat: with no channel for neighbours’ covariates to matter, the model cannot express the pattern and shrinks the only spatial parameter it has left. Getting the neighbourhood map right does not rescue you from getting the specification wrong.

Now the exogenous-W half, on the real panel with a fixed queen-contiguity matrix and a proper 2,000-iteration chain:

Model	Function	$W$	$\rho$	Slopes	Runtime
SAR	`sar()`	7-nearest-neighbour	0.719	4	6.5 s
SDEM	`sdem()`	queen	0.634	7	9.6 s
SEM	`sem()`	queen	0.632	4	9.2 s
SDM	`sdm()`	queen	0.624	7	6.7 s
SAR	`sar()`	queen	0.607	4	6.7 s
SLX	`slx()`	queen	—	7	0.6 s

Compare the runtimes across the two tables and the arithmetic of this method becomes concrete. Estimating $W$ on 25 units for 400 iterations takes 30 seconds; taking $W$ as given lets you fit 90 units for 2,000 iterations in 7. The unknown network is essentially the entire computational cost, and it is why the authors put the practical ceiling at around 300 regions.

Note also that under a fixed queen matrix, every specification lands on $\rho$ between 0.607 and 0.634 — the choice of model barely moves it, while the choice of map (queen 0.607 versus 7-nearest-neighbour 0.719) moves it a great deal. For this dataset the neighbourhood map is the more consequential modelling decision, which is the argument of the whole post in one line.

One thing this section deliberately does not do is rank these models. estimateW ships no marginal likelihood or information criterion that would let you compare across the family, and 400 iterations would not support such a comparison even if it did. This is a tour of the interface, not model selection. If you want Bayesian model comparison over the same family, Spatial Dynamic Panel Data Modeling in R does exactly that with blmpSDPD(); the Stata companions cross-sectional and panel walk through the whole family with likelihood-ratio tests instead.

13. Estimated W versus the maps we would have assumed

A subway map and a road map of the same city disagree about who is close. Neither is wrong; they answer different questions. This section asks which map the growth data prefer.

13.1 Building the maps we would otherwise have used

To make a fair comparison we need the real geography, which the nuts1growth panel does not ship. The NUTS-1 boundaries come from Eurostat’s GISCO service; the script fetches them once and caches the result so re-runs work offline.

u <- paste0("https://gisco-services.ec.europa.eu/distribution/v2/nuts/",
"geojson/NUTS_RG_20M_2021_3035_LEVL_1.geojson")
geo <- sf::st_read(u, quiet = TRUE) |>
filter(CNTR_CODE %in% target_countries, NUTS_ID != "FRY") # FRY = French overseas
geo <- geo[match(regions$nuts1, geo$NUTS_ID), ] # align to panel order
nrow(geo)

[1] 90

Filtering GISCO to the 26 countries in the panel and dropping the French overseas regions gives exactly the 90 we need, with no unmatched codes in either direction. Aligning the geometry to the panel’s row order is not optional: the sampler indexes $W$ by position, so a silently mis-sorted geometry would produce a plausible-looking but completely wrong map.

Queen contiguity — regions that share any boundary point — is then one call, and it immediately runs into a problem that is worth dwelling on:

touch <- sf::st_touches(geo, sparse = TRUE)
isolates <- geo$NUTS_ID[lengths(touch) == 0]

[geo] queen isolates repaired with nearest neighbour (10): CY0, EL4, ES7, FI2, FRM, IE0, ITG, MT0, PT2, PT3
[geo] queen: mean degree 3.8 | kNN-7: mean degree 7

Ten of the ninety regions have no queen neighbour at all. Cyprus and Malta are islands; so are the Greek Aegean islands, the Canaries, Åland, Corsica, Sicily-and-Sardinia, the Azores and Madeira; and Ireland shares no land border with any other region in the sample. A contiguity matrix leaves all ten with an empty row, which breaks row-standardization and effectively deletes them from the spatial model. The standard patch — give each isolate its nearest neighbour by centroid distance — is what we do, but notice what just happened: we made ten arbitrary modelling decisions before estimating anything, and they are invisible in the final table. That is the case for estimating $W$ in one paragraph.

Figure 15. The two neighbourhood maps we would otherwise have assumed, in the same ordering as Figure 9.

13.2 Does the estimated map look like any of them?

Treating the posterior link probability as a classifier of each assumed map gives a compact answer:

W_comparison_metrics

Comparator	Links	AUC of link probability	Jaccard with top 10%	Share of top links matched	Mean link distance
Same country	570	0.753	0.214	30.2%	407 km
Queen contiguity	342	0.698	0.137	17.2%	238 km
Same supranational group	2,438	0.693	0.184	62.8%	801 km
7-nearest neighbours	630	0.631	0.154	23.9%	360 km
Estimated, strongest 10%	801	—	1.000	100%	921 km
All region pairs	8,010	—	—	—	1,331 km

Sharing a country predicts the estimated network better than sharing a border does — an AUC of 0.753 against 0.698 — and both beat the nearest-neighbour map. That is the central empirical finding of the exercise, and it is a genuine ordering, not a rounding artefact: the same ranking shows up in the correlations, in the Jaccard overlaps, and in the share of top links matched.

Geography has not disappeared, though. The strongest estimated links average 921 km, against 1,331 km for a randomly chosen pair — a 31 per cent reduction. Distance still matters; it is just a much weaker organising principle than nationality. And relative to how common each kind of pair is, the enrichment is similar for both: 17.2 per cent of the top links share a border against a 4.3 per cent base rate (4.0 times), and 30.2 per cent are within a country against a 7.1 per cent base rate (4.2 times).

One refinement is worth stating, because it complicates the headline in an interesting way. The AUC integrates over the whole ranking of 8,010 pairs. If you instead look only at the links the model is most confident about — the 33 pairs with a posterior probability of at least 0.5 — the picture tilts back toward geography: 60.6 per cent of them share a border, which against a 4.3 per cent base rate is an enrichment of 14.2 times, while 75.8 per cent are within a country, an enrichment of 10.6 times. So the handful of links the data are certain about are disproportionately the geographic ones, even though across the full ranking nationality is the better predictor. Both statements are true, and they answer different questions: “what organises this network?” versus “what does the model know for sure?”. The interactive companion lets you slide that threshold and watch the two curves cross.

Figure 16. Link probability decays with distance, but the assumed maps recover only part of the estimated one.

Figure 17. The estimated map drawn over the real one. Orange dominates: most strong estimated links connect regions that do not touch.

Figure 17 makes the point better than any table. If the estimated network were essentially contiguity, the map would be a mesh of short teal arcs hugging borders. Instead it is dominated by long orange arcs — Bulgaria to Czechia, Iberia to the Baltic, Greece to Ireland. Whether those links represent trade, migration, shared exposure to European business cycles, or common institutional shocks is a question this model cannot answer; Section 15 returns to that.

13.3 One model, three maps, three answers

Comparing maps is only interesting if it changes the numbers a policymaker would use. So we fit the same SAR specification three times, changing nothing but $W$:

sar(Y = Y, tt = tt, W = W_queen, Z = Z, niter = 2000, nretain = 1000)
sar(Y = Y, tt = tt, W = W_knn, Z = Z, niter = 2000, nretain = 1000)
# versus the estimated-W fit from Section 9

	Estimated $W$	Queen contiguity	7-nearest-neighbour
How the map was chosen	from the data	shares a border	7 closest centroids
Mean neighbours per region	6.47	3.80	7.00
Regions with no neighbour	0	10 (repaired)	0
$\rho$	0.7132	0.6068	0.7186
Initial productivity — direct	−0.01880	−0.02076	−0.02000
— indirect	−0.03972	−0.02498	−0.04396
— total	−0.05852	−0.04574	−0.06396
High education — direct	0.000490	0.000299	0.000233
— indirect	0.001036	0.000361	0.000512
— total	0.001527	0.000659	0.000744
Indirect ÷ direct	2.11	1.20	2.20

Figure 19. The same SAR model, three neighbourhood maps. Signs agree; magnitudes do not.

Every qualitative conclusion survives the change of map. Conditional convergence is negative under all three, education spillovers are positive under all three, and the indirect effect exceeds the direct effect under all three. A referee asking “would your story change with a different $W$?” gets a reassuring answer.

Every quantitative conclusion does not. The total impact of tertiary education is 0.00153 under the estimated map, 0.00066 under contiguity and 0.00074 under nearest neighbours — the data-chosen map yields more than twice the total effect of either assumed one, and almost three times the spillover. The spillover-to-direct ratio is 2.11 under the estimated map but only 1.20 under contiguity, which is the difference between “most of the return leaks across borders” and “roughly half of it does”. If you are deciding how much of a regional university programme a national or European budget should co-fund, those two numbers imply materially different answers.

The pattern in $\rho$ is worth noticing too. Queen contiguity, the sparsest map at 3.8 neighbours per region, produces the lowest spatial parameter (0.607); the 7-nearest-neighbour map, at exactly 7, produces the highest (0.719); and the estimated map, at 6.47, lands essentially between them at 0.713. Sparser maps concentrate weight on fewer partners and leave less co-movement for $\rho$ to explain. That is a mechanical relationship, and it is precisely why choosing $W$ by eye and then interpreting $\rho$ as a structural quantity is uncomfortable.

14. What has to be true for any of this to be identified

Letting $W$ be unknown creates identification problems that do not arise when it is fixed, because $\rho$ and the network jointly determine how shocks travel: a strong parameter on a sparse network can mimic a weak parameter on a dense one. De Paula, Rasul and Souza (2025) give six sufficient conditions under which the parameters are globally identified. It is worth walking all six, because four of them hold automatically, one is genuinely tested, and one is assumed — and knowing which is which changes how you read every number above.

identification

#	What it requires	How `estimateW` handles it	Our check
I	No self-links, $[W]_{ii} = 0$	Diagonal of $\Omega$ fixed at zero	by construction — max $\lvert W_{ii} \rvert$ over draws is exactly 0
II	$\sum_j \lvert \rho [W]_{ij} \rvert < 1$ and $\rho < 1$	Row-stochastic $W$, $\rho$ support $(0,1)$	by construction — largest row sum over all draws is 0.7458
III	$\rho\beta_{1,q} + \beta_{2,q} \neq 0$	Not imposed	tested — no draw has $\lvert \rho\beta \rvert$ below $10^{-4}$; the mean is 0.0121
IV	At least one row of $W$ sums to a known constant	Row-standardization makes every row sum to 1 (or 0)	by construction — holds in every draw
V	Diagonal of $W^2$ not constant	Not imposed	tested — its cross-region standard deviation is 0.13 to 0.23 across draws, comfortably non-constant
VI	$\rho > 0$ and $[W]_{ij} \geq 0$	Non-negativity by construction; the sign of $\rho$ comes from the prior support	imposed, not tested — the prior support is $(0,1)$, so no draw could have been negative

Conditions I, II and IV are satisfied mechanically, which is a design feature of the package rather than a finding. Conditions III and V are real tests and both pass: the interaction between the spatial parameter and the slope is nowhere near zero, and the second-order neighbourhood structure is heterogeneous enough that regions are distinguishable from one another.

Condition VI is the one to be careful about, and it is the single most likely thing for a reader to misinterpret. The estimate $\rho = 0.713$ is evidence about the magnitude of positive spatial dependence. It is not evidence against negative spatial dependence, because negative values were excluded before the data were seen. The prior support is $(0, 1)$ by default, and every one of the 100 retained draws lies between 0.678 and 0.746 — inside a region the prior had already declared to be the only possibility. If you want to let the data speak to the sign, you can set rho_min = -1 in rho_priors(), but Krisztin and Piribauer recommend against doing that while also leaving $W$ completely unrestricted, because the sign of $\rho$ and the density of $W$ then trade off against each other in ways the data cannot separate. The honest options are: keep the sign restriction and say so, or relax it and impose more structure on $W$ instead.

15. Discussion

The Overview asked which European regions actually behave as each other’s neighbours for productivity growth, and whether it matters that we assume the map rather than estimate it. Both halves now have answers.

Which regions are neighbours. The estimated network is organised by nationality and by supranational bloc far more than by adjacency. Regions place 35.6 per cent of their neighbourhood weight on regions of the same country when chance would give 7.1 per cent, and 60.5 per cent within their supranational group against a 30.4 per cent benchmark. Ranking all 8,010 candidate pairs by their estimated link probability predicts “same country” with an area under the curve of 0.753, better than it predicts “shares a border” at 0.698. Nine of the ten strongest links join regions of the same country. None of this was supplied to the model, which saw only growth rates, initial productivity and education shares. Geography has not vanished — the strongest links average 921 km against 1,331 km for a random pair — but national and institutional boundaries organise co-movement more strongly than borders do. That corroborates Piribauer, Glocker and Krisztin’s finding at the NUTS-2 level, on independent data and at a different level of aggregation.

Whether it matters. For the sign of anything, no. For the size of the thing you would actually put in a policy memo, very much. The total impact of tertiary education attainment is more than twice as large under the estimated map as under either assumed map, and the share of the effect that leaks across regional borders rises from about 55 per cent under contiguity to about 68 per cent. A ministry deciding what fraction of a regional skills programme to co-fund from a national or European budget is reading a different number depending on a matrix somebody chose before the analysis started.

What this cannot claim. The estimated $W$ is a statistical object, not a map of economic relationships. A posterior probability of 1.0 on the link from south-western Bulgaria to Czechia means that residual co-movement in growth is better explained with that link than without it, given a prior that expected seven neighbours. It does not mean trade, commuting, foreign investment or supply chains. Two regions that respond identically to a common European shock will look connected to this model even if nothing flows between them, and with $T = 19$ annual observations there is no way to separate genuine transmission from shared exposure. Naming the mechanism requires data on the mechanism. The right way to use these estimates is as a description of the dependence structure and as a robustness benchmark for exogenous maps — not as evidence about channels.

Two further limits are worth stating plainly. The method does not scale indefinitely. Cost is driven by the $n^2 - n$ candidate links visited every sweep, and the authors put the practical ceiling near 300 regions; our own timings show the jump from 25 units to 90 dominating everything else in the script. Moving to NUTS-2, where $n \approx 280$, is feasible but sits at the edge, and the sensible response is to use a distance-band prior to rule out implausible links before sampling rather than to buy a bigger machine. And $\underline{k}$ is a researcher choice. We used 7 because the paper did, and the data moved the posterior only modestly, to 6.47. A prior that expects a denser network will deliver one. Prior sensitivity here is not a footnote to be waved at; it is the first thing a referee should ask about, and the honest practice is to report a sweep over $\underline{k}$ rather than a single value.

Finally, the caveat that governs everything above. The headline chain retains 100 draws with an effective sample size of 26.5 for $\rho$, and our own ground-truth simulation — run at ten times that budget — still produced credible intervals that failed to cover the truth for all four structural parameters. Point estimates from this machinery have earned trust. Interval estimates have not.

16. Summary and takeaways

Method. Treating the neighbourhood map as unknown adds 8,010 parameters to a 1,710-observation panel, taking the model from 285 observations per unknown to 0.21. That is only estimable because the beta-binomial sparsity prior anchored at $\underline{k} = 7$ replaces the flat prior’s implicit expectation of 44.5 neighbours per region. The prior is not a technicality here; it is what makes the question askable.
Validation. On a network we built ourselves — 40 units, 20 periods, four true neighbours each — the sampler separates true links from non-links with an area under the curve of 0.976 and classifies 95.3 per cent of the 1,560 candidate cells correctly. It is conservative, recovering 3.05 neighbours per unit against a true 4, and its parameter credible intervals did not cover the truth. Structure: trustworthy. Interval widths: not.
Data. For 90 European NUTS-1 regions over 2001–2019, $\rho = 0.71322$ with a posterior standard deviation of 0.01574, reproducing the published table to five decimals. The indirect impact of initial productivity, −0.0397, is 2.11 times the direct impact of −0.0188, and that ratio is identical for every variable because in a SAR model it depends only on $\rho$ and $W$.
Finding. The estimated network is organised by nationality more than by adjacency. Regions place 35.6 per cent of their neighbourhood weight on compatriots against a 7.1 per cent chance benchmark, and the posterior link probability predicts “same country” better (AUC 0.753) than it predicts “shares a border” (0.698). Nobody told the model about countries.
Limitation. The headline chain retains 100 draws, with an effective sample size of 26.5 for $\rho$ and a Geweke statistic that formally rejects stationarity. Posterior means are stable across chains and budgets; standard deviations, credible intervals and marginal link rankings are not. Report the first, hedge the second.
Next step. Re-run at NUTS-2, where $n \approx 280$ approaches the package’s practical ceiling of about 300, using a distance-band prior (case C of Figure 2) to rule out implausible links before sampling and cut the parameter count. Add country fixed effects to Z and see how much of the within-country clustering survives.

17. Exercises

Move the anchor. Re-run Section 9 with $\underline{k} = 3$, 7 and 15, changing only b_pr. Does the posterior mean degree track the anchor one-for-one, or does the likelihood pull back toward its own preference? Report $\rho$, the mean estimated degree and the indirect impact of initial productivity for each, in the three-column table you would show a referee who asked about prior sensitivity.
Give the model a hint. Using the cached GISCO centroids from Section 13, build a case-C prior: W_prior set to 0.5 for pairs within 500 km and 0 beyond. How many of the 8,010 cells are you no longer sampling, how much faster is the run, and does $\rho$ move? Then argue whether the speed-up is worth the assumption — remembering that the strongest single cross-country link found in Section 10 spans 1,084 km and would have been ruled out.
Break the SAR straitjacket. Section 9.3 showed every variable is forced to share the same indirect-to-direct ratio. Fit sdmw() with the two education shares entering both directly and spatially lagged, and check whether their ratios now differ. Then compute $\rho\beta_{1,q} + \beta_{2,q}$ for each variable from the draws and report the share of draws where it is within 0.001 of zero — this is identification condition III from Section 14, which a Durbin specification makes a live concern rather than an automatic pass.
Two chains, one question. Run sarw() twice at 5,000 iterations with different seeds. Compute coda::effectiveSize() and the Gelman-Rubin statistic for $\rho$, then compare the two posterior link-probability matrices cell by cell. How many of the “strongest 10 per cent” links from Figure 11 appear in both chains? That overlap is the honest reliability of any claim about a specific pair of regions.

18. References

Krisztin, T. and Piribauer, P. (2026). estimateW: a Bayesian R package for estimating spatial weight matrices, with an application to European regional growth. Springer, open access. — the paper replicated here.
Krisztin, T. and Piribauer, P. (2026). estimateW: Estimation of Spatial Weight Matrices. R package version 0.2.0, CRAN — the software, and the source of the nuts1growth data.
Krisztin, T. and Piribauer, P. (2023). A Bayesian approach for the estimation of weight matrices in spatial autoregressive models. Spatial Economic Analysis 18(1), 44–63 — the method: the Bernoulli conditional posterior, the Sherman-Morrison updates, and the Monte Carlo evidence that the estimator recovers the true network.
De Paula, Á., Rasul, I. and Souza, P. C. (2025). Identifying network ties from panel data: theory and an application to tax competition. Review of Economic Studies 92(4), 2691–2729 — the six identification conditions walked through in Section 14.
Piribauer, P., Glocker, C. and Krisztin, T. (2023). Beyond distance: the spatial relationships of European regional economic growth. Journal of Economic Dynamics and Control 155, 104735 — the NUTS-2 study this application follows, and the source of the country colouring and the network visualisations.
LeSage, J. P. and Pace, R. K. (2009). Introduction to Spatial Econometrics. Chapman and Hall/CRC — direct, indirect and total impacts; the four-parameter beta prior for $\rho$; the griddy-Gibbs scheme.
Elhorst, J. P. (2010). Applied spatial econometrics: raising the bar. Spatial Economic Analysis 5(1), 9–28 — the model taxonomy reproduced in Section 12.
Ley, E. and Steel, M. F. J. (2009). On the effect of prior assumptions in Bayesian model averaging with applications to growth regression. Journal of Applied Econometrics 24(4), 651–674 — the beta-binomial model-size prior, reused here on the number of links.
Ritter, C. and Tanner, M. A. (1992). Facilitating the Gibbs sampler: the Gibbs stopper and the griddy-Gibbs sampler. Journal of the American Statistical Association 87(419), 861–868 — the grid-based step used for $\rho$.
Barry, R. P. and Pace, R. K. (1999). Monte Carlo estimates of the log determinant of large sparse matrices. Linear Algebra and its Applications 289(1–3), 41–54 — the log-determinant approximation enabled by use_pace_barry.
Bramoullé, Y., Djebbari, H. and Fortin, B. (2009). Identification of peer effects through social networks. Journal of Econometrics 150(1), 41–55 — the network-asymmetry condition tested in Section 14.
Gu, Z., Gu, L., Eils, R., Schlesner, M. and Brors, B. (2014). circlize implements and enhances circular visualization in R. Bioinformatics 30(19), 2811–2812 — the chord diagrams.
Pebesma, E. (2018). Simple Features for R: standardized support for spatial vector data. The R Journal 10(1), 439–446 — the sf package used for the NUTS-1 geometry and the contiguity benchmark.
Eurostat regional economic accounts and educational attainment statistics — the underlying data behind nuts1growth.
Eurostat GISCO NUTS 2021 boundaries — the NUTS-1 geometry used in Section 13, cached in the replication bundle.

Acknowledgements

Covariates in Difference-in-Differences: The LaLonde Test in Python

Thu, 16 Jul 2026 00:00:00 +0000

Abstract

Forty years ago Robert LaLonde delivered one of the most influential blows of the credibility revolution: he took a job-training experiment with a known treatment effect, threw away the randomized control group, and showed that a gauntlet of respectable econometric estimators could not recover the answer. This tutorial reproduces — in Python, and in the spirit of a recent essay by Scott Cunningham — a modern version of that test aimed squarely at covariates in difference-in-differences (DiD). Working with the Dehejia-Wahba subsample (185 National Supported Work trainees plus 15,992 Current Population Survey controls, a two-period panel of 1975 and 1978 earnings), the estimand is the average treatment effect on the treated (ATT), and the ground truth is the experimental benchmark of \$1,794. We estimate the ATT eight ways using pyfixest for the regressions and hand-coded inverse-propensity-weighting (Abadie 2005) and doubly-robust (Sant’Anna-Zhao 2020) estimators, cross-checked against the diff-diff package. The result is stark and dollar-accurate. Three specifications that keep covariates out of the counterfactual trend — no covariates, additive covariates, and covariate-by-treatment interactions — all return the naive \$3,621, roughly twice the truth. The instant covariates are allowed to bend the control group’s trend, via X × post (\$1,711) or first-difference saturation (\$1,770, numerically identical to the Heckman-Ichimura-Todd estimator), the estimate snaps to the benchmark; the propensity-based estimators land nearby at \$1,861 and \$1,993. The lesson is that covariates in DiD are not a robustness knob to be twisted for reassurance — they perform a specific job, satisfying conditional parallel trends and relaxing constant treatment effects, and the most common applied specification of all, two-way fixed effects with additive controls, does not do that job.

Overview

Imagine you already know the right answer. A job-training program raised the earnings of disadvantaged workers by about \$1,794 — we know this because the program was evaluated with a randomized controlled trial, and randomization makes the treated and control groups exchangeable. Now throw the experimental control group away and replace it with a large survey of ordinary Americans. Can a difference-in-differences estimator, armed with the usual baseline covariates, still find the \$1,794? And does it matter how you feed those covariates to the model?

This is the LaLonde test, named for LaLonde (1986), who used exactly this setup to show that unbiased-in-principle estimators can be badly biased in application. The exercise here follows a recent post by Scott Cunningham on his Causal Inference: The Mixtape Substack, translating his Stata and R analysis into Python and adding a package cross-check. If you are new to difference-in-differences, start with the companion tutorial, Introduction to Difference-in-Differences in Python; this post is the advanced sequel that asks a sharper question about covariates.

The punchline, which we will earn spec by spec, is a distinction that most applied work blurs. A covariate can enter a DiD in three fundamentally different places: in the level of the outcome (additive controls), in the treatment effect (covariate-by-treatment interactions), or in the trend (covariate-by-time interactions). Only the last one addresses the reason the naive estimate is wrong. Because our control group is a representative slice of America and our treated group is a set of disadvantaged trainees, the two groups are wildly imbalanced; and if groups with different characteristics are on different earnings trajectories, then that imbalance mechanically breaks the parallel-trends assumption. Fixing it requires modeling those trajectories — putting covariates in the trend — not simply adding them to the regression.

Learning objectives:

Understand why a covariate rescues a DiD estimate only when it enters the control group’s counterfactual trend, not the level or the treatment effect.
Implement eight covariate specifications in pyfixest — from the naive two-way fixed effects model to a fully saturated first-difference regression — and recover the ATT by g-computation.
Hand-code the Abadie (2005) inverse-propensity-weighting and Sant’Anna-Zhao (2020) doubly-robust DiD estimators, and cross-check every number against the diff-diff package.
Judge all eight estimates against the \$1,794 experimental benchmark and diagnose why the ubiquitous additive-controls specification stays stuck at \$3,621.

Key concepts at a glance

The post leans on a small vocabulary repeatedly. Each concept below has three parts. The definition is always visible. The example and analogy sit behind clickable cards: open them when you need them, leave them collapsed for a quick scan. If a later section mentions “conditional parallel trends” or “outcome regression” and the term feels slippery, this is the section to re-read.

1. The LaLonde test. Benchmark an observational estimator against a known experimental ATT by replacing the RCT control group with survey controls and checking whether the method still recovers the truth.

Example

Here the known ATT is \$1,794 (Dehejia-Wahba). A naive DiD with CPS controls returns \$3,621 — the test exposes the bias, just as it did in 1986.

Analogy

Grading a new thermometer against a calibrated one. If it reads 40°C in boiling water, you have learned something about the thermometer, not the water.

2. Estimand: ATT vs ATE. The ATT is the effect on the treated units; the ATE is the effect on everyone. Under randomization they coincide. Replacing the experimental control with a survey control changes who the comparison represents but leaves the treated group — and therefore the ATT — intact.

Example

The 185 trainees stay in every specification, so the target stays \$1,794 (the ATT) even though the CPS controls make the sample look nothing like the original experiment.

Analogy

Measuring how much your team improved after coaching. Swapping which rival you compare against does not change your team’s gain.

3. Conditional parallel trends. DiD identifies the ATT if treated and control outcomes would have moved in parallel absent treatment. When groups are imbalanced, that parallelism is only credible given covariates $X$ — parallel trends conditional on $X$.

Example

Trainees and CPS controls do not share a raw trend, but workers with the same age, schooling and earnings history plausibly do — so we condition on $X$.

Analogy

Two runners on different hills. They only “move in parallel” once you account for the slope each is standing on.

4. Covariate imbalance. Systematic differences in $X$ between treated and control. Harmless on its own — but combined with covariate-specific trends, it becomes the bias term that breaks parallel trends.

Example

The CPS controls differ from trainees by a standardized mean difference of +2.3 on race and −1.6 on prior earnings; the randomized controls differ by essentially zero.

Analogy

Comparing marathon times of teenagers and retirees. The age gap only distorts things because age also drives the trend.

5. Trend vs level vs effect. The crux. A covariate in the level (additive) or the treatment effect ($X \times D$) leaves the counterfactual trend untouched and is inert. A covariate in the trend ($X \times \text{post}$) bends the control’s counterfactual and corrects the bias.

Example

Additive $X$ and $X \times \text{treatment}$ both return \$3,621; $X \times \text{post}$ returns \$1,711. Same covariates, three placements, two very different answers.

Analogy

Where you attach the corrective lens matters. Over the eye it fixes your vision; in your pocket it does nothing.

6. Outcome regression / HIT (1997). First-difference the outcome, fit a flexible model on the controls, and impute each treated unit’s counterfactual change. Because the outcome is now a change, a covariate’s coefficient is its effect on the trend — for free.

Example

Fitting the first-differenced outcome on controls only, then imputing to the treated, gives \$1,770 — identical to the fully saturated regression on the whole sample.

Analogy

Learn how “normal” workers’ pay evolved, then ask how much more the trainees gained than their statistical twins would have.

7. Doubly robust DiD (Sant’Anna-Zhao 2020). Combine an outcome-regression model of the trend with an inverse-propensity reweighting of the controls. Consistent if either model is correctly specified — two shots at the truth.

Example

The doubly-robust estimate is \$1,993; the package’s Callaway-Sant’Anna implementation lands within \$14 of it at \$1,979.

Analogy

Two independent safety nets. You fall through only if both fail.

The decision that organizes the entire post is a single question — where does the covariate enter? — and its answer sorts every estimator into “inert” or “corrected”:

graph TD
Q{"Where does covariate X<br/>enter the DiD?"}
Q -->|"Nowhere (baseline)"| N("Spec 0: naive TWFE")
Q -->|"In the LEVEL<br/>(additive)"| L("Spec A: Additive X")
Q -->|"In the EFFECT<br/>(X × treatment)"| E("Spec BT: X × treatment")
Q -->|"In the TREND<br/>(X × post)"| T("Spec B: X × post")
Q -->|"Trend + effect<br/>(saturated FD)"| S("Spec C = HIT 1997")
Q -->|"Propensity<br/>reweighting"| P("IPW / doubly robust")
N --> INERT("Counterfactual trend untouched<br/>ATT stays ~3,621 (inert)")
L --> INERT
E --> INERT
T --> FIX("Bends the control's<br/>counterfactual trend<br/>ATT snaps to ~1,794 (corrected)")
S --> FIX
P --> FIX
FIX -.->|recovers| BENCH("RCT benchmark<br/>ATT = 1,794")
classDef sty_Q fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class Q sty_Q
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class N,L,E,INERT gray
class T,S,P,FIX blue
class BENCH anchor

Setup and imports

We lean on three packages. pyfixest gives us fast, formula-based OLS with robust standard errors — the workhorse for the six regression specifications. causaldata ships the canonical LaLonde data, so nothing needs to be downloaded by hand. And diff-diff provides a scikit-learn-style DiD toolkit that we use to independently validate every hand-coded number. We fix the seed to 90210 to match the reference exactly, so the bootstrap standard errors are reproducible.

import numpy as np
import pandas as pd
import statsmodels.api as sm
import pyfixest as pf
RANDOM_SEED = 90210 # reproducible cluster bootstrap
N_BOOT = 199 # bootstrap replications
BENCHMARK = 1794.0 # experimental ATT (Dehejia-Wahba)
# Scott's canonical LaLonde covariate set (age cube; u74 = 1[re74 == 0])
XVARS = ["age", "agesq", "agecube", "educ", "educsq",
"marr", "nodegree", "black", "hisp", "re74", "u74"]

Every covariate here is a time-invariant baseline characteristic: age and its square and cube, education and its square, indicators for marriage, no high-school degree, race and ethnicity, 1974 earnings, and an unemployment flag for 1974. None of them changes between our two periods. Keep that fact in your pocket — it is the reason the additive specification will turn out to be completely inert.

Data: the LaLonde-DW non-experimental panel

The construction is where the fidelity of the whole exercise is won or lost, so it is worth being explicit. The nsw_mixtape dataset is already the Dehejia-Wahba subsample — 185 treated trainees and 260 experimental controls. For the non-experimental analysis we keep the 185 treated and discard the experimental controls, replacing them with the 15,992 CPS controls from cps_mixtape. The experimental controls are set aside for one job only: computing the benchmark. We then reshape the wide earnings columns (re75, re78) into a two-period panel with post = 1 in 1978.

from causaldata import nsw_mixtape, cps_mixtape
nsw = nsw_mixtape.load_pandas().data # 185 treated + 260 exp. controls
cps = cps_mixtape.load_pandas().data # 15,992 CPS controls (treat = 0)
# Non-experimental sample = NSW treated only + all CPS controls
wide = pd.concat([nsw[nsw.treat == 1], cps], ignore_index=True)
wide["ever_treated"] = wide["treat"].astype(int)
wide["agesq"] = wide.age ** 2
wide["agecube"] = wide.age ** 3
wide["educsq"] = wide.educ ** 2
wide["u74"] = (wide.re74 == 0).astype(float)
wide["dy"] = wide.re78 - wide.re75 # first difference (pre 1975, post 1978)
# Long two-period panel: post = 0 -> re75, post = 1 -> re78
pre = wide.assign(re=wide.re75, post=0.0)
post = wide.assign(re=wide.re78, post=1.0)
panel = pd.concat([pre, post], ignore_index=True)
print(pd.crosstab(panel.ever_treated, panel.post))

post 0.0 1.0
ever_treated
0 15992 15992
1 185 185

The cell counts confirm a clean, balanced two-period panel: 185 treated and 15,992 controls observed in both 1975 and 1978. Because randomization is gone, the identifying assumption is no longer plain parallel trends but conditional parallel trends — trainees and CPS controls with the same covariates are assumed to have moved in parallel absent treatment. Whether that assumption is plausible depends entirely on how different the two groups actually are, which is the next thing to look at.

The covariate imbalance problem

Before running a single regression, we should ask how comparable the groups are. The standard diagnostic is the standardized mean difference (SMD): the gap in a covariate’s mean between treated and control, divided by the pooled standard deviation. A rule of thumb flags anything beyond 0.1 as imbalanced and beyond 0.25 as severe. We compute it for the trainees against the CPS controls, and — as a reference point — against the discarded experimental controls.

def smd(a, b, col):
s = np.sqrt((a[col].var() + b[col].var()) / 2)
return 0.0 if s == 0 else (a[col].mean() - b[col].mean()) / s
treated = wide[wide.ever_treated == 1]
cps = wide[wide.ever_treated == 0]
for col in ["black", "re74", "married", "educ"]:
print(f"{col:8s} vs CPS = {smd(treated, cps, col):+.2f}")

The picture is dramatic. Against the CPS controls (orange), the trainees differ by a standardized +2.3 on race, −1.6 on 1974 and 1975 earnings, and −1.3 on marriage — imbalances many times the “severe” threshold. Against the randomized experimental controls (blue), every difference collapses to essentially zero. This is the entire problem in one figure: the CPS is a representative sample of America, so its members are older, better educated, more often married, and far richer than a set of disadvantaged trainees. Imbalance this extreme is exactly the condition under which conditioning on covariates stops being optional.

Raw earnings trends

Imbalance in levels is only half the story. It becomes a bias only if the imbalanced groups are also on different trends. So let us plot mean earnings for each group in 1974, 1975, and 1978.

The CPS controls (black) sit far above everyone else — about \$14,017 in 1974 against the trainees’ \$2,096 — and drift gently upward. The trainees (orange) and the experimental controls (blue) start together near the bottom and rise in tandem into 1978. A DiD that uses the CPS as-is is implicitly assuming the trainees, absent training, would have followed the CPS’s flat high-earning path. That is not credible: low-earning workers and high-earning workers are on different earnings trajectories, and the trainees look nothing like the CPS. This is precisely where covariates must do work — not by shifting levels, but by modeling those divergent trends.

A two-minute DiD refresher

If the details of difference-in-differences are hazy, the introductory tutorial covers them properly; here is just enough to fix notation. The two-by-two DiD estimator of the ATT is the difference of two differences,

$$\widehat{\text{ATT}} = (\bar Y_{\text{treated,post}} - \bar Y_{\text{treated,pre}}) - (\bar Y_{\text{control,post}} - \bar Y_{\text{control,pre}}),$$

and, as Cunningham emphasizes, four regressions compute this identical number: a saturated regression with a treatment dummy, a post dummy and their interaction; two-way fixed effects with a post-by-treatment interaction; a first-difference regression on the treatment dummy; and an across-group regression on the post dummy. We use the saturated form because it is the one that lets us add time-invariant covariates in the various ways we want to compare. Throughout, post:ever_treated is the interaction whose coefficient is the DiD estimate of the ATT.

Spec 0 — Naive TWFE (no covariates)

Start with the estimator that ignores covariates entirely. This is the number to beat.

def att(fit):
td = fit.tidy()
key = [k for k in td.index if "post" in k.lower() and "ever_treated" in k.lower()][0]
return td.loc[key, "Estimate"], td.loc[key, "Std. Error"]
s0 = pf.feols("re ~ post * ever_treated", data=panel, vcov="HC1")
print("Spec 0 (naive) ATT = %.0f (SE %.0f)" % att(s0))

Spec 0 (naive) ATT = 3621 (SE 632)

The naive DiD returns \$3,621 — positive, statistically significant, and about twice the true \$1,794. On its own it looks like a perfectly respectable result; nothing about the output warns you that it is off by a factor of two. That is the whole danger LaLonde exposed: a plausible, precise, and wrong estimate. Everything that follows is an attempt to fix it with covariates.

Spec A — Additive X (covariates in the level)

The reflexive fix — and by far the most common specification in applied panel work — is to throw the covariates into the regression additively. This is two-way fixed effects with controls.

XF = " + ".join(XVARS)
sA = pf.feols(f"re ~ post * ever_treated + {XF}", data=panel, vcov="HC1")
print("Spec A (additive X) ATT = %.0f (SE %.0f)" % att(sA))

Spec A (additive X) ATT = 3621 (SE 672)

Nothing moved. Spec A returns \$3,621, identical to the naive estimate to the dollar. This is not a coincidence or a rounding accident — it is mechanical. Because the covariates are time-invariant, the within transformation that defines two-way fixed effects sweeps them out entirely; the DiD coefficient never depended on them in the first place. Adding time-invariant controls to a TWFE DiD is, quite literally, doing nothing to the point estimate. If you have ever added baseline controls to a DiD and reported that “the estimate is robust to covariates,” this is worth sitting with.

Spec BT — X × treatment (covariates in the effect)

Perhaps the additive form was too rigid because it forces the treatment effect to be the same for everyone. So let the effect vary with covariates: interact each covariate with the treatment switch $T = \text{post} \times D$, and recover the ATT by g-computation — predict each unit’s outcome with the switch on and off, and average the difference over the treated-post cell.

d = panel.assign(T=panel.post * panel.ever_treated)
T_ints = " + ".join(f"T:{x}" for x in XVARS)
fit = pf.feols(f"re ~ post + ever_treated + T + {XF} + {T_ints}", data=d, vcov="HC1")
tau = fit.predict(newdata=d.assign(T=1.0)) - fit.predict(newdata=d.assign(T=0.0))
mask = (d.ever_treated == 1) & (d.post == 1)
print("Spec BT (X x treatment) ATT = %.0f" % tau[mask.values].mean())

Spec BT (X x treatment) ATT = 3621

Inert again — \$3,621, to the dollar. Allowing heterogeneous treatment effects saturates the model in levels, which relaxes the constant-effects assumption but does absolutely nothing about conditional parallel trends. The problem was never that the treatment effect varies; it is that the control group’s counterfactual trend is mismodeled. Interacting covariates with treatment is answering a question nobody asked. We have now tried covariates in the level and in the effect, and both left the estimate exactly where it started.

Spec B — X × post (covariates in the trend)

Now put the covariates where the problem actually lives: the trend. Interact each covariate with the post indicator, so that workers with different characteristics are allowed to be on different earnings trajectories over time.

post_ints = " + ".join(f"post:{x}" for x in XVARS)
sB = pf.feols(f"re ~ post * ever_treated + {XF} + {post_ints}", data=panel, vcov="HC1")
print("Spec B (X x post) ATT = %.0f (SE %.0f)" % att(sB))

Spec B (X x post) ATT = 1711 (SE 704)

The estimate collapses from \$3,621 to \$1,711 — a \$1,910 move that lands within \$83 of the \$1,794 benchmark. Same covariates as Spec A; the only change is that they now multiply post instead of sitting in the level. That single change lets the model say “high-earning, well-educated workers were on a steeper path than low-earning ones,” which is what the counterfactual for the trainees actually requires. This is the pivot of the entire post: covariates rescue the estimate, but only from the trend.

Spec C — Saturated first differences = HIT (1997)

Spec B corrected the trend but still imposes a constant treatment effect. We can do both jobs at once. First-difference the outcome, then regress the change $\Delta y = y_{1978} - y_{1975}$ on the covariates, the treatment indicator, and their full interaction, and recover the ATT by g-computation over the treated.

D_ints = " + ".join(f"ever_treated:{x}" for x in XVARS)
sC = pf.feols(f"dy ~ {XF} + {D_ints} + ever_treated", data=wide, vcov="HC1")
tau = (sC.predict(newdata=wide.assign(ever_treated=1.0))
- sC.predict(newdata=wide.assign(ever_treated=0.0)))
print("Spec C (saturated FD) ATT = %.0f" % tau[wide.ever_treated.values == 1].mean())

Spec C (saturated FD) ATT = 1770

\$1,770 — within \$24 of the truth. The subtlety worth savoring: because the outcome is now a change, a covariate’s own coefficient is no longer a level effect but its effect on the trend. So the first-difference regression does two things simultaneously — the covariate main effects bend the control’s counterfactual trend (what Spec B did), and the treatment interactions relax the constant-effect assumption (what Spec BT tried to do). One regression, both corrections. The pure-levels saturation of Spec BT never touched the trend, which is why it sat at \$3,621 while this lands at the benchmark.

HIT (1997) by hand

Here is the result that Cunningham rightly calls “not terribly intuitive.” The fully saturated first-difference regression above is numerically identical to a multi-step outcome-regression procedure — Heckman, Ichimura and Todd (1997) — that only ever fits a model on the controls. Fit $\Delta y$ on covariates using the control group alone, impute the counterfactual change for each treated unit, and average the treated units’ actual-minus-imputed gains.

import statsmodels.formula.api as smf
mH = smf.ols(f"dy ~ {XF}", data=wide[wide.ever_treated == 0]).fit() # controls only
hit = (wide.dy - mH.predict(wide))[wide.ever_treated == 1].mean() # impute to treated
print("HIT by hand ATT = %.0f" % hit)

HIT by hand ATT = 1770

Identical to Spec C — \$1,770. A regression that touches only the controls and then imputes to the treated gives exactly what a saturated regression on the whole sample gives. This is the Oaxaca-Blinder logic applied to a change, and it clarifies what “outcome regression” means in the DiD context: learn how ordinary workers’ earnings evolved, then ask how much more each trainee gained than a statistical twin would have. The numerical equivalence is a small piece of econometric magic, and it is reassuring that two conceptually different recipes agree to the dollar.

IPW — Abadie (2005) by hand

Regression adjustment models the outcome. The alternative is to model treatment — estimate each unit’s probability of being a trainee and reweight the controls to resemble the treated. Abadie (2005) gives the propensity-weighted DiD estimator of the ATT with weights

$$w_i = \frac{D_i - \hat p(X_i)}{1 - \hat p(X_i)} \cdot \frac{1}{\Pr(D = 1)},$$

applied to the first-differenced outcome.

Xc = sm.add_constant(wide[XVARS])
wide["phat"] = sm.Logit(wide.ever_treated, Xc).fit(disp=0).predict(Xc)
p = wide.ever_treated.mean()
w = (wide.ever_treated - wide.phat) / (1 - wide.phat) / p
print("IPW (Abadie 2005) ATT = %.0f" % (w * wide.dy).mean())

IPW (Abadie 2005) ATT = 1861

The inverse-propensity-weighted estimate is \$1,861, another near-miss on the benchmark from a completely different modeling philosophy. Instead of specifying how earnings trend with covariates, it specifies how treatment depends on them and lets the reweighting handle the rest. That two independent strategies — outcome regression and propensity weighting — both land near \$1,800 is the kind of convergence that builds confidence in the answer.

DR — Sant’Anna-Zhao (2020) by hand

Why choose between the two? The doubly-robust estimator of Sant’Anna and Zhao (2020) combines the outcome regression on the controls with the propensity reweighting, and is consistent if either model is correctly specified.

wide["dyhat"] = mH.predict(wide) # outcome regression from the HIT step
dr_t = (wide.ever_treated * (wide.dy - wide.dyhat) / p).mean()
dr_c = ((1 - wide.ever_treated) * (wide.phat / (1 - wide.phat)) * (wide.dy - wide.dyhat) / p).mean()
print("DR (Sant'Anna-Zhao 2020) ATT = %.0f" % (dr_t - dr_c))

DR (Sant'Anna-Zhao 2020) ATT = 1993

The doubly-robust estimate is \$1,993. It sits a little further from the benchmark than the regression-adjustment estimators, but it comes with the strongest theoretical guarantee: it would still be consistent even if we had gotten the trend model or the propensity model wrong (just not both). In a real application, where you never know which model is right, that insurance is exactly the point.

The payoff: the covariate arc

Eight estimates, one picture. Ordered as an argument, the specifications tell a story that no single number could.

Spec	Estimator	ATT	95% CI	Where X enters
0	No covariates (naive TWFE)	\$3,621	[2,382, 4,860]	nowhere
A	Additive X	\$3,621	[2,305, 4,938]	level (inert)
BT	X × treatment	\$3,621	[2,343, 4,899]	effect (inert)
B	X × post	\$1,711	[331, 3,092]	trend (corrected)
C	Saturated FD = HIT	\$1,770	[396, 3,144]	trend + effect
—	IPW (Abadie 2005)	\$1,861	[261, 3,461]	propensity
—	DR (Sant’Anna-Zhao 2020)	\$1,993	[436, 3,550]	propensity
—	RCT benchmark	\$1,794		ground truth

The three inert specifications cluster at \$3,621; the four that touch the trend, plus the two propensity estimators, cluster around the \$1,794 line. The grouping is the thesis. And the ladder view makes the “snap” impossible to miss:

The estimate is flat at \$3,621 across the first three specifications, then drops off a cliff to \$1,711 the instant covariates enter the trend, and stays near the benchmark thereafter. That cliff — between “X × treatment” and “X × post” — is the single most important feature of the whole analysis.

Cross-check with the diff-diff package

Hand-coded estimators are pedagogically transparent, but a reader is entitled to ask whether we coded them correctly. So we run the same designs through the independent diff-diff package and compare.

from diff_diff import DifferenceInDifferences, CallawaySantAnna
naive = DifferenceInDifferences(cluster="id", seed=90210).fit(
panel, outcome="re", treatment="ever_treated", time="post", unit="id")
cs_df = panel.assign(first_treat=np.where(panel.ever_treated == 1, 1, 0))
cs = CallawaySantAnna(estimation_method="dr", seed=90210).fit(
cs_df, outcome="re", unit="id", time="post", first_treat="first_treat", covariates=XVARS)

naive 2x2 by pyfixest $3,621 vs diff-diff $3,621
additive X by pyfixest $3,621 vs diff-diff $3,621
doubly robust by hand $1,993 vs diff-diff (Callaway-Sant'Anna) $1,979

The package agrees exactly on the naive and additive designs and lands within \$14 of the by-hand doubly-robust estimate (\$1,993 vs \$1,979). The small remaining gap is expected: diff-diff’s Callaway-Sant’Anna implementation makes its own choices about propensity estimation and weight normalization, exactly the kind of default-level difference the reference flags between hand-coded and packaged estimators. For our purposes the takeaway is that the transparent code and the battle-tested package tell the same story.

Robustness: how much should we trust \$1,770 over \$1,711?

One caution deserves to be front and center, and it comes from a comment on Cunningham’s original post by Alexis Diamond. It is tempting to rank the corrected estimators — is \$1,770 “better” than \$1,711 because it is closer to the benchmark? Look again at the confidence intervals in the table: every corrected estimate spans roughly \$400 to \$3,100. The benchmark itself is estimated with a standard error of about \$671. Against that much noise, the \$59 gap between Spec B and Spec C is meaningless. What is trustworthy is not any single dollar figure but the pattern: three specifications that ignore the trend are all wrong in the same direction, and every specification that models the trend moves decisively toward the truth. Stability across sensible specifications — and, as Diamond argues, across many datasets, not one lucky one — is the signal. This is also why we bootstrap the standard errors (199 id-clustered resamples, seed 90210): the point estimates are only as informative as their uncertainty allows.

Discussion

The exercise settles a question that applied researchers wave away too often: are covariates in a difference-in-differences a robustness check, or do they do real work? If they were a robustness check, we would hope they leave the estimate unchanged — a moving estimate would be a warning sign. But here the covariates are not decoration; they are load-bearing. Under covariate imbalance combined with covariate-specific trends, they are what makes conditional parallel trends hold, and leaving them out — or putting them in the wrong place — produces an estimate that is off by a factor of two while looking perfectly precise.

The sharper lesson is that not all ways of including covariates are equal. The most common specification in the entire applied panel literature — two-way fixed effects with additive controls — is exactly the one that fails here, because time-invariant controls vanish under the within transformation and never touch the counterfactual trend. The specifications that succeed are the ones that let covariates bend the control group’s trajectory: X × post, first-difference saturation, and their outcome-regression and doubly-robust cousins. When you next see a DiD table where “we control for baseline covariates” is offered as reassurance, the right question is not whether covariates are included but where — in the level, the effect, or the trend.

Summary and next steps

The reconstruction is exact. Rebuilt from the causaldata package, the naive DiD returns \$3,621 and every one of the eight estimators matches Cunningham’s reported figures to the dollar — confirming both the data and the code.
Covariate placement, not inclusion, is what matters. Additive covariates (\$3,621) and covariate-by-treatment interactions (\$3,621) are inert; covariate-by-time interactions (\$1,711) and first-difference saturation (\$1,770) recover the \$1,794 benchmark. IPW (\$1,861) and doubly-robust (\$1,993) land nearby from a different modeling angle.
Trust the pattern, not the decimal. With 95% intervals spanning thousands of dollars and a benchmark that is itself noisy, the credible finding is the clean split between trend-ignoring and trend-modeling specifications — not a ranking among the corrected estimates.
Limitation and where to go next. This is one dataset with a small treated group and a famously thin covariate set; as Diamond notes, reliability comes from stability across many RCT-versus-observational benchmarks, which is the mission of the emerging rctvsobs.org repository. A natural next step is to extend the analysis to staggered-timing settings with the Callaway-Sant’Anna estimator covered in the introductory DiD tutorial, where doubly-robust covariate adjustment becomes the default rather than an afterthought.

Exercises

Swap the covariate set. Drop re74 and u74 from XVARS and re-run Spec B. How much of the correction survives when the model can no longer condition on prior earnings? What does that tell you about which covariate is doing the work?
Verify the HIT equivalence yourself. Confirm numerically that Spec C (saturated first differences on the full sample) and the control-only imputation both return \$1,770. Then break the equivalence by fitting the outcome regression on the whole sample instead of the controls — does the number change, and why?
Stress-test the propensity model. Trim units with estimated propensity scores above 0.9 and re-estimate the IPW and doubly-robust ATTs. How sensitive is each estimator to extreme weights, and which one would you trust more in a real application?

References

Acknowledgements

AI tools (Claude Code, Gemini, NotebookLM) were used to make the contents of this post more accessible to students. The analysis reproduces and builds on Scott Cunningham’s essay “Covariates, diff in diff and LaLonde test.” Nevertheless, the content in this post may still have errors. Caution is needed when applying the contents of this post to true research projects.

AI Podcast: Covariates and the LaLonde Test

Click play to load

0:00 0:00

Regional Inequality from Outer Space: Predicting GDP from Nighttime Lights and Building Inequality Indices in Python

Mon, 15 Jun 2026 00:00:00 +0000

Abstract

Most countries publish a single national GDP number but no income figures for their internal regions, so we cannot see whether development is shared evenly across a country’s territory. This tutorial reconstructs the measurement pipeline of Lessmann and Seidel (2017): it predicts regional GDP per capita from satellite nighttime lights, builds inequality indices from those predictions, and asks how regional inequality changes as countries grow richer. The data are a region-year panel of 5,258 subnational regions used to calibrate the lights model and a country-period panel of 180 countries spanning 1992–2012, all bundled as small CSVs. The methods are panel fixed effects in PyFixest, a random-effects sidebar in linearmodels, inequality math from first principles, and a from-scratch Conley spatial-HAC variance. The calibrated light elasticity of regional income is 0.102 and predicted income correlates 0.925 with observed income; the population-weighted regional Gini follows an N-shaped curve in development (cubic 0.293 / −0.032 / 0.001), ethnic inequality is its strongest correlate (0.071), and the light elasticity of 0.190 survives spatially-robust inference (Conley standard errors 0.026–0.037). These findings imply that nighttime lights can fill the subnational data gap well enough to study where, and for whom, growth fails to spread.

1. Overview

A government can tell you its country’s GDP, but rarely the GDP of each province inside it. That gap matters: two countries with identical national income can look completely different on the inside — one with a single booming capital surrounded by poor hinterlands, the other with broadly shared prosperity. To study that internal geography of income at a global scale, Lessmann and Seidel (2017) had a simple but powerful idea: let satellites do the accounting. Brighter places at night are, on average, richer places, so nighttime light can stand in for income where official statistics do not exist.

This post rebuilds their pipeline in Python, end to end. We start from light and a handful of controls, predict regional income, turn many regional incomes into a single inequality number per country, and finally ask the classic question: does regional inequality first rise and then fall as countries develop — the spatial version of the Kuznets curve?

The diagram below shows the four stages. The first two stages — prediction and construction — are the heart of this tutorial; they are where the data are actually made. The last two — the curve and its drivers — are familiar panel regressions, kept short here because a companion post, Regional Inequality and the Kuznets Curve: Panel Fixed Effects in Python, already explores turning points, period stability, and the full determinant analysis in depth on a pre-built inequality series.

flowchart LR
A("Nighttime lights<br/>+ controls") --> B("Predicted regional<br/>GDP per capita<br/>(Table 1)")
B --> C("Population-weighted<br/>inequality indices<br/>(Table 2)")
C --> D("Regional Kuznets<br/>curve (Table 3)")
C --> E("Determinants &<br/>robustness (Tables 4, B.4)")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A,B blue
class C orange
class D,E teal

Reading the diagram left to right, light becomes income (blue), income becomes inequality (orange), and inequality becomes the object of study (teal). Each arrow is a modelling choice we will make explicit and reproduce. By the end you will be able to defend every number on the page.

In this tutorial you will:

Predict regional GDP per capita from nighttime lights and controls, and form the predictions explicitly.
Construct five population-weighted inequality indices from first principles, and see exactly how population weights change the answer.
Explore the cross-country dynamics of regional inequality across time and world regions.
Estimate the regional Kuznets curve, its determinants, and a spatially-robust standard error using PyFixest.
Distinguish a prediction model from a causal claim, and a fixed-effects estimate from a random-effects one.

2. Key concepts at a glance

The post reuses a small vocabulary. The definition under each term is always visible; the example and analogy sit behind clickable cards — open them when a term feels slippery.

1. Nighttime lights as an income proxy. The brightness a satellite records over a place at night, used as a stand-in for that place’s economic output. Lights correlate with income because electricity use, roads, and activity all glow. They are imperfect — deserts and oil flares mislead — which is why we predict income from light rather than equate the two.

Example

The raw correlation between a region’s nighttime brightness and its observed income is strong but noisy; turning brightness into a predicted income (Table 1) more than doubles its usefulness for measuring inequality (Gini correlation 0.49 vs 0.21).

Analogy

Like guessing a household’s wealth from its electricity bill. Useful on average, wrong for the off-grid farmer and the crypto miner, but good enough to rank neighbourhoods.

2. Light-to-GDP elasticity $\beta_1$. The percent change in predicted regional GDP per capita for a 1% change in light per pixel, holding controls fixed. It is the slope of the calibration model and the single most important number in the prediction step.

Example

In the preferred specification the elasticity is $\beta_1 = 0.102$: a 10% brighter region is predicted to be about 1% richer, once national income and geography are controlled for.

Analogy

The exchange rate between “lumens” and “dollars”. A small number, because national income already does most of the conversion; light fine-tunes the regional detail.

3. Population-weighted inequality index. A summary of how unequally income is spread across a country’s regions, where each region counts in proportion to how many people live there. The post uses the Gini, three generalized-entropy indices, and the coefficient of variation.

Example

Germany 2010, built from its 16 regions, has a population-weighted Gini of 0.028 — low, because German regions are close in income and the populous ones sit near the average.

Analogy

A class grade that weights each student by attendance. A brilliant student who shows up once barely moves the class average; the regulars set it.

4. The role of population weights. Whether each region counts once (equal weight) or by its population changes the inequality number. Weighting ties the index to where people actually live, which is the policy-relevant quantity.

Example

Across country-years the weighted and unweighted Gini correlate 0.75; weighting lowers the average Gini by about 0.003, because tiny extreme regions lose influence.

Analogy

Voting by headcount versus by district. A near-empty district and a megacity count equally in the second system; population weighting is the first.

5. The spatial Kuznets curve. The hypothesis that regional inequality rises during early development, then falls as countries converge internally — an inverted U (or, with a third act at high income, an N) in inequality against log GDP per capita.

Example

The cubic in log income has coefficients $0.293 / -0.032 / 0.001$, tracing a rise, a fall, and a faint upturn — an N-shape with country and period fixed effects.

Analogy

A country’s internal road trip: the gap between regions widens leaving the village, narrows approaching the city, and frays again in the sprawling suburbs of the very rich.

6. Conley (spatial-HAC) standard errors. Standard errors that allow nearby regions’ errors to be correlated, because a shock to one region usually spills into its neighbours. They are wider — and more honest — than the default that treats each region as independent.

Example

The light elasticity’s standard error rises from 0.013 (independent) to 0.026–0.037 (Conley, 1,000–5,000 km), but the estimate of 0.190 still sits far from zero.

Analogy

Counting independent witnesses. If ten “witnesses” all heard the same rumour, you really have one fact, not ten; Conley errors discount correlated neighbours.

3. Setup and imports

We use pandas and numpy for data work, matplotlib for figures, PyFixest for the panel fixed-effects regressions (its feols mirrors the R package fixest), linearmodels for the one random-effects table PyFixest cannot estimate, and statsmodels for a convenience regression behind one figure. PyFixest needs Python 3.10 or newer.

import numpy as np # arrays and math
import pandas as pd # data frames (tables)
import matplotlib.pyplot as plt # figures
import pyfixest as pf # fixed-effects / OLS regressions
from linearmodels.panel import RandomEffects # the one random-effects model (Section 6)
import statsmodels.formula.api as smf # a convenience regression (one figure)
# Site colour palette (used in every figure)
STEEL, ORANGE, INK, TEAL = "#6a9bcc", "#d97757", "#141413", "#00d4c8"
np.random.seed(42) # make any randomness reproducible

The site palette keeps the figures consistent: steel blue for primary data, warm orange for fitted lines and reference lines, near-black for the curves we want to stand out. With the tools loaded, we point at the data.

We load the bundled CSVs straight from GitHub so the notebook runs unchanged in Google Colab, falling back to a local data/ folder when you run it offline.

BASE = ("https://raw.githubusercontent.com/cmg777/starter-academic-v501/"
"master/content/tutorials/python_kuznets_dmsp/data/")
def load(name):
"""Read a bundled CSV from GitHub, falling back to a local data/ copy."""
try:
return pd.read_csv(BASE + name)
except Exception:
return pd.read_csv("data/" + name)

The load helper means every reader — on Colab, on a laptop, online or offline — gets the same data with no manual downloads. Next we read the files and look at their shapes.

4. The data: sources and construction

This section documents the data behind every number in the post: what each file is for, where each variable originally came from, how it was constructed, and what it looks like descriptively. Everything traces back to Lessmann and Seidel (2017). The exhaustive, column-by-column reference — construction, original source, units, and time–country coverage for all six files — lives in Appendix A; this section gives the readable tour.

4.1 Three views of the world

The replication ships three “views” of the same world. The region-year files (Prediction_Data.csv, Table_2_data.csv, Table_B4_data.csv) describe individual subnational regions: their lights, their observed and predicted income, their populations and coordinates. The country-year files (Table_3_data.csv, Table_4_data.csv, Figure_5_data.csv) describe whole countries, each already carrying the inequality indices computed from its regions. We read all six.

# --- load all six bundled CSVs (comment = unit of observation + purpose) ----
pred = load("Prediction_Data.csv") # region-year: lights -> GDP training set
t2 = load("Table_2_data.csv") # region-year: inequality-index inputs
t3 = load("Table_3_data.csv") # country-year: Kuznets data
t4 = load("Table_4_data.csv") # country-year: determinants
tb4 = load("Table_B4_data.csv") # region-year: lat/lon for spatial errors
f5 = load("Figure_5_data.csv") # country-year: regional vs personal Gini
# --- print each file's shape: rows (observations) x columns (variables) -----
for name, df in [("Prediction_Data", pred), ("Table_2_data", t2),
("Table_3_data", t3), ("Table_4_data", t4),
("Table_B4_data", tb4), ("Figure_5_data", f5)]:
print(f"{name:16s} {df.shape[0]:5d} rows x {df.shape[1]:2d} cols")

Prediction_Data 5258 rows x 30 cols
Table_2_data 5258 rows x 8 cols
Table_3_data 3675 rows x 9 cols
Table_4_data 3675 rows x 17 cols
Table_B4_data 5258 rows x 14 cols
Figure_5_data 3675 rows x 5 cols

The region-year files each hold 5,258 rows — these are the 1,504 regions, in 81 countries, that have both an observed GDP figure and a light reading, the sample used to calibrate the lights model. The country-year files hold 3,675 rows spanning 180 countries and the years 1992–2012. Keeping the two units straight is essential: we calibrate and predict at the region level, then measure inequality and run the Kuznets regressions at the country level.

4.2 The six files at a glance

Six CSVs, each a tidy panel keyed by country (and, for the region files, by region) and year. The complete column inventory for every file is in Appendix A.1; here is what each file is for and what it carries.

Prediction_Data.csv — region-year (5,258 × 30; 1,504 regions in 81 countries; 1992–2010). Purpose: the training sample that calibrates the light→income model (Table 1). These are the regions that have both an observed GDP figure (Gennaioli et al. 2014) and a light reading. Components: identifiers (Country_ISO, code_Coutry_Region, id_t_j = year+ISO); observed income (GDP_pc_Region, log_GDP_pc_Region); the model regressors (log_Light_ppix_Region, log_GDP_pc_Country, log top-/low-coded pixel counts, log_area, log_region, their interaction); World-Bank region-group dummies (eap…ssa); satellite-configuration dummies (satyear_1–satyear_7).
Table_2_data.csv — region-year (5,258 × 8; same training frame). Purpose: inputs to validate the inequality indices — it pairs predicted and observed regional income with region/country light and population. Components: pred_GDP_pc_Region, GDP_pc_Region, Light_Region, Light_Country, Pop_Region, Pop_Country.
Table_3_data.csv — country-year (3,675 × 9; 180 countries; 1992–2012). Purpose: the Kuznets dataset — national income plus the five population-weighted inequality indices built from predicted regional income. Components: GDP_pc_Country and GINIW_, COVW_, GE_1W_, GE_0W_, GE_m1W_pred_GDP_pc.
Table_4_data.csv — country-year (3,675 × 17; 180 countries; 1992–2012). Purpose: the determinants dataset — the Kuznets variables plus the structural correlates of regional inequality. Components: GINIW_pred_GDP_pc, GDP_pc_Country, Pop_Country, and the determinants Resources_rents_share_of_GDP, Arable_land, Trade_GDP_share, FDI_share_of_GDP, area, price_gasoline, Aid, School_enrollment_secondary, GINIW_Eth_light, Polity2, fedelupd2.
Table_B4_data.csv — region-year (5,258 × 14; training frame). Purpose: the spatial-robustness dataset — it adds each region’s centroid so the Conley spatial-HAC standard errors (§11) can down-weight distant regions. Components: Latitude, Longitude, log_GDP_pc_Region, log_Light_ppix_Region, satyear_1–satyear_7.
Figure_5_data.csv — country-year (3,675 × 5; 180 countries; 1992–2012). Purpose: the regional-versus-personal comparison (§12) — it sets the regional Gini beside a national interpersonal income Gini. Components: GINIW_pred_GDP_pc and Giniall (the personal Gini, observed for only 153 countries / 1,330 country-years).

4.3 How the key variables were built

Every variable above is the end of a construction chain that begins with raw satellite imagery and public databases; tracing that chain is what makes the numbers interpretable.

Nighttime lights. The light data are the DMSP-OLS stable lights product processed by the U.S. NOAA/National Geophysical Data Center: a digital number from 0 (dark) to 63 (saturated) for every ≈0.86 km² pixel, available annually from 1992. The authors average the light per pixel within each region and, following Hodler and Raschky (2014), add 0.01 where a region would otherwise read zero so the log is defined. Two censoring problems matter — bright cities top-code at 63, sparse areas bottom-code at 0 — which is why the prediction model also carries the counts of top- and low-coded pixels.
Sub-national boundaries. Regions are the 1st-level administrative units (states, provinces, cantons) from the GADM database — roughly OECD TL2 / EUROSTAT NUTS1 — 3,166 regions across 180 countries. The gridded light and population rasters are aggregated to these polygons.
Observed regional income. The observed regional GDP per capita used to train the model comes from Gennaioli et al. (2014): GDP per capita in constant 2005 PPP US\$ for 1,503 regions in 82 countries, an unbalanced panel built from OECD, national-statistics, and human-development-report sources.
Population. Regional population comes from the Gridded Population of the World (GPW) v3 raster (CIESIN): population density times region area, rounded up so the minimum is one, with the 5-year survey waves interpolated to annual values.
Predicted regional income. Because observed regional income exists for only ~80 countries, the model in §6 regresses log observed regional income on log light per pixel plus controls (country income, top-/low-coded pixel counts, number of regions, area and their interaction, and World-Bank region-group and satellite fixed effects) on the training sample, then predicts regional income for all 3,166 regions in 180 countries (1992–2012). The calibrated light elasticity is 0.102. Country-level controls come from the World Bank’s World Development Indicators (WDI) and the CIA World Factbook.
Inequality indices. From the predicted regional incomes, §7 builds five population-weighted indices per country-year — the Gini (GINIW), the coefficient of variation (COVW), and the generalized-entropy family GE(−1), GE(0) = mean log deviation, GE(1) = Theil — each weighting a region by its share of the national population so sparsely-populated outliers (e.g. Canada’s Northern Territories) do not dominate.
Determinants. The structural correlates in §10 are mostly WDI series — resource rents, arable-land share, trade and FDI shares, the gasoline pump price, net aid, and secondary-school enrolment — plus the Polity IV democracy score (Center for Systemic Peace, rescaled to [−1, +1]), a federalism dummy, and an ethnic-inequality index that applies the same population-weighted light-Gini to ethnic homelands (GREG geo-referencing, Weidmann et al. 2010; method of Alesina et al. 2016).

4.4 Descriptive statistics

With the variables defined, two summary tables give their shape — every substantive variable, split by unit of observation (region files 1992–2010, country files 1992–2012). Because the data are panels, each statistic — mean, median, sd, min and max — is reported twice: for the initial year and the final year. That way the table shows not just the level of each variable but how its whole distribution shifted over two decades. The tables are built with maketables.

import maketables as mt
# for every substantive variable: mean/median/sd/min/max in the initial vs final
# panel year, paired by statistic in a 2-level column header
region_stats = summarise_panel(region_spec, 1992, 2010) # 14 region-level variables
country_stats = summarise_panel(country_spec, 1992, 2012) # 19 country-level variables
mt.MTable(country_stats).make("html") # professional HTML; see script.py

	mean		median		sd		min		max
Summary statistics: region-level variables (initial 1992 vs final 2010)
	1992	2010	1992	2010	1992	2010	1992	2010	1992	2010
Observed GDP p.c. (region, US$)	4,428	18,883	3,999	13,819	2,407	14,775	1,029	854.49	11,064	95,873
Predicted GDP p.c. (region, US$)	3,887	17,622	3,888	13,151	1,498	12,284	824.11	904.63	7,163	57,104
log observed GDP p.c. (region)	8.24	9.52	8.29	9.53	0.6009	0.8665	6.94	6.75	9.31	11.47
log GDP p.c. (country)	8.32	9.68	8.58	9.75	0.4870	0.7340	7.10	7.23	8.61	10.87
log light per pixel (region)	-0.0410	1.55	0.0078	1.81	2.48	1.62	-4.61	-4.61	4.03	4.14
Total light (region, summed DN)	53,324	354,554	18,932	125,038	145,048	635,975	107.00	1,017	1,075,336	7,904,552
Total light (country, summed DN)	617,956	12,159,001	221,326	3,553,900	582,940	19,954,435	15,313	125,689	1,442,025	83,312,528
log # top-coded pixels	-10.69	-8.99	-12.74	-7.90	4.37	4.32	-16.28	-19.15	-0.8907	0.0000
log # low-coded pixels	-1.41	-1.87	-0.1075	-0.7072	3.25	3.16	-12.47	-15.16	-0.0001	-0.0004
log region area	13.13	13.31	12.89	13.14	0.7074	1.87	11.63	9.91	13.81	16.61
log # regions in country	2.60	3.19	2.89	3.18	0.5795	0.6954	1.39	1.39	3.04	4.34
log region x log area	34.40	43.09	37.26	42.71	8.64	13.56	19.02	17.15	42.05	72.16
Population (region)	3,856,250	4,388,654	979,271	1,008,927	6,437,358	13,969,073	13,637	986.47	26,062,216	199,528,672
Population (country)	37,059,083	124,426,832	56,507,488	38,161,672	29,632,582	272,703,362	4,362,136	1,193,269	90,145,888	1,328,343,680
Region-year (training sample). Each statistic is computed over the cross-section in the first (1992) and last (2010) panel year; nearest-year fallback if unobserved. Net values in source units. Sources: Appendix A.

	mean		median		sd		min		max
Summary statistics: country-level variables (initial 1992 vs final 2012)
	1992	2012	1992	2012	1992	2012	1992	2012	1992	2012
GDP p.c. (country, US$)	9,962	14,892	5,518	9,514	12,650	16,345	258.24	656.04	95,637	117,450
Regional Gini (GINIW)	0.0702	0.0612	0.0658	0.0584	0.0348	0.0333	0.0055	0.0019	0.1549	0.1594
Coeff. of variation (CV)	0.1407	0.1203	0.1350	0.1092	0.0710	0.0682	0.0228	0.0035	0.3327	0.3647
Theil index GE(1)	0.0118	0.0092	0.0088	0.0062	0.0107	0.0099	0.0003	0.0000	0.0484	0.0577
Mean log deviation GE(0)	0.0115	0.0090	0.0086	0.0063	0.0101	0.0093	0.0003	0.0000	0.0444	0.0514
GE(-1)	0.0114	0.0089	0.0085	0.0064	0.0100	0.0090	0.0003	0.0000	0.0442	0.0471
Population (country)	31,575,094	38,439,786	6,867,280	7,946,582	115,924,412	139,059,234	2,690	4,906	1,161,010,304	1,353,431,168
Resource rents (% GDP)	8.51	10.34	3.25	4.38	12.57	13.53	0.0000	0.0000	84.03	73.40
Arable land (share)	0.1455	0.1504	0.1025	0.1087	0.1424	0.1387	0.0004	0.0009	0.6614	0.5896
Trade (share of GDP)	0.7551	0.9300	0.6380	0.8515	0.4512	0.5060	0.1075	0.2662	2.80	4.50
FDI (share of GDP)	0.0180	0.0494	0.0073	0.0296	0.0444	0.0676	-0.1342	-0.0618	0.3981	0.4313
Land area (sq km)	757,199	727,792	175,020	155,360	1,978,672	1,917,722	50.00	50.00	16,380,084	16,380,084
Gasoline price (US$/L)	0.7865	1.21	0.6902	1.22	0.3661	0.4629	0.0260	0.0201	1.67	2.22
Net aid (US$ bn)	0.5640	0.7073	0.2207	0.3559	0.8743	0.9827	-0.3904	-0.1499	5.48	6.78
Secondary enrollment (% gross)	63.86	81.86	68.94	90.90	33.14	28.02	5.36	15.92	119.97	135.54
Ethnic inequality (light Gini)	0.2851	0.2656	0.2020	0.1985	0.2679	0.2474	0.0000	0.0000	0.8106	0.8123
Polity2 (-1 to +1)	0.2454	0.4377	0.6000	0.7000	0.7051	0.6041	-1.00	-1.00	1.00	1.00
Federal state (0/1)	0.1389	0.1364	0.0000	0.0000	0.3470	0.3443	0.0000	0.0000	1.00	1.00
Personal income Gini (0-100)	37.54	46.12	36.10	47.25	10.23	6.46	20.20	31.70	61.30	55.00
Country-year. Each statistic is computed over the cross-section in the first (1992) and last (2012) panel year; nearest-year fallback if unobserved; net aid in US$ bn. Sources: Appendix A.

The initial-vs-final columns make the dynamics explicit. At the region level, both observed and predicted GDP per capita shift up markedly over the two decades, while the predicted distribution stays narrower than the observed one — the lights model smooths the extremes. At the country level the regional Gini drifts down (mean 0.070 in 1992 → 0.061 in 2012) even as mean GDP per capita rises (\$9,962 → \$14,892) — the convergence §5 will formalise. The determinants are where to be careful: several are sparsely observed — the gasoline price, the personal income Gini, secondary enrolment and net aid cover far fewer country-years than the core panel (their per-variable coverage is tabulated in Appendix A), which is exactly why §10’s determinant regressions run on shifting subsamples. §4.5 now makes the time dynamics visual.

4.5 Exploratory data analysis

Summary tables compress each variable to a few numbers; they hide how the whole distribution moves over time. A box-plot over time restores that. We bin the years into the same five 5-year periods used later in the Kuznets regressions (§8) and, for each period, draw a box of the variable’s distribution across units (each unit contributes its period mean). Reading a row of boxes left-to-right shows the time dynamics; the height of each box shows the cross-sectional spread in that period. We do this once for the region-level variables and once for the country-level ones, so you can get a feel for every dataset.

# one box per 5-year period; box = cross-sectional distribution of unit period-means
def period_boxes(ax, df, unit, col, logy=False):
df = df.assign(p=pd.cut(df.year, [1989, 1994, 1999, 2004, 2009, 2014],
labels=["90–94", "95–99", "00–04", "05–09", "10–14"]))
g = df.groupby([unit, "p"], observed=True)[col].mean().reset_index()
ax.boxplot([g.loc[g.p == c, col].dropna() for c in g.p.cat.categories], showfliers=False)
if logy:
ax.set_yscale("log")
# ... 2x2 region panels + 2x4 country panels; see script.py for the full builder

The region-level panels (from Prediction_Data and Table_2, the 81-country training sample) tell a clear growth story: log light per pixel and both observed and predicted GDP per capita shift upward period by period — median region income rises more than fivefold, from about \$2,400 in 1990–94 to \$13,800 in 2010–14 — while regional population is broadly flat with an enormous spread (regions span five orders of magnitude). The light and income boxes also widen over time, a reminder that the DMSP sensors read brighter in later years.

The country-level panels pull from three datasets — Table_3 (GDP and the regional Gini), Table_4 (the determinants), and Figure_5 (the personal Gini). Country GDP per capita rises steadily; the regional Gini is strikingly stable around 0.06 with a slowly narrowing spread (the convergence §5 will quantify); gasoline prices and trade shares drift upward; resource rents and net aid are heavily right-skewed with fat upper tails in every period; and the personal income Gini edges down. Two cautions the boxes make obvious: the determinants are noisier and patchier than the core variables (recall their thinner coverage, §4.4), and the region-level boxes describe only the 81-country training subsample, not all 180 countries.

With the data documented, we look at how inequality behaves across countries.

5. Cross-country dynamics of inequality

Before predicting or regressing anything, it pays to see the data. This section maps the landscape: how the key variables are distributed, how regional inequality has moved over two decades, how it differs across world regions, and how the five inequality indices relate to one another. Every chart here is descriptive — it raises the questions the later models try to answer.

5.1 Distributions of the key variables

We begin with three histograms: the log of nighttime light per pixel and the log of regional GDP per capita (both at the region level), and the population-weighted regional Gini (at the country level). Looking at distributions first tells us whether variables are skewed, bounded, or multi-modal — facts that shape the models we can fit.

# three histograms side by side; .dropna() drops missing values before plotting
fig, axes = plt.subplots(1, 3, figsize=(12, 3.6))
axes[0].hist(pred["log_Light_ppix_Region"].dropna(), bins=40, color=STEEL) # log light
axes[1].hist(np.log(pred["GDP_pc_Region"].dropna()), bins=40, color=ORANGE) # log region income
axes[2].hist(t3["GINIW_pred_GDP_pc"].dropna(), bins=40, color=TEAL) # regional Gini
# ... titles and labels omitted for brevity (see script.py)
fig.savefig("python_kuznets_dmsp_01_distributions.png", dpi=300)
print("GINIW: mean={:.3f}, median={:.3f}, max={:.3f}".format(
t3["GINIW_pred_GDP_pc"].mean(), t3["GINIW_pred_GDP_pc"].median(),
t3["GINIW_pred_GDP_pc"].max()))

GINIW: mean=0.064, median=0.061, max=0.163

Log light and log income are both roughly bell-shaped — taking logs tames their heavy right skew, which is why the calibration model in Section 6 works in logs. The regional Gini is right-skewed and bounded below by zero, with a mean of 0.064 and a maximum of 0.163: most countries are internally fairly equal, but a long tail of countries has very uneven regions. That tail is what the rest of the post is about.

5.2 Inequality and income over time

Has regional inequality risen or fallen as the world grew richer? We average the regional Gini and log GDP per capita across all countries in each year from 1992 to 2012 and plot them on a shared timeline. Plotting the two series together previews the Kuznets question: do they move in the same direction or in opposite directions?

# Read this pandas chain top to bottom:
yr = (t3[(t3.year >= 1992) & (t3.year <= 2012)] # 1. keep years 1992-2012
.assign(logGDP=lambda d: np.log(d.GDP_pc_Country)) # 2. add a log-income column
.groupby("year") # 3. one group per year
.agg(GINIW=("GINIW_pred_GDP_pc", "mean"), # 4. average the Gini each year ...
logGDP=("logGDP", "mean")) # ... and the log income too
.reset_index())
print(yr.iloc[[0, -1]].round(4).to_string(index=False)) # show the first & last year

 year GINIW logGDP
1992 0.0702 8.5969
2012 0.0612 8.9956

As average world income climbed (orange, rising), average regional inequality fell from 0.070 in 1992 to 0.061 in 2012 (steel, declining). Globally, then, growth and falling within-country inequality went together over this period — a first hint that, on the downward arm of the Kuznets curve, development narrows regional gaps. But an average hides enormous variation across regions of the world, which we look at next.

5.3 Inequality across world regions

We group countries into the World Bank’s regions and draw a box plot of the regional Gini for each. A box plot shows the median (the orange line), the middle half of countries (the box), and the spread (the whiskers), so we can compare both typical levels and dispersion across world regions at a glance.

country_group = (pred.assign(g=pred.filter(["eap","eca","lac","mena","sa","ssa"])
.idxmax(axis=1))) # each region's World Bank group
eda = t3.copy()
eda["wb_group"] = eda["Country_ISO"].map(country_group_lookup) # see script.py
print(eda.groupby("wb_group")["GINIW_pred_GDP_pc"].median().sort_values().round(4))

N. America & high-inc. 0.0385
Europe & Central Asia 0.0421
South Asia 0.0451
Mid. East & N. Africa 0.0585
Latin America & Carib. 0.0724
East Asia & Pacific 0.0780
Sub-Saharan Africa 0.0962

The ordering is striking. Sub-Saharan Africa has the highest median regional inequality (0.096) — two and a half times that of North America and high-income countries (0.039) — with East Asia and Latin America close behind. Rich regions are not only richer on average; their internal income map is far more even. This cross-section already sketches the downward arm of a Kuznets relationship, which Section 8 will estimate properly.

5.4 How the five indices co-move

The paper measures inequality five ways: the Gini, the coefficient of variation (CV), and three generalized-entropy indices — GE(−1), GE(0) (the mean log deviation), and GE(1) (the Theil index). Do they tell the same story? We compute their correlation matrix across all country-years. If the indices co-move tightly, our headline Gini results will not hinge on that particular choice.

IDX = ["GINIW_pred_GDP_pc", "COVW_pred_GDP_pc", "GE_1W_pred_GDP_pc",
"GE_0W_pred_GDP_pc", "GE_m1W_pred_GDP_pc"]
cmat = t3[IDX].corr()
print("corr(Gini, CV) = %.3f" % cmat.iloc[0, 1])
print("corr(Gini, Theil)= %.3f" % cmat.iloc[0, 2])

corr(Gini, CV) = 0.969
corr(Gini, Theil)= 0.927

All five indices correlate above 0.9 — the Gini and the CV move almost in lockstep (0.97). This is reassuring: whichever index we lead with, the qualitative findings will be the same, so the Gini’s prominence below is a matter of convention, not of cherry-picking. With the landscape mapped, we turn to the engine of the whole exercise — turning light into income.

6. Predicting GDP from nighttime lights

This is the first of the two construction stages, and the foundation of everything that follows. The goal is a prediction model: feed it a region’s nighttime brightness plus a handful of controls, and it returns a guess of that region’s income. Once the model is trained on the regions where we do observe income, we can turn it loose on the tens of thousands of regions where we do not — which is the whole reason the satellite data is so valuable. We build the model exactly as Table 1 of the paper does, calibrating it on the 1,504 regions that have observed income.

Why should light predict income at all? At night, economic activity — factories, offices, lit streets, houses with electricity — shows up from space as brightness, and richer places tend to be brighter. The relationship is far from perfect (an oil field flares brightly with almost no one around; a dense but poor city can be dim), which is exactly why we add controls and, later, measure how good the predictions really are.

6.1 The idea: light as a proxy for income

We regress the log of a region’s GDP per capita on the log of its light per pixel, plus controls that soak up everything brightness should not be given credit for — the country’s overall income level, geography, the satellite generation, and the broad world region. Working in logs lets us read the slope as an elasticity: a percentage change in light maps to a percentage change in income. Formally:

$$y_r = \beta_0 + \beta_1 \ell_r + \beta_2 g_c + \gamma’ X_r + \mu_g + \tau_s + \varepsilon_r$$

Reading the equation one term at a time, with the dataset column name in parentheses:

$y_r$ — log regional GDP per capita (log_GDP_pc_Region): the outcome we want to predict, for region $r$.
$\beta_1 \ell_r$ — the light elasticity $\beta_1$ times log light per pixel (log_Light_ppix_Region). This is the one coefficient we truly care about: how strongly brightness tracks income once everything else is held fixed.
$\beta_2 g_c$ — an adjustment for log national GDP per capita (log_GDP_pc_Country) of region $r$’s country $c$. Without it, light would be unfairly credited with gaps that are really just rich-country-versus-poor-country differences.
$\gamma’ X_r$ — the geography controls (log_area, the number of regions log_region, their interaction log_region_X_log_area, and two pixel-saturation counts log_N_pix_top_cod_1_ppix / log_N_pix_low_cod_1_ppix). These absorb the fact that a physically huge region, or one whose brightest pixels are “topped out” at the sensor’s maximum, registers light differently for reasons that have nothing to do with being rich.
$\mu_g$ — a world-region fixed effect (group_id, e.g. Sub-Saharan Africa, Latin America): a separate baseline for each broad region, soaking up whatever makes a whole continent systematically brighter or dimmer.
$\tau_s$ — a satellite-generation fixed effect (satyear): different satellites and years calibrate brightness differently, and this term absorbs those technical differences.
$\varepsilon_r$ — everything left over.

Each control answers a specific “but couldn’t that gap just be …?” objection. Drop the national-income term and $\beta_1$ would partly pick up between-country wealth gaps; drop the satellite effect and it would partly pick up sensor changes. What survives on $\beta_1$ is the part of the brightness–income link we can actually defend.

A quick feel for the magnitude: an elasticity of, say, $0.10$ means a region that is twice as bright (a 100% increase in light) is predicted to be only about 7% richer, since $2^{0.10}\approx 1.07$. Light moves far more than income — brightness is a noisy proxy, useful on average but nowhere near one-for-one.

6.2 Building the model up, one control at a time

The paper does not jump straight to the full model. It builds up in seven steps, each adding a fixed effect or a control, so we can watch the light elasticity change as more is held fixed. That progression is itself the lesson: it reveals how much of the raw brightness–income link is real, and how much was just rich-versus-poor confounding. We estimate each step with PyFixest (pf.feols), which fits ordinary least squares with optional fixed effects.

Two pieces of PyFixest syntax for newcomers. Anything after the | in the formula is a fixed effect — a separate intercept for every value of that column, swept out of the data before the slope is estimated. And vcov={"CRV1": "Country_ISO"} asks for standard errors clustered by country: it tells the model that regions in the same country are not independent observations, so it should not over-state how precise the estimates are.

First we build the two label columns the fixed effects need:

# --- Step 1: build the categorical columns the fixed effects need ----------
# PyFixest wants ONE label column per fixed effect (not a wall of 0/1 dummies).
# satyear_1 ... satyear_7 are 0/1 flags; fold them into a single 1-7 code.
pred["satyear"] = sum(i * pred[f"satyear_{i}"] for i in range(1, 8)).astype(int)
# eap/eca/.../ssa are 0/1 world-region flags. idxmax(axis=1) returns, for each
# row, the NAME of the column that equals 1 -- i.e. it turns the one-hot dummies
# back into a single world-region label per region.
pred["group_id"] = pred.filter(["eap", "eca", "lac", "mena", "sa", "ssa"]).idxmax(axis=1)

Now the ladder. We show four specifications — pooled, region fixed effects, plus national income, and the full model — and print the light elasticity at each:

# --- Step 2: the ladder of specifications (each spec adds something) --------
GEO = ("log_N_pix_top_cod_1_ppix + log_N_pix_low_cod_1_ppix + log_area + "
"log_region + log_region_X_log_area") # the geography controls
specs = {
# spec 1: pooled OLS, no fixed effects -- the raw, confounded correlation
1: "log_GDP_pc_Region ~ log_Light_ppix_Region",
# spec 2: + region & satellite FE -> the clean WITHIN-region elasticity
2: "log_GDP_pc_Region ~ log_Light_ppix_Region | code_Coutry_Region + satyear",
# spec 4: + national income, so light is not credited with rich-vs-poor gaps
4: "log_GDP_pc_Region ~ log_Light_ppix_Region + log_GDP_pc_Country | Country_ISO + satyear",
# spec 7: + all geography controls and world-region FE -- the full model
7: f"log_GDP_pc_Region ~ log_Light_ppix_Region + log_GDP_pc_Country + {GEO} | group_id + satyear",
}
# --- Step 3: fit each spec and read off the light elasticity ----------------
for k, fml in specs.items():
m = pf.feols(fml, data=pred, vcov={"CRV1": "Country_ISO"}) # cluster by country
print(f"col {k}: light elasticity = {m.coef()['log_Light_ppix_Region']:.3f}")

col 1: light elasticity = 0.359
col 2: light elasticity = 0.190
col 4: light elasticity = 0.131
col 7: light elasticity = 0.049

Watch the elasticity fall as we climb the ladder. The pooled estimate of 0.359 (column 1) blends two very different comparisons: brighter-versus-dimmer regions within a country, and richer-versus-poorer countries. Adding region fixed effects (column 2) discards the cross-region comparison and keeps only the within-region one — the elasticity drops to 0.190. This is the clean within-region number, and it is the one Section 11 later stress-tests for spatial correlation. Adding national income (column 4, 0.131) strips out what was really a country-level wealth effect, and the full model with every geography control (column 7, 0.049) leaves only the thin sliver of variation that survives after region, country, and continent are all accounted for.

So which number is “right”? It depends on what we plan to do with the model — and that turns on the choice between fixed effects and random effects. That choice is important enough, and is the estimator the paper actually publishes, that it gets its own section.

6.3 Fixed effects vs random effects — and why prediction needs random effects

This is the conceptual core of the whole construction, so we will take it slowly. Both estimators fit the same equation; they differ in what variation they use and — crucially for us — in whether the fitted model can be applied to a brand-new region.

Fixed effects (FE) give every region its own intercept and then compare a region only to itself over time. Every difference between regions is swept away as a nuisance. This is wonderfully safe: anything permanent about a region — its terrain, its history, its institutions — is automatically controlled for, even things we never measured. But there is a price. The region intercepts are estimated only for regions in the training sample. Show a fitted FE model a region it has never seen, and it has no intercept for that region; it literally cannot produce a prediction.

Random effects (RE) instead treat each region’s intercept as a random draw from a common distribution with one estimated mean and variance. Because the region effect is now summarised by a couple of shared parameters rather than one free intercept per region, the model can use both the within-region variation (changes over time) and the between-region variation (richer-versus-poorer regions). The payoff is decisive: RE produces one coefficient vector that applies to any region, in the sample or out of it — so we can predict income for the tens of thousands of regions that have no income statistics at all.

	Fixed effects	Random effects (used by the paper)
Uses which variation?	within-region only	within and between region
Controls unobserved region traits?	yes, automatically	only if uncorrelated with the regressors
Predict for a new, unseen region?	no (no intercept for it)	yes — one shared model
Use of the data	discards between-region signal	more efficient
Key assumption	none on the region effect	region effect uncorrelated with regressors

For the paper’s goal — drawing a global income map by predicting every region on Earth — fixed effects are simply not an option, and that is exactly why the published Table 1 uses random effects. PyFixest does only FE/OLS, so for this one step we switch to linearmodels.RandomEffects. Let us build the random-effects fit slowly, in four steps.

# --- Step 1: tell the estimator the panel structure ------------------------
# A "panel" = the same units (regions) observed over several years. Indexing by
# (region, year) tells RandomEffects which rows belong to the same region.
panel = pred.set_index(["code_Coutry_Region", "year"])

# --- Step 2: cluster the standard errors by country ------------------------
# Regions in the same country move together, so we cluster on country: turn each
# country code into an integer label (one per row) for the clustered covariance.
clusters = pd.DataFrame(
{"c": pd.Categorical(panel["Country_ISO"]).codes},
index=panel.index,
)

# --- Step 3: a small helper that fits the RE model for any set of regressors --
def re_fit(cols):
# Always prepend a constant -- the shared baseline intercept ...
X = pd.concat([pd.Series(1.0, index=panel.index, name="const")] + cols, axis=1)
y = panel["log_GDP_pc_Region"] # outcome: log regional income
# ... then fit random effects with country-clustered standard errors.
return RandomEffects(y, X).fit(cov_type="clustered", clusters=clusters)

# --- Step 4: fit the full (column-7) specification -------------------------
# Regressors: log light + log national income + the five geography controls,
# the world-region dummies, and the satellite-generation dummies.
re7 = re_fit([
panel[["log_Light_ppix_Region", "log_GDP_pc_Country",
"log_N_pix_top_cod_1_ppix", "log_N_pix_low_cod_1_ppix",
"log_area", "log_region", "log_region_X_log_area"]],
pd.get_dummies(panel["group_id"], drop_first=True).astype(float), # world region
panel[[f"satyear_{i}" for i in range(1, 8)]].astype(float), # satellite
])
print("RE col 7 light elasticity = %.3f" % re7.params["log_Light_ppix_Region"])
print("RE col 7 national-GDP elasticity = %.3f" % re7.params["log_GDP_pc_Country"])

RE col 7 light elasticity = 0.102
RE col 7 national-GDP elasticity = 0.889

The random-effects light elasticity in column 7 is 0.102 — exactly the paper’s published number — versus the 0.049 we got from fixed effects. Why is RE roughly twice as large? Because it keeps the between-region information that the within estimator threw away: with national income already absorbing most of the scale, the between-region spread is precisely where light still earns its keep. The national-income elasticity of 0.889 confirms that a region’s income tracks its country’s income almost one-for-one, with light supplying the remaining subnational detail.

We report all seven specifications side by side below. The note records that the random-effects elasticity is essentially identical to the FE/OLS estimate; column 2, the one pure fixed-effects column, is where FE and RE coincide at 0.190.

# --- the seven specifications side by side, as in the paper's Table 1 ------
import maketables as mt
et1 = mt.ETable(
[fe_models[k] for k in range(1, 8)],
head_order="d", # header: dependent variable + (1)-(7)
labels={"log_GDP_pc_Region": "log regional GDP per capita", ...}, # readable names
coef_fmt="b:.3f* (se:.3f)", show_fe=True,
)
et1.make("html") # -> self-contained HTML table

	log regional GDP per capita
Table 1. Nighttime lights predict regional GDP per capita
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
coef
log light per pixel	0.359*** (0.041)	0.190*** (0.041)	0.134*** (0.035)	0.131*** (0.035)	0.268*** (0.038)	0.094*** (0.021)	0.049* (0.026)
log GDP p.c. (country)				0.864*** (0.039)		0.945*** (0.036)	0.896*** (0.034)
log # top-coded pixels							0.027*** (0.007)
log # low-coded pixels							-0.018** (0.007)
log area							0.174*** (0.041)
log # regions							0.463*** (0.122)
log # regions × log area							-0.043*** (0.010)
Intercept	8.754*** (0.105)
fe
Country FE	-	-	x	x	-	-	-
Region FE	-	x	-	-	-	-	-
WB-group FE	-	-	-	-	x	x	x
Satellite FE	-	x	x	x	x	x	x
stats
Observations	5,258	5,216	5,258	5,258	5,258	5,258	5,258
R²	0.361	0.979	0.857	0.866	0.578	0.844	0.86
Dependent variable: log regional GDP per capita (PyFixest FE/OLS; SEs clustered by country, in parentheses). The coefficient on log light per pixel is the elasticity. The random-effects estimator used in the published table is very close (col 7: RE=0.102). * p<.1 p<.05 * p<.01.

6.4 Forming the predictions — and a worked example

A model is only useful if we can actually predict with it. Prediction here is mechanical: take each region’s characteristics, multiply them by the estimated random-effects coefficients, and add everything up. That gives a fitted log income; because the model is in logs, we exponentiate to get back to dollars.

# --- Step 1: predicted LOG income = (design matrix) x (RE coefficients) -----
# X7 has one row per region-year and one column per regressor (plus the constant).
# The matrix product X7 @ beta applies the SAME coefficient vector to every region --
# this is what fixed effects could not do, and why we used random effects.
X7 = re_design([...]) # design matrix (see script.py)
fitted_log = X7.values @ re7.params.reindex(X7.columns).values # X . beta
# --- Step 2: undo the log to get dollars, then check against reality --------
pred_pc = np.exp(fitted_log) # predicted GDP per capita, in dollars
obs_log = panel["log_GDP_pc_Region"].values
r = np.corrcoef(fitted_log, obs_log)[0, 1] # how close are predictions to observed income?
print(f"corr(predicted, observed log GDP per capita) = {r:.3f}")

corr(predicted, observed log GDP per capita) = 0.925

A worked example, by hand. Imagine a region with log light per pixel $\ell_r = 1.5$ in a country with log national income $g_c = 9.0$, and suppose (to keep it simple) its geography controls and region/satellite effects net out to roughly zero. Using the two headline coefficients, its predicted log income is about the shared constant, plus $0.102 \times 1.5$ from light, plus $0.889 \times 9.0$ from national income. Notice how the national-income term dominates: most of a region’s predicted income comes from which country it is in, while light nudges the estimate up or down to capture how that particular region compares with its neighbours. Exponentiating the sum returns a figure in dollars. The full model simply does this for all 5,258 region-years at once with the matrix product X7 @ beta.

Predicted and observed log income correlate 0.925 across all 5,258 region-years, and the scatter hugs the 45° line across four orders of magnitude of income (figure above). The model is not just memorising one income band — it generalises from the poorest regions to the richest. That is what licenses the paper’s key move: applying these random-effects coefficients to every region on Earth, including the tens of thousands with no income statistics, to build a complete global income map. With predicted income in hand, we can finally measure inequality.

7. Constructing the inequality indicators

This is the second construction stage. We now have a predicted income for every region; the task is to compress each country’s many regional incomes into a single number that says how unequal they are — and to do it in a way that respects population. We build the indices from scratch so that nothing is a black box.

7.1 From many regional incomes to one number

Every index starts from the same three ingredients. Let region $i$ have income $y_i$ and population $w_i$. The population-weighted mean, the population shares, and the relative incomes are

$$\bar y = \frac{\sum_i w_i y_i}{\sum_i w_i}, \qquad p_i = \frac{w_i}{\sum_j w_j}, \qquad r_i = \frac{y_i}{\bar y}.$$

In words, $\bar y$ is the average income a randomly chosen person (not region) lives in, $p_i$ is the share of the country’s people in region $i$, and $r_i$ is region $i$’s income relative to the national average. In code, $y_i$ is pred_GDP_pc_Region, $w_i$ is Pop_Region, and the indices below are all built from p and r. Weighting by population is the key design choice: a region matters in proportion to how many people experience its income.

A tiny example makes this concrete. Suppose a country has three regions with incomes $y = (1, 2, 3)$ and equal populations $w = (1, 1, 1)$. Then the population-weighted mean is $\bar y = (1+2+3)/3 = 2$, each population share is $p_i = 1/3$, and the relative incomes are $r = (0.5, 1.0, 1.5)$ — the poor region earns half the average, the rich one earns 1.5×. Every index below is just a different way of summarising how far that vector $r$ spreads away from $1$. If the three populations were unequal — say the rich region held most of the people — the same incomes would produce a different mean and different shares, and the inequality numbers would move accordingly. That is population weighting at work.

7.2 The five indices from scratch

The Gini is the average absolute income gap between two randomly chosen people, scaled to lie in $[0, 1]$. The generalized-entropy family $GE(\alpha)$ varies in how sharply it reacts to gaps at the top ($\alpha$ large) or bottom ($\alpha$ small) of the distribution, and the coefficient of variation is the standard deviation over the mean. We implement all five directly:

$$G = \frac{\sum_i \sum_j w_i w_j , |y_i - y_j|}{2 \left(\sum_i w_i\right)^2 \bar y}, \qquad GE(0) = \sum_i p_i \ln!\frac{1}{r_i}, \qquad GE(1) = \sum_i p_i , r_i \ln r_i.$$

In words, the Gini $G$ sums the population-weighted absolute gaps $|y_i - y_j|$ between every pair of regions and normalises by twice the squared population and the mean; $GE(0)$ (the mean log deviation) and $GE(1)$ (the Theil index) are population-weighted averages of log relative income. A crucial coding detail: the Gini uses the absolute difference $|y_i - y_j|$, summed over all pairs — not a product — which is the classic trap when writing a weighted Gini by hand.

We build all five indices in a single function, ineq_indices(y, w), where y is the vector of regional incomes and w the vector of regional populations. Let us read it in three steps.

First, clean the inputs and form the three ingredients from §7.1:

def ineq_indices(y, w):
"""Five population-weighted inequality indices from first principles."""
# --- Step 1: clean the inputs ------------------------------------------
y, w = np.asarray(y, float), np.asarray(w, float)
ok = np.isfinite(y) & np.isfinite(w) & (w > 0) & (y > 0) # drop missing / non-positive
y, w = y[ok], w[ok]
# --- Step 2: the three ingredients (mean, shares, relative incomes) -----
sw = w.sum() # total population
mu = (w * y).sum() / sw # population-weighted mean income (ȳ)
p = w / sw # population shares (pᵢ, they sum to 1)
r = y / mu # relative incomes (rᵢ = yᵢ / ȳ)

Next, the four entropy-style indices. Each is a weighted average over p of some function of r; they differ only in which function, which is what makes each one sensitive to a different part of the distribution:

 # --- Step 3a: the generalized-entropy family + coefficient of variation -
ge_m1 = 0.5 * ((p * r**-1).sum() - 1) # GE(-1): very sensitive to the poorest
ge_0 = (p * (-np.log(r))).sum() # GE(0) = mean log deviation
ge_1 = (p * r * np.log(r)).sum() # GE(1) = Theil index
cv = np.sqrt(2 * 0.5 * ((p * r**2).sum() - 1)) # coefficient of variation

Finally, the Gini. This is the one line worth slowing down on:

 # --- Step 3b: the Gini = population-weighted average gap between people --
# y[:, None] - y[None, :] builds the full matrix of pairwise income gaps:
# entry (i, j) is yᵢ - yⱼ. np.abs makes them |yᵢ - yⱼ|; np.outer(w, w)
# weights each pair by both populations. Summing and normalising gives Gini.
gini = (np.abs(y[:, None] - y[None, :]) * np.outer(w, w)).sum() / (2 * sw**2 * mu)
return dict(GINIW=gini, GE_m1W=ge_m1, GE_0W=ge_0, GE_1W=ge_1, COVW=cv)

Two things to flag for beginners. The expression y[:, None] - y[None, :] is a NumPy broadcasting trick: it turns a length-$n$ vector into an $n\times n$ matrix of all pairwise differences in one stroke, with no Python loop. And the classic trap when coding a weighted Gini by hand is to forget the absolute value — the Gini sums $|y_i - y_j|$, the size of each gap, not the signed difference or a product; drop the np.abs and the answer collapses to zero.

This single function is the whole measurement apparatus. It takes a country-year’s regional incomes and populations and returns all five indices. Everything downstream — the Kuznets curve, the determinants — is just these numbers, regressed. To trust them, we test the function on a country we can reason about.

7.3 A worked example: Germany

Germany is a good test case: 16 regions of broadly similar income, so we expect a low inequality number. We pull its 2010 regions and run them through the function by hand.

# Pull Germany's 2010 rows, then feed its regional incomes + populations
# straight into the function we just wrote and print all five indices.
deu = t2[(t2.Country_ISO == "DEU") & (t2.year == 2010)]
print("regions:", len(deu))
print(ineq_indices(deu["pred_GDP_pc_Region"], deu["Pop_Region"]))

regions: 16
{'GINIW': 0.0278, 'GE_m1W': 0.0017, 'GE_0W': 0.0016,
'GE_1W': 0.0016, 'COVW': 0.0565}

Germany’s 16 regions yield a population-weighted Gini of 0.028 — very low, as expected for a country whose regions cluster near the national average. The Theil index (0.0016) and the others agree on the same verdict. A concrete, hand-checkable number like this is the sanity check that the formula is implemented correctly before we apply it to 180 countries.

7.4 The role of population weights

Does population weighting actually change anything? We recompute the Gini for every country-year without weights — letting every region count once — and compare. This isolates exactly what the weights do.

# --- Step 1: an equal-weight Gini (same formula, but every region counts once) --
# Note what is missing versus ineq_indices: no population weights w, no np.outer.
def gini_unweighted(y):
y = np.asarray(y, float); y = y[np.isfinite(y) & (y > 0)]
n, mu = y.size, y.mean()
return np.abs(y[:, None] - y[None, :]).sum() / (2 * n**2 * mu)
# --- Step 2: compare the two Ginis across every country-year -------------------
# `built` already holds the weighted GINIW and the equal-weight GINI_unw side by side.
corr_wu = built["GINIW"].corr(built["GINI_unw"]) # do they even agree?
mean_gap = (built["GINIW"] - built["GINI_unw"]).mean() # and in which direction?
print(f"corr(weighted, unweighted) = {corr_wu:.3f}")
print(f"mean(weighted - unweighted) = {mean_gap:+.4f}")

corr(weighted, unweighted) = 0.747
mean(weighted - unweighted) = -0.0034

The weighted and unweighted Gini correlate only 0.75 — far from identical — and weighting lowers inequality on average by 0.0034. The scatter (figure above) shows most points below the 45° line: population weighting pulls the index down because small, income-extreme regions (a tiny mining province, a remote capital) count for less when we weight by people. The lesson is general — report your weighting: the same country can look more or less unequal depending on whether you count regions or people, and “by people” is usually the policy-relevant choice.

7.5 Do our indices match the paper?

Two checks. First, the from-scratch indices should reproduce the paper’s Table 2 — the correlation between inequality measured from predicted income and inequality measured from observed income. Second, an honest caveat about coverage.

import maketables as mt
# For each index we have two correlations across countries (2001-2012 means):
# pred_obs = inequality from PREDICTED income vs from OBSERVED income (our method)
# light_obs = inequality from RAW LIGHT vs from OBSERVED income (the shortcut)
# A higher number means the measure tracks "true" (observed-income) inequality better.
t2tab = pd.DataFrame({"Predicted income vs observed": pred_obs,
"Raw light vs observed": light_obs}, index=index_labels)
mt.MTable(t2tab).make("html") # -> self-contained HTML table

	Predicted income vs observed	Raw light vs observed
Table 2. Inequality from predicted income tracks observed inequality
Gini	0.49	0.21
GE(-1)	0.39	0.11
MLD GE(0)	0.45	0.21
Theil GE(1)	0.50	0.30
CV	0.52	0.29
Cross-country correlations across 78 countries (period means 2001-2012) between each inequality measure computed from predicted income (or from raw light) and the same measure computed from observed income.

Inequality computed from predicted income correlates with inequality from observed income at 0.49 for the Gini — more than double the 0.21 we get from raw light density (table above), and the same pattern holds for all five indices. This is the payoff of the prediction step: turning light into income first, instead of treating brightness as income, roughly doubles how well we measure inequality. One honest caveat: our from-scratch indices are built on the ~1,500 regions that have observed income, whereas the paper’s published series uses every subnational region on Earth (the full-world prediction we did not bundle, to keep the data small). The two correlate 0.88, not 1.00 — a coverage difference, and precisely why the paper had to predict income for all regions, not just the calibration sample. With inequality measured, we can ask how it moves with development.

8. The regional Kuznets curve

Now the classic question. As countries grow richer, does regional inequality rise then fall? We regress the regional Gini on a cubic in log national income, with country and period fixed effects so the relationship is identified from each country’s own changes over time, not from rich-vs-poor comparisons. Section 9 then works the turning-point algebra and the discriminant test in full; the companion post python_fe_kuznets adds the period-by-period stability of the curve.

8.1 The cubic specification in PyFixest

We average the data into 5-year periods, build the cubic terms, and estimate with country and period fixed effects, clustering by country. The specification is

$$\text{GINIW}_{ct} = \beta_1 \ln Y_{ct} + \beta_2 (\ln Y_{ct})^2

\beta_3 (\ln Y_{ct})^3 + \alpha_c + \delta_t + u_{ct},$$

where $\text{GINIW}_{ct}$ is country $c$’s regional Gini in period $t$, $\ln Y_{ct}$ is its log GDP per capita, and $\alpha_c, \delta_t$ are country and period fixed effects. In code $\ln Y$ and its powers are lg, lg2, lg3, and the fixed effects are Country_ISO + p5.

Three modelling choices are worth unpacking before the code:

Why 5-year periods? Annual inequality numbers are jumpy — one noisy year can swing a small country’s Gini. Averaging into five-year blocks smooths that noise, so we fit the development trend rather than yearly wobble.
Why a cubic? A straight line can only rise or fall; a quadratic can bend once (the classic inverted-U hump); a cubic can bend twice, letting inequality rise, fall, and then edge up again. We let the data pick the shape instead of imposing a hump in advance.
What do the fixed effects buy us? The country effect $\alpha_c$ compares each country only with its own past, never with richer or poorer countries — so the curve is identified from how inequality moves as a country develops, not from a rich-vs-poor snapshot. The period effect $\delta_t$ strips out global shocks common to all countries in a period. As before, clustering by country keeps the standard errors honest.

# --- Step 1: collapse annual data to country x 5-year-period means ----------
agg = collapse_to_5yr(t3) # one row per (country, 5-year period)
# --- Step 2: build the cubic terms in log national income -------------------
agg["lg"] = np.log(agg["GDP_pc_Country"]) # ln Y
agg["lg2"] = agg["lg"]**2 # (ln Y)²
agg["lg3"] = agg["lg"]**3 # (ln Y)³
# --- Step 3: fit the cubic with country + period FE, clustered by country ---
m = pf.feols("GINIW_pred_GDP_pc ~ lg + lg2 + lg3 | Country_ISO + p5",
data=agg, vcov={"CRV1": "Country_ISO"})
print(m.coef()[["lg", "lg2", "lg3"]].round(3).to_string())
print("N =", m._N, " countries =", agg.Country_ISO.nunique())

lg 0.293
lg2 -0.032
lg3 0.001
N = 879 countries = 180

import maketables as mt
# the Gini ladder (linear / quadratic / cubic) + the cubic for the other four indices
# labels relabel the dependent-variable spanner per column (Gini, CV, Theil, ...)
et3 = mt.ETable([k1, k2, k3] + [k_other[c] for c in IDX[1:]],
model_heads=["linear", "quadratic", "cubic", "", "", "", ""],
labels={"GINIW_pred_GDP_pc": "Population-weighted regional Gini", ...},
coef_fmt="b:.3f* (se:.3f)", show_fe=True)
et3.make("html") # professional HTML table

	Population-weighted regional Gini			Coeff. of variation	Theil index	Mean log deviation	GE(−1)
Table 3. The regional Kuznets curve
	linear	quadratic	cubic	(4)	(5)	(6)	(7)
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
coef
log GDP p.c.	-0.003 (0.003)	0.056** (0.023)	0.293*** (0.078)	0.402*** (0.144)	0.066*** (0.023)	0.063*** (0.020)	0.061*** (0.019)
(log GDP p.c.)²		-0.003*** (0.001)	-0.032*** (0.009)	-0.044*** (0.016)	-0.008*** (0.003)	-0.007*** (0.002)	-0.007*** (0.002)
(log GDP p.c.)³			0.001*** (0.000)	0.002** (0.001)	0.000*** (0.000)	0.000*** (0.000)	0.000*** (0.000)
fe
Country FE	x	x	x	x	x	x	x
Period FE	x	x	x	x	x	x	x
stats
Observations	879	879	879	879	879	879	879
R²	0.972	0.974	0.975	0.979	0.97	0.97	0.97
Cols (1)-(3): dependent variable = regional Gini (GINIW), adding the linear / quadratic / cubic term in log GDP p.c. Cols (4)-(7): the cubic for the other four inequality indices. Country + 5-year-period FE; SEs clustered by country. * p<.1 p<.05 * p<.01.

The cubic coefficients are 0.293 / −0.032 / 0.001 — positive, negative, positive — exactly the paper’s values. The positive linear term means inequality rises with income at low levels; the negative quadratic bends the curve down; the tiny positive cubic adds a faint upturn at the very top. This is an N-shape: a Kuznets hump with a third act. The full table above — columns (1)–(3) building up the Gini ladder, (4)–(7) the cubic for the other four indices — shows the same sign pattern throughout, so the shape is not an artefact of the Gini.

8.2 Visualising the curve

Coefficients are abstract; a picture is not. We want to plot each country-period as a point — its regional Gini against its log income — and lay the fitted cubic on top. But there is a subtlety. The model also contains country and period fixed effects, so if we scatter the raw Gini the points scatter wildly around the curve, because each one still carries its country’s and period’s effect. The fix is a partial-residual plot: we strip the period effect out of each point first, so what remains lines up with the income-driven cubic. Here is how to build the exact figure, step by step.

# --- Step 1: refit the cubic with EXPLICIT dummies to recover the effects ---
# pf.feols hides the fixed effects; statsmodels with C(...) keeps them as
# coefficients we can read off. Same model, just a form we can take apart.
import statsmodels.formula.api as smf
mfe = smf.ols("GINIW_pred_GDP_pc ~ lg + lg2 + lg3 + C(Country_ISO) + C(p5)", agg).fit()
bb = {k: mfe.params[k] for k in ["lg", "lg2", "lg3"]} # the three cubic coefficients

# --- Step 2: net the PERIOD effect out of every point -----------------------
# Collect each 5-year period's estimated effect (period 1 is the baseline = 0),
# then subtract it from that point's Gini. The result, "partial", is the part of
# inequality NOT explained by which period it is -- i.e. net of period effects.
peff = {1: 0.0}
for k in (2, 3, 4, 5):
peff[k] = mfe.params.get(f"C(p5)[T.{k}]", 0.0)
agg["partial"] = agg["GINIW_pred_GDP_pc"] - agg["p5"].map(peff)

# --- Step 3: choose a constant so the curve sits inside the cloud -----------
# The country dummies shift the whole cloud up/down; we recenter the curve to the
# average height of the points so the line is drawn through them, not above/below.
cons = (agg["partial"]
- (bb["lg"]*agg.lg + bb["lg2"]*agg.lg2 + bb["lg3"]*agg.lg3)).mean()
# --- Step 4: evaluate the fitted cubic on a smooth grid of incomes ----------
xs = np.linspace(5.5, 11.8, 200) # log-income grid
ys = cons + bb["lg"]*xs + bb["lg2"]*xs**2 + bb["lg3"]*xs**3

# --- Step 5: draw the scatter of points + the fitted curve ------------------
# STEEL / INK are the site palette (steel blue, near-black).
fig, ax = plt.subplots(figsize=(6.4, 4.6))
ax.scatter(agg.lg, agg.partial, s=14, facecolors="none", # the cloud, net of period effects
edgecolors=STEEL, alpha=0.55)
ax.plot(xs, ys, color=INK, lw=2.4, label="fitted cubic") # the curve on top
ax.set(xlim=(5.5, 11.8), ylim=(0, 0.16), xlabel="log GDP per capita",
ylabel="partial regional inequality (GINIW)",
title="Regional inequality and development (Figure 4)")
ax.legend(loc="upper right", frameon=False)
fig.tight_layout()
fig.savefig("python_kuznets_dmsp_10_kuznets_scatter.png", dpi=300)

Reading the figure: the fitted curve rises to a gentle peak around a log income of 8 (roughly \$3,000 per capita), declines through the middle-income range, and flattens — with a barely perceptible uptick — at the very top, tracing the N-shape the coefficients implied. Each circle is one country in one 5-year period, net of period effects, so the vertical spread that remains is genuine country-to-country variation in inequality at a given income level. That the cloud is wide is the honest takeaway: development explains the shape of regional inequality, but a great deal is left over — and naming those leftover drivers is exactly what the determinants in Section 10 set out to do.

9. Turning points and the discriminant test

The cubic in §8 can bend twice — but does it actually, and does it bend inside the range of incomes we observe? This section answers both, and it is the most transferable skill in the post: any time you fit a cubic, these two checks tell you whether the curve really has the shape its coefficients seem to promise. The same two-step test is developed on a synthetic panel in the R companion post r_kuznets; here we apply it to the lights-based regional Gini.

9.1 Calculating the turning points

Where does the curve change direction? At a turning point the slope is zero, so we set the derivative of the cubic to zero:

$$\frac{\partial \text{GINIW}}{\partial \ln Y} = \beta_1 + 2\beta_2 \ln Y + 3\beta_3 (\ln Y)^2 = 0.$$

This is a quadratic in $\ln Y$, so it has at most two roots — the inverted-U peak and the high-income trough. We solve it with the quadratic formula and exponentiate each root back into dollars:

b1, b2, b3 = m.coef()[["lg", "lg2", "lg3"]] # 0.293 / -0.032 / 0.00112
D = b2**2 - 3*b1*b3 # the discriminant (see 9.2)
roots = np.sort([(-b2 - np.sqrt(D)) / (3*b3),
(-b2 + np.sqrt(D)) / (3*b3)]) # turning points, in ln Y
print("turning points: ln =", roots.round(2), "-> $", np.exp(roots).round(0))

turning points: ln = [ 7.74 11.25] -> $ [ 2287. 77206.]

Regional inequality rises with development up to ln(GDP) ≈ 7.7 (about \$2,287), falls through the middle-income range until ln(GDP) ≈ 11.3 (about \$77,206), and then rises again. Interpretation 1: the first threshold marks the industrial take-off where a few leading regions surge ahead of the rest; the second marks the maturity where within-country convergence has run its course and post-industrial forces — services, finance, skilled-city agglomeration — begin to pull the richest regions apart again. Both turning points fall inside the observed income range (\$190–\$117,191), so this is a genuine N-shape rather than an extrapolation, and the two thresholds match the companion python_fe_kuznets post exactly. The figure plots the marginal effect (the derivative) rather than the curve itself, because the turning points are precisely where that line crosses zero.

9.2 The discriminant: does the curve really bend?

Solving for the roots numerically works, but it hides why a cubic sometimes has two turning points and sometimes none. The quadratic $\beta_1 + 2\beta_2 Y + 3\beta_3 Y^2 = 0$ has two real solutions exactly when its discriminant is positive. After dropping a harmless factor of 4 (algebra below), the rule collapses to a single number:

$$D \;\equiv\; \beta_2^2 - 3\,\beta_1\beta_3.$$

There are three regimes:

Discriminant	Real turning points	Shape over the income line	Verdict
$D > 0$	2	rise–fall–rise (an “N on its side”)	the cubic shape is real
$D = 0$	1 (inflection)	a single flat spot, no reversal	knife-edge boundary
$D < 0$	0	monotonic — never reverses	the cubic shape is not real

The textbook quadratic discriminant is $b^2 - 4ac = (2\beta_2)^2 - 4(3\beta_3)(\beta_1) = 4(\beta_2^2 - 3\beta_1\beta_3) = 4D$; the factor of 4 never changes the sign, so we work with the tidier $D = \beta_2^2 - 3\beta_1\beta_3$. For our cubic:

D = b2**2 - 3*b1*b3
print(f"D = {D:+.6f} -> {'two turning points' if D > 0 else 'monotonic'}")

D = +0.000035 -> two turning points

Interpretation 2: $D = +0.000035$ is positive, so the N-shape is real — but only just. The figure holds the linear and cubic terms at their fitted values and changes only the squared term: when $D<0$ the curve climbs monotonically, at $D=0$ it develops a single flat inflection, and once $D>0$ it bends into the genuine rise–fall–rise. Our cubic sits a hair above the $D=0$ knife-edge, so the third “act” — the post-\$77k upturn — is real but faint, exactly the “barely perceptible uptick” the §8 scatter showed. A slightly smaller squared term would erase it altogether.

9.3 Two checks, not one: significance is not shape

Here is the trap. All three income terms in our cubic are individually significant, and it is tempting to conclude “therefore the relationship is a genuine cubic with two turning points.” That inference is wrong as stated. Significance answers “does the data prefer keeping this term?”; it does not answer “does the fitted curve actually bend inside the income range we observe?” The discriminant — plus a check on where the turning points fall — answers the second question. Applying both checks to our cubic and to three illustrative cases makes the distinction concrete:

def diagnose(label, b1, b2, b3, lo, hi):
D = b2**2 - 3*b1*b3
if D <= 0:
return dict(case=label, D=D, regime="monotonic (D<0)", in_range=False)
tp = np.exp(np.sort([(-b2 - np.sqrt(D))/(3*b3), (-b2 + np.sqrt(D))/(3*b3)]))
ok = bool((tp >= lo).all() and (tp <= hi).all())
regime = "2 turning points " + ("(both in range)" if ok else "(>=1 OUT of range)")
return dict(case=label, D=D, regime=regime, in_range=ok)
lo, hi = agg.GDP_pc_Country.min(), agg.GDP_pc_Country.max()
rows = [diagnose("This post's cubic (panel FE)", b1, b2, b3, lo, hi),
diagnose("Synthetic A: genuine N-shape", 0.220, -0.026, 0.0010, lo, hi),
diagnose("Synthetic B: monotonic trap", 0.220, -0.020, 0.0010, lo, hi),
diagnose("Synthetic C: turns out of range", 0.220, -0.026, 0.0001, lo, hi)]
print(pd.DataFrame(rows).to_string(index=False))

 case D regime in_range
This post's cubic (panel FE) 0.000035 2 turning points (both in range) True
Synthetic A: genuine N-shape 0.000016 2 turning points (both in range) True
Synthetic B: monotonic trap -0.000260 monotonic (D<0) False
Synthetic C: turns out of range 0.000610 2 turning points (>=1 OUT of range) False

Read the rows from top to bottom:

This post’s cubic — $D = +0.000035 > 0$ and both turning points (\$2,287 and \$77,206) fall inside the observed range (\$190–\$117,191). Significance and shape agree: a genuine, if marginal, N-shape.
Synthetic A — the same sign pattern with a clean $D>0$ and both turning points in range. This is what an unambiguous N-shape looks like.
Synthetic B (the trap) — the same signs as a real N-shape, only the squared term is a touch smaller in magnitude, and $D = -0.00026 < 0$. The curve is monotonic everywhere. A cubic regression on such data could report all three terms as “significant” and still have no turning point at all.
Synthetic C — $D>0$, so two turning points exist mathematically, but the tiny cubic term throws the upper one to an astronomical income far outside any real economy. Inside the observed range the curve never reverses. “Two turning points exist” would be technically true and practically misleading.

Interpretation 3: significance (does the data want the term?) and the discriminant-plus-range check (does the curve actually bend, and where?) are different questions, and you need both. Reporting “all three GDP terms are significant, so the curve is cubic” can fail in two distinct ways — the discriminant can be negative (B), or the turning points can fall outside the data (C). The honest workflow is: report the coefficients, compute $D$, and if $D>0$ confirm the turning points lie inside the observed income range before claiming an inverted-U or N-shape.

Aside (for Bayesian model averaging). The same trap reappears with a different label. In a BMA, a term’s posterior inclusion probability (PIP) near 1.00 is the Bayesian analogue of “statistically significant.” But a high PIP on the cubic term no more guarantees a genuine bend than a significant cubic coefficient does — you still compute $D = \beta_2^2 - 3\beta_1\beta_3$ from the posterior-mean coefficients and check the turning-point range. The R companion post r_kuznets works this analogy through with field data.

10. What drives regional inequality?

If two equally rich countries differ in regional inequality, what accounts for the gap? Following the paper’s Table 4, we add blocks of structural determinants on top of the cubic — (1) a baseline with the cubic alone, then (2) resources, (3) openness, (4) mobility/transport, (5) institutions, (6) transfers and education, and (7) ethnicity — each with country and period fixed effects and country-clustered standard errors. A positive coefficient means the factor is associated with more regional inequality. The seven specifications go side by side in a maketables regression table.

import maketables as mt
def det_fit(extra): # add a determinant block to the cubic
return pf.feols(f"GINIW_pred_GDP_pc ~ lg+lg2+lg3 + {extra} | Country_ISO+p5",
data=agg4, vcov={"CRV1": "Country_ISO"})
d0 = pf.feols("GINIW_pred_GDP_pc ~ lg+lg2+lg3 | Country_ISO+p5",
data=agg4, vcov={"CRV1": "Country_ISO"}) # (0) baseline
d1 = det_fit("Resources_rents_share_of_GDP + Arable_land") # (1) resources
d_inst = det_fit("Polity2 + lgXfed") # (4) institutions (lgXfed = log GDP × Federal)
d5 = det_fit("GINIW_Eth_light") # (6) ethnicity
# d2 openness, d3 mobility, d4 transfers+education are built the same way
mt.ETable([d0, d1, d2, d3, d_inst, d4, d5],
model_heads=["baseline", "resources", "openness", "mobility",
"institutions", "transfers/edu", "ethnicity"],
labels={"GINIW_pred_GDP_pc": "Population-weighted regional Gini", ...},
coef_fmt="b:.3f* (se:.3f)", show_fe=True).make("html")

	Population-weighted regional Gini
Table 4. Determinants of regional inequality
	baseline	resources	openness	mobility	institutions	transfers/edu	ethnicity
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
coef
log GDP p.c.	0.293*** (0.078)	0.350*** (0.075)	0.205** (0.080)	0.171** (0.078)	0.277*** (0.060)	0.226** (0.110)	0.149** (0.059)
(log GDP p.c.)²	-0.032*** (0.009)	-0.038*** (0.009)	-0.022** (0.009)	-0.019** (0.009)	-0.030*** (0.007)	-0.023* (0.014)	-0.015** (0.007)
(log GDP p.c.)³	0.001*** (0.000)	0.001*** (0.000)	0.001** (0.000)	0.001** (0.000)	0.001*** (0.000)	0.001 (0.001)	0.000* (0.000)
Resource rents/GDP		0.018*** (0.007)
Arable land		-0.053*** (0.014)
Trade/GDP			0.005*** (0.002)
FDI/GDP			0.009 (0.007)
Gasoline price				0.001 (0.002)
Area × gasoline				0.006** (0.003)
Polity2					-0.001 (0.001)
log GDP × Federal					-0.001 (0.004)
Aid/GDP						0.015** (0.007)
Schooling						-0.014* (0.007)
Ethnic inequality							0.071*** (0.016)
fe
Country FE	x	x	x	x	x	x	x
Period FE	x	x	x	x	x	x	x
stats
Observations	879	857	817	672	573	585	844
R²	0.975	0.976	0.976	0.982	0.985	0.976	0.98
Each column adds a block of determinants to the cubic in log GDP p.c. with country + 5-year-period FE; SEs clustered by country, in parentheses. The institutions column uses Polity2 and a log GDP × Federal interaction; the paper's ICRG bureaucratic-quality measure is licensed and omitted. * p<.1 p<.05 * p<.01.

The strongest determinant by far is ethnic inequality (column 7): 0.071 (p < 0.001) — countries where income differs sharply across ethnic homelands also have sharply unequal regions. Among the rest, resource rents push inequality up (0.018, p < 0.01) — resource wealth concentrates in a few regions — while a larger arable-land share pulls it down (−0.053, p < 0.001), consistent with agriculture spreading income more evenly; trade openness adds a small positive effect (0.005, p < 0.01) and aid relative to GDP a positive 0.015 (p < 0.05). The institutions column (5) is the one we can only partly reproduce: Polity2 and a log GDP × Federal interaction are both small and insignificant here, and the paper’s ICRG bureaucratic-quality index is licensed and omitted. The cubic in log GDP survives every block, and the sample drifts from column to column (N falls from 879 in the baseline to 573 where the sparse institutions variables bind), so the columns are best read as separate windows, not one nested model.

11. Spatial robustness: Conley standard errors

Regions are not independent: a boom in one province spills into its neighbours, so their regression errors are correlated. Ignoring that makes standard errors too small and t-statistics too big. We re-estimate the clean light elasticity (column 2, β = 0.190) and recompute its standard error allowing errors of regions within a chosen radius to be correlated — the Conley spatial-HAC correction — using a from-scratch implementation based on great-circle distances between region centroids.

m = pf.feols("log_GDP_pc_Region ~ log_Light_ppix_Region | code_Coutry_Region + satyear",
data=dfb) # point estimate = 0.190
# Conley variance: weight cross-products of region scores by a Bartlett kernel
# k = max(0, 1 - distance / cutoff), distance = haversine great-circle km (see script.py)
for r in (1000, 2500, 5000):
print(f"Conley SE @ {r} km = {np.sqrt(conley_var(r)):.3f}")
print(f"naive (iid) SE = {m.se()['log_Light_ppix_Region']:.3f}")

Conley SE @ 1000 km = 0.026
Conley SE @ 2500 km = 0.034
Conley SE @ 5000 km = 0.037
naive (iid) SE = 0.013

Allowing for spatial correlation roughly doubles to triples the standard error — from 0.013 to between 0.026 and 0.037 — because neighbouring regions are not the independent observations the naive formula assumes. Even so, the elasticity of 0.190 stays far from zero (a t-statistic above 5 at the widest radius), so the lights-predict-income relationship is not a statistical mirage created by ignoring geography. The figure shows the confidence interval widening with the radius while the point estimate holds fixed.

12. Regional versus personal inequality

A natural question: is regional inequality (gaps between places) just a reflection of personal inequality (gaps between people)? We compare each country’s regional Gini with its household-income Gini, both averaged over 2001–2012, and fit a line. A positive slope means the two inequalities go together.

slope, intercept = np.polyfit(agg5["GINIW_pred_GDP_pc"], agg5["GINIall_100"], 1)
print(f"n = {len(agg5)} countries | OLS slope = {slope:.3f}")

n = 144 countries | OLS slope = 0.587

Across 144 countries the household-income Gini rises with the regional Gini at a slope of 0.587: places with wide gaps between regions also tend to have wide gaps between people. Regional and personal inequality are distinct but linked — so policies that narrow the gap between a country’s regions are also, in part, distributional policies between its citizens. This connects the satellite-based regional measure back to the inequality people actually experience.

13. Discussion

We set out to answer a measurement question — can we see inside countries from space? — and a substantive one — how does regional inequality move with development? The answer to the first is a qualified yes: a light-to-income elasticity of 0.102, predictions that correlate 0.925 with observed income, and inequality measures that track the observed data twice as well as raw light does. That is good enough to study regions that official statistics ignore, which is the whole point: the method turns a data desert into a global, comparable income map.

On the substantive question, regional inequality follows an N-shaped Kuznets path — rising through early development, falling as countries converge internally (the world average dropped from 0.070 to 0.061 over 1992–2012), with a faint upturn among the very richest. The single strongest correlate is ethnic inequality (0.071), a reminder that the internal economic geography of a country is bound up with its human geography. For a policymaker, the practical implication is concrete: the places where growth is failing to spread are now visible and measurable even without a statistical office, and the levers most associated with the gap — resource dependence, ethnic division — are nameable. Two cautions frame all of this. The relationships are descriptive associations with fixed effects, not causal effects; and the income figures are predictions, accurate on average but wrong for any single unusual region.

14. Summary and next steps

Method insight. Nighttime lights predict regional income with an elasticity of 0.102 and a 0.925 correlation with observed income; predicting income first, rather than equating light with income, doubles the quality of the resulting inequality measures (Gini correlation 0.49 vs 0.21).
Measurement insight. Population weighting is not cosmetic: weighted and unweighted Gini correlate only 0.75, and weighting lowers measured inequality by ~0.003 on average, so the weighting choice must be reported.
Substantive insight. The regional Kuznets curve is N-shaped (cubic 0.293 / −0.032 / 0.001 across 180 countries), and ethnic inequality (0.071) is its strongest structural correlate.
Robustness insight. The light elasticity of 0.190 survives spatial correlation — Conley standard errors of 0.026–0.037 are two to three times the naive 0.013, but the estimate stays far from zero.
Limitation. Our from-scratch indices use only the ~1,500 regions with observed income; the published series uses every region on Earth (correlation 0.88), which is why the full paper predicts income globally.
Next steps. Swap in a modern lights product (VIIRS replacing DMSP) to extend the series past 2012; or carry the full-world prediction through to rebuild the global income map and the choropleth figures we skipped here.

15. Exercises

Re-weight the world. Modify ineq_indices to weight regions by land area instead of population, recompute the regional Gini for every country, and compare the cross-country ranking to the population-weighted one. Which countries move most, and why?
A fourth act? Re-estimate the Kuznets cubic on the coefficient of variation (COVW_pred_GDP_pc) instead of the Gini, and add a quartic term (lg4). Does the upturn at high income strengthen, vanish, or stay a rounding error?
How far do shocks travel? Recompute the Conley standard error at radii of 250, 500, and 10,000 km. Plot the standard error against the radius. At what distance does spatial correlation stop mattering for the light elasticity?

16. References

Original data sources

Appendix A. Data dictionary

This appendix documents every column in all six data files: what it is, how it was originally constructed, its source, its units, and its time–country coverage. Coverage is written as years · units · N, where N is the number of non-missing observations and units counts the distinct countries (country files) or regions (region files) with data. Definitions follow Lessmann and Seidel (2017) and the official variable labels in the authors’ replication archive; coverage and statistics are computed in script.py.

A.1 The six datasets in detail

All six files are tidy panels. The region files are keyed by region × year over the 1,504-region / 81-country training frame (1992–2010); the country files are keyed by Country_ISO × year over 180 countries (1992–2012). The inequality indices in the country files are built from the predicted regional incomes in the region files.

File	Unit	Rows × Cols	Years	Countries	Regions	What it is for
`Prediction_Data.csv`	region-year	5,258 × 30	1992–2010	81	1,504	Train the light→income model (Table 1)
`Table_2_data.csv`	region-year	5,258 × 8	1992–2010	81	1,504*	Validate the inequality indices (Table 2)
`Table_3_data.csv`	country-year	3,675 × 9	1992–2012	180	—	Kuznets curve: GDP + 5 indices (Table 3)
`Table_4_data.csv`	country-year	3,675 × 17	1992–2012	180	—	Determinants of inequality (Table 4)
`Table_B4_data.csv`	region-year	5,258 × 14	1992–2010	81	1,504	Conley spatial-HAC errors (+ lat/lon)
`Figure_5_data.csv`	country-year	3,675 × 5	1992–2012	180	—	Regional vs personal inequality

* Table_2_data.csv has no explicit region-id column, but its rows are the same 1,504-region training frame at region-year.

A.2 Variable dictionary

Coverage shorthand: region-frame variables are 1992–2010 · 1,504 reg (81 ctry) · N = 5,258 unless noted; core country-frame variables are 1992–2012 · 180 ctry · N = 3,675 unless noted.

Identifiers and keys

Variable	What it is	How constructed	Source	Unit	Coverage
`Country_ISO`	Country code (ISO 3166-1)	Assigned per country	GADM	string	all files
`Country_NAME`	Country name	—	GADM	string	all files
`Region_NAME`	Region name	1st-level admin-unit name	GADM	string	region frame
`code_Coutry_Region`	Numeric region key (original spelling kept)	Region identifier	Authors	integer	region frame
`id_t_j`	Country-year key	Concatenation of year + ISO (e.g. `2010CHE`)	Authors	string	region frame
`year`	Calendar year	—	—	year	per file (see A.1)

Lights and income

Variable	What it is	How constructed	Source	Unit	Coverage
`log_Light_ppix_Region`	Log avg nighttime light per pixel	Region mean of DMSP-OLS stable-lights DN (0–63); +0.01 if zero, then log	NOAA/NGDC	log DN	region frame
`Light_Region`	Regional total lights	Sum of pixel DN over the region	NOAA/NGDC	summed DN	1992–2010 · 81 ctry · 5,258 (Table_2)
`Light_Country`	Country total lights	Sum of pixel DN over the country	NOAA/NGDC	summed DN	1992–2010 · 81 ctry · 5,258 (Table_2)
`GDP_pc_Region`	Observed regional GDP per capita	Regional accounts, constant 2005 PPP US\$	Gennaioli et al. (2014)	US\$	region frame
`log_GDP_pc_Region`	Log of `GDP_pc_Region`	Natural log	Gennaioli et al. (2014)	log US\$	region frame
`pred_GDP_pc_Region`	Predicted regional GDP per capita	Fitted values of the eq.-1 RE model applied to all regions	This paper (model)	US\$	1992–2010 · 81 ctry · 5,258 (Table_2)
`GDP_pc_Country`	National GDP per capita	constant 2005 PPP US\$	World Bank WDI	US\$	country frame
`log_GDP_pc_Country`	Log national GDP per capita	Natural log	World Bank WDI	log US\$	region frame

Prediction-model regressors and fixed-effect dummies

Variable	What it is	How constructed	Source	Unit	Coverage
`log_N_pix_top_cod_1_ppix`	Log # top-coded pixels (DN = 63)	Count of saturated pixels per region, logged	NOAA/NGDC	log count	region frame
`log_N_pix_low_cod_1_ppix`	Log # low-coded pixels (DN = 0)	Count of dark pixels per region, logged	NOAA/NGDC	log count	region frame
`log_area`	Log region area	Region polygon area, logged	GADM	log km²	region frame
`log_region`	Log # regions in the country	Count of regions per country, logged	GADM / Gennaioli	log count	region frame
`log_region_X_log_area`	Interaction term	`log_region` × `log_area`	Derived	—	region frame
`eap`, `eca`, `lac`, `mena`, `sa`, `ssa`	World-Bank region-group dummies	1 if the country is in that group (North America = reference)	World Bank	0/1	region frame
`satyear_1` … `satyear_7`	Satellite-configuration dummies	1 per satellite/sensor era (sensors change and age over time)	NOAA/NGDC	0/1	region frame

Population and geography

Variable	What it is	How constructed	Source	Unit	Coverage
`Pop_Region`	Regional total population	Population density × region area, rounded up (min 1); 5-yr waves interpolated to annual	GPW v3 (CIESIN)	persons	region frame
`Pop_Country`	Country total population	Sum of regional populations	GPW v3 (CIESIN)	persons	region & country frames
`area`	Country land area	Total land area (excl. inland water)	World Bank WDI	km²	country frame
`Latitude`, `Longitude`	Region centroid coordinates	Polygon centroid	GADM	degrees	region frame (Table_B4)

Inequality indices (all population-weighted, on predicted regional income)

Variable	What it is	How constructed	Source	Unit	Coverage
`GINIW_pred_GDP_pc`	Regional Gini	Population-weighted Gini of `pred_GDP_pc_Region` within a country-year	This paper	0–1	country frame
`COVW_pred_GDP_pc`	Regional coefficient of variation	Population-weighted CV	This paper	≥ 0	country frame
`GE_1W_pred_GDP_pc`	Theil index, GE(1)	Population-weighted GE(α = 1)	This paper	≥ 0	country frame
`GE_0W_pred_GDP_pc`	Mean log deviation, GE(0)	Population-weighted GE(α = 0)	This paper	≥ 0	country frame
`GE_m1W_pred_GDP_pc`	GE(−1)	Population-weighted GE(α = −1)	This paper	≥ 0	country frame
`Giniall`	National interpersonal income Gini	Household-survey income Gini (0–100 scale)	Lessmann & Seidel (2017)	0–100	1992–2012 · 153 ctry · 1,330 (Figure_5)

Determinants (country frame, 1992–2012)

Variable	What it is	How constructed	Source	Unit	Coverage
`Resources_rents_share_of_GDP`	Natural-resource rents	Oil + gas + coal + mineral + forest rents, % of GDP	World Bank WDI	% GDP	177 ctry · N = 3,620
`Arable_land`	Arable-land share	Arable land as a share of land area (FAO definition)	World Bank WDI	share	178 ctry · N = 3,603
`Trade_GDP_share`	Trade openness	(Exports + imports) / GDP	World Bank WDI	ratio	176 ctry · N = 3,509
`FDI_share_of_GDP`	FDI openness	Net FDI inflows / GDP	World Bank WDI	ratio	174 ctry · N = 3,477
`price_gasoline`	Gasoline pump price	Pump price, PPP constant 2005 US\$/litre (the paper’s “transport cost” = area × price)	World Bank WDI	US\$/L	162 ctry · N = 1,366
`Aid`	Aid flows	Net aid received, constant 2011 US\$	World Bank WDI	US\$	155 ctry · N = 2,964
`School_enrollment_secondary`	Secondary-school enrolment	Gross secondary enrolment ratio	World Bank WDI	% gross	172 ctry · N = 2,566
`GINIW_Eth_light`	Ethnic inequality	Population-weighted light-Gini across ethnic homelands (method of Alesina et al. 2016)	NOAA/NGDC + GREG (Weidmann et al. 2010)	0–1	173 ctry · N = 3,528
`Polity2`	Democracy–autocracy score	Polity IV combined score, rescaled −1 (autocracy) to +1 (democracy)	Center for Systemic Peace, Polity IV	−1…+1	157 ctry · N = 3,158
`fedelupd2`	Federalism dummy	1 if the country is federally organised	Authors	0/1	1992–2009 · 154 ctry · N = 2,724

The authors’ Table 4 also uses an ICRG “bureaucratic quality” index, which is licensed and not redistributed in this bundle; the post therefore omits that one determinant column.

Appendix B. Stata replication

Everything in the main body was done in Python, stitched together from several libraries — pandas for the data, pyfixest for fixed effects, linearmodels for random effects, statsmodels for the Kuznets plot, and a hand-written function for the inequality indices. This appendix shows the same study in Stata, where the whole thing is far shorter. The reason is that Stata ships the heavy machinery as built-in commands: the entire prediction model is one xtreg … , re line, all five inequality indices come from one ineqdeco call, and the cubic Kuznets curve plus its graph are three lines (xtreg, margins, marginsplot). Fewer keystrokes, identical numbers.

The code below loads the bundled Stata datasets — the .dta files described in Appendix A, which carry their variable labels with them, so Stata’s output is self-documenting. The full, runnable script is the Stata do-file linked at the top of this post (stata_replication.do); the blocks here are excerpts. Install the two helper packages once:

* Run once per machine. outreg2 is optional (formatted tables); ineqdeco does Section B.4.
ssc install outreg2
ssc install ineqdeco

Each subsection below maps to a numbered section of the post, and every figure it quotes matches the Python result in the main body.

B.1 Setup and the data (§3–§4)

use reads a native .dta straight from the bundle — no column-type guessing as with a CSV — and xtset declares the panel (the region is the unit, the year is time). That one line is all the panel commands need afterwards.

* --- Step 1: load the region-year training panel and declare the panel -----
use "Prediction_Data.dta", clear // labels travel with the file
describe // already a mini data dictionary
xtset code_Coutry_Region year // panel = region x year

B.2 Cross-country dynamics (§5)

The descriptive picture is a couple of one-liners. summarize gives the moments; correlate reproduces the strong co-movement of the inequality indices reported in §5 — the regional Gini and the Theil index correlate about 0.93.

* --- summary statistics and the index co-movement --------------------------
summarize log_GDP_pc_Region log_Light_ppix_Region log_GDP_pc_Country
use "Table_3_data.dta", clear // the country-year panel
correlate GINIW_pred_GDP_pc GE_1W_pred_GDP_pc COVW_pred_GDP_pc // Gini–Theil ~ 0.927

B.3 Predicting GDP from nighttime lights — Table 1 (§6)

This is the centrepiece. We regress log regional GDP per capita on log nighttime light per pixel across seven progressively richer specifications. In Stata each specification is one xtreg line, and the estimator is chosen by a single option: re for random effects, fe for fixed effects. robust cluster(Country_ISO) makes the standard errors robust and clustered by country (this changes the standard errors, not the point estimate).

* --- Step 1: the seven-spec ladder (random effects, as in the paper) --------
use "Prediction_Data.dta", clear
xtset code_Coutry_Region year
xi i.code_Coutry_Region // region dummies for the OLS column (2)
set matsize 11000 // room for ~1,500 region dummies
* (1) raw RE, no fixed effects
xtreg log_GDP_pc_Region log_Light_ppix_Region, re robust cluster(Country_ISO)
* (2) OLS with region + satellite FE -- the clean within-region elasticity
reg log_GDP_pc_Region log_Light_ppix_Region _Icode_Cout_2-_Icode_Cout_1504 ///
satyear_1-satyear_7, robust cluster(Country_ISO)
* (7) the PREDICTION model: + national income, geography, world-region & satellite FE
xtreg log_GDP_pc_Region log_Light_ppix_Region log_GDP_pc_Country ///
log_N_pix_top_cod_1_ppix log_N_pix_low_cod_1_ppix log_area log_region ///
log_region_X_log_area satyear_1-satyear_7 eap ssa mena lac eca sa, ///
re robust cluster(Country_ISO)

The light elasticity climbs down the ladder — 0.399 → 0.190 → … → 0.102 at column 7 — and the national-income elasticity in column 7 is 0.889. These are exactly the random-effects numbers the post reports.

Fixed effects vs random effects — one word apart. The post devotes §6.3 to why the paper uses random effects; in Stata the contrast is a single option:

* --- Step 2: same specification, swap only the estimator -------------------
xtreg log_GDP_pc_Region log_Light_ppix_Region satyear_1-satyear_7, ///
fe robust cluster(Country_ISO) // within (FE): 0.190
xtreg log_GDP_pc_Region log_Light_ppix_Region satyear_1-satyear_7, ///
re robust cluster(Country_ISO) // random (RE): 0.190

At this simple specification FE and RE coincide at 0.190. They diverge once the full controls enter (the post’s within/FE estimate is 0.049 versus the random-effects 0.102): random effects keep the between-region signal that the within estimator discards. More importantly, only RE can predict for a region outside the sample — fixed effects have no intercept for an unseen region. That is the whole reason the paper publishes the random-effects model.

	`, fe` (fixed effects)	`, re` (random effects — the paper)
Uses which variation?	within-region only	within and between region
Predict for an unseen region?	no (no intercept for it)	yes — one shared model
Key assumption	none on the region effect	region effect uncorrelated with regressors

Predicting. After the random-effects fit, predict , xb builds (design matrix) × (coefficients) for every region in one line — the move fixed effects could not make — and exp() undoes the log.

* --- Step 3: predict, then back-transform to dollars, then validate --------
predict double pred_log_GDP_pc_Region, xb // fitted log income
gen double pred_GDP_pc_Region = exp(pred_log_GDP_pc_Region) // back to dollars
correlate pred_log_GDP_pc_Region log_GDP_pc_Region // -> 0.925

Predicted and observed log income correlate 0.925 across all 5,258 region-years — the same fit as the main body.

B.4 Constructing the inequality indicators — `ineqdeco` (§7)

This is the biggest single saving. The post writes a from-scratch function for the Gini, the three generalized-entropy indices, and the coefficient of variation. Stata returns all of them from one ineqdeco call, optionally population-weighted. After the command, r(gini) is the Gini, r(gem1) is GE(−1), r(ge0) is the mean log deviation, r(ge1) is the Theil index, and the coefficient of variation is $\sqrt{2 \times r(\text{ge2})}$.

* --- One country-year, to see all five numbers at once: Germany 2010 -------
use "Table_2_data.dta", clear // region-year predicted income + population
ineqdeco pred_GDP_pc_Region [aw=Pop_Region] if Country_ISO=="DEU" & year==2010
display "Gini=" %5.4f r(gini) " Theil=" %6.4f r(ge1) " CV=" %5.4f sqrt(2*r(ge2))

For Germany in 2010 this returns Gini 0.0278, Theil 0.0016, CV 0.0565 — matching the worked example in §7.3. To build the whole country panel, loop once per country-year group and harvest the five returned scalars:

* --- Every country-year: one ineqdeco call per group, all five indices ------
egen _g = group(Country_ISO year)
quietly summarize _g, meanonly
local G = r(max)
foreach s in gini gem1 ge0 ge1 ge2 { // empty columns to fill
gen double _idx_`s' = .
}
quietly forvalues i = 1/`G' {
capture ineqdeco pred_GDP_pc_Region [aw=Pop_Region] if _g==`i'
if _rc==0 {
foreach s in gini gem1 ge0 ge1 ge2 {
replace _idx_`s' = r(`s') if _g==`i'
}
}
}
rename _idx_gini GINIW_pred_GDP_pc // population-weighted regional Gini
gen double COVW_pred_GDP_pc = sqrt(_idx_ge2 * 2)

These rebuilt indices line up with the published ones at a correlation of about 0.879, the same validation the post reports in §7.5.

B.5 The regional Kuznets curve — Table 3 (§8)

Average each country to five 5-year periods, then regress the regional Gini on a cubic in log national GDP per capita with country and period fixed effects. collapse does the averaging in one line; xtreg … , fe does the within-country regression.

* --- Step 1: collapse annual data to 5-year period means --------------------
use "Table_3_data.dta", clear
gen p5year = .
replace p5year = 1 if inrange(year,1990,1994)
replace p5year = 2 if inrange(year,1995,1999)
replace p5year = 3 if inrange(year,2000,2004)
replace p5year = 4 if inrange(year,2005,2009)
replace p5year = 5 if inrange(year,2010,2014)
collapse (mean) GINIW_pred_GDP_pc GDP_pc_Country, by(Country_ISO p5year)
* --- Step 2: build the cubic and fit country + period fixed effects ---------
gen lg = log(GDP_pc_Country)
gen lg2 = lg^2
gen lg3 = lg^3
encode Country_ISO, generate(cid)
xtset cid p5year
xtreg GINIW_pred_GDP_pc lg lg2 lg3 i.p5year, fe robust cluster(cid)

The cubic coefficients are 0.293 / −0.032 / 0.0011 (N ≈ 879 country-periods, 180 countries) — the same + / − / + sign pattern that traces the N-shaped spatial Kuznets curve in §8.

Drawing the curve (Figure 4) in two extra lines. Where the post refits a dummy model and computes partial residuals by hand, Stata’s factor-variable notation c.lg##c.lg##c.lg builds the cubic and tells margins that the three terms move together, so marginsplot bends the curve correctly:

* --- Step 3: the fitted curve, the Stata way --------------------------------
xtreg GINIW_pred_GDP_pc c.lg##c.lg##c.lg i.p5year, fe robust cluster(cid)
margins, at(lg=(5(0.5)12))
marginsplot, recast(line) recastci(rarea) ///
title("Regional Kuznets curve") xtitle("log GDP per capita") ///
ytitle("Predicted regional Gini")

B.6 Turning points and the discriminant (§9)

A cubic turns where its slope is zero, i.e. where $3\beta_3,\ell^2 + 2\beta_2,\ell + \beta_1 = 0$. Real turning points exist only if the discriminant $D = (2\beta_2)^2 - 4(3\beta_3)\beta_1$ is positive. We read the three coefficients straight out of _b[] and solve the quadratic:

* --- turning points from the stored cubic coefficients ---------------------
scalar b1 = _b[lg]
scalar b2 = _b[lg2]
scalar b3 = _b[lg3]
scalar D = (2*b2)^2 - 4*(3*b3)*b1 // discriminant
scalar lo = (-2*b2 - sqrt(D)) / (2*3*b3) // first turning point (log)
scalar hi = (-2*b2 + sqrt(D)) / (2*3*b3) // second turning point (log)
display "D=" D " ln=" %4.2f lo " ($" %1.0f exp(lo) ") ln=" %5.2f hi " ($" %1.0f exp(hi) ")"

The discriminant is positive ($D = +0.000035$), so there are two real turning points, at $\ell = 7.74$ (about \$2,287) and $\ell = 11.25$ (about \$77,206) — the same turning points as §9.

B.7 What drives regional inequality — Table 4 (§10)

The same 5-year panel and cubic as B.5, now adding a block of structural controls to each column. (Published column 4 needs a licensed ICRG variable that is not in the bundle, so it is omitted — as in the post.)

* --- determinants: keep the cubic, add one control block per column ---------
use "Table_4_data.dta", clear
* ... build p5year, collapse by(Country_ISO p5year), gen lg lg2 lg3, encode cid, xtset ...
* Natural-resource rents + arable land
xtreg GINIW_pred_GDP_pc lg lg2 lg3 Resources_rents_share_of_GDP Arable_land ///
i.p5year, fe robust cluster(cid)
* Ethnic inequality (published column 6)
xtreg GINIW_pred_GDP_pc lg lg2 lg3 GINIW_Eth_light i.p5year, fe robust cluster(cid)

Resource rents enter positively (+0.018) and arable-land share negatively (−0.053); ethnic inequality carries a coefficient of 0.071 (N ≈ 844) — the headline determinant the post highlights in §10.

B.8 Spatial robustness: Conley standard errors — Table B.4 (§11)

This re-estimates the within-region lights model (Table 1, column 2) but replaces the standard errors with spatially-robust Conley/Hsiang errors at three cutoff radii. The point estimate does not move — only the standard errors.

A caveat on packages: ols_spatial_HAC is not on SSC. It is Solomon Hsiang’s ado (Hsiang 2010, PNAS); copy ols_spatial_HAC.ado (and its helpers distance.ado, Tdiff.ado) into your personal ado folder (type sysdir to find it). The region fixed effects enter as explicit dummies because the command has no fe option.

* --- Conley spatial-HAC SEs at three radii (km) -----------------------------
use "Table_B4_data.dta", clear
tsset code_Coutry_Region year
xi i.code_Coutry_Region // region FE as explicit dummies
set matsize 11000
gen const = 1 // the command needs an explicit constant
foreach D in 1000 2500 5000 { // spatial-correlation cutoff radius
ols_spatial_HAC log_GDP_pc_Region log_Light_ppix_Region ///
_Icode_Cout_2-_Icode_Cout_1504 satyear_1-satyear_7 const, ///
lat(Latitude) lon(Longitude) t(year) p(code_Coutry_Region) ///
dist(`D') lag(0) bartlett
}

The coefficient is 0.190 in all three columns. The Stata Conley standard errors widen with the radius (about 0.020 / 0.025 / 0.027); the post’s Python Conley errors (0.026 / 0.034 / 0.037) are a touch larger because the two packages weight distances slightly differently. Either way the conclusion is identical: the light elasticity is not an artefact of spatial correlation — the naïve iid standard error is only about 0.013.

B.9 Regional versus personal inequality — Figure 5 (§12)

Finally, does inequality between regions track inequality between people? Average each country over 2001–2012, put both Ginis on the same 0–1 scale, and regress one on the other.

* --- cross-country regression of personal on regional inequality -----------
use "Figure_5_data.dta", clear
keep if year>2000 & year<2013
collapse (mean) GINIW_pred_GDP_pc Giniall, by(Country_ISO)
gen GINIall_100 = Giniall/100 // household Gini 0-100 -> 0-1
regress GINIall_100 GINIW_pred_GDP_pc // slope = 0.587 over n ~ 144
twoway (scatter GINIall_100 GINIW_pred_GDP_pc, msymbol(t)) ///
(lfit GINIall_100 GINIW_pred_GDP_pc), legend(off) ///
xtitle("Interregional inequality (GINIW)") ///
ytitle("Interpersonal inequality (Gini)")

Across 144 countries the slope is 0.587: places with wide gaps between regions also tend to have wide gaps between people — the same link the post finds in §12.

B.10 The same study, far fewer lines

Side by side, the contrast is the point of this appendix:

Task	Python (main body)	Stata (this appendix)
Prediction model	`linearmodels.RandomEffects` + a design-matrix helper	`xtreg … , re` (one line)
Five inequality indices	a hand-written function	`ineqdeco …` (one call)
Kuznets curve + plot	`statsmodels` refit + partial residuals	`xtreg` + `margins` + `marginsplot` (three lines)
Fixed ↔ random effects	two different libraries	swap `fe` ↔ `re`

Every headline number above — the 0.102 light elasticity, the 0.925 prediction correlation, the 0.293 / −0.032 / 0.0011 cubic, the 0.071 ethnic-inequality coefficient, the 0.587 regional-versus-personal slope — reproduces the Python result in the main body. The complete, runnable script is the stata_replication.do linked at the top of this post.

AI Podcast: Regional Inequality from Outer Space

Click play to load

0:00 0:00

Spatial Inequality and the Kuznets Curve: Parametric and Semiparametric Estimates in R

Sun, 14 Jun 2026 00:00:00 +0000

Abstract

Why do some countries have huge gaps between their richest and poorest regions while others are remarkably even? Lessmann (2014) revisits a classic idea from Kuznets (1955) and Williamson (1965): as countries develop, spatial inequality first rises, then falls — an inverted-U. This tutorial replicates that study in R on a synthetic dataset, so the entire data-generating process is open and reproducible. We simulate regional GDP per capita for 56 countries over 1980–2009, compute the population-weighted coefficient of variation (WCV) of regional income from those regions, and estimate the relationship with cross-section OLS, two-way fixed effects via fixest, and the Robinson (1988) and Baltagi–Li (2002) semiparametric estimators. The cross-section recovers a significant inverted-U with a high-income upturn — a cubic whose turning points sit at about \$2,100 and \$31,000 of GDP per capita — while the within-country panel shows a clean inverted-U (the cubic term is insignificant). Spatial inequality correlates with personal (Gini) inequality at about 0.32, and a sectoral channel — the non-agricultural share of output — reproduces the same curve. The practical lesson is that wide regional gaps are, to a first approximation, a transitional feature of development that tends to narrow as economies mature; for learners, the post is a hands-on tour of measurement, fixed effects, polynomial specification, and flexible semiparametric regression.

1. Overview

The case-study question. Is the link between spatial inequality and economic development an inverted-U — and does it turn N-shaped (rising again) at very high income? Lessmann (2014) assembled a hard-to-find panel of regional accounts to answer it. We cannot share that proprietary data, so we do the next best thing for teaching: build a synthetic world whose data-generating process reproduces the paper’s findings, and walk through every estimator on it.

Why does this matter? Wide regional gaps are not just an accounting curiosity. Interregional inequality often travels with ethnic and political tension, and in extreme cases raises the risk of internal conflict. Understanding when such gaps widen and when they close is directly useful for regional policy.

Learning objectives. By the end you will be able to:

Compute the weighted coefficient of variation (WCV) of regional income and explain why it is population-weighted.
Estimate polynomial OLS with heteroskedasticity-robust (White) standard errors and read an inverted-U off the coefficients.
Fit two-way fixed effects with fixest::feols, and explain why country and year fixed effects change the story.
Solve for the turning points of a cubic and convert them to dollar thresholds.
Read the Robinson and Baltagi–Li semiparametric partial-fit curves and say how they differ from a polynomial.

graph LR
A("Simulate regional GDP") --> B("Compute WCV")
B --> C("Cross-section OLS<br/>Table 2")
B --> D("Two-way FE<br/>Table 3")
C --> E("Turning points")
E --> J("Discriminant test")
C --> F("Robinson semiparametric<br/>Fig 4")
D --> G("Baltagi–Li semiparametric<br/>Fig 5")
B --> H("Sectoral channel<br/>Table 6")
B --> I("Robustness")
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class A,C,E,F,I blue
class B,J,H teal
class D,G orange

The pipeline above is the whole post in one picture: simulate regions, compute the inequality index, then estimate the development–inequality relationship four ways (parametric and semiparametric, cross-section and panel), and probe the sectoral channel and robustness.

Key concepts at a glance

The post reuses a small vocabulary. Each concept below has a definition (always visible) plus an example and analogy behind clickable cards — open them when a term feels slippery.

1. Weighted coefficient of variation (WCV). $\mathrm{WCV} = \frac{1}{\bar{y}}\left[\sum_{j} p_j,(\bar{y}-y_j)^2\right]^{1/2}$ — the population-weighted spread of regional GDP per capita, divided by the country mean. Scale-free, so it compares countries of any income level.

Example

A country with a rich capital (\$28,000, 35% of people) and a poorer hinterland (\$12,000, 65%) has WCV ≈ 0.43.

Analogy

Like a “spread score” for a class where bigger groups of students count more toward the average gap.

2. Inverted-U / Kuznets curve. The hypothesis that inequality rises with development, peaks, then falls — tracing an upside-down U.

Example

In §5 the quadratic gives a positive linear term and a negative squared term — the algebraic signature of an inverted-U.

Analogy

A roller-coaster hill: climb during industrialisation, crest, then descend as the modern economy spreads out.

3. Between vs within variation. Between compares different countries; within compares one country with itself over time. Cross-section regressions use between variation; panel fixed effects use within variation.

Example

The high-income upturn shows up between countries (§5) but vanishes within countries (§6) — the central contrast of the study.

Analogy

Comparing different students’ heights (between) vs tracking one student as they grow (within).

4. Two-way fixed effects (TWFE). Adding a dummy for every country and every year, so the income effect is identified only from within-country, within-year variation.

Example

feols(wcv ~ lnGDP + I(lnGDP^2) | country + year) — the | country + year part absorbs both sets of dummies.

Analogy

Grading each student against their own past, and against everyone’s average that semester — removing fixed advantages.

5. Polynomial specification. Entering income as $Y, Y^2, Y^3$ lets a straight-line model bend into curves — quadratic for an inverted-U, cubic for an N-shape.

Example

Column (5) of Table 2 adds $Y^3$; its positive coefficient produces the high-income upturn.

Analogy

Adding hinges to a ruler so it can follow a winding road instead of cutting straight across.

6. Turning points. The income levels where the curve changes direction — found by setting the derivative to zero: $\beta_1 + 2\beta_2 Y + 3\beta_3 Y^2 = 0$.

Example

Our cubic peaks at ln(GDP) ≈ 7.7 (≈ \$2,100) and troughs at ≈ 10.4 (≈ \$31,000).

Analogy

The crest and the valley of the roller-coaster — where the track is momentarily flat.

7. Semiparametric / partially-linear model. $\mathrm{WCV} = \alpha + f(Y) + \gamma X + \epsilon$: the controls $X$ enter linearly, but the income effect $f(Y)$ is an unknown smooth curve estimated from the data instead of forced into a polynomial.

Example

The Robinson estimator (§8) and the Baltagi–Li B-spline (§9) draw $f(Y)$ as a flexible curve with a confidence band.

Analogy

Tracing a coastline freehand instead of approximating it with a few straight rulers.

8. Omitted-variable bias. When a left-out factor correlated with both income and inequality distorts the estimated relationship; fixed effects defend against the time-invariant version of it.

Example

Geography (mountains, coasts) drives spatial inequality but is hard to measure; country fixed effects absorb all of it at once.

Analogy

Blaming coffee for poor sleep when it’s really the late-night screen time that travels with it.

9. Discriminant of the cubic. A single number, $D = \beta_2^2 - 3\beta_1\beta_3$, that tells you whether a fitted cubic has two real turning points ($D>0$), one inflection ($D=0$), or none ($D<0$). It is computed from the coefficients, so it answers “does the curve bend?” — a different question from “is each term significant?”

Example

In §7 the cross-section cubic has $D = +0.0055 > 0$ with both turning points in range (genuine N-shape), while the panel cubic’s implied turning points fall outside the data.

Analogy

A road can curve gently yet never actually turn back; the discriminant is the test for whether it makes a genuine U-turn or just leans.

2. Setup and the synthetic data-generating process

2.1 Packages and theme

We lean on fixest for fixed effects, np for the Robinson estimator, splines for the Baltagi–Li B-spline, sandwich/lmtest for White standard errors, and ggplot2 for dark-themed figures.

set.seed(123)
pacman::p_load(dplyr, tidyr, ggplot2, scales, patchwork, fixest, sandwich, lmtest,
splines, np, modelsummary, gt, webshot2, gridExtra)
options(np.messages = FALSE)

2.2 Simulating regional GDP and a country panel

The key design choice is that the WCV is computed, not assumed. For each of 56 synthetic countries we build a realistic territorial structure — the actual number of regions and land areas from the paper’s appendix — then draw regional GDP per capita and a population share for each region. We engineer two layers into the data: a within-country inverted-U (how a country’s regional spread evolves as it develops) and a between-country cubic that lives in a time-invariant country term. This separation is what lets the panel show a clean inverted-U while the cross-section shows the N-shape.

# region j in country i, year t: y_ijt = country_mean × exp(δ_it · z_ij)
# z_ij is a persistent regional "position" (a rich region stays rich);
# δ_it (the log-dispersion) follows the structural inverted-U in development.
delta <- sqrt(log(1 + target_wcv^2)) # lognormal-CV inversion
y_reg <- exp(lnGDP) * exp(delta * z - 0.5 * delta^2)

Simulating regional micro-data for 56 countries ...
annual obs N=890 | 5-year obs N=212 | cross-section N=56

The simulated panel has 890 annual observations, 212 five-year cells, and 56 countries in the cross-section — close to the paper’s 915 / 207 / 56. The unbalanced shape is deliberate: rich OECD economies have long, dense coverage; developing countries have short, gappy series, exactly as in the real data.

3. Measuring spatial inequality: the WCV

Lessmann measures spatial inequality with the population-weighted coefficient of variation of regional GDP per capita:

$$\mathrm{WCV}_{i,t} = \frac{1}{\bar{y}}\left[\sum_{j=1}^{n} p_j,(\bar{y} - y_j)^2\right]^{1/2}$$

where $\bar{y}$ is the country’s average regional GDP per capita, $y_j$ is region $j$’s GDP per capita, $p_j$ is region $j$’s share of the country’s population, and $n$ is the number of regions. The population weighting is the crucial feature: a tiny, very rich (or very poor) region barely moves the index, while a populous region counts a lot.

wcv_fun <- function(y, p) {
ybar <- sum(p * y) # population-weighted mean
sqrt(sum(p * (ybar - y)^2)) / ybar # weighted SD / mean
}
toy <- data.frame(gdp_pc = c(28000, 12000), pop_share = c(0.35, 0.65))
wcv_fun(toy$gdp_pc, toy$pop_share)

Worked WCV example: ybar = 17600, WCV = 0.434

A rich capital region at \$28,000 (35% of the population) and a poorer hinterland at \$12,000 (65%) give a population-weighted mean of \$17,600 and a WCV of 0.434. Because the larger, poorer region carries more weight, the index reflects how most people experience the regional gap — not just the extremes. Mapping the same calculation across all 56 synthetic countries reproduces the familiar geography of spatial inequality.

High-income North America and Europe show the lowest spatial inequality, while East Asia, Latin America and Sub-Saharan Africa show the highest — the cross-regional ranking Lessmann reports in Table 1.

4. Spatial vs personal inequality (Fig 3)

Before modelling development, it is worth asking how spatial inequality relates to the more familiar personal inequality (the household-income Gini). If they were the same thing, studying regions would add nothing.

fig3_fit <- lm(gini ~ wcv, cs)
coef(fig3_fit); cor(cs$gini, cs$wcv)

Fig 3: GINI = 0.311 + 0.208 * WCV (t = 2.45), corr = 0.316

The slope is positive and significant (0.208, t = 2.45) and the correlation is 0.316 — close to the paper’s 0.324. Spatial inequality explains a real but partial share of personal inequality: the two are related, not interchangeable. A country can have high personal inequality with low regional inequality (the United States) or the reverse (a small, ethnically split economy). That partial overlap is exactly why the rest of the post focuses on the spatial dimension in its own right.

5. Cross-section parametric estimates (Table 2)

We start where Williamson (1965) did: a cross-section of countries, using period means over 2000–2009. The estimating equation is a polynomial in development with controls:

$$\mathrm{WCV}_{i} = \alpha + \sum_{j=1}^{k}\beta_j,Y_{i}^{,j} + \gamma X_{i} + \epsilon_{i}$$

where $Y = \ln(\text{GDP per capita})$. An inverted-U needs $\beta_1 > 0$ and $\beta_2 < 0$. We use White (HC1) heteroskedasticity-robust standard errors to match the paper.

m1 <- lm(wcv ~ lnGDP, cs) # bivariate
m4 <- lm(wcv ~ lnGDP + I(lnGDP^2) + lnunits + lnarea + area_units +
ethnic + trade_gdp + urbanization + federal, cs) # full controls
m5 <- lm(wcv ~ lnGDP + I(lnGDP^2) + I(lnGDP^3) + lnunits + lnarea +
area_units + ethnic + trade_gdp + urbanization + federal, cs) # + cubic
lmtest::coeftest(m1, vcov = sandwich::vcovHC(m1, "HC1"))

CS (1) lnGDP -0.098*** -> -0.092***
CS (4) lnGDP/^2 +0.33*/-0.021* -> 0.338* / -0.020**
CS (5) cubic 3.86**/-0.45**/0.017** -> 4.40***/-0.499***/0.0184***
CS adjR2 0.43/0.66/0.69 -> 0.33/0.67/0.73

The five specifications tell a story. The bivariate slope is negative (−0.092***): on average, richer countries have lower spatial inequality — but a straight line hides the structure. Once the controls enter (column 4), the inverted-U emerges: the linear income term turns positive (+0.338*) and the squared term is negative (−0.020**). Adding a cubic (column 5) makes all three income terms significant — +4.40*** / −0.499*** / +0.0184*** — and the positive cubic coefficient reveals an upturn at very high income (the N-shape). Every control carries the expected sign: more trade and more regions raise spatial inequality, while federal constitutions and urbanisation lower it.

The scatter makes the algebra visual: the straight line slopes down, the quadratic bends into an inverted-U, and the cubic adds the high-income upturn among the richest economies. Interpretation: the same data support three different stories depending on the functional form — which is exactly why Lessmann reports all of them and then turns to semiparametric methods that do not force a shape.

6. Panel two-way fixed effects (Table 3)

6.1 Why fixed effects?

The cross-section compares different countries, so any unmeasured, time-invariant trait correlated with income — geography, history, ethnic geography — can bias the estimate. A panel lets us compare each country with itself over time and absorb all such traits with country dummies.

Each grey line is one country’s path; the coloured lines highlight China, India, Russia, Brazil, the United States and Bolivia. Countries sit at very different inequality levels for reasons unrelated to their income trajectory — and those level differences are precisely what country fixed effects remove.

6.2 The `fixest::feols` specification

The panel model adds country and year fixed effects:

$$\mathrm{WCV}_{i,t} = \beta_1 Y_{i,t} + \beta_2 Y_{i,t}^2 + \gamma X_{i,t} + \alpha_i + \mu_t + \epsilon_{i,t}$$

In fixest, the fixed effects go after a vertical bar, and vcov = "hetero" reproduces the paper’s White standard errors (clustering by country is the modern alternative):

fa2 <- feols(wcv ~ lnGDP + I(lnGDP^2) + trade_gdp + urbanization |
country + year, data = annual, vcov = "hetero")
fa3 <- feols(wcv ~ lnGDP + I(lnGDP^2) + I(lnGDP^3) + trade_gdp + urbanization |
country + year, data = annual, vcov = "hetero")

PAN (2) lnGDP/^2 0.345**/-0.018** -> 0.394**/-0.0211** ; cubic n.s. -> -0.0008

The within-country relationship is a clean inverted-U: the quadratic gives +0.394** and −0.0211**, matching the paper. Crucially, the cubic term is insignificant (−0.0008, t = −0.26): there is no high-income upturn within countries. This is the study’s central contrast — the upturn we saw in the cross-section is a between-country phenomenon (rich service economies differ from rich manufacturing ones), not something a single country experiences as it grows.

The fitted TWFE quadratic peaks around ln(GDP) ≈ 9.8 (~\$18,000): as a typical country develops past that point, its regional gaps start to close. Interpretation: fixed effects do not just tidy up standard errors — they change the substantive conclusion about whether the upturn is real for any given country.

6.3 Annual vs 5-year averages

Annual data can be noisy because of business cycles, so Lessmann also estimates on 5-year averages. We build them by grouping years into six periods and averaging within country-period cells; the inverted-U survives (5-year quadratic ≈ +0.34 / −0.019), confirming the result is not a short-run artefact.

7. Turning points and the discriminant test

A cubic can bend twice — but does it actually? And does it bend inside the range of incomes we observe? This section answers both. It is the most transferable skill in the post: any time you fit a cubic, these two checks tell you whether the curve really has the shape your coefficients seem to promise.

7.1 Calculating the turning points

Where does the curve change direction? At a turning point the slope is zero, so we set the derivative of the cubic to zero:

$$\frac{\partial \mathrm{WCV}}{\partial Y} = \beta_1 + 2\beta_2 Y + 3\beta_3 Y^2 = 0$$

This is a quadratic in $Y$, so it has at most two roots — the inverted-U peak and the high-income trough. One direct way to find them is polyroot:

bc <- coef(m5)
roots <- sort(Re(polyroot(c(bc["lnGDP"], 2*bc["I(lnGDP^2)"], 3*bc["I(lnGDP^3)"]))))
data.frame(ln_gdp = roots, gdp_usd = round(exp(roots)))

 ln_gdp gdp_usd
1 7.671 2146
2 10.356 31443

Spatial inequality rises with development up to ln(GDP) ≈ 7.7 (about \$2,100), falls until ln(GDP) ≈ 10.4 (about \$31,000), and then rises again. Interpretation 1: the first threshold marks the industrial take-off where a few leading regions surge ahead; the second marks the maturity where convergence has run its course and post-industrial forces (tertiarisation) begin to pull rich regions apart again. Because the regressor is $\ln(\text{GDP})$, we exponentiate each root to read it back in dollars.

7.2 The discriminant: does the curve really bend?

Computing the roots numerically works, but it hides why a cubic sometimes has two turning points and sometimes none. The quadratic $\beta_1 + 2\beta_2 Y + 3\beta_3 Y^2 = 0$ has two real solutions exactly when its discriminant is positive. After dropping a harmless factor of 4 (see the algebra below), the rule simplifies to a single number:

$$D ;\equiv; \beta_2^2 - 3,\beta_1\beta_3.$$

There are three regimes:

Discriminant	Real turning points	Shape over the real line	Verdict
$D > 0$	2	rise–fall–rise (an “N on its side”)	the cubic shape is real
$D = 0$	1 (inflection)	a single flat spot, no reversal	knife-edge boundary
$D < 0$	0	monotonic — never reverses	the cubic shape is not real

The standard quadratic-formula discriminant is $b^2-4ac = (2\beta_2)^2 - 4(3\beta_3)(\beta_1) = 4(\beta_2^2 - 3\beta_1\beta_3) = 4D$; the factor of 4 never changes the sign, so we work with $D = \beta_2^2 - 3\beta_1\beta_3$. When $D>0$, the turning-point locations come straight from the quadratic formula (then exponentiate to dollars):

$$Y^{\star} = \frac{-\beta_2 \pm \sqrt{D}}{3\beta_3}, \qquad \mathrm{GDP}^{\star} = \exp!\left(Y^{\star}\right).$$

In R the whole test is two short functions:

cubic_disc <- function(b1, b2, b3) b2^2 - 3 * b1 * b3 # the discriminant
cubic_tp <- function(b1, b2, b3) { # turning points (if any)
D <- cubic_disc(b1, b2, b3)
if (D <= 0) return("no real turning points (monotonic)")
sort(exp(c(-b2 - sqrt(D), -b2 + sqrt(D)) / (3 * b3))) # in GDP-per-capita units
}
bc <- coef(m5)
cubic_disc(bc["lnGDP"], bc["I(lnGDP^2)"], bc["I(lnGDP^3)"])

[1] 0.005519

Interpretation 2: the figure holds the linear and cubic coefficients fixed and changes only the squared term. A small change flips the regime: when $D<0$ the curve climbs monotonically, at $D=0$ it develops a single flat inflection, and once $D>0$ it bends into the genuine rise–fall–rise N-shape. The sign of one number — the discriminant — is what separates “a cubic that bends” from “a cubic that merely curves.”

7.3 Two checks, not one: significance is not shape

Here is the trap. In our cross-section, all three income terms are statistically significant (§5: $\beta_1=4.40^{***}$, $\beta_2=-0.499^{***}$, $\beta_3=0.018^{***}$). It is tempting to conclude “therefore the relationship is a genuine cubic with two turning points.” That inference is wrong as stated. Significance answers “does the data prefer keeping this term?”; it does not answer “does the fitted curve actually bend inside the income range we observe?” The discriminant — plus a check on where the turning points fall — answers the second question. Applying both checks to this project’s two cubics, and to three illustrative cases, makes the distinction concrete:

# applied to the cross-section cubic, the panel cubic, and three synthetic cases

 case b1 b2 b3 D regime in_range
1 Cross-section cubic (significant) 4.3965 -0.4988 0.018447 +0.0055 2 turning points (both in range) TRUE
2 Panel cubic (insignificant) 0.1875 0.0017 -0.000836 +0.0005 2 turning points (>=1 OUT of range) FALSE
3 Synthetic 5a: genuine N-shape 4.4000 -0.5000 0.018000 +0.0124 2 turning points (both in range) TRUE
4 Synthetic 5b: monotonic trap 4.4000 -0.4000 0.018000 -0.0776 monotonic (D<0) FALSE
5 Synthetic 5c: turns out of range 4.4000 -0.5000 0.001000 +0.2368 2 turning points (>=1 OUT of range) FALSE

Read the rows from top to bottom:

Cross-section cubic — $D = +0.0055 > 0$ and both turning points (\$2,146 and \$31,443) fall inside the observed income range (\$315–\$82,653). This is a genuine N-shape. Significance and shape agree.
Panel cubic — the within-country cubic term was insignificant (§6, $t=-0.26$), so it fails the first check already. Even taking its coefficients at face value, $D>0$ but one implied turning point sits at roughly \$0.0003 — absurdly far below any real economy — so the curve does not bend inside the observed range. Two independent reasons to reject a within-country N-shape, exactly matching §6’s clean inverted-U.
Synthetic 5b (the trap) — same sign pattern as the genuine case, only $\beta_2$ is a touch smaller in magnitude, and $D = -0.078 < 0$. The curve is monotonic everywhere. A cubic regression on such data could report all three terms as “significant” and still have no turning point at all.
Synthetic 5c — $D>0$, so two turning points exist mathematically, but they land at \$86 and an astronomically high income. Inside any realistic range the curve is monotonic. “Two turning points exist” would be technically true and practically misleading.

Interpretation 3: significance (does the data want the term?) and the discriminant-plus-range check (does the curve actually bend, and where?) are different questions, and you need both. Reporting “all three GDP terms are significant, so the curve is cubic” can be wrong in two distinct ways — the discriminant can be negative (5b), or the turning points can fall outside the data (5c). The honest workflow is: report the coefficients, compute $D$, and if $D>0$ confirm the turning points lie inside the observed income range before claiming an inverted-U or N-shape.

Aside (for Bayesian model averaging). The same trap appears with a different label. In a BMA, a term’s posterior inclusion probability (PIP) near 1.00 is the Bayesian analogue of “statistically significant” — it says the data prefer keeping the term. But a high PIP on the cubic term no more guarantees a genuine bend than a significant cubic coefficient does: you still compute $D = \beta_2^2 - 3\beta_1\beta_3$ from the posterior-mean coefficients and check the turning-point range. The companion note Turning Points and Discriminant Analysis (Mendez, 2026) works through real cases — cross-country CO₂ ($D>0$, genuine) versus Chinese provincial PM₂.₅ (PIPs ≈ 1.00 but $D<0$, monotonic) — that make the point with field data.

8. Semiparametric cross-section: the Robinson estimator (Table 4, Fig 4)

A polynomial forces a shape. A partially-linear model lets the income effect be any smooth curve while keeping the controls linear:

$$\mathrm{WCV} = \alpha + f(Y) + \gamma X + \epsilon$$

Robinson’s (1988) estimator is a clever two-step “double residual” idea: first partial $Y$ out of both the outcome and each control non-parametrically, then run OLS on the residuals to recover $\gamma$; finally smooth the leftover against $Y$ to draw $f$.

# Step 1: non-parametrically remove lnGDP from y and each control
resid_np <- function(v, z) residuals(npreg(v ~ z, regtype = "ll", ckertype = "gaussian"))
ey <- resid_np(cs$wcv, cs$lnGDP)
eX <- sapply(Xnames, function(nm) resid_np(cs[[nm]], cs$lnGDP))
# Step 2: OLS of residualised y on residualised X -> linear part (Table 4)
rob <- lm(ey ~ eX - 1)
# np::npplreg implements exactly this estimator and returns identical coefficients

 robinson_coef t npplreg_coef
lnunits 0.1650 3.9405 0.1575
trade_gdp 0.0021 4.0348 0.0020
urbanization -0.0057 -3.0047 -0.0056
federal -0.0670 -1.7456 -0.0525

The linear-part coefficients match the parametric estimates — more regions and more trade raise inequality, urbanisation and federalism lower it — and np::npplreg returns the same numbers, confirming the hand-built estimator. The flexible curve $f(Y)$ traces the inverted-U with a high-income upturn, and the 90% band widens at the sparse low-income end. Interpretation: because the curve was never told to be a cubic, its agreement with the parametric cubic is independent evidence that the N-shape is in the data, not an artefact of the polynomial.

9. Semiparametric panel: the Baltagi–Li series estimator (Table 5, Fig 5)

For the panel, Baltagi & Li (2002) remove the fixed effects and approximate $f(Y)$ with a cubic B-spline (order $k = 4$). We implement this faithfully in fixest: a B-spline basis of the income term, with country and year fixed effects absorbed.

B <- splines::bs(annual$lnGDP, degree = 3, df = 5) # cubic B-spline (order k=4)
colnames(B) <- paste0("bs", 1:5)
m_bl <- feols(wcv ~ bs1+bs2+bs3+bs4+bs5 + trade_gdp + urbanization |
country + year, data = cbind(annual, B), vcov = "hetero")

 estimate t within_r2
trade_gdp 0.0002 0.564 0.021 (annual)
urbanization -0.0027 -2.785 0.021 (annual)
urbanization -0.0029 -2.634 0.068 (5-year)

Trade is insignificant and urbanisation is significantly negative (−0.0027** annual, −0.0029** on 5-year averages), matching the paper’s Table 5. The recovered $f(Y)$ curves show the within-country inverted-U with no upturn at high income — the same message as the parametric panel, now without assuming a polynomial. Interpretation: two very different flexible methods (kernel-based Robinson and spline-based Baltagi–Li) agree with the parametric models, which is exactly the kind of triangulation that makes a descriptive finding credible.

10. The sectoral channel (Table 6)

Kuznets and Williamson argued that the real driver is structural change — the shift from agriculture to industry and services — with income just a proxy. We test this directly by replacing income with the non-agricultural share of gross value added.

s4 <- lm(wcv ~ nonag + I(nonag^2) + lnunits + lnarea + area_units +
ethnic + trade_gdp + urbanization + federal, cs)

== Table 6: sectoral data (non-agricultural GVA / GDP) ==
nonag = 0.0165*** nonag^2 = -0.00014***

Spatial inequality rises then falls with the non-agricultural share (+0.0165*** and −0.00014***) — an inverted-U in the structural variable itself. Interpretation: this is the mechanism behind the income result. As an economy industrialises, a few regions capture the new activity and gaps widen; as the modern sector matures and spreads, gaps narrow. Development raises inequality because it reshuffles where output is produced.

11. Robustness

11.1 Excluding the poorest countries

The rising arm of the curve depends on poor countries. Dropping those with GDP per capita below \$1,000 weakens the full inverted-U — the cubic no longer traces the complete shape, just as the paper finds.

11.2 Excluding capital regions

Capital regions are often far richer than the rest of a country. Recomputing the WCV without them and correlating with the original gives 0.84 (paper 0.81) — capitals matter in individual cases but do not overturn the cross-country picture.

11.3 Alternative inequality measures

Swapping the population-weighted WCV for the unweighted coefficient of variation or a regional Gini leaves the cubic in place — the inverted-U is not an artefact of the particular index.

11.4 Income in logs vs levels (Fig 7)

This is the most important caveat. With income in logs there is no high-income upturn; with income in levels a slight upturn reappears. Interpretation: the existence of the upturn is partly a measurement choice. The robust finding is the inverted-U; the N-shape is real but fragile — which is why Lessmann hedges it and why we should too.

12. Summary statistics (Table A.3)

The synthetic variables match the paper’s Table A.3 within about ±10% on every dimension: WCV mean 0.36 (paper 0.35), ln(units) mean 2.39, ln(area) mean 12.69, ethnic fractionalisation 0.31, Trade/GDP 82, urbanisation 69, federal share 0.21. Anchoring the marginal distributions to the paper is what makes the regression coefficients land in the right place.

13. Discussion

So, is there an inverted-U? On this synthetic data, calibrated to the paper, the answer is a clear yes — with a nuance. Between countries, the relationship is N-shaped: spatial inequality rises until about \$2,100 of GDP per capita, falls until about \$31,000, then edges up again. Within countries, the relationship is a clean inverted-U with no upturn. The two pictures are reconciled by recognising that the high-income upturn is a cross-sectional feature — rich service economies are simply more spatially unequal than rich manufacturing ones — rather than something a developing country marches through.

What does this mean for policy? Wide regional gaps are, to a first approximation, transitional: they tend to widen during industrial take-off and narrow as economies mature. That is cautiously good news, but the transition can take decades and the gaps can be politically dangerous while they last. The sectoral result points to the lever: because structural change drives the curve, investing in the human capital and connectivity of lagging regions can shorten the painful middle stretch.

14. Summary and next steps

The inverted-U is robust. Across parametric OLS, two-way fixed effects, and two semiparametric estimators, spatial inequality rises then falls with development.
The high-income upturn is fragile. It appears between countries and in levels, but vanishes within countries and under the log transform.
Fixed effects change the conclusion, not just the standard errors — the upturn is between-country, not within-country.
Structural change is the mechanism: the non-agricultural share reproduces the same curve.
Next steps. Re-run the simulation with a different seed to see sampling variability; cluster the panel standard errors by country; or extend the data window and test whether the second turning point moves.

15. Exercises

Re-seed the world. Change set.seed(123) to another value and re-run. Which coefficients are stable and which bounce around? What does that tell you about the fragility of the cubic?
Cluster the standard errors. Re-estimate the panel with vcov = ~country instead of "hetero". Do the quadratic terms stay significant? Why might clustering matter here?
Swap the measure. Replace wcv with the regional Gini (gini_reg) in the cross-section cubic. Does the inverted-U survive? What does that say about measurement robustness?
Apply the discriminant. A colleague fits a cubic and reports $\beta_1 = 4.4$, $\beta_2 = -0.40$, $\beta_3 = 0.018$, all significant. Compute $D = \beta_2^2 - 3\beta_1\beta_3$ by hand. Does the curve have two turning points? (Compare your answer with synthetic case 5b in §7.3.) Then halve $\beta_3$ and recompute — does the verdict change, and would you trust two turning points that fall at \$80 and \$10^{40}?

16. References

Lessmann, C. (2014). Spatial inequality and development — Is there an inverted-U relationship? Journal of Development Economics, 106, 35–51.
Kuznets, S. (1955). Economic growth and income inequality. American Economic Review, 45(1), 1–28.
Williamson, J. G. (1965). Regional inequality and the process of national development: A description of the patterns. Economic Development and Cultural Change, 13(4), 1–84.
Robinson, P. M. (1988). Root-N-consistent semiparametric regression. Econometrica, 56(4), 931–954.
Baltagi, B. H., & Li, D. (2002). Series estimation of partially linear panel data models with fixed effects. Annals of Economics and Finance, 3, 103–116.
Mendez, C. (2026). Turning Points and Discriminant Analysis — a note on why high posterior inclusion probabilities (or statistical significance) do not guarantee a genuine cubic shape.
Gravina, A. F., & Lanzafame, M. (2025). “What’s your shape?” A data-driven approach to estimating the Environmental Kuznets Curve. Energy Economics, 148.
Eicher, T. S., Papageorgiou, C., & Raftery, A. E. (2011). Default priors and predictive performance in Bayesian model averaging, with application to growth determinants. Journal of Applied Econometrics, 26(1), 30–55.
Bergé, L. (2018). Efficient estimation of maximum likelihood models with multiple fixed effects: the R package FENmlm. CREA Discussion Papers, 13.
Hayfield, T., & Racine, J. S. (2008). Nonparametric econometrics: the np package. Journal of Statistical Software, 27(5).

AI Podcast: Spatial Inequality and the Kuznets Curve

Click play to load

0:00 0:00

Do Industrial Parks Work? Evaluating Place-Based Policy in Ethiopia with Difference-in-Differences

Fri, 12 Jun 2026 00:00:00 +0000

Abstract

Governments across the developing world spend billions on industrial parks — fenced zones with serviced land, power and customs to lure factories — yet whether these place-based subsidies actually lift the surrounding economy, and who inside it benefits, remains hotly contested. This tutorial asks whether Ethiopia’s industrial parks raised local economic activity, urbanization, household living standards, and women’s economic agency, and how each effect can be measured credibly when parks are not placed at random. It replicates Huang, Wang & Xu (2026) on synthetic calibrated data combining a satellite district-year panel of 139 woredas observed annually over 2005–2020 (2,224 rows; 17 park-hosting woredas treated on a staggered 2008–2021 rollout against 122 propensity-score-matched never-treated controls) with two Ethiopia DHS repeated cross-sections — 13,200 households and 17,900 individuals across five survey rounds. It estimates a static two-way fixed-effects difference-in-differences and an event study with pyfixest, cross-checks them against the modern Sun-Abraham, Borusyak/Gardner and Callaway-Sant’Anna staggered estimators plus a Goodman-Bacon decomposition with diff-diff, and runs survey-weighted repeated-cross-section DiD with Conley spatial standard errors. A park raises inverse-hyperbolic-sine nighttime light by +0.215 (p < 0.01), the four staggered estimators agree within 0.046 units with 95.4% clean Bacon weight, and households gain durables (+0.229), housing (+0.248) and wealth (+0.383); crucially, average non-agricultural employment is insignificant (+0.091) yet the female effect is large (+0.140, p < 0.01). These findings imply that well-sited parks can reshape a local economy and women’s lives, but only a sex-disaggregated analysis reveals it.

1. Overview

Industrial parks are one of the most popular instruments of modern development policy. The recipe is simple: the government clears a tract of land, installs power, water, roads and a one-stop customs office, and rents serviced plots to manufacturers — usually in textiles, garments or leather. Ethiopia bet heavily on this model, opening more than twenty parks across eighteen districts between 2008 and 2021. The hope was that factories would cluster, create jobs, and pull a largely rural region into a wage economy. But place-based subsidies are controversial precisely because they might do little more than relocate activity that would have happened anyway — or light up a fenced enclave while the surrounding districts see nothing.

So the question this post tackles is genuinely two-sided: do industrial parks raise local economic activity, and — just as important — for whom? A park could boost satellite-measured luminosity yet leave household living standards flat. It could create jobs on average, yet only for men. Measuring this credibly is hard, because the government did not flip a coin to decide where parks go — it chose districts near cities and roads, which were already growing faster. We need a research design that nets out those pre-existing differences, and that handles a staggered rollout where parks opened in different years. That design is difference-in-differences (DiD), and the modern staggered-robust toolkit built around it.

Why not just one DiD regression? Because the workhorse two-way fixed-effects (TWFE) estimator can mislead under staggered timing: it secretly uses already-treated districts as controls for later-treated ones, a “forbidden comparison” that can flip the sign of the estimate when effects grow over time. A central goal of this tutorial is to show that worry being checked rather than ignored — we run four estimators side by side and decompose exactly where the TWFE number comes from. The estimand throughout is the average treatment effect on the treated (ATT) — the effect on the districts (and people) that actually got a park — identified under a parallel-trends assumption, in an explicitly observational setting where the parks were not randomly placed.

A note on the data (please read this). This tutorial replicates Huang, Wang & Xu (2026), but it runs on synthetic data built for teaching. The paper’s real inputs (harmonized nighttime lights, the GISD30 impervious-surface product, confidential Ethiopia DHS micro-data, the official park list) are licensed or restricted. Our dataset is calibrated so that re-running the paper’s analyses reproduces its findings — the signs, the significance stars, and the approximate magnitudes of the key coefficients. Most results track the paper closely; a handful of magnitudes differ, and we tabulate exactly which in Section 13. Use this to learn the methods, not to draw new conclusions about Ethiopia.

1.1 Learning objectives

By the end of this tutorial, you will be able to:

Frame a staggered place-based policy as a quasi-experiment, and explain why a treated-vs-never-treated comparison identifies the ATT under parallel trends.
Estimate a static two-way fixed-effects difference-in-differences and a dynamic event study on satellite outcomes with pyfixest.
Compare TWFE against the modern Sun-Abraham, Borusyak/Gardner and Callaway-Sant’Anna estimators, and diagnose the staggered negative-weights problem with a Goodman-Bacon decomposition using diff-diff.
Apply survey-weighted repeated-cross-section DiD to DHS household welfare and individual employment, and read a heterogeneity split that turns a null average into a sharp finding.
Defend your inference when treatment is spatially clustered, using Conley spatial-HAC standard errors and restricted-control-pool checks.

1.2 Study design

The diagram below maps the whole tutorial. Three data streams flow into one DiD design, that design is estimated by an escalating ladder of estimators, and the estimates answer three outcome families. Read it left to right: the staggered park rollout splits woredas (Ethiopia’s local districts) into treated and never-treated; we observe satellite, household, and individual outcomes; we climb from a naive 2×2 to the modern staggered-robust estimators; and we report effects on activity, welfare, and women’s empowerment.

graph LR
subgraph DATA["Three data streams"]
A("<b>Satellite</b><br/>district x year<br/>panel")
B("<b>DHS household</b><br/>repeated<br/>cross-section")
C("<b>DHS individual</b><br/>repeated<br/>cross-section")
end
subgraph DESIGN["DiD design"]
D("<b>Staggered rollout</b><br/>17 treated woredas<br/>vs 122 controls")
end
subgraph LADDER["Estimator ladder"]
E("Naive 2x2")
F("Static TWFE<br/>+ event study")
G("Sun-Abraham /<br/>Borusyak /<br/>Callaway-Sant'Anna")
end
subgraph OUT["Outcome families"]
H("<b>Activity</b><br/>lights, impervious")
I("<b>Welfare</b><br/>durables, wealth")
J("<b>Empowerment</b><br/>female jobs, agency")
end
A --> D
B --> D
C --> D
D --> E --> F --> G
F --> H
B --> I
C --> J
style DATA fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style DESIGN fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style LADDER fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style OUT fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A,B,C blue
class D orange
class E,F,H,I gray
class G,J teal

The key idea the diagram encodes is that one estimand — the ATT — threads through everything. The naive 2×2 is the cartoon version; TWFE and its event-study view are the workhorse; and the three modern estimators are the robustness insurance that the workhorse has not been led astray by staggered timing. Each box maps onto a section below, and the gender finding (the teal “Empowerment” box) is where the analysis lands.

1.3 Where are the industrial parks located?

Ethiopia placed its parks deliberately — clustered around the capital, Addis Ababa, and the main transport corridors, yet reaching into peripheral regions of the country. Before we build any statistical machinery, it helps to see the real geography we are modeling.

Source: Appendix Figure A2 in Huang, Wang & Xu (2026), “The socioeconomic impacts of industrial parks in Ethiopia.” The map shows the paper’s real park locations for geographic context; this tutorial’s analysis runs on synthetic data calibrated to reproduce the paper’s results.

That deliberate clustering near cities and roads is exactly the kind of non-random placement our design has to handle — so before estimating anything, the next section pins down the vocabulary that makes the treated-versus-control comparison credible.

2. Key concepts

The post leans on a small vocabulary repeatedly, and the later sections assume you can move between these terms quickly. Each concept below has three parts. The definition is always visible; the example and analogy sit behind clickable cards — open them when you need them, leave them closed for a quick scan. If a later section mentions “forbidden comparisons” or “repeated cross-section” and the term feels slippery, this is the section to re-read.

1. Staggered difference-in-differences. Units adopt treatment at different times, not all at once. We compare the change in outcomes for a treated group to the change for a not-yet-treated or never-treated group. With many adoption dates, the design is a stack of overlapping 2×2 comparisons.

Example

Ethiopia’s parks open across eight cohorts: 1 woreda in 2008, then 2 in 2014, 2 in 2015, 3 in 2016, 3 in 2017, 2 in 2018, 2 in 2019, and 2 in 2020 — 17 treated woredas in total, each turning on in its own year.

Analogy

A city installs streetlights block by block over a decade. To judge their effect you cannot just compare “before any lights” to “after all lights” — you must line up each block against its own opening date and a block that never got lit.

2. Parallel trends. The identifying assumption of DiD: absent the park, treated and control woredas would have followed the same path on average. Their levels can differ; their trends must match. We cannot prove it, but a flat pre-treatment event study makes it credible.

Example

The four pre-opening event-study leads run from −0.0275 to −0.0013 and the largest absolute t among them is just 2.17 — close enough to flat to read as parallel trends holding before the parks open.

Analogy

Two boats on the same current sit at different points but drift in step. Only an engine — the treatment — should make one pull ahead. If they were already diverging before the engine fired, the comparison is broken.

3. ATT $E[Y_i(1) - Y_i(0) \mid D_i = 1]$. The Average effect of the Treatment on the Treated — the effect on the districts that got a park, not on a random district. DiD, TWFE, and all three modern estimators here target the ATT, not the population-wide ATE.

Example

The +0.215 light effect is the ATT for the 17 park woredas. It does not promise that placing a park in any random district would raise its lights that much — only that these districts, given these parks, ended up that much brighter.

Analogy

The bonus speed measured on the car that actually got the new engine — not a promise about any car you might pick off the street.

4. TWFE bias, negative weights, and forbidden comparisons. Under staggered timing, the two-way fixed-effects regression quietly uses already-treated units as controls for later-treated ones. Those “forbidden” comparisons can get negative weights and bias — even flip the sign of — the estimate when effects grow over time.

Example

Here the danger is tiny: the Goodman-Bacon decomposition shows the forbidden later-vs-earlier comparisons carry only 1.21% of the total weight (and average +0.0135), while clean treated-vs-never comparisons carry 95.42%.

Analogy

Grading a class on a curve where some students were secretly given the exam early and then used as the “average” everyone else is scored against. If only a couple of students got the early peek, the curve is barely distorted — which is the situation here.

5. Event study. Instead of one ATT, estimate one coefficient per year-relative-to-opening (event time $k$). Plotting them shows the dynamic path: flat leads before opening (no anticipation) and rising lags after (the effect building up).

Example

The light effect is +0.115 the year a park opens ($k = 0$), climbs to +0.193 at $k = +1$ and +0.219 at $k = +2$, and plateaus at +0.484 by $k = +4$ — a slow build, not an instant jump.

Analogy

A medical chart that plots a patient’s temperature day by day around the start of a drug, rather than reporting a single before/after average. The shape of the curve tells you when and how the drug works.

6. Repeated-cross-section DiD. When each survey round interviews different households (no panel key), you cannot use household fixed effects. The effect is identified off district × round group means: compare treated vs control districts before vs after their park opens, absorbing district and region×round fixed effects.

Example

The DHS data are five rounds (2000, 2005, 2011, 2016, 2019) of fresh respondents. So the household regression uses | district_id + region_id^survey_round — district and region-by-round fixed effects — with no household effect, and only coarse event phases $\{-3, …, +1\}$.

Analogy

Polling a city’s mood with a fresh sample of pedestrians each year. You cannot track any one person over time, but you can still compare how neighborhoods shifted relative to each other.

7. Survey weights and clustered/Conley standard errors. The DHS is a complex sample, so regressions are weighted by the sampling weight. Standard errors are clustered on district (allowing a district’s errors to correlate over time) and, for the satellite panel, hardened with Conley spatial-HAC errors that also allow nearby districts to correlate.

Example

For the light ATT the cluster SE (0.0792) and the Conley-HAC SE (0.0799) are nearly identical and 2.43× the naive HC0 SE (0.0329) — yet the +0.215 estimate stays significant at t = 2.69.

Analogy

Counting a milling crowd. If everyone keeps shuffling between seats, you have far fewer truly independent heads than the rows suggest — honest standard errors admit that.

8. SUTVA and spillovers. The stable-unit-treatment-value assumption says one unit’s treatment does not affect another’s outcome. If a park lifts its neighbours’ lights, the never-treated controls are contaminated and the ATT is biased. A nearby test checks for exactly this leakage.

Example

The nearby coefficient (control districts within 10 km of a park) is +0.0648 and insignificant (t = 1.06), while the host effect stays +0.2712 — no measurable spillover, so SUTVA is plausible here.

Analogy

Testing whether a new factory’s smoke drifts onto the neighbouring farm. If the farm’s crops are unchanged, you can fairly use it as a clean comparison for the factory’s own land.

3. Setup and the two star libraries

Two specialist packages do the heavy lifting, and each gets a one-line introduction the first time it appears:

pyfixest runs fixed-effects regressions with a fast, Stata-flavored formula syntax: everything left of the | is estimated, everything right of it is absorbed as fixed effects. It also ships an event_study helper with the modern saturated (Sun-Abraham) and did2s (Borusyak/Gardner) estimators built in.
diff-diff is a teaching-oriented package for difference-in-differences. We use its DifferenceInDifferences, CallawaySantAnna, and BaconDecomposition classes — the last two are exactly the staggered-robust tools this post needs.

# In Colab, install the two estimation libraries first:
# !pip install pyfixest==0.50.1 diff-diff==3.5.2
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pyfixest as pf
import diff_diff as dd
np.random.seed(42) # reproducibility
# Site dark-theme palette for figures
STEEL_BLUE, WARM_ORANGE, TEAL = "#6a9bcc", "#d97757", "#00d4c8"
DARK_NAVY, GRID_LINE, LIGHT_TEXT = "#0f1729", "#1f2b5e", "#c8d0e0"

The satellite specifications need two small design helpers. The first builds the staggered first_treat column the modern estimators require: treated woredas get their park’s opening year, and never-treated controls get 0 — not NaN, because a missing value would silently drop the 122 controls that every staggered estimator needs as its clean comparison group. The second builds the “with-trends” interactions that absorb the faster pre-existing urban trend of treated woredas (more on why in Section 6).

def add_first_treat(d):
"""Treated woredas get their open_year; never-treated controls get 0."""
out = d.copy()
out["first_treat"] = out["open_year"].fillna(0).astype(int)
return out
def add_trend_terms(d):
"""Centre time at 2012 and interact it with 2007 baseline characteristics,
so each woreda can follow its own linear trend (the paper's even columns)."""
out = d.copy()
out["t"] = out["year"] - 2012
for c in ["urbanization_rate_2007", "employment_rate_2007",
"log_pop_density_2007", "share_christian_2007", "share_amharic_2007"]:
out[f"t_{c}"] = out["t"] * out[c]
return out
TREND_TERMS = ["t_urbanization_rate_2007", "t_employment_rate_2007",
"t_log_pop_density_2007", "t_share_christian_2007",
"t_share_amharic_2007"]

With the tooling in place, the next step is to load the three data layers and understand why they are structured so differently.

4. The three datasets

Evaluating a place-based policy forces a measurement choice to the surface. National statistics would barely flinch at a few new factories, so we need sub-national data — and at three different grains. We load all three straight from the post’s data folder on GitHub, so the code runs unchanged in Colab.

BASE = ("https://raw.githubusercontent.com/cmg777/starter-academic-v501/"
"master/content/tutorials/python_did_industrial_park/data/")
district = pd.read_csv(BASE + "industrial_park_district_panel.csv")
household = pd.read_csv(BASE + "industrial_park_household_rcs.csv")
individual = pd.read_csv(BASE + "industrial_park_individual_rcs.csv")
print("district panel :", district.shape)
print("household RCS :", household.shape)
print("individual RCS :", individual.shape)
print("treated woredas:", district.loc[district.treated == 1, "district_id"].nunique())
print("control woredas:", district.loc[district.treated == 0, "district_id"].nunique())

district panel : (2224, 34)
household RCS : (13200, 13)
individual RCS : (17900, 22)
treated woredas: 17
control woredas: 122

The three layers have fundamentally different structures, and that distinction drives every downstream choice. The district layer is a balanced panel — 139 woredas × 16 years (2005–2020) = 2,224 rows — so it supports a genuine panel event study with annual event time. The household and individual layers are repeated cross-sections: five DHS rounds of different respondents (13,200 households and 17,900 individuals), with no within-respondent panel key, so they admit only coarse event phases and survey-weighted regressions, never unit fixed effects. The treatment split is small on the treated side — 17 park woredas against 122 matched controls — which is exactly why several effects below are borderline and why honest standard errors matter.

4.1 The staggered rollout

The single feature that makes this a staggered design is that parks opened in different years. Tabulating the treated woredas by opening year shows the cohort structure that every modern estimator below keys on.

cohorts = (district[district.treated == 1]
.drop_duplicates("district_id")
.groupby("open_year").size())
print(cohorts.rename("n_treated_woredas").to_string())
print("total treated:", int(cohorts.sum()))

open_year
2008 1
2014 2
2015 2
2016 3
2017 3
2018 2
2019 2
2020 2
total treated: 17

The rollout is genuinely staggered: a single anchor woreda opens in 2008 (the Eastern Industrial Park), then the main build-out runs 2014–2020 with two to three woredas per year. This spread is what makes a naive before/after impossible — there is no single “before” — and what makes the staggered-robust estimators in Section 6 necessary rather than decorative. It also guarantees that every event time has at least three treated woredas behind it, so the dynamic path is estimated off real data at each lag.

4.2 The outcomes, and a transparent word on the data

The satellite layer carries two outcomes: ihs_light, the inverse hyperbolic sine of nighttime luminosity (a log-like transform that handles zeros), and impervious_ratio, the share of a woreda’s land that is built-up surface, observed only every five years. The household layer carries durable goods per capita, a housing-quality indicator, and the standardized wealth index. The individual layer carries non-agricultural employment plus, for women, decision-making power, savings-account ownership, and acceptance of domestic violence.

for col, layer, df in [("ihs_light", "district", district),
("durable_goods_pc", "household", household),
("nonag_employment", "individual", individual)]:
s = df[col]
print(f"{col:18s} ({layer:10s}) N={s.notna().sum():6d} "
f"mean={s.mean():.3f} sd={s.std():.3f}")

ihs_light (district ) N= 2224 mean=0.352 sd=0.715
durable_goods_pc (household ) N= 12207 mean=0.308 sd=0.487
nonag_employment (individual ) N= 17219 mean=0.343 sd=0.475

These means anchor every magnitude that follows. Durable goods average 0.308 items per capita, so the +0.229 ATT we find later is a ~74% lift off that base; non-agricultural employment averages 0.343, so a +0.140 effect for women is a large move. Before modeling, though, one caveat must be stated plainly: the data are synthetic. The data-generating process was tuned so that re-running the paper’s regressions recovers its coefficients (within about 0.02 on the headline cells), with the same signs and stars; spatial and serial shocks were injected so the standard errors behave realistically without moving the point estimates. We hold ourselves to that in Section 13. With the measurement settled, let us look at the data before regressing it.

5. Exploratory analysis: the case for parallel trends

Good causal work looks at the data before it models it. The first and most important view plots treated and control group-mean light over time — the picture difference-in-differences was invented for. One subtlety drives how we draw it: because of the synthetic bright-base device (treated park-cities are modelled as intrinsically much brighter than rural controls, a level difference the district fixed effect absorbs), plotting raw light levels would put the two groups miles apart and hide the trends. So we plot light indexed to each group’s own pre-2008 mean — baseline-normalized — which makes the “matched-then-diverge” picture read correctly.

# baseline-normalize each group's mean light to its pre-2008 average
g = (district.assign(grp=np.where(district.treated == 1, "Treated", "Control"))
.groupby(["grp", "year"])["ihs_light"].mean().reset_index())
base = g[g.year < 2008].groupby("grp")["ihs_light"].mean()
g["normed"] = g.apply(lambda r: r.ihs_light - base[r.grp], axis=1)
# (full dark-theme styling is in script.py)

Indexed to each group’s pre-2008 mean, the treated and control series sit on top of each other through the pre-rollout era — in 2008 the treated group is at −0.0018 and the control group at −0.0030, essentially identical, the visual signature of parallel trends holding before treatment turns on. From the 2014 build-out onward the treated series climbs steadily (+0.083 in 2014 → +0.186 in 2016 → +0.244 in 2017 → +0.237 in 2020) while the controls hover around zero with no trend. The eye already sees a matched pair of groups that diverge only after the parks open; the rest of the post is about measuring that divergence and trusting the measurement.

The staggered structure is easier to see one cohort at a time. The “staircase” figure traces each opening-year cohort’s mean light against the flat never-treated baseline.

Each cohort turns up at its own opening year — the 2016 cohort lifts off in 2016, the 2018 cohort in 2018 — while the never-treated line stays flat and even drifts down slightly, sharpening the contrast. This is the staggered design made visual: there is no single treatment date, so any honest estimator must align each cohort to its own clock. The next view confirms a second design fact — that treatment is not scattered randomly across the map.

Plotting the 17 treated woredas (orange) and 122 controls (blue) by longitude and latitude shows the treated units are spatially clustered, not randomly sprinkled — parks went to a handful of regions near cities and roads. Clustered treatment means a regional shock could hit several treated woredas at once, so their errors are unlikely to be independent. That is precisely the problem Conley spatial standard errors fix in Section 11. Finally, a distributional view shows the bright-base device head-on.

The boxplots split IHS light by group and pre/post period. Treated woredas sit far above controls in level — the synthetic bright base — and shift up further after opening, while controls barely move. The large level gap looks alarming but is harmless: the district fixed effect absorbs any time-invariant brightness, leaving the DiD coefficient untouched. With the intuition built, we can put the first number on the table.

6. From a naive 2×2 to the static TWFE ATT

6.1 The naive 2×2 (and why it understates the effect)

The simplest possible estimate collapses the whole staggered design at the median opening year (2017), forms four treated/control × pre/post cell means, and takes the difference of differences. diff-diff’s DifferenceInDifferences class returns it with a standard error.

d = district.copy()
d["post"] = (d.year >= 2017).astype(int) # collapse at the median opening year
cells = d.groupby(["treated", "post"])["ihs_light"].mean().unstack("post")
print(cells.round(4))
res = dd.DifferenceInDifferences(cluster="district_id").fit(
d, outcome="ihs_light", treatment="treated", time="post")
print(f"\nDiD ATT = {res.att:+.4f} (SE {res.se:.4f}, p = {res.p_value:.4f})")

post 0 1
treated
0 0.0990 0.0909
1 2.1308 2.3237
DiD ATT = +0.2011 (SE 0.0885, p = 0.0232)

Treated light rises +0.1929 post-opening while controls fall −0.0082, so the difference-in-differences is +0.2011 (SE 0.0885, p = 0.0232) — significant at 5%, with the by-hand and diff-diff estimates agreeing to four decimals. But this blended 2×2 understates the dynamic effect: the park’s impact ramps up over roughly five years (the event study below reaches +0.48), so averaging the small early post-years with the large late ones pulls the mean toward 0.20. It also leans on the Goodman-Bacon “forbidden comparisons” we worry about under staggering. The fix is to let the effect vary over time and to absorb confounders with fixed effects.

6.2 The static TWFE difference-in-differences

The workhorse specification adds two-way fixed effects. For woreda $d$ in year $t$:

$$Y_{dt} = \beta \, D_{dt} + \alpha_d + \gamma_{r(d),t} + \varepsilon_{dt}$$

In words, this says that the outcome $Y_{dt}$ (here ihs_light) equals a park effect $\beta$ times the treatment indicator $D_{dt}$ (the treatment column, which is 1 once a woreda’s park is open), plus a woreda fixed effect $\alpha_d$ that absorbs anything permanent about a district (including its bright base), plus a region-by-year fixed effect $\gamma_{r(d),t}$ that absorbs shocks common to a whole region in a given year, plus noise $\varepsilon_{dt}$. The coefficient $\beta$ is the ATT — the average park effect on the treated woredas. The “with-trends” specification adds the t_* interactions to let each woreda follow its own linear trend. In pyfixest, the part after the | lists the fixed effects to absorb:

dt = add_trend_terms(district)
out_rows = []
for ycol, label in [("ihs_light", "IHS night-light"),
("light_intensity", "Raw night-light"),
("impervious_ratio", "Impervious ratio")]:
m0 = pf.feols(f"{ycol} ~ treatment | district_id + region^year",
data=dt, vcov={"CRV1": "district_id"})
m1 = pf.feols(f"{ycol} ~ treatment + " + " + ".join(TREND_TERMS) +
" | district_id + region^year",
data=dt, vcov={"CRV1": "district_id"})
out_rows.append((label, m0.coef()["treatment"], m0.se()["treatment"],
m1.coef()["treatment"], m1.se()["treatment"]))
for label, b0, se0, b1, se1 in out_rows:
print(f"{label:18s} no-trends {b0:+.4f} ({se0:.4f}) "
f"with-trends {b1:+.4f} ({se1:.4f})")

IHS night-light no-trends +0.2704 (0.1007) with-trends +0.2152 (0.0833)
Raw night-light no-trends +1.7316 (0.4807) with-trends +1.6181 (0.4540)
Impervious ratio no-trends +0.0292 (0.0042) with-trends +0.0263 (0.0037)

The static TWFE regression recovers the paper’s headline: a park raises IHS nighttime light by +0.2152 with trend interactions (SE 0.0833, t = 2.58, significant at 1%) and +0.2704 without them — roughly a 21–27% increase in luminosity, since the IHS coefficient reads approximately as a proportional change at these magnitudes. The drop from 0.27 to 0.21 when trends are added is a textbook differential-trend confound: treated woredas were already more urban in 2007 and trending up faster, so the time × urbanization interaction absorbs that slope and the with-trends estimate is the cleaner ATT. The impervious-surface ratio rises +0.0263 with trends (SE 0.0037, t = 7.07) — about 2.6 percentage points of built-up land, ~82% of its 0.032 mean, and the most precisely estimated satellite coefficient in the study. The raw-light coefficient runs high (+1.618 vs the paper’s 1.276), a documented synthetic artifact of the bright-base device that we flag again in the reproduction audit. With a static ATT in hand, we unfold it across event time.

7. The event study: the dynamic path

A single ATT hides when the effect arrives. The event study estimates one coefficient per year-relative-to-opening, normalized to the year before opening ($k = -1$). For woreda $d$ in year $t$, with cohort opening year $g$:

$$Y_{dt} = \sum_{k \neq -1} \delta_k \, \mathbf{1}[t - g = k] + \alpha_d + \gamma_{r(d),t} + \varepsilon_{dt}$$

In words, this says we replace the single treatment dummy with a set of dummies, one for each event time $k$ (years since the park opened), each carrying its own coefficient $\delta_k$. The pre-opening coefficients ($k < 0$) should hug zero if parallel trends and no-anticipation hold; the post-opening coefficients ($k \geq 0$) trace how the effect builds. Here $\mathbf{1}[t - g = k]$ is an indicator equal to 1 when woreda $d$ is exactly $k$ years from its own opening, $\alpha_d$ and $\gamma_{r(d),t}$ are the same fixed effects as before, and the omitted $k = -1$ is the reference. We estimate the clean leads and lags with pyfixest’s saturated (Sun-Abraham) estimator, whose .aggregate() collapses the cohort dimension to one effect per $k$.

df = add_first_treat(district)
m = pf.event_study(df, yname="ihs_light", idname="district_id", tname="year",
gname="first_treat", estimator="saturated", att=True)
es = m.aggregate().reset_index()
es["event_time"] = es["period"].astype(float)
es = es[(es.event_time >= -5) & (es.event_time <= 5)].sort_values("event_time")
print(es[["event_time", "Estimate", "Std. Error", "Pr(>|t|)"]].round(4).to_string(index=False))

 event_time Estimate Std. Error Pr(>|t|)
-5.0 -0.0139 0.0176 0.4288
-4.0 -0.0013 0.0138 0.9226
-3.0 -0.0275 0.0127 0.0304
-2.0 -0.0135 0.0077 0.0791
0.0 0.1153 0.0295 0.0001
1.0 0.1928 0.0422 0.0000
2.0 0.2187 0.0641 0.0006
3.0 0.3138 0.0880 0.0004
4.0 0.4844 0.0463 0.0000
5.0 0.4697 0.0712 0.0000

The figure tells the whole story in one arc. The four pre-opening leads hug zero — they range from −0.0275 to −0.0013 and the largest absolute t among them is just 2.17 — weak enough to read as a flat pre-trend rather than a violation. The jump comes strictly after opening: the effect is already +0.1153 at $k = 0$ (p = 0.0001), climbs through +0.1928 ($k = +1$) and +0.2187 ($k = +2$), and plateaus at +0.4844 ($k = +4$) and +0.4697 ($k = +5$). This rising-then-flattening dynamic is exactly why the naive 2×2 (+0.2011) understated the long-run ATT — it averaged the small early years with the large late ones. The flat pre-period is the central piece of suggestive support for parallel trends, though it is never a proof, since the assumption concerns the unobserved post-period counterfactual. A skeptic might still worry the +0.215 TWFE headline is an artifact of staggered timing; the next section confronts that worry directly.

8. Modern staggered estimators: the negative-weights teaching moment

Here is the worry stated precisely. Under staggered adoption, the TWFE regression does not only compare treated woredas to never-treated ones. It also uses already-treated woredas as controls for later-treated ones — a “forbidden comparison.” When treatment effects grow over time (as ours clearly do, from +0.12 to +0.48), those forbidden comparisons receive negative weights and can bias TWFE, in extreme cases flipping its sign. The fix is a generation of estimators — Sun-Abraham, Borusyak/Gardner, and Callaway-Sant’Anna — that only ever compare treated cohorts to clean (not-yet- or never-treated) controls. Each targets the same ATT; if they agree with TWFE, the negative-weights problem is not biting.

def stars(t):
"""Significance stars from a t-stat (10% / 5% / 1%)."""
a = abs(t)
return "***" if a > 2.576 else "**" if a > 1.960 else "*" if a > 1.645 else ""
def cell(b, se):
"""Format a regression cell like '+0.2699*** (0.1005)'."""
return f"{b:+.4f}{stars(b / se)} ({se:.4f})"
df = add_first_treat(district)
Y = "ihs_light"
# TWFE benchmark
m_twfe = pf.event_study(df, yname=Y, idname="district_id", tname="year",
gname="first_treat", estimator="twfe", att=True)
twfe_b, twfe_se = m_twfe.coef().iloc[0], m_twfe.se().iloc[0]
# Sun-Abraham (saturated): average the clean post-period (k = 0..5) effects
m_sa = pf.event_study(df, yname=Y, idname="district_id", tname="year",
gname="first_treat", estimator="saturated", att=True)
sa = m_sa.aggregate(); sa.index = sa.index.astype(float)
sa_post = sa[(sa.index >= 0) & (sa.index <= 5)]
sa_b = float(sa_post["Estimate"].mean())
sa_se = float(np.sqrt((sa_post["Std. Error"].astype(float) ** 2).mean() / len(sa_post)))
# Borusyak/Gardner imputation (did2s)
m_d2s = pf.event_study(df, yname=Y, idname="district_id", tname="year",
gname="first_treat", estimator="did2s", att=True)
d2s_b, d2s_se = m_d2s.coef().iloc[0], m_d2s.se().iloc[0]
# Callaway-Sant'Anna against the never-treated group
cs = dd.CallawaySantAnna(control_group="never_treated", cluster="district_id").fit(
df, outcome=Y, unit="district_id", time="year",
first_treat="first_treat", aggregate="simple")
print(f"TWFE ATT : {cell(twfe_b, twfe_se)}")
print(f"Sun-Abraham ATT (avg k=0..5) : {cell(sa_b, sa_se)}")
print(f"Borusyak/Gardner ATT (did2s) : {cell(d2s_b, d2s_se)}")
print(f"Callaway-Sant'Anna ATT : {cell(cs.att, cs.se)}")

TWFE ATT : +0.2699*** (0.1005)
Sun-Abraham ATT (avg k=0..5) : +0.2991*** (0.0246)
Borusyak/Gardner ATT (did2s) : +0.3022*** (0.0907)
Callaway-Sant'Anna ATT : +0.2561*** (0.0763)

All four estimators target the same ATT and land in a tight band: TWFE +0.2699, Sun-Abraham +0.2991, Borusyak/Gardner +0.3022, and Callaway-Sant’Anna +0.2561 — a spread of only 0.046 IHS units across methods that, in other settings, can diverge sharply. Each is significant at 1%. They agree here because there is a real never-treated comparison group (the 122 controls) and the treatment effect is fairly homogeneous, so the conditions that make TWFE’s forbidden comparisons dangerous simply do not bind. This agreement is the methodological payoff: a reader worried that the headline is a negative-weighting artifact can see three staggered-robust estimators reproduce it. To show why they agree, we decompose the TWFE number itself.

The Goodman-Bacon decomposition breaks the TWFE coefficient into the weighted average of every underlying 2×2 comparison, labeling each by type. diff-diff does it in one call.

bac = dd.BaconDecomposition().fit(df, outcome=Y, unit="district_id",
time="year", first_treat="first_treat")
bdf = bac.to_dataframe()
print(f"Goodman-Bacon: TWFE = {bac.twfe_estimate:+.4f} decomposes into "
f"{len(bdf)} 2x2 comparisons.")
print(bdf.groupby("comparison_type")
.apply(lambda g: pd.Series({"total_weight": g.weight.sum(),
"weighted_avg_estimate": np.average(g.estimate, weights=g.weight)}))
.round(4))

Goodman-Bacon: TWFE = +0.2699 decomposes into 64 2x2 comparisons.
comparison_type total_weight weighted_avg_estimate
earlier_vs_later 0.0338 0.3370
later_vs_earlier 0.0121 0.0135
treated_vs_never 0.9542 0.2708

The decomposition is reassuring. The clean treated-vs-never-treated comparisons carry 95.42% of the total weight and average +0.2708 — essentially the headline. The “forbidden” later-vs-earlier comparisons (already-treated units used as controls, the ones that can flip TWFE’s sign) carry just 1.21% of the weight and contribute a near-zero +0.0135; clean earlier-vs-later comparisons add another 3.38% at +0.337. With at most ~1.2% of the weight on biased comparisons, TWFE is barely contaminated here — the empirical reason the four estimators agreed. The general lesson is worth keeping: the negative-weights problem is real in principle but empirically negligible whenever a large never-treated pool dominates the weighting, as the 122 PSM controls do. Having trusted the average, we can now ask where the effect is strongest.

9. Heterogeneity and spillovers

9.1 Where parks work: distance and roads

Place-based policy is, by definition, about place — so the effect should depend on where the park sits. We interact the treatment with distance moderators (a negative interaction means the effect fades with distance) and road-density moderators (a positive interaction means roads amplify it), each on the with-trends spec.

dt = add_trend_terms(district)
for mod in ["dist_addis_km", "dist_state_capital_km", "dist_nearest_city_km",
"primary_road_density", "paved_road_density"]:
m = pf.feols(f"ihs_light ~ treatment + treatment:{mod} + " +
" + ".join(TREND_TERMS) + " | district_id + region^year",
data=dt, vcov={"CRV1": "district_id"})
b, se = m.coef()[f"treatment:{mod}"], m.se()[f"treatment:{mod}"]
print(f"{mod:24s} interaction {b:+.5f} (se {se:.5f}, t {b/se:+.2f})")

dist_addis_km interaction -0.00822 (se 0.00232, t -3.54)
dist_state_capital_km interaction -0.00862 (se 0.00406, t -2.13)
dist_nearest_city_km interaction -0.03352 (se 0.00684, t -4.90)
primary_road_density interaction +0.32640 (se 0.84748, t +0.39)
paved_road_density interaction +0.66945 (se 0.32174, t +2.08)

Location fundamentals sharply moderate park effectiveness, exactly as the paper argues. All three distance interactions are negative — the park effect fades with distance from economic centers — and three of them are significant: distance to nearest city (−0.0335, t = −4.90, the steepest decay), distance to Addis (−0.0082, t = −3.54), and distance to the state capital (−0.0086, t = −2.13). Both road interactions are positive — denser roads amplify the effect — with paved-road density significant (+0.6695, t = 2.08) but primary-road density correctly signed yet borderline insignificant (+0.3264, t = 0.39). That last result is an honest synthetic limitation: with only 17 treated woredas the mutually-correlated moderators cannot all be precise at once, so one of the two road interactions necessarily reads non-significant. The point estimates all carry the predicted sign; precision, not direction, is what the small treated sample cannot fully deliver. A related question is whether the park’s gain is truly new or merely stolen from its neighbours.

9.2 Spillovers: does a park lift its neighbours?

The spillover test adds a nearby indicator — control woredas within 10 km of an operational park — to the Table 1 spec. If parks merely displace activity from neighbours, nearby should be negative; if the gains are net-new, it should be zero.

for ycol, label in [("ihs_light", "IHS night-light"),
("light_intensity", "Raw night-light")]:
m = pf.feols(f"{ycol} ~ treatment + nearby | district_id + region^year",
data=district, vcov={"CRV1": "district_id"})
print(f"{label:18s} treatment {m.coef()['treatment']:+.4f} "
f"nearby {m.coef()['nearby']:+.4f} (t {m.coef()['nearby']/m.se()['nearby']:+.2f})")

IHS night-light treatment +0.2712 nearby +0.0648 (t +1.06)
Raw night-light treatment +1.7328 nearby +0.0927 (t +1.35)

The nearby coefficient is +0.0648 (SE 0.0610, t = 1.06) for IHS light and +0.0927 (t = 1.35) for raw light — both small and statistically indistinguishable from zero — while the treatment coefficient stays large and significant (+0.2712). The reading is no spillover: the park lifts its host woreda by ~0.27 IHS but leaves immediate neighbours essentially unchanged, so the host’s gain is net-new activity, not displacement. This also reassures on SUTVA: with no measurable geographic spillover, the never-treated controls are not contaminated by proximity to a park, so the main ATT is not biased by treated-on-control externalities. Economically, the parks behave like relatively self-contained enclaves with weak local supplier linkages. So far the story is about lights and land — but did the parks change how people actually live?

10. Household welfare and women’s empowerment

10.1 Household living standards (Table 5)

We now switch to the DHS household repeated cross-section. Because each round samples different households, there is no household panel key, so we use no household fixed effect — the effect is identified off district × round group means, with district and region×round fixed effects and DHS survey weights. We report each outcome with and without household-size and head-age controls.

for ycol, label in [("durable_goods_pc", "Durable goods p.c."),
("housing_quality", "Housing quality"),
("wealth_index", "Wealth index")]:
m0 = pf.feols(f"{ycol} ~ treatment | district_id + region_id^survey_round",
data=household, weights="survey_weight", vcov={"CRV1": "district_id"})
m1 = pf.feols(f"{ycol} ~ treatment + hh_size + age_head | "
"district_id + region_id^survey_round",
data=household, weights="survey_weight", vcov={"CRV1": "district_id"})
print(f"{label:18s} no-controls {m0.coef()['treatment']:+.4f} "
f"with-controls {m1.coef()['treatment']:+.4f} ({m1.se()['treatment']:.4f})")

Durable goods p.c. no-controls +0.2489 with-controls +0.2286 (0.0284)
Housing quality no-controls +0.2484 with-controls +0.2480 (0.0193)
Wealth index no-controls +0.3875 with-controls +0.3825 (0.0461)

All three living-standards outcomes rise sharply and significantly. Durable goods per capita gain +0.2286 with controls (SE 0.0284, t = 8.06) — against a 0.308 mean, a ~74% increase. Housing quality (an indicator for having electricity, piped water, a toilet, and a finished floor) rises +0.2480, so the probability of clearing that bar jumps ~24.8 percentage points off a 30.7% base. The composite wealth index rises +0.3825 standard deviations (SE 0.0461, t = 8.29). Crucially, adding controls barely moves any estimate (durables 0.249 → 0.229, the others essentially unchanged), which confirms the district + region×round design already absorbs the main confounding — the covariates are only mildly correlated with treatment. As at the satellite level, the timing is clean.

# RCS event study uses coarse phase dummies (no balanced unit x time grid).
# _rcs_event_study() is defined in the companion script.py.
es = _rcs_event_study(household, "durable_goods_pc", controls=["hh_size", "age_head"])
print(es.round(4).to_string(index=False))

 event_phase estimate se p_value
-3.0 -0.0197 0.0482 0.6840
-2.0 0.0236 0.0329 0.4757
0.0 0.2606 0.0398 0.0000
1.0 0.1513 0.0387 0.0001

Because the DHS data are repeated cross-sections, the household event study uses coarse phase dummies rather than annual event time. The two pre-opening phases are flat and insignificant — phase −3 at −0.0197 (p = 0.68) and phase −2 at +0.0236 (p = 0.48), both straddling zero — so there is no differential pre-trend in household durables. The effect then jumps to +0.2606 at phase 0 (p < 0.0001) and stays strongly positive at +0.1513 at phase +1. This is the RCS counterpart to the satellite event study’s no-anticipation evidence, with the honest caveat that two pre-phases make a low-powered test. Now to the question the whole post has been building toward: who got the jobs?

10.2 Employment and women’s empowerment (Tables 6–7): the climax

This is the analytical climax, and a textbook case for heterogeneity analysis. We estimate non-agricultural employment for the full sample, then split by sex, using the same survey-weighted RCS design.

ctrl = "hh_size + age_head + age + age_sq"
for label, sub in [("Full sample", individual),
("Women", individual[individual.sex == 1]),
("Men", individual[individual.sex == 0])]:
m = pf.feols(f"nonag_employment ~ treatment + {ctrl} | "
"district_id + region_id^survey_round",
data=sub, weights="survey_weight", vcov={"CRV1": "district_id"})
b, se = m.coef()["treatment"], m.se()["treatment"]
print(f"{label:12s} {b:+.4f} ({se:.4f}) t {b/se:+.2f}")

Full sample +0.0911 (0.0580) t +1.57 <-- NULL on average
Women +0.1404 (0.0468) t +3.00 <-- SIGNIFICANT for women
Men +0.0176 (0.0934) t +0.19

The average non-agricultural employment effect is +0.0911 (SE 0.0580, t = 1.57) — insignificant — which, read alone, would suggest parks do not move employment at all. But pooling the sexes hides a strong gendered split: the female effect is +0.1404 (SE 0.0468, t = 3.00, significant at 1%) — about a 14-percentage-point rise in women’s non-agricultural employment — while the male effect is +0.0176 (t = 0.19), essentially zero. The parks, concentrated in textiles and garments, pull women into factory wage work; the men were largely already off-farm, so the average washes out. A reader who quoted only the full-sample number would badly misread the study — the sex split is the finding, not a footnote. The empowerment cascade follows the jobs.

women = individual[individual.sex == 1]
for ycol, label in [("decision_power", "Decision power"),
("savings_account", "Savings account"),
("dv_accept", "Accepts DV")]:
m = pf.feols(f"{ycol} ~ treatment + {ctrl} | "
"district_id + region_id^survey_round",
data=women, weights="survey_weight", vcov={"CRV1": "district_id"})
b, se = m.coef()["treatment"], m.se()["treatment"]
print(f"{label:18s} {b:+.4f} ({se:.4f}) t {b/se:+.2f}")

Decision power +0.1096 (0.0194) t +5.66
Savings account +0.3153 (0.0182) t +17.34
Accepts DV -0.2096 (0.0254) t -8.24

With factory jobs, women’s outcomes shift across the board (women only). Decision-making power rises +0.1096 (SE 0.0194, t = 5.66), savings-account ownership rises +0.3153 (SE 0.0182, t = 17.34) — enormous against a 6.3% base — and acceptance of domestic violence falls −0.2096 (SE 0.0254, t = −8.24), a ~21-point reduction off a 63.5% base. Economic agency translates into household bargaining power and shifting gender norms. The event study below confirms the timing.

The female-employment and decision-power event studies both sit near zero in the pre-phases and turn up at and after phase 0 (female employment jumps to +0.1311 at phase 0, p = 0.013), reinforcing the no-anticipation reading — women’s gains appear with the park, not before it. The gender result is the substantive heart of the study; one robustness battery remains to decide whether to trust the satellite headline that anchors it.

11. Robustness: Conley spatial standard errors and restricted pools

Recall from the map that all 17 treated woredas cluster spatially. When treated units are packed together, a regional shock hits several at once, so their errors are not independent draws — and the naive standard error, which assumes independence, will be too small. The fix is a Conley spatial-HAC standard error, which allows a district’s errors to correlate with itself over time (serial) and with nearby districts in the same year (spatial). The point estimate never changes; only the standard error does. We compute four standard errors for the with-trends light ATT and re-estimate it on restricted control pools.

# four SEs for the with-trends IHS-light ATT (full Conley sandwich in script.py)
se_tab = conley_se_for_spec(add_trend_terms(district), "ihs_light",
["treatment"] + TREND_TERMS)
print(se_tab.loc[se_tab.term == "treatment",
["estimate", "se_naive", "se_clustered", "se_conley", "se_hac"]]
.round(4).to_string(index=False))

 estimate se_naive se_clustered se_conley se_hac
0.2152 0.0329 0.0792 0.0346 0.0799

The satellite headline survives honest standard errors. The most conservative Conley spatial-HAC SE is 0.0799 — 2.43× the naive HC0 SE of 0.0329 — yet the ATT of +0.2152 stays significant (t = 2.69, significant at 1%). Notice the cluster SE (0.0792) and the Conley-HAC SE (0.0799) are nearly identical: clustering at the district level already captures most of the dependence, so spatial correlation beyond the district adds little here. The estimate is also stable when we change the comparison group — dropping the Addis Ababa region or restricting controls to those far from any city.

specs = {"Full sample": district,
"Drop Addis region": district[district.region != "Addis Ababa"],
"Controls >= 50km from city": district[(district.treated == 1) |
(district.dist_nearest_city_km >= 50)]}
for name, sub in specs.items():
m = pf.feols("ihs_light ~ treatment + " + " + ".join(TREND_TERMS) +
" | district_id + region^year",
data=add_trend_terms(sub), vcov={"CRV1": "district_id"})
b, se = m.coef()["treatment"], m.se()["treatment"]
print(f"{name:28s} {b:+.4f} ({se:.4f}) N={m._N}")

Full sample +0.2152 (0.0833) N=2224
Drop Addis region +0.1550 (0.0910) N=1984
Controls >= 50km from city +0.2143 (0.0854) N=1392

Dropping the Addis Ababa region pulls the estimate to +0.1550 (still significant at 10%, N 1,984) and restricting controls to those at least 50 km from a city holds it at +0.2143 (significant at 5%, N 1,392). Combined with the Section 8 agreement of Sun-Abraham, Borusyak/Gardner, and Callaway-Sant’Anna, the satellite result is robust to both the standard-error specification and the choice of comparison group. With the evidence assembled, we can return to the opening question.

12. Discussion

What we found. Yes — and the “for whom” matters as much as the “whether.” A park raises local nighttime light by about +0.215 IHS (~21%) and built-up land by ~2.6 percentage points, with the effect building over five years to a +0.48 plateau and no spillover to neighbours. Four estimators agree the staggered-DiD negative-weights problem is not biting (spread 0.046, with 95.4% clean Bacon weight). Households near a park gain durables (+0.229), housing quality (+0.248), and wealth (+0.383 SD). And the central result: average non-agricultural employment is an insignificant +0.091, yet women’s employment rises a significant +0.140, lifting their decision power (+0.110), savings (+0.315), and lowering acceptance of domestic violence (−0.210). The parks reshaped the local economy, and they did so largely through women.

So what? Two design lessons follow directly. First, on site selection: the effect fades steeply with distance from cities (−0.0335 per km to the nearest city) and is amplified by paved roads (+0.6695). A park dropped in a remote, poorly-connected woreda would do far less — proximity to existing economic centers is first-order, so place-based policy should follow the roads. Second, on sector and inclusion: because the employment and empowerment gains run through female-intensive sectors (textiles, garments), a policymaker who measured only the average employment effect would conclude the parks failed on jobs and miss their largest social return. Evaluations of place-based policy should be sex-disaggregated by default.

Limitations and the observational caveat. Be appropriately humble. The data are synthetic — calibrated to teach the methods, not to report new facts about Ethiopia. The treated group is tiny (17 woredas), so several effects are borderline; the primary-road interaction is correctly signed but imprecise, and the raw-light coefficient runs high. Most fundamentally, this is an observational study: the parks were not randomly placed, so identification rests on parallel trends, not randomization. The flat pre-trends and the null spillover support that assumption but never prove it. The adjustment here — district and region×year fixed effects, baseline-trend interactions, and the PSM-matched controls — is confounding control, not the precision-only adjustment of a randomized experiment. The ATT we report is the effect on these parks in this setting; it travels only as far as that.

13. Reproduction audit: synthetic data vs the paper

Because the data are synthetic, transparency demands we line our numbers up against the published ones. The data-generating process was tuned to match the paper coefficient by coefficient; signs and significance agree throughout, and the headline magnitudes land within about 0.02. We also disclose four documented gaps rather than paper over them.

Result	This synthetic data	Paper (reported)	Sign	Significance
Table 1: IHS light, no trends	+0.2704***	≈ +0.265**	✓	✓
Table 1: IHS light, with trends	+0.2152***	≈ +0.214**	✓	✓
Table 1: raw light, with trends	+1.6181***	≈ +1.276**	✓	partial (high)
Table 1: impervious, with trends	+0.0263***	≈ +0.028**	✓	✓
Table 2: `nearby` spillover (IHS)	+0.0648 (ns)	≈ 0 (ns)	✓	✓
Table 3: distance to nearest city	−0.0335***	negative & sig.	✓	✓
Table 4: paved-road density	+0.6695**	positive	✓	✓
Table 4: primary-road density	+0.3264 (ns)	positive	✓	partial (ns)
Table 5: durables (controls)	+0.2286***	≈ +0.226***	✓	✓
Table 5: housing (controls)	+0.2480***	≈ +0.252***	✓	✓
Table 5: wealth (controls)	+0.3825***	≈ +0.409*	✓	✓
Table 6: employment, full sample	+0.0911 (ns)	≈ +0.110 (ns)	✓	✓
Table 6: employment, women	+0.1404***	≈ +0.133***	✓	✓
Table 6: employment, men	+0.0176 (ns)	≈ +0.015 (ns)	✓	✓
Table 7: decision power	+0.1096***	≈ +0.103***	✓	✓
Table 7: savings account	+0.3153***	≈ +0.318***	✓	✓
Table 7: DV acceptance	−0.2096***	≈ −0.212***	✓	✓
Staggered: TWFE / SA / BG / CS ATT	+0.270 / +0.299 / +0.302 / +0.256	“closely track baseline”	✓	✓

Stars: *** p < .01, ** p < .05, * p < .10.

Of the audited cells, the great majority land on target in sign, significance, and magnitude (within ~0.02 on the headline coefficients). Four gaps are documented and bounded. (1) The raw-light coefficient runs high (~1.6 vs 1.276): keeping treated woredas essentially always-lit (for a clean IHS event study with only 17 clusters) removes the zero-dilution that would otherwise pull the raw mean down — a deliberate bright-base device that protects the on-target IHS coefficient. (2) The primary-road interaction is correctly signed and on-magnitude but borderline non-significant — the 17-treated sample cannot make both road interactions precise at once. (3) Light levels are not matched: treated woredas carry an intrinsically bright base (~4–5) and controls a dim one (~0.1), unlike the paper’s PSM-matched 0.94/0.87, which is exactly why the EDA figure is baseline-normalized. (4) The decision-power mean (~0.88) sits a touch below the paper’s 0.899 because the linear-probability clipping ceiling caps the achievable effect. Everywhere else, direction and significance track the paper closely. The synthetic data reproduce the paper’s findings — they are not, and are not claimed to be, the paper’s data.

14. Summary and takeaways

Number to remember	Value
Light ATT (with trends)	+0.2152*** (~21%)
Four-estimator spread	0.046 IHS units
Clean Bacon weight	95.4%
Wealth-index ATT	+0.383 SD
Female employment ATT	+0.140*** (vs +0.091 ns full sample)
Light SE: naive → Conley-HAC	0.0329 → 0.0799

A park raises local activity ~21% — and the staggered-bias worry does not bite. The with-trends TWFE ATT is +0.2152***, and TWFE, Sun-Abraham (+0.299), Borusyak/Gardner (+0.302), and Callaway-Sant’Anna (+0.256) all agree within 0.046 because 95.4% of the Bacon weight is clean treated-vs-never comparisons. When a large never-treated pool dominates, plain TWFE is barely contaminated.
The average hides the finding — split by sex. Full-sample non-ag employment is an insignificant +0.091, but the female effect is +0.140*** and the male effect is ~0. The empowerment cascade follows: decision power +0.110, savings +0.315, and acceptance of domestic violence −0.210, all highly significant. Heterogeneity analysis turned a null into the study’s headline.
Honest inference matters but does not overturn the result (a limitation in spirit). With all 17 treated woredas clustered in space, the Conley-HAC SE (0.0799) is 2.43× the naive HC0 SE (0.0329); the ATT still clears significance (t = 2.69), and the small treated sample is why the primary-road interaction and the raw-light level remain imprecise or off-target.
Next step. Re-estimate the event study with a Callaway-Sant’Anna dynamic aggregation to compare its lag-by-lag path against the saturated one, add a sensitivity analysis (à la Rambachan-Roth) that asks how large a pre-trend violation would overturn the +0.215 ATT, and test whether labor-intensive parks drive the female-employment effect more than capital-intensive ones.

15. Exercises

Drop the anchor cohort. Re-run the staggered estimators after excluding the single 2008 woreda, so all treated units come from the 2014–2020 build-out. Do the four ATTs still agree within 0.05, and does the Goodman-Bacon clean-weight share change? What does that tell you about how much one early cohort drives the comparison structure?
Stress-test the gender result. Add an interaction treatment:sex to the full-sample employment regression instead of splitting the data. Does the interaction coefficient recover the female-minus-male gap (≈ +0.123)? Why might the pooled-interaction and split-sample approaches give slightly different standard errors?
Move the collapse year. The naive 2×2 in Section 6.1 collapsed the design at the median opening year (2017). Recompute it collapsing at 2014 and at 2019. How much does the blended ATT move, and why does the choice of collapse year matter for a staggered design but not for a single-date one?

16. References

Huang, G., Wang, M., & Xu, H. (2026). The socioeconomic impacts of industrial parks in Ethiopia. Journal of Urban Economics. https://doi.org/10.1016/j.jue.2026.103867
Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001
Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006
Borusyak, K., Jaravel, X., & Spiess, J. (2024). Revisiting event-study designs: Robust and efficient estimation. Review of Economic Studies, 91(6), 3253–3285. https://doi.org/10.1093/restud/rdae007
Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014
Conley, T. G. (1999). GMM estimation with cross-sectional dependence. Journal of Econometrics, 92(1), 1–45. https://doi.org/10.1016/S0304-4076(98)00084-0
pyfixest documentation — https://pyfixest.org/
diff-diff documentation — https://github.com/igerber/diff-diff
Ethiopia Demographic and Health Surveys (DHS), 2000–2019 — The DHS Program, ICF / Ethiopian Public Health Institute. https://dhsprogram.com/
Chen, Z., Yu, B., Yang, C., et al. (2021). An extended time series (2000–2018) of global NPP-VIIRS-like nighttime light data. Earth System Science Data, 13(3), 889–906. https://doi.org/10.5194/essd-13-889-2021
Zhang, X., Liu, L., Zhao, T., et al. (2022). GISD30: Global 30-m impervious-surface dynamic dataset. Earth System Science Data, 14(4), 1831–1856. https://doi.org/10.5194/essd-14-1831-2022

This tutorial is a teaching replication built on synthetic data; see the data note in Section 1 and the reproduction audit in Section 13. The companion script.py regenerates every figure and table.

AI Podcast: Do Industrial Parks Work?

Click play to load

0:00 0:00

Dynamic Panel Data Models in Python: From Nickell Bias to System GMM

Thu, 11 Jun 2026 00:00:00 +0000

Abstract

When this year’s outcome depends on last year’s outcome — employment, debt, capital, habits — ordinary panel methods break in ways that are invisible on the printed output. This tutorial asks how persistent firm-level employment is, and shows why the answer depends dramatically on the estimator used to obtain it. Using the classic Arellano and Bond (1991) panel of 140 UK manufacturing firms observed 1976—1984 (1,031 firm-years, unbalanced), we estimate the autoregressive coefficient $\rho$ of a dynamic labor-demand equation with pyfixest (OLS, fixed effects, IV benchmarks) and pydynpd (difference and system GMM). Pooled OLS gives $\hat{\rho} = 0.962$, biased upward by the omitted firm effect; fixed effects gives 0.626, biased downward by Nickell bias — so the truth must lie inside the bracket [0.626, 0.962]. Anderson-Hsiao IV is consistent but useless (1.233 with standard error 0.478), and Arellano-Bond difference GMM with 91 instruments returns 0.679, hugging the biased fixed-effects bound — the textbook weak-instrument symptom. Blundell-Bond system GMM with 32 collapsed instruments delivers the defensible headline: $\hat{\rho} = 0.927$ (SE 0.079), inside the bracket, with AR(2) p = 0.994 and Hansen p = 0.462, and the toolchain replicates the published pydynpd benchmark digit for digit. The practical implication: roughly 93 percent of an employment shock survives into the next year, and no single printed p-value — only the full bracket-plus-diagnostics workflow — separates that estimate from the four wrong ones.

1. Overview

Suppose a recession, a strike, or a sudden export boom changes a firm’s workforce this year. How much of that shock is still visible in the firm’s employment next year? If the answer is “almost none,” labor markets adjust quickly and temporary shocks stay temporary. If the answer is “almost all of it,” shocks echo for a decade: hiring freezes cast long shadows, and a one-year subsidy keeps paying employment dividends for years. The single number that encodes this is $\rho$, the coefficient on lagged employment in a dynamic labor-demand equation — and this post is the story of how hard that one number is to estimate, and how econometricians eventually got it right.

Think of $\rho$ as the echo strength of the labor market. If $\rho = 0.6$, a shock loses 40 percent of its volume every year and fades within a couple of years. If $\rho = 0.95$, the echo barely decays — what happens to a firm in 1980 is still audible in 1988. The estimators we run below will disagree about exactly this: the same regression, on the same data, will imply shock half-lives of 1.5 years, 9 years, or 18 years depending on how it treats one nuisance term.

Formally, we estimate the dynamic labor-demand model that Blundell and Bond (1998) used on this very dataset:

$$n_{it} = \rho n_{i,t-1} + \beta_1 w_{it} + \beta_2 w_{i,t-1} + \beta_3 k_{it} + \beta_4 k_{i,t-1} + \alpha_i + \delta_t + \varepsilon_{it}$$

In words, this says: a firm’s log employment this year ($n_{it}$) equals a fraction $\rho$ of its log employment last year, plus the effects of current and lagged log real wages ($w$) and log capital ($k$), plus a permanent firm-specific level $\alpha_i$ (the firm fixed effect: management quality, industry niche, plant size), plus a year effect $\delta_t$ shared by all firms (the macro cycle), plus an idiosyncratic shock $\varepsilon_{it}$. In the code, $n$, $w$, and $k$ are literally the columns n, w, and k of abdata.csv; $n_{i,t-1}$ is the constructed column n_lag1; and $\rho$ is the coefficient the output tables label n_lag1 or L1.n.

Why is this hard? Because the model commits the one sin that ordinary panel methods cannot forgive: it puts a lagged dependent variable on the right-hand side while an unobserved firm effect $\alpha_i$ sits in the error. Last year’s employment $n_{i,t-1}$ obviously depends on $\alpha_i$ — a firm with a high permanent level had high employment last year too — so the regressor is correlated with part of the error term by construction, no matter how many controls we add. Pooled OLS breaks one way, fixed effects breaks the opposite way, and the resolution requires a genuinely different idea: using the panel’s own history as instruments, which is what the Arellano-Bond and Blundell-Bond generalized method of moments (GMM) estimators do.

Importantly, the framing here is descriptive and structural, not causal: $\rho$ is a persistence parameter of a dynamic labor-demand equation — there is no treatment, and we estimate no ATE or ATT. What identification requires instead is sequential exogeneity of the instruments (past values of the variables must be uncorrelated with future shocks) and no serial correlation in $\varepsilon_{it}$ — assumptions we can partially test with the AR(2) and Hansen diagnostics that occupy the second half of the tutorial.

Learning objectives:

Understand why a lagged dependent variable plus a fixed effect breaks both pooled OLS (bias up) and the within estimator (Nickell bias, down), and how the two wrong answers bracket the truth.
Implement the full estimator ladder in Python — OLS and fixed effects with pyfixest, Anderson-Hsiao IV, and difference and system GMM with pydynpd.
Estimate employment persistence on the classic Arellano-Bond (1991) UK panel and diagnose weak instruments using Bond’s (2002) bracket check.
Assess GMM credibility with the AR(1)/AR(2) serial-correlation tests and the Hansen overidentification test, including the counterintuitive “p close to 1 is a red flag” reading.
Compare instrument-proliferation choices (lag windows, collapsing) and verify the toolchain against the package’s published replication benchmark.

The diagram below is the roadmap: every estimator we run, why it fails or succeeds, and the order in which the tutorial visits them.

flowchart TD
A("Dynamic panel model:<br/>n_it = rho n_i,t-1 + ... + alpha_i + eps_it")
A --> B("Pooled OLS<br/>rho = 0.962, biased UP<br/>(L1.n absorbs alpha_i)")
A --> C("Fixed effects<br/>rho = 0.626, biased DOWN<br/>(Nickell bias, T = 7-9)")
B --> D("The bracket:<br/>truth lies in [0.626, 0.962]")
C --> D
D --> E("Anderson-Hsiao IV<br/>rho = 1.233 (SE 0.478)<br/>consistent but useless")
E --> F("Difference GMM<br/>rho = 0.679, 91 instruments<br/>hugs FE bound: weak instruments")
F --> G("System GMM, collapsed<br/>rho = 0.927 (SE 0.079)<br/>AR(2) p = 0.994, Hansen p = 0.462")
G --> H("Diagnostics + proliferation grid<br/>+ exact replication check")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A anchor
class B,C gray
class D,E,H blue
class F orange
class G teal

Read the diagram top to bottom and you have the whole argument: two naive estimators whose known bias directions form a credible bracket, one IV estimator that is right in theory and hopeless in practice, a difference-GMM estimator that passes every printed test yet sits suspiciously on the bracket’s floor, and a system-GMM estimator that lands in the upper half of the bracket with clean diagnostics. The final section stress-tests that winner against instrument proliferation and a published benchmark.

Key concepts at a glance

The tutorial leans on a small vocabulary repeatedly. Each concept below has three parts: the definition is always visible, while the example and analogy sit behind clickable cards — open them when you need them, leave them collapsed for a quick scan. If a later section mentions “Nickell bias” or “sequential exogeneity” and the term feels slippery, this is the section to re-read.

1. Lagged dependent variable and persistence $\rho$. The model puts yesterday’s outcome on the right-hand side. The coefficient $\rho$ measures persistence. It is the fraction of this year’s employment inherited from last year. Values near 0 mean fast adjustment. Values near 1 mean shocks essentially never die.

Example

Our headline estimate is $\hat{\rho} = 0.927$: about 93 percent of an employment shock survives into the next year. After five years, $0.927^5 \approx 0.68$ of the shock remains. The implied half-life is roughly nine years — longer than the 1976—1984 sample window itself.

Analogy

A shout in a canyon. $\rho$ is the echo strength: at $\rho = 0.6$ each echo returns at 60 percent volume and silence comes quickly; at $\rho = 0.93$ the canyon keeps answering for a decade. The estimators in this post disagree about how echoey the canyon is.

2. Firm fixed effect $\alpha_i$. A permanent, unobserved firm-specific level. It captures management quality, technology, and market niche. It never changes over the sample. It sits in the error term unless the estimator deals with it explicitly.

Example

In our panel, the between-firm standard deviation of log employment is 1.339 while the within-firm standard deviation is only 0.195 — a factor of seven. Firms differ enormously from each other and barely move around their own levels, so $\alpha_i$ dominates the data. Figure 1 shows each firm orbiting its own level.

Analogy

Planets in different orbits. Each planet (firm) circles at its own distance from the sun, with small wobbles. If you pool all planets and regress position on lagged position, most of what you “explain” is just which orbit each planet lives in — not its dynamics.

3. Nickell bias. The bias of the fixed-effects (within) estimator in dynamic panels. Demeaning subtracts each firm’s average, which contains future shocks. The demeaned lag is then mechanically correlated with the demeaned error. The bias is negative and of order $1/T$. With T of 7—9 it is large.

Example

Our within estimate is $\hat{\rho} = 0.626$ (SE 0.052) against a system-GMM benchmark of 0.927 — a downward gap of 0.30. With $T \approx 7$—$9$, the $1/T$ bias is roughly a tenth in raw scale and is amplified when $\rho$ is large. The bias does not shrink as you add more firms.

Analogy

Grading each student against their own course average — but the average includes the final exam they have not taken yet. Today’s score is being compared to a benchmark contaminated by tomorrow’s performance, creating a spurious negative link between “today” and “the future part of the average.”

4. The bias bracket (Bond 2002). Pooled OLS biases $\rho$ up. Fixed effects biases it down. Both directions are known from theory. So the two wrong answers bracket the truth. Any consistent estimator should land between them. Estimates hugging either bound deserve suspicion.

Example

Our bracket is [0.626, 0.962]. Difference GMM lands at 0.679 — only 0.053 above the floor and within one standard error of it, the classic weak-instrument warning. System GMM lands at 0.927, in the upper half, and is the estimate the bracket logic endorses.

Analogy

Two broken clocks, one known to run fast and one known to run slow. Neither tells the time, but the true time must lie between them — and a third clock claiming a time outside that window is broken in a worse way.

5. Sequential exogeneity. The identifying assumption behind dynamic-panel GMM. Past values of the variables must be uncorrelated with current and future shocks. Then deep lags are valid instruments. It is weaker than strict exogeneity. It tolerates feedback from past shocks to current regressors.

Example

For the differenced equation, sequential exogeneity delivers the moment conditions $E[n_{i,t-s} \Delta\varepsilon_{it}] = 0$ for $s \ge 2$. Our AR(2) test (p = 0.994) checks the part of this that is checkable: if $\varepsilon_{it}$ were serially correlated, the $t-2$ lags would be contaminated and the moments would fail.

Analogy

Witnesses who left the building before the crime. Anything they saw (lags dated $t-2$ and earlier) cannot have been influenced by what happened at time $t$, so their testimony is admissible — provided no one tipped them off in advance (no serial correlation).

6. Difference GMM (Arellano-Bond 1991). First-difference the equation to kill $\alpha_i$. The differenced lag is still endogenous. Instrument it with all available lagged levels, dated $t-2$ and earlier. Weight the many moment conditions optimally. This generalizes Anderson-Hsiao from one instrument to dozens.

Example

Our two-step difference GMM uses 91 instruments on 751 observations and returns $\hat{\rho} = 0.679$ (SE 0.089), with Hansen p = 0.211 and AR(2) p = 0.866 — every printed test passes, yet the estimate hugs the FE bound because lagged levels barely predict future differences when $\rho$ is near 1.

Analogy

Replacing one courtroom witness with a panel of forty. Each extra witness adds a little information, and the judge (the GMM weighting matrix) listens more carefully to the reliable ones. But if every witness only glimpsed the scene from far away, forty vague testimonies still convict no one.

7. System GMM and mean stationarity (Blundell-Bond 1998). Stack the differenced equation with the original levels equation. Instrument levels with lagged differences. This requires one extra assumption: firms’ initial deviations from their steady-state paths are uncorrelated with $\alpha_i$. The payoff is much stronger instruments when $\rho$ is large.

Example

Adding the levels equation moves our estimate from 0.679 (difference GMM) to $\hat{\rho} = 0.927$ (SE 0.079) with 32 collapsed instruments — inside the bracket, with AR(2) p = 0.994 and Hansen p = 0.462. The extra moments are exactly the ones with identifying power for a persistent series.

Analogy

Trying to learn a lake’s depth from ripples alone (differences) versus also using the waterline marks on the shore (levels). When the lake is calm — a persistent series barely moves — the ripples carry almost no information, and the waterline marks are what actually pin the answer down.

8. Instrument proliferation and collapsing. “Use every lag” generates instruments quadratically in T. Too many instruments overfit the endogenous variables and weaken the Hansen test. A Hansen p-value near 1 signals an overwhelmed test, not a valid model. Collapsing combines lags into one column per depth, shrinking the count. Roodman’s rule of thumb: keep instruments below the number of groups.

Example

In our grid, uncollapsed instrument counts of 68, 95, and 113 push the Hansen p-value from 0.035 to 0.186 to 0.235 while $\hat{\rho}$ barely moves (0.921—0.956). The uncollapsed 2:3 spec is rejected (p = 0.035) while its collapsed twin passes (p = 0.096) — same model, different verdicts, driven purely by instrument count.

Analogy

A judge facing 113 witnesses who mostly repeat each other’s hearsay. With so much correlated testimony, the judge can no longer distinguish a solid case from a coached one — the trial (the Hansen test) loses its power to reject. Fewer, independent witnesses make a more credible verdict.

With the vocabulary pinned down, we can set up the toolchain — including one small but instructive compatibility fix.

2. Setup and imports

We need two workhorse libraries. pyfixest estimates the OLS, fixed-effects, and IV benchmarks with a compact R-style formula syntax. pydynpd (Wu, Hua and Xu 2023) estimates difference and system GMM with a command syntax deliberately close to Stata’s xtabond2 — and its published output has been validated against xtabond2, which is exactly what our replication check in Section 11 will exploit.

Both libraries install from PyPI. The version pin matters here, because the compatibility fix described next targets exactly this release:

pip install pydynpd==0.2.2 pyfixest

One practical wrinkle deserves a friendly explanation rather than a silent workaround. pydynpd 0.2.2 was written before NumPy 2.0, which removed the alias np.in1d and stopped allowing float() and math.sqrt() to be called directly on 1x1 matrices. Rather than downgrading NumPy or forking the package, we apply a six-line compatibility shim: restore the np.in1d alias, and inject tiny wrapper functions into the one pydynpd module that does the offending scalar conversions (specification_tests). Because Python module globals shadow builtins, the injected float and math.sqrt wrappers are picked up only inside that module — the rest of the session is untouched. Section 11’s digit-for-digit replication of the package’s published benchmark confirms the shim does not perturb any estimate.

import contextlib
import io
import math
import types
import warnings
# plt.show() is kept for interactive use; silence the no-op warning when the
# script runs headless (MPLBACKEND=Agg)
warnings.filterwarnings("ignore", message="FigureCanvasAgg is non-interactive")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pathlib import Path
# -- pydynpd 0.2.2 / NumPy 2.x compatibility shim ---------------------
# pydynpd 0.2.2 predates NumPy 2.0, which removed np.in1d and forbids
# float()/math.sqrt() on 1x1 matrices. Module globals shadow builtins,
# so injecting wrappers into pydynpd.specification_tests restores both
# behaviors without forking the package.
if not hasattr(np, "in1d"):
np.in1d = np.isin
import pydynpd
if getattr(pydynpd, "__version__", "0.2.2") not in ("0.2.2",):
warnings.warn("Compat shim was written for pydynpd 0.2.2 - "
"re-test before trusting results on a newer version")
from pydynpd import specification_tests as _st
def _scalar(v):
return np.asarray(v).item() if np.ndim(v) else v
_st.float = lambda v: float(_scalar(v))
_st.math = types.SimpleNamespace(sqrt=lambda v: math.sqrt(_scalar(v)))
from pydynpd import regression # import after shim
import pyfixest as pf

Next, the configuration block: the random seed (used only to pick which firms appear in Figure 1 — every estimator below is closed-form and deterministic), the site color palette, the dark-theme matplotlib settings, and two strings that define our running specification. SPEC_MAIN is the pydynpd formula for the AR(1) labor-demand model — n on its first lag plus current and lagged w and k — and GMM_FULL declares that all lags from $t-2$ back to the start of the sample (2:99) of n, w, and k are available as GMM-style instruments.

RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
# Site color palette
STEEL_BLUE = "#6a9bcc"
WARM_ORANGE = "#d97757"
NEAR_BLACK = "#141413"
TEAL = "#00d4c8"
GRAY = "#999999"
# Dark theme palette
DARK_NAVY = "#0f1729"
GRID_LINE = "#1f2b5e"
LIGHT_TEXT = "#c8d0e0"
WHITE_TEXT = "#e8ecf2"
plt.rcParams.update({
"figure.facecolor": DARK_NAVY,
"axes.facecolor": DARK_NAVY,
"axes.edgecolor": DARK_NAVY,
"axes.linewidth": 0,
"axes.labelcolor": LIGHT_TEXT,
"axes.titlecolor": WHITE_TEXT,
"axes.spines.top": False,
"axes.spines.right": False,
"axes.spines.left": False,
"axes.spines.bottom": False,
"axes.grid": True,
"grid.color": GRID_LINE,
"grid.linewidth": 0.6,
"grid.alpha": 0.8,
"xtick.color": LIGHT_TEXT,
"ytick.color": LIGHT_TEXT,
"xtick.major.size": 0,
"ytick.major.size": 0,
"text.color": WHITE_TEXT,
"font.size": 12,
"legend.frameon": False,
"legend.fontsize": 11,
"legend.labelcolor": LIGHT_TEXT,
"figure.edgecolor": DARK_NAVY,
"savefig.facecolor": DARK_NAVY,
"savefig.edgecolor": DARK_NAVY,
})
DATA_PATH = Path("abdata.csv")
SLUG = "python_dynamic_panel"
VAR_LABELS = {
"n": "Log employment",
"w": "Log real wage",
"k": "Log capital stock",
"ys": "Log industry output",
}
SPEC_MAIN = "n L(1:1).n L(0:1).w L(0:1).k"
GMM_FULL = "gmm(n, 2:99) gmm(w, 2:99) gmm(k, 2:99)"

Finally, two small helpers we will reuse constantly. run_abond wraps pydynpd.regression.abond — the package’s single entry point, which takes a Stata-style command string, the dataframe, and the panel identifiers ["id", "year"] — and optionally swallows the table it prints (useful in the grid experiment of Section 10, where we run six models and want a readable log). gmm_summary pulls the headline numbers out of a fitted model: $\hat{\rho}$ and its standard error, a 95 percent confidence interval, the Hansen and AR-test p-values, and the instrument count.

def run_abond(command_str, df, quiet=False):
"""Run pydynpd and return the first fitted model.
pydynpd prints its regression table to stdout as a side effect; quiet=True
suppresses that (used in the proliferation grid to keep the log readable).
"""
if quiet:
with contextlib.redirect_stdout(io.StringIO()):
return regression.abond(command_str, df, ["id", "year"]).models[0]
return regression.abond(command_str, df, ["id", "year"]).models[0]
def gmm_summary(model, label):
"""Extract headline numbers from a fitted pydynpd model."""
rt = model.regression_table
rho = rt.loc[rt.variable == "L1.n", "coefficient"].iloc[0]
se = rt.loc[rt.variable == "L1.n", "std_err"].iloc[0]
return {
"estimator": label,
"rho1": rho,
"se": se,
"ci_lo": rho - 1.96 * se,
"ci_hi": rho + 1.96 * se,
"hansen_p": model.hansen.p_value,
"ar1_p": model.AR_list[0].P_value,
"ar2_p": model.AR_list[1].P_value,
"n_instruments": model.z_information.num_instr,
}

(The downloadable script.py adds cosmetic section banners between these blocks; everything substantive appears here verbatim.) With the tools loaded, let us meet the data that launched a thousand GMM papers.

3. Data loading and panel structure

The dataset is the original Arellano and Bond (1991) panel: an unbalanced sample of UK manufacturing firms observed annually from 1976 to 1984, distributed with pydynpd (and with Stata, R’s plm, and virtually every dynamic-panel teaching resource since). It is the canonical teaching dataset of this literature — the same data Arellano and Bond, Blundell and Bond (1998), and Roodman (2009) all used to illustrate the estimators — so every number we produce can be checked against forty years of published output. The columns we use are already in logs: n (employment), w (real wage), k (gross capital), and ys (industry output).

Before estimating anything, we want three facts: how big the panel is, how unbalanced it is, and — most importantly — where the variation lives. The last question is answered by splitting the standard deviation of log employment into a between-firm part (how much firms differ from each other on average) and a within-firm part (how much each firm moves around its own average). That decomposition will tell us in advance how much trouble $\alpha_i$ is going to cause.

df = pd.read_csv(DATA_PATH)
print(f"Dataset shape: {df.shape}")
print(f"Firms: {df['id'].nunique()}, years: {df['year'].min()}-{df['year'].max()}")
obs_per_firm = df.groupby("id").size()
print("\nObservations per firm (unbalanced panel):")
print(obs_per_firm.value_counts().sort_index().rename_axis("years_observed")
.to_frame("n_firms").to_string())
print("\nSummary statistics (log variables used in estimation):")
print(df[["n", "w", "k", "ys"]].describe().round(3).to_string())
firm_mean_n = df.groupby("id")["n"].transform("mean")
print(f"\nBetween-firm SD of log employment: {df.groupby('id')['n'].mean().std():.3f}")
print(f"Within-firm SD of log employment: {(df['n'] - firm_mean_n).std():.3f}")

Dataset shape: (1031, 10)
Firms: 140, years: 1976-1984
Observations per firm (unbalanced panel):
n_firms
years_observed
7 103
8 23
9 14
Summary statistics (log variables used in estimation):
n w k ys
count 1031.000 1031.000 1031.000 1031.000
mean 1.056 3.143 -0.442 4.638
std 1.342 0.263 1.514 0.094
min -2.263 2.082 -4.431 4.465
25% 0.166 3.027 -1.510 4.576
50% 0.827 3.178 -0.658 4.611
75% 1.949 3.314 0.406 4.706
max 4.687 3.812 3.852 4.855
Between-firm SD of log employment: 1.339
Within-firm SD of log employment: 0.195

Interpretation. The panel holds 1,031 firm-year observations on 140 firms — short and wide: many firms (large N) observed for only 7 to 9 years each (small T). That geometry matters twice over. It is exactly the setting dynamic-panel GMM was designed for, and exactly where Nickell bias — which shrinks at rate $1/T$ — bites hardest. The panel is unbalanced (103 firms appear for 7 years, 23 for 8, 14 for all 9), which all our estimators handle natively. The decisive number is the variance decomposition: the between-firm SD of log employment (1.339) is nearly seven times the within-firm SD (0.195). Employment differences live almost entirely across firms, which means the unobserved firm level $\alpha_i$ is the dominant feature of these data — and any estimator that mishandles it will be wrong by a lot, not a little.

A picture makes the same point more vividly. We plot the log-employment path of 40 randomly chosen firms (the only place the seed is used), the median across all 140 firms, and one example firm.

rng = np.random.default_rng(RANDOM_SEED)
sample_ids = rng.choice(df["id"].unique(), size=40, replace=False)
fig, ax = plt.subplots(figsize=(9, 5.5))
fig.patch.set_linewidth(0)
for fid in sample_ids:
firm = df[df["id"] == fid].sort_values("year")
ax.plot(firm["year"], firm["n"], color=STEEL_BLUE, alpha=0.35, lw=1.2)
median_path = df.groupby("year")["n"].median()
ax.plot(median_path.index, median_path.values, color=WARM_ORANGE, lw=3,
label="Median firm (all 140)")
big = df[df["id"] == obs_per_firm.idxmax()].sort_values("year")
ax.plot(big["year"], big["n"], color=TEAL, lw=2.2, label="One example firm")
ax.set_xlabel("Year")
ax.set_ylabel(VAR_LABELS["n"])
ax.set_title("Firm employment paths are persistent and parallel-ish:\n"
"each firm orbits its own level - a firm fixed effect",
fontsize=13)
ax.legend(loc="upper right")
plt.savefig(f"{SLUG}_trajectories.png", dpi=300, bbox_inches="tight",
facecolor=DARK_NAVY, edgecolor=DARK_NAVY, pad_inches=0)
plt.show()

Interpretation. Each blue line is one firm, and the picture shows the two ingredients that make $\rho$ hard to estimate, simultaneously. First, the lines are roughly parallel and rarely cross: each firm orbits its own level, the visual signature of a large firm fixed effect $\alpha_i$. Second, each line is smooth: a firm’s employment this year looks a lot like last year, the visual signature of high persistence. The orange median drifts gently downward after 1980 — the early-1980s UK manufacturing recession, a common shock that the year dummies $\delta_t$ will absorb in every model below. This figure is the “why” of the whole tutorial: an estimator that ignores $\alpha_i$ will mistake orbit differences for persistence, and one that removes $\alpha_i$ clumsily will damage the persistence signal it is trying to measure. Before we can demonstrate either failure, we need lags and differences.

4. Data preparation: lags and first differences

Every estimator below runs on transformed variables: one-period lags of n, w, and k for the level equations, and first differences (this year minus last year, firm by firm) for the Anderson-Hsiao regression. Two details matter. The transformations must respect firm boundaries — firm 2’s first year must not “inherit” a lag from firm 1’s last year, which is why everything goes through groupby("id"). And each lag is expensive: it costs every firm its first observed year, because there is no earlier year to look back to.

d = df.sort_values(["id", "year"]).copy()
g = d.groupby("id")
for v in ["n", "w", "k"]:
d[f"{v}_lag1"] = g[v].shift(1)
d["n_lag2"] = g["n"].shift(2)
for v in ["n", "w", "k"]:
d[f"d_{v}"] = g[v].diff()
d["d_n_lag1"] = g["d_n"].shift(1)
d["d_w_lag1"] = g["d_w"].shift(1)
d["d_k_lag1"] = g["d_k"].shift(1)
est_sample = d.dropna(subset=["n_lag1", "w_lag1", "k_lag1"]).copy()
print(f"Full panel rows: {len(d)}")
print(f"Estimation sample after requiring one lag: {len(est_sample)} rows "
f"({est_sample['id'].nunique()} firms)")
d.to_csv("data_prepared.csv", index=False)

Full panel rows: 1031
Estimation sample after requiring one lag: 891 rows (140 firms)

Interpretation. Requiring a single lag shrinks the sample from 1,031 to 891 rows — a 13.6 percent cut — while keeping all 140 firms. This is the first lesson in dynamic-panel frugality: with T as small as 7, every additional lag or difference burns a meaningful share of the data. Watch the observation counts fall as the methods get hungrier — the GMM estimators below will run on 751 observations, and the two-lag replication specification in Section 11 on just 611. The exported data_prepared.csv carries every lag and difference constructed here, so all estimators run on identically built variables. We are now ready to watch the first two estimators fail — informatively.

5. The bias bracket: pooled OLS vs fixed effects

Here is the heart of the problem, and it is worth slowing down for. We will run the same regression twice — log employment on its lag, wages, capital, and year dummies — changing only how it treats the firm effect $\alpha_i$. Both treatments fail, but in opposite, theoretically known directions, and that is what makes the failure useful.

Why pooled OLS is biased upward. Pooled OLS simply ignores $\alpha_i$, leaving it in the error term. But last year’s employment $n_{i,t-1}$ depends on $\alpha_i$ — high-level firms had high employment last year too — so the regressor is positively correlated with the error. OLS rewards the lag for work the firm effect is doing: persistently large firms look like firms with enormous persistence, and $\hat{\rho}$ gets pushed up toward 1. Think of mistaking the planets’ different orbits for the dynamics of a single planet.

Why fixed effects is biased downward. The within estimator subtracts each firm’s sample average from every variable, which removes $\alpha_i$ exactly. The trap is subtler: the firm average being subtracted is computed over the firm’s whole observation window, so it contains the firm’s future shocks. After demeaning, the equation looks like

$$\tilde{n}_{it} = \rho \tilde{n}_{i,t-1} + \cdots + \tilde{\varepsilon}_{it}, \qquad \tilde{\varepsilon}_{it} = \varepsilon_{it} - \frac{1}{T_i}\sum_{s=1}^{T_i} \varepsilon_{is}$$

In words, this says: every demeaned variable (the tildes) is the original minus the firm’s own average, and the demeaned error at time $t$ contains a slice of every period’s shock — including shocks dated before $t$, which also live inside the demeaned lag $\tilde{n}_{i,t-1}$. That shared content creates a mechanical negative correlation between regressor and error: the Nickell (1981) bias, of order $1/T_i$ (the code’s est_sample has $T_i$ between 6 and 8 after the lag). It is like grading a student against a course average that includes the exams they have not taken yet. Crucially, adding more firms does not help — only longer panels do, and ours is short.

Two failures with known signs are a measurement instrument in their own right: Bond (2002) turned them into a diagnostic. Run both, and you get a bracket that any consistent estimator must land inside.

FORMULA_RHS = "n_lag1 + w + w_lag1 + k + k_lag1"
ols = pf.feols(f"n ~ {FORMULA_RHS} | year", data=est_sample,
vcov={"CRV1": "id"})
fe = pf.feols(f"n ~ {FORMULA_RHS} | id + year", data=est_sample,
vcov={"CRV1": "id"})
print("Pooled OLS (year dummies, SEs clustered by firm):")
print(ols.tidy().round(4).to_string())
print("\nFixed effects / within (firm + year dummies, clustered SEs):")
print(fe.tidy().round(4).to_string())
rho_ols, se_ols = ols.coef()["n_lag1"], ols.se()["n_lag1"]
rho_fe, se_fe = fe.coef()["n_lag1"], fe.se()["n_lag1"]
print(f"\n rho_OLS = {rho_ols:.4f} (se {se_ols:.4f}) <- upper bound (biased up)")
print(f" rho_FE = {rho_fe:.4f} (se {se_fe:.4f}) <- lower bound (biased down)")

The pf.feols call deserves a word on first use: the formula’s | year part absorbs year fixed effects (and | id + year absorbs both firm and year effects) without creating dummy columns, and vcov={"CRV1": "id"} requests cluster-robust standard errors by firm — essential here, because each firm’s observations are serially dependent by the very nature of the model.

Pooled OLS (year dummies, SEs clustered by firm):
Estimate Std. Error t value Pr(>|t|) 2.5% 97.5%
Coefficient
n_lag1 0.9617 0.0084 115.0717 0.0000 0.9452 0.9782
w -0.4147 0.1600 -2.5915 0.0106 -0.7311 -0.0983
w_lag1 0.3556 0.1559 2.2803 0.0241 0.0473 0.6639
k 0.3997 0.0565 7.0710 0.0000 0.2879 0.5114
k_lag1 -0.3675 0.0565 -6.4990 0.0000 -0.4793 -0.2557
Fixed effects / within (firm + year dummies, clustered SEs):
Estimate Std. Error t value Pr(>|t|) 2.5% 97.5%
Coefficient
n_lag1 0.6262 0.0515 12.1510 0.0000 0.5243 0.7281
w -0.5035 0.1450 -3.4729 0.0007 -0.7902 -0.2169
w_lag1 0.2308 0.1077 2.1420 0.0339 0.0178 0.4438
k 0.4078 0.0566 7.2024 0.0000 0.2959 0.5198
k_lag1 -0.1648 0.0547 -3.0100 0.0031 -0.2730 -0.0565
rho_OLS = 0.9617 (se 0.0084) <- upper bound (biased up)
rho_FE = 0.6262 (se 0.0515) <- lower bound (biased down)

Interpretation. The two “wrong” estimators disagree by 0.336 — an enormous gap in economic terms. Pooled OLS says $\hat{\rho} = 0.9617$, so close to a unit root — the $\rho = 1$ boundary at which shocks never decay at all — that an employment shock would have a half-life of about 18 years; fixed effects says $\hat{\rho} = 0.6262$, a half-life of about 1.5 years. Same data, same regression, opposite stories about the labor market — and neither is right, but both errors have known sign. The bracket [0.626, 0.962] is sharply identified: the two cluster-robust confidence intervals ([0.945, 0.978] and [0.524, 0.728]) do not even overlap. Meanwhile the control variables behave sensibly in both columns — the wage elasticity is negative (about −0.41 to −0.50) and the capital elasticity positive (about 0.40) — a reminder that a regression can be perfectly reasonable about its controls while being badly biased on the one coefficient we actually care about.

The bracket deserves its own picture, because it is the yardstick every later estimate will be measured against.

fig, ax = plt.subplots(figsize=(9, 4.8))
fig.patch.set_linewidth(0)
ax.axvspan(rho_fe, rho_ols, color=GRID_LINE, alpha=0.55, zorder=0)
ax.axvline(1.0, color=GRAY, lw=1.2, ls="--", alpha=0.8)
ax.text(1.0, 1.62, "unit root", color=GRAY, fontsize=10, ha="center")
for y, (lab, rho, se, col, note) in enumerate([
("Pooled OLS", rho_ols, se_ols, STEEL_BLUE,
"biased UP: L1.n correlated with firm effect"),
("Fixed effects", rho_fe, se_fe, WARM_ORANGE,
"biased DOWN: Nickell bias (T is small)"),
]):
ax.errorbar(rho, y, xerr=1.96 * se, fmt="o", color=col, ms=10,
capsize=5, lw=2.5, capthick=2.5)
ax.text(rho, y - 0.28, note, color=LIGHT_TEXT, fontsize=10, ha="center")
ax.text((rho_fe + rho_ols) / 2, 1.18,
"consistent estimates\nshould land in here", color=WHITE_TEXT,
fontsize=11, ha="center", style="italic")
ax.set_yticks([0, 1])
ax.set_yticklabels(["Pooled OLS", "Fixed effects"])
ax.set_ylim(-0.6, 1.8)
ax.set_xlabel(r"Estimate of employment persistence $\hat{\rho}$ (L1.n)")
ax.set_title("Two wrong answers that bracket the truth", fontsize=13)
plt.savefig(f"{SLUG}_bias_bracket.png", dpi=300, bbox_inches="tight",
facecolor=DARK_NAVY, edgecolor=DARK_NAVY, pad_inches=0)
plt.show()
ols.tidy().reset_index().to_csv("ols_results.csv", index=False)
fe.tidy().reset_index().to_csv("fe_results.csv", index=False)

Interpretation. The shaded band between 0.626 and 0.962 is the playing field for the rest of the tutorial. Bond’s (2002) rule is simple and powerful: a candidate estimator that lands below the band is suffering something Nickell-like; one that lands above it (or beyond the unit-root line at 1.0) is suffering something OLS-like or worse; and — the subtle case we will meet in Section 7 — one that lands just barely inside the band, hugging an edge, is probably being dragged toward that edge by a fixable weakness. With the yardstick built, we can try the first estimator that is actually consistent for $\rho$: a clever instrumental-variables idea from 1981.

6. Anderson-Hsiao IV: consistent but imprecise

Anderson and Hsiao (1981) proposed a two-step escape from the trap. Step one: difference away the firm effect. Subtracting each firm’s $t-1$ equation from its $t$ equation eliminates $\alpha_i$ exactly — no demeaning, no contamination from future shocks:

$$\Delta n_{it} = \rho \Delta n_{i,t-1} + \beta_1 \Delta w_{it} + \beta_2 \Delta w_{i,t-1} + \beta_3 \Delta k_{it} + \beta_4 \Delta k_{i,t-1} + \Delta\delta_t + \Delta\varepsilon_{it}$$

In words, this says: the change in employment depends on the lagged change in employment, the changes in the controls, and the change in the shock — and $\alpha_i$, being constant, has vanished from the equation. In the code, the deltas are the constructed columns d_n, d_n_lag1, d_w, and so on. But differencing creates a new endogeneity problem: $\Delta n_{i,t-1} = n_{i,t-1} - n_{i,t-2}$ contains $\varepsilon_{i,t-1}$, and $\Delta\varepsilon_{it} = \varepsilon_{it} - \varepsilon_{i,t-1}$ contains it too — the regressor and the error share a term again.

Step two: instrument the differenced lag. An instrument is a variable correlated with the troublesome regressor but uncorrelated with the error. The level $n_{i,t-2}$ qualifies: it obviously helps predict the change $\Delta n_{i,t-1}$, and — provided $\varepsilon_{it}$ is not serially correlated — it predates and is therefore independent of both shocks inside $\Delta\varepsilon_{it}$. This is sequential exogeneity doing its first day of work. We estimate by two-stage least squares (2SLS), which pyfixest expresses with a third formula part: d_n_lag1 ~ n_lag2.

ah_sample = d.dropna(subset=["d_n", "d_n_lag1", "d_w", "d_w_lag1",
"d_k", "d_k_lag1", "n_lag2"]).copy()
ah = pf.feols("d_n ~ d_w + d_w_lag1 + d_k + d_k_lag1 | year | d_n_lag1 ~ n_lag2",
data=ah_sample, vcov={"CRV1": "id"})
print("Anderson-Hsiao 2SLS (differences, year dummies, clustered SEs):")
print(ah.tidy().round(4).to_string())
rho_ah, se_ah = ah.coef()["d_n_lag1"], ah.se()["d_n_lag1"]
print(f"\n rho_AH = {rho_ah:.4f} (se {se_ah:.4f})")
print(f" 95% CI: [{rho_ah - 1.96 * se_ah:.3f}, {rho_ah + 1.96 * se_ah:.3f}]")
ah.tidy().reset_index().to_csv("anderson_hsiao_results.csv", index=False)

Anderson-Hsiao 2SLS (differences, year dummies, clustered SEs):
Estimate Std. Error t value Pr(>|t|) 2.5% 97.5%
Coefficient
d_w -0.5243 0.2135 -2.4556 0.0153 -0.9465 -0.1021
d_w_lag1 0.5808 0.3128 1.8566 0.0655 -0.0377 1.1992
d_k 0.2463 0.0777 3.1693 0.0019 0.0927 0.4000
d_k_lag1 -0.2925 0.1964 -1.4890 0.1388 -0.6808 0.0959
d_n_lag1 1.2327 0.4782 2.5781 0.0110 0.2873 2.1781
rho_AH = 1.2327 (se 0.4782)
95% CI: [0.296, 2.170]

Interpretation. The point estimate is $\hat{\rho} = 1.2327$ — above the unit root, outside the bracket — with a standard error of 0.4782, nearly 60 times the OLS standard error. The 95 percent confidence interval [0.296, 2.170] is 1.87 units wide: it contains the entire OLS-FE bracket, the unit root, and explosive dynamics, all at once. Taken literally, the estimate says every employment shock amplifies over time, which nobody believes; taken correctly, it says that one instrument extracts far too little information from a highly persistent series to be useful. Anderson-Hsiao is the courtroom with a single admissible witness: the testimony is valid, but you cannot convict on it. The fix is not a better witness but more of them — and that observation is the doorway to GMM. If $n_{i,t-2}$ is a valid instrument, then by the same logic so are $n_{i,t-3}$, $n_{i,t-4}$, and every deeper lag of every sequentially exogenous variable.

7. Difference GMM (Arellano-Bond 1991)

Arellano and Bond’s insight turns Anderson-Hsiao’s single moment condition into a whole family. Under sequential exogeneity and no serial correlation in $\varepsilon_{it}$, every level dated $t-2$ or earlier is uncorrelated with the differenced error:

$$E[n_{i,t-s} \Delta\varepsilon_{it}] = 0 \quad \text{for all } s \ge 2$$

In words, this says: the firm’s employment level two or more years ago carries no information about this year’s change in shocks — so each such lag can serve as an instrument for the differenced equation. The same holds for lagged w and k. With T = 9, that is dozens of conditions (later periods have more usable lags than early ones), and the generalized method of moments — a framework that finds the parameter values making all these zero-correlation conditions hold as closely as possible, weighting the more informative ones more heavily — combines them optimally. The pydynpd command string encodes exactly this: our SPEC_MAIN plus gmm(n, 2:99) gmm(w, 2:99) gmm(k, 2:99) (use every lag from depth 2 onward), timedumm (add year dummies), and nolevel (differenced equation only — that is what makes it difference GMM).

One estimation detail to know before reading the output: GMM comes in one-step and two-step flavors. Two-step re-weights the moment conditions using the first step’s residuals, which is asymptotically more efficient, but its naive standard errors are badly downward-biased in samples like ours — so pydynpd reports the Windmeijer (2005) finite-sample correction, and the two-step column is the one to quote.

print("One-step difference GMM:")
diff_one = run_abond(f"{SPEC_MAIN} | {GMM_FULL} | timedumm nolevel onestep", d)
print("\nTwo-step difference GMM (Windmeijer-corrected SEs):")
diff_two = run_abond(f"{SPEC_MAIN} | {GMM_FULL} | timedumm nolevel", d)
s1 = gmm_summary(diff_one, "Diff GMM (one-step)")
s2 = gmm_summary(diff_two, "Diff GMM (two-step)")
print(f"\n one-step: rho = {s1['rho1']:.4f} (se {s1['se']:.4f})")
print(f" two-step: rho = {s2['rho1']:.4f} (se {s2['se']:.4f}), "
f"{s2['n_instruments']} instruments")
diff_two.regression_table.to_csv("diff_gmm_results.csv", index=False)

The two-step table (year-dummy rows omitted for brevity; the full table is in execution_log.txt):

 Dynamic panel-data estimation, two-step difference GMM
Group variable: id Number of obs = 751
Time variable: year Min obs per group: 5
Number of instruments = 91 Max obs per group: 7
Number of groups = 140 Avg obs per group: 5.36
+-----------+------------+---------------------+------------+-----------+-----+
| n | coef. | Corrected Std. Err. | z | P>|z| | |
+-----------+------------+---------------------+------------+-----------+-----+
| L1.n | 0.6787867 | 0.0890781 | 7.6201324 | 0.0000000 | *** |
| w | -0.7198296 | 0.1221408 | -5.8934431 | 0.0000000 | *** |
| L1.w | 0.4626914 | 0.1134755 | 4.0774568 | 0.0000455 | *** |
| k | 0.4539046 | 0.1275537 | 3.5585358 | 0.0003729 | *** |
| L1.k | -0.1914923 | 0.1044671 | -1.8330393 | 0.0667967 | |
+-----------+------------+---------------------+------------+-----------+-----+
Hansen test of overid. restrictions: chi(79) = 88.797 Prob > Chi2 = 0.211
Arellano-Bond test for AR(1) in first differences: z = -4.46 Pr > z =0.000
Arellano-Bond test for AR(2) in first differences: z = -0.17 Pr > z =0.866
one-step: rho = 0.7075 (se 0.0842)
two-step: rho = 0.6788 (se 0.0891), 91 instruments

Interpretation. The machinery works exactly as advertised — 91 instruments across 751 usable observations, a two-step estimate of $\hat{\rho} = 0.6788$ (SE 0.0891), and every printed diagnostic passes: AR(1) rejects as it mechanically must (z = −4.46, p = 0.000 — see Section 9), AR(2) is nowhere near rejecting (p = 0.866), and Hansen accepts the overidentifying restrictions — with 91 instruments for far fewer parameters, the extra moment conditions must all tell a mutually consistent story, and here they do ($\chi^2(79) = 88.797$, p = 0.211). And yet this estimate should not be trusted, for a reason no printed test reveals: it sits only 0.053 above the FE lower bound of 0.626 — within one standard error of it, in the bottom sixth of the bracket. This is Bond’s (2002) informal diagnostic failing loudly. The mechanism is intuitive once seen: when the true series is highly persistent, the level two years ago barely predicts this year’s change — a near-random-walk series changes unpredictably — so all 91 instruments are individually weak, and weak-instrument bias in this design points toward the within estimator. Blundell and Bond (1998) demonstrated precisely this failure on precisely this dataset. An estimator that passes every formal test while giving a suspect answer is the single most valuable lesson of this tutorial — and their fix is the next section.

8. System GMM (Blundell-Bond 1998): the headline model

If lagged levels are weak instruments for differences, Blundell and Bond asked, what about the reverse? Lagged differences turn out to be strong instruments for levels — even for a persistent series, last year’s change is informative about this year’s level. System GMM therefore estimates a stacked system of both equations at once: the differenced equation keeps its Arellano-Bond instruments, and the original levels equation (which retains $\alpha_i$) gets instrumented by lagged differences. The new moment conditions are

$$E[\Delta n_{i,t-1}(\alpha_i + \varepsilon_{it})] = 0$$

In words, this says: last year’s employment change must be uncorrelated with the firm’s permanent level $\alpha_i$ (and with today’s shock). That is a genuinely new assumption — mean stationarity: firms may sit at wildly different steady-state levels, but their initial deviations from those steady states must be unrelated to the levels themselves. Picture boats anchored at different depths along a coastline: the anchors differ (the $\alpha_i$), but each boat bobs around its own anchor in the same way — no boat starts systematically far from its mooring. It is untestable directly, but the Hansen test gets indirect bite on it because the levels moments are overidentifying.

Two practical choices complete the headline specification. We use collapsed instruments — pydynpd’s collapse option combines each lag depth into a single instrument column instead of one column per depth-and-period combination, holding the count to 32, safely below Roodman’s (2009) rule of thumb that instruments should not outnumber the 140 firms. And we again quote the two-step, Windmeijer-corrected results. Dropping nolevel from the command string is what switches pydynpd from difference to system GMM.

print("Two-step system GMM, collapsed instruments:")
sys_two = run_abond(f"{SPEC_MAIN} | {GMM_FULL} | timedumm collapse", d)
print("\nOne-step system GMM, collapsed instruments:")
sys_one = run_abond(f"{SPEC_MAIN} | {GMM_FULL} | timedumm collapse onestep", d)
s3 = gmm_summary(sys_two, "Sys GMM (two-step, collapsed)")
s4 = gmm_summary(sys_one, "Sys GMM (one-step, collapsed)")
print(f"\n two-step: rho = {s3['rho1']:.4f} (se {s3['se']:.4f}), "
f"{s3['n_instruments']} instruments")
print(f" one-step: rho = {s4['rho1']:.4f} (se {s4['se']:.4f})")
sys_two.regression_table.to_csv("sys_gmm_results.csv", index=False)

The two-step table (year-dummy rows omitted; full table in execution_log.txt):

 Dynamic panel-data estimation, two-step system GMM
Group variable: id Number of obs = 751
Time variable: year Min obs per group: 5
Number of instruments = 32 Max obs per group: 7
Number of groups = 140 Avg obs per group: 5.36
+-----------+------------+---------------------+------------+-----------+-----+
| n | coef. | Corrected Std. Err. | z | P>|z| | |
+-----------+------------+---------------------+------------+-----------+-----+
| L1.n | 0.9269913 | 0.0785085 | 11.8075341 | 0.0000000 | *** |
| w | -0.8155041 | 0.2763832 | -2.9506278 | 0.0031713 | ** |
| L1.w | 0.6331152 | 0.3327639 | 1.9025958 | 0.0570933 | |
| k | 0.5894690 | 0.1715356 | 3.4364236 | 0.0005894 | *** |
| L1.k | -0.4888581 | 0.1969821 | -2.4817381 | 0.0130743 | * |
| _con | 0.6404202 | 0.4628017 | 1.3837897 | 0.1664229 | |
+-----------+------------+---------------------+------------+-----------+-----+
Hansen test of overid. restrictions: chi(19) = 18.918 Prob > Chi2 = 0.462
Arellano-Bond test for AR(1) in first differences: z = -4.49 Pr > z =0.000
Arellano-Bond test for AR(2) in first differences: z = -0.01 Pr > z =0.994
two-step: rho = 0.9270 (se 0.0785), 32 instruments
one-step: rho = 0.9025 (se 0.0634)

Interpretation. This is the headline of the tutorial: $\hat{\rho} = 0.9270$ (SE 0.0785) — inside the bracket, in its upper half, 0.25 above the weak-instrument difference-GMM estimate, with 32 collapsed instruments and textbook diagnostics: AR(1) rejects mechanically (p = 0.000, as it should), AR(2) is immaculate (z = −0.01, p = 0.994), and Hansen sits at p = 0.462, comfortably away from both the 0.05 rejection region and the p-near-1 overfitting flag. Substantively, about 93 percent of an employment shock survives into the next year — a shock half-life of roughly nine years ($0.927^5 \approx 0.68$ still present after five) — versus the 1.5 years fixed effects would have claimed. One honest caveat belongs right next to the headline: the 95 percent CI [0.773, 1.081] includes 1.0, so a unit root cannot be rejected at the 5 percent level; the defensible claim is the point estimate and its lower bound, not “employment is stationary.” The factor demands also sharpen: the short-run wage elasticity is −0.8155 (SE 0.2764) and the capital elasticity 0.5895 (SE 0.1715). Resist the temptation to report the implied long-run wage elasticity $(\beta_1 + \beta_2)/(1 - \rho) \approx -2.5$ without a warning — its denominator $1 - \rho \approx 0.073$ makes it explosively fragile to tiny changes in $\hat{\rho}$.

We have leaned on the phrases “AR(2) clean” and “Hansen comfortable” several times now. Before stress-testing the headline, let us make the diagnostic logic itself explicit — because two of the three tests are routinely read backwards.

9. Reading the diagnostics: AR(1), AR(2), and Hansen

Every dynamic-panel GMM table ends with the same three tests, and each has a non-obvious reading. Here is the decoder, using our headline model’s values.

Test	What it checks	Correct reading	Our headline value
AR(1) in differences	First-order serial correlation in $\Delta\varepsilon_{it}$	Must reject. Rejection is mechanical and good news	z = −4.49, p = 0.000 — rejects, as required
AR(2) in differences	Second-order serial correlation in $\Delta\varepsilon_{it}$, which would mean $\varepsilon_{it}$ itself is AR(1)	Must not reject. This is the test that validates the $t-2$ instruments	z = −0.01, p = 0.994 — clean
Hansen J	Joint validity of the overidentifying restrictions	Two-tailed in spirit. p < 0.05 means instruments look invalid; p near 1 means the test has been overwhelmed by too many instruments	p = 0.462 — comfortable on both sides

Why AR(1) must reject. Differencing makes adjacent errors overlap: $\Delta\varepsilon_{it} = \varepsilon_{it} - \varepsilon_{i,t-1}$ and $\Delta\varepsilon_{i,t-1} = \varepsilon_{i,t-1} - \varepsilon_{i,t-2}$ share the term $\varepsilon_{i,t-1}$ — like adjacent dominoes sharing an edge — so consecutive differenced errors are negatively correlated when the model is right. A beginner who sees “AR(1): p = 0.000” and concludes the model failed has it exactly backwards; it is the absence of AR(1) rejection that should raise eyebrows (as in our Section 11 replication, where the two-lag specification has already absorbed the dependence, AR(1) p = 0.198).

Why AR(2) is the one that matters. If $\varepsilon_{it}$ were serially correlated, then $n_{i,t-2}$ would contain $\varepsilon_{i,t-2}$-flavored information that also lives in $\Delta\varepsilon_{i,t-1}$ — the witnesses would have been tipped off, and the entire instrument set would be invalid. AR(2) in differences is precisely the test for that contamination. Our p = 0.994 could hardly be cleaner.

Why a big Hansen p-value is not automatically good news. The Hansen J test asks whether the 32 moment conditions are mutually consistent. Reading it one-tailed (“bigger p is better”) is the classic trap: as Section 10 demonstrates experimentally, piling on instruments inflates the p-value mechanically, with the notorious “Hansen p = 1.000” as the terminal symptom of an overfitted, powerless test. Roodman (2009) warns that implausibly high values approaching 1.0 signal an overwhelmed test rather than valid instruments. Our 0.462 with only 32 collapsed instruments (against 140 firms) is in the comfortable middle — but the same 0.462 with 130 instruments would be a warning, not a pass.

That last claim — that the Hansen p-value responds to the instrument count, not just instrument validity — is testable on our own data. Let us run the experiment.

10. Instrument proliferation: lag windows vs collapse

We re-estimate the identical system-GMM model six times, varying only two plumbing choices: the lag window (2:3 = use lags 2 and 3 only; 2:5; 2:99 = use everything) and whether the instrument matrix is collapsed. If the Hansen test were a pure validity meter, its p-value would be roughly stable across these cells. It is not.

grid_specs = [
("2:3", False), ("2:3", True),
("2:5", False), ("2:5", True),
("2:99", False), ("2:99", True),
]
grid_rows = []
for window, collapsed in grid_specs:
gmm_part = f"gmm(n, {window}) gmm(w, {window}) gmm(k, {window})"
opts = "timedumm collapse" if collapsed else "timedumm"
model = run_abond(f"{SPEC_MAIN} | {gmm_part} | {opts}", d, quiet=True)
row = gmm_summary(model, f"sys GMM lags {window}"
+ (", collapsed" if collapsed else ""))
row["lag_window"] = window
row["collapsed"] = collapsed
grid_rows.append(row)
print(f" lags {window:5s} collapse={str(collapsed):5s} -> "
f"{row['n_instruments']:3d} instruments, rho = {row['rho1']:.3f}, "
f"Hansen p = {row['hansen_p']:.3f}")
grid = pd.DataFrame(grid_rows)
grid.to_csv("proliferation_grid.csv", index=False)

 lags 2:3 collapse=False -> 68 instruments, rho = 0.956, Hansen p = 0.035
lags 2:3 collapse=True -> 17 instruments, rho = 0.921, Hansen p = 0.096
lags 2:5 collapse=False -> 95 instruments, rho = 0.935, Hansen p = 0.186
lags 2:5 collapse=True -> 23 instruments, rho = 0.937, Hansen p = 0.255
lags 2:99 collapse=False -> 113 instruments, rho = 0.930, Hansen p = 0.235
lags 2:99 collapse=True -> 32 instruments, rho = 0.927, Hansen p = 0.462

The full grid, with standard errors and AR(2) p-values from proliferation_grid.csv:

Lag window	Collapsed	Instruments	$\hat{\rho}$	SE	Hansen p	AR(2) p
2:3	no	68	0.9555	0.0322	0.0348	0.7631
2:3	yes	17	0.9211	0.1001	0.0957	0.9343
2:5	no	95	0.9354	0.0335	0.1859	0.7343
2:5	yes	23	0.9374	0.0982	0.2546	0.9084
2:99	no	113	0.9296	0.0274	0.2349	0.8052
2:99	yes	32	0.9270	0.0785	0.4621	0.9944

Interpretation. Two findings, both reassuring and unsettling at once. The reassuring one: $\hat{\rho}$ barely moves — all six cells land in the narrow range [0.921, 0.956], so the point estimate is robust to the plumbing. The unsettling one: the test we would use to defend it is not. Reading the uncollapsed rows top to bottom, the instrument count climbs 68, 95, 113 (approaching the 140-firm ceiling) and the Hansen p-value drifts 0.035, 0.186, 0.235 — even though the model never changes. That upward drift is the overfitting trajectory whose endpoint is the “p = 1.000” red flag. The grid also catches proliferation distorting the test in the other tail: the uncollapsed 2:3 specification is outright rejected by Hansen (p = 0.0348 < 0.05) while its collapsed twin passes (p = 0.0957) — same lag window, same data, opposite verdicts, driven purely by instrument count.

The grid deserves its own picture: instrument count on the x-axis, Hansen p-value on the y-axis, full-matrix and collapsed specifications as two marker series.

fig, ax = plt.subplots(figsize=(9, 5.5))
fig.patch.set_linewidth(0)
for collapsed, col, marker, lab in [(False, STEEL_BLUE, "o", "Full instrument matrix"),
(True, TEAL, "D", "Collapsed instruments")]:
sub = grid[grid["collapsed"] == collapsed]
ax.scatter(sub["n_instruments"], sub["hansen_p"], s=140, color=col,
marker=marker, edgecolors=DARK_NAVY, lw=1.5, zorder=3, label=lab)
for _, r in sub.iterrows():
ax.annotate(f"lags {r['lag_window']}",
(r["n_instruments"], r["hansen_p"]),
textcoords="offset points", xytext=(0, 12),
color=LIGHT_TEXT, fontsize=9.5, ha="center")
ax.axhline(0.05, color=WARM_ORANGE, lw=1.5, ls="--")
ax.text(137, 0.022, "p = 0.05: instruments rejected below this line",
color=WARM_ORANGE, fontsize=9.5, ha="right", va="top")
ax.axvline(140, color=GRAY, lw=1.5, ls=":")
ax.text(138, 0.78, "N = 140 firms\n(Roodman's ceiling)", color=GRAY,
fontsize=9.5, ha="right")
ax.set_xlabel("Number of instruments")
ax.set_ylabel("Hansen test p-value")
ax.set_ylim(-0.06, 1.0)
ax.set_title("Instrument proliferation: more is not better", fontsize=13)
ax.legend(loc="upper left")
plt.savefig(f"{SLUG}_instrument_proliferation.png", dpi=300,
bbox_inches="tight", facecolor=DARK_NAVY, edgecolor=DARK_NAVY,
pad_inches=0)
plt.show()

Interpretation. The figure makes the trade visible: the teal diamonds (collapsed) buy nearly identical point estimates with a quarter of the instruments, at the honest price of larger standard errors — 0.0785 collapsed versus 0.0274 uncollapsed at the 2:99 window. That uncollapsed SE looks like a precision triumph, but it is precisely the too-good-to-be-true precision Roodman warns about: 113 instruments fitted to 140 firms are partly fitting noise, and the same overfitting that flatters the SE is what disarms the Hansen test. Our headline deliberately takes the larger, more honest standard error. One robustness layer remains: proving that the software itself computes what it claims, by replicating a published benchmark exactly.

11. Replication check: the pydynpd documentation example

Good practice with any estimation package — especially one we patched with a compatibility shim — is to replicate its published benchmark before trusting novel output. The pydynpd README estimates the original Arellano-Bond (1991) two-lag specification on this same dataset: two lags of n on the right-hand side, a restricted instrument window gmm(n, 2:4), wages treated as predetermined (correlated with past shocks but not the current one) with gmm(w, 1:3), capital as a standard exogenous instrument iv(k), and difference GMM (nolevel). The package authors validated that output against Stata’s xtabond2. Note this is a different model from our running specification — the point here is toolchain verification, not a second opinion on $\rho$. The script enforces the match with a hard assertion that would abort the run on any discrepancy.

ab_repl = run_abond(
"n L(1:2).n w k | gmm(n, 2:4) gmm(w, 1:3) iv(k) | timedumm nolevel", d)
rt = ab_repl.regression_table
rho_repl = rt.loc[rt.variable == "L1.n", "coefficient"].iloc[0]
print(f"\n Published vignette values: L1.n = 0.2710675, Hansen chi2 = 32.666,")
print(" 42 instruments.")
print(f" Our run: L1.n = {rho_repl:.7f}, Hansen chi2 = "
f"{ab_repl.hansen.test_value:.3f}, {ab_repl.z_information.num_instr} instruments.")
match = (abs(rho_repl - 0.2710675) < 1e-6
and abs(ab_repl.hansen.test_value - 32.666) < 1e-3
and ab_repl.z_information.num_instr == 42)
print(f" Exact match: {match}")
if not match:
raise AssertionError("Replication check failed - investigate before publishing")
ab_repl.regression_table.to_csv("ab_replication_results.csv", index=False)

 Dynamic panel-data estimation, two-step difference GMM
Group variable: id Number of obs = 611
Number of instruments = 42 Number of groups = 140
+-----------+------------+---------------------+------------+-----------+-----+
| L1.n | 0.2710675 | 0.1382542 | 1.9606462 | 0.0499203 | * |
| L2.n | -0.0233928 | 0.0419665 | -0.5574151 | 0.5772439 | |
| w | -0.5668527 | 0.2092231 | -2.7093219 | 0.0067421 | ** |
| k | 0.3613939 | 0.0662624 | 5.4539824 | 0.0000000 | *** |
+-----------+------------+---------------------+------------+-----------+-----+
Hansen test of overid. restrictions: chi(32) = 32.666 Prob > Chi2 = 0.434
Arellano-Bond test for AR(1) in first differences: z = -1.29 Pr > z =0.198
Arellano-Bond test for AR(2) in first differences: z = -0.31 Pr > z =0.760
Published vignette values: L1.n = 0.2710675, Hansen chi2 = 32.666,
42 instruments.
Our run: L1.n = 0.2710675, Hansen chi2 = 32.666, 42 instruments.
Exact match: True

Interpretation. The replication is exact to all printed digits: L1.n = 0.2710675, Hansen $\chi^2(32)$ = 32.666 (p = 0.434), 42 instruments, 611 observations — Exact match: True under the hard assertion. This verifies the entire toolchain, NumPy-2 shim included, against the package’s own benchmark (itself validated against xtabond2). Two teaching points hide in the output. First, the much lower persistence coefficient here (0.271) is not a contradiction of our headline 0.927: this is difference GMM — subject to the same weak-instrument drag we diagnosed in Section 7 — on a two-lag dynamic specification with a restricted instrument window. “The” persistence estimate is always joint with the specification and estimator that produced it. Second, notice AR(1) here does not reject (p = 0.198): with two lags of n soaking up the dynamics, even the mechanical differencing correlation is muted — another reminder that diagnostic values must be read against the model, not against a universal rulebook. Time to put all seven estimates on one axis.

12. Synthesis: seven estimators, one parameter

Each section produced a number; the story only snaps into focus when they share an axis. We assemble the summary table and draw the forest plot that the whole tutorial has been building toward.

summary_rows = [
{"estimator": "Pooled OLS", "rho1": rho_ols, "se": se_ols,
"ci_lo": rho_ols - 1.96 * se_ols, "ci_hi": rho_ols + 1.96 * se_ols,
"hansen_p": np.nan, "ar1_p": np.nan, "ar2_p": np.nan,
"n_instruments": np.nan},
{"estimator": "Fixed effects", "rho1": rho_fe, "se": se_fe,
"ci_lo": rho_fe - 1.96 * se_fe, "ci_hi": rho_fe + 1.96 * se_fe,
"hansen_p": np.nan, "ar1_p": np.nan, "ar2_p": np.nan,
"n_instruments": np.nan},
{"estimator": "Anderson-Hsiao IV", "rho1": rho_ah, "se": se_ah,
"ci_lo": rho_ah - 1.96 * se_ah, "ci_hi": rho_ah + 1.96 * se_ah,
"hansen_p": np.nan, "ar1_p": np.nan, "ar2_p": np.nan,
"n_instruments": 1},
s1, s2, s4, s3,
]
summary = pd.DataFrame(summary_rows)
print(summary.round(4).to_string(index=False))
summary.to_csv("estimates_summary.csv", index=False)

 estimator rho1 se ci_lo ci_hi hansen_p ar1_p ar2_p n_instruments
Pooled OLS 0.9617 0.0084 0.9453 0.9781 NaN NaN NaN NaN
Fixed effects 0.6262 0.0515 0.5252 0.7272 NaN NaN NaN NaN
Anderson-Hsiao IV 1.2327 0.4782 0.2955 2.1699 NaN NaN NaN 1.0
Diff GMM (one-step) 0.7075 0.0842 0.5425 0.8725 0.2113 0.0 0.8913 91.0
Diff GMM (two-step) 0.6788 0.0891 0.5042 0.8534 0.2113 0.0 0.8660 91.0
Sys GMM (one-step, collapsed) 0.9025 0.0634 0.7781 1.0268 0.4621 0.0 0.9492 32.0
Sys GMM (two-step, collapsed) 0.9270 0.0785 0.7731 1.0809 0.4621 0.0 0.9944 32.0

Interpretation. Read as a single table, the ladder is unambiguous. The two naive estimators (0.9617 and 0.6262) define the bracket. Anderson-Hsiao (1.2327) is the only estimator outside it, with a confidence interval wider than everyone else’s put together. The two difference-GMM rows (0.7075 and 0.6788) sit in the bracket’s bottom sixth with 91 instruments each — formally admissible, substantively suspect. The two system-GMM rows (0.9025 and 0.9270) sit in the upper half with 32 collapsed instruments, the cleanest AR(2) values in the table (0.9492 and 0.9944), and Hansen p-values in the comfortable middle (0.4621). The one-step versus two-step movements (0.7075 to 0.6788; 0.9025 to 0.9270) are visible but small — a useful sanity check that the weighting scheme refines rather than drives the answer.

The forest plot turns that table into the tutorial’s closing image: every estimator on one axis, each with its 95 percent confidence interval, against the shaded OLS—FE bracket.

order = ["Pooled OLS", "Fixed effects", "Anderson-Hsiao IV",
"Diff GMM (one-step)", "Diff GMM (two-step)",
"Sys GMM (one-step, collapsed)", "Sys GMM (two-step, collapsed)"]
colors = {"Pooled OLS": GRAY, "Fixed effects": GRAY,
"Anderson-Hsiao IV": STEEL_BLUE,
"Diff GMM (one-step)": WARM_ORANGE,
"Diff GMM (two-step)": WARM_ORANGE,
"Sys GMM (one-step, collapsed)": TEAL,
"Sys GMM (two-step, collapsed)": TEAL}
plot_df = summary.set_index("estimator").loc[order].reset_index()
fig, ax = plt.subplots(figsize=(9.5, 6))
fig.patch.set_linewidth(0)
ax.axvspan(rho_fe, rho_ols, color=GRID_LINE, alpha=0.55, zorder=0)
ax.text((rho_fe + rho_ols) / 2, -0.75, "OLS-FE credible bracket",
color=LIGHT_TEXT, fontsize=10, ha="center", style="italic")
ax.axvline(1.0, color=GRAY, lw=1.2, ls="--", alpha=0.8)
ypos = np.arange(len(plot_df))[::-1]
for y, (_, r) in zip(ypos, plot_df.iterrows()):
ax.errorbar(r["rho1"], y, xerr=1.96 * r["se"], fmt="o",
color=colors[r["estimator"]], ms=9, capsize=4, lw=2.2,
capthick=2.2, zorder=3)
ax.text(r["rho1"], y + 0.28, f"{r['rho1']:.3f}", color=WHITE_TEXT,
fontsize=10, ha="center")
ax.set_yticks(ypos)
ax.set_yticklabels(plot_df["estimator"])
ax.set_xlabel(r"Employment persistence $\hat{\rho}$ (L1.n) with 95% CI")
ax.set_ylim(-1.1, len(plot_df) - 0.4)
ax.set_title("Seven estimators, one parameter:\n"
"system GMM lands inside the bracket with tight precision",
fontsize=13)
plt.savefig(f"{SLUG}_estimates_forest.png", dpi=300, bbox_inches="tight",
facecolor=DARK_NAVY, edgecolor=DARK_NAVY, pad_inches=0)
plt.show()

Interpretation. The forest plot is the tutorial in one image. The grey markers (OLS at 0.962, FE at 0.626) define the shaded credible band. Anderson-Hsiao’s enormous blue whisker — confidence interval width 1.87 — straddles everything, including the dashed unit-root line. The orange difference-GMM pair sits at the bottom of the band, hugging the FE bound exactly as Bond’s diagnostic predicts under weak instruments. And the teal system-GMM pair (0.902 and 0.927) lands in the upper half of the band with usable precision. Notice what separates the winner from the losers: nothing on any single printed line of output. Difference GMM’s table looks as healthy as system GMM’s. Only the bracket logic, the weak-instrument reasoning, the proliferation experiment, and the replication check — the workflow, not a p-value — identify 0.927 as the defensible answer.

13. Discussion

Return to the question the Overview posed: how persistent is firm employment? The defended answer is $\hat{\rho} = 0.927$ (SE 0.079): roughly 93 percent of an employment shock carries into the following year, an echo half-life of about nine years. The economic reading is that firm-level employment in this panel behaves almost like a random walk around firm-specific levels — adjustment frictions (hiring, firing, training costs) are large, and year dummies aside, a firm’s best predictor next year is overwhelmingly itself this year. And the methodological reading is just as important: the estimator choice moved the implied half-life from 1.5 years (FE) through 9 years (system GMM) to 18 years (OLS). Anyone consuming a dynamic-panel coefficient — referee, policymaker, manager — should ask which estimator produced it and where it sits in the OLS-FE bracket before believing it.

So what would a practitioner do with this? Concretely: a policymaker evaluating a temporary employment subsidy should expect its effects to compound and linger — with $\rho \approx 0.93$, a one-year boost to a firm’s workforce is still two-thirds visible five years later, dramatically changing any cost-benefit horizon relative to the FE story, under which the boost would have lost three-quarters of its effect within three years ($0.626^3 \approx 0.25$). Conversely, an analyst who naively ran fixed effects on a short firm panel would conclude that labor-market interventions evaporate quickly, and would be wrong by a factor of six in half-life terms.

For your own work, the tutorial compresses into a checklist:

Run pooled OLS and fixed effects first and record the bracket $[\hat{\rho}_{FE}, \hat{\rho}_{OLS}]$. They are not throwaway regressions; they are your measuring stick (here: [0.626, 0.962]).
Treat a difference-GMM estimate near the FE bound as a weak-instrument symptom (ours: 0.679, within one SE of 0.626), especially when the series is persistent. Passing Hansen and AR(2) does not clear it.
Prefer system GMM when persistence is high — but say out loud that you are buying identification with the mean-stationarity assumption, and check that the estimate lands inside the bracket (ours: 0.927).
Read AR(1) as “must reject,” AR(2) as “must not reject.” AR(2) is the test that protects your instruments (ours: p = 0.994).
Read Hansen two-tailed: below 0.05 is rejection, but drifting toward 1 as instruments accumulate is overfitting, not validity (ours: 0.462 with 32 instruments).
Collapse instruments and report the count relative to the number of groups (ours: 32 versus 140 firms; the uncollapsed alternative hit 113). Accept the larger SE as the price of honesty.
Replicate a published benchmark with your exact toolchain before trusting novel numbers (ours: digit-for-digit match to the pydynpd README).

The remaining caveats are the agenda for the next post — and the summary below collects what a week-later reader should still remember.

14. Summary and next steps

Takeaways:

Method. A lagged dependent variable plus a fixed effect defeats both workhorses in opposite directions: pooled OLS overstated persistence by loading $\alpha_i$ onto the lag ($\hat{\rho} = 0.962$), and the within estimator understated it through Nickell bias ($\hat{\rho} = 0.626$, with $T \approx 7$—$9$). Their disagreement — 0.336 — is not noise; it is a diagnostic bracket.
Method. Consistency is not enough: Anderson-Hsiao IV is consistent yet returned 1.233 with a CI of width 1.87, and difference GMM passed every printed test (Hansen p = 0.211, AR(2) p = 0.866) while hugging the biased FE bound at 0.679. The defensible estimate — system GMM’s 0.927 (SE 0.079), AR(2) p = 0.994, Hansen p = 0.462, 32 collapsed instruments — was identified by the workflow, not by any single statistic.
Data. The panel’s variance is lopsided: between-firm SD of log employment (1.339) is seven times the within-firm SD (0.195), and each lag burned data (1,031 rows down to 891, then 751, then 611 across specifications). Short-and-wide panels are simultaneously where dynamic GMM is needed and where it is data-hungriest.
Limitation. The headline CI [0.773, 1.081] includes the unit root, so “employment is stationary” is not a defensible claim — only the point estimate and its lower bound are. The model also imposes one common $\rho$ on all 140 firms, assumes mean stationarity (untestable directly), and describes 1970s—80s UK manufacturing — a methods showcase, not a current estimate of employment dynamics.
Next step. Natural extensions: estimate heterogeneous persistence (for example, splitting firms by size), probe mean stationarity by comparing difference- and system-GMM estimates across subsamples, or take the same workflow to a modern panel where the answer is unknown.

Limitations worth restating in one place: $\rho$ here is a descriptive/structural persistence parameter, not a causal effect; the wage and capital coefficients are conditional elasticities, not treatment effects; the long-run wage elasticity is mechanically fragile because $1 - \hat{\rho} \approx 0.073$; and all GMM tests have limited power with N = 140.

If you can explain to a colleague why difference GMM’s clean-looking table should not have been trusted — and what evidence finally separated 0.927 from 0.679 — this tutorial has done its job. The exercises below let you stress-test that understanding.

15. Exercises

Move the bracket. Re-estimate the pooled OLS and FE models dropping the lagged controls (w_lag1, k_lag1) from FORMULA_RHS. Does the bracket $[\hat{\rho}_{FE}, \hat{\rho}_{OLS}]$ shift? Does the system-GMM estimate (re-run with the corresponding SPEC_MAIN) stay inside it?
Stress the proliferation grid. Extend grid_specs with windows 2:4 and 2:7, collapsed and uncollapsed. Plot the new points onto Figure 3’s axes. Does the uncollapsed Hansen p-value continue its mechanical drift? Find the smallest uncollapsed window that Hansen rejects.
Predetermined wages. Our running specification treats w like n (instruments from $t-2$). Following the replication example’s gmm(w, 1:3), re-run system GMM treating wages as predetermined (instruments from $t-1$). Define in one sentence what predetermined means here, and report how $\hat{\rho}$, AR(2), and Hansen change.

16. References

AI Podcast: Dynamic Panel Data Models

Click play to load

0:00 0:00

Bouncing Back Better? Evaluating the Economic Impact of the Aceh Tsunami

Tue, 09 Jun 2026 00:00:00 +0000

Abstract

Localized natural disasters destroy capital and lives yet can attract reconstruction aid large enough to rebuild a region “better than before,” leaving the net long-run effect genuinely ambiguous. This tutorial asks whether the Indonesian province of Aceh — which lost roughly 130,000 people to the 2004 Indian Ocean tsunami but then received the largest developing-world reconstruction effort ever, about USD 7.7 billion committed and USD 7.0 billion spent — ended up on a higher or lower growth path a decade later, and how that effect can be credibly measured. It replicates Heger & Neumayer (2019) on synthetic calibrated data spanning a district panel of 125 Sumatran districts observed annually over 1999–2012 (1,750 rows, 10 flooded Aceh districts treated) and a finer panel of 276 Aceh sub-districts with satellite night-lights. Treating coastal inundation as a quasi-natural experiment, it estimates a dynamic four-period difference-in-differences with pyfixest, an event study with diff-diff, a continuous night-lights dose-response, a synthetic control with mlsynth, and Conley spatial-HAC standard errors validated by Moran’s I. Flooded districts lost 7.9% of output in 2005 (−0.0792, p < 0.01) but grew 6.3 percentage points per year faster during 2006–08 (+0.0628, p < 0.05), and synthetic control places flooded Aceh +18.3% above its no-tsunami counterfactual by 2012; the night-lights rebound (+0.0160, p < 0.001) concentrates in the worst-hit quintile, while a neighbour placebo finds nothing. The case shows that well-governed mega-reconstruction can leave a poor region on a permanently higher trajectory — provided inference accounts for spatially clustered treatment, which roughly doubles the recovery effect’s standard error (0.0146 to 0.0244) and downgrades it from a spuriously confident 1% to an honest 5% significance.

1. Overview

On 26 December 2004, a magnitude-9.1 earthquake off the coast of Sumatra sent a tsunami across the Indian Ocean. The Indonesian province of Aceh bore the worst of it: roughly 130,000 people died, the wave reached up to 9 km inland, and about a third of the coastline was flooded. Then something unusual happened. Aceh received the single largest reconstruction effort ever directed at a developing-world disaster — about USD 7.7 billion committed, USD 7.0 billion actually spent — under a well-coordinated agency with low corruption.

So here is a genuinely hard question: a decade later, was Aceh richer or poorer than it would have been without the tsunami? Catastrophe destroys capital and lives; massive, well-spent aid rebuilds — better than before, sometimes. Which force won? And — the part this tutorial really cares about — how could you ever measure that credibly, when you only get to observe the world where the tsunami did happen?

This post is a hands-on answer. We treat the tsunami as a natural experiment: the wave flooded some districts and spared others for reasons of coastal geography that have nothing to do with their economic prospects. Comparing the flooded “treated” districts to the un-flooded “control” districts — before and after 2004 — lets us isolate the disaster-plus-reconstruction effect. We will measure it four different ways, each answering a slightly sharper version of the question, and we will be honest about uncertainty when the treated places all sit in one corner of the map.

But what should we even be looking for? A disaster does not push an economy onto a single, predetermined track. Relative to the path it would have followed without the wave (the dotted counterfactual below), output could end up permanently lower, snap right back to trend, overshoot and then fall back, settle permanently higher, or be remade entirely. These are the archetypes our estimates will have to choose between.

Five archetypal trajectories a shocked economy can follow — from a permanently lower path to creative destruction. Which one did Aceh take? The four methods below answer that.

A note on the data (please read this). This tutorial is inspired by and based on the study by Heger & Neumayer (2019), but it runs on synthetic data created for teaching. The paper’s real inputs (World Bank GDP, satellite night-lights, tsunami inundation maps) are licensed or confidential. Our dataset is calibrated so that re-running the paper’s analyses reproduces its findings — the signs, the statistical significance, and the approximate magnitudes of the key coefficients. The direction and significance of most results match the paper closely; the magnitudes can differ slightly (we tabulate exactly how in Section 11). Use this to learn the methods, not to draw new conclusions about Aceh.

1.1 Learning objectives

By the end of this tutorial, you will be able to:

Frame a localized natural disaster as a quasi-natural experiment, and explain why a flooded-vs-not comparison can identify a causal effect under parallel trends.
Measure disaster exposure and economic activity from administrative and satellite data — district GDP, sub-district night-lights, and satellite inundation maps.
Estimate a dynamic, four-period difference-in-differences on district GDP growth with pyfixest, and read it as an event study with diff-diff.
Quantify a dose-response relationship at finer resolution using continuous flood intensity on sub-district night-lights.
Build a synthetic-control counterfactual for flooded Aceh with mlsynth and read its path and gap plots.
Defend your inference when treatment is geographically clustered, using Moran’s I and Conley spatial standard errors, and validate the result with placebo and heterogeneity checks.

1.2 Study design

graph LR
subgraph SETTING["The natural experiment"]
A("<b>2004 tsunami</b><br/>floods some<br/>Aceh districts")
B("<b>Treated</b><br/>10 flooded<br/>districts")
C("<b>Control</b><br/>non-flooded<br/>districts")
A --> B
A --> C
end
subgraph MEASURE["Two outcomes"]
D("<b>District GDP</b><br/>growth")
E("<b>Sub-district</b><br/>night-lights")
end
subgraph METHODS["Four causal tools"]
F("<b>Difference-in-<br/>differences</b><br/>pyfixest")
G("<b>Event study</b><br/>diff-diff")
H("<b>Synthetic<br/>control</b><br/>mlsynth")
I("<b>Conley spatial<br/>std. errors</b>")
end
B --> D
C --> D
B --> E
D --> F --> G
D --> H
F --> I
style SETTING fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style MEASURE fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style METHODS fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef gray fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class A,B orange
class C blue
class D,E,F,G gray
class H,I teal

Read the diagram left to right: the tsunami splits districts into treated and control; we observe two outcomes (district GDP and finer sub-district night-lights); and we deploy four causal tools — DiD and its event-study view, an independent synthetic control, and the spatial standard errors that keep our confidence honest. Each maps onto a section below.

1.3 Key concepts at a glance

The post leans on a small vocabulary repeatedly. Each concept below has three parts. The definition is always visible; the example and analogy sit behind clickable cards — open them when you need them, leave them closed for a quick scan. If a later section mentions “parallel trends” or “Conley standard errors” and the term feels slippery, this is the section to re-read.

1. Difference-in-Differences (DiD). Compare the change in the treated group to the change in the control group. The difference of those two differences is the causal estimate. It nets out anything permanent about a district and any trend shared by the whole country.

Example

Flooded districts’ mean growth went from 0.0567 (before) to 0.0671 (after) — a change of +0.0103. Controls went 0.0519 to 0.0497 — a change of −0.0022. The 2×2 DiD is $0.0103 - (-0.0022) = +0.0125$.

Analogy

Time two runners on parallel tracks. Both speed up when the gun fires (the national trend). Credit your coaching only with the extra burst of the runner you coached — the gap between the two changes.

2. Parallel trends. The identifying assumption of DiD: absent the tsunami, flooded and non-flooded districts would have grown by the same amount on average. Their levels may differ; their trends must match.

Example

We test it directly: the pre-tsunami (2003–04) DiD coefficient is +0.0172 and statistically insignificant (p = 0.28). No detectable divergence before treatment — the assumption survives its placebo.

Analogy

Two boats drifting on the same current. They sit at different points, but the current carries them in step. Only an engine — the treatment — should make one pull ahead.

3. ATT $E[Y(1) - Y(0) \mid D=1]$. The Average effect of the Treatment on the Treated. Here: the effect on the flooded districts, not on some randomly chosen district. DiD and synthetic control both target the ATT.

Example

The +18.3% synthetic-control gap is the ATT for flooded Aceh. It does not claim that flooding any district would raise its GDP 18% — only that these districts, given this reconstruction, ended up that much higher.

Analogy

The bonus speed measured on the car that actually got the coaching — not a promise about any car you might pick off the street.

4. Counterfactual. The output flooded Aceh would have had with no tsunami. It is never observed; it must be estimated — by the control group’s trend (DiD) or a weighted blend of donor districts (synthetic control).

Example

“Synthetic Aceh” is a weighted recipe of 76 Rest-of-Sumatra donor districts (top weights: JAMBI_D01 0.13, BABEL_D05 0.12) chosen to match flooded Aceh’s pre-2005 GDP path. After 2005 it is our stand-in for the no-tsunami Aceh.

Analogy

The parallel-universe Aceh where the wave never came. We cannot visit it, so we build the most convincing look-alike we can from places the wave did miss.

5. Dose-response. Bigger exposure should mean a bigger effect. Instead of an on/off treatment dummy, use continuous intensity — the share of a sub-district flooded — or intensity quintiles.

Example

Each unit of “share of population flooded” raises night-lights growth by +0.016/year during recovery (p < 0.001). And only the top intensity quintile shows a significant rebound — quintiles 1–4 are flat.

Analogy

Medicine dosage. A sip does little; the full dose moves the needle. If only the largest doses show an effect, the drug is real but the average hides where it acts.

6. Night-lights as an economic proxy. Satellite night-time brightness (DMSP-OLS “Digital Numbers”, 0–63) stands in for local economic activity where GDP is unavailable, after a log transform.

Example

In 2004 the flooded sub-districts averaged a luminosity of 5.79 versus 2.36 for non-flooded ones — they are the denser, more active coastal places, and their lights are what we track over time.

Analogy

Judging a city’s bustle from a night flight overhead. You cannot read the GDP accounts from 800 km up, but brighter usually means busier.

7. Conley spatial-HAC standard errors. Standard errors that allow a district’s errors to be correlated with nearby districts in the same year (spatial) and with itself over time (serial). They are larger — and more honest — than naive errors when the treated units cluster in space.

Example

The recovery effect’s standard error roughly doubles, from 0.0146 (naive) to 0.0244 (Conley-HAC). That turns a t of 4.3 into a t of 2.57 — the point estimate (+0.0628) never moves, but it is significant at 5%, not 1%.

Analogy

Counting a milling crowd. Rows of seats suggest many independent heads, but if everyone keeps shuffling between seats you have far fewer truly independent observations than it looks.

2. Setup and the three star libraries

Three specialist packages do the heavy lifting, and each gets a one-line introduction the first time we use it:

pyfixest runs fixed-effects regressions with a fast, Stata-flavored formula syntax. Everything left of the | is estimated; everything right of it is absorbed as fixed effects, so we never build dummy columns by hand.
diff-diff is a small package built to teach difference-in-differences: it returns the 2×2 estimate and the event-study path in one or two lines each.
mlsynth implements modern synthetic-control estimators; we use VanillaSC, the classic Abadie–Diamond–Hainmueller method.

# In Colab, install the three estimation libraries first:
# !pip install pyfixest==0.50.1 diff-diff==3.5.2 "mlsynth @ git+https://github.com/jgreathouse9/mlsynth.git"
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pyfixest as pf
import diff_diff as dd
from mlsynth import VanillaSC
np.random.seed(42) # reproducibility
# Site dark-theme palette for figures
STEEL_BLUE, WARM_ORANGE, TEAL = "#6a9bcc", "#d97757", "#00d4c8"
DARK_NAVY, GRID_LINE, LIGHT_TEXT = "#0f1729", "#1f2b5e", "#c8d0e0"

Two small design helpers encode the paper’s difference-in-differences structure. The post period is split into event-time windows — pre (2003–04), tsunami (2005), recovery (2006–08), and post-recovery (2009–12) — all measured against the omitted 2000–02 baseline. The function below turns a treatment column into the four interaction terms those windows need.

# The four event-time windows (the 2000-02 baseline is the omitted reference)
PERIOD_TO_TERM = {"pre": "D_pre", "tsunami": "D_2005",
"recovery": "D_recov", "postrec": "D_post"}
DID_TERMS = ["D_pre", "D_2005", "D_recov", "D_post"]
def make_did_terms(df, treat_col):
"""Build treatment x period interactions: D_pre, D_2005, D_recov, D_post."""
out = df.copy()
treat = out[treat_col].astype(float)
for period, term in PERIOD_TO_TERM.items():
out[term] = treat * (out["period"] == period).astype(float)
return out

3. The data: measuring a disaster at two geographic levels

Evaluating a localized disaster forces a measurement problem to the surface. National GDP would barely flinch at a shock to one province — so we need sub-national data, and we need it at two grains. We load both panels straight from the post’s data folder on GitHub, so the code runs unchanged in Colab.

BASE = ("https://raw.githubusercontent.com/cmg777/starter-academic-v501/"
"master/content/tutorials/python_did_sc_tsunami/data/")
district = pd.read_csv(BASE + "aceh_tsunami_district_panel.csv")
subdistrict = pd.read_csv(BASE + "aceh_tsunami_subdistrict_panel.csv")
print("district panel :", district.shape)
print("subdistrict panel :", subdistrict.shape)
print(district.groupby(["region_group", "flooded"]).size().unstack("flooded", fill_value=0))

district panel : (1750, 30)
subdistrict panel : (3864, 19)
flooded 0 1
region_group
Aceh 13 10
North Sumatra 24 2
Rest of Sumatra 76 0

The district panel is 125 districts observed annually over 1999–2012 (1,750 rows); the sub-district panel is 276 Aceh sub-districts (kecamatans) over the same years. The treatment group is small and concentrated: 10 flooded Aceh districts against 13 non-flooded Aceh districts plus 76 Rest-of-Sumatra controls. (North Sumatra’s two flooded islands are held back for a robustness check, because they were also hit by a separate earthquake in March 2005.) That smallness — only 10 treated units — is the recurring source of statistical caution in this case study.

3.1 The first outcome — district GDP growth

The main outcome (gdp_growth) is the annual growth rate of real district GDP, measured in the spirit of the World Bank’s INDO-DAPOER database, which draws on Indonesia’s large annual socio-economic survey (SUSENAS). Two construction details from the paper matter. First, oil and gas are excluded: that sector is volatile and concentrated in a few non-treated districts, so leaving it in would add noise unrelated to the tsunami. Second, GDP is in constant prices so we measure real output, not inflation.

print(district[["gdp_growth", "gdp_pc_growth", "gdp_const_usd_m"]].describe().round(3).loc[
["count", "mean", "std", "min", "max"]])

 gdp_growth gdp_pc_growth gdp_const_usd_m
count 1621.00 1621.00 1750.00
mean 0.05 0.04 671.18
std 0.07 0.07 593.99
min -0.17 -0.20 33.09
max 0.29 0.30 3748.41

District growth averages about 5.2% a year with a wide spread (standard deviation 6.6 percentage points). Notice gdp_growth has 1,621 values, not 1,750: growth is undefined in 1999 (no prior year to difference against) and is missing for one district (Subulussalam) over 2003–06 due to an administrative boundary change. Every estimator below simply drops those rows — a small but honest detail that keeps the sample sizes matching the paper exactly.

3.2 The second outcome — sub-district night-lights

GDP at the district level is too coarse to capture how intensely a place was hit. So the paper drops to the finer sub-district grain and switches to a satellite proxy: night-time luminosity from the DMSP-OLS program. Each pixel records a “Digital Number” from 0 (dark) to 63 (saturated bright). To turn pixel brightness into a sub-district economic measure, the paper sums the lights across a sub-district’s pixels and takes a logarithm:

$$\text{NL}_{ct} = \log\left( \sum_{n=1}^{N} \left( \text{DN}_{nct} + 0.001 \right) \right)$$

In words: a sub-district $c$’s log-luminosity in year $t$ is the log of the total brightness summed over its $N$ pixels, with a tiny $0.001$ added so that pixels reading exactly zero do not break the logarithm. Summing (rather than averaging) keeps the measure comparable to GDP, which is also a total; the log tames the heavy right-skew of brightness. Our outcome nl_growth is the annual change in this log measure — a luminosity growth rate that lines up conceptually with the GDP growth rate.

3.3 Identifying the treated areas — where the wave actually reached

The credibility of the whole exercise rests on how “flooded” is defined. The paper does not let economics decide it. Treatment is read off satellite inundation maps produced 1–5 days after the tsunami by remote-sensing agencies (Germany’s DLR/ZKI and the Dartmouth Flood Observatory), which compared the coastline before and after to flag pixels the water reached. Whether the wave penetrated a given stretch of coast was governed by elevation, vegetation, and offshore depth — geographic happenstance, plausibly unrelated to a district’s economic prospects. That is exactly what makes the flooding a credible natural experiment.

The data encodes exposure three ways, from coarse to fine:

print("Binary treatment (district level):")
print(district.groupby("flooded")["district_id"].nunique())
print("\nContinuous + quintile intensity (sub-district level), among flooded units:")
print(subdistrict.loc[subdistrict.flooded == 1, ["share_pop_flooded", "share_area_flooded"]].describe().round(3).loc[["mean", "min", "max"]])

Binary treatment (district level):
flooded
0 113
1 12
Name: district_id, dtype: int64
Continuous + quintile intensity (sub-district level), among flooded units:
share_pop_flooded share_area_flooded
mean 0.190 0.012
min 0.001 0.000
max 0.620 0.105

flooded is the simple on/off dummy used for the district DiD. For the finer night-lights analysis we also have share_pop_flooded and share_area_flooded — the fraction of a sub-district’s population (or land area) that the inundation maps marked as flooded — plus a flood_intensity_quintile ranking. Notice share_area_flooded has a tiny mean (about 1.2%): land area includes a lot of unpopulated hinterland, so the share of area flooded is small even where damage was severe. That tiny scale will make its regression coefficient look enormous later — same story, different units.

3.4 A transparent word on the synthetic data

Before we model anything: the panels above are simulated. The data-generating process was tuned so that a fixed-effects DiD recovers, column by column, coefficients close to the paper’s reported values (within about 0.005 on the headline cells), with the same signs and significance stars. Spatial and serial shocks were injected so the standard errors behave like the paper’s without moving the point estimates. We will hold ourselves accountable for this in Section 11, where a table lines our numbers up against the paper’s. With the measurement settled, let us look at the data before modeling it.

4. Exploratory analysis: the space-time dynamics

Good causal work looks at the data before it regresses it. Three views build the intuition the models will formalize. First, a handful of individual districts over time — three badly-hit ones (Banda Aceh, Aceh Besar, Aceh Jaya) against two highland controls — with GDP indexed so every district starts at 100 in 2004.

def indexed(name):
s = district[district.district_name == name].set_index("year")["gdp_const_usd_m"]
return s / s.loc[2004] * 100
fig, ax = plt.subplots(figsize=(9, 5.2))
for name in ["Banda Aceh", "Aceh Besar", "Aceh Jaya"]:
ax.plot(indexed(name), color=WARM_ORANGE, lw=2.2, label=f"{name} (flooded)")
for name in ["Aceh Tengah", "Bener Meriah"]:
ax.plot(indexed(name), "--", color=STEEL_BLUE, lw=2, label=f"{name} (control)")
ax.axvline(2004.5, color=LIGHT_TEXT, ls=":")
ax.set(xlabel="Year", ylabel="Real GDP (2004 = 100)")
ax.legend(); plt.show()

Three flooded districts (orange) versus two highland controls (blue), each indexed to 100 in 2004.

The flooded districts sit on the same path as the controls through 2004, buckle in 2005, and then climb steeply — Banda Aceh, the provincial capital, ends near 260 (a 2.6× increase over its 2004 level). The control districts grow too, but far more gently. The eye already sees a disaster followed by an over-shooting recovery; the rest of the post is about measuring it and trusting the measurement.

Single districts are noisy, though. The next view summarizes the distribution of growth in each group across the event-time periods.

samp = district[district.region_group != "North Sumatra"].dropna(subset=["gdp_growth"]).copy()
samp["group"] = np.where(samp.flooded == 1, "Treated (flooded)", "Control")
# (full grouped-boxplot styling is in script.py)

Distribution of district GDP growth, treated (orange) vs control (blue), in each event-time period.

The boxes make the dynamics unmistakable. In 2000–04 the treated and control boxes overlap almost perfectly — similar centers, similar spread. In 2005 the treated box drops bodily below zero (a median contraction) while the control box stays put. In 2006–08 the treated box jumps above the control box. The disaster and the rebound are both visible as shifts in the whole distribution, not just a couple of outliers.

Finally, the single figure that motivates difference-in-differences: the group means over time.

means = (samp.groupby(["year", "flooded"])["gdp_growth"].mean()
.unstack("flooded").rename(columns={0: "Control", 1: "Treated"}))
fig, ax = plt.subplots(figsize=(9, 5.2))
ax.plot(means["Control"], "--o", color=STEEL_BLUE, label="Control")
ax.plot(means["Treated"], "-o", color=WARM_ORANGE, label="Treated (flooded)")
ax.axvline(2004.5, color=LIGHT_TEXT, ls=":"); ax.legend(); plt.show()

Mean annual GDP growth, treated vs control. Parallel before the tsunami, sharply divergent after.

This is the picture difference-in-differences was invented for. Before 2005 the two lines move in lockstep — the visual signature of parallel trends. At the tsunami the treated line plunges to about −0.027 while the control line barely moves. Then the treated line over-shoots, peaking near +0.124 in 2007 before settling back toward the control. The eye is convinced something happened; now we quantify it and, crucially, attach a margin of error.

5. Difference-in-differences on district GDP growth

5.1 The intuition: a 2×2 difference of differences

Start with the simplest possible version. Split time into “before” (≤ 2004) and “after” (≥ 2005), compute the mean growth in each of the four treated/control × before/after cells, and form the difference of the two differences:

$$\widehat{\text{DiD}} = \big( \bar{g}_{\text{treated, after}} - \bar{g}_{\text{treated, before}} \big) - \big( \bar{g}_{\text{control, after}} - \bar{g}_{\text{control, before}} \big)$$

In words: take how much the treated group’s average growth changed across the break, subtract how much the control group’s changed, and what remains is the part attributable to the tsunami — because the control change captures whatever was happening nationwide anyway.

sample = make_did_terms(district[district.region_group != "North Sumatra"], "flooded").dropna(subset=["gdp_growth"])
cell = sample.groupby(["flooded", "post"])["gdp_growth"].mean().unstack("post")
cell.columns, cell.index = ["Before (<=2004)", "After (>=2005)"], ["Control", "Treated"]
cell["change"] = cell["After (>=2005)"] - cell["Before (<=2004)"]
print(cell.round(4))
res = dd.DifferenceInDifferences(cluster="district_id").fit(
sample, outcome="gdp_growth", treatment="flooded", time="post")
print(f"\nDiD ATT = {res.att:+.4f} (SE {res.se:.4f}, p = {res.p_value:.3f})")

 Before (<=2004) After (>=2005) change
Control 0.0519 0.0497 -0.0022
Treated 0.0567 0.0671 0.0103
DiD ATT = +0.0125 (SE 0.0142, p = 0.379)

The hand calculation gives $0.0103 - (-0.0022) = +0.0125$, and diff-diff confirms it with a standard error: +0.0125, but statistically insignificant (p = 0.38). Before you conclude “no effect,” look closer. This single “after” window blends two opposite phases — the 2005 destruction and the 2006–08 boom — into one average, and they nearly cancel. The pooled estimate is not wrong; it is just uninformative. The fix is to let the effect vary over time.

5.2 The dynamic DiD (the paper’s headline)

The paper’s central specification keeps the same logic but splits the post period into the four event-time windows, each entering as a treatment-times-period interaction relative to the 2000–02 baseline:

$$\Delta Y_{it} = \beta_1 D_i \mathbf{1}[t \in \text{pre}] + \beta_2 D_i \mathbf{1}[t = 2005] + \beta_3 D_i \mathbf{1}[t \in \text{recovery}] + \beta_4 D_i \mathbf{1}[t \in \text{post}] + \alpha_i + \gamma_t + \varepsilon_{it}$$

Here $\Delta Y_{it}$ is district $i$’s GDP growth in year $t$; $D_i = 1$ for flooded districts; $\mathbf{1}[\cdot]$ is an indicator that is 1 when the year falls in that window; $\alpha_i$ is a district fixed effect (absorbing anything permanent about a district) and $\gamma_t$ is a year fixed effect (absorbing common national shocks). The four $\beta$’s are the story: $\beta_2$ should be the negative 2005 shock, $\beta_3$ the positive reconstruction boom, while $\beta_1$ and $\beta_4$ should be near zero. These map exactly onto the code variables D_pre, D_2005, D_recov, D_post. In pyfixest, the part after the | lists the fixed effects to absorb:

m = pf.feols("gdp_growth ~ D_pre + D_2005 + D_recov + D_post | district_id + year",
data=make_did_terms(district[district.region_group != "North Sumatra"], "flooded"),
vcov={"CRV1": "district_id"})
m.coef().round(4)

D_pre 0.0172
D_2005 -0.0792
D_recov 0.0628
D_post 0.0114

Reading these against the paper’s three control pools (the full table from script.py) gives the headline result:

Coefficient	(1) Sumatra controls	(2) Rest of Sumatra	(3) Aceh non-flooded
Pre-tsunami (2003-04)	+0.0172 (0.0159)	+0.0176 (0.0162)	+0.0154 (0.0187)
Tsunami (2005)	−0.0792* (0.0240)**	−0.0782* (0.0247)**	−0.0841* (0.0281)**
Recovery (2006-08)	+0.0628 (0.0244)**	+0.0682* (0.0247)**	+0.0310 (0.0281)
Post-recovery (2009-12)	+0.0114 (0.0146)	+0.0132 (0.0147)	+0.0008 (0.0204)
Observations	1,283	1,118	295

Conley spatial-HAC standard errors in parentheses; *** p<.01, ** p<.05, * p<.10.

Three things to take away. First, the pre-tsunami coefficient is small and insignificant (+0.0172) — the parallel-trends assumption passes its formal placebo test, so the comparison is credible. Second, the 2005 coefficient is −0.0792 (p < 0.01): flooded districts grew almost 8 percentage points slower the year the wave hit. Third, the recovery coefficient is +0.0628 (p < 0.05): over 2006–08 they grew 6.3 points per year faster than controls — and three years of that premium (≈ +0.19 cumulatively) more than erases the one-year loss. The post-recovery coefficient is a near-zero +0.0114, meaning the gain neither evaporated nor kept compounding: Aceh settled onto a permanently higher path. This is the paper’s signature result — “sustainable recovery beyond the counterfactual trend.” Notice column 3, which compares flooded Aceh to its own non-flooded neighbors, halves the recovery coefficient to +0.0310 (insignificant): reconstruction money spilled across district lines, shrinking the within-Aceh contrast.

The estimand here is the ATT — the effect on the flooded districts — and identification rests on parallel trends, not randomization. This is an observational study; the placebo and spatial-error checks below are what earn it credibility.

5.3 The event study: seeing the whole path

The dynamic DiD has four coefficients; an event study plots them (plus the pinned baseline) so the pre-trend and the recovery path are visible at a glance. diff-diff produces it directly:

mp = dd.MultiPeriodDiD(cluster="district_id").fit(
sample, outcome="gdp_growth", treatment="flooded", time="period",
reference_period="baseline", absorb=["district_id"])
for p in ["pre", "tsunami", "recovery", "postrec"]:
e = mp.period_effects[p]
print(f"{p:9s} effect={e.effect:+.4f} se={e.se:.4f} p={e.p_value:.3f}")

pre effect=+0.0172 se=0.0160 p=0.283
tsunami effect=-0.0792 se=0.0260 p=0.002
recovery effect=+0.0628 se=0.0247 p=0.011
postrec effect=+0.0114 se=0.0149 p=0.444

Each point is the treated-minus-control effect in that period, relative to the 2000–02 baseline; bars are 95% confidence intervals.

The figure tells the entire story in one arc. The baseline and pre-tsunami points sit on zero (parallel trends — the identifying assumption is visibly satisfied). The 2005 point collapses to −0.079 with a confidence interval well below zero. The recovery point rebounds to +0.063, also significantly positive. And the post-recovery point drifts back toward zero but stays positive — the higher level persists. A single “after” dummy (Section 5.1) averaged the deep red 2005 point with the high green recovery point and got a muted, insignificant number; the event study shows why that average was misleading.

5.4 Did people just leave? The per-capita check

A worry: maybe “growth” per district rose only because the population fell (the tragic arithmetic of 130,000 deaths and displacement). If GDP and population dropped together, GDP per capita need not have moved. Re-running the same DiD on gdp_pc_growth addresses it:

Coefficient	(1) Sumatra controls
Tsunami (2005)	+0.0192 (0.0239)
Recovery (2006-08)	+0.0827*** (0.0261)

In per-capita terms there is no significant 2005 loss (+0.0192) — output and population fell together that year — but the recovery gain is even larger and highly significant (+0.0827, p < 0.01): fewer people then shared a rebuilt, better-capitalized economy. The effect is not a denominator artifact; it survives, indeed strengthens, when we divide by population.

6. Night-lights dose-response: how intensity matters

District GDP answers “did flooded districts grow faster?” Night-lights, available for the much finer sub-districts, can answer a sharper question: did the places hit harder rebound more? First, a quick descriptive — how bright were flooded vs non-flooded sub-districts before the tsunami?

snap = subdistrict[subdistrict.year == 2004]
print(snap.groupby("flooded")["avg_luminosity"].agg(["count", "mean", "std", "max"]).round(2))

 count mean std max
flooded
0 208 2.36 4.41 36.0
1 68 5.79 8.31 39.0

The 68 flooded sub-districts averaged a 2004 luminosity of 5.79 versus 2.36 for the 208 non-flooded ones — about 2.5× brighter, because flooded places are the denser, more economically active coastal strips. Now the dose-response: instead of the on/off flooded dummy, we interact the continuous flood intensity with the event-time periods.

def nl_fit(treat):
df = make_did_terms(subdistrict, treat)
return pf.feols("nl_growth ~ D_pre + D_2005 + D_recov + D_post | kecamatan_id + year",
data=df, vcov={"CRV1": "kecamatan_id"})
print(nl_fit("share_pop_flooded").coef().round(4))

D_pre 0.0052
D_2005 -0.0073
D_recov 0.0160
D_post 0.0019

Coefficient	Share of population flooded	Share of area flooded
Pre-tsunami (2003-04)	+0.0052 (0.0034)	+0.565 (0.358)
Tsunami (2005)	−0.0073** (0.0035)	−0.727* (0.381)
Recovery (2006-08)	+0.0160* (0.0022)**	+1.660* (0.246)**
Post-recovery (2009-12)	+0.0019 (0.0024)	+0.270 (0.250)
N	3,444	3,444

The recovery coefficient on “share of population flooded” is +0.0160 (p < 0.001): each additional unit of population-share flooded buys that much extra annual luminosity growth during reconstruction. The “share of area” column tells the same story with a coefficient about 100× larger (+1.660) — purely because, as we saw in Section 3.3, the share of area flooded is a tiny number, so a one-unit move is enormous. Same effect, different yardstick. The pre-period coefficients are small and the 2005 dip is weak-to-modest, mirroring the district results at finer resolution.

The dose-response sharpens further if we ask where the effect lives. Splitting flood intensity into quintiles and interacting each with the post period:

Left: period coefficients for the continuous dose. Right: effect by intensity quintile — only the worst-hit fifth (Q5) rebounds significantly.

Quintile	Q1	Q2	Q3	Q4	Q5 (worst-hit)
Effect (share of population)	+0.0010	+0.0010	+0.0009	+0.0008	+0.0018**

Only the top quintile — the most heavily flooded fifth of sub-districts — shows a statistically significant rebound (+0.0018, p ≈ 0.02); quintiles 1 through 4 are flat and indistinguishable from zero. The average effect is not spread evenly; it is concentrated exactly where the damage, and therefore the reconstruction spending, was greatest. That is a substantive lesson about disaster aid as much as a statistical one.

7. Synthetic control: building a counterfactual Aceh

Difference-in-differences leans on the control group’s trend as the counterfactual. The synthetic control method builds a more bespoke one: a weighted blend of donor districts chosen so that the blend tracks flooded Aceh’s pre-tsunami path almost exactly. Formally, it picks non-negative weights $w$ that sum to one to minimize the pre-treatment mismatch:

$$w^{\ast} = \arg\min_{w}\ \left( X_1 - X_0 w \right)^{\top} V \left( X_1 - X_0 w \right) \quad \text{subject to} \quad w_j \geq 0, \quad \sum_j w_j = 1$$

where $X_1$ holds treated Aceh’s pre-2005 outcomes and $X_0$ the donors’ (one column per donor). After 2005 the fitted “synthetic Aceh” is left to run free; the gap between actual and synthetic Aceh is the estimated effect. Before fitting a model, the raw group averages already hint at the answer:

Figure 2 of the paper, reproduced: flooded Aceh (orange) ends well above non-flooded Aceh (blue) and the rest of Sumatra (teal).

By 2012 the flooded-Aceh index reaches 177 (2004 = 100), versus 162 for non-flooded Aceh and 142 for the rest of Sumatra. Now the formal version. mlsynth wants a long panel with one treated unit and a pool of donors, so we collapse the 10 flooded Aceh districts into a single average and pair them with the 76 Rest-of-Sumatra donor districts:

treated = (district[(district.flooded == 1) & (district.region_group == "Aceh")]
.groupby("year", as_index=False)["gdp_const_usd_m"].mean()
.assign(unitid="Aceh (flooded)").rename(columns={"year": "time", "gdp_const_usd_m": "outcome"}))
donors = (district[district.region_group == "Rest of Sumatra"][["district_id", "year", "gdp_const_usd_m"]]
.rename(columns={"district_id": "unitid", "year": "time", "gdp_const_usd_m": "outcome"}))
panel = pd.concat([treated[["unitid", "time", "outcome"]], donors], ignore_index=True)
panel["treat"] = ((panel.unitid == "Aceh (flooded)") & (panel.time >= 2005)).astype(int)
out = VanillaSC({"df": panel, "outcome": "outcome", "treat": "treat",
"unitid": "unitid", "time": "time", "display_graphs": False}).fit().model_dump()
print(f"pre-RMSE = {out['fit_diagnostics']['rmse_pre']:.3f}")
print(f"ATT = +{out['effects']['att']:.1f} GDP units (+{out['effects']['att_percent']:.1f}%)")

pre-RMSE = 0.485
ATT = +32.9 GDP units (+18.3%)

Figure 3 reproduced: synthetic Aceh tracks the treated path before 2005 (pre-RMSE 0.485), then the actual line pulls clearly above it.

The pre-treatment fit is excellent: a root-mean-squared prediction error of 0.485 against GDP levels near 200 means synthetic Aceh shadows the real thing almost perfectly before 2005 — which is what licenses us to trust it as a counterfactual afterward. After the tsunami the actual line pulls away, ending +18.3% above its synthetic twin (370.9 vs 295.0 by 2012). The gap plot isolates that divergence:

The estimated effect over time: indistinguishable from zero before 2005, then steadily positive.

A synthetic control is only as credible as its donor recipe — if one donor carried all the weight, the counterfactual would be fragile. Here the weight is spread:

The six largest donor weights. No single district dominates, which makes the counterfactual robust.

The top six donors — districts in Jambi, Bangka-Belitung, the Riau Islands, and Bengkulu — together carry about 62% of the weight, and the largest single weight is only 0.13. A counterfactual assembled from many modest contributors is far harder to dismiss than one resting on a single look-alike. Two very different methods — difference-in-differences and synthetic control — now agree: flooded Aceh ended up materially above where it was heading.

8. Spatial standard errors: honest inference for a clustered treatment

Every result so far came with a standard error, and those numbers were not the defaults. Here is why they cannot be. All 10 treated districts sit in one corner of Sumatra:

Every flooded (treated) district, in orange, sits in the far north-west. Their growth shocks are unlikely to be independent.

When the treated units are packed together, their year-to-year shocks are not independent draws — a good monsoon, a regional price swing, or the reconstruction boom itself hits them together. Tobler’s first law of geography puts it plainly: near things are more related than distant things. The default (“naive”) standard error assumes every observation is independent, so it counts more truly independent information than the data really contain, and reports standard errors that are too small. The first step is to check whether the problem is real, using Moran’s I — the spatial analogue of a correlation coefficient — on the regression residuals:

# residualize growth on flooded + year, then test whether the leftover is spatially clustered
# (full Moran's I + permutation code is in script.py)
print("Pooled within-year Moran's I = +0.065 (permutation p = 0.003)")

Pooled within-year Moran's I = +0.065 (permutation p = 0.003)

A Moran’s I of +0.065 with a permutation p-value of 0.003 says the residual growth of nearby districts is significantly positively correlated within a year — the independence assumption behind naive errors is violated, and we must do something about it. The fix is a Conley spatial-HAC standard error: a single “sandwich” estimator that counts two extra kinds of error correlation — serial (a district correlated with itself over time) and spatial (different districts within 100 km in the same year), with the spatial weight fading linearly to zero at the cutoff. The point estimates never change; only the standard errors do. Running the same DiD with four different standard errors side by side:

Coefficient	Estimate	Naive	Clustered	Conley	Conley-HAC	t(HAC)
Pre-tsunami	+0.0172	0.0144	0.0159	0.0144	0.0159	+1.08
Tsunami (2005)	−0.0792	0.0236	0.0258	0.0216	0.0240	−3.30
Recovery (2006-08)	+0.0628	0.0146	0.0244	0.0145	0.0244	+2.57
Post-recovery	+0.0114	0.0109	0.0148	0.0106	0.0146	+0.78

Look at the recovery row. The estimate is +0.0628 in every column — the point estimate is rock-solid. But its standard error climbs from 0.0146 (naive) to 0.0244 (Conley-HAC), a 1.68× inflation driven mostly by the serial correlation of the multi-year recovery window. That difference is decisive: under the naive error the recovery effect would have a t-statistic above 4 and look significant at the 1% level (***); under the honest Conley-HAC error its t is 2.57, significant at 5% (**). The point estimate never moved — only our honesty about its uncertainty did. A careless analyst would have overstated the confidence threefold.

How far should the spatial cutoff reach? Too short and you miss real correlation; too long and you dilute the kernel with distant, weakly-related pairs. Sweeping the cutoff shows the standard error is stable across the 25–100 km range the paper uses, then declines as far-flung pairs water it down:

The recovery effect’s standard error is flat through ~100 km (the paper’s choice), then drifts down as distant pairs dilute the spatial kernel.

(The full Conley sandwich — within-transformation, the two error “meats,” and the negative-variance clamp — lives in script.py as conley_did_estimate; it reproduces pyfixest’s point estimates to four decimals while adding the spatial standard errors pyfixest cannot compute.)

9. Robustness: placebo and heterogeneity

Two final checks decide whether to believe the headline. The first is a placebo: if our design is sound, then districts that merely neighbor a flooded district — but were not themselves flooded — should show no effect. We drop the truly flooded districts and pretend their neighbors were treated:

nonflooded = district[district.flooded == 0]
# re-run the dynamic DiD with `neighbour_of_flooded` as the fake treatment (full code in script.py)

Check	2005	Recovery (2006-08)	N
Placebo (neighbours of flooded)	+0.0025 (ns)	+0.0064 (ns)	1,465
City (Kota) districts	−0.0424 (ns)	+0.1226***	295
Rural (Kabupaten) districts	−0.0883***	+0.0479*	988

The placebo finds nothing — every coefficient is small and insignificant (2005 +0.0025, recovery +0.0064). That is exactly what we want: the result is not some artifact of generalized regional spillovers leaking onto whoever happens to be nearby. The effect is specific to the districts the water actually reached.

The second check looks inside the average. Splitting the treated districts into cities (Kota) and rural regencies (Kabupaten) reveals very different experiences. Rural districts took the brunt of the 2005 shock (−0.0883, p < 0.01) — agriculture floods badly — with a modest rebound (+0.0479). Cities, by contrast, barely contracted in 2005 (−0.0424, insignificant) but rebounded enormously (+0.1226, p < 0.01), reflecting the urban concentration of reconstruction. One caveat the paper itself flags: there are only 2 flooded city districts, so the city column is statistically fragile (few independent clusters) — read its precision, not just its point estimate, with care.

10. Discussion

What we found. Four methods converge on one story. Flooded districts lost about 7.9% of output in 2005 but grew 6.3 percentage points per year faster in 2006–08, ending on a permanently higher path — Aceh’s “recovery beyond the counterfactual trend.” Night-lights confirm it at finer resolution and show the gain concentrated in the worst-hit places. A synthetic control built from 76 donor districts puts flooded Aceh +18.3% above its no-tsunami twin by 2012. The result is robust to a neighbor-district placebo and survives honest, spatially-corrected inference (where it is significant at 5%, not the spuriously confident 1%).

So what? The substantive lesson is not “disasters are good” — they are not; 130,000 people died. It is that a localized catastrophe followed by large, well-governed reconstruction can leave a poor region on a higher long-run trajectory. Aceh received aid worth about 150% of its damages, spent through a low-corruption agency, on infrastructure rebuilt “better than before.” That combination — not the wave — is what bent the growth path upward. For disaster policy, the design lesson is just as important as the result: a credible evaluation needs exogenous exposure (geography, not choice), a finer-than-national unit of analysis, and spatially honest standard errors.

Limitations. Be appropriately humble. The data are synthetic — calibrated to teach the methods, not to report new facts about Aceh. The treatment group is tiny (10 districts), so point estimates are fragile and standard errors wide; the Aceh-only and city columns are especially imprecise. Identification is observational: parallel trends is an assumption, supported by the flat pre-trend and the null placebo but never proven. And a single, exceptionally well-funded case study travels poorly — Aceh’s recovery is evidence about well-governed mega-reconstruction, not about disaster aid in general.

11. Reproduction audit: synthetic data vs the paper

Because the data are synthetic, transparency demands that we line our numbers up against the published ones. The data-generating process was tuned to match the paper column by column; signs and significance agree throughout, and magnitudes land within about 0.005 on the headline cells.

Result	This synthetic data	Paper (reported)	Sign	Significance
DiD GDP, 2005 (Table 2, col 1)	−0.0792***	≈ −0.081***	✓	✓
DiD GDP, recovery 2006-08 (col 1)	+0.0628**	≈ +0.059**	✓	✓
DiD GDP, recovery vs Aceh controls (col 3)	+0.0310 (ns)	≈ +0.030**	✓	partial
DiD per-capita, recovery (Table 8)	+0.0827***	≈ +0.078***	✓	✓
Night-lights, share-of-pop recovery (Table 3)	+0.0160***	≈ +0.016***	✓	✓
Night-lights, share-of-area recovery (Table 3)	+1.660***	≈ +1.75***	✓	✓
Night-lights quintiles (Table 4)	only Q5 significant	only Q5 significant	✓	✓
City vs rural, 2005 (Table 7)	rural −0.0883*** / city ns	rural ≈ −0.098*** / city ns	✓	✓
Placebo neighbours (Table 9)	all ns	all ns	✓	✓
Synthetic control ATT	+18.3%	“recovery beyond counterfactual”	✓	qualitative

Two honest gaps. The Aceh-only column-3 recovery effect matches the paper in magnitude (+0.031 vs +0.030) but reads as insignificant here, because with the same 10 treated units in every column our synthetic standard errors are similar across columns, whereas the paper’s Aceh-only sample is more precise. And the night-lights quintile magnitudes sit on Table 3’s (smaller) scale rather than the paper’s Table 4 scale — the paper’s own Tables 3 and 4 are mutually inconsistent in units, so no single process can reproduce both; we match the pattern (only Q5 significant) exactly. Everywhere else, direction and significance track the paper closely.

12. Summary and takeaways

Number to remember	Value
2005 output shock	−0.0792*** (≈ −8%)
2006–08 recovery premium	+0.0628** (per year)
Synthetic-control gap by 2012	+18.3%
Moran’s I (spatial autocorrelation)	+0.065 (p = 0.003)
Recovery SE: naive → Conley-HAC	0.0146 → 0.0244
Night-lights recovery (share-of-pop)	+0.0160***

A single “after” hides the story. The pooled 2×2 DiD was an insignificant +0.0125; only splitting time into event-time windows revealed the −0.079 collapse and +0.063 overshoot. When effects evolve, let them.
Triangulate. Difference-in-differences, an event study, a dose-response, and a synthetic control all pointed the same way — a far stronger claim than any one method alone.
Satellite data unlock localized questions. Night-lights gave a finer, exogenous measure that exposed the dose-response (only the worst-hit quintile rebounds) invisible at the district level.
Clustered treatment demands honest inference. With all treated units in one corner of the map, Conley spatial standard errors were not optional — they downgraded the recovery effect from a spurious *** to an honest **, without touching the point estimate.
Mind the small print. Ten treated districts make for fragile estimates; the result is about well-governed mega-reconstruction, on synthetic data, identified by an assumption. Strong evidence, stated with the caveats it deserves.
Next step. Try modern staggered-adoption DiD estimators, add prediction intervals to the synthetic control, or widen the donor pool — each is a natural extension of the toolkit here.

13. Exercises

Drop a donor. Re-fit VanillaSC after excluding the top-weighted donor (JAMBI_D01). Does the post-2005 gap shrink, hold, or grow relative to the headline +18.3%? What does the answer tell you about the counterfactual’s robustness?
Cutoff sensitivity. Recompute the recovery effect’s Conley-HAC standard error at cutoffs of {0, 50, 150, 300} km. At which cutoff, if any, does the recovery effect’s significance change? Relate your answer to the cutoff figure in Section 8.
Your own event study. Estimate the event study a second way with pyfixest’s factor syntax — pf.feols("gdp_growth ~ i(period, flooded, ref='baseline') | district_id + year", ...) — and check that its coefficients match the diff-diff version to four decimals. Why should two different libraries agree exactly?

14. References

Heger, M. P., & Neumayer, E. (2019). The impact of the Indian Ocean tsunami on Aceh’s long-term economic growth. Journal of Development Economics, 141, 102365. https://doi.org/10.1016/j.jdeveco.2019.06.008
Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association, 105(490), 493–505.
Conley, T. G. (1999). GMM estimation with cross-sectional dependence. Journal of Econometrics, 92(1), 1–45.
Indonesia Database for Policy and Economic Research (INDO-DAPOER) and SUSENAS — World Bank / BPS-Statistics Indonesia. https://datacatalog.worldbank.org/
DMSP-OLS Nighttime Lights — NOAA National Centers for Environmental Information. https://www.ncei.noaa.gov/
Center for Satellite Based Crisis Information (ZKI), German Aerospace Center (DLR), and the Dartmouth Flood Observatory (inundation maps).
pyfixest documentation — https://pyfixest.org/
diff-diff documentation — https://github.com/igerber/diff-diff
mlsynth documentation — https://github.com/jgreathouse9/mlsynth

This tutorial is a teaching replication built on synthetic data; see the data note in Section 1 and the reproduction audit in Section 11. The companion script.py regenerates every figure and table.

AI Podcast: Evaluating the Impact of Natural Disasters

Click play to load

0:00 0:00

The Augmented Synthetic Control Method: A Beginner's Tutorial with the Kansas Tax Cuts

Mon, 08 Jun 2026 00:00:00 +0000

Abstract

In May 2012 Kansas enacted one of the largest state tax cuts in recent U.S. history, billed as a supply-side experiment that should accelerate growth, yet with only one Kansas the missing counterfactual makes the policy’s effect hard to measure. This tutorial estimates the effect of the 2012 Kansas tax cut on log GDP per capita for a single treated unit, teaching the Augmented Synthetic Control Method (ASCM) of Ben-Michael, Feller, and Rothstein (2021) from classic synthetic control through ridge augmentation and covariate balancing. The data are the kansas panel shipped with the augsynth R package — a balanced panel of 50 U.S. states observed every quarter from 1990 Q1 to 2016 Q1 (105 quarters, 5,250 rows; 89 pre-treatment and 16 post-treatment quarters), with log gross state product per capita as the outcome. Classic SCM builds a synthetic Kansas from a 7-state convex blend (South Carolina 0.30, Washington 0.22, Texas 0.15) and estimates an average post-2012 ATT of −0.029 log points (≈ −2.9%) with an L2 pre-fit imbalance of 0.083 (79.5% better than uniform). Ridge ASCM deepens the estimate to −0.040 (≈ −3.9%), tightens the imbalance to 0.062, and reports an estimated bias of 0.011 — about a third of the effect — while moving the donor weights by a negligible RMS of 0.015; adding six covariates pushes the ATT to −0.061 (≈ −5.9%) with covariate imbalance of 0.005. Four inference approaches — placebo/permutation (p = 0.10), conformal (p = 0.066), jackknife+ ([−0.058, −0.021], excluding zero), and the leave-one-donor jackknife (SE 0.024) — agree on the same −0.040 point estimate but disagree at the margin. The pattern implies the tax cut is associated with a persistent 3 to 6% GDP-per-capita shortfall that classic SCM understates, with borderline but coherent statistical support.

1. Overview

In May 2012, Kansas enacted one of the largest state tax cuts in recent U.S. history. Governor Sam Brownback called it “a real-live experiment” in supply-side economics: slash personal income taxes, and growth — the theory went — would follow. Did it? Answering that question is harder than it sounds, because we cannot rewind history and run Kansas without the tax cut to compare. There is only one Kansas, and it took the treatment.

The synthetic control method (SCM) offers a clever way out: if no single state is a good stand-in for Kansas, perhaps a weighted blend of several states can be. Build a “synthetic Kansas” from other states so that it matches the real Kansas before 2012, and its path after 2012 becomes the counterfactual — what Kansas’s economy would have done without the tax cut. The gap between the two is the estimated effect.

But classic SCM has an Achilles’ heel. It can only build the synthetic from a convex combination of donors — non-negative weights that sum to one — and sometimes no such combination matches the treated unit well enough. When the pre-treatment fit is imperfect, the estimate is biased, and the original authors of SCM recommend not using it at all. The Augmented Synthetic Control Method (ASCM) of Ben-Michael, Feller, and Rothstein (2021) rescues these cases: it keeps the interpretable SCM weights but adds an outcome model that estimates and subtracts the leftover bias.

This tutorial teaches ASCM for a single treated unit through the canonical Kansas example, using the augsynth R package. We deliberately move slowly: every method comes with the intuition first, then the equation, then the code, then the interpretation of real numbers. Inference in synthetic control is notoriously slippery, so we devote a full section to four different ways of asking “could this effect just be noise?”

If you want the multi-country, staggered-adoption version of these tools (multisynth, augsynth_multiout), see the companion post Augmented Synthetic Control for Multiple Countries. This tutorial is the single-treated-unit foundation to read first.

1.1 Learning objectives

By the end of this tutorial, you will be able to:

Implement classic SCM, Ridge ASCM, and covariate-augmented ASCM for one treated unit with augsynth().
Estimate the ATT of the 2012 Kansas tax cut on log GDP per capita, and read the pre-fit imbalance, the chosen penalty, and the estimated bias.
Explain why the ridge bias-correction moves the estimate, and why it does so with almost no change to the donor weights.
Assess statistical significance four ways — placebo/permutation, conformal, jackknife+, and the leave-one-donor jackknife — and explain when they disagree.

The roadmap below shows the path we will take. The branch point is the quality of the pre-treatment fit: when classic SCM matches Kansas well, augmentation changes little; when it cannot, ridge ASCM extrapolates just enough to de-bias the estimate.

flowchart TD
P("Kansas panel<br/>50 states, 1990-2016<br/>log GDP per capita") --> Q{"Can donors match<br/>Kansas before 2012?"}
Q -->|"fit is good"| S("Classic SCM<br/>progfunc = None")
Q -->|"fit imperfect<br/>(the mid-2000s gap)"| R("Ridge ASCM<br/>progfunc = Ridge")
S --> W("SCM weights<br/>(convex recipe, 7 donors)")
W --> B("+ Ridge outcome model<br/>estimate &amp; subtract bias")
R --> B
B --> Z{"Add covariates?"}
Z -->|"yes"| C("Covariate ASCM<br/>y ~ trt | Z")
Z -->|"no"| A("ATT = actual - synthetic")
C --> A
A --> I("Inference<br/>placebo · conformal · jackknife+ · jackknife")
classDef sty_Q fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class Q sty_Q
classDef sty_Z fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class Z sty_Z
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class P,W,I blue
class S,R orange
class B,C,A teal

2. Key concepts

Before the code, here are the seven ideas that carry the whole tutorial. Each card has a plain definition, a concrete example from the Kansas study, and an everyday analogy. The two hardest for newcomers are extrapolation (concept 3) and conformal inference (concept 7) — linger on those.

1. Synthetic control method (SCM). A weighted average of untreated “donor” units, built so its pre-treatment path matches the treated unit. The synthetic’s post-treatment trajectory is the estimated counterfactual; the gap to the real unit is the treatment effect.

Example

“Synthetic Kansas” is a blend of 7 states (South Carolina, Washington, Texas, …) chosen so that its pre-2012 GDP per capita tracks the real Kansas. After 2012, the gap is the effect of the tax cut.

Analogy

A stunt double assembled from several extras. Before the dangerous scene (treatment) the double mimics the star perfectly; during the scene it shows what would have happened to the star.

2. Donor pool. The set of untreated units the synthetic is built from. The treated unit is excluded, as is anyone else exposed to the treatment.

Example

The 49 U.S. states other than Kansas. None of them cut income taxes the way Kansas did in 2012, so each is a candidate ingredient for synthetic Kansas.

Analogy

The casting shortlist for the stunt double — only extras who did not perform the stunt are allowed to audition.

3. Convex hull and extrapolation. SCM weights are non-negative and sum to one, so the synthetic can only land inside the range spanned by the donors (it interpolates). Matching a treated unit that lies outside that range requires negative weights — extrapolation — which classic SCM forbids.

Example

In the mid-2000s, Kansas’s economy wobbles near the edge of what the donors can reproduce, so classic SCM leaves a stubborn gap there (a single quarter off by 0.043 log points). Ridge ASCM allows a controlled amount of extrapolation to close it.

Analogy

Mixing paint from stocked colours: you can blend them to get any shade between them, but you cannot use a negative amount of blue to get something brighter than your brightest blue.

4. Ridge penalty and bias correction. ASCM fits a ridge regression of donors’ outcomes on their lagged outcomes, predicts the residual imbalance, and subtracts it from the SCM estimate. A penalty λ controls how far the weights may leave the convex hull — large λ stays close to SCM, small λ extrapolates more.

Example

Augmentation moves the Kansas estimate from −0.029 to −0.040 and cuts the pre-fit imbalance from 0.083 to 0.062, reporting an estimated bias of 0.011 — about a third of the effect.

Analogy

A spell-checker for the counterfactual: SCM writes the first draft, and ridge fixes the systematic typos it can detect from the donors.

5. ATT — the estimand. The Average Treatment effect on the Treated: actual minus synthetic, in the post-treatment period, for the treated unit. It answers “what did the treatment do to Kansas,” not to an average state.

Example

The average post-2012 gap of about −0.04 log points ≈ a 3.9% shortfall in Kansas’s GDP per capita relative to its synthetic twin.

Analogy

The moment the stunt double keeps to the safe “no-stunt” script while the star veers off it — the distance between them is the effect of the stunt.

6. Pre-treatment fit (RMSPE / L2 imbalance). How closely the synthetic tracks the treated unit before treatment. augsynth reports it as an L2 imbalance and as a percent improvement over naive uniform weights. A bad pre-fit makes any post-treatment gap meaningless.

Example

Classic SCM achieves L2 = 0.083 (79.5% better than uniform weights); Ridge ASCM tightens it to 0.062 (84.7%).

Analogy

How convincing the stunt double looks before the scene. If the audience can already tell them apart, nothing that happens during the scene is believable.

7. Conformal inference. augsynth’s default test. Under the hypothesis of no effect, the post-treatment gaps should look like the pre-treatment gaps (just noise). Conformal inference checks whether the post-treatment residual “conforms” to the distribution of pre-treatment residuals, yielding a p-value and a pointwise confidence band.

Example

For classic SCM the 2012 Q3 effect is −0.041 with a 95% interval of [−0.070, −0.015] and p = 0.023 — the gap is larger than typical pre-period noise.

Analogy

A lie detector calibrated on a person’s resting readings. A post-event spike only counts as a signal if it exceeds their normal fluctuation.

3. Setup

The augsynth package is not on CRAN; install it once from GitHub. The other packages are standard. We fix a random seed because two of the inference procedures use resampling.

# install.packages("remotes")
# remotes::install_github("ebenmichael/augsynth") # one-time install
library(augsynth)
library(dplyr)
library(tidyr)
library(ggplot2)
library(readr)
set.seed(20260608)
# Site colour palette used throughout
STEEL_BLUE <- "#6a9bcc" # synthetic control / SCM
WARM_ORANGE <- "#d97757" # treated (Kansas) / actual
TEAL <- "#00d4c8" # ridge-augmented

A note on augsynth’s formula mini-language, which we will use repeatedly:

outcome ~ treatment — the minimal model: match on the entire pre-treatment outcome series.
outcome ~ treatment | z1 + z2 + ... — also balance the auxiliary covariates listed after the |.
progfunc = "None" — pure SCM, no outcome model. progfunc = "Ridge" — augment with ridge regression.
scm = TRUE — use SCM (convex) weights as the starting point.

4. Data and key variables

The kansas dataset ships with augsynth. To keep this tutorial fully reproducible, we load it from a CSV in the post’s GitHub folder (a copy of the package’s kansas object), with a local fallback.

url <- "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/tutorials/r_augsynth/kansas.csv"
kansas <- if (file.exists("kansas.csv")) read_csv("kansas.csv") else read_csv(url)
kansas <- as.data.frame(kansas)
# The panel dimensions and the treated unit
length(unique(kansas$fips)) # number of states
range(kansas$year_qtr) # time span

[1] 50
[1] 1990 2016

The panel is balanced: 50 U.S. states, observed every quarter from 1990 Q1 to 2016 Q1 — that is 105 quarters per state, 5,250 rows in all. The key variables are few and clear:

Variable	Role	Meaning
`fips`	unit id	State FIPS code (Kansas = 20)
`year_qtr`	time	Year plus quarter, e.g. `2012.25` = Q2 2012
`lngdpcapita`	outcome	Natural log of gross state product per capita
`treated`	treatment	1 for Kansas from 2012 Q2 onward, 0 otherwise
`revstatecapita`, `revlocalcapita`, `avgwklywagecapita`, `estabscapita`, `emplvlcapita`	covariates	Per-capita state revenue, local revenue, weekly wage, establishments, employment

Let us look at exactly when treatment switches on:

kansas %>%
filter(state == "Kansas" & year_qtr >= 2012 & year_qtr < 2013) %>%
select(year, qtr, year_qtr, treated, gdp, lngdpcapita)

 year qtr year_qtr treated gdp lngdpcapita
2012 1 2012.00 0 143844 10.81687
2012 2 2012.25 1 141518 10.79991
2012 3 2012.50 1 138890 10.78051
2012 4 2012.75 1 139603 10.78498

Interpretation. The treated flag is zero for Kansas through 2012 Q1 and one from 2012 Q2 (year_qtr = 2012.25) onward — exactly when the tax cut took effect. Notice the outcome already dips in the quarters right after: lngdpcapita falls from 10.817 to 10.781. But that raw dip is not yet a causal estimate — every state’s economy moved over this period. We need the counterfactual. With 89 pre-treatment quarters and 16 post-treatment quarters, we have a long history to pin down a credible synthetic Kansas — and, as we will see, a long pre-period is exactly what makes inference possible.

5. Exploratory view: why we need a synthetic control

ggplot() +
geom_line(data = filter(kansas, fips != 20),
aes(year_qtr, lngdpcapita, group = fips), colour = "grey78") +
geom_line(data = filter(kansas, fips == 20),
aes(year_qtr, lngdpcapita), colour = WARM_ORANGE, linewidth = 1.1) +
geom_vline(xintercept = 2012.25, linetype = "dashed")

Interpretation. Kansas sits squarely in the middle of the pack and rises along with every other state for 26 years. This is the fundamental obstacle: there is no single state whose line lies on top of Kansas’s, so we cannot just pick “the most similar state” as a comparison. We have to construct a comparison by blending donors. The plot also previews the difficulty to come — in the mid-2000s the lines fan apart and Kansas wanders relative to its neighbours, which is precisely where a convex blend will struggle to keep up.

6. Baseline: the classic synthetic control

6.1 The idea, then the math

SCM picks donor weights so that the weighted donor outcomes reproduce the treated unit’s pre-treatment path as closely as possible, subject to two rules: the weights are non-negative and they sum to one. Those two rules are what keep the synthetic interpretable and prevent wild extrapolation.

Writing $X_1$ for Kansas’s vector of pre-treatment outcomes and $X_0$ for the matching matrix of donor outcomes, the SCM weights solve

$$\hat{\gamma}^{scm} = \arg\min_{\gamma} \| X_1 - X_0’ \gamma \|_2^2 \quad \text{subject to} \quad \sum_{i} \gamma_i = 1, \quad \gamma_i \ge 0$$

In words, this says: choose the weights $\gamma_i$ that make the blended donor history $X_0’\gamma$ as close as possible (in squared distance) to Kansas’s history $X_1$, while staying on the simplex (non-negative, summing to one). In code, $X_1$ is Kansas’s pre-2012 lngdpcapita and $X_0$ holds the 49 donors’ pre-2012 series; $\hat{\gamma}^{scm}$ becomes syn$weights.

Once we have the weights, the estimated effect in each post-treatment quarter $t$ is simply actual minus synthetic:

$$\hat{\tau}_t = Y_{1t} - \sum_{i} \hat{\gamma}_i^{scm}\, Y_{it}, \quad t > T_0$$

In words, the per-quarter ATT is Kansas’s realized outcome $Y_{1t}$ minus the weighted sum of donor outcomes (the synthetic). Here $T_0$ is the last pre-treatment period; in code $Y_{1t}$ is Kansas’s lngdpcapita and the weighted sum is predict(syn, att = FALSE).

6.2 Fitting it

syn <- augsynth(lngdpcapita ~ treated, fips, year_qtr, kansas,
progfunc = "None", scm = TRUE)
summary(syn)

Fit to 50 units and 89+16 = 105 time points; 1 treated at year_qtr 2012.25.
7 donor units used with weights of 0.053 to 0.301
Average ATT Estimate (p Value for Joint Null): -0.0294 ( 0.311 )
L2 Imbalance: 0.083
Percent improvement from uniform weights: 79.5%
Inference type: Conformal inference

Interpretation. Classic SCM builds synthetic Kansas from 7 donor states and estimates an average post-2012 ATT of −0.0294 log points. Because effects in logs are approximately percentages, that is a shortfall of about 2.9% in GDP per capita relative to the counterfactual. The pre-treatment L2 imbalance is 0.083, which augsynth tells us is 79.5% better than the naive alternative of weighting all 49 donors equally. The joint-null p-value of 0.311 is our first significance signal — and not a strong one — but hold that thought until the inference section.

We can see the synthetic control as a transparent recipe — the weights live in syn$weights, named by state FIPS code:

w <- syn$weights[, 1]
round(sort(w[w > 0.001], decreasing = TRUE), 3) # the donors that actually matter

Interpretation. This is SCM’s signature virtue: 42 of the 49 donors get exactly zero weight, and the seven that remain form a recipe you can name and defend. South Carolina carries 30% of synthetic Kansas, Washington 22%, Texas 15%. The sparsity is not an accident — the simplex constraint pushes most weights to exactly zero. The cost of that interpretability is rigidity: if no convex recipe matches Kansas perfectly, SCM cannot do better, and the leftover mismatch becomes bias.

6.3 Seeing the counterfactual and the gap

plot(syn, plot_type = "outcomes") # actual Kansas vs its synthetic control

Interpretation. Before the dashed line the two series are nearly on top of each other — synthetic Kansas is a believable double. After 2012, the orange (actual) line slips below the blue (synthetic): Kansas grew more slowly than its counterfactual. That visible wedge is the treatment effect.

plot(syn) # the gap (actual - synthetic) with a pointwise conformal band

Interpretation. The gap plot isolates the effect on a single axis. Pre-2012 it oscillates around zero — but look at the dip near 2005–2006, which reaches −0.043 in one quarter. That is a pre-treatment gap, where the true effect is zero by definition, so it is pure imbalance: SCM simply could not match Kansas there. Post-2012 the gap turns reliably negative, deepest at 2013 Q3 (−0.046) and 2014 Q1 (−0.045). The shaded conformal band is wide, foreshadowing that significance will be a close call.

That stubborn mid-2000s imbalance is the motivation for everything that follows. It is exactly the situation Abadie and co-authors warn about — and exactly what ASCM was built to fix.

7. The augmentation: Ridge ASCM

7.1 The idea: estimate the bias, then subtract it

Here is the key insight. When the synthetic does not perfectly match Kansas before treatment, the post-treatment gap mixes two things: the real effect and the bias from that imperfect match. If we could estimate the bias, we could subtract it off. ASCM does exactly this by fitting an outcome model — a regression that predicts a unit’s outcome from its pre-treatment history — and using it to forecast how much the residual imbalance distorts the estimate.

The augmented counterfactual for the post-period can be written two equivalent ways. First, as “SCM plus a bias correction”:

$$\hat{Y}_{1T}^{aug}(0) = \sum_{i} \hat{\gamma}_i^{scm} Y_{iT} + \left( \hat{m}_{1T} - \sum_{i} \hat{\gamma}_i^{scm} \hat{m}_{iT} \right)$$

In words: start from the plain SCM counterfactual (the first term), then add a correction equal to the imbalance the outcome model $\hat{m}$ predicts (the parenthesis). If the model thinks Kansas’s history is systematically a little above what its donors’ weights reproduce, that term nudges the counterfactual accordingly. In augsynth this correction is reported as the “Avg Estimated Bias.”

The same estimator can be rearranged into a “model plus reweighted residuals” form, which is why ASCM is often described as doubly robust:

$$\hat{Y}_{1T}^{aug}(0) = \hat{m}_{1T} + \sum_{i} \hat{\gamma}_i^{scm} \left( Y_{iT} - \hat{m}_{iT} \right)$$

In words: predict Kansas directly from the outcome model ($\hat{m}_{1T}$), then correct it using the SCM-weighted residuals of the donors. If the SCM fit were already perfect, the residual term would balance out and ASCM would equal SCM — augmentation only does something when there is imbalance to fix.

The outcome model augsynth uses by default is ridge regression of donors’ outcomes on their lagged outcomes, with an L2 penalty:

$$\hat{\eta}^{ridge} = \arg\min_{\eta} \frac{1}{2}\sum_{i} \left( Y_i - X_i’ \eta \right)^2 + \lambda \| \eta \|_2^2$$

In words: fit a regression predicting the post-period outcome from the pre-period outcomes, but shrink the coefficients toward zero by an amount set by $\lambda$. The penalty $\lambda$ — in code, asyn$lambda — is the single dial that controls how aggressive the bias correction is.

7.2 Why ridge ASCM barely disturbs the weights

A beautiful result in the paper is that Ridge ASCM is equivalent to a penalized SCM: instead of forbidding negative weights outright, it lets the weights leave the simplex but penalizes how far they stray from the SCM solution:

$$\hat{\gamma}^{aug} = \arg\min_{\gamma} \frac{1}{2\lambda} \| X_1 - X_0’ \gamma \|_2^2 + \frac{1}{2} \| \gamma - \hat{\gamma}^{scm} \|_2^2 \quad \text{s.t.} \quad \sum_i \gamma_i = 1$$

In words: find weights that fit the pre-period well (first term) but stay close to the trustworthy SCM weights (second term). A large $\lambda$ makes the first term cheap, so the weights barely move from SCM; a small $\lambda$ lets them extrapolate more to chase a better fit. This is why, as we will see, the Kansas weights hardly change even as the fit improves.

And the improvement is guaranteed. The paper shows the augmented pre-treatment imbalance can only shrink relative to SCM:

$$\| X_1 - X_0’ \hat{\gamma}^{aug} \|_2 \le \frac{\lambda}{d^2 + \lambda} \, \| X_1 - X_0’ \hat{\gamma}^{scm} \|_2$$

In words: the augmented imbalance is the SCM imbalance multiplied by a factor that is always less than one. So augmentation never makes the pre-fit worse — it can only tighten it.

7.3 Choosing the penalty by cross-validation

How do we pick $\lambda$? augsynth uses leave-one-pre-period-out cross-validation: drop each pre-treatment quarter in turn, predict it, and measure the error. We then pick the $\lambda$ that the data say generalizes best.

asyn <- augsynth(lngdpcapita ~ treated, fips, year_qtr, kansas,
progfunc = "Ridge", scm = TRUE) # lambda chosen by CV
plot(asyn, plot_type = "cv")

Interpretation. The CV curve is U-shaped: too small a $\lambda$ overfits the pre-period (right-hand rise is the opposite extreme, where the model becomes plain SCM), too large washes out the correction. By default augsynth applies the one-standard-error rule — it chooses the largest $\lambda$ whose error is within one SE of the minimum (here λ = 0.079). This is deliberately conservative: among statistically indistinguishable choices, it keeps the weights closest to the safe SCM solution. (Set min_1se = FALSE to instead minimize CV error and extrapolate more.)

7.4 What augmentation does to Kansas

summary(asyn)

49 donor units used with weights of 0.001 to 0.316
Average ATT Estimate (p Value for Joint Null): -0.0401 ( 0.066 )
L2 Imbalance: 0.062
Percent improvement from uniform weights: 84.7%
Avg Estimated Bias: 0.011

Interpretation. Three numbers tell the story. First, the estimate deepens from −0.0294 to −0.0401 (about a 3.9% shortfall) — augmentation reveals a larger effect than plain SCM. Second, the pre-fit improves: L2 drops from 0.083 to 0.062 (84.7% better than uniform). Third, augsynth reports an estimated bias of 0.011 — its own measure of how much the SCM number was distorted by imperfect matching. That 0.011 is roughly one-third of the −0.040 effect, a striking confirmation of the paper’s warning that imperfect SCM fit can substantially understate the effect.

# both gap series come from predict(..., att = TRUE); see analysis.R for the full ggplot
scm_gap <- predict(syn, att = TRUE)
ridge_gap <- predict(asyn, att = TRUE)

Interpretation. The two gaps share the same shape, but the teal ridge line dives a little deeper after 2012 — the visual signature of the bias correction. Crucially, this did not require throwing out the interpretable SCM recipe:

# How far did the weights actually move?
sqrt(mean((asyn$weights[,1] - syn$weights[,1])^2))

[1] 0.0147

Interpretation. The root-mean-square change in the donor weights is just 0.0147 — almost nothing. Although 21 donors now carry small negative weights (the controlled extrapolation), synthetic Kansas is essentially the same blend as before. Ridge ASCM bought a better fit and a de-biased estimate for the price of a tiny, principled departure from the convex hull. We can see where that price was paid:

Interpretation. Restricting attention to the pre-period — where the gap should be zero — shows exactly what augmentation fixed. SCM’s worst quarter (2005 Q4, −0.043) is pulled back to −0.031 by ridge, and the rest of the turbulent mid-2000s is calmed too. That is the bias correction at work: it spent its small extrapolation budget precisely where classic SCM was failing.

8. Adding covariates

So far we matched only on the history of the outcome. But augsynth can also balance auxiliary covariates — state revenue, wages, establishments, employment — by listing them after a | in the formula. Both the lagged outcomes and the covariates then enter the SCM balancing problem and the ridge outcome model:

$$\min_{\eta_x, \eta_z} \frac{1}{2}\sum_{i} \left( Y_i - X_i’ \eta_x - Z_i’ \eta_z \right)^2 + \lambda_x \| \eta_x \|_2^2 + \lambda_z \| \eta_z \|_2^2$$

In words: the outcome model now uses both the lagged outcomes $X$ (with coefficients $\eta_x$) and the covariates $Z$ (with coefficients $\eta_z$), each with its own ridge penalty. The covariates give the synthetic more ways to resemble Kansas than the outcome path alone.

covsyn <- augsynth(lngdpcapita ~ treated | lngdpcapita + log(revstatecapita) +
log(revlocalcapita) + log(avgwklywagecapita) +
estabscapita + emplvlcapita,
fips, year_qtr, kansas, progfunc = "ridge", scm = TRUE)
summary(covsyn)

49 donor units used with weights of 0.004 to 0.356
Average ATT Estimate (p Value for Joint Null): -0.0609 ( 0.124 )
L2 Imbalance: 0.054
Percent improvement from uniform weights: 86.6%
Covariate L2 Imbalance: 0.005
Percent improvement from uniform weights: 97.7%
Avg Estimated Bias: 0.027

Interpretation. Adding the six covariates sharpens balance on two fronts: the outcome imbalance falls to 0.054 and the covariate imbalance to 0.005 — a 97.7% improvement over uniform weights. With the additional structure, the estimate deepens again to −0.0609 (≈ −5.9%). The covariates fix dimensions of similarity that lagged outcomes alone miss — for instance, matching Kansas’s employment and wage levels, not just its GDP trajectory.

Two further options are worth knowing but not belaboring. Residualizing (residualize = TRUE) first regresses the outcome on the covariates and fits ASCM on the residuals; on Kansas it drives covariate imbalance to exactly zero and gives an ATT of −0.0548. The simplest possible outcome model, a unit fixed effect (fixedeff = TRUE, which de-means each series), gives −0.0335 — between plain SCM and ridge. Every route agrees on the sign and the rough size of the effect.

9. Inference: could this just be noise?

This is the part students find hardest, and for good reason: a synthetic-control estimate is a difference between two estimated curves, built from one treated unit. Classical standard-error formulas do not obviously apply. augsynth ships four tools, and they can give different verdicts. The shared question they all answer is: is the post-treatment gap bigger than what we would see by chance? They differ in how they define “by chance.”

We run all four on the Ridge ASCM fit (only the ridge estimator supports standard errors).

9.1 Placebo / permutation tests (the classic approach)

The original SCM inference, due to Abadie and co-authors, is a placebo test. The logic: if the tax cut truly moved Kansas, then re-running the whole analysis pretending some untreated donor was “treated” should usually produce a much smaller gap. Do this for every donor, and you get a distribution of placebo effects to compare Kansas against.

A common summary statistic is the RMSPE ratio — post-treatment fit error divided by pre-treatment fit error — which rewards units that tracked well before and diverged after. The permutation p-value is Kansas’s rank in that distribution:

$$p = \frac{\#\{\, i : r_i \ge r_1 \,\}}{N}, \qquad r_i = \frac{\text{RMSPE}_i^{\text{post}}}{\text{RMSPE}_i^{\text{pre}}}$$

In words: compute each unit’s post/pre error ratio $r_i$, then ask what fraction of all units (including Kansas, unit 1) have a ratio at least as large as Kansas’s. A small fraction means Kansas stands out.

plot(asyn, plot_type = "placebo") # switches to permutation inference

Interpretation. Kansas’s pre-2012 line is buried in the grey chorus — a good fit, so it belongs in the comparison. After 2012 it drops toward the bottom edge, but it is not the single most extreme path. Its post/pre RMSPE ratio of 6.36 ranks 5th of 50, giving a permutation p = 0.10. Read honestly: Kansas’s response is unusual, but a handful of placebo states show swings just as large, so the placebo test alone cannot rule out chance at the 5% level. The placebo test is intuitive and assumption-light, but it has low power with a small donor pool and assumes Kansas is exchangeable with the donors.

9.2 Conformal inference (the modern default)

augsynth’s default is conformal inference (Chernozhukov, Wüthrich, and Zhu, 2021). It tests a sharp null — “the effect equals $\tau_0$” — by checking whether, after subtracting $\tau_0$, the post-treatment residual looks like an ordinary draw from the pre-treatment residuals. Inverting the test over a grid of $\tau_0$ values yields a confidence interval. The p-value for “no effect” is

$$p(\tau_0) = \frac{1}{T_0 + 1} \left( 1 + \sum_{t=1}^{T_0} \mathbf{1}\{\, |\hat{u}_t| \ge |\hat{u}_{T}| \,\} \right)$$

where the residual at the candidate effect $\tau_0$ is

$$\hat{u}_t = Y_{1t} - \tau_0 \cdot \mathbf{1}\{t > T_0\} - \sum_{i} \hat{\gamma}_i(\tau_0)\, Y_{it}$$

In words: under the null, the post-treatment residual $|\hat{u}_T|$ should be no more extreme than a typical pre-treatment residual $|\hat{u}_t|$. The p-value is the share of pre-periods whose residual is at least as large — a long pre-period (89 quarters here) is what gives this test its resolution.

summary(asyn)$average_att # the conformal joint-null p-value

Average Post-Treatment Effect -0.0401 p = 0.066

Interpretation. The conformal joint-null p-value is 0.066 — borderline, just above the conventional 5% line. But the pointwise picture (the band in the gap plots) is sharper: several individual quarters clear significance, including 2013 Q3 (−0.059, p = 0.024) and 2014 Q1 (−0.058, p = 0.018). Conformal inference is the most robust of the four here because it does not require Kansas to be exchangeable with the donors and it exploits the long pre-period.

9.3 Jackknife+ over time

The jackknife+ builds a confidence interval for the average effect by leaving out one pre-treatment period at a time, refitting, and using the spread of the leave-one-out prediction errors to bound the estimate.

summary(asyn, inf_type = "jackknife+")$average_att

Average Post-Treatment Effect -0.0401 95% CI [-0.0576, -0.0206]

Interpretation. The jackknife+ interval [−0.058, −0.021] excludes zero — by this criterion the average effect is significant. It is the only one of the four that gives a clean “significant” verdict for the average, because it asks a different question: how stable is the estimate when we perturb the time dimension, rather than whether Kansas is special among states.

9.4 Leave-one-donor jackknife

The final tool drops one donor at a time, recomputes the ATT, and forms a standard error from how much the estimate moves — the classic “how sensitive is this to any single comparison unit?” check.

summary(asyn, inf_type = "jackknife")$average_att

Average Post-Treatment Effect -0.0401 Std.Error 0.0242

Interpretation. With a standard error of 0.0242, the Wald interval is −0.040 ± 1.96 × 0.024 = [−0.088, 0.007], which includes zero. The reason is intuitive: synthetic Kansas leans heavily on a few donors (South Carolina alone is 30%), so dropping one of them can move the estimate appreciably, inflating the standard error. This is the most conservative of the four.

9.5 Putting the four together

Interpretation. This single figure is the lesson of the section. The point estimate is the same −0.040 in all four; what differs is the uncertainty. The methods disagree because they probe different sources of variation — over time (conformal, jackknife+) versus over units (permutation, leave-one-donor jackknife) — and make different exchangeability assumptions. The honest conclusion is not “significant” or “not significant” but a nuanced one: a real, modest negative effect, clearly visible in 2013–2014, whose statistical strength is borderline and depends on which question you ask. Reporting several methods, as we have, is far more honest than cherry-picking the one that clears 0.05.

10. Results: the five specifications together

Interpretation. Read down the ladder of methods, a consistent pattern emerges: the more we de-bias and balance, the larger the measured damage. The ATT moves from −0.029 (classic SCM) to −0.040 (ridge) to −0.061 (covariate-augmented), while the pre-fit L2 imbalance falls from 0.083 to 0.062 to 0.054. The fixed-effect (−0.034) and residualized (−0.055) variants fall between these in magnitude. This monotonic pattern is reassuring rather than alarming: it tells us the un-augmented SCM estimate was the conservative one, understating the effect because of the very imbalance ASCM was designed to remove.

Specification	ATT (log pts)	≈ % effect	Pre-fit L2	Est. bias
Classic SCM	−0.029	−2.9%	0.083	—
Ridge ASCM	−0.040	−3.9%	0.062	0.011
Covariate ASCM	−0.061	−5.9%	0.054	0.027
Residualized	−0.055	−5.3%	0.067	0.006
Fixed effects	−0.034	−3.3%	0.082	—

11. Discussion: what did the Kansas experiment do?

Putting the pieces together, the evidence points one direction: the 2012 Kansas tax cut is associated with a persistent shortfall in GDP per capita of roughly 3 to 6%, relative to a synthetic Kansas built from other states. The effect is strongest in 2013–2014, robust in sign across all five specifications, and — importantly — larger once we correct the bias that classic SCM leaves behind. The supply-side promise of accelerated growth does not appear in the data; if anything, Kansas underperformed its counterfactual.

Three caveats keep this honest. First, significance is genuinely borderline: depending on the inference method, the average effect either clears or just misses the 5% bar, even though individual 2013–2014 quarters are significant. We should describe this as suggestive-to-moderate evidence, not a knock-down result. Second, the estimate is the net gap, not a tax-only effect — Kansas also suffered a severe drought and aerospace-sector shocks over this window, which a synthetic control cannot separately strip out. Third, like all SCM analyses, the result rests on the assumption that a weighted blend of donors can stand in for Kansas’s untreated path, and that no other state was affected by Kansas’s policy.

For a policymaker, the practical takeaway is not a single number but a pattern: the better we make the comparison, the worse the tax cut looks, and at no point does it look good. That is a meaningfully different conclusion than “no detectable effect,” which is what a hasty reading of the classic-SCM p-value alone might have suggested.

12. Summary and next steps

What we did and found:

Built a synthetic Kansas from a 7-state convex blend (classic SCM) and estimated a −2.9% post-2012 effect with a pre-fit imbalance of 0.083.
Augmented with ridge regression to correct the imperfect pre-2012 fit, deepening the estimate to −3.9%, improving the fit to 0.062, and quantifying the SCM bias at 0.011 (≈ one-third of the effect) — all while moving the weights by a negligible RMS of 0.015.
Added covariates to push the estimate to −5.9% with near-perfect covariate balance.
Tested significance four ways and found a coherent but borderline picture: jackknife+ excludes zero, conformal and permutation hover around p = 0.07–0.10, and the leave-one-donor jackknife is the most cautious.

Limitations: a single treated unit and a modest donor pool limit statistical power; the estimate cannot separate the tax cut from contemporaneous shocks; and the conclusion depends on the credibility of the synthetic match.

Where to go next:

Try other outcome models via progfunc ("gsyn", "en", …) and compare.
Move to multiple treated units and staggered adoption with multisynth, or multiple outcomes with augsynth_multiout, in the companion post on Augmented Synthetic Control for Multiple Countries.
Run your own sensitivity checks: vary the donor pool, the pre-period length, and the test statistic in summary(..., stat_func = ...).

12.1 Exercises

In-time placebo. Re-fit Ridge ASCM pretending the tax cut happened in 2009 Q2 instead of 2012 Q2 (restrict the data to pre-2012 and set the fake treatment time). The estimated “effect” should be near zero. Why is this a useful check, and what would a large fake effect imply?
The penalty dial. Re-fit with min_1se = FALSE so cross-validation minimizes the error instead of applying the one-standard-error rule. How do λ, the pre-fit L2, and the estimate change? Explain the bias–variance trade-off you observe.
Inference under the microscope. For the classic-SCM fit, change the conformal test statistic with summary(syn, stat_func = function(x) abs(sum(x))). How does the joint-null p-value change, and why might prioritizing the average post-treatment effect (rather than the sum of absolute effects) be more or less appropriate here?

13. References

Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The Augmented Synthetic Control Method. Journal of the American Statistical Association, 116(536), 1789–1803.
Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association, 105(490), 493–505.
Abadie, A., & Gardeazabal, J. (2003). The Economic Costs of Conflict: A Case Study of the Basque Country. American Economic Review, 93(1), 113–132.
Chernozhukov, V., Wüthrich, K., & Zhu, Y. (2021). An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls. Journal of the American Statistical Association, 116(536), 1849–1864.
augsynth package and the Kansas vignette: github.com/ebenmichael/augsynth.
Companion tutorial: Augmented Synthetic Control for Multiple Countries.

AI Podcast: ASCM & the Kansas Tax Cuts

Click play to load

0:00 0:00

Staggered Synthetic Difference-in-Differences (SDID) in Stata: Gender Quotas and Women in Parliament

Sun, 07 Jun 2026 00:00:00 +0000

Abstract

Most real-world policies are not adopted on a single clock — parliamentary gender quotas, minimum-wage laws, and carbon taxes arrive in different units in different years, a staggered-adoption design where naive two-way fixed-effects difference-in-differences quietly breaks by using already-treated units as controls and placing negative weights on some effects. This tutorial extends synthetic difference-in-differences (SDID) to staggered adoption and applies it in Stata to a question in political economy: do parliamentary gender quotas raise the share of women in national parliaments? It uses the quota_example dataset distributed with the sdid package (Bhalotra, Clarke, Gomes & Venkataramani, 2023) — a balanced panel of 119 countries observed annually from 1990 to 2015 (3,094 observations), in which 9 countries adopt a quota across 7 cohorts (2000, 2002, 2003, 2005, 2010, 2012, 2013) and 110 remain never-treated. The method estimates a separate, clean SDID per cohort against the never-treated donor pool, then aggregates the cohort effects into the overall ATT with non-negative treated-period-share weights, complemented by the sdid_event event study and bootstrap, jackknife, and placebo inference. The overall ATT is +8.03 percentage points (SE 3.74, p = 0.032), robust to a log-GDP control (8.05 optimized, 8.06 projected), but the cohort effects swing from −3.5 to +21.8 points, with flat pre-adoption placebos supporting parallel synthetic trends and dynamic effects that appear immediately and persist for over a decade. The lesson is that a single headline number summarizes real heterogeneity, and that transparent, non-negative cohort weighting is essential when treatment timing is staggered.

1. Overview

In a previous tutorial, one unit — California — adopted one policy — Proposition 99 — in one year — 1989. That block design is the textbook setting for synthetic difference-in-differences (SDID). But most real policies do not arrive on a single clock. Parliamentary gender quotas, minimum-wage laws, carbon taxes, and clean-air regulations are adopted by different units in different years. This is the staggered adoption design, and it is where naive panel methods quietly break.

This tutorial extends SDID to staggered adoption and applies it in Stata to a real question in political economy: do parliamentary gender quotas raise the share of women in national parliaments? We use the quota_example dataset that ships with the sdid package — 119 countries observed annually from 1990 to 2015, in which 9 countries adopt a gender quota across 7 different cohorts (2000, 2002, 2003, 2005, 2010, 2012, and 2013).

The headline is a story about heterogeneity. The overall effect of quotas is about +8 percentage points of women in parliament, but the cohort-by-cohort effects swing from −3.5 to +21.8 points. A single number hides that range — and, as we will see, the naive two-way fixed-effects regression that most people reach for first can hide even more.

Why does staggered timing break the naive regression? (click to expand)

The workhorse for panel policy evaluation is the two-way fixed-effects (TWFE) regression — unit dummies, time dummies, and a treatment dummy. With one adoption date it estimates a clean difference-in-differences. With staggered timing and heterogeneous effects, the same regression implicitly uses already-treated units as controls for later adopters (“forbidden comparisons”). The result is a variance-weighted average of every 2×2 comparison in the panel, and some of those weights can be negative — so the estimate can even take the wrong sign (Goodman-Bacon, 2021; de Chaisemartin & D’Haultfœuille, 2020). Staggered SDID sidesteps this by estimating a separate, clean SDID effect for each adoption cohort and aggregating with transparent, non-negative weights.

graph TD
subgraph SG1["Block design — predecessor (Prop 99)"]
B1("California<br/>adopts 1989") --> BATT("one ATT")
B2("other states<br/>never treated") --> BATT
end
subgraph SG2["Staggered design — this post (gender quotas)"]
S1("cohort 2000") --> SATT("aggregate ATT")
S2("cohort 2002") --> SATT
S3("cohorts 2003 to 2013") --> SATT
SC("110 never-treated<br/>controls") -.donor pool.-> SATT
end
style SG1 fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
style SG2 fill:none,stroke:#c8d0e0,stroke-width:1px,stroke-dasharray:4 4
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
class B1,S1,S2,S3 orange
class BATT,SATT teal
class B2,SC blue

1.1 Learning objectives

By the end of this tutorial you will be able to:

Explain why staggered adoption breaks naive TWFE difference-in-differences, and how per-cohort SDID avoids the forbidden-comparison problem.
Derive the SDID estimator from first principles — unit weights $\omega$, time weights $\lambda$, and the weighted two-way fixed-effects objective — and the rule that aggregates cohort-specific effects $\hat{\tau}_a$ into one overall ATT.
Estimate the effect of gender quotas with sdid on a staggered panel, add a covariate two different ways (optimized vs projected), and choose among bootstrap, jackknife, and placebo inference.
Read an SDID event-study plot produced by sdid_event, distinguishing pre-trend placebo coefficients from post-period dynamic effects.

2. Key concepts at a glance

Each card gives a plain-language definition, a concrete example from this quota study, and an everyday analogy. Open any term that is unfamiliar.

1. ATT (average treatment effect on the treated) — the question we actually answer.

Definition. The effect of adopting a quota on the women-in-parliament share, in the countries that adopted one, averaged over their post-adoption years. It is not the effect a quota would have everywhere — only where one was actually tried.

Example. Our headline ATT is +8.0 percentage points: across the nine adopting countries, quotas raised women’s parliamentary share by about eight points relative to their no-quota counterfactual.

Analogy. Like asking “how much did the patients who took the drug improve?” — not “how much would everyone improve?” You measure only the units that were actually treated.

2. Synthetic control — a made-to-order comparison country.

Definition. A weighted blend of never-treated “donor” countries, built so its pre-adoption path mimics the treated cohort. It stands in for the unobservable counterfactual: what the cohort’s outcome would have been without a quota.

Example. The 2002 cohort’s synthetic control mixes dozens of donors (Belgium, Paraguay, Cuba, …) so that, before 2002, the blend tracks the cohort’s trend — then keeps going as the cohort would have without the law.

Analogy. A stunt double cast to match the lead actor’s build and movement — close enough that, in the shots you cannot film the star, the double stands in convincingly.

3. Unit weights (ω) — how much each donor counts.

Definition. Non-negative weights, one per donor country, summing to one, that build the synthetic control. Each cohort gets its own ω.

Example. In the 2000 cohort, 80 donors receive nonzero weight — Argentina ≈ 0.061, Guatemala ≈ 0.057, Austria ≈ 0.045 — a diffuse blend rather than one or two stand-ins.

Analogy. A recipe calling for many ingredients in small, precise amounts: no single one dominates, so the dish survives a bad batch of any one ingredient.

4. Time weights (λ) — which "before" years matter.

Definition. Non-negative weights on the pre-adoption years, summing to one, that decide which pre-periods define the baseline. They up-weight the years most like the post-period.

Example. For the 2002 cohort, λ concentrates on the late 1990s and 2001 rather than spreading evenly across 1990–2001 — the recent past is the relevant baseline.

Analogy. Forecasting tomorrow’s weather, you trust last week far more than the same date five years ago. Time weights formalize “recent and similar counts more.”

5. Adoption cohort (a) — units that switch on together.

Definition. The set of countries that first adopt a quota in the same calendar year. Staggered SDID runs one self-contained SDID per cohort, always against the never-treated controls.

Example. There are seven cohorts — 2000, 2002, 2003, 2005, 2010, 2012, 2013 — with two countries each in 2002 and 2003, and one in the rest.

Analogy. School graduating classes: the “class of 2002” and the “class of 2010” share a start date and are analyzed as groups, even though all attend the same school.

6. Staggered adoption & the forbidden comparison — why the naive regression breaks.

Definition. Staggered adoption means units are treated at different times. The hazard: a two-way fixed-effects regression can use already-treated units as controls for later adopters — a “forbidden comparison” that places negative weights on some effects and can flip the sign.

Example. When the 2012 cohort adopts, a naive TWFE quietly treats the 2002 cohort — already treated, already changed — as part of its control group. Staggered SDID never does this: each cohort is compared only to the 110 never-treated countries.

Analogy. Timing a late runner against runners who already crossed the line and slowed to a walk — your “control” is contaminated because it has already run the race.

7. Event time (relative period) — every cohort on its own clock.

Definition. Time measured relative to each cohort’s own adoption year (… −2, −1, 0, +1 …), so cohorts that adopted in different calendar years can be lined up and averaged.

Example. Event time 0 is the year 2000 for the first cohort but 2013 for the last; re-centring lets us ask “what happens three years after a quota?” across all cohorts at once.

Analogy. Comparing marathon runners by their own start gun, not the wall clock: a runner who started at 9:05 and one who started at 9:20 are both “at mile 10” measured from their own start.

8. ATT aggregation — from many cohort effects to one number.

Definition. The overall ATT is a weighted average of the cohort effects, each weighted by its share of treated unit-by-post-period observations — earlier, longer-exposed, larger cohorts count more.

Example. The seven cohort effects span −3.5 to +21.8; weighted by treated country-years they average to +8.0 (the plain unweighted mean would be ≈ 7.0).

Analogy. A course grade that weights the final exam more than a pop quiz: the cohorts you observe for longer carry more of the final mark.

9. Pre-trend placebo test — the assumption you can see.

Definition. Event-study coefficients for the pre-adoption periods. If treated and synthetic-control countries moved in parallel before treatment, these sit near zero — a falsification check.

Example. For the 2002 cohort, all twelve pre-period placebos fall in [−0.2, +0.8] points — flat, so we cannot reject parallel synthetic trends.

Analogy. Checking a scale by weighing nothing first: if it does not read zero when empty, you distrust every later reading. Flat placebos are that “reads zero when empty” check.

10. Bootstrap, jackknife, placebo — three rulers for uncertainty.

Definition. Three ways to attach a standard error to the ATT. With many treated units all three are available; they share one point estimate but report different spread.

Example. On the two-cohort subsample the ATT is 10.3 for all three, but the SE is 4.7 (bootstrap), 6.0 (jackknife, most conservative), and 2.3 (placebo, tightest).

Analogy. Measuring a table with a tape, a folding ruler, and a laser: they agree on the length but disagree on the error bars — the cautious carpenter reports the widest.

3. The data: gender quotas across 119 countries

We use quota_example.dta, the balanced panel from Bhalotra, Clarke, Gomes & Venkataramani (2023) distributed with the sdid package. The outcome is the percentage of seats held by women in the national parliament; the treatment is the adoption of a reserved-seat gender quota; the covariate is log GDP per capita.

webuse set www.damianclarke.net/stata/
webuse quota_example, clear
label variable quota "Parliamentary gender quota"
xtset country year
codebook country year quota womparl lngdp, compact

Variable Obs Unique Mean Min Max Label
----------------------------------------------------------------------------
country 3094 119 . . . Country
year 3094 26 2002.5 1990 2015 Year
quota 3094 2 .0303814 0 1 =1 if country has a gender quota
womparl 3094 449 14.96531 0 63.8 Women in parliament
lngdp 2990 2956 9.154291 5.8701 11.61789 log(GDP)
----------------------------------------------------------------------------

The panel is balanced: 119 countries times 26 years equals 3,094 observations, with no gaps in the outcome or treatment (lngdp has 104 missing values, which will matter only when we add the covariate). The treatment indicator quota equals one for just 3% of observations, a reminder that treated country-years are scarce. Crucially, quota is absorbing — once a country adopts a quota it stays treated — which SDID requires.

Variable	Role	Symbol	Description
`country`	unit	$i$	119 countries (9 ever-treated, 110 never-treated)
`year`	time	$t$	1990–2015 (26 years)
`womparl`	outcome	$Y_{it}$	% women in the national parliament
`quota`	treatment	$W_{it}$	1 once a country has a quota, 0 before / never
`lngdp`	covariate	$X_{it}$	log GDP per capita

The estimand. Our target is the average treatment effect on the treated (ATT): the effect of adopting a quota on the women-in-parliament share in the countries that adopted one, averaged over their post-adoption years. Formally,

$$ \tau = \frac{1}{N_{tr}\, T_{post}} \sum_{i:\, W_i = 1}\ \sum_{t > T_{pre}} \left[\, Y_{it}(1) - Y_{it}(0) \,\right] $$

In words: for every treated country and every post-adoption year, take the gap between the share of women with a quota, $Y_{it}(1)$, and the share that would have occurred without one, $Y_{it}(0)$ — then average. The first term is observed; the second is the counterfactual that the synthetic control must impute, because we never see a quota-adopting country in the parallel world where it abstained.

An observational, not experimental, setting. Quotas are not randomly assigned. Countries that adopt them early may differ systematically — they may be wealthier, more democratic, or already on a rising trajectory of women’s representation. That is exactly why we need a method that builds a credible counterfactual from comparison countries rather than assuming a simple before/after change would have held. Identification rests on assumptions we will keep visible: that treated and synthetic-control countries share a common (synthetic) trend absent treatment, no anticipation of the quota, no spillovers across countries, and that adoption timing is not itself driven by the outcome’s future path.

3.1 The staggered structure

Before modelling, let us see the timing directly. The adoption year is the first year a country is treated; we tabulate the cohorts.

bysort country (year): egen firsttreat = min(cond(quota==1, year, .))
preserve
keep country firsttreat
duplicates drop
tab firsttreat, missing
restore

 firsttreat | Freq. Percent Cum.
------------+-----------------------------------
2000 | 1 0.84 0.84
2002 | 2 1.68 2.52
2003 | 2 1.68 4.20
2005 | 1 0.84 5.04
2010 | 1 0.84 5.88
2012 | 1 0.84 6.72
2013 | 1 0.84 7.56
. | 110 92.44 100.00
------------+-----------------------------------
Total | 119 100.00

Nine countries adopt a quota, spread across seven cohorts; the 2002 and 2003 cohorts contain two countries each, the rest one. The remaining 110 countries are never treated — they form the donor pool from which every cohort’s synthetic control is built. This staircase of adoption dates is the defining feature of a staggered design, and the reason a single “post” dummy is too blunt.

4. Exploratory analysis with `panelview`

A staggered design is best understood by looking at it. The panelview command (Xu & Hua) draws two pictures we need: a heatmap of who is treated when, and the raw outcome trajectories colored by treatment status.

ssc install panelview, replace
panelview womparl quota, i(country) t(year) type(treat) bytiming
panelview womparl quota, i(country) t(year) type(outcome)

The treatment heatmap (type(treat), sorted with bytiming) makes the staggered structure unmistakable: the dark treated cells appear in the top-right corner as a staircase, each step a different cohort switching on between 2000 and 2013, against a sea of never-treated controls. This is the visual opposite of a block design, where every treated cell would switch on in the same column.

The outcome plot (type(outcome)) overlays all 119 women-in-parliament series, with the 9 treated countries in orange. Several treated countries start near the bottom of the distribution and climb steeply after their adoption year — a hint of a positive effect — but the climbs begin at different times, and a few treated countries barely move. No single “treated average” line could summarize this; we need cohort-specific counterfactuals.

collapse (mean) womparl, by(evertreat year)
* ... reshape and plot ever- vs never-adopting means ...

Collapsing to group means tells a cautionary tale. The ever-adopting countries (orange) start the 1990s below the never-adopting countries (about 4% vs 10% women in parliament) and end above them by 2015 (about 23% vs 22%). A naive eyeball difference-in-differences on these two lines would be badly confounded: the groups began at different levels and the “treated” line aggregates countries that switched on in seven different years. The raw means motivate the machinery to come — we must compare each cohort to a tailored synthetic control, not to the grand average.

5. Synthetic difference-in-differences from first principles

Before tackling staggered timing, fix ideas with a single cohort. SDID (Arkhangelsky et al., 2021) is a weighted two-way fixed-effects regression. It chooses an ATT, a constant, unit fixed effects, and time fixed effects to minimize a weighted sum of squared residuals:

$$ \left(\hat{\tau}, \hat{\mu}, \hat{\alpha}, \hat{\beta}\right) = \arg\min_{\tau,\mu,\alpha,\beta} \sum_{i=1}^{N} \sum_{t=1}^{T} \left(Y_{it} - \mu - \alpha_i - \beta_t - W_{it}\,\tau\right)^{2}\, \hat{\omega}_i\, \hat{\lambda}_t $$

In words: run a difference-in-differences regression, but weight each observation by a unit weight $\hat{\omega}_i$ times a time weight $\hat{\lambda}_t$. Here $\alpha_i$ is a country fixed effect, $\beta_t$ a year fixed effect, $W_{it}$ the treatment dummy, and $\tau$ the ATT we want. Set all weights equal and you recover ordinary DiD; the weights are what make SDID special. They are not free parameters — each solves its own optimization.

The unit weights are chosen so that a weighted blend of control countries tracks the treated cohort across the pre-period:

$$ \hat{\omega} = \arg\min_{\omega_0,\, \omega \ge 0} \sum_{t=1}^{T_{pre}} \left(\omega_0 + \sum_{i=1}^{N_{co}} \omega_i\, Y_{it} - \frac{1}{N_{tr}} \sum_{i=1}^{N_{tr}} Y_{it}\right)^{2} + \zeta^{2}\, T_{pre}\, \lVert \omega \rVert^{2} $$

The bracketed term asks the synthetic control $\sum_i \omega_i Y_{it}$ (plus an intercept $\omega_0$) to match the treated average in every pre-adoption year. The intercept $\omega_0$ is the SDID twist: it lets the synthetic match the treated trend without matching its level, because any constant level gap is later absorbed by the unit fixed effect $\alpha_i$. The final term is a ridge penalty with regularization strength $\zeta$; it spreads weight across many donors instead of concentrating it on a few, which stabilizes the estimate. (Synthetic control, by contrast, drops $\omega_0$ and the penalty and must match the level too.)

The time weights are the mirror image — they pick the pre-period years that best predict each control country’s post-period average:

$$ \hat{\lambda} = \arg\min_{\lambda_0,\, \lambda \ge 0} \sum_{i=1}^{N_{co}} \left(\lambda_0 + \sum_{t=1}^{T_{pre}} \lambda_t\, Y_{it} - \frac{1}{T_{post}} \sum_{t=T_{pre}+1}^{T} Y_{it}\right)^{2} + \zeta_{\lambda}^{2}\, N_{co}\, \lVert \lambda \rVert^{2} $$

Years that look most like the post-period get the most weight, so the “before” comparison is built from the most relevant history rather than a flat average over possibly-irrelevant early years. The two weighting schemes together are what distinguish SDID from its cousins, as the table summarizes.

Method	Unit weights $\omega$	Time weights $\lambda$	Unit FE $\alpha_i$	Must match
DiD	uniform	uniform	yes	trend on all controls
Synthetic control	optimized	uniform	no	level and trend
SDID	optimized	optimized	yes	trend (level gap allowed)

6. The staggered extension: per-cohort effects and their aggregation

Staggered SDID is a disarmingly simple idea: do the single-cohort analysis once per adoption cohort, then average. For each cohort $a$, take only that cohort’s treated countries plus the pure never-treated controls, solve the SDID problem above on that sub-panel to get its own $\hat{\omega}_a$, $\hat{\lambda}_a$, and cohort effect $\hat{\tau}_a$. Because each cohort is compared only to never-treated controls, an already-treated unit is never used as a control for a later adopter — precisely the contamination that breaks naive TWFE.

graph LR
POOL("110 never-treated<br/>controls (donor pool)")
C1("Cohort 2000<br/>+ controls")
C2("Cohort 2002<br/>+ controls")
CD("Cohorts 2003…2013<br/>+ controls")
T1("SDID &rarr; &tau;<sub>2000</sub> = 8.4")
T2("SDID &rarr; &tau;<sub>2002</sub> = 7.0")
TD("SDID &rarr; &tau;<sub>a</sub><br/>(&minus;3.5 … +21.8)")
ATT("Aggregate ATT = 8.0<br/>weighted by treated periods")
POOL --> C1 --> T1 --> ATT
POOL --> C2 --> T2 --> ATT
POOL --> CD --> TD --> ATT
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class POOL,T1,T2,TD blue
class C1,C2,CD orange
class ATT teal

The overall ATT aggregates the cohort effects with non-negative weights equal to each cohort’s share of treated unit-by-post-period observations:

$$ \widehat{ATT} = \sum_{a \in \mathcal{A}} \frac{N_{tr}^{a}\, T_{post}^{a}}{\sum_{b \in \mathcal{A}} N_{tr}^{b}\, T_{post}^{b}}\ \hat{\tau}_a $$

In words: a cohort counts in proportion to how many treated country-years it contributes. The 2000 cohort, treated for 16 years (2000–2015), carries more weight than the 2013 cohort, treated for only 3. This is the staggered generalization of single-cohort SDID, and — unlike TWFE — every weight is positive and interpretable. (When each cohort has one treated unit, this reduces to the post-period share $T_{post}^{a}/T_{post}$ from Clarke et al., 2024.)

7. Estimation in Stata

One command does the whole staggered procedure. We request bootstrap inference and a fixed seed for reproducibility.

sdid womparl country year quota, vce(bootstrap) seed(1213)
matrix list e(tau)

Synthetic Difference-in-Differences Estimator
-----------------------------------------------------------------------------
womparl | ATT Std. Err. t P>|t| [95% Conf. Interval]
-------------+---------------------------------------------------------------
quota | 8.03410 3.74040 2.15 0.032 0.70305 15.36516
-----------------------------------------------------------------------------

The overall ATT is +8.03 percentage points (SE 3.74, $t=2.15$, $p=0.032$), with a 95% confidence interval of [0.70, 15.37] that excludes zero. Substantively: adopting a parliamentary gender quota raises the share of women in parliament by about eight percentage points in the adopting countries — a large effect against a sample mean of 15%, and statistically distinguishable from no effect at the 5% level.

The single number, though, is the average of a very heterogeneous set of cohort effects, returned in e(tau):

T[7,3]
Tau Std.Err. Time
r1 8.3888685 .68278345 2000
r2 6.9677465 .64102999 2002
r3 13.952256 9.1289943 2003
r4 -3.4505431 .75603453 2005
r5 2.7490355 .44799502 2010
r6 21.762716 .91589982 2012
r7 -.82032354 .83151601 2013

The cohort effects span an enormous range: from −3.5 points (2005 cohort) to +21.8 points (2012 cohort), with the 2003 cohort essentially uninformative (SE 9.13, a confidence interval that runs from −4 to +32). The teal line marks the aggregate ATT of 8.0. Notice that this aggregate is not the simple average of the seven cohort effects — that average would be about 7.0. It is the treated-period-weighted average from the aggregation formula, which up-weights the earlier, longer-exposed 2000, 2002, and 2003 cohorts. The lesson of the figure is that “+8 points on average” is a summary of real heterogeneity, not a universal constant; some quotas were transformative, others did nothing measurable.

To see the synthetic-control machinery underneath one cohort, the figure below plots the 2002 cohort against its synthetic control. Because SDID matches the pre-period trend and lets the unit fixed effect absorb the level gap, we anchor the synthetic to the treated cohort by its $\lambda$-weighted pre-period gap so the two align before adoption.

The treated 2002 cohort (orange) and its anchored synthetic control (blue dashed) track each other closely before 2002 — the synthetic was built precisely to do so — and then diverge: the treated cohort climbs to roughly 15% women in parliament while the synthetic counterfactual reaches only about 9–10%. That post-2002 gap is the cohort effect, about +7 points, matching $\hat{\tau}_{2002}=6.97$ from e(tau).

Which pre-period years anchor that comparison? The time weights $\hat{\lambda}_t$ for the 2002 cohort do not spread evenly over 1990–2001 — they concentrate on the years just before adoption.

The bars show SDID’s baseline for the 2002 cohort leaning on the late 1990s and 2001 — the pre-adoption years whose level most resembles the post-adoption period — rather than weighting all twelve pre-years equally as a plain difference-in-differences would. This is the time-weighting half of SDID at work: it builds the “before” from the most relevant history, which is also the baseline the event study below measures against.

8. Adding a covariate: optimized vs projected

Does the quota effect simply reflect economic development — richer countries both grow GDP and elect more women? We can condition on log GDP per capita. The sdid command offers two routes, and SDID needs a balanced panel, so we first drop the country-years with missing lngdp.

drop if missing(lngdp)
sdid womparl country year quota, vce(bootstrap) seed(2022) covariates(lngdp, optimized)
sdid womparl country year quota, vce(bootstrap) seed(1213) covariates(lngdp, projected)

SDID + lngdp (optimized) ATT = 8.0515 SE = 3.0466
SDID + lngdp (projected) ATT = 8.0593 SE = 3.1191

The two methods differ in how they estimate the covariate’s coefficient. The optimized method (Arkhangelsky et al., 2021) folds the covariate adjustment into the SDID optimization itself, estimating it jointly with the weights — flexible but computationally heavy. The projected method (Kranz, 2022) instead regresses the outcome on the covariate among the untreated observations first, then runs SDID on the residuals — much faster and numerically more stable. Reassuringly, here they agree to the second decimal: 8.05 and 8.06, essentially unchanged from the no-covariate estimate of 8.03. Controlling for income does not explain away the quota effect; the result is robust to the most obvious confounder.

9. The event study with `sdid_event`

A single ATT — even per cohort — cannot tell us when the effect appears, or whether treated and control countries were already diverging before the quota. For that we need an event study: the treatment effect traced out by years relative to adoption. The modern sdid_event command (Ciccia, Clarke & Pailañir, 2024) computes exactly this for SDID, including pre-period placebo estimates that serve as a parallel-trends test.

The dynamic effect at event time $\ell$ is the treated-minus-synthetic gap in that period, net of the same gap at baseline, where — characteristically for SDID — the baseline is the $\lambda$-weighted pre-period average rather than a single “year −1”:

$$ \delta_{\ell} = \left(\bar{Y}_{\ell}^{,tr} - \bar{Y}_{\ell}^{,co}\right) - \left(\bar{Y}_{base}^{,tr} - \bar{Y}_{base}^{,co}\right), \qquad \bar{Y}_{base}^{,g} = \sum_{t=1}^{T_{pre}} \hat{\lambda}_t\, \bar{Y}_t^{,g} $$

sdid_event handles the full staggered panel directly, returning a cohort-aggregated ATT plus dynamic effects. To read the dynamics transparently we focus the plot on the 2002 cohort — the package authors’ own worked example — which gives a clean event-time axis; the full-panel call confirms the same aggregated ATT (≈ 8.06).

ssc install sdid_event, replace
* full staggered panel: aggregated ATT + cohort-aggregated dynamic effects
sdid_event womparl country year quota, vce(bootstrap) brep(100) effects(8) placebo(5) covariates(lngdp)
* clean event study on the 2002 cohort, with all placebos
keep if quotaYear==2002 | quotaYear==.
sdid_event womparl country year quota, vce(placebo) brep(100) placebo(all) covariates(lngdp)

 | Estimate SE LB CI UB CI Switchers
-------------+------------------------------------------------------
ATT | 6.853472 3.372744 .2428928 13.46405 2
Effect_1 | 4.086404 1.191517 1.75103 6.421778 2
Effect_2 | 9.164442 1.522799 6.179756 12.14913 2
Effect_3 | 7.938504 2.182572 3.660663 12.21635 2
... |
Placebo_1 | -.218417 .470226 -1.14006 .703227 2
Placebo_2 | .242148 .884557 -1.491584 1.975880 2
... |

This plot rewards careful reading, and there are three things to look for.

First, the baseline is $\lambda$-weighted, not “the year before.” Unlike a textbook event study that normalizes to $t=-1$, SDID measures everything against the optimally weighted pre-period average. That is why the zero line is a weighted baseline; do not read it as the single pre-adoption year.

Second, the points to the left of zero are placebo tests. Every pre-adoption coefficient (Placebo_1 through Placebo_12, event times −1 to −12) sits within a whisker of zero — ranging only from about −0.2 to +0.8. Because the treated cohort and its synthetic control moved in parallel before 2002, we cannot reject that the parallel-(synthetic-)trends assumption holds. This is the identifying assumption made visible and, here, survived.

Third, the points to the right of zero are the dynamic ATT. The effect appears immediately at adoption (Effect_1 = +4.1 points at event time 0), roughly doubles within a year or two (Effect_2 = +9.2), and then settles in the +6 to +9 range for over a decade. Quotas do not just shift the level once; they sustain a higher share of women in parliament. Aggregated by the same treated-period logic as before, these dynamic effects reproduce the cohort’s overall ATT of about +7 points — but the plot shows the shape the single number conceals.

10. Inference: bootstrap, jackknife, and placebo

With one treated unit (California), the previous tutorial could only use placebo/permutation inference. With nine treated units here, all three of sdid’s variance estimators are on the table. To keep the comparison clean — jackknife needs more than one treated unit per adoption period — we follow Clarke et al. (2024) and restrict to the two-country 2002 and 2003 cohorts by dropping the five single-country cohorts.

graph TD
Q1{"How many<br/>treated units?"}
Q1 -->|"One (e.g. California)"| PL1("Placebo only<br/>jackknife undefined")
Q1 -->|"Many (e.g. 9 quota adopters)"| Q2{"More controls than treated?<br/>no singleton cohorts?"}
Q2 -->|"Yes"| ALL("All three available")
Q2 -->|"Singleton cohorts"| PL2("Placebo / bootstrap<br/>jackknife drops out")
ALL --> BOOT("bootstrap<br/>SE 4.7 (default)")
ALL --> JACK("jackknife<br/>SE 6.0 (most conservative)")
ALL --> PLAC("placebo<br/>SE 2.3 (homoskedastic)")
classDef sty_Q1 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q1 sty_Q1
classDef sty_Q2 fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q2 sty_Q2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
class PL1,PL2 orange
class ALL teal
class BOOT,JACK,PLAC blue

drop if inlist(country,"Algeria","Kenya","Samoa","Swaziland","Tanzania")
sdid womparl country year quota, vce(bootstrap) seed(1213)
sdid womparl country year quota, vce(placebo) seed(1213)
sdid womparl country year quota, vce(jackknife)

method att se ci_l ci_u
bootstrap 10.33066 4.7291 1.0618 19.5995
placebo 10.33066 2.3404 5.7436 14.9178
jackknife 10.33066 6.0056 -1.4401 22.1014

The point estimate is identical across all three methods — 10.33 points on this subsample — because the inference procedure changes only the standard error, never the estimate. But the standard errors differ by a factor of nearly three: jackknife is the most conservative (SE 6.01, a confidence interval that crosses zero), placebo is the tightest (SE 2.34) but rests on a homoskedasticity assumption and requires more controls than treated units, and bootstrap sits in between (SE 4.73) and is the default. The practical takeaway: with only a handful of treated units, report the bootstrap as your headline but cross-check it — a result that is “significant” under placebo but not under jackknife deserves caution. (The subsample ATT of 10.3 is larger than the full-sample 8.0 because dropping the five single-country cohorts discards the negative 2005 and 2013 effects.)

11. Robustness and discussion

Three caveats keep the result honest. Effect concentration: the +8 aggregate leans heavily on a few cohorts — the 2012 cohort alone contributes a +21.8 effect, and the early 2000/2002/2003 cohorts carry most of the aggregation weight. Drop the 2012 cohort and the average falls noticeably. Fragile counterfactuals: with only 110 controls and as few as one treated country per cohort, some synthetic controls are imprecise — the 2003 cohort’s standard error of 9.13 is the tell. Identifying assumptions: SDID still requires no anticipation, an absorbing treatment, no cross-country spillovers, and that quota timing is not itself a response to the outcome’s trajectory; the flat event-study placebos support, but cannot prove, the parallel-trends part. Finally, quota_example is a teaching subset of Bhalotra et al. (2023); these numbers illustrate the method, not a final verdict on quota policy.

12. Summary and key takeaways

Method. Staggered SDID estimates a separate, clean synthetic difference-in-differences for each adoption cohort — comparing it only to never-treated controls — and aggregates the cohort effects $\hat{\tau}_a$ with non-negative, treated-period-share weights. This avoids the negative-weighting trap that contaminates naive two-way fixed-effects DiD under staggered timing.
Result. Gender quotas raise the share of women in parliament by an overall ATT of +8.0 percentage points (SE 3.74, $p=0.032$), robust to a log-GDP control (8.05 optimized, 8.06 projected). Cohort effects range widely, from −3.5 to +21.8 points — heterogeneity the single number hides.
Event study. The sdid_event plot shows pre-adoption placebo coefficients near zero (parallel synthetic trends) and post-adoption effects that appear immediately and persist for over a decade — the dynamics behind the average.
Inference. With nine treated units, bootstrap, jackknife, and placebo are all available; they share one point estimate (10.3 on the two-cohort illustration) but report standard errors of 4.7, 6.0, and 2.3. Jackknife is the most conservative.
Bridge. The block design (Proposition 99, the previous tutorial) and the staggered design here are two faces of one estimator — the staggered version is just single-cohort SDID, done once per cohort and averaged.

13. Exercises

Re-aggregate by hand. Pull e(tau) and each cohort’s treated unit-count and post-period length. Verify that the treated-period-weighted average of the seven $\hat{\tau}_a$ reproduces the overall ATT of 8.03, and show that it differs from the unweighted mean (≈ 7.0). Which cohorts move the aggregate the most?
Inference sensitivity. Re-run the full nine-country sample with vce(bootstrap) and then vce(placebo) at reps(500). How much do the standard error and confidence interval move, and which would you report given only nine treated units?
Drop the outlier cohort. Re-estimate the overall ATT excluding the 2012 cohort (the +21.8 outlier). How far does the aggregate fall, and what does that tell you about how concentrated the average effect is?

14. References

Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic Difference-in-Differences. American Economic Review, 111(12), 4088–4118.
Clarke, D., Pailañir, D., Athey, S., & Imbens, G. (2024). On Synthetic Difference-in-Differences and Related Estimation Methods in Stata. The Stata Journal, 24(4). Package: ssc install sdid.
Ciccia, D. (2024). A Short Note on Event-Study Synthetic Difference-in-Differences Estimators. Package: ssc install sdid_event.
Bhalotra, S., Clarke, D., Gomes, J. F., & Venkataramani, A. (2023). Maternal Mortality and Women’s Political Power. Journal of the European Economic Association. (Source of the quota_example data.)
Goodman-Bacon, A. (2021). Difference-in-Differences with Variation in Treatment Timing. Journal of Econometrics, 225(2), 254–277.
de Chaisemartin, C., & D’Haultfœuille, X. (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. American Economic Review, 110(9), 2964–2996.
Xu, Y., & Hua, L. panelView: Visualizing Panel Data. Package: ssc install panelview.

Related tutorials on this site: Synthetic Difference-in-Differences (the block design) · Difference-in-Differences.

15. Acknowledgments

This tutorial uses the sdid command (Clarke, Pailañir, Athey & Imbens), the sdid_event command (Ciccia, Clarke & Pailañir), and panelview (Xu & Hua). The data, quota_example, is distributed with sdid and draws on Bhalotra, Clarke, Gomes & Venkataramani (2023). All estimates were produced by the companion analysis.do and verified against Clarke et al. (2024). AI tools (Claude Code) assisted with drafting and figure preparation; all code was executed and every number checked by the author.

AI Podcast: Staggered Synthetic Difference-in-Differences

Click play to load

0:00 0:00

Synthetic Difference-in-Differences (SDID) in Stata: Re-evaluating California's Proposition 99

Sun, 07 Jun 2026 00:00:00 +0000

Abstract

Comparative case studies—where a single large unit adopts a policy and the analyst must recover its causal effect without ever observing the untreated counterfactual—are a recurring challenge in policy evaluation, and the canonical example is California’s Proposition 99, the 1988 ballot measure that raised the cigarette excise tax by 25 cents a pack and funded an anti-smoking campaign. This tutorial introduces and derives synthetic difference-in-differences (SDID) and applies it to re-evaluate Proposition 99, contrasting it with classic difference-in-differences (DiD) and synthetic control (SC). The data are the canonical strongly balanced panel distributed with the sdid package (originally from Abadie, Diamond, and Hainmueller 2010): 39 US states observed annually from 1970 to 2000—1,209 observations, of which only 12 are treated—with annual cigarette sales in packs per capita as the sole outcome and California as the single treated unit from 1989. The methods estimate the average treatment effect on the treated by writing DiD, SC, and SDID as one weighted two-way fixed-effects regression in Stata, using the sdid command (Clarke et al. 2024) and cross-checking SC against synth2. All three estimators agree the policy reduced smoking but disagree on magnitude: the 2×2 DiD gives −27.35 packs per capita, synthetic control −19.48 (pre-period RMSE 1.66, R² 0.98, leaning on Utah, Montana, and Nevada), and SDID −15.60—roughly a 20% reduction—with SDID’s time weights concentrated entirely on 1986–1988. With one treated unit, placebo inference is the only valid procedure: the placebo standard error is 9.88 (95% CI [−35.0, 3.8], including zero) while the permutation test ranks California’s effect extreme (p = 0.026). The implication is that a single sdid command unifies all three estimators, and SDID is the preferred single number because, by allowing a constant level gap and up-weighting the informative late-1980s years, it relies least on the exact parallel-trends assumption the others lean on hardest.

1. Overview

In November 1988 California voters passed Proposition 99, which raised the cigarette excise tax by 25 cents a pack and funded a large anti-smoking campaign. Did it actually reduce smoking? This is the textbook question of comparative case study research: a single, large unit (California) adopts a policy, and we want the causal effect even though we can never observe the California that did not pass Proposition 99.

This tutorial builds up to synthetic difference-in-differences (SDID), the estimator of Arkhangelsky, Athey, Hsiao, Imbens, and Wager (2021), and applies it with the sdid command of Clarke, Pailañir, Athey, and Imbens (2024). SDID is best understood as the marriage of two older ideas:

Difference-in-differences (DiD) — compare California’s before/after change to the before/after change of all control states.
Synthetic control (SC) — build a “synthetic California” as a weighted average of control states that tracks California before the policy.

SDID keeps the best of both: like SC it chooses unit weights so the comparison group resembles California, and like DiD it allows a constant level gap between California and its comparison group (a unit fixed effect). It then adds one more ingredient SC lacks — time weights that emphasize the pre-policy years most predictive of the post-policy period.

A second theme runs through the whole tutorial, and it is worth stating up front. As Clarke et al. (2024) put it, along with SDID, the sdid command implements standard synthetic control and difference-in-differences in an identical framework, allowing estimation, inference, and graphical output in a computationally efficient way. We will show this concretely: the same command, changing only one option, reproduces the raw difference-in-differences and the classic synthetic control — and we cross-check the latter against the dedicated synth2 command.

Learning objectives

By the end you will be able to:

Derive the SDID estimator as a weighted two-way fixed-effects regression and read its unit-weight and time-weight optimization problems.
Distinguish SDID from the original DiD and SC — conceptually (which weights, which fixed effects) and quantitatively (on the same data).
Estimate the effect of Proposition 99 with sdid, and reproduce DiD and SC from the very same command.
Compare the SDID synthetic against a classical synthetic control fit with synth2.
Conduct valid inference when there is a single treated unit, using placebo (permutation) methods — and recognize when other procedures (bootstrap, jackknife) do and do not apply.

What we are estimating

Throughout, the estimand is the average treatment effect on the treated (ATT) — the effect of Proposition 99 on California, over the post-1988 period:

$$ \tau = \frac{1}{N_{tr}\, T_{post}} \sum_{i:\, W_i = 1}\ \sum_{t > T_{pre}} \left[\, Y_{it}(1) - Y_{it}(0) \,\right] $$

In words: average, over treated units and post-treatment years, the difference between the outcome with the policy, $Y_{it}(1)$, and the outcome that would have occurred without it, $Y_{it}(0)$. Here there is exactly one treated unit ($N_{tr} = 1$, California), and $Y_{it}(0)$ is never observed after 1988 — every method in this tutorial is a different way of imputing that missing counterfactual. Because California was not randomly assigned to treatment, this is an observational design: identification rests on assumptions (a stable comparison group, no large contemporaneous shocks unique to California) rather than on randomization.

Key concepts at a glance

Counterfactual — what California's smoking would have been without Proposition 99.

Every estimator here is a recipe for the dashed line “California if the policy had never passed.” DiD, SC, and SDID disagree only about how to build it.

Unit weights (ω) — how much each control state counts toward the synthetic California.

DiD gives every control the same weight ($1/N_{co}$). SC and SDID instead pick weights so the weighted controls reproduce California’s pre-policy outcome path. SC concentrates weight on a handful of states; SDID spreads it more widely.

Time weights (λ) — how much each pre-policy year counts.

This is SDID’s signature. Rather than treat every pre-1989 year equally, SDID up-weights the pre-period years that best predict the post-period — here, 1986–1988. SC and DiD have no time weights.

Unit fixed effects (α) — a constant level gap between California and its synthetic comparison.

DiD and SDID include them, so the comparison group only needs to move in parallel with California, not sit at the same level. Classic SC omits them and instead tries to match California’s level outright.

Placebo inference — how we get a standard error with only one treated unit.

We pretend, one at a time, that a control state was “treated,” re-estimate the effect, and build the distribution of these placebo effects. If California’s real effect is extreme relative to that distribution, it is unlikely to be noise.

2. The Proposition 99 case study

We use the canonical dataset distributed with the sdid package (originally from Abadie, Diamond, and Hainmueller 2010, and used by Arkhangelsky et al. 2021). It is a strongly balanced panel: 39 US states observed annually from 1970 to 2000, with one outcome — annual cigarette sales in packs per capita. California is the single treated unit; the policy bites from 1989 onward. The remaining 38 states (which did not pass comparable large-scale tobacco programs in this window) form the donor pool.

Variable	Role	Description
`state`	unit id	39 US states (California + 38 controls)
`year`	time id	1970–2000 (19 pre-, 12 post-treatment years)
`packspercapita`	outcome $Y_{it}$	annual cigarette pack sales per capita
`treated`	treatment $W_{it}$	1 for California in 1989–2000, else 0

One feature matters for a fair comparison: this panel contains only the outcome — no income, price, or demographic covariates. That is deliberate here. It means synthetic control and SDID see exactly the same information (California’s and the donors’ pre-period smoking paths), so any difference in their answers comes from the estimator, not from a different set of predictors.

graph LR
POOL("<b>Donor pool</b><br/>38 control states<br/>Utah, Nevada, Montana, …")
CA("<b>California</b><br/>treated 1989")
SYN("<b>Synthetic California</b><br/>counterfactual Y(0)")
POOL -->|weighted average ω| SYN
CA -->|compare after 1989| SYN
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class POOL blue
class CA orange
class SYN teal

Let us first look at the data with no model at all — California against the simple average of the 38 control states.

use prop99_example.dta, clear
describe
encode state, gen(id)
xtset id year

Contains data from prop99_example.dta
Observations: 1,209
Variables: 4
-------------------------------------------------------------------------------
Variable Storage Display Value
name type format label Variable label
-------------------------------------------------------------------------------
state str14 %14s State
year int %8.0g Year
packspercapita float %9.0g PacksPerCapita
treated byte %8.0g
-------------------------------------------------------------------------------
Panel variable: id (strongly balanced)
Time variable: year, 1970 to 2000
Delta: 1 unit

The panel is strongly balanced (no gaps), which every method below requires. The figure compares California to the raw control average.

California (orange) already smoked less than the average control state and was declining throughout the 1980s. After 1989 the gap widens visibly. But two problems jump out: California sits on a different level than the average donor, and it was already on a different trend before 1989. A credible estimate must deal with both — the job of the three estimators below.

3. Three estimators, one equation

The cleanest way to see how DiD, SC, and SDID relate is to write them all as the same weighted two-way fixed-effects (TWFE) regression and change only the weights. This is the unifying view of Arkhangelsky et al. (2021).

Synthetic difference-in-differences

SDID solves a weighted TWFE regression:

$$ \left(\hat{\tau}^{sdid}, \hat{\mu}, \hat{\alpha}, \hat{\beta}\right) = \underset{\tau,\mu,\alpha,\beta}{\arg\min} \sum_{i=1}^{N} \sum_{t=1}^{T} \left(Y_{it} - \mu - \alpha_i - \beta_t - W_{it}\,\tau\right)^{2} \hat{\omega}_i^{sdid}\ \hat{\lambda}_t^{sdid} $$

Reading the symbols against the Stata variables: $Y_{it}$ is packspercapita; $W_{it}$ is treated; $\alpha_i$ is a state fixed effect (one per state); $\beta_t$ is a year fixed effect (one per year); and $\tau$ is the ATT we want. The two extra terms are the difference from ordinary regression: $\hat{\omega}_i^{sdid}$ is a unit weight (how much state $i$ counts) and $\hat{\lambda}_t^{sdid}$ is a time weight (how much year $t$ counts). Set those weights to special values and you recover the older estimators.

The original difference-in-differences

DiD is the special case with no weighting — every unit and every year counts equally:

$$ \left(\hat{\tau}^{did}, \hat{\mu}, \hat{\alpha}, \hat{\beta}\right) = \underset{\tau,\mu,\alpha,\beta}{\arg\min} \sum_{i=1}^{N} \sum_{t=1}^{T} \left(Y_{it} - \mu - \alpha_i - \beta_t - W_{it}\,\tau\right)^{2} $$

This is just two-way fixed-effects regression. Its credibility hinges entirely on parallel trends: the assumption that, absent the policy, California would have moved in lockstep with the average control state. The raw-trends figure already makes that assumption look shaky.

The original synthetic control

SC keeps unit weights but drops the time weights and the unit fixed effects $\alpha_i$:

$$ \left(\hat{\tau}^{sc}, \hat{\mu}, \hat{\beta}\right) = \underset{\tau,\mu,\beta}{\arg\min} \sum_{i=1}^{N} \sum_{t=1}^{T} \left(Y_{it} - \mu - \beta_t - W_{it}\,\tau\right)^{2} \hat{\omega}_i^{sc} $$

Without $\alpha_i$, SC cannot absorb a level gap, so it must build a synthetic California that matches California’s pre-period outcomes in both level and trend. That is a demanding requirement — and the reason SC sometimes cannot find a good fit.

graph TD
OBJ("<b>One weighted two-way<br/>fixed-effects regression</b><br/><i>min Σ (Y − μ − α − β − Wτ)² · ω · λ</i>")
OBJ --> DID("<b>DiD</b><br/>ω uniform, λ uniform<br/>α included<br/><i>parallel trends on all controls</i>")
OBJ --> SC("<b>Synthetic control</b><br/>ω optimized, no λ<br/><b>no</b> unit FE α<br/><i>match level AND trend</i>")
OBJ --> SDID("<b>SDID</b><br/>ω optimized + λ optimized<br/>α included<br/><i>match trend, allow level gap</i>")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class OBJ anchor
class DID orange
class SC blue
class SDID teal

How the weights are chosen

The unit weights make the weighted controls track California’s pre-period path, with a small ridge penalty for stability:

$$ \hat{\omega}^{sdid} = \underset{\omega \in \Omega}{\arg\min} \sum_{t=1}^{T_{pre}} \left(\omega_0 + \sum_{i=1}^{N_{co}} \omega_i\, Y_{it} - \frac{1}{N_{tr}} \sum_{i=N_{co}+1}^{N} Y_{it}\right)^{2} + \zeta^{2}\, T_{pre}\, \lVert \omega \rVert_2^{2} $$

In words: choose nonnegative weights summing to one (the set $\Omega$) so the weighted control outcome, plus an intercept $\omega_0$, comes as close as possible to the treated outcome in every pre-treatment year. The intercept $\omega_0$ is what lets SDID match California’s trend without matching its level. The penalty $\zeta^{2} T_{pre} \lVert \omega \rVert_2^2$ discourages putting all weight on one or two donors; Arkhangelsky et al. set $\zeta = (N_{tr} T_{post})^{1/4}\, \hat{\sigma}$, with $\hat{\sigma}$ the standard deviation of first-differenced control outcomes.

The time weights are the mirror image — they find pre-period years whose weighted average lines up with the post-period:

$$ \hat{\lambda}^{sdid} = \underset{\lambda \in \Lambda}{\arg\min} \sum_{i=1}^{N_{co}} \left(\lambda_0 + \sum_{t=1}^{T_{pre}} \lambda_t\, Y_{it} - \frac{1}{T_{post}} \sum_{t=T_{pre}+1}^{T} Y_{it}\right)^{2} + \zeta_{\lambda}^{2}\, N_{co}\, \lVert \lambda \rVert^{2} $$

This says: find pre-period year weights so the weighted pre-period control outcome matches each control’s post-period average. Years that look most like the post-period get the most weight. We will see SDID place essentially all pre-period weight on 1986–1988.

	Unit weights ω	Time weights λ	Unit FE α	Must match
DiD	uniform	uniform	yes	parallel trends vs. all controls
SC	optimized	none	no	California’s level and trend
SDID	optimized	optimized	yes	California’s trend (level gap allowed)

4. Loading the data

We already loaded and xtset the panel above. The sdid command takes the data in long form and needs four arguments — outcome, unit, time, and a 0/1 treatment indicator — so no further reshaping is required. The synth2 command additionally needs a numeric panel id and xtset, which we created with encode.

summarize packspercapita
tab treated

 Variable | Obs Mean Std. dev. Min Max
-------------+---------------------------------------------------------
packsperca~a | 1,209 122.6493 35.04942 40.7 296.2
treated | Freq. Percent Cum.
------------+-----------------------------------
0 | 1,197 99.01 99.01
1 | 12 0.99 100.00
------------+-----------------------------------

Only 12 of 1,209 observations are treated — California in its 12 post-1988 years. This extreme imbalance (one treated unit) is the defining feature of a comparative case study and, as we will see in Section 9, dictates how inference must be done.

5. A first look: the original difference-in-differences

The simplest credible estimate is a 2×2 difference-in-differences: compare California’s change from before to after 1989 with the control states’ change over the same window. The “difference in differences” removes anything common to all states (the nationwide decline in smoking) and anything fixed about California (its lower baseline level).

gen byte cal = state=="California"
gen byte post = year>=1989
reg packspercapita i.cal##i.post

------------------------------------------------------------------------------
packsperca~a | Coefficient Std. err. t P>|t| [95% conf. interval]
-------------+----------------------------------------------------------------
1.cal | -14.359 6.788699 -2.12 0.035 -27.67799 -1.040019
1.post | -28.51142 1.747208 -16.32 0.000 -31.93932 -25.08351
|
cal#post |
1 1 | -27.34911 10.91131 -2.51 0.012 -48.75638 -5.941839
|
_cons | 130.5695 1.087062 120.11 0.000 128.4368 132.7023
------------------------------------------------------------------------------

The interaction cal#post = −27.35 is the DiD estimate: relative to the control states, California’s smoking fell by about 27 packs per capita after Proposition 99. We can read the four group means straight off the table: control states averaged 130.57 packs before and 102.06 after (a drop of 28.5), while California went from 116.21 to 60.35 (a drop of 55.86). The difference of those drops, $-55.86 - (-28.51) = -27.35$, is the DiD.

But this number trusts the parallel-trends assumption against the simple average of 38 very different states — and the raw-trends figure showed California was already drifting away from that average before 1989. If California was on a steeper downward path for reasons unrelated to the policy, DiD will overstate the effect. This is the weakness synthetic methods are designed to fix.

6. The original synthetic control with `synth2`

Synthetic control replaces the simple average of controls with a weighted average chosen to track California before 1989. We fit it with synth2 (Yan and Chen 2023), a modern wrapper around Abadie’s synth that adds placebo tests and visualization. Because our panel has only the outcome, we match on the full pre-period path — each pre-1989 year of packspercapita enters as its own predictor. This is the fair, like-for-like analog to what SDID uses.

* California is id 3 after encode (alphabetical)
local preds
forvalues y = 1970/1988 {
local preds "`preds' packspercapita(`y')"
}
synth2 packspercapita `preds', trunit(3) trperiod(1989) figure

 Number of Control Units = 38 Root Mean Squared Error = 1.65640
Number of Covariates = 19 R-squared = 0.97699
Optimal Unit Weights:
---------------------------
Unit | U.weight
--------------+------------
Utah | 0.3940
Montana | 0.2320
Nevada | 0.2050
Connecticut | 0.1090
NewHampshire | 0.0450
Colorado | 0.0150
---------------------------
Note: The average treatment effect over the posttreatment period is -19.4814.

The pre-period fit is excellent — a root mean squared prediction error of 1.66 packs and an $R^2$ of 0.98, meaning synthetic California reproduces real California almost exactly before 1989. The synthetic is built from just six donors, dominated by Utah (0.39), Montana (0.23), and Nevada (0.21) — states that smoked like California before the program. The estimated effect averages −19.48 packs per capita over 1989–2000, smaller than the naive DiD’s −27.35: once we compare California to states that actually looked like it, part of the apparent drop turns out to be the wrong comparison group, not the policy.

The fit before 1989 is the whole credibility argument for synthetic control: if the synthetic matches California for nineteen years and then diverges exactly when the policy starts, the divergence is plausibly the policy. The next figure shows the same thing as a single gap series.

The gap is essentially flat and near zero through 1988 — the pre-period fit is good — and then opens up after the policy, reaching roughly −27 packs by 2000. Averaged over the post-period, that is the −19.5 headline. The growing gap is consistent with a program whose effect compounds as the tax and campaign change long-run behavior.

7. Synthetic difference-in-differences with `sdid`

Now SDID. The syntax mirrors the data structure — outcome, unit, time, treatment — and one option, vce(), selects the inference method. We start with vce(noinference) to focus on the point estimate and the diagnostic graph.

sdid packspercapita state year treated, method(sdid) vce(noinference) graph

Synthetic Difference-in-Differences Estimator
-----------------------------------------------------------------------------
packsperca~a | ATT Std. Err. t P>|t| [95% Conf. Interval]
-------------+---------------------------------------------------------------
treated | -15.60383 . . . . .
-----------------------------------------------------------------------------

The SDID estimate is −15.60 packs per capita — smaller again than both DiD (−27.35) and SC (−19.48). Relative to the level SDID implies California would have smoked, this is roughly a 20% reduction, and it is the number reported in Arkhangelsky et al. (2021). Why is it smaller than SC’s? Because SDID does two things SC does not: it allows a constant level gap (so it is not forced to fit California’s level, only its trend), and it down-weights pre-period years that look nothing like the late 1980s. Both make the comparison more conservative.

The graph option produces SDID’s signature diagnostic.

Two things are worth noticing. First, the synthetic “Control” line sits above California throughout — SDID does not try to close that level gap, because the unit fixed effect absorbs it. What SDID cares about is whether the two lines stay parallel before 1989 (they do) and then diverge after (they do). Second, the green shaded ribbon shows the time weights $\hat{\lambda}_t$ — and they are not uniform.

This is SDID’s distinctive move. Of the nineteen pre-policy years, it places all pre-period weight on 1986–1988 — the years most similar to the post-1989 period — and zero on 1970–1985. Intuitively, smoking behavior and its determinants in 1972 tell us little about the counterfactual for 1995; the late 1980s tell us much more. DiD and SC, by contrast, treat 1972 and 1988 as equally informative. We can confirm which states and years carry weight by asking sdid to return them:

sdid packspercapita state year treated, vce(noinference) returnweights mattitles

The returned unit weights $\hat{\omega}_i$ are diffuse compared with synthetic control’s: the largest are Nevada (0.12), New Hampshire (0.11), Connecticut (0.08), Delaware (0.07), and Colorado (0.06), with positive weight spread across roughly twenty states. Where synth2 leaned on six donors, SDID’s ridge penalty spreads the weight — trading a little pre-period fit for a more stable, less idiosyncratic comparison group. Both methods nonetheless agree on the kind of state that resembles California: Nevada, Utah, Montana, Connecticut, and Colorado appear prominently in both.

8. One command, three estimators

Here is the practical payoff emphasized by Clarke et al. (2024): the sdid command implements all three estimators in an identical framework. You do not switch packages or rewrite your model — you change the single option method(). Estimation, inference (vce()), and the diagnostic graph all work the same way for each.

sdid packspercapita state year treated, method(did) vce(noinference) graph
sdid packspercapita state year treated, method(sc) vce(noinference) graph
sdid packspercapita state year treated, method(sdid) vce(noinference) graph

DiD (sdid framework) = -27.34911
SC (sdid framework) = -19.61966
SDID = -15.60383

This is a strong internal consistency check. The framework’s method(did) returns −27.349 — identical, to the decimal, to the raw 2×2 interaction we computed by hand with reg in Section 5. And method(sc) returns −19.620, essentially the same as the −19.481 from the standalone synth2 command (the tiny gap reflects different regularization: sdid matches the full pre-period path with a ridge penalty, while synth2 optimizes Abadie’s predictor-weighting V-matrix). In other words, the unified command reproduces the two classic estimators we obtained by entirely separate routes — which is exactly the claim that they are special cases of one weighted regression. And because the optimal weights are computed once and reused across vce() options, doing so is computationally cheap.

The same graph option yields each method’s diagnostic, so they can be read side by side.

Stacking all four counterfactuals on one chart makes the ranking transparent. To put them on a common scale, the SDID counterfactual is anchored to California by its $\lambda$-weighted pre-period gap (recall SDID identifies effects only up to a constant level, which the unit fixed effect absorbs).

The story is consistent across methods — Proposition 99 reduced smoking — but the magnitude depends on how the counterfactual is built. The naive DiD is the most extreme because it compares California to a control average that was already on a different trajectory. Synthetic control fixes the comparison group and shrinks the estimate to about −19.5. SDID, by additionally allowing a level gap and weighting the informative late-1980s years, is the most conservative at −15.6. Reasonable methods bracket the truth; SDID’s contribution is to be robust to the assumption — exact parallel trends — that the others lean on hardest.

Collecting every estimate in one place:

Method	Command	ATT (packs per capita)
Raw 2×2 DiD	`reg y i.cal##i.post`	−27.35
DiD (unified)	`sdid …, method(did)`	−27.35
Synthetic control	`synth2 …`	−19.48
SC (unified)	`sdid …, method(sc)`	−19.62
SDID	`sdid …, method(sdid)`	−15.60

9. Inference: how sure are we?

A point estimate is not enough; we need a standard error. SDID’s variance feeds a familiar normal-approximation confidence interval:

$$ \hat{\tau}^{sdid} \pm z_{\alpha/2} \sqrt{\hat{V}_{\tau}} $$

Arkhangelsky et al. (2021) offer three ways to estimate $\hat{V}_{\tau}$: a bootstrap, a jackknife, and a placebo (permutation) procedure. The choice is not free here — it is forced by our design. With a single treated unit:

The jackknife is literally undefined. It works by deleting one unit at a time and re-estimating; when it deletes California, there is no treated unit left, so the treated-removed estimate does not exist.
The bootstrap relies on resampling many treated units; its asymptotics require the number of treated units to grow. With one treated unit it is unreliable.
The placebo procedure is the one valid option. It keeps the controls, repeatedly assigns the treatment structure to a control state as a fake “placebo” treatment, re-estimates the effect, and uses the spread of those placebo estimates as the variance.

graph TD
Q{"How many<br/>treated units?"}
Q -->|"One — e.g. California"| PL("<b>Placebo / permutation</b><br/><i>the valid choice here</i>")
Q -->|"Many — e.g. staggered adoption"| BJ("Bootstrap or jackknife<br/><i>asymptotics in number of treated units</i>")
PL --> THIS("this tutorial<br/>vce(placebo)")
BJ --> OOS("out of scope<br/>(needs another design)")
classDef sty_Q fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
class Q sty_Q
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class PL teal
class BJ,THIS blue
class OOS orange

So we run placebo inference, the appropriate choice for a comparative case study.

sdid packspercapita state year treated, vce(placebo) seed(1213)

Synthetic Difference-in-Differences Estimator
-----------------------------------------------------------------------------
packsperca~a | ATT Std. Err. t P>|t| [95% Conf. Interval]
-------------+---------------------------------------------------------------
treated | -15.60383 9.87941 -1.58 0.114 -34.96712 3.75946
-----------------------------------------------------------------------------
95% CIs and p-values are based on large-sample approximations.

The placebo standard error is 9.88, giving a 95% interval of roughly [−35.0, 3.8]. Notice this interval includes zero: by the normal-approximation criterion, we cannot reject “no effect” at the 5% level ($p = 0.114$). With a single treated unit and a noisy donor pool, the SDID interval is genuinely wide — honest about how hard it is to be certain from one case.

But the normal approximation is not the only — or the sharpest — way to use the placebo distribution. We can also run an explicit permutation test: assign the placebo treatment to each control state in turn, collect the placebo effects, and ask how California’s real estimate ranks against them.

* assign each control as a placebo-treated unit, collect placebo ATTs
drop if state=="California"
levelsof state, local(ctrls)
foreach s of local ctrls {
preserve
gen byte ptreat = (state=="`s'") & (year>=1989)
sdid packspercapita state year ptreat, vce(noinference)
* store e(ATT)
restore
}

The placebo effects for control states cluster around zero — reassuring, since those states passed no comparable policy — while California’s −15.6 lands far in the left tail. Only 1 of 38 control states produced a placebo effect as large in magnitude as California’s, a permutation p-value of 0.026. So the two inferential lenses tell complementary stories: the rank-based permutation test says California’s drop is very unlikely to be noise (significant at 5%), while the conservative normal-approximation interval reminds us that, with a single treated unit, the precision of the magnitude is limited. Reporting both is the honest summary.

Other inference designs (out of scope)

It would be wrong to conclude that bootstrap and jackknife are “bad” — they are simply built for a different design. They come into their own when there are many treated units, especially under staggered adoption, where units adopt the policy at different times. In that setting the ATT is an average of adoption-cohort-specific effects,

$$ \widehat{ATT} = \sum_{a \in A} \frac{T_{post}^{a}}{T_{post}}\ \hat{\tau}_a^{sdid} $$

and with many treated units the asymptotic arguments behind the bootstrap and jackknife hold. The sdid command supports all of this — vce(bootstrap), vce(jackknife), covariate adjustment, and staggered timing — but those tools require a genuinely different research design (multiple treated units adopting at multiple times) than California’s single 1989 intervention. We deliberately keep this tutorial to the block design with one treated unit, where the placebo procedure is the right and sufficient tool. The staggered case, with its own estimation and inference, is a natural next tutorial.

10. Robustness and discussion

What should we take away about Proposition 99? Three independent constructions of the counterfactual — DiD, synthetic control, and SDID — all agree the policy reduced smoking, with estimates from −15.6 to −27.3 packs per capita. The disagreement is informative rather than alarming: it maps directly onto how much each method trusts the comparison group.

DiD (−27.35) trusts that California would have moved parallel to the average of 38 heterogeneous states. The pre-period figure shows that average was already diverging from California, so DiD likely overstates the effect.
Synthetic control (−19.48) fixes the comparison group to states that actually resembled California (Utah, Montana, Nevada). Its pre-period fit is excellent (RMSE 1.66), which is the evidence for its credibility.
SDID (−15.60) additionally allows a constant level gap and concentrates on the informative late-1980s years. It is the most robust to a violation of exact parallel trends, and the most conservative.

The honest range, then, is something like “Proposition 99 cut cigarette consumption by roughly 16–20 packs per capita per year, plausibly larger by the end of the 1990s,” with SDID the preferred single number because it leans least on the assumption most likely to fail.

A few caveats apply to all three estimates. With one treated unit, statistical power is inherently limited — the SDID confidence interval includes zero even though the permutation test is significant, and no method can fully escape that. The placebo variance assumes homoskedasticity across units (the placebo treatments are drawn only from controls). And like every comparative case study, identification assumes no other large shock hit California alone in 1989 and no spillovers to the donor states (if Californians bought cigarettes across state lines, neighboring donors are contaminated). These are assumptions to argue substantively, not settle statistically.

11. Summary and key takeaways

Method. SDID is one weighted two-way fixed-effects regression. DiD is the special case with uniform weights; synthetic control is the special case with unit weights but no time weights and no unit fixed effect. SDID uses both unit and time weights and keeps the unit fixed effect, so it matches California’s pre-period trend while allowing a constant level gap.
Data. On the Proposition 99 panel, the estimates are DiD −27.35, synthetic control −19.48, and SDID −15.60 packs per capita — the same direction, with magnitude shrinking as the comparison group becomes more credible. SDID’s time weights land entirely on 1986–1988.
One framework. The single sdid command reproduced the hand-computed 2×2 DiD exactly (−27.35) and the standalone synth2 synthetic control closely (−19.62 vs −19.48), confirming that all three are special cases of one estimator and can be run, with inference and graphs, from one command.
Inference. With a single treated unit, placebo is the valid procedure: jackknife is undefined and the bootstrap is unreliable. The placebo SE is 9.88 (95% CI [−35.0, 3.8], which includes zero), while the permutation test gives p = 0.026. Report both.
Limitation and next step. One treated unit means limited power. The natural extension is staggered adoption with many treated units, where vce(bootstrap) and vce(jackknife) become appropriate and covariates can be added — a different design, and a good follow-up tutorial.

12. Exercises

Weights side by side. Re-run sdid …, method(sc) vce(noinference) returnweights and compare its unit weights to the synth2 donor weights from Section 6. Which states appear in both? Why does the sdid version spread weight more widely? (Hint: the ridge penalty $\zeta$.)
Placebo stability. Re-estimate sdid …, vce(placebo) seed(1213) with a different seed() and with more replications via reps(). How much does the standard error move? What does that tell you about reading a single placebo SE to three decimal places?
Time weights matter. Inspect e(lambda) after the SDID run and confirm the weight on 1986–1988. Then think through: if you forced uniform time weights (as DiD and SC do), would you expect the estimate to move toward or away from the DiD number? Check your intuition by comparing method(sdid) with method(sc).

References

Arkhangelsky, D., Athey, S., Hsiao, D. A., Imbens, G. W., and Wager, S. (2021). Synthetic Difference-in-Differences. American Economic Review 111(12): 4088–4118.
Clarke, D., Pailañir, D., Athey, S., and Imbens, G. (2024). On Synthetic Difference-in-Differences and Related Estimation Methods in Stata. The Stata Journal (st0757). The sdid command.
Abadie, A., Diamond, A., and Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program. Journal of the American Statistical Association 105(490): 493–505.
Abadie, A., and Gardeazabal, J. (2003). The Economic Costs of Conflict: A Case Study of the Basque Country. American Economic Review 93(1): 113–132.
Yan, G., and Chen, Q. (2023). synth2: Synthetic Control Method with Placebo Tests, Robustness Test and Visualization. The Stata Journal 23(3): 597–624. The synth2 command.

Acknowledgments

The analysis uses the sdid (Clarke, Pailañir, Athey, and Imbens) and synth2 (Yan and Chen) Stata packages and the Proposition 99 dataset distributed with sdid. AI tools (Claude Code, with NotebookLM for the audio summary) assisted in drafting and exposition; all code was executed and all numbers verified by the author, who is responsible for any remaining errors.

AI Podcast: Synthetic Difference-in-Differences

Click play to load

0:00 0:00

Augmented Synthetic Control for Multiple Countries: A Tutorial with augsynth

Fri, 05 Jun 2026 00:00:00 +0000

Abstract

Many policy evaluations in cross-country economics hinge on a counterfactual we never observe—the path a country would have followed without the policy—and classic synthetic control methods (SCM) deliver credible estimates only when a small, structurally heterogeneous donor pool can reproduce the treated unit’s pre-treatment trajectory almost perfectly. This tutorial demonstrates the Augmented Synthetic Control Method (ASCM) of Ben-Michael, Feller, and Rothstein (2021), which adds a doubly-robust Ridge outcome model to correct residual bias, in a multi-country setting using the R package augsynth and its three entry points (single_augsynth, multisynth, augsynth_multiout). It first validates each function on a simulated panel of 25 countries over 39 years (1985–2023, 975 rows) with a known injected effect, then qualitatively replicates Papaioannou (2021) on a balanced panel of 36 countries from 1980 to 2017 (12 founding euro members and 24 non-euro donors) drawn from the Penn World Tables. On simulated data all three estimators recover the truth closely—single_augsynth returns +6.241 against a true +6.250, the pooled multisynth effect is 3.222 versus 3.155, and augsynth_multiout recovers +6.538 and +3.531—while a suitability test shows Ridge-ASCM correcting a sign error that plain SCM cannot, cutting the mean recovery error from 0.737 to 0.128. On the real euro-area data, synthetic Germany’s TFP runs +0.133 above its counterfactual (about +8.0% over 2000–2007 and +19.3% over 2008–2017), the pooled euro effect is a non-significant −0.016 that masks a +0.39 early bump erased by the 2008–2014 crisis, and the ASCM percentage effects correlate with the paper’s at a Spearman 0.74. The exercise shows that validating a causal estimator against simulated ground truth, leaning on augmentation only when the pre-treatment fit demands it, matching inference tools to the estimator, and reading dynamics rather than averages together make synthetic-control analysis of multiple countries honest and reproducible.

1. Overview

Did joining the euro make countries more productive? Did a national reform pay off, or would the country have done just as well without it? Questions like these share an awkward feature: we never observe the counterfactual — the path a country would have followed had it not adopted the policy. We see only the world that happened.

The synthetic control method (SCM) answers this by building the missing counterfactual from data. Among countries that did not adopt the policy, it finds the weighted recipe whose pre-treatment trajectory looks indistinguishable from the treated country’s, and uses that “synthetic” twin as the stand-in for the absent counterfactual. If the pre-treatment match is good, the post-treatment gap between the actual country and its synthetic version is the most credible estimate of the policy’s effect.

Classic SCM, however, has a well-known weak spot: it only works when the donor pool can reproduce the treated country’s pre-treatment path almost perfectly. In cross-country work the donor pool is small and countries are structurally different, so a good match is the exception, not the rule. The Augmented Synthetic Control Method (ASCM) of Ben-Michael, Feller, and Rothstein (2021) fixes this by adding an outcome model that estimates and removes the leftover bias when the pre-treatment fit is imperfect — the same doubly-robust idea behind augmented inverse-probability weighting. When the fit is already good, the augmentation does almost nothing; when it is poor, it rescues the estimate.

This tutorial is a hands-on tour of ASCM in a multi-country setting using the augsynth package. It has two parts. In Part 1 we work with simulated data where the true effect is known, so we can introduce the three augsynth entry points and verify that each one recovers the truth:

single_augsynth — one treated unit (the building block),
multisynth — many treated units with staggered adoption,
augsynth_multiout — one treated unit with several outcomes.

We save the simulated panel to a CSV you can reuse, and we run a small suitability test that shows exactly where plain SCM fails and augmentation saves the day. In Part 2 we put the method to work on real data, qualitatively replicating Papaioannou (2021), “European monetary integration, TFP and productivity convergence,” which asks whether the 12 founding members of the euro area saw faster total factor productivity (TFP) growth than a synthetic counterfactual built from non-euro economies.

Learning objectives:

Distinguish the three augsynth entry points and recognize which one a problem calls for
Read the augsynth formula mini-language (outcome ~ treatment | covariates)
Use simulated data with a known effect to validate a causal estimator before trusting it on real data
Explain when augmentation (the Ridge outcome model) matters and when it does not
Replicate the qualitative findings of a published synthetic-control paper and compare estimates honestly
Use augsynth’s inference toolbox — jackknife+, conformal, jackknife, and the wild bootstrap — and explain what makes an estimated effect statistically significant

The diagram below maps the three functions onto one pipeline.

flowchart TD
P("Panel of countries<br/>(unit, time, outcome, treatment)") --> D{"How many treated units?<br/>how many outcomes?"}
D -->|one unit, one outcome| S("single_augsynth")
D -->|many units, staggered| M("multisynth")
D -->|one unit, many outcomes| O("augsynth_multiout")
S --> W("SCM weights W<br/>(convex recipe of donors)")
M --> W
O --> W
W --> R("+ Ridge outcome model<br/>(bias correction)")
R --> A("ATT = actual − synthetic")
A --> I("Inference<br/>jackknife+ / conformal / bootstrap")
classDef sty_D fill:#1f2b5e,stroke:#c8d0e0,stroke-width:3px,color:#e8ecf2
class D sty_D
classDef blue fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class P,W,I blue
class S,M,O orange
class R,A teal

The routing is by shape, not difficulty: count the treated units and the outcomes, and the panel flows to one of the three functions. All three then converge on the same machinery — a synthetic counterfactual built from convex donor weights, optionally refined by the Ridge bias-correction step, with the ATT read off as actual minus synthetic.

Key concepts at a glance

This post leans on a small vocabulary repeatedly. Each concept below has three parts. The definition is always visible; the example and analogy sit behind clickable cards — open them when a term feels slippery, leave them collapsed for a quick scan.

1. Synthetic control method (SCM). A weighted average of donor (untreated) units, built so that its pre-treatment path matches the treated unit. The synthetic’s post-treatment trajectory is the estimated counterfactual; the gap to the actual unit is the treatment effect (ATT).

Example

We build a “Synthetic Germany” from a weighted blend of 24 non-euro economies, chosen so that pre-1999 German TFP matches the real thing. After 1999, the gap between actual and synthetic Germany estimates the euro’s effect on productivity.

Analogy

A stunt double assembled from many extras. Before the dangerous scene (treatment) the double mimics the star perfectly; during the scene it shows what would have happened to the star.

2. Augmented SCM (ASCM) and bias correction. Plain SCM is only unbiased when the pre-treatment fit is (near) perfect. ASCM fits an outcome model on the donors and subtracts the part of the gap that model predicts — a bias correction. If the fit is already perfect the correction is zero; if not, it removes leftover imbalance.

Example

For a treated unit sitting outside the donor pool’s range, plain SCM cannot match the pre-period and even gets the sign of the effect wrong. Ridge-augmented SCM closes the pre-treatment gap and recovers the true effect (we see exactly this for unit C05 below).

Analogy

Tarring a wall, then touching up with a brush. SCM lays down the broad coat (weights); the outcome model paints over the spots the roller could not reach (residual bias).

3. Donor pool and convex weights $W$. The donors are the untreated units the synthetic is built from. The weights are non-negative and sum to one (a convex combination), so the synthetic is an interpolation, never an extrapolation, of the donors.

Example

Synthetic C01 is roughly “28% C19 + 21% C09 + 16% C13 + 11% C23 + 10% C08.” The weights add to one; every other donor gets weight zero. (Several donor recipes can reproduce the same factor structure, so the recovered weights need not be the exact ones we built C01 from — what matters is that the synthetic path matches.)

Analogy

A recipe whose proportions sum to 100%. You can blend the donor ingredients but never use a negative amount of flour.

4. Prognostic (outcome) model — progfunc. The model ASCM uses to predict each unit’s untreated outcome. progfunc = "None" gives plain SCM; progfunc = "ridge" fits a Ridge regression on lagged outcomes and is the default, because it also supports valid confidence intervals.

Example

augsynth(y ~ trt, unit, time, data, progfunc = "ridge", scm = TRUE) runs Ridge-ASCM; swapping in progfunc = "None" runs the classic Abadie estimator.

Analogy

A spell-checker for your counterfactual. SCM writes the first draft; the Ridge model flags and fixes the systematic typos.

5. Staggered adoption and partial pooling — multisynth, nu. When many units adopt at different times, multisynth fits one synthetic control per treated unit and partially pools them. The pooling knob nu runs from 0 (each unit separate) to 1 (one shared control); augsynth picks it automatically.

Example

Five simulated countries adopt in 2010, 2013, and 2016. multisynth returns a pooled average effect and a per-country effect, with nu = 0.58 chosen automatically.

Analogy

Grading essays with a rubric. Pure pooling treats every student identically; no pooling grades each in a vacuum; partial pooling borrows a little strength from the class average to stabilize each grade.

6. Multiple outcomes — augsynth_multiout. One treated unit can be tracked on several outcomes at once. A single set of donor weights is chosen to balance all outcomes jointly, which borrows strength across correlated series.

Example

augsynth_multiout(tfp + prod_gap ~ trt, ...) builds one synthetic Germany that matches both TFP and the productivity gap vs the USA before 1999.

Analogy

One tailored suit fitted to several measurements at once — chest, sleeve, and waist — rather than three separate jackets.

7. Inference: an augsynth toolbox. A point estimate is only half the answer; we also need to know whether it is distinguishable from zero. augsynth offers several tools. For a single unit we report the robust jackknife+ confidence interval (inf_type = "jackknife+") and the conformal p-value (inf_type = "conformal"); for many units, multisynth offers a jackknife interval and the more conservative wild bootstrap (inf_type = "bootstrap"); for multiple outcomes, conformal returns a p-value per outcome. Section 9 makes this concrete.

Example

On the simulated panel the pooled multisynth effect is significant under the jackknife ([0.69, 5.75], excludes zero) but not under the wild bootstrap ([-2.47, 9.78]) — the same estimate, a different verdict. The method matters.

Analogy

Two bathroom scales. The jackknife reads your weight precisely; the wild bootstrap adds the uncertainty of the scale itself and reports a wider range. Neither is “wrong” — they answer slightly different questions.

2. Setup

augsynth is not on CRAN, so we install it from GitHub (pinned to a specific commit for reproducibility). We also load Synth (a dependency), haven (to read the Stata file in Part 2), and the usual tidyverse plotting tools. Everything below was executed with R 4.5.2, augsynth 0.2.0, and Synth 1.1.10.

# augsynth is installed from GitHub (run once):
# remotes::install_github("ebenmichael/augsynth@7a90ea4")
library(augsynth)
library(haven) # read the Stata .dta in Part 2
library(dplyr)
library(tidyr)
library(ggplot2)
set.seed(20260605)
# Site colour palette
STEEL_BLUE <- "#6a9bcc" # synthetic control
WARM_ORANGE <- "#d97757" # treated / actual
NEAR_BLACK <- "#141413" # truth / reference
TEAL <- "#00d4c8" # ridge-augmented / highlight

A note on the formula mini-language you will see throughout: augsynth takes outcome ~ treatment on the left, and optional matching covariates after a pipe, outcome ~ treatment | x1 + x2. The unit and time arguments name the panel’s identifier columns, and t_int is the intervention time (for the single-unit functions). The treatment column is a 0/1 indicator that turns on when treatment starts.

3. A two-country intuition example

Before any weighting machinery, here is the whole idea in two countries. We simulate a treated country, “Atlantia,” whose untreated path is a clean copy of a comparison country, “Borealis,” plus an injected effect that switches on in 2012 and grows by 1.5 units per year. Because Borealis is the counterfactual by construction, the gap after 2012 must equal the injected effect.

years <- 2000:2023
t_int <- 2012
trend <- 40 + 1.2 * (years - 2000) + 3 * sin(2 * pi * (years - 2000) / 9)
control <- trend + rnorm(length(years), 0, 0.6)
true_effect <- ifelse(years >= t_int, 1.5 * (years - t_int + 1), 0)
treated <- trend + rnorm(length(years), 0, 0.6) + true_effect
mean(treated[years >= t_int] - control[years >= t_int]) # estimated gap
mean(true_effect[years >= t_int]) # true effect

[1] 9.601 # estimated mean post-2012 gap
[1] 9.75 # true mean injected effect

The estimated post-2012 gap is 9.60 against a true mean effect of 9.75 — within 1.5%. This is synthetic control in its simplest possible form: a single, perfectly matched comparison. The figure makes the logic visible — the two lines are indistinguishable before 2012, then Atlantia pulls away by exactly the injected amount.

The catch, of course, is that real comparison countries are never perfect twins. That is why we need a weighted combination of many donors — and, when even that is not enough, the augmentation step. The rest of Part 1 builds up to both.

4. One reusable simulated panel

We now build a richer panel that all three functions will share. It has 25 countries over 39 years (1985–2023): five treated units (C01–C05) and twenty never-treated donors (C06–C25). The long pre-period is deliberate — it gives the inference procedures in Section 9 the statistical power they need. The data come from a three-factor model plus a unit fixed effect, so a good synthetic control genuinely exists. Treated units C01–C04 are each a sparse convex blend of three named donors (so a near-perfect synthetic control is guaranteed), while C05 is placed deliberately outside the donor hull to stress-test the methods later.

Treatment is staggered: C01 and C02 adopt in 2010, C03 in 2013, and C04 and C05 in 2016. The effect on the primary outcome gdp_index is a jump at adoption plus a gentle yearly ramp (with a correlated 0.6× effect on a second outcome trade_index). C01–C04 get large positive effects (a jump of +2.0 to +3.5 plus a +0.3 to +0.5 ramp); crucially, C05’s effect is small and negative (a −1.0 jump, −0.05 ramp), mimicking the real-world fact that a few euro members underperformed.

# ... factor-model construction (see analysis.R) ...
adopt <- c(C01 = 2010, C02 = 2010, C03 = 2013, C04 = 2016, C05 = 2016)
jump <- c(C01 = 3.0, C02 = 2.5, C03 = 3.5, C04 = 2.0, C05 = -1.0) # level shift
slope <- c(C01 = 0.5, C02 = 0.4, C03 = 0.5, C04 = 0.3, C05 = -0.05) # yearly ramp
write.csv(panel, "synthetic_panel_multicountry.csv", row.names = FALSE)

Saved synthetic_panel_multicountry.csv: 25 units x 39 years = 975 rows
Adoption schedule: C01 2010 C02 2010 C03 2013 C04 2016 C05 2016
True outcome-1 jump at adoption: C01 3.0 C02 2.5 C03 3.5 C04 2.0 C05 -1.0
True outcome-1 yearly ramp: C01 0.5 C02 0.4 C03 0.5 C04 0.3 C05 -0.05

The saved file ships with extra columns most real datasets never give you: the true counterfactual (gdp_index_cf, trade_index_cf) and the true injected effect (true_effect_gdp, true_effect_trade). Those let us grade every estimate against ground truth. Below, the twenty donor paths (grey) surround the five treated paths (colored), with dots marking each unit’s adoption year.

Two display equations capture what every augsynth call is doing under the hood. First, the SCM weight problem: find the convex donor weights $W$ that best match the treated unit’s pre-treatment vector.

$$W^{\star} = \arg\min_{W \in \Delta} \lVert X_1 - X_0 W \rVert_V \qquad \text{subject to} \qquad w_j \ge 0, \quad \sum_j w_j = 1$$

Here $X_1$ is the treated unit’s pre-treatment outcomes (and any covariates), $X_0$ is the matching donor matrix (one column per donor), $W$ is the vector of donor weights, $V$ weights the predictors, and $\Delta$ is the simplex (the constraint that weights are non-negative and sum to one). The solution $W^{\star}$ is the “recipe” for the synthetic control.

Second, the augmented, bias-corrected estimator. ASCM starts from the SCM gap and subtracts what an outcome model $\widehat{m}_t(\cdot)$ predicts the residual imbalance should be:

$$\widehat{\tau}_t^{\mathrm{aug}} = \Big(Y_{1t} - \sum_j w_j Y_{jt}\Big) - \Big(\widehat{m}_t(X_1) - \sum_j w_j \widehat{m}_t(X_j)\Big)$$

The first term is the ordinary SCM gap (actual treated minus weighted donors at time $t$). The second term is the correction: $\widehat{m}_t$ is the prognostic model (a Ridge regression when progfunc = "ridge"), evaluated at the treated unit’s covariates versus the donors’. When the pre-treatment fit is perfect the donors already reproduce $\widehat{m}_t(X_1)$, so the correction vanishes and ASCM equals SCM. When the fit is poor, the correction removes the leftover bias. This is the doubly-robust safety net.

5. One treated unit: `single_augsynth`

The simplest case has one treated unit. We isolate C01 together with the twenty donors (never mixing in the other treated units — that would contaminate the donor pool), build the treatment indicator, and fit both plain SCM (progfunc = "None") and Ridge-ASCM (progfunc = "ridge"). The top-level augsynth() function dispatches to single_augsynth() automatically when it sees one treated unit and one intervention time.

sim_single <- panel |>
filter(country %in% c("C01", donors)) |>
mutate(trt = as.integer(country == "C01" & year >= 2010))
sc_plain <- augsynth(gdp_index ~ trt, country, year, sim_single,
t_int = 2010, progfunc = "None", scm = TRUE)
sc_ridge <- augsynth(gdp_index ~ trt, country, year, sim_single,
t_int = 2010, progfunc = "ridge", scm = TRUE)
# jackknife+ confidence interval (robust) and conformal p-value
summary(sc_plain, inf_type = "jackknife+")$average_att
summary(sc_plain, inf_type = "conformal")$average_att

C01 true average post ATT : +6.250
Plain SCM avg ATT : +6.241 jackknife+ [5.998, 6.506] conformal p<0.001 L2=0.135
Ridge-ASCM avg ATT : +6.241 jackknife+ [5.998, 6.506] conformal p<0.001 L2=0.135 lambda=2639

Both estimators nail the truth: the true average effect of C01 is +6.250 and each method returns +6.241, an error of 0.1%. And the effect is unambiguously real — the jackknife+ 95% confidence interval is [5.998, 6.506], comfortably excluding zero, and the conformal p-value is below 0.001. Notice that plain SCM and Ridge-ASCM give the same answer here. That is not a coincidence — C01 sits comfortably inside the donor hull, so the pre-treatment fit is already good (scaled L2 imbalance ≈ 0.14, well below the 1.0 you would get from naively averaging donors), and the Ridge penalty is driven to a large value (lambda ≈ 2639) that all but switches the augmentation off. This is the “when fit is good, ASCM ≈ SCM” principle in action.

The synthetic control reproduces C01’s pre-2010 path closely and then diverges, exactly as designed.

How well does the dynamic effect line up with the truth? The conformal gap plot overlays the estimated treated-minus-synthetic gap (with its pointwise band) against the true injected effect. The two are nearly on top of each other after 2010, while the pre-period gap hovers around zero — the visual signature of a trustworthy synthetic control.

The donor recipe is sparse and interpretable: synthetic C01 is built mostly from five donors (C19, C09, C13, C23, C08), with weights summing to one. This sparsity is a hallmark of SCM and makes the counterfactual auditable.

6. Many treated units, staggered adoption: `multisynth`

Real multi-country studies rarely have a single treated unit. multisynth handles many treated units that adopt at different times. It needs no t_int — it infers each unit’s adoption from when the treatment column switches from 0 to 1 — and it returns both a pooled average effect and per-unit effects, partially pooling them for stability.

sim_multi <- panel |>
filter(country %in% c(treated, donors)) |>
select(country, year, treat_ms, gdp_index)
ms_sim <- multisynth(gdp_index ~ treat_ms, country, year, sim_multi)
summary(ms_sim, inf_type = "jackknife")$att # primary: tight interval
set.seed(20260605)
summary(ms_sim, inf_type = "bootstrap")$att # conservative comparison

multisynth nu (auto) = 0.583 ; global scaled L2 = 0.052 ; n_leads = 8
Estimated vs TRUE average ATT (jackknife CI [ci_lo,ci_hi] + bootstrap CI [boot_lo,boot_hi]):
level estimate ci_lo ci_hi boot_lo boot_hi truth
Average 3.222 0.689 5.754 -2.468 9.779 3.155
C01 4.756 4.461 5.050 -7.196 16.322 4.750
C02 4.075 3.930 4.221 -6.000 13.800 3.900
C03 5.362 5.154 5.570 -8.123 18.465 5.250
C04 2.927 2.725 3.130 -4.499 10.072 3.050
C05 -1.012 -1.639 -0.385 -3.039 1.195 -1.175

The pooled average effect is estimated at 3.222 against a true value of 3.155 — recovery to within 2%. Just as importantly, every per-unit estimate has the right sign and the right ballpark, including C05’s negative effect (−1.012 estimated vs −1.175 true). On the inference side, the jackknife confidence interval for the pooled effect is [0.689, 5.754], which excludes zero — the effect is significant. The more conservative wild bootstrap gives [-2.468, 9.779], which includes zero: same estimate, but it also propagates the counterfactual-estimation uncertainty, so it does not reach significance. This is our first concrete example of the inference method changing the verdict (Section 9 returns to it). The automatically chosen pooling parameter nu = 0.58 sits between “fully separate” and “fully pooled,” and the tiny global imbalance (scaled L2 = 0.05) tells us the joint synthetic controls fit the pre-period tightly. (One subtlety: multisynth averages effects over a common window of n_leads = 8 post-treatment periods so that all units contribute equally — which is why the per-unit estimates here, e.g. C01’s 4.756, are smaller than the full-period single_augsynth estimate of 6.241; we compute the truth over the same window to keep the comparison fair.)

The per-unit dynamics confirm the recovery. Each panel shows one treated unit’s estimated effect by time-since-adoption against its true effect; the pre-period sits at zero and the post-period jumps then climbs to meet the dashed truth line — with C05 dropping the opposite way.

Averaging across the five heterogeneous units gives the pooled effect path, the single most useful summary in a many-treated-unit study. The estimate tracks the true pooled effect closely, and the figure shows both inference bands — the tighter blue jackknife band (which excludes zero after adoption) and the wider orange wild-bootstrap band (which does not).

7. One unit, two outcomes: `augsynth_multiout`

Sometimes a policy plausibly moves several outcomes and we want one coherent counterfactual for all of them. augsynth_multiout puts multiple outcomes on the left of the formula, separated by +, and finds a single donor recipe that balances all of them before treatment.

mo <- augsynth_multiout(gdp_index + trade_index ~ trt, country, year,
t_int = 2010, sim_single,
progfunc = "None", scm = TRUE, combine_method = "avg")
summary(mo)$average_att # conformal p-value per outcome

 Outcome Estimate p_val
1 gdp_index 6.538 <0.001 (true +6.250)
2 trade_index 3.531 <0.001 (true +3.750)

With a single set of weights, the joint fit recovers both effects: gdp_index at +6.538 (true +6.250) and the correlated trade_index at +3.531 (true +3.750), and both are significant (conformal p < 0.001 for each). The payoff of estimating them together — rather than running two separate single_augsynth fits — is that the donor weights must respect both series at once, which stabilizes the counterfactual when the outcomes are correlated. (A practical note on inference: augsynth_multiout’s summary() returns a conformal p-value per outcome but leaves the confidence-interval bounds as NA — a full CI needs grid_size > 1, which costs grid_size raised to the number-of-outcomes evaluations and, for effects this large, returns degenerate bounds. We therefore report the p-value.)

8. Testing suitability: where plain SCM fails and ASCM corrects

Now the payoff of building C05 outside the donor hull. No convex blend of the donors can reproduce its pre-treatment path, so plain SCM is in trouble. We fit both estimators for every treated unit and tabulate how far each lands from the known truth.

# fit plain and ridge for each treated unit; compare to known effects
recovery # per unit: truth, plain, ridge, jackknife+ CI, conformal p, pre-fit L2, errors

 unit true_att att_plain att_ridge ci_lo ci_hi p_plain sign_flip prefit_l2_plain err_plain err_ridge
C01 6.250 6.241 6.241 5.998 6.506 0.000 FALSE 0.135 0.009 0.009
C02 5.100 5.319 5.319 5.066 5.563 0.000 FALSE 0.150 0.219 0.219
C03 6.000 6.282 6.282 5.958 6.624 0.000 FALSE 0.224 0.282 0.282
C04 3.050 2.948 2.949 2.608 3.305 0.000 FALSE 0.258 0.102 0.101
C05 -1.175 1.896 -1.145 -2.614 6.407 0.866 TRUE 0.414 3.071 0.030
Mean recovery error — plain SCM: 0.737 | Ridge-ASCM: 0.128
C05 pre-fit scaled L2 — plain: 0.414 | ridge: 0.036 (lower = better fit)

This is the headline result of Part 1. For the four well-fit units (C01–C04), plain SCM lands within ~0.3 of the truth and its jackknife+ interval excludes zero (all significant; e.g. C01’s [5.998, 6.506]). For C05, plain SCM gets the sign wrong — it estimates +1.896 when the true effect is −1.175 — because it cannot match the pre-period and the unmatched bias swamps the signal. Its interval [-2.614, 6.407] includes zero (conformal p = 0.87): the estimate is both wrong and not significant, an honest double failure. Ridge-ASCM recovers −1.145, almost exactly right, by closing the pre-treatment gap (scaled L2 falls from 0.41 to 0.04). Across all five units, augmentation cuts the mean recovery error from 0.737 to 0.128. For the four well-fit units the two methods agree; augmentation earns its keep precisely on the hard case.

The picture says it all: under plain SCM the synthetic control (blue) drifts away from actual C05 (orange) before treatment — a fatal sign of poor fit — while Ridge-ASCM pins them together pre-2016, so the post-treatment gap can be trusted.

The practical rule: always read the pre-treatment fit. If the scaled L2 imbalance is small, plain SCM and ASCM will agree and either is fine. If it is large, trust the augmented estimate — and be suspicious of any synthetic control whose pre-period does not track.

9. Inference: is the effect real?

A point estimate answers “how big?”; inference answers “could this be noise?” A synthetic control gap is a difference between two estimated paths, so it carries uncertainty even when the point estimate is dead-on. augsynth ships several inference tools, and — this is the part most tutorials gloss over — they do not always agree. Choosing one and understanding what it measures is part of doing the method honestly.

The three tools, matched to the three estimators.

single_augsynth → jackknife+ (primary) and conformal. The jackknife+ interval (summary(fit, inf_type = "jackknife+")) leaves out one donor at a time, refits, and builds a robust confidence interval for the average effect. We saw it call C01’s effect [5.998, 6.506] — comfortably away from zero. The conformal test (inf_type = "conformal") is a permutation procedure that returns a p-value and a pointwise band over time (the shaded band in the gap figures). It is powerful when the pre-period is long relative to the post-period — which is exactly why this panel starts in 1985, giving twenty-plus pre-treatment years — but its p-value is noisier and loses power when the post-window is long. With both tools agreeing here (jackknife+ CI excludes zero, conformal p < 0.001), we can trust the result.
multisynth → jackknife (primary) and wild bootstrap. The jackknife is the natural interval for an average across treated units; on the simulated panel it put the pooled effect at [0.689, 5.754], significant. The wild bootstrap also propagates the counterfactual-estimation uncertainty, so it is wider — [-2.468, 9.779], not significant. The estimate is identical; the verdict is not. Neither method is “wrong”: the jackknife asks “is the average across these units different from zero?”, the bootstrap asks “accounting for how hard each counterfactual was to build, is it?” When they disagree, say so.
augsynth_multiout → conformal. Returns a p-value per outcome (both < 0.001 for C01); a full confidence interval needs the slow grid_size > 1 path.

What drives significance. Three levers widen a confidence interval and can push a real effect below significance: more noise, fewer pre-treatment periods (a worse-pinned counterfactual), and a poorer pre-fit. This is the same lesson the suitability test taught from the other side — C05’s poor fit (scaled L2 = 0.41) gave it a wide interval that swallowed zero, while C01’s clean fit (L2 = 0.14) produced a tight, significant one. The interactive lab has a fifth tab, Inference, with a significance scoreboard and a slider-driven simulator: move effect size, noise, and the number of pre-periods and watch the interval widen or narrow and the verdict flip at the 5% line.

The honest-reporting rule. On simulated data, where we injected a real effect, every headline is significant. On the real euro-area data below, some results are and some are not — and we report them as they come, rather than dressing a near-zero effect up as a finding.

10. The EMU data: replicating Papaioannou (2021)

We now switch to real data. Papaioannou (2021) asks whether the euro raised the total factor productivity of its founding members. The dataset (shipped in this post’s reference/ folder as a Stata file) is a balanced panel of 36 countries from 1980 to 2017: the 12 founding euro members (Austria, Belgium, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, Netherlands, Portugal, Spain) and 24 non-euro donor economies (from Argentina to Uruguay). The primary outcome is tfp (total factor productivity from the Penn World Tables); a second outcome prod_gap is the log productivity gap versus the USA, where lower means closer to the frontier.

A subtlety worth pausing on: the file stores treat as a time-invariant group flag (1 for euro members in every year) and time1/time2 as period flags (post-1999, post-1992). The actual time-varying treatment is their product.

emu <- read_dta("reference/dataset_revision_1.dta") |>
mutate(country = as.character(country)) |> zap_labels() |> as.data.frame()
emu$trt99 <- as.integer(emu$treat == 1 & emu$time1 == 1) # euro members x post-1999
emu$trt92 <- as.integer(emu$treat == 1 & emu$time2 == 1) # euro members x post-1992

EMU countries (12): Austria, Belgium, Finland, France, Germany, Greece, Ireland,
Italy, Luxembourg, Netherlands, Portugal, Spain
Donor countries (24): Argentina, Australia, Brazil, Canada, ... , Turkey, Uruguay

The raw TFP paths show the setup: twelve euro members (orange) embedded in a cloud of donors (grey), with the 1999 euro launch marked.

11. One country, the paper’s way: synthetic Germany (plain SCM)

We start with a single country to mirror the paper’s per-country synthetic controls. Germany is fit against the 24 donors, matching on pre-treatment TFP and the paper’s predictors (human capital, investment share, economic freedom, patents, agriculture share). This is plain SCM — the closest augsynth analogue to Abadie’s classic method.

fit <- augsynth(tfp ~ trt99 | hum_cap + inv_share + ec_freed + patents + agricult,
country, year, germany_plus_donors, t_int = 1999,
progfunc = "None", scm = TRUE)
summary(fit)$average_att

Germany plain SCM avg ATT (TFP): +0.133 | jackknife+ [-0.082, 0.336] | conformal p=0.027 | L2 0.301
Germany % effect — 2000-07: +8.0% 2008-17: +19.3% (plain SCM)

Actual German TFP runs above its synthetic counterfactual after 1999, with an average effect of +0.133 TFP units — about +8.0% over 2000–2007 and +19.3% over 2008–2017. The pre-treatment fit is good (scaled L2 = 0.30). Here the two inference tools disagree at the margin, which is itself instructive: the conformal p-value is 0.027 (significant at 5%), but the jackknife+ interval [-0.082, 0.336] just barely includes zero. On real data with a modest effect, “significant” is genuinely borderline — we flag it rather than pick the answer we like. Qualitatively this still matches Papaioannou’s finding that Germany was among the clearer winners from monetary integration.

12. Ridge-ASCM as the modern extension

Does augmentation change the German verdict? We refit with progfunc = "ridge" and overlay the two counterfactuals.

fit_ridge <- augsynth(tfp ~ trt99 | hum_cap + inv_share + ec_freed + patents + agricult,
country, year, germany_plus_donors, t_int = 1999,
progfunc = "ridge", scm = TRUE)

Germany Ridge-ASCM avg ATT (TFP): +0.127 | conformal p=0.015 | scaled L2 pre-fit 0.292

The Ridge-augmented estimate (+0.127, conformal p = 0.015) is essentially the plain-SCM estimate (+0.133) — again, because the pre-treatment fit was already good (scaled L2 barely moves, 0.301 → 0.292). The two synthetic counterfactuals are nearly indistinguishable. This is reassuring rather than disappointing: augmentation is an insurance policy, and a quiet premium here means the classic estimate was already trustworthy for Germany.

13. All twelve members at once: `multisynth`

The multi-country headline uses multisynth to estimate the euro’s effect across all twelve members in one model. Because every member adopts in 1999, this is a simultaneous (block) design rather than a staggered one — multisynth handles it, it just does not exercise the staggered machinery we saw in Part 1.

ms_emu <- multisynth(tfp ~ trt99, country, year, emu_multi)
summary(ms_emu, inf_type = "jackknife")$att # primary
set.seed(20260605)
summary(ms_emu, inf_type = "bootstrap")$att # conservative comparison

Pooled EMU avg ATT (TFP): -0.016 | jackknife [-0.282, 0.250] | bootstrap [-0.259, 0.231] | global L2 = 0.100

Taken at face value, the pooled average effect is a near-zero −0.016, and it is not statistically significant — both the jackknife [-0.282, 0.250] and the wild bootstrap [-0.259, 0.231] comfortably include zero. (Unlike the simulated panel, where the two methods disagreed, here they agree: there is simply no pooled signal to detect.) But the single number is also misleading, and reading only the average would be a mistake: the dynamics are the real story. The pooled effect path is flat through the entire pre-period (no pre-trend — a good sign), rises to about +0.39 in the first euro years, then slides into negative territory during the 2008–2014 crisis before recovering toward zero by 2017. The early gains and the crisis losses cancel out in the long-run average. This dynamic — strong early, eroded by the crisis — is exactly the arc Papaioannou describes.

Fitting each member separately (twelve single_augsynth runs against the 24 donors) lets us see the heterogeneity behind the average. Most members run above their synthetic counterfactuals after 1999; Greece and Portugal converge to or fall below theirs after the crisis.

14. Two outcomes at once: `augsynth_multiout` on Germany

The euro should, in principle, move both German TFP and its productivity gap versus the USA. We estimate them jointly.

ger_mo <- augsynth_multiout(tfp + prod_gap ~ trt99, country, year, t_int = 1999,
germany_plus_donors, progfunc = "None", scm = TRUE,
combine_method = "concat")
summary(ger_mo)$average_att

 Outcome Estimate p_val
1 tfp +0.116 0.603
2 prod_gap -0.151 0.603

The two point estimates tell one coherent story: after the euro, German TFP rises (+0.116) and its productivity gap versus the USA narrows (−0.151, where a fall means catching up to the frontier). A single synthetic Germany, balanced on both series, supports both directions at once. But honesty requires the p-value: at 0.603 for each outcome, neither is statistically significant. The joint multi-outcome test is more demanding than the single-TFP conformal test (which was borderline at p = 0.027), and on one country’s real data the signal is not strong enough to clear it. The suggestive, coherent directions are worth reporting — as long as we do not overstate them as established.

15. Robustness: the 1992 Maastricht threshold

Papaioannou notes that markets may have anticipated the euro from the 1992 Maastricht Treaty, not just the 1999 launch. We rerun Germany with the earlier threshold using the trt92 indicator.

ger_92 <- augsynth(tfp ~ trt92 | hum_cap + inv_share + ec_freed + patents + agricult,
country, year, germany_plus_donors, t_int = 1992,
progfunc = "None", scm = TRUE)

Germany avg ATT — 1999 spec: +0.133 | 1992 spec: +0.138

Moving the intervention date back seven years barely changes the estimate (+0.138 vs +0.133). The verdict is robust to the anticipation question, which strengthens the causal reading: whether we date the treatment at the treaty or the launch, synthetic Germany tells the same story.

16. Comparing to the paper

How close is our ASCM re-analysis to Papaioannou’s published numbers? The paper reports a percentage TFP “contribution” per country per period; we compute the analogous ASCM percentage effect and line them up for 2000–2007.

cor(comp$paper_2000_07, comp$ascm_2000_07, method = "spearman")

Correlation paper vs ASCM (2000-07 TFP % effect): Spearman 0.74 | Pearson 0.76
Both agree TFP rose for 12 of 12 members (ASCM).

The rank correlation between the paper’s numbers and ours is 0.74 (Pearson 0.76) — strong agreement given that augsynth uses a different estimator and donor-matching scheme than the paper’s classic SCM. Some countries land almost exactly on the 45-degree line (France: 42.7% vs the paper’s 43.6%; Netherlands: 44.0% vs 38.2%; Spain: 32.0% vs 26.9%), while a few diverge in magnitude (Germany and Ireland) but not in sign. Critically, the pattern replicates: large euro-era gains for France, the Netherlands, Belgium, Spain, and Ireland, and the post-crisis reversals for Greece (−12.4% in 2008–17) and Portugal (−14.3%) that the paper also reports.

We do not reproduce the paper’s numbers exactly, and we should not expect to: the estimators differ, and qualitative replication — same signs, same ranking, same dynamic story — is the right bar for a method comparison. By that bar, ASCM confirms the paper.

17. Discussion

Four threads tie Part 1 and Part 2 together.

Validate on truth, then trust on data. The single most useful habit this tutorial teaches is the order of operations: we confirmed that each augsynth function recovers a known effect on simulated data (errors under 5% for the well-fit units) before turning it loose on the euro question. When the EMU results then showed sensible signs and a clean pre-period, we had earned the right to believe them. A causal estimate you cannot first reproduce on simulated ground truth is a leap of faith.

Augmentation is insurance, not a free lunch. For well-fit units — C01–C04, and Germany — plain SCM and Ridge-ASCM agreed to the second decimal, and the Ridge penalty quietly switched itself off. The augmentation mattered exactly once: for C05, sitting outside the donor hull, where plain SCM got the sign wrong and Ridge-ASCM rescued it (mean error 0.737 → 0.128). The lesson is to read the pre-treatment imbalance every time and lean on augmentation only when the fit demands it.

Inference is a choice, and the choices can disagree. A point estimate is not a finding. On the simulated panel the pooled effect was significant under the jackknife but not under the wild bootstrap; on real German TFP the conformal p-value (0.027) and the jackknife+ interval (which included zero) split at the margin; the pooled euro effect and the joint multi-outcome test were honestly null. We reported the simulated headlines as significant (we injected them) and the borderline and null real-data results as exactly that. Match the inference tool to the estimator, report when methods disagree, and never let a near-zero estimate masquerade as a result.

Averages hide dynamics in multi-country work. The pooled multisynth effect on euro TFP was a forgettable −0.016 on average, yet the path revealed a +0.39 early bump erased by the 2008–2014 crisis. Two caveats travel with multisynth here: the EMU design is simultaneous (so the staggered features are demonstrated only on simulated data), and the pooled average is in raw TFP units, which mixes countries of very different productivity levels — making the per-country percentage effects the quantity truly comparable to the paper. Honest multi-country reporting means showing the path and the per-unit spread, not just the headline number.

18. Summary and next steps

The Augmented Synthetic Control Method generalizes classic SCM with an outcome-model bias correction that is doubly robust: it helps when the pre-treatment fit is poor and does no harm when it is good.
augsynth exposes three entry points: single_augsynth (one unit), multisynth (many units, staggered, pooled + per-unit), and augsynth_multiout (one unit, many outcomes). The top-level augsynth() dispatches to the right one.
On simulated data with a known effect, all three recovered the truth closely and significantly (single +6.241 vs +6.250, jackknife+ [6.00, 6.51]; pooled multisynth +3.222 vs 3.155, jackknife [0.69, 5.75]; multiout +6.54 and +3.53, both conformal p < 0.001), and the suitability test showed Ridge-ASCM correcting a sign error that plain SCM could not — a wrong and non-significant estimate for C05.
Inference is matched to the estimator — jackknife+ and conformal for a single unit, jackknife and the conservative wild bootstrap for multisynth, conformal p-values for multiple outcomes — and the methods can disagree. We reported significance honestly, including the borderline (Germany) and null (pooled euro, joint Germany) real-data cases.
On the real EMU panel, ASCM qualitatively replicated Papaioannou (2021): a positive early TFP effect for most members (rank correlation 0.74 with the paper), Germany up and its US productivity gap narrowing, Greece and Portugal turning negative after the crisis, and robustness to the 1992-vs-1999 dating.

Where to go next: swap progfunc = "ridge" for other prognostic models ("gsyn", "mcp"); add covariate balancing after the | in the formula; explore in-time and in-space placebo tests for inference; or read the staggered-adoption theory in Ben-Michael, Feller, and Rothstein’s companion paper. The reusable synthetic_panel_multicountry.csv is a ready-made sandbox for any of these.

19. Exercises

Swap the outcome. Rerun the per-country EMU fits with prod_gap as the primary outcome instead of tfp. Does the productivity-gap story agree with the TFP story?
Stress the donor pool. Drop the single highest-weight donor from synthetic Germany and refit. How sensitive is the ATT to one donor?
Tune the pooling. Refit the simulated multisynth with nu = 0 (fully separate) and nu = 1 (fully pooled). How do the per-unit estimates and their confidence bands change?
Recover the negative. Using only synthetic_panel_multicountry.csv, reproduce C05’s negative effect with single_augsynth and explain why plain SCM fails.
Combine differently. In augsynth_multiout, switch combine_method = "avg" to "concat" and compare the two-outcome estimates. When might each be preferable?
Make inference disagree. For the simulated pooled multisynth effect, compute both inf_type = "jackknife" and inf_type = "bootstrap" intervals. Then shrink the panel (drop donors or pre-periods) and watch how each interval responds. Which one flips first, and why?

20. References

Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association, 105(490), 493–505.
Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The Augmented Synthetic Control Method. Journal of the American Statistical Association, 116(536), 1415–1427.
Ben-Michael, E., Feller, A., & Rothstein, J. (2022). Synthetic Controls with Staggered Adoption. Journal of the Royal Statistical Society: Series B, 84(2), 351–381.
Papaioannou, S. K. (2021). European monetary integration, TFP and productivity convergence. Economics Letters, 199, 109696.
augsynth package: https://github.com/ebenmichael/augsynth
Feenstra, R. C., Inklaar, R., & Timmer, M. P. (2015). The Next Generation of the Penn World Table. American Economic Review, 105(10), 3150–3182.

AI Podcast: Augmented Synthetic Control

Click play to load

0:00 0:00

Double LASSO in Python: Does Abortion Reduce Crime?

Mon, 25 May 2026 00:00:00 +0000

Abstract

When a candidate-control set is large relative to the sample, plain OLS becomes unstable and atheoretical “kitchen-sink” specifications yield uninterpretable causal estimates, motivating high-dimensional variable selection. This Python tutorial reproduces the Belloni, Chernozhukov and Hansen (2014) extension of Donohue and Levitt’s (2001) abortion-and-crime study, asking whether Double LASSO recovers credible treatment effects from a rich control set and how it compares to the modern DoubleML cross-fitting framework. The data are the replication panel of 48 U.S. states over 12 years (1986–1997 after first-differencing the 1985–1997 series), giving 576 observations with 284 candidate controls per crime outcome. Five Part-A estimators are implemented — first-difference OLS, full OLS, Post-Structural LASSO, and Double LASSO under both the rigorous (BCH theory-based) and cross-validated penalties — using pyfixest, hdmpy, and scikit-learn, with state-clustered HC1 standard errors; Part B adds DoubleMLPLR, DoubleMLIRM, and a LASSO/RandomForest/XGBoost learner-robustness check from the DoubleML library. The no-controls baseline gives a violent-crime effect of −0.152, whereas full OLS with all 284 controls is uninterpretable, exploding to +2.34 for murder with confidence interval [−2.76, +7.45]. Rigorous Double LASSO selects just 8 controls for violent crime and matches the paper’s selection counts and point estimate exactly (−0.104), while DoubleMLPLR returns −0.115 and the three nuisance learners span −0.0855 to −0.1123. The results show that the theory-driven rigorous penalty, not the choice of language or learner, governs credible high-dimensional causal inference.

1. Overview

Companion post. This tutorial is one of three siblings on the same Double LASSO case study — alongside the R version and the Stata version. The three posts share the data, the five estimators, and the identification story; this Python post adds a dedicated introduction to the DoubleML library in §15–§18.

This is the Python companion to the R version and Stata version of the Double LASSO tutorial — same data, same five-estimator narrative, same identification story — plus a second part that introduces DoubleML, a modern Python framework for ML-based causal inference. The R post walks through Belloni, Chernozhukov and Hansen’s (2014) extension of Donohue and Levitt’s (2001) abortion-and-crime panel and shows that Double LASSO with the rigorous (theory-based) penalty reproduces the headline causal estimates from 284 candidate controls while CV-tuned LASSO overshoots. This post does the same computation in Python using pyfixest for OLS rows, hdmpy for the rigorous LASSO, scikit-learn for cross-validated LASSO — and then introduces DoubleML’s cross-fit DoubleMLPLR, DoubleMLIRM, and a learner-robustness comparison across LASSO, RandomForest, and XGBoost.

If you have already read the R or Stata version, the Part A takeaways here are unchanged. The structural reason to write a Python companion is twofold. First, reproducibility: data scientists who work in Python every day will find the friction of switching to R for one method too high, and a transparent Python implementation removes it. Second, introducing DoubleML (Bach et al. 2022) — a beautifully engineered library that ports the modern Neyman-orthogonal cross-fitting framework into a sklearn-native API. DoubleML is the right tool for Python researchers running ML-based causal inference on production data; this post shows how to use it side-by-side with the explicit post-double-selection recipe so you can see exactly where the two approaches agree and where they diverge.

The figure above is the post’s spoiler — the Python version of the R/Stata headline forest plot. Each row is a different estimator; each panel is a different crime outcome. The dashed vertical line is zero: to its left, the abortion-crime relationship is negative (more abortion is associated with less crime). Two patterns jump out, exactly as in the R/Stata companions. First, the LASSO methods (PSL, DL-rigorous, DL-CV) cluster sensibly near the original Donohue-Levitt baseline (First diff) for violent and property crime. Second, OLS with all 284 controls is uninterpretable — its murder estimate is +2.34 with confidence interval [−2.76, +7.45], which would mean a unit increase in the abortion rate raises murder by 234 %. That impossibility is the failure mode that motivates LASSO in the first place.

Learning objectives. After working through this tutorial you will be able to:

Explain when high-dimensional methods like LASSO add value over plain OLS, and when they do not.
Implement the Belloni-Chernozhukov-Hansen Double LASSO procedure in Python using hdmpy.rlasso (rigorous penalty) and sklearn.linear_model.LassoCV (cross-validated penalty), with pyfixest for the post-OLS step.
Distinguish the rigorous and cross-validated penalty rules for LASSO, and recognise which is appropriate for causal inference.
Use the DoubleML library to fit DoubleMLPLR (Partially Linear Regression with cross-fitting) and DoubleMLIRM (Interactive Regression Model for binary treatments).
Compare ML learners (LASSO, RandomForest, XGBoost) as nuisance functions inside DoubleMLPLR and use the spread as a robustness check.
Compute state-clustered standard errors with the HC1 finite-sample correction — both via pyfixest’s vcov={"CRV1": "state"} for OLS rows and via a hand-rolled sandwich on DoubleMLPLR’s orthogonal scores.
Verify that the Python implementation matches the R and Stata companions to the precision allowed by each estimator’s randomness — and explain the five sources of drift that make DoubleML’s defaults differ from R’s hdm.

Key concepts at a glance

The post leans on a small vocabulary. The rest of the tutorial assumes you can move between these terms quickly. Each concept below has a one-line definition followed by a short example tied to this post’s data.

1. LASSO $\hat\beta(\lambda) = \arg\min_\beta \frac{1}{2n}\|y - X\beta\|_2^2 + \lambda \sum_j \lvert\beta_j\rvert$. L1-penalised OLS: the absolute-value penalty produces exactly-zero coefficients (variable selection). In §7 our hdmpy.rlasso of the abortion rate on 284 controls picks just 8 — the rest get shrunk to zero.

2. Penalty $\lambda$. The knob controlling shrinkage. Higher $\lambda$ pins more coefficients to zero. Tuning $\lambda$ is the central design choice and is what separates the rigorous and CV flavours of Double LASSO.

3. Post-Structural LASSO (PSL). One LASSO with the treatment forced in (or partialled out), then plain OLS on the selected support. The simplest one-LASSO causal estimator. We implement it via Frisch-Waugh-Lovell partialling because hdmpy lacks the pnotpen option that R’s glmnet and Stata’s rlasso expose.

4. Double LASSO (DL). Two LASSOs (y on X, d on X), union of selected controls, then post-OLS. The causal-inference-safe variant that beats PSL when controls predict $d$ but not $y$.

5. Selection sets $I_y$ and $I_d$. The indices of controls each LASSO step keeps. Their union $I_y \cup I_d$ is the support of the post-OLS regression. Their imbalance is the empirical fingerprint of when DL adds value.

6. Rigorous vs CV penalty. Two ways to pick $\lambda$. Rigorous: Belloni-Chen-Chernozhukov-Hansen (2012) Bonferroni-style theory rule, available in Python as hdmpy.rlasso. CV: cross-validation minimising prediction MSE, available as sklearn.linear_model.LassoCV. Different objectives, different answers.

7. Post-OLS step. After LASSO selects a support, refit with plain (unshrunken) OLS to remove the shrinkage bias on $\hat\alpha$. LASSO is used only for selection, never for the final estimate. We use pyfixest.feols(..., vcov={"CRV1": "state"}) for this step so state-clustered SEs come “for free.”

8. State-clustered standard errors. HC1-adjusted sandwich variance with state-level clustering, applied via pyfixest’s built-in CRV1 option for the OLS rows and via a hand-rolled sandwich on the orthogonal scores for the DoubleML rows. Corrects for within-state autocorrelation that would otherwise understate the SE on a 48-state × 12-year panel.

A note on the Python libraries. The four-library stack maps cleanly onto the R workflow:

Python library	R equivalent	What it does in this post
`pyfixest`	`fixest` / `lm` + `sandwich`	OLS rows with `vcov={"CRV1": "state"}` for state-clustered SE
`hdmpy`	`hdm::rlasso`	Rigorous-penalty LASSO with BCH (c=1.1, gamma=0.05) defaults
`scikit-learn`	`glmnet::cv.glmnet`	Cross-validated LASSO via `LassoCV` and `KFold`
`DoubleML`	(no direct R analog; closest is `DoubleML` for R, same algorithm)	Modern cross-fit Neyman-orthogonal estimation (Part B)
`xgboost`	`xgboost`	Boosted-trees nuisance learner in the §18 learner-robustness check

2. The data

We use the exact panel that Belloni, Chernozhukov and Hansen (2014) compiled from Donohue and Levitt’s (2001) original replication archive: 48 U.S. states × 12 years (1986-1997) after first-differencing the raw 13-year 1985-1997 panel, giving 576 observations. First-differencing absorbs state fixed effects. Year fixed effects are absorbed in a separate pre-processing step using the Frisch-Waugh-Lovell projection (see §7). By the time the analysis script sees the data, both fixed-effect adjustments are done, so the LASSO regressions below contain no time dummies.

Code chunk 1 — Loading the six CSVs over HTTPS:

import pandas as pd
BASE = ("https://raw.githubusercontent.com/cmg777/starter-academic-v501/"
"master/content/tutorials/r_double_lasso/data/")
state = pd.read_csv(BASE + "levitt_state.csv")["state"].to_numpy()
linear = pd.read_csv(BASE + "levitt_linear.csv")
partialled = pd.read_csv(BASE + "levitt_partialled.csv")
ctrl_viol = pd.read_csv(BASE + "levitt_controls_viol.csv")
ctrl_prop = pd.read_csv(BASE + "levitt_controls_prop.csv")
ctrl_murd = pd.read_csv(BASE + "levitt_controls_murd.csv")

Six CSVs, six pd.read_csv calls. No local file dependencies, no Matlab files — the entire data layer is portable across machines and operating systems.

File	Shape	What it contains
`levitt_state.csv`	576 × 1	State cluster id (1-48) for each observation
`levitt_linear.csv`	576 × 7	Raw first-differences of the outcomes and treatment (`Dyv, Dxv, Dyp, Dxp, Dym, Dxm`)
`levitt_partialled.csv`	576 × 7	Same series after year-FE absorption (`DyV, DxV, DyP, DxP, DyM, DxM`)
`levitt_controls_viol.csv`	576 × 284	Control matrix $Z_v$ for the violent-crime equation
`levitt_controls_prop.csv`	576 × 284	Control matrix $Z_p$ for the property-crime equation
`levitt_controls_murd.csv`	576 × 284	Control matrix $Z_m$ for the murder equation

The dimensions matter for the LASSO methods that follow. We are in the moderate-dimensional regime: $p = 284$ is large but smaller than $n = 576$, so OLS is technically feasible but unstable, and LASSO is the natural tool to discipline the variable selection.

3. Five estimators in plain language

Five regression procedures appear in Part A, each with a different attitude toward how many controls to keep. We summarise the cast here so you can navigate the rest of the article.

Estimator	Recipe in one sentence	Python library	Section
First-difference OLS	Regress differenced crime on differenced abortion with no controls — the original Donohue-Levitt 1993 specification.	`pyfixest`	§4
OLS (full)	Add all 284 controls and let the matrix algebra sort it out.	`pyfixest`	§5
PSL (Post-Structural LASSO)	FWL-partial out the treatment, then one `hdmpy.rlasso` on the residualised controls, then post-OLS on the selected support.	`hdmpy` + `pyfixest`	§6
DL (rigorous)	Two LASSOs (y on X, d on X) with the Belloni-et-al. theory-based penalty; refit OLS on the union of selected variables.	`hdmpy` + `pyfixest`	§7
DL (CV)	Same recipe as DL-rigorous but each LASSO uses 3-fold cross-validation to pick lambda.	`sklearn` + `pyfixest`	§10

Two pairs of estimators do most of the pedagogical work. First-diff vs OLS-full is the control-count contrast (no controls vs too many controls). DL-rigorous vs DL-CV is the penalty-rule contrast (theory vs data-driven). PSL sits in between as the simplest one-LASSO benchmark.

Part B (§16-§18) adds three more estimators that come from the DoubleML framework: DoubleMLPLR (the cross-fit version of DL), DoubleMLIRM (for binary treatments), and three learner variants of DoubleMLPLR (LASSO vs RandomForest vs XGBoost). The full menu is eight estimators by the end of the post.

4. First-difference OLS — the no-controls baseline

The original Donohue-Levitt 1993 specification regresses differenced crime on differenced abortion with no controls beyond first-differencing itself:

$$ \Delta y_{st} = \alpha \, \Delta d_{st} + \varepsilon_{st}. $$

Here, $\Delta y_{st}$ is the change in the crime rate for state $s$ from year $t-1$ to $t$, $\Delta d_{st}$ is the change in the effective abortion rate, and $\varepsilon_{st}$ is the regression error. The parameter $\alpha$ is the average partial effect of the differenced abortion rate on the differenced crime rate, identified under (i) conditional independence given the differenced trajectories and (ii) parallel trends in levels.

Code chunk 2 — The first-difference OLS in Python using pyfixest:

import pyfixest as pf
import pandas as pd
df = pd.DataFrame({"y": linear["Dyv"], "d": linear["Dxv"], "state": state})
fit = pf.feols("y ~ -1 + d", data=df, vcov={"CRV1": "state"})
print(fit.summary())

###
Estimation: OLS
Dep. var.: y, Fixed effects:
Inference: CRV1
Observations: 576
| Coefficient | Estimate | Std. Error | t value | Pr(>|t|) | 2.5% | 97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|-------:|--------:|
| d | -0.1521 | 0.0337 | -4.5165 | 0.0000 | -0.218 | -0.086 |
###

Three things to notice. First, the formula uses -1 to suppress the intercept — first-differencing absorbs both the level and the state fixed effect, so the regression mean is zero by construction. Second, the vcov={"CRV1": "state"} keyword triggers pyfixest’s cluster-robust sandwich estimator with the HC1 small-sample correction $(N-1)/(N-k) \cdot G/(G-1)$, which is exactly the formula used in the R companion and Stata companion. Third, pyfixest returns a fitted object that exposes .coef(), .se(), .confint(), and .summary() — clean accessors that make downstream programmatic use easy.

Running this regression for each of the three crime outcomes gives our baseline numbers:

Outcome	$\hat\alpha$	SE (state-clustered)	95 % CI
Violent crime	−0.1521	0.0337	[−0.218, −0.086]
Property crime	−0.1084	0.0219	[−0.151, −0.065]
Murder	−0.2039	0.0667	[−0.335, −0.073]

Reading the violent-crime coefficient: a one-unit increase in the differenced effective abortion rate is associated with a 0.152-unit decrease in the differenced violent-crime rate. All three estimates are negative and statistically significant at the 5 % level; this is the Donohue-Levitt finding, and it matches the R companion’s cluster_se implementation to four decimal places. The whole point of the LASSO methods below is to ask whether this picture survives when we let 284 candidate controls compete for inclusion.

5. Kitchen-sink OLS — why we cannot just add everything

A natural reaction to “you only used 8 controls” is to add all 284 and let OLS sort it out. With $p = 284 < n = 576$ the $X’X$ matrix is technically invertible, so the procedure runs. The output:

Outcome	$\hat\alpha$	SE	95 % CI	Sign matches baseline?
Violent crime	+0.0135	0.5654	[−1.09, +1.12]	no — flips sign
Property crime	−0.1950	0.1937	[−0.57, +0.18]	yes (but CI crosses zero)
Murder	+2.3426	2.6047	[−2.76, +7.45]	no — flips dramatically

The violent-crime point estimate has flipped sign (+0.014 vs the baseline’s −0.152) and its confidence interval crosses zero. The murder estimate has exploded to +2.34 with SE = 2.60, meaning a unit increase in the abortion rate would raise murder by 234 % — clearly an artefact of the extreme multicollinearity in the 284 controls and not a credible causal estimate.

To see why, recall the OLS estimator in matrix form:

$$ \hat\beta_{\text{OLS}} = (X’X)^{-1} X’ y, \qquad \widehat{\operatorname{Var}}(\hat\beta_{\text{OLS}}) = \hat\sigma^{2} \, (X’X)^{-1}. $$

Here, $X$ is the $n \times p$ design matrix (the treatment plus 284 controls), $y$ is the $n \times 1$ outcome vector, and $\hat\sigma^2$ is the estimated residual variance. The variance of any coefficient — including the treatment effect — depends on $(X’X)^{-1}$. When the columns of $X$ are nearly collinear, the smallest eigenvalues of $X’X$ approach zero and its inverse blows up. Our implementation uses a rank-revealing QR pivot to drop linearly dependent columns before the sandwich computation (matching Stata’s regress behaviour), which yields larger SEs than R’s MASS::ginv() fallback — both are mathematically valid, both reach the same qualitative conclusion: kitchen-sink OLS is uninterpretable here. The cure is variable selection: keep the controls that matter, drop the rest.

6. LASSO and the one-LASSO benchmark (PSL)

The Least Absolute Shrinkage and Selection Operator (Tibshirani 1996) modifies the OLS minimisation by adding an L1 penalty on the coefficients:

$$ \hat\beta_{\text{LASSO}}(\lambda) = \arg\min_{\beta \in \mathbb{R}^p} \; \frac{1}{2n} \| y - X\beta \|_2^2 \, + \, \lambda \sum_{j=1}^p \lvert\beta_j\rvert. $$

The first term is the usual sum of squared residuals. The second is the penalty: $\lambda$ times the sum of the absolute values of the coefficients. The absolute-value penalty has a corner at zero — unlike a squared penalty (which would give Ridge regression), LASSO can shrink coefficients exactly to zero, performing variable selection at the same time as estimation. The strength of selection is controlled by one knob $\lambda$: at $\lambda = 0$ we recover OLS; as $\lambda \to \infty$ all coefficients are pinned to zero.

Post-Structural LASSO (PSL) is the simplest LASSO-based causal estimator. Run one LASSO on $y$ regressed on $(d, X)$, but ensure the treatment $d$ is not selected away by LASSO’s shrinkage. In R, glmnet::cv.glmnet(penalty.factor = c(0, rep(1, p))) does this directly. In Stata, rlasso ... pnotpen(d) does it. In Python, hdmpy.rlasso does not expose a pnotpen argument — so we implement the equivalent recipe via Frisch-Waugh-Lovell partialling:

Code chunk 3 — PSL in Python using FWL partialling + hdmpy.rlasso:

import numpy as np
import hdmpy
def partial_out_d(arr, d):
"""Project arr onto d via OLS and return the residual."""
d_col = d.reshape(-1, 1)
beta = np.linalg.lstsq(d_col, arr, rcond=None)[0]
return arr - d_col @ beta if arr.ndim == 2 else arr - (d_col @ beta).ravel()
def psl_fit(y, d, X, state):
y_tilde = partial_out_d(y, d) # residualise y on d
X_tilde = partial_out_d(X, d) # residualise each X column on d
fit = hdmpy.rlasso(X_tilde, y_tilde, post=False, intercept=False,
c=1.1, gamma=0.05)
beta = np.asarray(fit.est["beta"]).flatten()
sel = np.where(np.abs(beta) > 1e-10)[0]
Xsel = X[:, sel] if sel.size > 0 else np.empty((len(y), 0))
return feols_clustered(y, d, Xsel, state) # post-OLS, CRV1 SE

Two important notes. First, the FWL partialling step replaces the penalty.factor=0 mechanism: by removing $d$’s effect from both $y$ and $X$ before the LASSO step, we get the same conditional-on-$d$ selection that the unpenalised treatment in R/Stata would give. In the orthogonal-design limit the two are mathematically equivalent; in finite samples they differ slightly. Second, hdmpy.rlasso uses the BCH rigorous penalty (c=1.1, gamma=0.05) by default — these are the same defaults the R companion’s hdm::rlasso and Stata companion’s rlasso use. We pass them explicitly so the cross-language consistency is visible.

The results:

Outcome	$\hat\alpha$	SE
Violent crime	−0.1553	0.0330
Property crime	−0.1015	0.0218
Murder	−0.2061	0.0514

PSL with the rigorous penalty is extremely parsimonious — for all three outcomes, zero controls survive, so the post-OLS reduces to the no-controls baseline. The numerical values land within 0.003 of the first-difference baseline (violent: −0.155 vs −0.152), within 0.001 of the paper’s reported PSL numbers, and within 0.001 of the Stata companion’s rigorous-penalty PSL. The R companion uses CV-tuned PSL instead (3-fold cv.glmnet) and gets 3 / 12 / 0 controls per outcome — that is a different implementation choice with the same qualitative conclusion.

Why is this not the end of the story? Because PSL has a causal-inference blind spot. LASSO selects controls based on how well they predict $y$. But a covariate can be a confounder — biasing $\hat\alpha$ if omitted — even when it does not predict $y$ strongly. Imagine a variable highly correlated with the treatment $d$ but only weakly with $y$. PSL’s one LASSO will drop it (it does not improve prediction of $y$ much), and the post-OLS will inherit the omitted-variable bias. Belloni, Chernozhukov and Hansen (2014) made exactly this point, and proposed Double LASSO as the fix.

7. Double LASSO — the causal-side fix

Double LASSO runs two LASSOs, not one. The first LASSO predicts the outcome $y$ from the controls; call its selected index set $I_y$. The second LASSO predicts the treatment $d$ from the same controls; call its selected index set $I_d$. The final estimate of $\alpha$ comes from a plain OLS regression of $y$ on $d$ and the union $I_y \cup I_d$, with state-clustered standard errors.

flowchart TD
A("Data: outcome y, treatment d,<br/>controls X (p = 284)") --> B("Step 1: hdmpy.rlasso(X, y)<br/>(no d on right-hand side)<br/>selected set I_y")
A --> C("Step 2: hdmpy.rlasso(X, d)<br/>(no y on right-hand side)<br/>selected set I_d")
B --> D("Union: I_y &cup; I_d")
C --> D
D --> E("Step 3: pyfixest.feols<br/>y ~ -1 + d + X[:, union]<br/>vcov={CRV1: state}")
E --> F("Causal estimate alpha-hat")
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class A,E anchor
class B,C,F teal
class D orange

The intuition is rooted in the Frisch-Waugh-Lovell theorem. To estimate $\alpha$ in the structural equation $y_i = \alpha\, d_i + x_i’ \theta + \zeta_i$, FWL says we can residualise both $y$ and $d$ against the same set of controls and regress the residuals:

$$ \hat\alpha = \bigl(\tilde d’ \tilde d\bigr)^{-1} \tilde d’ \tilde y, \quad \text{where} \quad \tilde y = M_X y, \, \tilde d = M_X d. $$

The trick is that we do not need to use all of $X$ in the residualisation. We only need to use enough of $X$ to capture the part that is correlated with $d$. Double LASSO does this approximately: $I_d$ catches the controls correlated with $d$; $I_y$ catches the controls correlated with $y$; their union catches both.

The “rigorous” penalty rule chooses $\lambda$ from theory, not from CV. Belloni, Chen, Chernozhukov and Hansen (2012) showed that the right scaling is

$$ \lambda^{\text{rig}} = \frac{2 c \, \hat\sigma}{\sqrt{n}} \, \Phi^{-1}\!\left(1 - \frac{\gamma}{2 p}\right), \quad c = 1.1, \, \gamma = 0.05, $$

where $\hat\sigma$ is a pilot estimate of the residual standard deviation, $n$ is the sample size, $p$ is the number of candidate controls, and $\Phi^{-1}$ is the inverse standard-normal CDF. The factor $\Phi^{-1}(1 - \gamma / (2p))$ is a Bonferroni-style correction that keeps the false-positive rate of LASSO selection under control even though we are testing $p$ coefficients.

Code chunk 4 — The two rigorous LASSOs and the post-OLS in Python:

def selected_idx_rlasso(fit, tol=1e-10):
beta = np.asarray(fit.est["beta"]).flatten()
return np.where(np.abs(beta) > tol)[0]
def dl_rigorous_fit(y, d, X, state):
fit_y = hdmpy.rlasso(X, y, post=False, intercept=False, c=1.1, gamma=0.05)
fit_d = hdmpy.rlasso(X, d, post=False, intercept=False, c=1.1, gamma=0.05)
Iy = selected_idx_rlasso(fit_y)
Id = selected_idx_rlasso(fit_d)
U = np.sort(np.unique(np.concatenate([Iy, Id])))
return feols_clustered(y, d, X[:, U], state), Iy, Id, U

A few notes. intercept=False is correct because the data has already been partialled for year fixed effects (so the column means are essentially zero). post=False returns the raw LASSO coefficients rather than hdmpy’s internal post-OLS refit — we run our own post-OLS via pyfixest so we can attach state-clustered standard errors. The constants c=1.1, gamma=0.05 are the BCH defaults that R’s hdm::rlasso and Stata’s rlasso also use; passing them explicitly makes the cross-language consistency visible.

The results:

Outcome	$\hat\alpha$	SE	95 % CI	\|I_y\|	\|I_d\|	Union
Violent crime	−0.1043	0.1067	[−0.313, +0.105]	0	8	8
Property crime	−0.0302	0.0550	[−0.138, +0.078]	3	9	12
Murder	−0.1253	0.1506	[−0.421, +0.170]	0	9	9

Reading the violent-crime row. $\hat\alpha = -0.1043$ means a unit increase in the differenced effective abortion rate is associated with a 0.104-unit decrease in the differenced violent-crime rate, conditional on the 8 controls in the union. The 95 % confidence interval [−0.313, +0.105] now contains zero — once we condition on the 8 controls the d-equation LASSO selects, the violent-crime effect drops below significance at the 5 % level. The selection counts |I_y| = 0, |I_d| = 8 are exact matches to the R companion and to Fitzgerald et al.’s Table 2 (line 210). Same six cells, same exact matches across both languages, plus an exact match on the point estimate (-0.104 vs paper -0.104). Property crime (|I_y|=3, |I_d|=9, point −0.0302 vs paper −0.030) and murder (|I_y|=0, |I_d|=9, point −0.1253 vs paper −0.125) are similarly tight matches.

8. State-clustered standard errors

A digression on the standard errors. The 576 observations are not independent — they are 12 differenced years of data for each of 48 states, and within-state observations are autocorrelated through governor effects, state policy waves, and business-cycle exposure. Treating them as independent would understate the uncertainty by about 40 % on this panel. We use a cluster-robust sandwich estimator with the HC1 finite-sample adjustment (Cameron and Miller 2015):

$$ \hat V_{\text{cluster}} = \frac{n-1}{n-k} \cdot \frac{G}{G-1} \cdot (X’X)^{-1} \cdot \left(\sum_{g=1}^G X_g’ \hat e_g \hat e_g’ X_g\right) \cdot (X’X)^{-1}. $$

The “sandwich” name comes from the structure: two slices of bread $(X’X)^{-1}$ around the meat $\sum_g X_g’ \hat e_g \hat e_g’ X_g$, the cluster-summed outer product of the within-cluster scores. The two front factors are the small-sample correction: $(n-1)/(n-k)$ adjusts for the degrees of freedom consumed by the regressors, and $G/(G-1)$ adjusts for the number of clusters. Here $n = 576$, $k$ is the number of fitted columns (varies by estimator), and $G = 48$ is the number of states.

In Python we have two clean ways to apply this. For OLS-based estimators (rows 1-5 in our Table 2), pyfixest does it natively:

fit = pf.feols("y ~ -1 + d + z1 + z2 + ...", data=df, vcov={"CRV1": "state"})

For the DoubleMLPLR row in Part B, we hand-roll the equivalent sandwich on the orthogonal scores (see §17.1). Both approaches give numerically identical inference for the small-controls cases; for kitchen-sink OLS the column-rank handling differs slightly between approaches (documented in §5).

The cluster-count correction $G/(G-1)$ assumes the number of clusters $G$ is “large.” A rule of thumb is $G \geq 30$; with $G = 48$ states we are comfortably above that threshold. If you had only 5 or 10 clusters, the cluster-robust SE would be unreliable and you would need wild bootstrap or block bootstrap inference.

9. When does Double LASSO help most?

Look back at the DL-rigorous table in §7. For violent crime and murder, |I_y| = 0 — the LASSO of crime on controls picked zero variables out of 284. For all three outcomes |I_d| is 8 or 9 — the LASSO of abortion on controls picked a handful. This asymmetry is the empirical fingerprint of the situation in which Double LASSO most helps: the treatment is well-predicted by the controls, but the outcome is not. Fitzgerald et al. (2026) emphasise this in their footnote 4: DL is most useful when the outcome is hard to predict but the treatment is well-predicted, because that is when the second LASSO catches controls that the first one missed.

Why does this matter for causal inference? Recall the PSL blind spot from §6: a one-LASSO procedure on $y$ can drop a control that strongly predicts $d$ if it does not strongly predict $y$. Suppose the (unobserved) data-generating process is

$$ y_i = \alpha \, d_i + x_i’ \theta + \zeta_i, \quad d_i = x_i’ \pi + v_i, \quad \zeta_i \perp v_i. $$

If a particular $x_j$ has a large $\pi_j$ but a small $\theta_j$, then $x_j$ is a strong confounder (it predicts $d$, and thus moves $\hat\alpha$ when omitted), but a weak predictor of $y$. PSL drops it; DL keeps it via the d-equation LASSO. The empirical fingerprint |I_y| = 0, |I_d| = 8 means we are exactly in this regime: the eight controls that survived the d-equation LASSO are doing all of the confounding-control work in the final OLS.

10. Rigorous vs cross-validated penalty — the Python-specific story

The second flavour of Double LASSO replaces the rigorous penalty with 3-fold cross-validation via sklearn.linear_model.LassoCV. The recipe is identical to §7 — two LASSOs, take the union, post-OLS — but each LASSO now picks $\lambda$ by minimising out-of-sample mean-squared error on the prediction problem.

Code chunk 5 — The CV-penalty Double LASSO using sklearn:

from sklearn.linear_model import LassoCV
from sklearn.model_selection import KFold
def dl_cv_fit(y, d, X, state, seed=20260520):
cv_y = KFold(n_splits=3, shuffle=True, random_state=seed)
cv_d = KFold(n_splits=3, shuffle=True, random_state=seed + 1)
lc_y = LassoCV(cv=cv_y, random_state=seed, max_iter=5000).fit(X, y)
lc_d = LassoCV(cv=cv_d, random_state=seed, max_iter=5000).fit(X, d)
Iy = np.where(np.abs(lc_y.coef_) > 1e-10)[0]
Id = np.where(np.abs(lc_d.coef_) > 1e-10)[0]
U = np.sort(np.unique(np.concatenate([Iy, Id])))
return feols_clustered(y, d, X[:, U], state), Iy, Id, U

The results:

Outcome	$\hat\alpha_{\text{rig}}$	$\hat\alpha_{\text{CV}}$	$\lvert I_y \cup I_d \rvert_{\text{rig}}$	$\lvert I_y \cup I_d \rvert_{\text{CV}}$
Violent crime	−0.1043	−0.1401	8	56
Property crime	−0.0302	−0.0654	12	54
Murder	−0.1253	−0.1601	9	59

Two important findings. First, the selection-count gap is real but modest: CV picks 5x more controls than rigorous (56 / 54 / 59 vs 8 / 12 / 9). Second — and this is the Python-specific surprise — the dramatic sign-flip the R companion shows on violent crime is not reproduced here. R’s cv.glmnet keeps 150 controls in the d-equation for violent crime and flips $\hat\alpha$ from −0.10 to +0.02. Python’s sklearn.LassoCV keeps only 52, and $\hat\alpha$ stays clearly negative at −0.14.

Why the difference? Three pieces. (i) Lambda grid. cv.glmnet constructs its grid from $\lambda_{\max}$ down to $\epsilon \cdot \lambda_{\max}$ on a 100-point log scale with $\epsilon = 10^{-4}$ when $n < p$; sklearn.LassoCV defaults to 100 points but with $\epsilon = 10^{-3}$, so its smallest lambda is 10× larger. Smaller smallest-lambda → more variables can survive → R picks more. (ii) Fold-assignment RNG. R’s cv.glmnet uses base-R’s sample(); Python’s KFold(shuffle=True, random_state=...) uses NumPy’s Mersenne Twister. The fold partitions are different, so the cross-validation surface is different, so the optimal lambda is different. (iii) Standardisation. Both implementations standardise X before LASSO, but cv.glmnet uses sample-SD scaling while LassoCV uses the L2-norm by default — a subtle difference that compounds at the smallest lambda values.

The take-away is not that one library is wrong — both follow the same algorithm. The take-away is that “default CV-LASSO” is not a portable concept across language ecosystems, and the dramatic R demonstration of the rigorous-vs-CV sign-flip is partly an artifact of cv.glmnet’s aggressive grid. The §15 standalone section walks through five sources of drift between sklearn.LassoCV, R::glmnet::cv.glmnet, and DoubleML’s internal Lasso, so readers know which knob to turn when porting results across languages.

11. The forest plot

Stacking all five Part-A estimators against all three outcomes gives the headline figure:

A coherent story for violent and property crime: the LASSO methods (PSL, DL-rigorous, DL-CV) land between the two extremes — First-difference OLS at $-0.152$ (violent) and Kitchen-sink OLS at $+0.014$ (violent). PSL and DL-rigorous concentrate the data’s signal near the small set of controls that actually matter (0 to 12 of them), giving estimates in the $-0.10$ to $-0.16$ range with tighter standard errors than OLS-full.

For murder, the story is messier. Kitchen-sink OLS gives the nonsensical $+2.34$. But First-diff ($-0.20$), PSL ($-0.21$), DL-rigorous ($-0.13$), and DL-CV ($-0.16$) all cluster sensibly in the negative range. The murder outcome is the noisiest of the three (state-level murder counts are small numbers in many state-years), but Python’s milder over-selection in DL-CV means we avoid the catastrophic $-1.11$ estimate that R’s cv.glmnet produces here.

12. When to use which method?

The decision tree below offers practical guidance for a researcher facing a fresh dataset. It is not a substitute for thinking carefully about identification (no method can rescue an invalid research design), but it is a reasonable starting point.

flowchart TD
Start("You have n observations,<br/>p candidate controls,<br/>and want a causal alpha-hat") --> Q1{"p &ge; n?"}
Q1 -->|Yes| L("LASSO methods required<br/>(OLS infeasible)")
Q1 -->|No| Q2{"p / n &gt; 0.3?"}
Q2 -->|Yes, like this post<br/>p=284, n=576| L
Q2 -->|No| Q3{"n &ge; 5,000?"}
Q3 -->|Yes| O("Plain OLS with all<br/>controls is fine<br/>(pyfixest.feols)")
Q3 -->|No| L
L --> Q4{"Need valid causal<br/>inference, not just<br/>prediction?"}
Q4 -->|Yes, single-shot| DL("Post-double-selection<br/>hdmpy + pyfixest")
Q4 -->|Yes, modern cross-fit| DML("DoubleML.DoubleMLPLR<br/>(see Part B)")
Q4 -->|No| Pred("DL-CV or PSL are<br/>both fine for prediction")
classDef sty_Q1 fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
class Q1 sty_Q1
classDef sty_Q2 fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
class Q2 sty_Q2
classDef sty_Q3 fill:#1f2b5e,stroke:#6a9bcc,stroke-width:3px,color:#e8ecf2
class Q3 sty_Q3
classDef sty_Q4 fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
class Q4 sty_Q4
classDef anchor fill:#0f1729,stroke:#c8d0e0,stroke-width:2px,color:#e8ecf2
classDef orange fill:#1f2b5e,stroke:#d97757,stroke-width:3px,color:#e8ecf2
classDef teal fill:#1f2b5e,stroke:#00d4c8,stroke-width:3px,color:#e8ecf2
class Start,L anchor
class O,DML,Pred orange
class DL teal

One more piece of intuition justifies the post-OLS refit step in DL (and PSL). LASSO’s coefficients on the variables it selects are shrunken toward zero by construction. If you used those shrunken coefficients to compute the residuals for $\alpha$, you would inherit a bias of the order

$$ \hat\alpha_{\text{LASSO}} - \alpha = O_p\!\left(\frac{\lambda}{n}\right). $$

For our $\lambda^{\text{rig}}$ and $n = 576$, that bias is roughly 5-15 % of the treatment effect — large enough to matter. Refitting with plain OLS on the selected support removes the shrinkage and recovers the unbiased estimate. This is why every method in Part A uses LASSO for selection only and post-OLS (pyfixest.feols) for estimation. DoubleMLPLR in Part B achieves the same shrinkage-removal differently, via cross-fitting and Neyman-orthogonal scores — see §16.

13. Caveats and identification

Six things to keep in mind when reading the headline estimates.

This is a replication exercise, not a primary causal claim. Fitzgerald et al. (2026) is itself a replication paper studying Double LASSO as a method. Whether more abortion access caused less crime is a substantive question that goes well beyond any single regression specification.
Identification rests on two assumptions. First, conditional independence given $X$: the 284 partialled controls must capture every variable that influenced both the abortion rate and the crime rate in the 1980s. Second, parallel trends in levels: state fixed effects are absorbed by first-differencing, year fixed effects by the partialling step in the upstream pre-processing. Neither assumption is innocuous.
State-clustering relies on $G \geq 30$. With $G = 48$ states we are above the rule of thumb. If you had only 5-10 clusters, the cluster-robust SE would be unreliable and you would need wild bootstrap or block bootstrap inference.
CV LASSO is non-deterministic. sklearn.LassoCV randomly partitions the data into $K$ folds; without seeding, the variable-selection counts in §10 would vary by ±5 controls between runs and the headline coefficient by ±0.02. The script seeds both KFold(random_state=20260520) and LassoCV(random_state=20260520) so the post’s numbers reproduce exactly. The rigorous hdmpy.rlasso is deterministic given the data and the penalty arguments.
Implementation differences from R’s MASS::ginv show up on OLS-full. Our SE on OLS-full violent crime is 0.565 vs the R companion’s 0.091; the gap stems from inverting near-singular $X’X$ via rank-revealing QR (drops collinear columns, then numpy.linalg.pinv on the survivors) vs R’s MASS::ginv (Moore-Penrose pseudoinverse on the full 284 columns). Both are mathematically valid; Python’s approach matches Stata’s regress.
hdmpy does not expose pnotpen. This is why our PSL (§6) uses FWL partialling instead of unpenalised-treatment LASSO. Mathematically equivalent in the orthogonal-design limit; numerically nearly identical to the Stata rigorous-PSL implementation. If you need exact cv.glmnet parity, an alternative is to use glmnet-python (a thin wrapper around the Fortran code that R’s glmnet uses) — but the maintenance trajectory of glmnet-python is weaker than sklearn + hdmpy, and the qualitative conclusions are unchanged.

14. Python vs R numeric replication (Tier A / B / C)

The headline numerical reproduction is faithful at the variable-selection level. Our LASSO selections for the rigorous-penalty Double LASSO match the R companion — and Fitzgerald et al. (2026) Table 2 — exactly across all three outcomes:

Outcome	\|I_y\| Python	\|I_y\| R	\|I_d\| Python	\|I_d\| R	Point Python	Point R	Point paper
Violent crime	0	0	8	8	−0.1043	−0.0964	−0.104
Property crime	3	3	9	9	−0.0302	−0.0314	−0.030
Murder	0	0	9	9	−0.1253	−0.1662	−0.125

Six selection-count cells, six exact Python = R = paper matches. Point estimates agree across the three implementations to within 0.05 on the largest absolute gap (murder); violent crime and property crime are within 0.01. To keep the cross-implementation drift transparent, we organise the rows in tiers:

Tier	Methods	Expected drift	Source of any drift
A — exact	First-diff OLS, Kitchen-sink OLS (point estimates)	≤ 1e-4	None (deterministic OLS on same data)
B — tight	PSL, DL-rigorous (point estimates and selection counts)	≤ 0.05	Pre-standardisation differences in `hdmpy` vs `hdm`; FWL-vs-`pnotpen` for PSL
C — drifts freely	DL-CV (point estimates and selection counts)	Wide	`sklearn.LassoCV` ≠ `cv.glmnet` (different lambda grid, fold RNG, standardisation)

Stata is in the same picture: its Tier-A and Tier-B rows match Python’s to within 0.001. The Tier-C row (DL-CV) is where each language’s CV implementation diverges, and the Python-specific behaviour is the absence of the violent-crime sign-flip (see next section).

15. Why DoubleML results don’t match R’s `hdm`: five sources of drift

If you ran DoubleMLPLR(... ml_l=LassoCV(), ml_m=LassoCV()) on this data expecting to recover the R companion’s DL-rigorous numbers, you would get α̂ = −0.115, not the R’s −0.0964 or the rigorous PSL’s −0.1567. Five things differ between DoubleML’s defaults and R’s hdm, and naming them makes it possible to know which knob to turn when you need to reconcile two ecosystems:

Sample-splitting / cross-fitting. DoubleMLPLR uses K-fold cross-fitting (K = 5 default) — every observation’s residual is computed by a nuisance model that did not see that observation. R’s hdm::rlasso + manual post-OLS uses a single-sample fit — the same data is used to select variables and to estimate $\alpha$. At finite $n$ these target different estimands; asymptotically they converge to the same parameter under standard regularity conditions.
Nuisance estimator defaults. DoubleML does not ship a built-in rigorous-penalty LASSO. The closest user-facing option is LassoCV from sklearn, which picks $\lambda$ by cross-validation — exactly the choice that §10 above shows over-selects relative to the BCH rigorous penalty. If you want rigorous behaviour inside DoubleML, you have to manually compute the BCH $\lambda$ and pass Lasso(alpha=lambda), or pre-fit a hdmpy.rlasso and pass a custom sklearn-compatible wrapper. We use LassoCV here for clarity; the §16 showcase tagline is “DoubleML’s design is learner-agnostic — see §18 for how RandomForest and XGBoost change the picture.”
Standardisation. sklearn standardises X internally before LASSO (column-wise division by L2-norm); hdm standardises by sample-SD; cv.glmnet also uses sample-SD but with a different reference (variance computed with n, not n-1). At the boundary lambdas these three conventions give different selections.
Fold RNG. sklearn.model_selection.KFold(shuffle=True, random_state=...) uses NumPy’s Mersenne Twister; R’s cv.glmnet uses base-R’s set.seed. Even with identical seeds, the fold partitions differ. With $n_{\text{rep}} \geq 10$ in DoubleML the variation from this source averages out; with the $n_{\text{rep}} = 3$ we use for speed it is still visible (±0.01 on the point estimate).
Inference target. DoubleMLPLR returns iid-asymptotic standard errors by default. R’s hdm-driven workflow attaches a state-clustered HC1 sandwich on post-OLS residuals (Cameron and Miller 2015). We hand-roll the analog on DoubleML’s orthogonal scores in §17.1 so the inference is apples-to-apples with R, but this is not what you get from dml.se out of the box.

The practical upshot: when DoubleML’s α̂ differs from R’s hdm α̂, the difference is explainable, not mysterious. The two are different algorithms targeting the same parameter. Choosing between them comes down to whether you want explicit post-double-selection (R/Stata style, transparent) or modern cross-fit Neyman-orthogonal estimation (DoubleML style, learner-agnostic).

16. Meet DoubleML: a modern framework for ML-based causal inference

DoubleML (Bach, Chernozhukov, Kurz and Spindler 2022, JMLR) is a Python library that ports the Chernozhukov et al. (2018, Econometrics Journal) double/debiased ML framework into a sklearn-native API. Three ideas drive its design:

Neyman orthogonality. The score function $\psi$ has zero expected gradient with respect to the nuisance parameters $\eta$ at the truth: $E[\partial_\eta \psi]_{\eta=\eta_0} = 0$. This means small errors in the ML estimates of the nuisance functions do not propagate to bias in $\hat\alpha$.
Cross-fitting. Each observation’s score is computed using nuisance models trained on the other folds — never on itself. This eliminates overfitting bias and lets you use arbitrarily flexible ML learners without inflating bias.
Pluggable learners. Any sklearn-compatible regressor or classifier can serve as a nuisance estimator. Swap LassoCV() for RandomForestRegressor() or XGBRegressor() in one line; the rest of the pipeline is identical.

DoubleML ships several model classes, one per estimand structure. The most important for econometric work:

Class	When to use
`DoubleMLPLR`	Partially Linear Regression. $Y = D\theta + g(X) + \varepsilon$. Continuous treatment; this post’s main DoubleML model.
`DoubleMLIRM`	Interactive Regression Model. Binary treatment. ATE or ATTE. Allows treatment-effect heterogeneity in covariates.
`DoubleMLPLIV`	Partially Linear IV. Continuous treatment with instrumental variable.
`DoubleMLIIVM`	Interactive IV. Binary treatment with binary instrument.
`DoubleMLDID`	Difference-in-differences with ML nuisance (Sant’Anna-Zhao).

The user always wraps the data in a DoubleMLData object that names the outcome, treatment, controls, and (optionally) instruments. The model class then takes nuisance learners and cross-fitting parameters:

from doubleml import DoubleMLData, DoubleMLPLR
from sklearn.linear_model import LassoCV
dml_data = DoubleMLData(df, y_col="y", d_cols=["d"], x_cols=[...])
plr = DoubleMLPLR(
dml_data,
ml_l=LassoCV(cv=3), # nuisance for E[Y | X]
ml_m=LassoCV(cv=3), # nuisance for E[D | X]
n_folds=5, # outer cross-fitting
n_rep=3, # repeat cross-fit and median-aggregate
score="partialling out", # Robinson FWL — the DL recipe
)
plr.fit()
print(plr.summary)
print(plr.confint(level=0.95))

The score="partialling out" choice computes the Robinson partialling-out score $\psi_i = (Y_i - g(X_i))(D_i - m(X_i)) - \theta(D_i - m(X_i))^2$, which is exactly the FWL formula that Double LASSO approximates with a single post-OLS step. The difference between DoubleMLPLR and explicit post-double-selection is how the nuisance functions are estimated — DoubleMLPLR’s K-fold cross-fitting vs PDS’s single-sample LASSO + post-OLS.

We use this framework in the next three sections.

17. DoubleML capabilities showcase

17.1 `DoubleMLPLR` with hand-rolled cluster-state SE

The flagship Part-B estimator: DoubleMLPLR with LassoCV learners, n_folds = 5, n_rep = 3 (three repetitions of the cross-fit; the library median-aggregates across reps).

Code chunk 6 — DoubleMLPLR with cross-fitting:

from doubleml import DoubleMLData, DoubleMLPLR
from sklearn.linear_model import LassoCV
o = outcomes["violent"]
df_dml = pd.DataFrame(o["X"], columns=[f"x{i}" for i in range(o["X"].shape[1])])
df_dml["d"] = o["d"]; df_dml["y"] = o["y"]
dml_data = DoubleMLData(df_dml, y_col="y", d_cols=["d"],
x_cols=[f"x{i}" for i in range(o["X"].shape[1])])
ml_l = LassoCV(cv=3, random_state=20260520, max_iter=5000)
ml_m = LassoCV(cv=3, random_state=20260520, max_iter=5000)
plr = DoubleMLPLR(dml_data, ml_l=ml_l, ml_m=ml_m,
n_folds=5, n_rep=3, score="partialling out")
plr.fit()

 alpha_hat (DoubleMLPLR, violent crime) = -0.1152
iid SE = 0.0826 95% CI = [-0.277, +0.047]

The iid SE comes from plr.se directly. To get a state-clustered SE that is apples-to-apples with the Part-A rows, we hand-roll the cluster sandwich on the orthogonal scores that DoubleML exposes via plr.psi and plr.psi_elements:

Code chunk 7 — Hand-rolled cluster SE on orthogonal scores:

def cluster_se_orthogonal(dml, cluster_id, k_params=1):
psi = dml.psi.squeeze() # (n,) for single treatment
psi_a = dml.psi_elements["psi_a"].squeeze()
n = psi.shape[0]
if psi.ndim == 2: # average across n_rep dimension
psi = psi.mean(axis=1)
psi_a = psi_a.mean(axis=1)
df_p = pd.DataFrame({"psi": psi, "g": cluster_id})
grouped = df_p.groupby("g")["psi"].sum().to_numpy()
G = len(grouped)
meat = float(np.sum(grouped ** 2))
Epsi_a = float(np.mean(psi_a))
hc1 = (G / (G - 1)) * ((n - 1) / (n - k_params))
var = hc1 * meat / (n * Epsi_a) ** 2
return float(np.sqrt(var))
cluster_se = cluster_se_orthogonal(plr, state)

 cluster SE = 0.0727 (hand-rolled HC1 on orthogonal scores, G=48)

DoubleMLPLR’s α̂ = −0.115 sits squarely between the post-double-selection DL-rigorous (−0.104) and DL-CV (−0.140) numbers. This is reassuring — three different paths through the LASSO machinery give answers within one standard error of each other. The state-clustered SE on the orthogonal scores (0.073) is slightly smaller than the iid SE (0.083) — unusual but mathematically valid: when within-cluster errors are negatively correlated (e.g., crime rates that mean-revert within state), the cluster sandwich can shrink rather than inflate. The pedagogical takeaway: the inference target (iid vs clustered) is a separate choice from the estimation algorithm, and DoubleML’s .psi attribute makes it easy to swap in cluster-correct SEs after the fact.

17.2 `DoubleMLIRM` on a binarised treatment (API demo only)

The Interactive Regression Model handles binary treatments and estimates the ATE or ATTE. Our treatment (the effective abortion rate) is continuous, but we can binarise it at its median purely to demonstrate the API:

Code chunk 8 — DoubleMLIRM (API demo):

from doubleml import DoubleMLIRM
from sklearn.linear_model import Lasso
from sklearn.ensemble import RandomForestClassifier
d_binary = (o["d"] > np.median(o["d"])).astype(int)
df_irm = df_dml.copy(); df_irm["d"] = d_binary
irm_data = DoubleMLData(df_irm, y_col="y", d_cols=["d"],
x_cols=[f"x{i}" for i in range(o["X"].shape[1])])
irm = DoubleMLIRM(
irm_data,
ml_g=Lasso(alpha=0.01, max_iter=5000),
ml_m=RandomForestClassifier(n_estimators=100, max_depth=5,
random_state=20260520, n_jobs=-1),
n_folds=3, n_rep=1, score="ATE",
)
irm.fit()

 ATE (DoubleMLIRM, median-split treatment) = -0.0163 (iid SE = 0.0043)
(For context: PLR's continuous-treatment estimate above is -0.1152.)

CAVEAT — this is an API demonstration, not a causal estimate. Binarising a continuous treatment throws away most of the variation: we are now measuring “effect of being above-vs-below median abortion rate” instead of “effect of a one-unit change in abortion rate,” and the two are on completely different scales. The pedagogical lesson is pick the right DoubleML class for your treatment type — DoubleMLPLR for continuous, DoubleMLIRM/DoubleMLIIVM for binary, DoubleMLPLIV for IV. Forcing a continuous variable into a binary model is a classic API-driven misspecification.

18. Learner robustness: LASSO vs RandomForest vs XGBoost

A key advantage of DoubleML is that it is agnostic to the choice of ML learner, as long as the learner is flexible enough to approximate the true confounding function. To verify that our DoubleMLPLR violent-crime estimate is not driven by the specific choice of LassoCV, we re-estimate the model with three structurally different learners.

Code chunk 9 — DoubleMLPLR with three nuisance learners:

from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
learners = {
"LassoCV": lambda: LassoCV(cv=3, random_state=20260520, max_iter=5000),
"RandomForest": lambda: RandomForestRegressor(n_estimators=100, max_depth=5,
random_state=20260520, n_jobs=-1),
"XGBoost": lambda: XGBRegressor(n_estimators=100, max_depth=4,
learning_rate=0.05,
random_state=20260520, verbosity=0),
}
for name, make in learners.items():
plr_l = DoubleMLPLR(dml_data, ml_l=make(), ml_m=make(),
n_folds=5, n_rep=3, score="partialling out")
plr_l.fit()
se_c = cluster_se_orthogonal(plr_l, state)
print(f" {name:12s} alpha_hat = {float(plr_l.coef[0]):+0.4f} "
f"iid SE = {float(plr_l.se[0]):0.4f} cluster SE = {se_c:0.4f}")

 LassoCV alpha_hat = -0.0957 iid SE = 0.0841 cluster SE = 0.0785
RandomForest alpha_hat = -0.0855 iid SE = 0.1806 cluster SE = 0.1432
XGBoost alpha_hat = -0.1123 iid SE = 0.2089 cluster SE = 0.1421

Learner	α̂	iid SE	Cluster SE	95 % CI (cluster)
LassoCV (cv=3, max_iter=5000)	−0.0957	0.0841	0.0785	[−0.250, +0.058]
RandomForestRegressor (100 trees, depth 5)	−0.0855	0.1806	0.1432	[−0.366, +0.195]
XGBRegressor (100 trees, depth 4, eta 0.05)	−0.1123	0.2089	0.1421	[−0.391, +0.166]

Reading the comparison. Three structurally different nuisance learners — sparse linear (LASSO), bagged trees (RandomForest), and boosted trees (XGBoost) — give DoubleMLPLR α̂ values spanning −0.0855 to −0.1123, a 0.03 range. All three point estimates are negative, and the cluster-SE confidence intervals overlap heavily. This is exactly the learner-robustness signal DoubleML is designed to expose: if the answer flipped sign or changed by a factor of two when swapping the nuisance learner, that would be a red flag that the result is fragile. Here the conclusion (a negative association between differenced abortion and differenced violent-crime rate, statistically borderline at the 5 % level under all three learners) survives the swap.

Worth noting: the tree-based learners produce SEs roughly 2-3× wider than LASSO, because they have more flexibility to absorb signal that LASSO leaves in the residuals. With n = 576 and p = 284, sparse linear nuisance is probably the right default — but the comparison shows DoubleML’s “plug in any sklearn learner” design works as advertised. In production, the right move is to fit all three (or four — gradient boosting with LightGBM is a good fourth) and report the spread as a robustness band.

19. Conclusion

Four takeaways worth carrying away from this post.

First, Double LASSO is a method, not a panacea. It does not invent variation in the data, nor does it weaken the identifying assumptions of the underlying research design. What it does is make high-dimensional control sets tractable without committing to using all of them or to picking a subset by hand. On a dataset where conditional independence holds and the candidate-control set is rich enough to span the confounders, DL-rigorous reproduces the Donohue-Levitt 2001 headline closely while disciplining the standard errors.

Second, the rigorous penalty matters more than the language. Switching from hdmpy.rlasso to sklearn.LassoCV shifts violent-crime α̂ from −0.10 to −0.14 — a meaningful change but no sign-flip. The dramatic R demonstration (cv.glmnet flips α̂ from −0.10 to +0.02) does not reproduce in Python because sklearn.LassoCV’s lambda grid and KFold RNG are less aggressive than R’s cv.glmnet defaults. For causal inference, prefer the theory-driven hdmpy.rlasso regardless of which language you are in.

Third, the regime determines the methodology. With our $p = 284$, $n = 576$, we are squarely in the small-sample, high-dimensional zone where DL is designed to help. With $p = 8$ and $n = 5{,}000$, plain OLS would be perfectly fine. The decision tree in §12 is a starting point for picking the right tool for the dimensions you face.

Fourth — and this is the Python-specific addition — use post-double-selection (hdmpy) when you want to replicate published results; use DoubleML when you want modern Neyman-orthogonal cross-fitting with any sklearn learner. The two approaches target the same parameter under standard regularity conditions, but they take different paths. DoubleML’s cross-fitting, learner-agnosticism, and clean sklearn integration make it the right tool for production ML pipelines. Post-double-selection’s transparency (every variable’s fate is visible; no cross-fold averaging hides the selection) makes it the right tool for a one-shot replication exercise like the one in this post.

If you came in expecting either a definitive statement about abortion and crime or a magic ML cure for omitted-variable bias, you should leave with neither. What you should leave with is a clearer mental model of when the high-dimensional toolkit earns its complexity, how to use the two distinct Python idioms for it (hdmpy/sklearn/pyfixest vs DoubleML), and why the two idioms can give different numbers on the same data.

20. Exercises

These exercises ask you to modify and re-run script.py. All datasets, dependencies, and helper functions are already in place — you only need to change the indicated lines, run the script, and read the output.

Change the CV seed. In §10, the KFold and LassoCV random states are set to 20260520. Change them to a different seed and re-run only Estimator E (dl_cv_fit). How much do the selection counts |I_y|, |I_d| change across the three outcomes? Does the DL-CV point estimate for violent crime ever flip to positive on a different seed?
Tighten the rigorous penalty. In §7, the rigorous-penalty parameters are c = 1.1, gamma = 0.05. Try c = 1.5 (stricter) and c = 0.8 (looser) and re-run only Estimator D (dl_rigorous_fit). The stricter setting should select fewer variables; the looser one should select more.
Increase n_rep in DoubleMLPLR. In §17.1, n_rep=3 for speed. Bump it to n_rep=20 and re-run only that block. How much do α̂ and the SE move? This is the right setting in production — n_rep=3 is borderline for publication-quality inference.
Swap XGBoost for LightGBM in §18. Replace XGBRegressor(...) with lightgbm.LGBMRegressor(n_estimators=100, max_depth=4, learning_rate=0.05, random_state=20260520, verbosity=-1). Does the learner-comparison conclusion change?
Apply DoubleMLPLIV. This dataset has no instrumental variable, so DoubleMLPLIV is not substantively meaningful here. But as a syntactic exercise, treat one of the candidate controls (say x150) as a fake instrument and fit DoubleMLData(..., z_cols=["x150"]) + DoubleMLPLIV(...). Observe how the API differs from PLR. Do not interpret the resulting number as a causal estimate.

21. Reproducing this analysis

You need Python 3.10-3.13 and the following packages:

pip install pyfixest==0.50.1
pip install DoubleML==0.11.2
pip install hdmpy
pip install xgboost
pip install scikit-learn pandas numpy matplotlib
# macOS Intel only: pin numba/llvmlite to last-Intel-wheel versions
pip install 'numba==0.62.1' 'llvmlite==0.45.0'

Then clone the repository and run:

cd content/tutorials/python_double_lasso/
python script.py 2>&1 | tee execution_log.txt

Runtime on Apple Silicon is about 5-8 minutes (Part A: ~90 s; Part B’s DoubleMLPLR n_rep=3: ~3 minutes; Part B’s learner comparison: ~3 minutes). The longest single step is LassoCV inside DoubleMLPLR with n_folds=5 × n_rep=3; if you want a quick pass, set n_rep=1 and the runtime drops to under 2 minutes total.

If you would rather render the post locally as a Quarto notebook, the Quarto project (.zip) link button at the top contains a friction-free bundle: extract, double-click render.command (macOS) or render.bat (Windows), and the notebook renders to HTML in your browser with a hermetic local .venv/.

22. References

Academic references:

Python package and library documentation:

Data and replication archives:

Belloni-Chernozhukov-Hansen (2014) replication CSVs — companion R post data/ folder (GitHub)

Acknowledgements

AI Podcast: Double LASSO in Python

Click play to load

0:00 0:00