Introduction to Panel Data Methods in Python

Downloads

Each dataset is available as a labeled Stata .dta and its source file.

⇩ Download all data (ZIP)stata_codebook.do

Run stata_codebook.do in Stata once to attach long-form per-variable notes to the .dta files.

Load directly in code

Every file loads straight from GitHub (raw URLs). Swap the file name to load any dataset.

Stata

* Stata 14+ : `use` reads an https URL directly
global BASE "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_panel_intro/data/"
use "${BASE}raw_data.dta", clear
describe
notes

Python

!pip install -q pyreadstat
import pandas as pd
BASE = "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_panel_intro/data/"
df = pd.read_stata(BASE + "raw_data.dta")

# load every dataset at once
files = ["raw_data", "data_panel"]
data = {f: pd.read_stata(BASE + f + ".dta") for f in files}

# pyreadstat (richest metadata) reads LOCAL files -> download first
import pyreadstat, urllib.request
urllib.request.urlretrieve(BASE + "raw_data.dta", "raw_data.dta")
df, meta = pyreadstat.read_dta("raw_data.dta")

Copy and paste this snippet in Google Colab app. https://colab.research.google.com/notebooks/empty.ipynb

R

# R : haven::read_dta auto-downloads an https URL
library(haven)
BASE <- "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_panel_intro/data/"
df <- read_dta(paste0(BASE, "raw_data.dta"))

Overview & sources

Companion data for a beginner-friendly Python tutorial that walks through seven canonical panel-data estimators — pooled OLS, the between estimator, first-differences, the within (fixed-effects) estimator, two-way fixed effects, random effects, and Mundlak's correlated random effects (CRE) — on a single worker wage panel. The running question is whether union membership raises wages. The data are real NLSY-style observations on US workers from the quarcs-lab/data-open repository (isds/wage_panel_bob4.dta). The cross-sectional estimators report a union premium of 7–11 log points; once unobserved worker traits are netted out, the within estimators report roughly 21 log points — a near-tripling that illustrates selection on unobservables. The Hausman test fails to reject random effects (H = 1.79, p = 0.180) while the Mundlak term (−0.144, p = 0.072) hints at negative selection; both point toward CRE/Mundlak as the specification to lead with.

Data sources

Cite this data

Please cite this dataset as follows.

APA

Mendez, C. (2026). Introduction to Panel Data Methods in Python [Data set]. https://carlos-mendez.org/post/python_panel_intro/

Wooldridge, J. M. (2010). Econometric Analysis of Cross Section and Panel Data (2nd ed.). MIT Press. Hausman, J. A. (1978). Specification Tests in Econometrics. Econometrica, 46(6), 1251–1271. Mundlak, Y. (1978). On the Pooling of Time Series and Cross Section Data. Econometrica, 46(1), 69–85.

BibTeX

@misc{mendez2026pythonpanelintro,
  author       = {Mendez, Carlos},
  title        = {Introduction to Panel Data Methods in Python},
  year         = {2026},
  howpublished = {\url{https://carlos-mendez.org/post/python_panel_intro/}},
  note         = {Data set}
}

@book{wooldridge2010econometric,
  author    = {Wooldridge, Jeffrey M.},
  title     = {Econometric Analysis of Cross Section and Panel Data},
  edition   = {2nd}, publisher = {MIT Press}, year = {2010}
}
@article{hausman1978specification,
  author  = {Hausman, Jerry A.},
  title   = {Specification Tests in Econometrics},
  journal = {Econometrica},
  volume  = {46}, number = {6}, pages = {1251--1271}, year = {1978}
}
@article{mundlak1978pooling,
  author  = {Mundlak, Yair},
  title   = {On the Pooling of Time Series and Cross Section Data},
  journal = {Econometrica},
  volume  = {46}, number = {1}, pages = {69--85}, year = {1978}
}

Variable explorer search & filter all 10 variables

Type to filter by name or label, or use the chips to filter by type. Each row shows a mini distribution. Click a header to sort.

Cross-file variable index

Which file each variable appears in (● = present).

Construction & formulas

All estimators target the coefficient β on union in the panel model y_it = α_i + β x_it + u_it, where y is lwage, x is union, and α_i is the unobserved worker effect. They differ in which variation identifies β.

• Pooled OLS (POLS): OLS on every row, ignoring the panel — uses all variation.
• Between: OLS on worker means &ybar;_i on &xbar;_i — cross-sectional variation only.
• First-differences (FDFE): regress Δy_it on Δx_it; α_i cancels by differencing.
• Within / Fixed effects (FE): OLS on demeaned data &xtilde;_it = x_it − &xbar;_i; α_i vanishes by demeaning. With T = 2, FE ≈ FDFE.
• Two-way FE (TWFE): absorbs worker effects α_i and year effects δ_t.
• Random effects (RE): GLS treating α_i as a random draw uncorrelated with the regressors — a weighted blend of between and within.
• Correlated random effects (CRE / Mundlak): RE plus the worker mean &xbar;_i of each time-varying regressor; the within coefficient equals FE and the mean coefficient tests RE-vs-FE.

Between/within variance decomposition (per variable): between SD is SD(&xbar;_i), within SD is SD(x_it − &xbar;_i), and the between share is between² / (between² + within²). Hausman: H = (β_FE − β_RE)² / (V_FE − V_RE) ~ χ²(1).

The datasets

Switch datasets with the tabs. Each shows the full variable dictionary plus a sortable statistics table with mini distributions and data coverage.

Expand all datasets expand to search (Ctrl/⌘+F) or print across all datasets

Known limitations & caveats

• Two periods only in the analysis panel. data_panel keeps just 2010 and 2012 (T = 2) so the first-difference and within estimators coincide; with only two waves the fixed-effects estimate is power-limited and the Hausman test has low power. raw_data retains all five waves for extensions.
• Thin within variation. Union status is 93.9% between-worker and only 9.1% within; schooling has zero within-variation in the two-period window, so fixed effects mechanically drops it.
• 10 workers dropped. The analysis panel keeps 2,199 of the 2,209 raw workers; ten are dropped for missing values in lwage/union/age/schooling.
• Estimands differ. Within estimators (FDFE/FE/TWFE/CRE) identify the effect for union switchers under strict exogeneity; POLS/Between report a population-weighted association without a causal interpretation absent unconfoundedness.