Pooled PCA for Building Development Indicators Across Time

Downloads

Each dataset is available as a labeled Stata .dta and its source file.

⇩ Download all data (ZIP)stata_codebook.do

Run stata_codebook.do in Stata once to attach long-form per-variable notes to the .dta files.

Load directly in code

Every file loads straight from GitHub (raw URLs). Swap the file name to load any dataset.

Stata

* Stata 14+ : `use` reads an https URL directly
global BASE "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_pca2/data/"
use "${BASE}data.dta", clear
describe
notes

Python

!pip install -q pyreadstat
import pandas as pd
BASE = "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_pca2/data/"
df = pd.read_stata(BASE + "data.dta")

# load every dataset at once
files = ["data", "data_long"]
data = {f: pd.read_stata(BASE + f + ".dta") for f in files}

# pyreadstat (richest metadata) reads LOCAL files -> download first
import pyreadstat, urllib.request
urllib.request.urlretrieve(BASE + "data.dta", "data.dta")
df, meta = pyreadstat.read_dta("data.dta")

Copy and paste this snippet in Google Colab app. https://colab.research.google.com/notebooks/empty.ipynb

R

# R : haven::read_dta auto-downloads an https URL
library(haven)
BASE <- "https://raw.githubusercontent.com/cmg777/starter-academic-v501/master/content/post/python_pca2/data/"
df <- read_dta(paste0(BASE, "data.dta"))

Overview & sources

Companion data for a hands-on Python tutorial that builds a temporally comparable composite Human Development Index using pooled PCA and contrasts it with the naive per-period approach. The data are a South American subset of the Subnational Human Development Index from the Global Data Lab (Smits & Permanyer, 2019), with Education, Health, and Income sub-indices (plus the official SHDI and its underlying components) for 153 sub-national regions across 12 South American countries in 2013 and 2019. Pooled PCA stacks both periods and computes a single set of standardization parameters, eigenvector weights, and normalization bounds from the combined data, so a region's 2013 score is measured against the same yardstick as its 2019 score. The pooled PC1 captures 72.42% of variance with weights [0.5642, 0.5448, 0.6204] and registers a net development shift of +0.1439 that per-period PCA forces to zero by construction.

Data sources

Cite this data

Please cite this dataset as follows.

APA

Mendez, C. (2026). Pooled PCA for Building Development Indicators Across Time [Data set]. https://carlos-mendez.org/post/python_pca2/

Smits, J., & Permanyer, I. (2019). The Subnational Human Development Database. Scientific Data, 6, 190038. https://doi.org/10.1038/sdata.2019.38

BibTeX

@misc{mendez2026pythonpca2,
  author       = {Mendez, Carlos},
  title        = {Pooled PCA for Building Development Indicators Across Time},
  year         = {2026},
  howpublished = {\url{https://carlos-mendez.org/post/python_pca2/}},
  note         = {Data set}
}

@article{smits2019subnational,
  author  = {Smits, Jeroen and Permanyer, Iñaki},
  title   = {The Subnational Human Development Database},
  journal = {Scientific Data},
  volume  = {6}, pages = {190038}, year = {2019},
  doi     = {10.1038/sdata.2019.38}
}

Variable explorer search & filter all 31 variables

Type to filter by name or label, or use the chips to filter by type. Each row shows a mini distribution. Click a header to sort.

Cross-file variable index

Which file each variable appears in (● = present).

Construction & formulas

The composite index is built with pooled PCA on the three SHDI sub-indices (education, health, income), stacking both periods so a single set of parameters applies to all 306 region-year observations.

• Pooled standardization (z-score): Z = (X − μ_pool) / σ_pool, where the mean and SD are computed across all 306 rows (both years) rather than period by period — a fixed yardstick. Pooled means [0.6786, 0.8437, 0.7254], SDs [0.0814, 0.0472, 0.0749].
• Pooled covariance & eigen-decomposition: Σ = (1/nT) Zₜ Z, then Σ v_k = λ_k v_k. Eigenvalues [2.1726, 0.5631, 0.2643]; PC1 eigenvector (weights) [0.5642, 0.5448, 0.6204].
• Variance explained: λ_k / Σλ — PC1 = 72.42%, PC2 = 18.77%, PC3 = 8.81%.
• PC1 score (composite index): PC1_i = w₁Z_edu + w₂Z_health + w₃Z_income, the projection of each region-year onto the first eigenvector.
• Pooled normalization (0–1): HDI_i = (PC1_i − PC1_min) / (PC1_max − PC1_min), using the min/max across all 306 observations (a common scale across periods).

Wide vs long. data.csv stores each region once with year-suffixed columns; data_long.csv melts it so each (region, period) pair is a row, rounding the sub-indices/SHDI to 4 decimals and population to 1 decimal and adding a region_country display label (shortened region name + 3-letter country code). The official SHDI uses a geometric mean of its three components, SHDI = (Education × Health × Income)^(1/3), and serves as the external validation benchmark.

The datasets

Switch datasets with the tabs. Each shows the full variable dictionary plus a sortable statistics table with mini distributions and data coverage.

Expand all datasets expand to search (Ctrl/⌘+F) or print across all datasets

Known limitations & caveats

• Real data. Values are the Global Data Lab's published Subnational HDI for South America (Smits & Permanyer, 2019); they are not simulated.
• Two periods only. 2013 and 2019 are the minimum for temporal analysis; more periods would strengthen the pooled estimates and allow testing the constant-correlation assumption.
• Sample-relative index. The PCA-based HDI is relative to this specific 153-region sample — adding or removing regions changes every score, and Min-Max normalization is sensitive to outliers (Potaro-Siparuni, Guyana anchors the bottom).
• South America only. Correlation structure and eigenvector weights are specific to this region; other world regions can produce different weights.
• Income decline is genuine. Average income fell 2013→2019 (driven by Venezuela's collapse, 0.782→0.630), which is why a fixed (pooled) yardstick is needed to compare scores across time.