Empirical Research Guide · Machine Learning

Which machine-learning method fits your data?

The right method depends on your question and on your data, not on fashion. Decide first whether you want to predict an outcome or estimate a causal effect; then the tests below choose between LASSO, ridge, elastic net, random forest and XGBoost, or between OLS and double machine learning.

9 method guides9 real example datasetsPython · scikit-learn · DoubleML

Choose a dataset

Nine real datasets: five prediction problems and four causal questions. Each one leads to a different method.

Method guides

Nine routes to a finished paper

Each guide takes one real dataset through tuning, validation, diagnostics and writing, with Python code, real output and sample text for the paper.

No machine learning needed

OLS with robust standard errors

Few controls that enter linearly

With fewer than 10 controls and no sign of nonlinearity, OLS already adjusts for the confounders; report the DML estimate as a robustness check …

Lecture 02: OLS inference →
Design-based

Unobserved confounders

Selection on unobservables, no instrument

Machine learning cannot fix confounding by variables you do not observe. Use a research design: difference-in-differences, synthetic control or regression discontinuity …

Open the Causal Inference Guide →
How the tool decides. Prediction. (A1) A date column with at least 50 periods marks a time series: each series is tested with ADF and KPSS, I(1) series are differenced, and validation uses rolling-origin splits that never shuffle time. (A2) With fewer than 100 observations only regularised linear models are used. (A3) Otherwise a random forest is compared with LASSO by five-fold cross-validated RMSE; if the forest is at least 5% better, the data are treated as nonlinear. (A4) Linear: if any two predictors are correlated above 0.8, the elastic net; otherwise LASSO if it keeps fewer than half of the predictors at the one-standard-error penalty, and ridge if it keeps more. (A5) Nonlinear: a random forest below 1000 observations, XGBoost (gradient boosting) above. Causal effect. (B1) If confounders are unobserved and there is no instrument, machine learning cannot help. (B2) With fewer than 10 controls that enter linearly (the forest does not beat LASSO by 5% for the outcome or the treatment), OLS is enough. (B3) An instrument leads to DML-IV (the first-stage F should be at least 10); a binary treatment to the interactive model, after checking that the propensity scores lie in [0.05, 0.95]; a continuous treatment to the partially linear model. (B4) DML is estimated with LASSO and random-forest learners, five-fold cross-fitting; if the two differ by more than one standard error, the forest is reported. (B5) With at least 2000 observations and a question about who is affected most, a causal forest. These cut-offs are common rules of thumb, like T < 15 in the Panel Data Guide; report the choice and its robustness.