Sensitivity analysis and robustness checks for causal claims
The systematic process of testing whether a causal conclusion holds under various alternative assumptions, model specifications, and data perturbations to quantify its vulnerability to unobserved confounding or methodological fragility.
It directly mitigates strategic and financial risk by ensuring business decisions are based on reliable causal evidence rather than spurious correlations, which prevents costly misallocations of resources. It builds organizational credibility by demonstrating analytical rigor in high-stakes environments, enabling confident investment in initiatives like marketing spend or policy interventions.
1Careers
1Categories
8.7 Avg Demand
15%
Avg AI Risk
How to Learn Sensitivity analysis and robustness checks for causal claims
Focus on understanding the core threat to causal inference-unobserved confounding (Omitted Variable Bias). Grasp the purpose of basic robustness checks like varying model specifications (e.g., adding/removing control variables) and running placebos or falsification tests. Build the habit of always asking: 'What if my key finding is driven by something else?'
Move to formal sensitivity analysis frameworks. Learn to apply the Oster (2019) delta and coefficient stability method or the Cinelli & Hazlett (2020) omitted variable bias approach. Practice conducting leave-one-out analysis, sample splitting, and using alternative measures of the treatment/outcome. Common mistake: confusing a robustness check with a sensitivity analysis-a check often just reruns the same assumption; a test systematically quantifies how much assumption violation the result can withstand.
Master designing a comprehensive sensitivity analysis suite for a publication-grade study or a major business experiment. Integrate sensitivity results into the final interpretation (e.g., 'Our effect size is robust to an unobserved confounder X times stronger than any observed covariate'). Mentor teams by establishing institutional protocols for robustness, and link sensitivity outcomes to executive risk tolerance and decision thresholds.
Practice Projects
Beginner
Case Study/Exercise
Marketing Campaign Attribution Check
Scenario
A team claims a new ad campaign caused a 15% increase in sales based on a simple regression. You suspect seasonality or a concurrent product launch might explain it.
How to Execute
1. Collect data on the product launch timeline and monthly sales trends. 2. Re-run the original regression adding month fixed effects and a binary 'product launch' indicator as controls. 3. Conduct a placebo test by running the analysis on a pre-campaign period to see if the 'effect' appears spuriously. 4. Report whether the original effect survives these basic controls.
Intermediate
Project
Sensitivity Analysis for an A/B Test
Scenario
You have a statistically significant uplift from an A/B test on user engagement, but you need to assess how sensitive this result is to potential violations of the SUTVA (Stable Unit Treatment Value Assumption) or non-random attrition.
How to Execute
1. Apply the Oster (2019) method: calculate the R-squared change from adding a rich set of user covariates, then compute the delta (δ) that would be required for unobservables to explain away the effect. 2. Perform a Lee Bounds analysis to address potential differential attrition between treatment and control groups. 3. Report the range of the effect size under these different sensitivity scenarios.
Advanced
Project
Robustness Suite for a Quasi-Experimental Policy Evaluation
Scenario
Evaluating the causal impact of a new city-wide rental subsidy policy using a difference-in-differences design, where parallel trends may be questionable and spillover effects are possible.
How to Execute
1. Conduct a formal sensitivity analysis using the Cinelli & Hazlett (2020) framework to quantify how strong an unobserved confounder must be relative to observed controls to nullify the result. 2. Implement a triple-differences model with a placebo outcome and placebo group to test for pre-existing trends. 3. Perform spatial analysis to test for displacement effects. 4. Synthesize all findings into a single 'robustness scorecard' for the policy report, explicitly stating the conditions under which the causal claim holds.
Tools & Frameworks
Mental Models & Methodologies
Omitted Variable Bias (OVB) FrameworkOster (2019) Delta and Coefficient StabilityCinelli & Hazlett (2020) Sensitivity Analysis for OVBDifference-in-Differences (DiD) Event Study PlotsPlacebo and Falsification Tests
OVB is the foundational mental model. Oster's method quantifies robustness by calculating how much selection on unobservables (relative to observables) would be needed to explain away the effect. Cinelli & Hazlett provide a more formal 'robustness value' metric. Event studies and placebos are the workhorse checks for pre-trend validation and confounding detection.
Software & Platforms
R (packages: 'sensemakr', 'fixest', 'rdrobust', 'bounds')Stata (commands: 'sensemakr', 'sensemakr', 'plm')Python (libraries: 'linearmodels', 'statsmodels', 'scipy')Jupyter Notebooks / R Markdown
Use 'sensemakr' in R/Stata for Oster and Cinelli & Hazlett calculations. 'fixest' is optimal for fast, high-dimensional fixed effects models common in DiD. 'rdrobust' is essential for regression discontinuity designs. These tools should be embedded in reproducible report workflows (Rmd/Jupyter) to document every robustness test run.
Interview Questions
Answer Strategy
Structure your answer around the OVB framework. Start by stating you'd first run multiple alternative model specifications. Then, you'd apply a formal sensitivity method (e.g., Oster's) to calculate the 'robustness value'-the strength of hypothetical unobserved confounders needed to nullify the effect. Finally, you'd interpret this in business terms: 'The result is robust to confounders up to X times stronger than our observed variables, which, given our domain knowledge, suggests high confidence.'
Answer Strategy
Testing for practical problem-solving and intellectual honesty. Describe a specific instance where you used a placebo test, falsification test, or sensitivity analysis. Detail the method (e.g., 'I ran a triple-differences model with a placebo outcome'). The outcome should show nuance: perhaps the original effect attenuated but remained significant, or it was entirely explained away by the confounder. Emphasize how you communicated this revised finding to stakeholders.
Careers That Require Sensitivity analysis and robustness checks for causal claims