Assumptions Behind Statistical Methods

Contents
  1. 31. Regression to the Mean
  2. 32. Multicollinearity
  3. 33. Omitted Variable Bias
  4. 34. Overfitting
  5. 35. Data Leakage
  6. 36. Look-ahead Bias
  7. 37. Extrapolation Bias
  8. 38. P-value Misinterpretation
  9. 39. Effect Size Neglect
  10. 40. Underpowered Study
  11. 41. Multiple Comparisons

The “Sports Illustrated Jinx” has circulated among sports fans for years: whenever an athlete appears on the cover of Sports Illustrated, their subsequent performance drops, or they get injured. Countless athletes have refused to be on the cover because of it.

Athletes often appear on the cover while performing exceptionally well. Even without a cover appearance, an extreme performance may be followed by results closer to their usual level. Ignoring regression to the mean can make that later change appear to be an effect of the cover.

To test whether the cover has an effect, we would need to compare those athletes with others who reached a similar peak without appearing on it. The methods in this article likewise depend on the comparison group, when information was available, and whether a model’s assumptions hold.


31. Regression to the Mean

Nobel laureate Daniel Kahneman once told a story: a flight instructor opposed praising students, arguing that “every time I praise a student for a perfect landing, their next one is usually worse; but when I harshly scold them for a terrible landing, they improve next time. So punishment is more effective than praise.”

The instructor committed the regression fallacy:

  • After a student makes a “perfect landing” (an extreme high), the next performance regressing to the mean (getting worse) is a high-probability event, regardless of whether you praised them.
  • After a student makes a “terrible landing” (an extreme low), the next performance regressing to the mean (getting better) is also a high-probability event, regardless of whether you scolded them.

In the tech world, this error shows up in post-incident reviews: “After a severe system failure (extreme value), we deployed emergency patch A, and system stability improved significantly. Patch A worked.” Not necessarily. Systems naturally tend to regress toward normal after an extreme failure, unless the damage is permanent. Without a control group, you cannot tell whether the patch was effective or mean reversion simply ran its course.

How to spot it: if your intervention always happens when “things are especially bad” and then “things improve,” you need at least a control group before you can say anything.

32. Multicollinearity

In regression models, if predictor variables are highly correlated with each other, the model cannot accurately estimate each variable’s independent effect. Coefficients become unstable and may even flip sign (what should be a positive relationship gets estimated as negative).

Predicting house prices: square footage and number of rooms are highly correlated variables (bigger houses typically have more rooms). When you put both into a regression model, the model gets stuck: is it square footage or number of rooms that affects price? The coefficient estimates jump around, become unstable; you cannot say “each additional room increases the price by X dollars” because the effects of square footage and room count are entangled and inseparable.

In machine learning feature engineering, the harm of multicollinearity is often underestimated: you think adding more correlated features makes the model “richer,” when in reality you are making it more confused and interpretability drops to zero.

33. Omitted Variable Bias

This is the model-level version of confounding: your regression model omits a variable that simultaneously affects X and Y, so the model incorrectly attributes that omitted variable’s influence to the other variables you did include, causing coefficient bias.

Suppose you want to study the effect of education level (X) on salary (Y), but you forgot to control for “family socioeconomic status” (omitted variable Z). Z simultaneously influences education (wealthy families with resources more easily send children to higher education) and salary (the direct effects of social capital and connections). Result: your model’s coefficient estimate for education is biased upward because it credits the salary advantage brought by “family background” to “education level.”

Omitted variable bias is the “you don’t know what you don’t know” kind of problem: you did not include Z, you cannot see Z’s influence, you may not even be aware Z exists. The antidote is building a theoretical framework (thinking through what factors might affect Y before looking at data), not just looking at data.

34. Overfitting

An overfitted model performs well on training data but much worse on new data. It may have learned noise specific to the training sample, limiting how well it works elsewhere.

Imagine training a purchase prediction model on 2019 user behavior data: the model achieves 97% accuracy on 2019 data. In 2020, COVID dramatically changes all user behavior, and the model’s performance instantly collapses. Much of what it learned as “patterns” was noise specific to that particular time period, not generalizable user behavior logic.

Common overfitting signals: model performance on training data is far better than on validation data; too many features are used (especially when the number of features approaches or exceeds the number of data points); different random seeds produce wildly different results.

35. Data Leakage

Data leakage can produce unusually strong test results, so high accuracy alone does not show that a model will be useful.

Suppose you train an AI to predict whether a patient will develop pneumonia: 99% accuracy, and you think you are about to revolutionize healthcare. After deployment, the model completely fails. Investigation reveals: the training data included a feature called “whether the patient is taking antibiotics.” In the historical data, only diagnosed pneumonia patients were on antibiotics. The model found a shortcut: “antibiotics = pneumonia.” But in reality, you want to predict the condition before the patient takes medication; that feature simply does not exist at prediction time.

Similar forms of leakage in data preprocessing are more common and harder to detect:

  • Computing normalization means and standard deviations using the entire dataset (including the test set), then splitting into train/test sets: test set information has already seeped into the training process.
  • Using “last login time” to predict user churn: known in retrospect, but when the model goes live, you do not know this feature’s value before churn happens.

A model performing too good to be true is not good news; it is a signal that demands serious scrutiny.

36. Look-ahead Bias

Look-ahead bias is data leakage’s specific manifestation in time series analysis and financial backtesting: you use “future data that was unavailable at the time” to evaluate a strategy that “could have been executed at the time,” making historical performance look unrealistically good.

For example, a strategy designed with all of 2020’s data might then be evaluated against returns from that same year. A trader in March could not have known what happened in November; such a backtest overstates what was possible at the time.

In ML time series forecasting, this problem appears in feature engineering: you calculate a “7-day moving average,” but if the calculation includes the prediction date’s own data, you have introduced look-ahead bias. Strictly distinguishing “before the prediction point” from “the prediction date and after” is a fundamental practice in time series modeling.

37. Extrapolation Bias

Extrapolation means extending a model’s predictions beyond the range of its training data. The problem: a model performing well within the range it has seen does not mean it is valid outside that range.

You trained a recommendation system on Taiwanese user behavior data; accuracy and retention look great. You confidently launch it in Southeast Asia, and the results are dismal. Southeast Asian users have entirely different behavior patterns, cultural backgrounds, and usage contexts from Taiwanese users. The model has never seen that kind of data, and its “intelligence” is worthless outside the training distribution.

In trend forecasting, the extrapolation fallacy takes the form of extending a linear growth trend indefinitely: “We added 10,000 new users per month for the past six months, so in three years we’ll have 3.6 million users.” Growth curves are never straight lines: markets have boundaries, competitors emerge, user acquisition costs rise. “Extrapolating the current trend” is the laziest and most error-prone approach to prediction.

38. P-value Misinterpretation

The A/B test is done, p = 0.03. Someone says: “We are 97% confident the new version is better!”

This is the most widespread misunderstanding of p-values, common even in academia, let alone in product development.

The correct definition of a p-value: given that the null hypothesis (no difference between versions) is true, the probability of observing data “this extreme or more extreme.”

p = 0.03 means: if the two versions truly have no difference, there is only a 3% chance of producing this result.

It does not mean: “the probability that the null hypothesis is true is 3%,” nor does it mean “the probability that the new version is better is 97%.” Both interpretations are logically wrong: the p-value is computed under the assumption that the null hypothesis is true, and it has no ability to directly tell you the probability that the null hypothesis is true or false.

A practical way to understand it: p < 0.05 says “if nothing changed, a result this coincidental has only a 5% chance of appearing.” It gives you reason to doubt the assumption that “nothing changed,” but it does not equal “the new version is definitely better.”

39. Effect Size Neglect

An A/B test with n = 1,000,000 finishes, p < 0.0001 (highly significant): the new button’s click-through rate improved from 2.0000% to 2.0001%.

This result is statistically real; it is not noise. But is a 0.0001 percentage point improvement worth the additional code complexity, maintenance costs, and deployment risks?

Statistical significance and practical importance are two different things. With a large enough sample, any tiny difference can reach statistical significance. This is why reporting effect size (Cohen’s d, relative lift, etc.) is at least as important as p-values, if not more.

Effect size neglect and #40 “underpowered study” are two sides of the same coin: too small a sample means real effects go undetected (false negative); too large a sample means trivial effects get blown up to “significant” (treating noise as signal). Both problems must be addressed when designing experiments.

40. Underpowered Study

An A/B test with a few hundred people shows no significant difference. Team conclusion: “The new feature doesn’t work.”

But the truth may be: the effect is real; you just did not have a large enough sample to “see” it. This is a false negative (Type II Error): incorrectly accepting the null hypothesis.

The purpose of power analysis is to calculate, before the experiment begins: given the effect size you expect, how large a sample do you need to detect the effect with sufficient confidence if it truly exists? Running an A/B test without a power analysis is like “going to read the blackboard without knowing how poor your eyesight is.”

Many companies rush to stop a test after seeing “no significant difference,” treating it as proof that the feature is ineffective. But “failing to reject the null hypothesis” and “the null hypothesis being true” are two entirely different things.

41. Multiple Comparisons

Suppose you simultaneously A/B test 20 different button colors, each at a significance threshold of p < 0.05. Purely by probability, even if all 20 colors truly have zero effect on conversion, you would expect about 1 (20 x 5% = 1) to produce a false positive result with p < 0.05. If you only report that 1 “significant” result, you claim to have found the best button color, but that is mathematical inevitability, not a discovery.

Randomly combining variables and time intervals to run tests until some combination produces a “significant” result, then retroactively crafting an explanation: this is called P-Hacking. It is not necessarily deliberate fraud; more often it is self-deception: you genuinely believe the “significant” result because you do not realize how many tests you have already run.

The fix: determine your hypothesis before the experiment begins (pre-registration); if you must do multiple comparisons, use Bonferroni correction or FDR (False Discovery Rate) control; for every “significant” result, seriously ask yourself “how many tests did I run before finding this one?”


Accuracy, p-values, and backtest returns must be read alongside the methods that produced them: how the data was split, how many comparisons were made, and whether inputs were available at prediction time. A pneumonia model with 99% accuracy cannot be used if it depends on medication data that is not yet available when a prediction is needed.