From First Principles

About this blog

It is more important than ever to truly understand how statistical models are built, and what they are doing. Not just how to execute them, or how to refresh their input data, but to remember the history and derivations and how would someone change a model’s internals. A good mental model is crucial for responsible development. It also helps to build communal understanding that can create human innovation, and it make systems more sustainable.

This blog “From First Principles” unpacks concepts in statistics and explains them from their first principles using math and software. By building this background together, we will be well positioned to explain modeling strategy, understand history, and innovate in algorihms to make intelligent systems. Much of this blog is dedicated to causal inference and how to analyze experimental data, though I frequently write about extensions for how to make decision making engines in general.

Recent posts

Sustainable Data Science Systems

data science

A data scientist’s first unique value add is in generating actionable insights for a business. Later, they might scale their work by embedding it into a platform system, for…

Bayesian Universal Modeling

statistics

bayesian statistics

AB testing

regression

In the conditionally sufficient statistics post, a streaming service prompted households to turn on a new playback setting and measured how it changed hours watched and…

Every bootstrap is a reweighting

statistics

bayesian statistics

AB testing

regression

A team runs an AB test and reports a covariate-adjusted effect on hours streamed. A second team reports the difference in median hours. A third wants the probability that…

What are AI Benchmarks?

AI

In September 2026 OpenAI launched GPT-6 Astra with a table of benchmark results, including 96.0% on GPQA Diamond, 97.6% on FrontierMath Tier 4 (v2), and “74.1% on DeepSWE…

Conditionally Sufficient Statistics

causal inference

AB testing

regression

Experimentation has many special designs, each warranting a specialized type of model. Managing all of these can be difficult. Keeping the implementation sane is harder. In…

First Principles of Ranking

causal inference

ranking

AI

It is Friday night, and a member opens Netflix on their TV. Within a second, the home page fills with rows of titles: “Continue Watching”, “Trending Now”, “Because You…

What does it mean to be the average effect?

causal inference

experimentation

statistics

Oh what’s in a name?

Inference on Ratio Metrics

causal inference

AB testing

regression

Some business metrics need to be normalized. For example, a video streaming service might ask how many rebuffers a user experienced. But if the user played many videos, it…

It’s OK to let least squares output a probability

causal inference

AB testing

regression

In AB testing we use linear regression all the time to model data. This is very appropriate when the outcome variable, \(y\), is continuous. Many AB outcomes, however, are…

You want the posterior distribution, not the confidence interval

bayesian statistics

AB testing

regression

In this post we cover the misconception between a confidence interval and a posterior distribution. Despite the confusion though, the frequentist methods are entrenched in…

From OLS to Deep Learning

mathematical statistics

We often hear “deep learning is a bunch of matrix multiplications”. Is that true? If yes, then is it actually a linear operator like OLS?

The Variance of your Composite Metric

experimentation

statistics

This post will describe the analysis strategy for analyzing a composite metric, especially how to derive its variance. A composite metric is a new metric that is made by…

A Better, Robust CUPED

experimentation

statistics

We previously discussed the similarities and differences between CUPED and OLS. While CUPED is a computationally efficient implementation of a linear model, it has some…

Relative Treatment Effects, the Delta Method, and Fieller’s Method

experimentation

statistics

In AB testing, the classic average treatment effect refers to the absolute difference in means. Sometimes, we want to compute the treatment effect in terms of percentages…

Zero Inflated Causal Inference

experimentation

statistics

In this blog post we study how a large amount of zeros in experiment data can affect causal inference. In an experiment, we ideally target users who qualify to receive a…

Make Experimentation Planning Simple Again

experimentation

statistics

We want to learn as quickly as possible from experiments. But most experiments are designed as fixed horizon tests, meaning we meticulously plan for statistical power…

Creating CUPED from FWL

experimentation

statistics

CUPED is a well known technique to greatly increase statistical power in a controlled experiment. The CUPED formulation has similarities to the OLS model

Does your AB have controlled randomization?

experimentation

statistics

The SRM is a popular test to tell if your AB test is healthy. But it’s not enough. The SRM is powered by a chi square test underneath the hood, which compares observed count…

Aggregations and Arrival Times Affect Convergence

experimentation

statistics

In this post we study the convergence speed of an experiment. Over time, we accumulate sample size. As that sample size increases, statistical power is supposed to increase…

Errors in Experiments

experimentation

statistics

An experiment can make many errors. What error do we care about? How do we design an experiment that can make guarantees that those errors are small? While a randomized and…

Posthoc MDE

experimentation

statistics

Credits: This post is a summary of the World Bank blog post written by David McKenzie. There is no additional innovation in the post here. It is a natural follow up to our…

Statistical Power under Noncompliance

experimentation

statistics

This is an extension of a previous post on statistical power. In that post we derived the formula for statistical power using

Foundations in Statistical Power

experimentation

statistics

Statistical power is a concept that helps us plan for an effective experiment. Power is typically described in terms of a purely randomized and controlled trial, where we…

The Mathematics of the Elastic Net

mathematical statistics

Author’s note: This post is largely a rehash of many of the original elastic net and glmnet papers. I hope that having another voice describe the elegance of the elastic net…

Tikhonov Regularization and Gaussian Priors

mathematical statistics

In this post we will show that the maximum a-posteriori (MAP) estimator of a normal-normal is equal to the estimator from Tikhonov regularization.