A SMALL, INTERACTIVE TUTORIAL
Each observation asks your parameters to lie on a line. They disagree, so least squares finds a compromise. Add one observation and watch that compromise move.
This is parameter space: a point is a model . A line is one equation , not a fitted regression line in data space.
01 / EXPLORE
The short orange arrow is the new line’s normal. Faint ellipses are equal-loss contours of the existing system.
Start near zero: the purple influence estimate and teal exact answer should nearly overlap.
A prediction is . Its signed residual is . Since stays fixed, prediction change = residual change.
| Line / weight | Before | Exact after | Exact Δ | Influence Δ |
|---|
02 / CALCULATE
The influence function is a derivative at zero weight. Least squares also lets us compute the exact finite-weight answer, so we can see precisely where the approximation departs.
Use a sum of rank-one quadratics, with no hidden averaging or regularization. Kept observations have fixed weights .
Our starting normals span two dimensions, so is positive definite and the solution is unique. Each additional nonnegative weight preserves that.
Let describe the orange line. Its residual at the old solution gives its loss gradient. Kaczmarz would step along ; influence first accounts for the curvature of all the existing lines.
Solve this 2 × 2 system for ; there is no need to form a matrix inverse. Weakly constrained directions move more.
The new objective is . The first-order estimate and the exact rank-one update are:
The denominator is the correction that the influence approximation omits. Here measures how weakly the old system constrains the new normal; solve and take . For one added line, the exact path is straight but its motion saturates as the weight grows.
Subtract the two stationarity equations:
Applying the Sherman–Morrison rank-one identity yields the formula above. Equivalently, substitute it directly into this equation. In particular, : a very strong observation drives its own residual toward zero. Although the loss is quadratic in parameters, the solution is not linear in its weight.
Take a dot product with each line’s normal. The sign says which way its residual moves; compare absolute residuals to decide whether that line is fitted better.
The derivative at a nonzero weight is . The purple line always uses the derivative at .
03 / PROJECT
A full Kaczmarz step projects onto one line. With inconsistent observations, it generally keeps moving: there is no common intersection to settle on. Least squares balances the residuals instead.
The numerator is the signed error. Dividing by the squared normal length gives exactly the movement needed to reach the selected line. Here the original normals have unit length.
Form the two normal equations, including the new observation at its current weight:
Each row is another line in parameter space. Unlike the original observations, these two lines meet exactly at the weighted least-squares solution. The same projection formula, now using a row of , converges to that intersection.
The influence solve works the same way, with as the unknown and on the right. Two rows, two consistent lines, one influence direction.
Scaling a row by the square root of its weight leaves its full geometric projection unchanged. Cycling once through every positive-weight observation therefore does not implement weighted least squares. Weight-aware stochastic updates with diminishing steps, or extended Kaczmarz methods, address inconsistent systems. This demo uses normal equations to keep that distinction visible.
For a large or ill-conditioned problem, forming normal equations squares the condition number of the weighted design matrix. QR, SVD, LSQR, or an appropriate extended Kaczmarz method can be better choices. Here we solve only a positive-definite 2 × 2 system, and evaluate influence independently of the animated solver.
THE IDEA TO TAKE WITH YOU
The influence function is that response at infinitesimal weight. This quadratic example lets you compare it with the exact answer, one residual at a time.