Logistic regression or neural network? Same neuron

♣ Newbie

Imagine you want to guess whether it’ll rain tomorrow. You look at clues: there are clouds, it’s windy, the sun comes out in the afternoon. They don’t all count the same: clouds weigh a lot, wind a little and sun takes away. You add up what each clue weighs, and that sum moves a needle that goes from “definitely not” to “definitely yes”.

That scale is a logistic regression, a good old statistics method for answering yes-or-no questions. And it’s also a neuron, the building block of neural networks. Same scale, two names.

The most important part is still missing: learning. At first, the scale doesn’t know how much each clue weighs. You show it lots of past days and, every time it’s wrong, it gets told off, harder the surer it was. To avoid telling-offs, the scale adjusts its weights bit by bit. And this scale has a reassuring property: wherever you start, you always end up with the same weights.

♦ Intermediate

A logistic regression has three pieces, and they’re worth knowing because each one explains a different way it fails.

1. A weighted sum. Each column of your data is multiplied by a number, its weight, and everything is added up together with a constant. We’ll call the result the score. Everything the model knows lives there: if the signal is in a combination of two columns or in a curve, the model can’t see it unless you hand it over already built.

2. The sigmoid. It’s an S-shaped curve that squashes any score between 0 and 1: a very negative one gives almost 0, a very positive one almost 1, and zero gives exactly 0.5. It isn’t picked for looks: it comes from assuming the answer is a loaded coin whose bias depends on a weighted sum. That’s why its output can be read as a probability.

3. The penalty. It compares the probability with the right answer and punishes being very sure and wrong. In the jargon it’s called the loss; this one in particular is the log-loss or cross-entropy. To learn, the model checks which way the penalty goes down and moves the weights a little in that direction, over and over: that’s gradient descent.

And here’s the property that changes everything: for logistic regression, the penalty has a single bottom. Any way of going downhill ends up in the same place, so the random seed doesn’t matter. A network with hidden layers loses that guarantee: it has many valleys, and where you end up depends on where you start.

I checked: scikit-learn’s logistic regression and a neuron trained by hand give the same weights to the third decimal. A practical detail: many libraries add a brake by default that shrinks the weights, so if you want to read them, turn it off.

♠ Technical

Logistic regression is a generalized linear model: y ~ Bernoulli(p) with the canonical logit link, so logit(p) = wᵀx + b and p = σ(wᵀx + b). The sigmoid isn’t a design choice: it’s the inverse of the Bernoulli’s canonical link. A neuron with σ activation trained with binary cross-entropy (BCE) is exactly that model, with the same likelihood.

With the mean loss L(w) = (1/n) Σᵢ BCE(yᵢ, σ(wᵀxᵢ)), the gradient is ∇L = Xᵀ(p − y)/n and the Hessian is ∇²L = XᵀSX/n with S = diag(pᵢ(1 − pᵢ)). Since S has a positive diagonal, XᵀSX ⪰ 0: the loss is convex, and strictly convex if X has full rank. That’s why gradient descent, Newton or IRLS reach the same optimum. With separable data the optimum doesn’t exist (‖w‖ → ∞) and you need to regularize.

I checked it with scikit-learn without regularization (C=np.inf) against a neuron trained with gradient descent:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=2000, n_features=5,
                           flip_y=0.1, random_state=0)
clf = LogisticRegression(C=np.inf).fit(X, y)

w, b = np.zeros(5), 0.0
for _ in range(20000):
    # one neuron: linear + sigmoid
    p = 1 / (1 + np.exp(-(X @ w + b)))
    # gradient of the log-loss
    w -= 0.5 * X.T @ (p - y) / len(y)
    b -= 0.5 * (p - y).mean()

# the same weights
print(clf.coef_[0].round(3), w.round(3))

With scikit-learn 1.9.1, the weights match to the third decimal. It isn’t a fluke of this dataset: it’s the same convex problem solved along two paths.

♣ Newbie

If the scale at the end is the same, what’s so special about a big neural network? Whatever happens before the scale.

Think of a pot. If you cook with pre-chopped ingredients, you throw them in and you’re done. That’s what data in a table is like: age, amount, number of visits. Each column is already a meaningful clue, and the scale weighs them directly.

A photo is a whole potato. A single dot of colour doesn’t tell you whether there’s a cat. Someone has to chop the photo into useful clues, like “there are ears here” or “this looks like a whisker”. The layers of a neural network do exactly that: they learn to chop by themselves. People in the know call those manufactured clues features. In the end, everything lands in the same pot: the same scale as before.

♦ Intermediate

If the neuron at the end is the same, what a deep network adds is whatever happens before that neuron. A ten-layer network that answers yes or no is, in its last layer, a logistic regression. The difference is that its inputs aren’t your columns: they’re features, new combinations that the hidden layers learned from the data.

The classic example is XOR: four points at the corners of a square, with the class switching on each diagonal. A neuron can only draw a straight line, and no straight line splits them. But give it a new column, the product of the two columns, and the points split on their own. That new column is a feature: either you build it by hand, or a hidden layer builds it.

That gives the most useful rule of thumb I know. If your data is a table and the columns already mean something, the good features exist and the hidden layers have little to add. If it’s pixels, audio or text, no weighted sum of pixels can see a cat: someone has to build the features. They can be built by hidden layers trained on your data, by hundreds of trees adding up their votes (boosting, on tables) or by a model already trained on half the world’s data, of which you only adjust the end.

A practical detail: in deep learning libraries, the network returns the unsquashed score, and the sigmoid is applied inside the penalty, where the arithmetic doesn’t break with huge numbers.

♠ Technical

A binary network with L layers is p = σ(wᵀφ(x) + b), with φ = h⁽ᴸ⁻¹⁾ ∘ … ∘ h⁽¹⁾ learned jointly with w. Conditional on φ, the last layer is a logistic regression on the features h. The same holds for almost every binary classifier in use today: in binary boosting (XGBoost binary:logistic, LightGBM binary), p = σ(Σₖ fₖ(x)) with trees fₖ; in a fine-tuned pretrained model, φ is the embedding.

In XOR, with x ∈ {−1, 1}², the class is y = 1 if x₁x₂ > 0: no hyperplane in (x₁, x₂) separates it, but in (x₁, x₂, x₁x₂) the hyperplane x₁x₂ = 0 is enough. The lab’s hidden layer (8 tanh neurons) reaches an equivalent boundary without anyone telling it which product to look for, at the cost of |θ| = 33 instead of 4, of losing convexity and of W⁽¹⁾ no longer being readable.

♣ Newbie

If a network is “lots of scales in a row”, the obvious move would be to use the same needle, the S-shaped curve, in all of them. That’s how it was done for years, and it went badly.

To learn, a network gets a message at the end, “you were off by this much”, and that message has to travel backwards, from the last layer to the first, so each one can adjust its weights. It’s like the telephone game: each layer with that curve repeats the message at a quarter of the volume at most. After five people, the message arrives at less than half of one percent. With ten, not even a whisper gets through, and the first layers barely learn.

The fix was to swap the needle in the middle layers for one that doesn’t turn the volume down when the message is positive. The S-shaped curve didn’t retire: it went to work as a doorman. It’s still at the end of the network, to say “88% yes”, and inside large language models, deciding how much information gets through.

♦ Intermediate

If a network is “lots of logistic neurons stacked up”, the obvious move would be sigmoids in every layer. That’s how it was done for years, and it went badly.

The reason is the sigmoid’s slope. It’s 0.25 at most, right in the middle of the S, and almost zero at the ends, where the curve is already flat. To train a network, the signal saying how much and which way to move each weight (the gradient) is computed from the last layer to the first, multiplying by each layer’s slope: that’s backpropagation. With several sigmoids in a row, what reaches the first layer has a ceiling of 0.25 multiplied by itself once per layer, and that’s before counting how much the weights stretch or shrink it.

With 10 layers, the signal reaching the start is less than one millionth. The first layers barely learn. It’s the vanishing gradient problem, which Hochreiter described in 1991. In practice it was just as bad: in 2010, Glorot and Bengio trained networks with 5 hidden layers and test error was 82.61% with sigmoids, 27.15% with the hyperbolic tangent and 15.60% with the hyperbolic tangent plus a better initialization. In hidden layers, the sigmoid gave way to the ReLU, a function that lets positive values through as they are and sets negative ones to zero.

The sigmoid didn’t disappear: it changed jobs. Outside the chain of hidden layers it isn’t multiplied by itself dozens of times, and its 0-to-1 range is exactly what you need for a probability or a gate.

♠ Technical

With h⁽ˡ⁾ = σ(z⁽ˡ⁾) and z⁽ˡ⁾ = W⁽ˡ⁾h⁽ˡ⁻¹⁾ + b⁽ˡ⁾, backpropagation gives δ⁽ˡ⁾ = diag(σ′(z⁽ˡ⁾)) W⁽ˡ⁺¹⁾ᵀ δ⁽ˡ⁺¹⁾, and the first layer’s gradient is ∂L/∂W⁽¹⁾ = δ⁽¹⁾xᵀ. Since σ′ = σ(1 − σ) ≤ ¼, with n sigmoid layers ‖δ⁽¹⁾‖ ≤ (¼)ⁿ ∏ₗ ‖W⁽ˡ⁾‖ · ‖∂L/∂h⁽ⁿ⁾‖. Unless the weight norms make up for the factor of 4 per layer, the gradient decays exponentially with depth (Hochreiter, 1991 and 1998). If the |z| sit in the tails, σ′ ≈ 0 and the bound is much lower.

Glorot and Bengio (AISTATS 2010, Table 1) measured it with 5 hidden layers on Shapeset-3×2: test error of 82.61% with sigmoid, 27.15% with tanh and 15.60% with tanh and normalized initialization, Var(W) = 2/(nᵢₙ + nₒᵤₜ), now called Xavier or Glorot. Also, σ isn’t zero-centred: mean activations of 0.5 bias the updates and saturate the top layers. The ReLU, max(0, z), has derivative 1 for z > 0 and breaks the product of factors below ¼.

Outside the deep chain, σ keeps two roles: as the output link, p = σ(z), in binary and multi-label settings (one σ per class instead of softmax), and as a multiplicative gate. In an LSTM, the input gate is iₜ = σ(Wᵢ[hₜ₋₁, xₜ] + bᵢ). In SwiGLU (Shazeer, 2020), FFN(x) = (Swish(xW) ⊙ xV)W₂ with Swish(u) = u·σ(βu): LLaMA and PaLM use it in every transformer block.

♣ Newbie

You don’t buy a food processor to peel an orange. The machine is expensive, takes a while to set up and, if something goes wrong, you don’t know why. It pays off when you have to make bread for a whole village.

Models are the same. If your data is a table with columns that already mean something, the good old scale is usually enough, and you can explain every decision too: “I said no because of this column”. If your data is photos, sound or text, someone has to chop the ingredients, and that’s where a network pays off. One piece of advice that always works: try the scale first. It takes minutes and tells you how much the machine would have to win by to be worth it.

♦ Intermediate

Tables: the classic model is usually enough. Christodoulou and colleagues reviewed 71 clinical prediction studies with 282 comparisons between logistic regression and machine learning. They measured performance with the AUC, the probability that the model scores a positive case higher than a negative one. In the comparisons at low risk of bias (well-run studies), the difference was zero. Machine learning only came out ahead in the studies at high risk of bias. I read the paper’s abstract, not the full text. On large tables where nobody has to understand the decisions, trees adding up their votes usually beat neural networks, and logistic regression is still the baseline to start from.

Images, audio or text: the features have to be learned. But that doesn’t always take a huge model. In 2024, Bucher and Martini compared small fine-tuned models (already trained and then adjusted with a few hundred of your own examples) with large language models given no examples, only the instructions:

In two of the four tasks, the best language model with no examples scores below always guessing the most common class. The fine-tuned models perform well with between 200 and 500 labelled examples, according to the authors. A language model with no examples is good for getting going on day one and for labelling those first few hundred examples, but it usually isn’t what you want in production.

One last warning: a number between 0 and 1 isn’t automatically a probability you can trust. Logistic regression comes out well calibrated: if it says 80%, it’s right about 80% of the time. Modern deep networks, not so much, although it can be fixed afterwards with a single adjustment.

♠ Technical

Christodoulou et al. (J Clin Epidemiol, 2019): 71 studies and 282 comparisons. At low RoB (145), Δ logit(AUC) = 0.00 [−0.18, 0.18]; at high RoB (137), +0.34 [0.20, 0.47]. ML’s apparent advantage is concentrated where validation is weakest. I only read the abstract. On a benchmark of 45 tabular datasets, Grinsztajn et al. (2022) find that tree-based models remain state of the art on medium-sized data (around 10,000 rows): networks struggle with uninformative features and irregular target functions.

On text, Bucher and Martini (2024) compare the best fine-tuned model with the best zero-shot generative LLM (GPT-3.5, GPT-4 and Claude 3 Opus) on four tasks. In two of them, zero-shot falls below the majority-class baseline (0.73 and 0.50). According to the authors (§5.5), fine-tuning performs well with between 200 and 500 labels.