A Pattern Nobody Programmed
When a neural network trains on data, something curious happens that no engineer explicitly designed. The network learns to reproduce the broad, sweeping shapes of a function before it captures the fine, sharp details. Feed it a complex curve, and it will first approximate the gentle hills and valleys. Only later, if at all, does it begin to trace the jagged edges. This behavior has been observed so consistently across different architectures and tasks that researchers gave it a name: spectral bias, sometimes called the F-principle. The network appears to be doing one thing — learning a function — while actually doing something quite different: learning certain parts of that function much faster than others, in a specific order, according to rules it never announced.
The implications extend beyond academic curiosity. When a neural network makes a prediction, it outputs a number, a classification, or a generated image without indicating which frequency components it has learned well and which it has not. A new theoretical framework submitted in October 2026 offers an explanation of why this happens. [1] The work, titled “Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework,” appeared on arXiv and addresses five specific goals that earlier theoretical studies had left insufficiently resolved. [1]
The Mathematics of Selective Learning
The paper’s central insight begins with a reframing. Instead of analyzing a network’s behavior directly in the space of its weights and biases, the authors expand the network’s output as a general Fourier series in an orthogonal polynomial basis. This transformation converts the messy, nonlinear dynamics of gradient descent into a set of ordinary differential equations governing how each Fourier coefficient evolves during training. The key object in this analysis is a matrix called H, which captures how changes in the network’s parameters affect each coefficient. The eigenvalues of H are related to the speed at which different components are learned. Large eigenvalues mean fast learning. Small eigenvalues mean slow learning. The distribution of these eigenvalues — their decay pattern as you move from low-frequency to high-frequency components — is what produces spectral bias.
What makes this framework powerful is its generality. Earlier explanations of spectral bias often relied on specific assumptions about activation functions or data distributions. This work extends the analysis to general activation functions, including sigmoid and ReLU, and makes only weak assumptions about the data. It also avoids aliasing effects that plague finite-domain Fourier analysis, where the boundaries of the domain introduce artifacts that can distort the picture. The authors show that the graded structure of H — the way its eigenvalues decrease as frequency increases — provides an intuitive explanation for why networks prefer low-frequency components. The orthogonal polynomial theory then allows them to analyze how this bias depends on the activation function and, potentially, on the network architecture itself.
What the Network Hides From Itself
Here is where the gap becomes concrete. A network trained on a dataset containing both smooth trends and sharp transitions will report a single loss value. That number summarizes its overall performance. But the loss is not distributed evenly across frequencies. The network has essentially mastered the low-frequency components while barely beginning to learn the high-frequency ones. If you plot the true function and the network’s approximation, you will see the network tracing the broad strokes while smoothing over the details. It looks like a reasonable approximation. It is not a complete one.
The paper’s analysis of eigenvalue decay makes this precise. The authors derive bounds showing that the gradient of the Fourier coefficients with respect to the network parameters decreases exponentially with frequency. [1] For a sigmoid activation function, the decay follows a specific pattern determined by the properties of the activation function. For ReLU, the pattern differs. But in all cases, the high-frequency coefficients receive vanishingly small gradients during early training. The network is not failing to learn them because they are inherently difficult. It is failing because the gradient signal for those components is exponentially small. The network’s apparent competence is real but partial.
The Historical Arc of a Misunderstanding
This is not the first time a technology has presented a partial truth as a complete one. The instrument’s bias became part of the observation, and it took time and better instruments to separate the two.
Neural networks are instruments for learning functions from data. Their spectral bias is a form of chromatic aberration. It makes them see the low-frequency world clearly and the high-frequency world poorly. But unlike a telescope, which at least shows a blurry image when it fails to resolve fine detail, a neural network outputs a crisp prediction regardless of its confidence. There is no blur. There is no signal that the high-frequency components are missing. The output looks as sharp for a well-learned function as for a poorly-learned one. This is the structural deception: the network’s interface does not reflect its internal state. It presents all predictions with equal clarity, whether they rest on solid learning or on extrapolation from low-frequency approximations.
The paper’s framework helps explain why this happens by connecting the training dynamics to the properties of orthogonal polynomials. These polynomials — Legendre, Chebyshev, and others — form a basis for representing functions on an interval. When you expand a function in such a basis, the coefficients capture different scales of variation. The network’s gradient descent dynamics, expressed in this basis, show that the coefficients corresponding to slowly varying basis functions grow quickly, while those corresponding to rapidly varying functions grow slowly. The network is not choosing to learn this way. The structure of the dynamics matrix H makes it a mathematical consequence of gradient descent.
The Consequence Nobody Draws
The paper addresses five goals that earlier work had not fully met: explaining spectral bias asymptotically, extending it to general activation functions, making weak assumptions about data distribution, avoiding aliasing effects, and developing a framework applicable to arbitrary networks. The authors achieve these by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H. The graded structure of H provides the intuitive explanation. The orthogonal polynomial theory provides the analytical tools. The result is a unified account of why networks learn the way they do, applicable across architectures and activation functions.
But the consequence that deserves more attention is this: if spectral bias is a general property of gradient-descent training, then neural networks are systematically biased toward certain kinds of errors. They will be most accurate on problems where the important information lies in low-frequency components. They will be least accurate on problems where high-frequency details matter. And the output alone does not indicate which kind of problem you are facing. The bias is in the training dynamics, not in the final weights. Once training is complete, the weights encode a function that may or may not have learned the high frequencies, but they do not encode the history of how that function was learned.
This is the final gap: the appearance of completeness. A neural network that has learned only the low-frequency components of a function will still produce outputs for any input. It will extrapolate, interpolate, and generate outputs for inputs it has not seen. The user has no way to tell from the output alone how well the high-frequency components were learned. The network itself does not track its own learning progress. It was designed to minimize a loss function, and it did. The loss function did not specify that all frequencies matter equally. It simply summed the squared errors across the training data. The network minimized that sum as best it could, given the structure of the gradient-descent dynamics. The result is a partial approximation presented as a complete one, and the gap between the two is where the problem lives — not in any single prediction, but in the structure of the system itself.