The Subtle Instability of Deterministic Large Language Models: Unpacking the "Feynman Anomaly"

In September 2025, a seemingly simple experiment conducted by Horace He and his team at the Thinking Machines Lab inadvertently illuminated a fundamental fragility within even the most rigorously controlled large language models (LLMs). The researchers posed a query about the renowned physicist Richard Feynman to the Qwen3-235B model, sending the identical prompt a staggering 1,000 times. Crucially, they configured the model to operate at "temperature 0," a setting designed to ensure absolute determinism by compelling the AI to select the single most probable token at each step of generation. The expectation was a perfect replication: 1,000 identical outputs. However, the results defied this expectation, yielding a surprising 80 unique completions. This anomaly, dubbed the "Feynman Anomaly" by some in the AI research community, points to a deeper, often overlooked source of non-determinism in LLM inference.

The most striking aspect of the experiment was not merely the existence of variations, but their concentrated emergence. For the initial 102 tokens of generated text, all 1,000 completions remained in lockstep. Then, at token 103, a subtle divergence occurred: 992 of the outputs continued with the phrase "Queens, New York," while a mere 8 deviated to "New York City." This sudden bifurcation, with 8 out of 1,000 runs flipping at a single token after a prolonged period of perfect agreement, highlighted a critical vulnerability where seemingly minor computational fluctuations could cascade into noticeable differences in output. While He and his colleagues delved into the technical reasons behind why these outputs differed at all, this analysis seeks to explore a more nuanced question: how frequently should such token flips occur, and why do they manifest in sudden, concentrated spikes rather than a uniform distribution of errors?

The Unseen Arithmetic: Tracing the Source of Computational Noise

At its core, the operation of an LLM at temperature 0 relies on deterministic mathematical calculations. The model identifies the token with the highest logit, or raw score, and selects it. However, the digital execution of these calculations, particularly when dealing with floating-point numbers on modern hardware, introduces a form of inherent "noise." Unlike pure mathematics where addition is associative (a + (b + c) = (a + b) + c), the order in which a computer performs floating-point additions can subtly alter the final result.

An illustrative example provided by the research highlights this phenomenon. When summing 100,000 random single-precision (float32) numbers, researchers observed distinct outcomes depending on the summation order. A forward pass yielded approximately -90.82497, a reverse pass resulted in -90.82674, and a pairwise summation using NumPy’s algorithm produced -90.82513. While a double-precision (float64) reference calculation provided a more precise -90.82508, the mere variation across three summation orders for the same set of numbers underscores the sensitivity of floating-point arithmetic.

These billions of sums within a neural network’s architecture mean that the specific order in which hardware processes these operations becomes a significant factor. A popular, though not entirely accurate, explanation for this non-determinism often points to the random order in which GPU threads complete their tasks. However, the findings by He et al. suggest a more complex reality. Their analysis indicates that typical LLM forward passes exhibit minimal "atomic adds" – operations that are inherently synchronized and thus deterministic. Instead, the primary culprit identified is the lack of batch invariance in many computational kernels.

Batch invariance refers to whether the output of a computation remains the same regardless of the order in which elements within a batch are processed. In the context of LLMs, kernels that are not batch-invariant will alter their reduction order based on the batch size. This batch size, in a real-world serving environment, is dynamic and dependent on the concurrent requests from other users. Consequently, an LLM’s output can be influenced by the activity of unseen users, introducing a layer of dependency that undermines the ideal of deterministic inference. The researchers demonstrated that by implementing batch-invariant kernels, they could achieve perfectly identical outputs for all 1,000 Feynman completions. However, this deterministic solution comes with a performance trade-off: the unoptimized deterministic version of Qwen-3-8B on a single GPU took 55 seconds, significantly longer than the default vLLM’s 26 seconds, and even a moderately improved version took 42 seconds.

The Criticality of Near-Ties: A Mathematical Perspective on Instability

From a mathematical standpoint, the key to understanding these output variations lies in the concept of "near-ties" – situations where the logits of the top two most probable tokens are exceptionally close. Let $z_1$ and $z_2$ represent the two largest logits. The gap, defined as $M = z_1 – z_2 ge 0$, quantifies the difference between these scores. Numerical noise, introduced by the aforementioned computational factors, perturbs this gap by a small error, denoted as $delta$. A flip in the chosen token (i.e., a change in the argmax) only occurs if the perturbed gap becomes negative, meaning $M + delta < 0$.

This insight is crucial: tokens with a substantial gap (e.g., 8 logits) are virtually impervious to these minor numerical fluctuations. The risk of a flip is concentrated in positions where the logits are nearly equal. Averaging across all possible token positions, and considering $f(m)$ as the density of token positions exhibiting a gap of size $m$, the probability of a flip ($p$) can be approximated by the integral:

$p approx int P(delta < -m) f(m) dm$

Under the assumption of small noise and a roughly constant density $f(0)$ over the narrow range where $P(delta < -m)$ is non-negligible, the integral simplifies. The expected value of the absolute error, $E|delta|$, plays a pivotal role. For symmetric noise, the integral of $P(delta < -m)$ over $m ge 0$ is approximately $E|delta|/2$. This leads to a simplified formula for the flip probability:

$p approx f(0) cdot E|delta| / 2$

This elegant equation reveals that the probability of a token flip is a product of two factors: the "crowdedness" of the logit space around ties (represented by $f(0)$), which is intrinsic to the model and the specific text being generated, and the magnitude of the numerical noise ($E|delta|$), which is determined by the hardware, precision settings, and computational kernels used. This formulation aligns with related formalizations, such as the "background temperature" concept introduced by Messina and Scotta, which treats these perturbations as an effective temperature within the model’s decision-making process.

Empirical Validation: Simulating and Measuring the "Feynman Anomaly"

To validate this theoretical framework, the researchers first employed simulations. They generated synthetic gap distributions and introduced Gaussian noise of known variance ($sigma$). The core simulation code, remarkably concise at nine lines, allowed for rapid testing. The results demonstrated a strong correlation between the formula’s predictions and the simulated flip rates. For both exponential and half-normal gap distributions, the simulation matched the formula within a 4% margin of error for noise levels up to $sigma = 0.03$. Even at $sigma = 0.3$, the half-normal distribution maintained a close match, with the exponential case deviating slightly as the noise became comparable to the typical gap size. When the noise was significantly larger than the typical gap ($sigma = 1$), the formula’s accuracy decreased, as its core assumption of a flat density over the relevant range began to break down.

A critical test case involved distributions where the gap density vanished at zero ($f(0) = 0$). In such scenarios, the flip rate grows quadratically with noise ($sigma^2$) rather than linearly, indicating a far less sensitive system. The simulation confirmed this, matching the $sigma^2/4$ prediction within 8% for small noise levels. This distinction is vital for understanding whether real-world LLMs operate in a regime where ties are frequent and thus susceptible to noise, or if their internal representations naturally avoid such close calls.

To bridge the gap between simulation and reality, the researchers then tested the formula on a real LLM: GPT-2. They utilized the WikiText-2 dataset, processing 256 windows of 128 tokens each. By comparing a bfloat16 (bf16) forward pass with a full-precision float32 (fp32) pass, they could isolate the effect of lower-precision arithmetic, a common proxy for realistic serving noise on modern hardware. While their laptop setup couldn’t replicate the dynamic batch-size effects of a busy GPU server, it could accurately measure the logit error introduced by lower-precision kernels. The transformer body was processed in bf16, with the output projection in fp32, ensuring the final logits weren’t overly constrained by bf16’s coarser grid.

For each token position, they recorded the top-20 fp32 logits, the gap between the top two, the error incurred on that gap during the bf16 run, and whether the argmax decision had flipped. The analysis revealed a significant density of gaps near zero for GPT-2, with $f(0) approx 0.83$ per logit, and approximately 9% of positions having a gap smaller than 0.1 logits. The bf16 error on these near-ties averaged around 0.02 logits. Applying the formula, the predicted flip rate was approximately 0.01, a figure that closely matched their measured flip rate of 0.01. Further experiments, involving injecting controlled Gaussian noise into the real logits, corroborated these findings, demonstrating a consistent relationship between noise level and flip rate.

From Token Flips to Completion Divergence: The Amplification of Risk

The impact of individual token flips on the overall completion is a matter of probability. The survival probability of a complete generation, $S(t)$, after $t$ tokens, is the product of the per-token survival probabilities: $S(t) approx exp(-sum p_s)$, where $p_s$ is the survival probability at each step. If every token carried the same risk $p$, the survival probability would be $exp(-pt)$, and roughly half of all completions would diverge by token $ln(2)/p$.

However, real text generation is not characterized by uniform risk. The Feynman experiment provides a stark illustration: over the first 102 tokens, 1,000 runs yielded zero disagreements, implying a per-token flip rate of at most around $3 times 10^-5$ (using the "rule of three" 95% confidence bound). At token 103, the rate surged to 8 in 1,000, or 0.8%. This represents a more than 250-fold increase in risk at a single decision point. This indicates that instability is not evenly distributed but rather clusters around specific "knife-edge" tokens where the logits are precariously close.

The variability in the number of these knife-edge tokens across different prompts also significantly influences completion survival. To model this, a simulation was run assuming a mean per-token flip rate of 0.005, but with "fragility" (the propensity for knife-edge tokens) distributed log-normally across different hypothetical prompts. The results showed a dramatic difference in completion longevity. If every prompt had a consistent flip rate of 0.005, approximately half of the completions would diverge by token 139, with only 8% remaining identical at token 500. In contrast, when the mean flip rate was maintained at 0.005 but fragility varied, the median divergence point shifted to token 210, and a substantial 28% of completions still matched at token 500. This emphasizes that the average flip rate alone is an insufficient predictor of completion stability; the distribution of fragility across prompts is equally critical.

Implications and Future Directions

The findings from the "Feynman Anomaly" and subsequent analyses have several significant implications for the deployment and understanding of LLMs:

• Predicting "Flake" Rates: For systems with a known per-token flip rate $p$, the probability of a completion of length $L$ differing between runs can be estimated as $1 – exp(-p cdot L)$. For instance, with $p = 10^-3$ and 500-token answers, this yields a divergence probability of approximately 39%. Researchers and developers can now use measurements of $f(0)$ and the expected noise on their specific hardware stack to predict potential "flake" rates before deploying models.

• Rethinking Output Assertions: The inherent non-determinism, even at temperature 0, necessitates a shift in how LLM outputs are evaluated. Asserting exact equality between outputs from identical prompts should be avoided. Instead, comparisons should incorporate a tolerance or rely on semantic grading, acknowledging the possibility of minor, yet valid, variations.

  

• The Throughput-Determinism Trade-off: For applications where absolute determinism is paramount, the use of batch-invariant kernels offers a solution, as demonstrated by He et al. However, this comes at a cost to inference speed. This highlights a fundamental trade-off that developers must navigate: the speed and efficiency of current LLM serving frameworks versus the absolute predictability of their outputs. Organizations controlling their inference pipelines can invest in optimizing for determinism, while those relying on third-party APIs must assume a degree of variability.

• Understanding Model Behavior: The concept of a "background temperature" or effective temperature provides a useful lens for understanding how subtle computational differences can influence LLM behavior. It suggests that even in deterministic settings, a form of "stochasticity" is implicitly present due to the mechanics of computation.

In conclusion, the seemingly minor divergence observed in the Qwen3-235B experiment serves as a powerful reminder that perfect determinism in LLM inference is an idealized state, not always realized in practice. The interplay between the inherent structure of language models, the mathematical properties of floating-point arithmetic, and the specific computational environment creates a subtle yet persistent source of non-determinism. By quantifying this noise and understanding the concentration of risk around near-ties, researchers and engineers can better predict, manage, and ultimately mitigate the variability in LLM outputs, paving the way for more robust and reliable AI systems.