---
title: "LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training"
authors: ["Abhishek Tyagi", "Saurabh Hukerikar", "Nirmal Saxena", "Yanxiang Huang", "Philip Shirvani", "Chung-Hsuan Tung", "Yuhao Zhu"]
url: "https://arxiv.org/abs/2604.10390"
sections: 45
estimated_tokens: "16.2k"
---

## Contents
- I Introduction
  - This work makes the following key contributions:
  - Paper Overview:
- II Background
  - II-A Permanent Faults
  - II-B Silent Data Corruption (SDC)
  - II-C LLMs Training at Scale
  - II-D Intuitions About LLM Fault Resilience
  - II-E Related Work
- III Methodology
  - III-A From Physical Defects to Logical Faults
    - Assumption:
  - III-B RTL Fault Characterization
  - III-C Software Fault Site Parameterization
    - III-C 1 Spatial and Architectural Coordinates
    - III-C 2 Temporal Injection Parameters
    - III-C 3 Fault Intensity & Bit-Level Mechanics
  - III-D Instantiating a Fault Site: Fixed and Variable Parameters
    - III-D 1 Fixed Parameters
    - III-D 2 Variable Parameters
  - III-E Modeling Intermittent Faults With Error Rate
- IV Experiment Setup
  - IV-A Models and Dataset
    - Models:
    - Dataset:
  - IV-B Data Formats
  - IV-C Hardware, Software and Training Recipe
    - Hardware:
    - Software:
    - Training Recipe:
- V Results
  - RQ1: How do permanent faults propagate through training, and what makes them different from transient faults?
    - Weight divergence reveals what loss curves hide:
    - Failure mode prevalence across formats:
  - RQ2: How resilient is LLM training to permanent faults?
    - 1) Model training dynamics under fault.
    - 2) Impact on PPL.
    - 3) Downstream performance.
  - RQ3: How does the choice of numerical data format influence the resilience of LLM training to permanent hardware faults?
    - Data format strongly modulates resilience: FP8 > > BF16 > > FP16.
  - RQ4: How effective is loss-NaN checking in reducing the impact of permanent faults on training dynamics?
    - Cross-format summary.
- VI Limitations
- VII Conclusion
- References

## Abstract

Abstract Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience.

## I Introduction

The rapid growth of large language models (LLMs) has driven the deployment of massive-scale AI infrastructure, with state-of-the-art training runs now executing on clusters with tens of thousands GPUs spanning weeks to months [12]. As these systems scale, the reliability of the underlying hardware becomes increasingly crucial. Silent data corruption (SDC), faults that lead to incorrect computations without triggering detectable errors  [6, 14, 29], poses a serious threat to the correctness, stability, and reproducibility of LLM training [10]. Even minor deviations during training can compound through billions of updates, potentially degrading model accuracy, convergence behavior, or downstream model accuracy and/or performance in subtle and costly ways.

Most prior studies assessing the resilience of deep learning workloads have focused on transient faults [27, 30, 28]. These non-persistent faults, often caused by cosmic radiation or particle strikes manifest as bit-flips that momentarily corrupt the architectural state of processors.
However these analyses overlook another class of reliability challenges — permanent faults, which can manifest either continuously or intermittently depending on operating conditions.
Permanent faults stem from manufacturing defects that escape testing, latent defects that become active during the early operational lifetime, or silicon wear-out mechanisms [24].
Unlike transient faults, these errors can silently corrupt computations over long periods or intermittently manifest depending on workload activity, temperature, or voltage conditions. As semiconductor technology advances towards sub-5nm process nodes and with the growing architectural complexity of modern AI accelerators featuring billions of transistors, extensive on-chip interconnects, and heterogeneous compute fabrics, this class of faults is becoming increasingly prevalent.

Emerging evidence from LLM production training workloads indicates that permanent or marginal hardware defects that causes SDC introduces subtle numerical errors that do not trigger failures but quietly perturb gradients and parameters [7, 18, 9]. However their cumulative effect on training dynamics and final model behavior remains largely unexplored and not well understood. Therefore, the core focus of this work is on characterizing the sensitivity of LLM training to silent data corruption caused by permanent faults, with particular emphasis on their intermittent manifestations.

##### This work makes the following key contributions:

- •
We develop a fault model, a quantitative abstraction of how underlying hardware defects perturb architectural state, for permanent hardware defects, including their intermittent manifestations, relevant to modern GPUs, and demonstrate how it can be instantiated within NVIDIA’s Megatron-LM [26] framework to enable systematic fault injection in large-scale LLM training.
- •
We study how such permanent faults perturb the LLM training pipeline, analyzing their impact on loss trajectories, convergence behavior, and final model quality.
- •
We perform an end-to-end evaluation of LLM training simulating thousands of permanent faults across multiple precision types (FP16, BF16, FP8)
- •
We evaluate widely deployed runtime monitoring mechanism like NaN/Inf detectors and quantify its effectiveness and blind spots for detecting SDC induced by permanent and intermittent faults.

Understanding how these faults propagate through the training pipeline can inform fault-aware system design, more robust algorithmic methods to detect and errors, or the use of selective redundancy strategies.

##### Paper Overview:

Section [II](#S2) reviews permanent hardware fault mechanisms and their implications for modern LLM training. Section [III](#S3) presents LLM-PRISM, our methodology for characterizing SDC resilience in LLM pre-training. Section [IV](#S4) describes the experimental setup. Section [V](#S5) presents the main findings from a large-scale empirical study, spanning RTL-level fault characterization and end-to-end effects on training dynamics, including the behavior of NaN/Inf detection across precision formats. Section [VI](#S6) discusses limitations, and Section [VII](#S7) concludes with implications for fault-aware LLM training systems.

## II Background

### II-A Permanent Faults

Permanent faults arise from three distinct root causes: manufacturing test escapes, where defective devices pass production screening and only reveal themselves under field conditions; latent early-life defects, which surface during the initial operational period as marginal structures fail under stress; and aging-induced failures, which develop progressively over a chip’s lifetime as wear-out mechanisms such as BTI [11], Hot Carrier Injection (HCI) [15], TDDB [4], and Electromigration (EM) [1] degrade transistor device characteristics.
At the logical level, permanent faults may manifest as stuck-at faults in combinational logic, flip-flops, or RAM arrays, where a node is permanently fixed at a logic ’0’ or ’1’ regardless of the applied input — these are timing-independent faults whose effect is deterministic and persistent across all operating conditions. Partial degradation of transistors or interconnects introduces small delay defects that reduce the available timing margin in a circuit path. When these delays grow sufficiently large, they cause violations of setup or hold time constraints, leading to incorrect values being latched or propagated. Such faults are classified as timing-dependent permanent faults, or intermittent faults, exhibit marginality - they tend to manifest selectively at specific voltage or frequency operating points. Such timing-dependent, intermittent faults are critical concern in advanced process nodes and in compute elements and RAM arrays — where aggressive voltage/frequency scaling and current densities exacerbate marginal timing behavior.

### II-B Silent Data Corruption (SDC)

SDC denotes a class of hardware errors in which a system produces incorrect computation results without triggering explicit failure signals or program crashes. These errors typically manifest as bit flips in memory or logic, unintentionally altering the binary representations of parameters, activations, or gradients during LLM training. Unlike fail-stop errors that halt execution, SDC allows training to proceed with a corrupted state, perturbing intermediate computations and silently altering the optimization trajectory. The root causes of SDCs are diverse, spanning transient, intermittent, and permanent hardware faults that degrade execution reliability, with intermittent and permanent faults often arising from latent silicon defects, circuit aging, or aggressive voltage and frequency scaling that create “unhealthy nodes” which consistently or sporadically produce errors under high-stress conditions. In the context of LLM training, such silent perturbations can accumulate over billions of update steps, making SDC particularly problematic because it undermines both numerical correctness and the reproducibility of large-scale training runs.

### II-C LLMs Training at Scale

The training of Large Language Models (LLMs) has reached an unprecedented scale, necessitating distributed systems that span thousands of accelerators to handle models with hundreds of billions of parameters. State-of-the-art models require massive computational resources over extended periods; for instance, training Llama 3 405B utilized a cluster of 16,000 H100 GPUs [18], and other large-scale efforts similarly employ tens of thousands of chips. This massive parallelism typically combines data, tensor, and pipeline parallelism to partition the workload, so that a fault in a single node can perturb the global training run.
As the number of hardware components in a training cluster increases, the aggregate probability of encountering hardware failures, including SDCs, rises significantly. Empirical reports from major industry training runs indicate that SDC events are no longer rare anomalies but regular occurrences at scale: Meta attributed six unplanned job interruptions to SDC during a 54-day pre-training snapshot [18], and Google estimated that an SDC event occurs every week or two during Gemini training [9]. Compounding this issue, modern high-performance LLM training workloads push hardware utilization to its limits, which can activate latent defects in “marginal” hardware that would likely pass manufacturing screening stress tests [3].

### II-D Intuitions About LLM Fault Resilience

A widely held intuition is that LLM training may be inherently resilient to transient hardware errors: the optimization process is stochastic by design, gradient clipping and norm-based regularization act as implicit error dampeners, and the sheer volume of gradient updates across billions of parameters can dilute the impact of isolated corruptions. This intuition has merit for occasional transient faults, but it does not necessarily extend to permanent and intermittent hardware faults, where errors recur systematically across training steps rather than appearing as isolated perturbations. Given the scale and economic significance of large-scale LLM training infrastructure, characterizing the degree to which such permanent faults degrade convergence and destabilize training dynamics is therefore necessary to reason rigorously about the resilience properties required of large-scale LLM training infrastructure.

**TABLE I: Comparison of prior LLM fault-tolerance and resilience studies. We contrast studies along key dimensions including hardware realism, fault type, training phase, injection scope, distributed support, and scale. For our work, total compute time is estimated from 7,664 complete training runs by weighting model training time by the number of runs per model. Sun et al. [27] and Yu et al. [30] report total hours explicitly; Ma et al. [19] and Chen et al. [2] do not report total GPU hours, so their reported experimental bounds are listed instead.**
| Work | Hardware<br>Accurate | Fault Type | Ease of<br>Exp. | Target<br>Phase | Injection<br>Scope | Analyzes<br>Dynamics | Distributed<br>Support | Total Compute<br>/ Scale |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ma et al. [19] | ✓ | Real / Latent | $\times$ | Pre-train &<br>Fine-tune | Submodules, Gradients | ✓ | ✓ | Not reported<br>(30 nodes, 4.5k steps) |
| Chen et al. [2] | $\times$ | Transient | ✓ | Fine-tune &<br>Inference | Weights, Activations | $\times$ | $\times$ | Not reported<br>(5 epochs/run) |
| Sun et al. [27] | $\times$ | Transient | ✓ | Inference Only | Weights, Neurons | $\times$ | $\times$ | $\sim$4,800 hours<br>(13M+ runs) |
| Yu et al. [30] | $\times$ | Transient | ✓ | Training | Forward, Gradients,<br>Optimizer states | ✓ | ✓ | $>$5,000 hours<br>(300K+ runs) |
| Our Work | ✓ | Permanent | ✓ | Training | Weights, Inputs,<br>Gradients | ✓ | ✓ | $\approx$11,500 hours<br>(7,664 runs) |

### II-E Related Work

Table [I](#S2.T1) exposes the key limitations of prior LLM fault-tolerance and resilience studies, particularly their reliance on idealized fault models, short evaluation horizons, and limited coverage of end-to-end training dynamics.
These studies differ in fault realism, evaluation horizon, and whether they capture end-to-end training dynamics. Idealized bit-flip injections, short-horizon evaluations, and inference-only studies can miss failure modes that emerge under hardware-grounded faults and long-running pre-training.

Sun et al. [27] present a large fault-injection study, but focus only on inference. Yu et al. [30], Chen et al. [2], and Ma et al. [19] extend the scope to training, but remain limited by either short fine-tuning horizons or reliance on physically degraded hardware, which constrains reproducibility and the range of fault signatures that can be explored, and the range of training trajectories that can be observed. In particular, prior training studies are often too short to expose long-tail phenomena such as delayed loss spikes, divergence, or silent corruption that only become visible over extended runs.

Our work addresses these gaps by combining hardware-grounded fault models with software-based injection inside the training loop, enabling large-scale pre-training studies from initialization to convergence. This allows us to study not only average degradation, but also loss spikes, NaN/Inf anomalies, and divergence across the full optimization trajectory.

## III Methodology

The methodology first situates our approach in the fault-injection design space by contrasting RTL-level and model-level injection (Section [III-A](#S3.SS1)). It then derives hardware-grounded error signatures from representative GPU functional units under realistic stuck-at and timing-dependent defects (Section [III-B](#S3.SS2)). These signatures drive a software-level fault-injection framework embedded in the training pipeline and parameterized by a seven-element fault-site tuple that specifies where, when, and how faults occur (Section [III-C](#S3.SS3)). Finally, a Bernoulli error-rate parameter models intermittent activation and unifies permanent and intermittent faults within a single probabilistic framework (Section [III-E](#S3.SS5)).

### III-A From Physical Defects to Logical Faults

Permanent GPU defects, including intermittently activated ones under varying thermal and electrical stress, are a key source of SDC during LLM training. To study them systematically, we map physically plausible defect mechanisms to logical perturbations in the training pipeline. Rather than simulate faults at the hardware level, we simulate their observable error symptoms using statistical activation models that capture when and how errors appear over time. This approach preserves the realism of permanent and intermittent defects while enabling scalable end-to-end evaluation. We instantiate it in Megatron-LM, a production-grade training framework widely used for transformer models on GPUs.

##### Assumption:

In this work, a single underlying hardware defect is modeled at a time, i.e., at most one permanent fault is assumed to be active in the system over the duration of each training run; this simplifies attribution of observed behavior to a specific defect and reflects the fact that, at realistic fault rates, multi-defect interactions are relatively rare and can be studied in future work.

### III-B RTL Fault Characterization

Figure: Fig. 1: RTL characterization flow for error signature extraction.
Refer to caption: 2604.10390v1/x1.png

Figure: Fig. 2: Software level fault site tuple characterized with seven parameters divided in three broad categories: 1) Temporal 2) Spatial and Architectural 3) Fault Intensity & Bit-Level.
Refer to caption: 2604.10390v1/x2.png

Figure [1](#S3.F1) summarizes our RTL characterization flow. Rather than simulate the full LLM training workload at RTL — which is computationally intractable — we target the GPU functional units most exposed during transformer training: FMA paths in the CUDA cores, tensor cores, register files, and L1/L2 caches and SRAM arrays. Simulations are driven by computation traces extracted directly from the training workload and replayed through a commercial EDA tool to obtain cycle-accurate functional-unit activity under realistic input and toggle conditions.

We inject stuck-at faults sampled in proportion to the silicon area of each unit, ensuring the fault population reflects each block’s contribution to the overall GPU design. For each fault, we record its output error signature: the set of affected bits, the magnitude of the numerical deviation, and the corruption pattern at the functional-unit output. These signatures characterize the what of each fault — which bits are affected, at what magnitude, and in what pattern — while the where and when are determined by architectural reasoning about GPU data flow and training dynamics, as described in next subsection.

Figure: Fig. 3: Examples error signatures derived from RTL characterization.
Refer to caption: 2604.10390v1/figs/rtl-signatures.png

The error signatures in Figure [3](#S3.F3) are illustrative examples derived from single stuck-at fault RTL simulations. For Figure [3](#S3.F3)(a), the fault is injected into a tensor core datapath bit that feeds a subset of output tiles, producing localized $3\times 3$ patches in the matrix where all corruptions share the same flipped bit position, resulting in a sharply peaked bit-position distribution. For Figure [3](#S3.F3)(b), the fault is injected on an output bit of an FMA ALU, perturbing low- and mid-order bits in a pattern-dependent manner so that most outputs remain correct but a small number of operations exhibit sporadic, high-magnitude numeric outliers with a broader bit distribution. In case (c), the stuck-at fault is in L1 cache line such that an entire matrix row’s stored values are corrupted when accessed, yielding structured row-wise corruption and a wide bit-position distribution centered on high-order bits. We collected over 6,000 such error signatures from faults injected throughout the GPU memory hierarchy and compute datapaths providing a diverse empirical basis for the software-level fault injection.

### III-C Software Fault Site Parameterization

RTL fault injection operates on the registers and combinational data paths of a functional unit; the analogous injection targets in a software training loop are the tensors flowing through the computation graph — activations, gradients, and weights. Translating hardware-derived error signatures into this setting requires a structured parameterization that captures the
spatial, temporal, and numerical dimensions of a physical defect as quantities controllable within Megatron-LM.

The core challenge is that a single stuck-at defect has a fixed physical location, but its effect on training depends on which tensor is being computed, on which device, at which step, and at what numerical precision. A parameterization that spans all of these dimensions lets us reproduce the full diversity of hardware failure modes during distributed training with Megatron-LM.

To this end, we define a software-level fault site as a tuple of seven parameters organized into three categories (Figure [2](#S3.F2)). The fault intensity and bit-level parameters are drawn from the RTL-derived error signature distributions established above; the spatial and architectural
parameters follow from first-principles reasoning about how a fixed hardware defect interacts with distributed training computation; and the temporal parameters capture the stochastic activation behavior of intermittent defects, as detailed in Section [III-E](#S3.SS5). The seven parameters are described below:

#### III-C 1 Spatial and Architectural Coordinates

These three parameters specify which hardware and which part of the model is affected (Figure [2](#S3.F2)b).

a) Rank identifies the faulty GPU in the distributed cluster, simulating the common datacenter scenario where a single unhealthy device operates alongside otherwise healthy peers.

b) Layer pins the fault to a specific transformer layer and sub-module — Multi-Head Attention, MLP, or LayerNorm — enabling study of whether certain components are more susceptible to corruption than others.

c) Training Phase determines which tensor is corrupted: forward-pass activations (fwd_outputs), backward-pass input gradients (bwd_grad_inputs), or weight gradients (bwd_grad_weights). The distinction is consequential: an activation error is consumed and discarded, whereas a weight gradient error propagates directly into the model parameters,
causing persistent drift in the optimization trajectory.

#### III-C 2 Temporal Injection Parameters

These two parameters specify when and how often corruption occurs during training.

a) Fault Checkpoint sets the iteration at which injection begins, enabling controlled comparisons of model vulnerability at different stages of convergence — early training, when gradients are large and noisy, versus later stages, when the model is near a local minimum and more sensitive to perturbation.

b) Error Rate controls the fraction of eligible iterations that experience a fault, spanning the full persistence spectrum: a rate of 1.0 models a permanent stuck-at defect corrupting every iteration, while a low rate models an intermittent defect that activates under specific input or operating conditions. Section [III-E](#S3.SS5) describes how this parameter is derived and swept.

#### III-C 3 Fault Intensity & Bit-Level Mechanics

These two parameters specify the severity and structure of the numerical corruption applied to the target tensor.

a) Fault Density controls the proportion of corrupted elements within the target tensor tile. A low density models a localized defect; a higher density models a fault in shared logic — such as a cache line or register file entry — that corrupts multiple values simultaneously.

b) Bit-Flip Profile specifies which bits of the floating-point representation are flipped and how many. Flipping an exponent bit can shift a value by orders of magnitude (e.g., $3.141\rightarrow 3248$), whereas mantissa flips produce minor precision loss. By replaying the bit-flip patterns observed in the RTL error signatures, this parameter anchors the software fault model in hardware-measured corruption rather than uniform random bit-flip assumptions.

### III-D Instantiating a Fault Site: Fixed and Variable Parameters

Instantiating the fault-site tuple for a concrete injection campaign requires deciding which of its seven parameters reflect fixed properties of the physical defect and which must be resampled as the data stream changes across iterations. A physical defect is anchored to one location on one device, but the tensor it corrupts, the layer it affects, and the bit pattern it produces all depend on what computation happens to be scheduled on the defective unit. This distinction between defect-level properties and data-stream interactions determines which parameters are initialized once at
campaign start and which vary throughout training.

#### III-D 1 Fixed Parameters

a) Rank is fixed because a hardware defect resides on a specific physical device. A degraded functional unit on GPU $k$ remains faulty for the entire training run, and we assume at most one faulty GPU at any time.

b) Fault Checkpoint and c) Error Rate are fixed because they describe the temporal characteristics of the defect rather than any individual corruption event. The checkpoint defines the training window during which the fault is active — modeling, for example, a defect that emerges after thermal stress accumulates over extended operation. The error rate captures how frequently the defect propagates to a visible output error within that window.

#### III-D 2 Variable Parameters

a) Training Phase is sampled uniformly from
fwd_outputs, bwd_grad_inputs, and
bwd_grad_weights. This is an architectural choice: the forward pass performs one matrix computation per layer, whereas the backward pass performs two (input and weight gradients), so a uniform draw over three phases yields a natural 2:1 exposure ratio for backward versus forward computation without explicit timing measurements.

b) Layer is sampled uniformly across transformer layers. Because layers are structurally identical and occupy the same GPU for comparable durations, uniform sampling reflects equal exposure of a fixed defect to each layer’s computation.

c) Fault Injection Tile is sampled to match GPU execution granularity. Matrix operations are executed in fixed-size tiles; a defect corrupts whatever tile is scheduled on the faulty unit at activation. We therefore select a target tile at random and derive the intra-tile corruption pattern from the RTL error signature distributions.

d) Bit-Flip Profile is drawn from the RTL error signatures with single-bit flips weighted more heavily than multi-bit patterns.

### III-E Modeling Intermittent Faults With Error Rate

Figure: Fig. 4: Stochastic Intermittent Fault Activation Model
Refer to caption: 2604.10390v1/x3.png

To model intermittent faults — permanent hardware defects that only manifest on a subset of operations — we interpret the error-rate field in the fault tuple as the probability that a given spatial fault site produces a visible corruption during training. Whether a given operation produces a visible error depends on whether the input data exercises the defective logic path, or whether the operating conditions — voltage, frequency, or thermal stress — cross the threshold at which the defect becomes active. This combination of data-dependent and condition-dependent activation is what makes permanent faults behave intermittently at the software level.

The *error injection rate* $r\in(0,1]$ is defined for a given defect as the per-iteration probability that the defect produces a visible corruption at the software level, conditioned on the operation using the faulty resource. In other words, $r$ summarizes the defect’s *detectability*: the fraction of input patterns that exercise the faulty logic path and propagate an error to architecturally visible state. Each fault tuple is associated with a fixed rate $r$ for the duration of an injection campaign. At each eligible iteration, a Bernoulli trial with parameter $r$ decides whether the defect activates; on activation, the fault tuple is used to inject the error to the target tensor. A rate $r=1.0$ models a hard stuck-at defect that corrupts every eligible operation, whereas $r\ll 1$ models a timing‑marginal or latent defect that rarely manifests.

To study how activation frequency shapes training behavior, we treat $r$ as a hyperparameter and sweep it across a broad range. Figure [4](#S3.F4) visualizes the fault activation over training steps as a Bernoulli process with rate $r$, and the the corresponding probability density function $f(r)$. We construct PDF $f(r)$ based on prior reliability reports: these studies consistently observe that most permanent or intermittent defects activate rarely, while a smaller fraction exhibit much higher activation rates. We therefore choose a unimodal distribution with most probability mass concentrated at low $r$
(latent and marginal defects), but with a long tail covering higher
$r$ values corresponding to aging and severely degraded structures. For our experiments, we select a set of representative rates spanning these regions and run separate campaigns at each value, so that each campaign probes a distinct point in this activation-rate design space. This ties the error-rate parameter back to physical intuition about intermittent defects while providing a systematic knob for characterizing how faults of different activation frequencies affect LLM training stability.

## IV Experiment Setup

### IV-A Models and Dataset

##### Models:

We perform all fault-injection experiments on the GPT-2 family of decoder-only Transformers [25], using the 124M (GPT-2 Small) and 355M (GPT-2 Medium) variants. These models represent the foundational archetype for modern decoder-only LLMs (e.g., Llama, GPT-4), while remaining small enough to support thousands of full training runs. To study the effect of model
scaling under realistic training dynamics, we train both variants from scratch on the WikiText language modeling dataset [21]. This experimental design deliberately trades absolute model size for statistical rigor: by restricting to two GPT-2 architectures, we are able to run 7,664 complete fault-injected pre-training experiments, enabling fine-grained evaluation of a large space of
fault sites and activation rates.

##### Dataset:

All the experiments are carried out with WikiText [21] dataset and we measure the PPL of all the runs to see the impact of fault injections at the macro level.
A rigorous fault injection campaign requires thousands of targeted training iterations to achieve statistical significance across a massive configuration space (layer depth, error rate, data format etc). By utilizing a single, sufficiently complex dataset that thoroughly exercises the model’s computational graph, we eliminate dataset-induced variance as a confounding factor while keeping the computational overhead of exhaustive fault profiling tractable.

### IV-B Data Formats

To evaluate the impact of hardware faults across modern mixed-precision regimes, we run all fault-injection campaigns under three training formats: IEEE FP16, BF16 [16], and FP8 [23], which are the de facto low-precision standards. Rather than varying datasets or model sizes, we fix the architecture and data and sweep the numerical format, since the low-level propagation of a bit-flip through PyTorch primitives is governed primarily by the underlying representation. For example, flipping an exponent bit in an FP16 tensor can change a value by orders of magnitude, whereas the same spatial fault in a BF16 tensor, with a wider exponent and narrower mantissa, produces a different corruption profile; FP8 further alters this trade-off by aggressively compressing both range and precision. While increasing model scale is known to affect activation outlier distributions and thus quantization behavior [5], isolating the numerical format at a controlled scale allows us to rigorously probe a large space of hardware-level bit-flips and activation rates, yielding more direct, actionable insight into how representation choice shapes training-time fault sensitivity.

### IV-C Hardware, Software and Training Recipe

##### Hardware:

The fault injection training experiments were run on 16× NVIDIA H100 GPUs spread across two 8-GPU nodes, using data-parallel Megatron-LM training. Each experiment occupies all 16 GPUs across both nodes.

##### Software:

We utilize the open-source Megatron-LM [26] framework, which we specifically modified to integrate our targeted fault injection methodology during model training.

##### Training Recipe:

We employ a standard configuration using the Adam optimizer, linear warmup with cosine annealing, and sequence parallelism for distributed workloads.
Across all configurations, this setup yields 7,664 complete fault-injected pre-training runs from scratch (4,681 for GPT-2 Small and 2,983 for GPT-2 Medium), in which RTL-derived error signatures are used to construct the fault-site tuples and the error-rate parameter is swept across its range.

## V Results

Figure: Fig. 5: Training loss traces for four representative failure modes under permanent fault injection. Each subplot shows loss over training iterations (GPT2-Small with BF16 format); the dashed line is the fault-free baseline (3.352 nats / PPL = 28.54) and red triangles mark iterations where the loss calculation produced a NaN. Plot 1:(a)Spike-and-Recover: sparse NaN events are absorbed and the run converges to baseline. (b)Spike-and-Degrade: dense NaN events overwhelm recovery and loss diverges. (c)Silent Degradation: no NaN events, yet loss settles above baseline; the most operationally deceptive mode. (d)Gradual Drift: loss is visually indistinguishable from baseline while parameters are silently corrupted. Plot 2: Global weight divergence ($\|W_{\text{fault}}-W_{\text{baseline}}\|_{2}$) over training, for GPT-2Small with BF16. All fault rates produce monotonically increasing divergence after injection begins. Plot3: Distribution of the four failure modes across data formats and model scales. FP8 produces no Silent Degradation or Gradual Drift. Its outcomes are strictly binary (recover or crash). FP16 shows the highest Spike-and-Degrade fraction, consistent with its wider dynamic range enabling large-but-finite corruptions.
Refer to caption: 2604.10390v1/x4.png

### RQ1: How do permanent faults propagate through training, and what makes them different from transient faults?

Prior studies on transient faults during training typically inject a single bit flip and observes whether the training recovers or crashes [30, 2]. This probes resilience to a one-time perturbation, but not the behavior of a GPU that persistently corrupts computations over hundreds of thousands of iterations. Our pre-training runs are subjected to permanent fault injection, revealing four qualitatively distinct failure modes (Plot 1 in Figure [5](#S5.F5)), only the first of which would be observable under a transient fault model.

1) Spike-and-recover.
At low fault rates, corruptions are too infrequent to overwhelm the training dynamics. Each NaN event triggers mixed-precision recovery (loss-scale reduction and update skipping [22]), after which gradient descent proceeds with predominantly clean updates. The loss therefore exhibits isolated spikes followed by return to the fault-free trajectory. This is the regime that single-injection studies capture and typically interpret as *training is resilient* to the injected fault.

2) Spike-and-degrade.
At higher fault rates, NaN events become sufficiently frequent that the loss scale collapses and the skip-and-halve mechanism no longer restores stability. The optimizer is effectively stalled: many steps are skipped, the effective learning signal vanishes, and the loss rises monotonically as corrupted activations repeatedly propagate through the network. In contrast to spike-and-recover, fault rate here exceeds the threshold at which training dynamics can self-heal.

3) Silent degradation.
Faults are frequent and severe enough to corrupt the training state, but not enough to crash the job or consistently produce NaNs. Each faulty iteration perturbs activations or gradients while remaining finite, so weights are updated using biased signals and the loss drifts upward over time. The run completes without obvious instability indicators, making this behavior operationally difficult to detect.

4) Gradual drift.
Faults predominantly affect backward-pass quantities (e.g., gradient inputs), introducing small but systematic errors into each update step. These biases accumulate over long horizons, leading to a slow divergence from the fault-free trajectory without sharp loss spikes or NaNs. This mode is invisible to NaN-based monitoring and unlikely to be exposed by short fine-tuning runs, but becomes apparent over full pre-training durations.

##### Weight divergence reveals what loss curves hide:

Plot 1 in Figure [5](#S5.F5) shows the loss trajectories that a practitioner *would* observe, whereas Plot 2 reports the L2 distance between the faulty and fault-free parameter vectors throughout training. All permanent fault runs exhibit monotonically increasing parameter divergence once injection begins; the model does not rejoin the baseline trajectory. Runs with error rates $r\leq 0.005$ remain near-zero divergence, consistent with benign perplexity outcomes, while runs with $r\geq 0.01$ show steadily growing distance. Notably, even trajectories whose loss remains near baseline (Gradual Drift) accumulate substantial weight drift, indicating that loss alone can mask significant deviations in the training parameters.

##### Failure mode prevalence across formats:

Plot 3 summarizes how the four failure modes distribute across numerical formats and model scales. Two effects are most prominent: FP8 produces no instances of Silent Degradation or Gradual Drift: outcomes are effectively binary (recover or crash), which is consistent with its narrow dynamic range forcing faults either to saturate harmlessly or to trigger NaNs; in contrast, FP16 exhibits the largest fraction of Spike-and-degrade runs - its wider dynamic range permits large but finite corruptions that degrade the model
without necessarily causing a crash. These format-dependent patterns indicate that numerical representation is a first-order determinant of how permanent faults manifest at the training-loss level.

Figure: Fig. 6: Distribution of training outcomes under permanent fault injection across data formats for GPT2-Small(a) and GPT2-Medium(b). Left: final model PPL classified as Unchanged, Changed, or Crashed. Middle and right: fraction of runs encountering NaN/Inf in loss or weights/activations, with sub-bars showing the subset that also resulted in a PPL change. Lower-precision formats crash more but degrade silently less; larger models show a higher fraction of silent degradation.
Refer to caption: 2604.10390v1/x5.png

### RQ2: How resilient is LLM training to permanent faults?

##### 1) Model training dynamics under fault.

We first characterize the impact of fault injection along three complementary dimensions summarized in Figure [6](#S5.F6). The primary dimension is final model quality: we measure the perplexity (PPL) of the trained model and classify each run as *Unchanged* (PPL within 1% of the fault-free baseline of 28.54), *Changed* (PPL deviates by more than 1%), or *Crashed* (training does not produce a valid PPL). The 1% threshold is empirically motivated: repeated fault-free runs exhibit up to 1% natural variance, so smaller deviations cannot be confidently attributed to faults.
The second dimension records whether the training loss becomes NaN or Inf at any iteration, indicating a breakdown of the optimization objective. The third dimension tracks NaN/Inf events in weights or activations, capturing instability propagating through the network independently of the loss. For the
latter two dimensions, a sub-bar reports what fraction of affected runs also exhibit a PPL change, linking low-level numerical faults to observable model degradation.

Across all formats, the dominant trend is that LLM training is remarkably resilient to permanent hardware faults: 57–76% of runs complete with PPL indistinguishable from the fault-free baseline. Model behavior under failure, however, depends strongly on numerical precision. For GPT-2 Small, FP16’s remaining 28% of runs split between graceful degradation (11.0% PPL Changed) and crashes (16.7%), with NaN/Inf loss events (14.6%) serving as the primary propagation pathway: roughly 60% of FP16 runs that encounter a NaN/Inf loss still finish training but with elevated PPL.
BF16 and FP8 follow a different pattern: crash rates rise to 26.9% and 41.9%, while the fraction showing non-crashing PPL degradation falls to 6.3% and 0.5%. This inverse relationship between crash rate and PPL-change rate reflects a shift in failure mode: lower-precision formats have narrower dynamic ranges, so faults that
overflow to NaN typically produce catastrophic failure rather than recoverable, but degraded, solutions. Consequently, in BF16 and FP8,
NaN/Inf loss events almost always correspond to either a full crash or a near-baseline outcome, with little middle ground. The same trend appears for NaN/Inf in weights and activations, which closely track loss instability across formats, indicating that faults originate in the computational layers and propagate upward into the loss. GPT-2 Medium (Figure [6](#S5.F6)b) exhibits a similar qualitative pattern but with a larger Changed fraction (13.8% for FP16 vs. 11.0% for Small), suggesting that larger models provide more opportunities for faults to induce silent degradation rather than crashes.

##### 2) Impact on PPL.

We next examine which fault parameters drive variation in model quality. Figure [8](#S5.F8) reports the distribution of final PPL for fault-injected runs, stratified by fault rate, injection checkpoint, and fault phase for FP16, BF16, and FP8; the fault-free baseline is PPL = 28.54. The following observations summarize the main trends:

- Obsv. 1:
Most runs survive, but tail risk is severe.
Across all formats, the majority of runs converge near the baseline. For FP16 at fault rate 1.0, the median PPL is 28.58, yet the 95th percentile reaches 192.4, corresponding to converged but poor solutions rather than outright divergence. Permanent faults can therefore silently degrade models
without triggering obvious failures.
- Obsv. 2:
Low fault rates are benign.
At rates 0.001 through 0.01, PPL is statistically indistinguishable from the baseline across all three formats. Degradation appears only at rates 0.05 and above, consistent with the optimizer absorbing infrequent corruptions through subsequent updates. Format-dependent differences in distributional spread become pronounced at higher rates.
- Obsv. 3:
Earlier faults cause more damage.
Faults injected at step 2571 yield broader PPL distributions than those at steps 5142 or 7713. Early in training, the model has not yet reached a stable region of the loss landscape and is therefore more sensitive to perturbations.
- Obsv. 4:
Backward weight gradients are resilient; forward outputs and backward input gradients are not.
Faults in bwd_grad_wt have little effect on PPL, due to two mitigating factors: optimizer momentum smooths corrupted gradients over multiple steps, and in distributed training, gradients are averaged across GPUs, diluting the impact of a single faulty rank by a factor of $1/N$. In contrast, faults in fwd_out and bwd_grad_inp act on local computation before any cross-rank communication and thus propagate without attenuation.

##### 3) Downstream performance.

Perplexity is an imperfect proxy for task performance: models with similar PPL can differ substantially on downstream evaluations [20]. To test whether training-time corruption from permanent faults translates into real capability loss, we evaluate GPT-2 Medium models using the LM Evaluation Harness [8] on the Children’s Book Test (CBT) [13] and the Winograd Schema Challenge [17]. GPT-2 Small is excluded as it lacks sufficient capacity to produce meaningful scores on these tasks.

Figure [7](#S5.F7) shows that downstream degradation is not confined to runs with obviously corrupted loss curves.
The Spike-and-degrade models perform poorly across all benchmarks. The Spike-and-recover and Gradual Drift models, which maintain near-baseline PPL, still exhibit measurable accuracy drops on both tasks. These results confirm that permanent faults can harm learned representations in ways that are not visible in training perplexity alone, and that loss-curve monitoring is insufficient to detect all forms of training-time corruption induced by faulty hardware.

Figure: Fig. 7: Performance of faulty GPT2-Medium (BF16) models on downstream tasks. We classify the models in the four categories and run them on Children’s Book Test Dataset [13] (CN = Common Nouns and NE = Named Entities) and Winograd Schema Challenge [17]. We can see that even for cases like spike recover or gradual drifts, the downstream performance is inferior to that of the fault free baselines on downstream tasks due to a change in the training dynamics.
Refer to caption: 2604.10390v1/x6.png

### RQ3: How does the choice of numerical data format influence the resilience of LLM training to permanent hardware faults?

Figure: Fig. 8: Variation in PPL studied against fault rate, fault checkpoint and fault phase. We can see that as the fault rate increases, we can see a variation in the PPL. Also, earlier the fault is injected, higher chances that it will manifest as a change in PPL. And, when faults are injected in the gradients wrt inputs or the forward pass, we see a higher change in PPL.
Refer to caption: 2604.10390v1/x7.png

##### Data format strongly modulates resilience: FP8 > > BF16 > > FP16.

Figure [8](#S5.F8) shows a pronounced difference in PPL distributional spread across numerical formats. FP16 exhibits the widest distributions and most extreme outliers, especially at high fault rates. BF16 yields substantially tighter distributions under the same fault conditions, and FP8 is the most compact of the three. This ordering aligns with the structural properties of the formats: FP16 uses 1 sign, 5 exponent, and 10 mantissa bits, so a single exponent bit-flip can induce a large-magnitude change. BF16 retains 8 exponent bits but only 7 mantissa bits, matching the dynamic range of FP32 and reducing the worst-case impact of individual bit corruptions. FP8 further narrows the effective dynamic range, which bounds the magnitude of individual corruption events and provides a degree of fault containment at the cost of reduced numerical precision.

### RQ4: How effective is loss-NaN checking in reducing the impact of permanent faults on training dynamics?

Figure: Fig. 9: Effect of the loss NaN check across data formats. Left: outcome distribution by fault rate with NaN check enabled (solid) and disabled (hatched). Right: PPL deviation of non-diverged runs under both conditions. For FP16, the check converts many crashes into Spike-and-Recover events but extreme silent degradation persists. For BF16, the check mitigates both crashes and silent degradation. For FP8, outcomes are binary regardless of the check, with PPL deviations under 1%.
Refer to caption: 2604.10390v1/x8.png

The loss-NaN check is a standard mixed-precision safeguard: when a NaN or Inf is detected in the loss, the optimizer step is skipped and the dynamic loss scale is halved. We compare the training dynamics of all the three data formats under permanent faults and present our results in Figure [9](#S5.F9).
For FP16, the failure spectrum is broad: Crashed/Diverged, Spike-and-recover, and Mild Degradation all appear across fault rates. With the NaN check enabled, Spike-and-recover (blue) becomes the dominant non-benign outcome, whereas without it Crashed/Diverged (red) dominates for rates $\geq 0.05$. This indicates that, for FP16, the check successfully intercepts a substantial fraction of training anomalies that would otherwise be fatal.

A particularly consequential effect is that non-diverged FP16 runs exhibit PPL deviations of up to $\sim$600% above baseline (rightmost panel of Figure [9](#S5.F9)a), representing the largest silent degradation among all formats. The NaN check substantially tightens this distribution, but extreme outliers remain, indicating that some fault-induced corruptions accumulate below the NaN threshold and manifest as degraded model quality rather than as a detectable crash.

BF16 training (Figure [9](#S5.F9)b) exhibits a distinct risk profile that combines unfavorable aspects of both FP8 and FP16. Crash rates at high fault rates approach those of FP8, yet non-diverged runs can still suffer silent PPL degradation of up to $\sim$400%, a failure mode largely absent in FP8. The NaN check mitigates both classes of failure: it reduces Crashed/Diverged outcomes and suppresses the large PPL deviations among surviving runs. This dual effect makes the check particularly important for BF16: disabling it increases outright failures and exposes training to a class of invisible quality loss that neither crashes nor triggers obvious alarms. As in FP16, the check mainly converts otherwise fatal trajectories to Spike-and-recover behavior rather than eliminating underlying faults.

##### Cross-format summary.

FP8 fault outcomes are effectively binary: runs either absorb faults or diverge, so the NaN check primarily prevents crashes. FP16 training is most prone to persistent PPL degradation among surviving runs; the check reduces but does not eliminate this risk. BF16 combines high crash susceptibility with substantial silent degradation, both of which the check mitigates. Across formats and fault rates, fault phase is a stronger predictor of outcome than NaN checking: Forward Output and Backward Gradient Input faults consistently yield the highest crash rates ($\sim$60%) at matched fault rates), whereas Backward Gradient Weight faults are largely self-correcting. This phase dependence appears format-agnostic, suggesting it is driven by gradient-flow structure rather than by the numerical representation.

## VI Limitations

First, our evaluation is limited to relatively small-scale models, which may differ from the massive foundation models deployed in production clusters. We intentionally accept this trade-off: smaller models allow thousands of full training runs across diverse fault signatures, enabling a more rigorous characterization of SDC behavior.

Second, the fidelity of our software-level fault injection depends on the accuracy of the underlying hardware defect models. Our conclusions are valid only when the simulated error signatures and activation rates reflect real silicon degradation; arbitrary bit-flip patterns would yield misleading results. Accurate error-signature characterization is therefore a prerequisite for meaningful use of the methodology.

Third, our experiments target a specific distributed training configuration with sequence parallelism and may not capture fault propagation under alternative parallelism strategies. Different pipeline and tensor-parallel layouts induce different communication and synchronization patterns, causing SDC on one node to propagate, attenuate, or amplify differently. Characterizing how distribution strategy shapes the SDC blast radius is left to future work.

## VII Conclusion

This paper presents the first systematic characterization of large language model training sensitivity to SDC caused by permanent and intermittently manifesting hardware faults.
Our results show that, although SDC-induced perturbations to submodule computations and gradients are often numerically small, they can still steer optimization toward different local optima and yield models with meaningfully different learned weights and downstream behavior. We further demonstrate that SDCs can be highly evasive under conventional monitoring: in many cases, training loss appears well behaved, masking underlying corruption, while in extreme cases, the same class of faults can trigger sharp loss spikes and catastrophic divergence. We find that SDC impact is strongly dependent on the specific unhealthy node and fault site, and is tightly coupled to the local loss landscape of the training task, leading to highly heterogeneous resilience behavior even within a single cluster.
Therefore, this work establishes an empirical foundation for reasoning about detection strategies, resilience targets, and system-level reliability requirements for LLM training at scale.
Beyond the vulnerability assessment of the training loop, this work offers a methodology that future work can leverage towards integrating hardware-accurate fault models and realistic intermittent activation behavior with software-level training analysis to co-design hardware, runtime, and optimization algorithms for fault-aware large-scale AI systems.

## References

- [1]
J. R. Black (1969)
Electromigration—a brief survey and some recent results.
IEEE Transactions on Electron Devices 16 (4), pp. 338–347.
Note: DOI: 10.1109/T-ED.1969.16754
External Links: [Document](https://dx.doi.org/10.1109/T-ED.1969.16754),
[Link](https://doi.org/10.1109/T-ED.1969.16754)
Cited by: [§II-A](#S2.SS1.p1.1).
- [2]
Y. Chen, Z. Tan, A. K. Jaiswal, H. Qu, X. Zhao, Q. Lin, Y. Cheng, A. Kwong, Z. Cao, and T. Chen (2025)
Bit-flip error resilience in llms: a comprehensive analysis and defense framework.
In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,
pp. 10425–10435.
Note: DOI: 10.18653/v1/2025.emnlp-main.52
External Links: [Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.52),
[Link](https://doi.org/10.18653/v1/2025.emnlp-main.52)
Cited by: [§II-E](#S2.SS5.p2.1),
[TABLE I](#S2.T1),
[TABLE I](#S2.T1.14.2),
[TABLE I](#S2.T1.4.4.4.4),
[§V](#S5.SSx1.p1.1).
- [3]
S. Cui, A. Patke, Z. Chen, A. Ranjan, H. Nguyen, P. Cao, B. Bode, G. Bauer, S. Jha, C. Narayanaswami, et al. (2025)
Characterizing modern gpu resilience and impact in hpc systems: a case study of a100 gpus.
In 2025 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W),
pp. 1–6.
External Links: [Link](https://ieeexplore.ieee.org/document/11071550),
[Document](https://dx.doi.org/10.1109/DSN-W65791.2025.00031)
Cited by: [§II-C](#S2.SS3.p1.1).
- [4]
R. Degraeve, G. Groeseneken, R. Bellens, J. Ogier, M. Depas, P. J. Roussel, and H. E. Maes (1998)
New insights in the relation between electron trap generation and the statistical properties of oxide breakdown.
IEEE Transactions on Electron Devices 45 (4), pp. 904–911.
Note: DOI: 10.1109/16.662800
External Links: [Document](https://dx.doi.org/10.1109/16.662800),
[Link](https://doi.org/10.1109/16.662800)
Cited by: [§II-A](#S2.SS1.p1.1).
- [5]
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022)
Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.
Advances in neural information processing systems 35, pp. 30318–30332.
Note: DOI: 10.5555/3600270.3602468
External Links: [Document](https://dx.doi.org/10.5555/3600270.3602468),
[Link](https://doi.org/10.5555/3600270.3602468)
Cited by: [§IV-B](#S4.SS2.p1.1).
- [6]
H. D. Dixit, S. Pendharkar, M. Beadon, C. Mason, T. Chakravarthy, B. Muthiah, and S. Sankar (2021-02)
Silent data corruptions at scale.
arXiv preprint arXiv:2102.11245.
Note: DOI: 10.48550/arXiv.2102.11245
External Links: [Link](https://arxiv.org/abs/2102.11245),
[Document](https://dx.doi.org/10.48550/arXiv.2102.11245)
Cited by: [§I](#S1.p1.1).
- [7]
E. Elsen, C. Hawthorne, and A. Somani (2023-09)
The adventure of the errant hardware.
Note: Adept AI Blog
External Links: [Link](https://www.adept.ai/blog/sherlock-sdc/)
Cited by: [§I](#S1.p3.1).
- [8]
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2023-12)
A framework for few-shot language model evaluation.
Zenodo.
Note: DOI: 10.5281/zenodo.10256836
External Links: [Document](https://dx.doi.org/10.5281/zenodo.10256836),
[Link](https://zenodo.org/records/10256836)
Cited by: [§V](#S5.SSx2.SSS0.Px3.p1.1).
- [9]
Gemini Team, Google (2023-12)
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805.
Note: DOI: 10.48550/arXiv.2312.11805
External Links: [Link](https://arxiv.org/abs/2312.11805),
[Document](https://dx.doi.org/10.48550/arXiv.2312.11805)
Cited by: [§I](#S1.p3.1),
[§II-C](#S2.SS3.p1.1).
- [10]
N. George, S. Gurumurthi, V. Sridharan, H. D. Dixit, E. Goksu, B. Parthasarathy, A. Huffman, T. Macieira, A. Sinha, D. Liberty, L. Minwell, and R. S. Chappell (2026-01)
Silent Data Corruption in Artificial Intelligence: A Growing Challenge for Large-Scale Machine Learning .
IEEE Micro 46 (01), pp. 66–72.
Note: DOI: 10.1109/MM.2025.3645670
External Links: ISSN 1937-4143,
[Document](https://dx.doi.org/10.1109/MM.2025.3645670),
[Link](https://doi.ieeecomputersociety.org/10.1109/MM.2025.3645670)
Cited by: [§I](#S1.p1.1).
- [11]
T. Grasser, B. Kaczer, W. Goes, H. Reisinger, T. Aichinger, P. Hehenberger, P. Wagner, F. Schanovsky, J. Franco, M. T. T. Luque, et al. (2011)
The paradigm shift in understanding the bias temperature instability: from reaction–diffusion to switching oxide traps.
IEEE Transactions on Electron Devices 58 (11), pp. 3652–3666.
Note: DOI: 10.1109/TED.2011.2164543
External Links: [Document](https://dx.doi.org/10.1109/TED.2011.2164543),
[Link](https://doi.org/10.1109/TED.2011.2164543)
Cited by: [§II-A](#S2.SS1.p1.1).
- [12]
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024)
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783.
Note: DOI: 10.48550/arXiv.2407.21783
External Links: [Document](https://dx.doi.org/10.48550/arXiv.2407.21783),
[Link](https://doi.org/10.48550/arXiv.2407.21783)
Cited by: [§I](#S1.p1.1).
- [13]
F. Hill, A. Bordes, S. Chopra, and J. Weston (2015)
The goldilocks principle: reading children’s books with explicit memory representations.
arXiv preprint arXiv:1511.02301.
Note: DOI: 10.48550/arXiv.1511.02301
External Links: [Link](https://arxiv.org/abs/1511.02301),
[Document](https://dx.doi.org/10.48550/arXiv.1511.02301)
Cited by: [Fig. 7](#S5.F7),
[Fig. 7](#S5.F7.3.2),
[§V](#S5.SSx2.SSS0.Px3.p1.1).
- [14]
P. H. Hochschild, P. Turner, J. C. Mogul, R. Govindaraju, P. Ranganathan, D. E. Culler, and A. Vahdat (2021)
Cores that don’t count.
New York, NY, USA.
Note: DOI: 10.1145/3458336.3465297
External Links: ISBN 9781450384384,
[Link](https://doi.org/10.1145/3458336.3465297),
[Document](https://dx.doi.org/10.1145/3458336.3465297)
Cited by: [§I](#S1.p1.1).
- [15]
C. Hu, S. C. Tam, F. Hsu, P. Ko, T. Chan, and K. W. Terrill (1985)
Hot-electron-induced mosfet degradation-model, monitor, and improvement.
IEEE Journal of Solid-State Circuits 20 (1), pp. 295–305.
Note: DOI: 10.1109/T-ED.1985.21952
External Links: [Document](https://dx.doi.org/10.1109/T-ED.1985.21952),
[Link](https://doi.org/10.1109/T-ED.1985.21952)
Cited by: [§II-A](#S2.SS1.p1.1).
- [16]
D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, et al. (2019)
A study of bfloat16 for deep learning training.
arXiv preprint arXiv:1905.12322.
Note: DOI: 10.48550/arXiv.1905.12322
External Links: [Document](https://dx.doi.org/10.48550/arXiv.1905.12322),
[Link](https://doi.org/10.48550/arXiv.1905.12322)
Cited by: [§IV-B](#S4.SS2.p1.1).
- [17]
H. Levesque, E. Davis, and L. Morgenstern (2012)
The winograd schema challenge.
In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning,
Note: DOI: 10.5555/3031843.3031909
External Links: [Link](https://dl.acm.org/doi/10.5555/3031843.3031909),
[Document](https://dx.doi.org/10.5555/3031843.3031909)
Cited by: [Fig. 7](#S5.F7),
[Fig. 7](#S5.F7.3.2),
[§V](#S5.SSx2.SSS0.Px3.p1.1).
- [18]
Llama Team, AI @ Meta (2024-07)
The Llama 3 herd of models.
arXiv preprint arXiv:2407.21783.
Note: DOI: 10.48550/arXiv.2407.21783
External Links: [Link](https://arxiv.org/abs/2407.21783),
[Document](https://dx.doi.org/10.48550/arXiv.2407.21783)
Cited by: [§I](#S1.p3.1),
[§II-C](#S2.SS3.p1.1).
- [19]
J. J. Ma, H. Pei, L. Lausen, and G. Karypis (2025)
Understanding silent data corruption in llm training.
In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pp. 20372–20394.
Note: DOI: 10.18653/v1/2025.acl-long.996
External Links: [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.996),
[Link](https://doi.org/10.18653/v1/2025.acl-long.996)
Cited by: [§II-E](#S2.SS5.p2.1),
[TABLE I](#S2.T1),
[TABLE I](#S2.T1.1.1.1.2),
[TABLE I](#S2.T1.14.2).
- [20]
C. Meister and R. Cotterell (2021)
Language model evaluation beyond perplexity.
In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers),
pp. 5328–5339.
Note: DOI: 10.48550/arXiv.2106.00085
External Links: [Link](https://arxiv.org/abs/2106.00085),
[Document](https://dx.doi.org/10.48550/arXiv.2106.00085)
Cited by: [§V](#S5.SSx2.SSS0.Px3.p1.1).
- [21]
S. Merity, C. Xiong, J. Bradbury, and R. Socher (2016)
Pointer sentinel mixture models.
arXiv preprint arXiv:1609.07843.
Note: DOI: 10.48550/arXiv.1609.07843
External Links: [Document](https://dx.doi.org/10.48550/arXiv.1609.07843),
[Link](https://doi.org/10.48550/arXiv.1609.07843)
Cited by: [§IV-A](#S4.SS1.SSS0.Px1.p1.1),
[§IV-A](#S4.SS1.SSS0.Px2.p1.1).
- [22]
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. (2017)
Mixed precision training.
arXiv preprint arXiv:1710.03740.
Note: DOI: 10.48550/arXiv.1710.03740
External Links: [Link](https://arxiv.org/pdf/1710.03740),
[Document](https://dx.doi.org/10.48550/arXiv.1710.03740)
Cited by: [§V](#S5.SSx1.p2.1).
- [23]
P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, et al. (2022)
Fp8 formats for deep learning.
arXiv preprint arXiv:2209.05433.
Note: DOI: 10.48550/arXiv.2209.05433
External Links: [Document](https://dx.doi.org/10.48550/arXiv.2209.05433),
[Link](https://doi.org/10.48550/arXiv.2209.05433)
Cited by: [§IV-B](#S4.SS2.p1.1).
- [24]
S. Mitra, S. S. Banerjee, M. Dixon, M. Fuller, R. Govindaraju, P. Hochschild, E. X. Liu, B. Parthasarathy, and P. Ranganathan (2025)
Silent data corruption by 10× test escapes threatens reliable computing.
IEEE Design & Test 42 (6), pp. 40–53.
Note: DOI: 10.1109/MDAT.2025.3602741
External Links: [Document](https://dx.doi.org/10.1109/MDAT.2025.3602741),
[Link](https://doi.org/10.1109/MDAT.2025.3602741)
Cited by: [§I](#S1.p2.1).
- [25]
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)
Language models are unsupervised multitask learners.
OpenAI blog 1 (8), pp. 9.
External Links: [Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
Cited by: [§IV-A](#S4.SS1.SSS0.Px1.p1.1).
- [26]
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2019)
Megatron-lm: training multi-billion parameter language models using model parallelism.
arXiv preprint arXiv:1909.08053.
Note: DOI: 10.48550/arXiv.1909.08053
External Links: [Document](https://dx.doi.org/10.48550/arXiv.1909.08053),
[Link](https://doi.org/10.48550/arXiv.1909.08053)
Cited by: [1st item](#S1.I1.i1.p1.1),
[§IV-C](#S4.SS3.SSS0.Px2.p1.1).
- [27]
Y. Sun, Z. Coalson, S. Chen, H. Liu, Z. Zhang, S. Hong, B. Fang, and L. Yang (2025)
Demystifying the resilience of large language model inference: an end-to-end perspective.
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,
pp. 1127–1144.
Note: DOI: 10.1145/3712285.3759803
External Links: [Document](https://dx.doi.org/10.1145/3712285.3759803),
[Link](https://doi.org/10.1145/3712285.3759803)
Cited by: [§I](#S1.p2.1),
[§II-E](#S2.SS5.p2.1),
[TABLE I](#S2.T1),
[TABLE I](#S2.T1.14.2),
[TABLE I](#S2.T1.8.8.8.5).
- [28]
A. Tyagi, R. Jeyapaul, C. Zhou, P. Whatmough, and Y. Zhu (2024)
Characterizing soft-error resiliency in arm’s ethos-u55 embedded machine learning accelerator.
In 2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS),
pp. 96–108.
Note: DOI: 10.1109/ISPASS61541.2024.00017
External Links: [Document](https://dx.doi.org/10.1109/ISPASS61541.2024.00017),
[Link](https://doi.org/10.1109/ISPASS61541.2024.00017)
Cited by: [§I](#S1.p2.1).
- [29]
S. Wang, G. Zhang, J. Wei, Y. Wang, J. Wu, and Q. Luo (2023)
Understanding silent data corruptions in a large production CPU population.
In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP),
Note: Also published as ACM Transactions on Computer Systems (TOCS), 2024; DOI: 10.1145/3600006.3613149
External Links: [Document](https://dx.doi.org/10.1145/3600006.3613149),
[Link](https://doi.org/10.1145/3600006.3613149)
Cited by: [§I](#S1.p1.1).
- [30]
P. Yu, J. Gu, H. Han, D. Shen, B. Wen, and Y. Liu (2025)
Exploring and mitigating failure behavior of large language model training workloads in hpc systems.
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,
pp. 1165–1179.
Note: DOI: 10.1145/3712285.3759893
External Links: [Document](https://dx.doi.org/10.1145/3712285.3759893),
[Link](https://doi.org/10.1145/3712285.3759893)
Cited by: [§I](#S1.p2.1),
[§II-E](#S2.SS5.p2.1),
[TABLE I](#S2.T1),
[TABLE I](#S2.T1.10.10.10.3),
[TABLE I](#S2.T1.14.2),
[§V](#S5.SSx1.p1.1).