Title: Self-Improving Pretraining: using post-trained models to pretrain better models ArXiv: 2601.21343 Authors: Ellen Xiaoqing Tan, Jack Lanchantin, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, Ping Yu, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Xian Li, Olga Golovneva Sections: 70 Estimated tokens: 29.2k ## Contents - Introduction - 1 Self-Improving Pretraining for Safety, Factuality and Quality - 1.1 Method - 1.1.1 The sequence pretraining task: prefix-conditioned suffix generation - 1.1.2 Self-Improving pretraining using post-trained models - Suffix Rewriter - Suffix Judge - Policy Model Training - 1.2 Experiments - 1.2.1 Models and data - 1.2.2 Experimental Setup - Safety Experimental Setup - Factuality Experimental Setup - Quality Experimental Setup - Policy training variants and ablations - Continual pretraining experiments - From-scratch pretraining - Ablations - 1.2.3 Evaluations - 1.2.4 Results - Main results - Pretraining from-scratch results - 1.2.5 Analysis & ablations - Training objective - Suffix & rewrite vs. rollouts - Number of rollouts - Judge choice - Pivots in pairwise comparison judgments - 1.3 Related Work - Safety. - Factuality. - Reasoning and RL. - 1.4 Conclusion - 1.5 Discussion - Isn’t this slower than next token prediction pretraining? - Is making models safe always a good idea? - What else can this framework do? How do you generalize it? - 2 Thinking Mid-training: Reinforcement Learning of Interleaved Reasoning - 2.1 Method - 2.1.1 Mid-training Data Thinking Augmentation - 2.1.2 Thinking Mid-training - Thinking SFT Mid-training - Thinking RL Mid-training - 2.1.3 RL Post-Training - 2.2 Experiments - 2.2.1 Experimental Setup - Mid-train Data and Models. - Post-train Data and Models. - Training Framework. - 2.2.2 Evaluations - 2.2.3 Results - Mid-training Performance - Post-training Performance. - Data Efficiency of RL Mid-training. - 2.2.4 Ablations - 2.3 Related Work - 2.4 Conclusion - References - Appendix A Additional Judge experiments - A.1 Suffix judge comparisons for quality - A.2 Suffix judge comparisons for factuality - A.2.1 Factuality Prompts: with reference - Prompt Variants v1-v5 - A.2.2 Factuality Prompts: without reference - Prompt Variants v1-v4 - Appendix B Synthetic data generation - B.1 Unsafe test set - Appendix C Evaluation results - C.1 Evaluation prompts - C.2 Finegrained evaluation results ## Abstract Abstract Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model’s capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning. ## Introduction Large-scale pretraining on raw text followed by extensive fine-tuning on curated data is by now the classical paradigm to train large language models. This design has a core weakness: properties like safety, factuality, quality, and reasoning are typically layered on after pretraining. Even though the patterns acquired during pretraining largely dictate what the model can ultimately do, no guidance is provided at this early stage concerning these desirable properties that effort should be concentrated on developing them. Current models still exhibit various weaknesses, e.g. exhibiting factuality, safety and reasoning flaws. We hypothesize the current training pipeline is a fundamental cause of this issues, and fixing it could rectify some of these failings. In this work, we introduce self-improving pretraining: using stronger post-trained models to improve earlier stages of the training pipeline. At a high-level, our approach consists of using this existing strong model to inform earlier stage training of the current policy in two ways: by rewriting pretraining data to encourage desirable behaviors, and as a judge to reward desirable behaviors. Thus, in Section 1 we first introduce a reinforcement-learning–based pretraining approach that rewrites pretraining data, and evaluates candidate continuations using a strong judge model to improve safety, factuality, and overall generation quality directly during pretraining. In Section 2, we propose thinking mid-training, an intermediate training stage that rewrites (augments) pretraining data with interleaved reasoning traces and uses supervised learning and reinforcement learning with a judge to optimize the usefulness of these thoughts. Together, these methods show that incorporating post-trained model rewrites and judgments to guide the policy model early in the training process can substantially improve downstream capabilities. Across safety, factuality, quality, and challenging reasoning benchmarks, our approaches demonstrate large improvements over standard training pipelines, suggesting that leveraging stronger models to shape pretraining data and objectives is a promising direction for building more capable and reliable language models. ## 1 Self-Improving Pretraining for Safety, Factuality and Quality > Ensuring safety, factuality and overall quality in the generations of large language models is a critical challenge, especially as these models are increasingly deployed in real-world applications. The prevailing approach to addressing these issues involves collecting expensive, carefully curated datasets and applying multiple stages of fine-tuning and alignment. However, even this complex pipeline cannot guarantee the correction of patterns learned during pretraining. Therefore, addressing these issues during pretraining is crucial, as it shapes a model’s core behaviors and prevents unsafe or hallucinated outputs from becoming deeply embedded. To tackle this issue, we introduce a new pretraining method that streams documents and uses reinforcement learning (RL) to improve the next K generated tokens at each step. A strong, post-trained model judges candidate generations—including model rollouts, the original suffix, and a rewritten suffix—for quality, safety, and factuality. Early in training, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts. This approach builds higher quality, safer, and more factual models from the ground up. In experiments, our method gives 36.2% and 18.5% relative improvements over standard pretraining in terms of factuality and safety, and up to 86.3% win rate improvements in overall generation quality. Figure: Figure 1: Self-Improving pretraining: Our proposed model training streams pretraining documents and improves the next $K$ generated tokens (suffix, given prefix) at each step with RL. A strong previously post-trained model is used to judge generation candidates at each RL step for quality, safety and hallucination, where the candidates are: (i) $N$ rollouts from the current policy; (ii) the original suffix; and (iii) a rewrite of the suffix by the strong post-trained model. The rewrite can improve the pretrain data’s quality or safety; in the latter case as the prefix remains unsafe the model is always learning how to steer away to a safe suffix. At the start of training model rollouts (i) are low quality, so training relies on candidates (ii) and (iii); later in training the judge starts rewarding winning rollouts. Refer to caption: 2601.21343v3/x1.png Standard pretraining works by predicting the next token on large, usually human-written, corpora. Human-written documents vary widely in quality, safety – and to a degree factuality as well. A standard approach is to curate the training data by identifying and removing low quality documents, but issues likely remain (nguyen2025recycling). Typically, final pretrained models can still produce toxic, biased or otherwise unsafe responses. Further, no matter how factual the original training data is, trained models can still hallucinate due to high next token probabilities being seemingly plausible, but not being grounded in reality. In any case, simply removing all low-quality, unsafe or nonfactual data from pretraining contexts will also mean the model does not learn to steer towards quality, safety and factuality given these inputs, for example in dialogue with a human or given such low-quality documents as context at inference time. The standard approach tries to course correct for these issues during post-training, but this cannot guarantee to fix these patterns, which are inherently core behaviors of the pretrained model (itzhak2025planted). In this work we propose a new scheme for pretraining, quite different from the next token prediction paradigm, termed Self-Improving Pretraining. Our overall setup is depicted in [Figure 1](#S1.F1). First, we assume we have access to an existing strong post-trained model, typically trained from a previous iteration of the self-improving cycle. We use this strong model to help pretrain our policy model. Second, during pretraining, we apply and learn from sequence generation rather than next token prediction, so that the model can more accurately learn how to generate sequences, which is the goal during deployment. We argue that only addressing high quality, safe and factual sequence generation at post-training time may already be too late. Our method thus streams the pretraining data and at each step splits it into the most recent $N$ tokens (termed suffix), conditioned on the remaining earlier context (prefix). The existing post-trained model at this point is prompted to rewrite the suffix to steer away from potential unsafe or otherwise low-quality prefixes towards a high quality suffix, which can be used to pretrain our policy model. Third, the existing post-trained model is used as a judge to evaluate the original suffix, the rewrite, and rollouts from the current policy model. This is used to assign rewards and pretrain the policy via reinforcement learning (RL). Early in pretraining, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts. In our experiments, we find strong gains in performance across a broad set of different evaluations compared to standard next token prediction, in both from-scratch and continual pretraining settings. For example in the latter continual pretraining setting, we obtain win rates in generation quality of up to 86.3% over the standard pretraining baseline, and relative improvements of 36.2% and 18.5% in terms of factuality and safety. Similarly in the from-scratch setting we observe absolute gains in generation quality win rate of 31.1% and a 14.4% relative improvement in safety. We provide a detailed analysis and ablation studies of the optimization strategies that contribute to these wins. ### 1.1 Method #### 1.1.1 The sequence pretraining task: prefix-conditioned suffix generation We re-envision pretraining as a sequence learning task, rather than next token prediction. To this end, we segment the stream of pretraining data into chunks of size $N$, where the current chunk $x_{j}$ is termed the suffix, and contiguous chunks in the context are termed the prefix, denoted $x_{1,\dots,j-1}$. The sequence pretraining task is thus to generate a high quality sequence of length $N$ given the prefix: $$ $\bar{x}_{j}\sim\pi(*|x_{1,\dots,j-1}),$ $$ where $\pi$ is our policy model, to be trained. We limit this generation $\bar{x}_{j}$ to be $N$ tokens, and can compare it to the known suffix $x_{j}$ present in the pretraining data for judgment purposes. However, crucially, we should not expect or always desire an exact match with the suffix, and in fact in many cases will not want one, e.g. supposing the suffix is low-quality, unsafe or nonfactual. However, in the case of high-quality suffixes in the pretraining data, they can act directly as references that we would like our policy model to mimic. Our proposed sequence pretraining will thus makes use of an existing teacher model that can differentiate between these cases, as described in the next section. #### 1.1.2 Self-Improving pretraining using post-trained models Our self-improvement framework assumes we already have access to a fully trained (i.e., first pre- and then post-trained) model. This model has effectively absorbed information from across the entire pretrain and post-train datasets already – and this expertise can now be brought to bear on individual examples in the pretraining datasets to train a new model using an effectively superior training signal than the one from which it was trained itself. We consider using this fixed teacher model in two ways: as a rewriter and as a judge. ##### Suffix Rewriter Given a prefix $x_{1,\dots,j-1}$ and a suffix $x_{j}$, the task of the rewriter is to produce a rewrite of the suffix $\hat{x}_{j}$ that is superior to $x_{j}$ for policy training. Policy training would proceeed using the same suffix $x_{1,\dots,j-1}$ but with the rewrite $\hat{x}_{j}$ as the target. There are various ways that the rewrite $\hat{x}_{j}$ can be superior to the suffix $x_{j}$ during training: - • Overall quality: if the suffix is low quality, e.g. comes from a low quality part of the pretraining corpus, the rewriter can improve it, making the training target higher quality. - • Safety: if the prefix and suffix are unsafe, the rewriter can steer the model towards a safe suffix given an unsafe prefix. Note this is quite different to simply rewriting the whole original document, which would mean the model is no longer exposed to unsafe inputs. - • Augmentation: rewriting the data in various ways can improve performance, as has been shown in the offline setting of rewriting entire documents. This has been shown to improve diversity and knowledge (hao2025reformulation; allen2023physics), quality (nguyen2025recycling), and reasoning ability (wang2025thinking; ishibashi2025mining). Our setting allows the model to steer from natural input (prefix) data towards new augmentations (via a rewritten suffix). To build such a rewriter we can either directly prompt an existing post-trained model or fine-tune it further especially for this task. We detail our approach in [subsection 1.2](#S1.SS2). ##### Suffix Judge Given a prefix $x_{1,\dots,j-1}$ and possible completions $\bar{x}_{j}$, the task of our judge is to discern which completion is superior as a target for policy training. There are thus various ways that a judge can provide signal to improve the policy model, including: - • Overall quality: if the suffix, rewrite or certain rollouts are low quality, they will receive low reward. At the start of training, rollouts are likely to be poor and the suffix or rewrite may receive higher reward. After sufficient training, rollouts are more likely to receive high reward. - • Safety: if the prefix and suffix are unsafe, the rewrite or rollouts can steer the model towards a safe suffix given an unsafe prefix. Among the multiple policy rollouts the judge can choose between them to encourage safety amongst model generations. - • Factuality: similarly, after sufficient training, selecting the most factual generations among the rollouts can improve the factuality of the policy model. Similarly to building the rewriter, to build a judge we can either directly prompt an existing post-trained model, or further fine-tune it especially for this task. In our experiments, we consider both settings. We also consider judging each of the above — quality, safety and factuality — by prompting the post-trained judge model for each individually. The prompts we employ are given in [Figure 3](#S1.F3), [Figure 3](#S1.F3), and [Figure 4](#S1.F4). We detail our full approach in [subsection 1.2](#S1.SS2). ##### Policy Model Training Putting it all together, we train our policy model using the sequence pretraining task described in [subsection 1.1](#S1.SS1). We assume we have access to a post-trained model that can act as a suffix judge and a suffix rewriter, as described above. For each prefix, we consider several candidate completions during online training. We can consider (i) the original suffix, (ii) a rewritten suffix; and (iii) $K$ rollouts $\bar{x}_{j}^{k}$, $k=1,\dots,K$ from the current policy $\pi$. The suffix judge is used to provide rewards for online RL by scoring the provided completions. In our experiments we consider both online DPO (qi2024online; lanchantin2025bridging) and reward-filtered negative log-likelihood training (RF-NLL) (christiano2017deep), but other update algorithms are possible. Online DPO has shown performance comparable to GRPO (lanchantin2025bridging). Unlike GRPO, however, DPO is an off-policy algorithm that allows learning from sequences not generated from the current policy, such as the original suffix or rewrites, making it suitable for our approach. For online DPO we take the chosen completion as the highest scoring, and the rejected as the lowest scoring. For RF-NLL we simply take the highest scoring to conduct an NLL update. At the beginning of training, we expect the rollouts from the policy to be low quality. Hence, the original suffix and rewrite are most important at this stage. We thus expect that rewarding rollouts should be introduced after sufficient examples have already been seen. Using a rewriter, however, can improve training starting from the initial updates. In our experiments we consider various ablations of including candidate completions of type (i) original, (ii) rewrite and (iii) rollouts, as well as the number of rollouts $K$. ### 1.2 Experiments #### 1.2.1 Models and data Models. We primarily use the pretrained Llama2 1.4 billion parameter model as a baseline policy model (touvron2023llama), and conduct continual sequence pretraining from that checkpoint. Additionally, we conduct pretraining experiments where we train the same model from scratch by first re-initializing the weights. For the sequence pretraining task, we use chunk size $N=128$. Both suffix judge and rewriter need to have strong instruction-following capabilities, as such we compare two models: (1) fine-tuned Llama3.1-8B-Instruct (dubey2024llama); and (2) prompted GPT-OSS-120B (openai2025gptoss120bgptoss20bmodel). Data. We use the SlimPajama (SP, cerebras2023slimpajama) and RedPajama pretraining datasets (RP, weber2024redpajamaopendatasettraining). SP is a derivative of RP, created by applying more aggressive safety and quality filtering to produce a “slimmer” higher-quality dataset. Thus training only on SP can be considered as a baseline where the training only uses safe and high-quality samples. We use RP for training our method in the safety experiments. To ensure fairness in training, policy, judge, and rewriter models were trained and evaluated on non-overlapping subsets of the data. Judge training. To fine-tune Llama3-8B-Instruct for a judge role, we generate synthetic data from subsets of SP and RP with known rewards (i.e., safe vs. unsafe completions and higher vs. lower quality completions). For the quality task, we create the data by asking a Llama3.3-70B-Instruct (dubey2024llama) model to spoil the original suffix (see Appendix [Figure 13](#A1.F13)) extracted from SP. A pair of original and corrupted suffixes is then used to create two samples, by wrapping the pair or its flipped version in the quality judge prompt, given in [Figure 3](#S1.F3). For the safety task, we use the same model to filter safe and unsafe suffixes from RP by prompting it (see [Figure 3](#S1.F3)) to evaluate the safety of the suffix with 8 random seeds, and only using samples where all 8 judgments are safe or unsafe. We then use the same prompt to wrap suffixes for training. We generate 75,432 training and 4,096 validation samples for the quality task, and 3,192 and 512 for the safety task respectively. Rewriter training. We similarly fine-tune a rewriter from the Llama3-8B-Instruct model (for safety experiments only). We found this was necessary otherwise Llama would refuse to rewrite unsafe prompts. To generate synthetic data we follow the same filtering procedure as for judge safety task training. 73,080 safe and unsafe suffixes then used with the rewriter prompt template, provided in Appendix [Figure 14](#A1.F14). Policy training. Training on SP is conducted on 983,520 samples. Training on the RP dataset is conducted on 257,154 samples, that were filtered to include particularly unsafe content. Specifically, we applied tag-based filtering to extract unsafe documents. Validation and test data were further filtered to retain unsafe data by GPT-OSS-120B to ensure the prefix contains unsafe content. Filtering details are provided in Appendix [B.1](#A2.SS1). Figure: Figure 2: Suffix judge safety prompt. Figure: Figure 4: Factuality judge prompt. #### 1.2.2 Experimental Setup ##### Safety Experimental Setup Our pipeline involves the following three models: a judge, rewriter, and the policy model. Below we will summarize the setup for the components. Suffix judge. Recent studies provide strong evidence that LLM judges become more robust and effective when they generate their own Chain-of-Thought (CoT) analyses before producing final judgments (zhang2024generative; chen2025rm; whitehouse2025j1). To fine-tune Llama3.1-8B-Instruct to be a safety and quality judge incorporating reasoning, we use GRPO (shao2024deepseekmath) as our optimization algorithm. Unlike SFT, GRPO does not require generating high-quality synthetic CoT data, but fully relies on a signal from the final judgment, while incentivizing reasoning traces that result in correct judgments. To reward the judge model during training we rely on labels from synthetically generated data of judgments, rewarding a correctly categorized suffix with 1.0, and 0.0 for mismatching the label. We run GRPO training on the synthetically generated data, where the judge is simultaneously trained on two tasks: quality and safety. We set the global batch size to 256, with 16 generations per prompt under temperature $T=0.6$ and $top_p=0.6$. We train on 64 GPUs for 500 steps with $2.0e-07$ constant learning rate. The maximum prompt length is set to 3584 tokens, and the model can generate up to 512 new tokens. Figure: Figure 5: Suffix judge validation rewards on safety and quality tasks. Initial performance of the model is close to random chance on either task, achieving scores above $90\%$ by the end of training. Refer to caption: 2601.21343v3/figures/judge_safety_valid.png During training we observe that initially the safety task is easier to learn than quality, as the model plateaus at 0.94 average reward score at approximately 100 steps, while the reward for quality keeps growing until the end of the training, see [Figure 5](#S1.F5). Manually analyzing judgments, we found that the initial model tends to favor suffixes that feel more complete, rather than those that are more coherent with respect to the context. Training helps to fix this problem. At inference time to make a judgment we query the model twice, for safety using [Figure 3](#S1.F3), and for quality using [Figure 3](#S1.F3), and combine the results. For the safety judgment, this is a pointwise score, but for quality this is a pairwise judgment given two candidate responses, which outputs which is better of the two. For the latter during policy training we run all pairwise comparisons amongst candidates in the batch, assigning reward 0 or 1 in each case, and take the average of their rewards to obtain pointwise scores. For each rollout, our judge is prompted to evaluate safety and quality 5 times each with temperature $T=1.0$ and $top_p=0.6$. Suffix rewriter. Similarly, we train Llama3.1-8B-Instruct model with the GRPO algorithm. Our goal is to build a suffix rewriter that leaves safe high quality suffixes unchanged (hence the generative output would typically copy the suffix that is given in the input context), whereas for unsafe suffixes, they should be rewritten to be safe. Hence, to train the rewriter, the reward is assigned with the following method: - • If the model was prompted to rewrite a safe suffix, we return reward $1.0$ if the rewritten suffix $\bar{x}_{j}$ is an exact match of the given suffix $x_{j}$, otherwise we reward it with 0.0: $R_{\text{safe}}=\begin{cases}1.0&\text{ if }\bar{x}_{j}=x_{j}\text{\;,}\\ 0.0&\text{ otherwise\;.}\end{cases}$ (1) - • If the model was prompted to rewrite an unsafe suffix, the rewritten suffix is evaluated with the suffix judge based on quality $J_{\text{qual}}$ and safety $J_{\text{safe}}$, averaging judgments across $5$ random seeds: $R_{\text{unsafe}}=\frac{1}{2}\left(J_{\text{qual}}(\bar{x}_{j},x_{j}|x_{1,\dots,j-1})+J_{\text{safe}}(\bar{x}_{j})\right)\text{ \;.}$ (2) To train the suffix rewriter model we use same setup as for the suffix judge. We modify the maximum prompt length to 3968 tokens, and the model generation length to 128 new tokens to match our suffix length. We validate model performance on safe and unsafe subsets. We observe steady improvement on the copy task (exact match reward score on safe suffixes, [Figure 6](#S1.F6)), and use the final checkpoint that achieves token overlap percent plateaued at $98\%$, as shown in [Figure 7](#S1.F7). Figure: Figure 6: Suffix rewriter validation rewards on safe and unsafe suffixes of the RedPajama dataset. Initial performance of the model is close to random chance on the safety task (0.5 score on unsafe suffixes), and near zero on copying safe suffixes (exact match reward score of 0.1), but still increasing after 500 steps. Refer to caption: 2601.21343v3/figures/rewriter_safe.png Figure: Figure 7: Token Overlap in Suffix Rewriter Validation on RedPajama Dataset. We evaluate token overlap between original and rewritten suffixes for both safe and unsafe suffixes in the RedPajama dataset. Our objective is to produce safe rewrites that remain similar to the original suffix. Token overlap serves as a measure of this similarity. For safe suffixes, token overlap increases and approaches 1.0 as we optimize for exact matches. In contrast, token overlap for unsafe suffixes averages around 0.63 and remains close to its initial value, indicating less change (should not overlap). Refer to caption: 2601.21343v3/figures/rewriter_tok_safe.png ##### Factuality Experimental Setup Suffix judge. In the factuality training setting, we only consider using a judge, and not a rewriter. For the factuality judge, this is a pointwise judgment given one candidate response, and a reference answer. We use the original suffix from the training data as the reference. We use GPT-OSS-120B with the prompt given in [Figure 4](#S1.F4). In [subsection A.2](#A1.SS2), we conduct a detailed study using different strong post-trained models as the judge, and various prompt designs, comparing their performance. The suffix judge outputs whether the continuation has no hallucination (reward 1), possible hallucination (reward 0.5) or definite hallucination (reward 0). As in the safety experiments, we combine this reward with an overall quality score of the generation, by adding the quality scoring judge rewards. This is done in the same way as in [subsubsection 1.2.2](#S1.SS2.SSS2.Px1). GPT-OSS-120B is prompted with temperature $T=1.0$ and $top_p=1.0$. We also consider another variant of using a single pivot candidate for pairwise comparisons instead, resulting in $K$ judgments for each update, rather than $\binom{K}{2}$. ##### Quality Experimental Setup Suffix judge. In the quality training setting, we also only consider using a judge, and not a rewriter. For the quality judge, this is a pairwise judgment given two candidate responses, which outputs which is better. For this we use GPT-OSS-120B with the prompt given in [Figure 3](#S1.F3). We run all pairwise comparisons amongst candidates, assigning reward 0 or 1 in each case, and take the mean of their rewards to obtain pointwise scores. We also consider two other variants: (1) using the trained model from [subsubsection 1.2.2](#S1.SS2.SSS2.Px1) but only prompted for quality; and (2) using a single pivot candidate for pairwise comparisons instead, resulting in $K$ judgments for each update, rather than $\binom{K}{2}$. ##### Policy training variants and ablations We conduct a series of variants and ablations of policy training primarily in the safety pretraining setting. First, we conduct both from scratch and continued pretraining in this setting. ##### Continual pretraining experiments Self-Improving Pretraining models are trained with online DPO (unless said otherwise in ablations) with the global batch size 256, sampling 16 rollouts per prompt using temperature $T=1.0$ and $top_p=1.0$. We train on 64 GPUs for 2000 steps with cosine learning rate $lr=5.0e-06$, min ratio $0.1$, and $100$ warmup steps. The maximum sequence length is set to 2048 tokens, and the model generates $N=128$ new tokens for each rollout. For the safety task, the fine-tuned Llama3.1-8B-Instruct judge is used to select DPO pairs from 16 rollouts and the original suffix, while GPT-OSS-120B is used to judge 16 rollouts for the quality and factuality tasks. ##### From-scratch pretraining To pretrain from scratch, we use a similar setup, but increase the number of training steps to 21,000, increase the learning rate to $5.0e-04$, and the number of warmup steps to $2000$. In these experiments, we only use 1 rollout for training. ##### Ablations We also ablate various ways of doing the training with different loss functions and candidate generation pools during online training, all compared to next token prediction baselines. In particular, firstly we compare to: SFT on either (i) rewrites or (ii) (single) rollouts; which do not require a judge during training. For RL training, we use online DPO, which has shown performance comparable to GRPO (lanchantin2025bridging). As mentioned before, DPO is an off-policy algorithm that allows learning from sequences not generated from the current policy, such as the original suffix or rewrites, making it suitable for our approach. First, a baseline simple option is to use the rewrite as the chosen and the current rollout as the rejected in online DPO, which also does not require a judge, inspired by the approach in chen2024self. For our full Self-Improving Pretraining method using a judge, we compare online DPO with reward filtered (RF)-NLL. For RF-NLL we consider two flavors: rollout vs rewrite as candidates to be judged, or rollout vs. original suffix vs rewrite. For online DPO, we consider: (i) suffix vs 1 rollout, (ii) rewrite vs. 1 rollout, (iii) suffix vs. 16 rollouts; and (iv) 16 rollouts only. We also conduct a separate study of the effect of scaling the number of rollouts. For policy model generations during training we use a temperature of 1. For quality and factuality ablations, we study the effects of (i) a single rollout which does not require a judge during training, (ii) 2, 4, 8, 16 rollouts, (iii) suffix as pivot for 8 rollouts. We also compare using the trained judge from [subsubsection 1.2.2](#S1.SS2.SSS2.Px1), with GPT-OSS-120B as an online judge in the quality pretraining setting. #### 1.2.3 Evaluations We evaluate our models on a broad set of benchmarks, including standard evaluations and additional benchmarks focused on coherence, safety and factuality. For generation tasks, we use GPT-OSS-120B as a judge and judgments across 8 random seeds. For the policy model we use greedy generations. Generation quality. To evaluate the generation quality we use 1k samples from the test split of SP as data with safe prefixes, and 1k samples from the test split of filtered RP as data with unsafe prefixes. Generation quality is evaluated by comparing a sequence of length $N$ against baseline generations of Llama Base of the same length. We use GPT-OSS-120B as a suffix judge using the prompt given in [Figure 3](#S1.F3). We average judgments across 8 random seeds using a temperature of 0.7. In addition, we measure coherence, particularly in terms of repetition, independently using the prompt given in [Figure 16](#A3.F16). Note that the generation quality score (win rate) is hence 50.0 for Llama Base given it is used as the baseline in the pairwise comparison. Standard Evaluations. We use a set of standard evaluation tasks to measure the pretrained policy model’s general reasoning abilities. In particular, we average performance across the following datasets: BoolQ (clark2019boolq), PIQA (bisk2020piqa), SIQA (sap2019socialiqa), HellaSwag (zellers2019hellaswag), ARC easy and challenge (clark2018think), OpenBookQA (mihaylov2018can), and 5-shot performance on the aggregated MMLU benchmark (hendrycks2020measuring). Safety. The policy model’s safety is evaluated as a weighted average across five datasets: the RP test split, RealToxicityPrompts (gehman2020realtoxicityprompts), ToxiGen (hartvigsen2022toxigen), and the XStest safe and unsafe sets (rottger2024xstest). In each case, safety is evaluated with GPT-OSS-120B as a judge using the prompt given in [Figure 3](#S1.F3). We use majority vote over N predictions with a temperature of 1. Factuality. The policy model’s factuality is evaluated as a weighted average across five datasets: the RP test split, FActScore (min2023factscore), HaluEval (li2023halueval), which are generation tasks, and the TruthfulQA multiple-choice tasks MC1 and MC2 (lin2022truthfulqa). We evaluate on the QA, dialogue, summarization tasks in HaluEval with the provided ground-truth answers as reference. For FActScore, the provided wikipedia text is used as ground-truth reference for the GPT judge. For the RP test split, FActScore, and HaluEval, the evaluation is done with the corresponding judge prompts given in [Figure 4](#S1.F4), [Figure 17](#A3.F17), [Figure 18](#A3.F18), respectively. We again use GPT-OSS-120B as a judge, using a temperature of 0.7. **Table 1: Main results: continued pretraining results for overall quality, factuality and safety training, compared to standard next token prediction (Llama Base 1.4B and Pretrain Baseline).** | Pretraining for Quality | Generation Quality
Std. Prefix
Unsafe Prefix | Standard Evals (Avg) | Coherence Eval | | | --- | --- | --- | --- | --- | | Llama Base | 50.0 | | 47.6 | 50.1 | | Trained on SlimPajama | | | | | | Llama Pretrain Baseline | 49.0 | | 46.8 | 49.4 | | Self-Improving Pretraining | 86.3 | | 50.8 | 87.9 | | | | | | | | Pretraining for Factuality | Generation Quality
Std. Prefix
Unsafe Prefix | Standard Evals (Avg) | Factuality Evals (Avg) | | | Llama Base | 50.0 | | 47.6 | 42.3 | | Trained on SlimPajama | | | | | | Llama Pretrain Baseline | 49.0 | | 46.8 | 44.0 | | Self-Improving Pretraining | 84.0 | | 50.5 | 57.6 | | | | | | | | Pretraining for Safety | Generation Quality
Std. Prefix
Unsafe Prefix | Standard Evals (Avg) | Safety Evals (Avg) | | | Llama Base | 50.0 | 50.0 | 47.6 | 76.9 | | Trained on SlimPajama | | | | | | Llama Pretrain Baseline | 49.0 | 44.9 | 46.8 | 77.0 | | Trained on RedPajama | | | | | | Llama Pretrain Baseline | 54.5 | 52.6 | 47.9 | 75.5 | | Self-Improving Pretraining | 73.6 | 77.7 | 49.1 | 91.1 | #### 1.2.4 Results ##### Main results [Table 1](#S1.T1) summarizes our main results in the continued pretraining setting when optimizing for quality, factuality and safety. We find that all three objectives significantly improve over the initial and continually pretrained baselines in several metrics. Self-Improving Pretraining provides superior generation quality over standard (SlimPajama test set) prefixes, and higher scores on standard pretraining evaluations in all three cases. A breakdown of the standard evaluations can be found in [Table 3](#S1.T3). When optimizing for quality, we see the largest gains in generation quality on standard prefixes, with a win rate of 86.3% over the baseline generations, and a 87.9% win rate in terms of coherence. When optimizing for factuality, we also see significant gains in quality (84.0% win rate), and more importantly, an improvement in factuality evaluations from 42.3 to 57.6. The breakdown in to individual factuality tasks can be found in [Table 4](#S1.T4), where we observe wins in every individual benchmark tested. When optimizing for safety, we also see significant gains in quality for unsafe prefixes (77.7% win rate), as well as significant improvements in safety evaluations with an average increase from 76.9 to 91.1. The breakdown into individual safety tasks is given in [Table 5](#S1.T5). Again, we observe wins in most individual benchmarks tested. We show further detailed results in [Table 24](#A3.T24) which highlight that, for example, optimizing for safety does not optimize for factuality, or vice-versa. This indicates that if you want to optimize for both, both must be factored into the rewards provided during training. We also report the performance of the larger Llama-3.1 8B Base model, to show that our results are not a distillation effect owing to our reward model’s size. Optimizing for safety and quality with Self-Improving Pretraining outperforms the 8B model with a 1.4B model. Figure: Figure 8: Rollout chosen rate on the training data during from-scratch pretraining (left) and continued pretraining (right). Initially RL reward for rollouts is low, and suffix or rewrite completions are chosen for training more often. As the model improves, RL rewards high-quality rollouts, resulting in higher rollout chosen rates. Refer to caption: 2601.21343v3/x2.png ##### Pretraining from-scratch results The previous results are from continued pretraining from the initial Llama baseline model. Potentially, our Self-Improving Pretraining could provide much larger improvements if used earlier in pretraining, for example by making the model learn safety measures earlier on in training. We compare 4 training setups in the safety pretraining setting: - • Pretrain Baseline (model trained on RedPajama suffixes); - • Pretrain on Rewrites; - • Self-Improving Pretraining: RF-NLL (suffix vs. rewrite); - • Self-Improving Pretraining: RF-NLL (rollout vs. rewrite). In these experiments, we only use 1 rollout for training. [Table 2](#S1.T2) summarizes quality and safety evaluation results. NLL pretraining on rewritten suffixes outperforms baseline training on safety evaluations, but does not improve on overall quality. Using the fine-tuned Llama3.1-8B-Instruct suffix judge promotes generations that are better in both quality and in safety, resulting in improved performance for our models. Self-Improving Pretraining using RF-NLL (rollout vs. rewrite) has a generation quality win rate of 32.4, compared to the next-token prediction baseline win rate of only 1.3 – a huge improvement. Simultaneously, safety evaluations improve from 85.2 to 97.5. **Table 2: Pretraining (from scratch) results: Comparison of overall quality and safety outcomes for 1.4B models trained on RedPajama from scratch (21k steps), versus next token prediction approaches (Pretrain Baseline and Pretrain on Rewrites).** | Pretraining for Safety (from scratch) | Generation Quality
Std. Prefix
Unsafe Prefix | Safety Evals (Avg) | | | --- | --- | --- | --- | | Pretrain Baseline | 1.3 | 2.4 | 85.2 | | Pretrain on Rewrites | 1.6 | 2.4 | 96.7 | | Self-Improving Pretraining: RF-NLL (suffix vs. rewrite) | 5.3 | 25.8 | 96.4 | | Self-Improving Pretraining: RF-NLL (rollout vs. rewrite) | 32.4 | 12.1 | 97.5 | **Table 3: Standard pretraining evaluation tasks: continued pretraining results compared to standard next token prediction on standard evaluation tasks. All continually trained models use SlimPajama except in the safety setting which uses RedPajama.** | | BoolQA | PIQA | Hellaswag | ARC-e | ARC-c | OBQA | SIQA | MMLU | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Llama Base | 64.6 | 74.8 | 47.9 | 66.6 | 32.3 | 27.2 | 41.0 | 26.4 | | Trained on SlimPajama | | | | | | | | | | Llama Pretrain Baseline | 59.6 | 74.2 | 47.7 | 65.3 | 31.3 | 27.0 | 42.2 | 26.7 | | Pretraining for Quality | | | | | | | | | | Self-Improving Pretraining | 69.1 | 75.8 | 51.7 | 69.4 | 35.7 | 30.0 | 46.1 | 28.3 | | Pretraining for Factuality | | | | | | | | | | Self-Improving Pretraining | 70.3 | 75.1 | 51.1 | 69.1 | 35.1 | 29.0 | 46.8 | 27.9 | | Pretraining for Safety | | | | | | | | | | Trained on RedPajama | | | | | | | | | | Llama Pretrain Baseline | 64.0 | 74.3 | 49.2 | 66.9 | 32.8 | 26.6 | 41.5 | 27.5 | | Self-Improving Pretraining | 65.7 | 75.6 | 49.6 | 69.0 | 34.8 | 27.4 | 44.1 | 26.7 | **Table 4: Factuality tasks: continued pretraining results compared to standard next token prediction on factuality tasks.** | Pretraining for Factuality | SlimPajama | FActScore | HaluEval | HaluEval | HaluEval | Truthful QA | TruthfulQA | | --- | --- | --- | --- | --- | --- | --- | --- | | | (pointwise) | (pairwise) | dialogue | QA | summarization | MC1 | MC2 | | Llama Base | 36.6 | 50.0 | 50.0 | 50.1 | 50.0 | 22.4 | 35.9 | | Trained on SlimPajama | | | | | | | | | Llama Pretrain Baseline | 35.4 | 48.9 | 50.8 | 51.4 | 61.5 | 21.5 | 35.5 | | Self-Improving Pretraining | 63.5 | 69.3 | 54.6 | 58.5 | 84.7 | 27.7 | 42.5 | **Table 5: Safety tasks: continued pretraining results compared to standard next token prediction on safety tasks.** | Pretraining for Safety | RealToxicityPrompts | RedPajama test | XStest safe | XStest unsafe | Toxigen | | --- | --- | --- | --- | --- | --- | | Llama Base | 88.1 | 68.0 | 85.2 | 39.5 | 80.1 | | Trained on RedPajama | | | | | | | Llama Pretrain Baseline | 87.1 | 67.4 | 87.6 | 35.0 | 82.0 | | Self-Improving Pretraining | 96.0 | 93.4 | 88.4 | 49.0 | 93.1 | #### 1.2.5 Analysis & ablations ##### Training objective [Table 6](#S1.T6) provides ablation results on variants of the Self-Improving Pretraining training objective in the safety optimization case. First, we find that continued pretraining using standard next token prediction on RedPajama lowers the performance compared to the initial baseline on safety evaluations slightly (from 76.9 to 75.5), while standard evaluations are similar or slightly improved (47.6 vs. 47.9). As RedPajama contains unsafe contexts this is not unexpected. Continued pretraining on the cleaner SlimPajama keeps the safety evaluations more or less unchanged (76.9 vs. 77.0), although standard evaluations drop. Next, training with SFT on rewrites or a single rollout without a judge gives little improvement in quality for the former (52.7 of safe and 50.6 on unsafe prefixes), and large deterioration for the latter (dropping to 2.0 and 0.2 on safe and unsafe prefixes), which is expected (i.e., model collapse). Upon inspection of the model generations, we found that the model trained on a single rollout collapsed to generating meaningless - but safe - sequences of words or symbols. In contrast online DPO with the rewrite as chosen and current rollout as rejected gives slightly improved standard and safety evaluations (48.8 and 77.7 respectively). Overall, however, with our full Self-Improving Pretraining method using a post-trained suffix judge, we find much larger gains – particularly in the online DPO case, and for larger numbers of rollouts. We find applying RF-NLL improves safety evaluations over the baseline (85.0 vs. 76.9) but is only on par with the improvement found using SFT on rewrites, which does not use a judge, while both do not give significant gains in generation quality. For online DPO however, we see major boosts in generation quality. Online DPO using rewrites and a single rollout improves generation quality from 50.0 to 60.0 on standard prefixes, and from 50.0 to 87.2 on unsafe prefixes. Increasing to 16 rollouts gives even larger gains on standard prefixes (from 50.0 to 73.6), and on overall safety evaluations (from 76.9 to 91.1). **Table 6: Training objective ablations: detailed ablations of Self-Improving Pretraining in the safety training setting, training on RedPajama.** | Method / Ablation | Generation Quality
Std. Prefix
Unsafe Prefix | Standard Evals (Avg) | Safety Evals (Avg) | | | --- | --- | --- | --- | --- | | Llama Base | 50.0 | 50.0 | 47.6 | 76.9 | | Llama Pretrain Baseline | 54.5 | 52.6 | 47.9 | 75.5 | | Training without a Judge | | | | | | SFT (rewrite) | 52.7 | 50.6 | 48.4 | 86.5 | | SFT (1 rollout) | 2.0 | 0.2 | 29.5 | 99.5 | | Online DPO (chosen: rewrite, reject:rollout) | 53.6 | 83.1 | 48.8 | 77.7 | | Self-Improving Pretraining | | | | | | RF-NLL (rollout vs. rewrite) | 49.0 | 51.8 | 48.3 | 85.0 | | RF-NLL (suffix vs. rewrite vs. 1 rollout) | 50.1 | 51.1 | 48.8 | 84.6 | | Online DPO (suffix vs. 1 rollout) | 55.7 | 84.7 | 48.4 | 82.5 | | Online DPO (rewrite vs. 1 rollout) | 60.2 | 87.2 | 48.5 | 81.9 | | Online DPO (suffix vs 16 rollouts) | 73.6 | 77.7 | 49.1 | 91.1 | | Online DPO (suffix vs rewrite vs 16 rollouts) | 72.5 | 75.4 | 49.1 | 88.9 | | Online DPO (suffix as a pivot for 16 rollouts) | 59.6 | 51.9 | 48.8 | 89.0 | | Online DPO (16 rollouts) | 71.1 | 72.0 | 49.7 | 88.9 | ##### Suffix & rewrite vs. rollouts In both the continual and from-scratch pretraining settings, we find that early in training the model relies on the original and rewritten suffixes more often for supervision. As the model improves the judge picks rollouts more and more frequently, see [Figure 8](#S1.F8). Later in training RL rewards high-quality rollouts, resulting in a higher rollout chosen rate. ##### Number of rollouts We report ablation results on the number of rollouts used in online DPO for quality, factuality, and safety training in [Figure 9](#S1.F9). We generally find improved performance across all benchmarks with an increasing number of rollouts, where we experimented with between 1 and 16 rollouts. We did not experiment past 16 rollouts due to the increased compute required, but we expect further gains. Furthermore, similar trends can be seen in generation quality and standard evaluations, as shown in Appendix [Table 17](#A3.T17), where more rollouts lead to better final performance across all benchmarks tested. Detailed standard task results are also given in Appendix [Table 18](#A3.T18) and [Table 19](#A3.T19). ##### Judge choice As mentioned in [subsubsection 1.2.1](#S1.SS2.SSS1), we experiment with two types of judges: one fine-tuned specifically for a target task such as quality, and another used directly via prompting without training. In [Table 7](#S1.T7) we compare these two judges when they are used for quality training. We find that the prompted GPT-OSS-120B model generally performs better, but the finetuned Llama judge is not far behind, demonstrating that we can purpose-train a smaller model for this goal. A detailed breakdown of results across standard tasks can be found in Appendix [Table 20](#A3.T20). Figure: Figure 9: Ablation results on the number of rollouts in online DPO training for models trained for Quality (left), Factuality (middle), and Safety (right). Refer to caption: 2601.21343v3/x4.png **Table 7: Judge comparison: evaluation results on generation quality and coherence for ablations of using GPT-OSS-120B as judge versus using our finetuned llama3 judge during online DPO training. The number of rollouts used is 8 in these experiments.** | Pretraining for Quality | Generation Quality | Standard Evals (avg) | Coherence Eval | | --- | --- | --- | --- | | Self-Improving Pretraining (finetuned Llama3 as judge) | 72.1 | 49.6 | 72.7 | | Self-Improving Pretraining (GPT-OSS-120B as judge) | 84.3 | 51.1 | 86.8 | ##### Pivots in pairwise comparison judgments We also experiment with speeding up pairwise quality judgments by instead using a pivot. That is, one generation is selected and then all generations in the training batch are compared only against this pivot generation to produce rewards. Results are given in Appendix [Table 21](#A3.T21), [Table 22](#A3.T22) and [Table 23](#A3.T23) for various settings. Overall we find deterioration in performance from using pivots, leaving how to make judgments faster while maintaining quality an open question. ### 1.3 Related Work Pretraining of neural language models stretches back to the work of bengio2003neural, and language modeling itself stretches back to at least shannon1948mathematical. Subsequent work then built both masked language modeling (collobert2011natural; peters2018deep; devlin2019bert) and next token prediction systems (dai2015semi; raffel2020exploring; radford2018improving). The latter has now become the dominant paradigm due to the ability to extend to generating full sequences autoregressively. Despite rapid progress, particularly by scaling (brown2020language; achiam2023gpt), there remain unanswered questions in key areas of generalization, for example safety, factuality and reasoning. ##### Safety. Training on all available pretraining data will inevitably include unsafe human written data, from toxicity through to bias and harms. Simply filtering the pretraining data of unsafe content can impoverish the model, and will make it unable to handle unsafe inputs (xu2020recipes). As with other issues, one approach is to attempt to fix these problems in post-training (dinan2019build; xu2021bot; bai2022constitutional). However, due to poor generalization issues still remain typically when considering out-of-distribution inputs, as is shown by jailbreak attacks (zou2023universal). It should also be noted that fine-grained control of safety is likely a better choice than simply removing capabilities (yi2025goodbadfailurellms). korbak2023pretraining is an early work incorporating safety into pretraining, which reported success with control tokens which incorporate human preferences. More recently, min2023factscore also use a combination of rewriting and special tokens, and report encouraging results. shilov2025beyond proposes a different approach, whereby they alter the training scheme altogether. They split the model’s weights into retain and forget subsets, and guide specific knowledge into the forget subset during training. ##### Factuality. A number of works have tried to address factuality at post-training time with various approaches. tian2023finetuning; lin2024flame; zhang2024self mostly focused on supervised fine-tuning (SFT) and offline RL approaches such as DPO (rafailov2023direct). chen2025learning and chen2025train built specific rewards using retrieval tools to provide measures of factuality for RL training. ##### Reasoning and RL. Standard pretraining already gives reasoning capabilities, including chain-of-thought emergence (kojima2022large). These traits are further amplified via post-training, particularly through reinforcement learning on verifiable rewards (RLVR) (deepseekai2025deepseekr1incentivizingreasoningcapability). The success of improving reasoning at post-training time has encouraged researchers to try to move post-training techniques further upstream to either mid-training or pretraining. Recent works have augmented pretraining with thinking tokens (wang2025thinking; fujii2025rewriting), and incorporated RL for optimizing thoughts for the next token (dong2025reinforcement; hatamizadeh2025rlp) or the next set of tokens (yu2024codepmp; li2025reinforcement; team2025kimi). ### 1.4 Conclusion Our work re-envisions pretraining by using a strong post-trained model to provide superior supervision signals. This works in two ways: (i) by providing rewrites on the original streaming pretrain data; and (ii) by acting as a judge. We showed that such a self-improving setup can improve the factuality, safety and overall generation quality of pretrained models. ### 1.5 Discussion Here we discuss some common questions about our approach. ##### Isn’t this slower than next token prediction pretraining? Self-Improving Pretraining is indeed slower than standard next token prediction, especially when using rollouts. However, using rewrites and suffixes only, which can work at the start of pretraining, might not be that much slower. Nevertheless, our thinking follows that of chung2023dontteach: training methods should be designed to exploit future increases in compute, favoring incentive-based objectives over explicit skill instruction. Hence, using strong post-trained models as judges may prove to be a winner in the long run, especially as pretraining hits a “data wall” where increased compute with next token prediction does not offer gains, in the case that we have “run out of data”. ##### Is making models safe always a good idea? We showed how our approach can make models safer, but indeed there may be cases where safe generations are not the goal. An example is generating a movie script with dialogue from bad actors, which would necessitate the ability to generate unsafe text. During training, one way to get around this is the use of control tokens, or some other method of fine-grained control of safety, i.e. to train for both safe and unsafe cases, given the control token which can be switched on/off at inference time. We believe this might actually be a better choice than simply removing capabilities (yi2025goodbadfailurellms). As mentioned earlier, korbak2023pretraining is an early work incorporating safety into pretraining, which reported success with control tokens which incorporate human preferences. ##### What else can this framework do? How do you generalize it? We showed that safety, factuality and general quality can be optimized in our framework, e.g. simply by providing different LLM-as-judge prompts. An obvious approach to combine all three methods at the same time is to sum the rewards from the prompts, or potentially combine them into a single prompt. We already showed that combining quality and safety or quality and factuality works, so we believe this should not be difficult. Ideally we would prefer a more generic judge prompt that can capture all these skills well at the same time. Going further, there are other aspects of a powerful model one may wish for pretraining to also capture, i.e. other skills! – an obvious one being stronger reasoning ability. Training chain-of-thought can also fit fairly well into our framework, i.e. switching between rewrites from a strong post-trained model earlier in pretraining (in this case, to rewrite the original suffix to contain chain-of-thought), and then switching to improving rollouts later in training. See [subsection 1.3](#S1.SS3) for existing related work in the area of chain-of-thought augmentation and reinforcement learning. We will address this topic in the next part of the paper. ## 2 Thinking Mid-training: Reinforcement Learning of Interleaved Reasoning > Large language models are typically trained in two stages: pretraining on raw text followed by post-training for instruction-following and reasoning. This creates a fundamental gap where reasoning capabilities must be acquired almost entirely during post-training, as pretraining data lacks explicit reasoning traces. We introduce thinking mid-training, an intermediate training phase that bridges this gap by teaching models to reason on augmented pretraining corpora. Our approach consists of three components: (1) a data augmentation strategy that uses a teacher model to enrich pretraining text with interleaved “thoughts”, or intermediate reasoning steps inserted at semantically appropriate positions, (2) supervised fine-tuning on the augmented corpus to teach model how to interleave thoughts, and (3) reinforcement learning with an LLM judge to optimize the utility of thoughts for predicting subsequent text. Experiments on Llama-3-8B demonstrate that thinking mid-training substantially improves post-training effectiveness: our full pipeline achieves an average accuracy of 0.38 across challenging reasoning benchmarks (GSM8K, MATH-500, AMC23, Olympiad, GPQA-Diamond), compared to 0.12 for direct RL post-training on the base model, a 3.2$\times$ improvement, and more than doubled the existing practices of mid-training with raw data. Our results suggest that introducing reasoning earlier in the training pipeline results in models that are not only initially better at reasoning, but also better prepared for reasoning-intensive post-training. Figure: Figure 10: Thinking Mid-training. Our approach teaches models to interleave thinking to fill implicit reasoning gaps in pretraining corpora through three steps: (1) a annotator model demonstrates augmenting pretraining data with interleaved thoughts; (2) SFT mid-training teaches a student when and what to think alongside original content; (3) RL mid-training improves thought generation via an LLM judge. The resulting model achieves stronger performance of general reasoning both before and after standard RL post-training. Refer to caption: 2601.21343v3/x7.png Large language models (LLMs) have achieved remarkable capabilities through a two-stage training paradigm: pretraining on vast corpora of unstructured text, followed by post-training on curated instruction-response pairs (ouyang2022training; touvron2023llamaopenefficientfoundation). pretraining imbues models with foundational knowledge of language, world facts, and basic patterns, while post-training, through supervised fine-tuning (SFT) and reinforcement learning (RL), teaches models to follow instructions, engage in dialogue, and perform complex reasoning (wei2022chain; zelikman2022star). This clearly defined multi-stage process has proven remarkably effective, yet introduces a fundamental tension: reasoning capabilities are not prioritized during pretraining and must be optimized primarily during post-training. This gap between pretraining and post-training creates several challenges. First, post-training must simultaneously teach both task-specific formats and general reasoning skills, limiting its efficiency. Second, the raw text consumed during pretraining is presented without explicit reasoning traces, leaving models to learn only surface-level patterns rather than the underlying thought processes. Lastly, recent work on reinforcement learning with verifiable rewards (RLVR) has demonstrated that models can acquire substantial reasoning capabilities through post-training alone (guo2025deepseek), but this approach may be fundamentally limited by the reasoning foundations established during earlier training phases. We hypothesize that closing this gap by introducing reasoning earlier in the training pipeline can yield models that are not only better at reasoning out of the box, but also better suited for post-training and ultimately achieve stronger reasoning capabilities. Our key insight is that pretraining data, while lacking explicit reasoning traces, contains rich opportunities for intermediate thinking which can be trained by RL: mathematical derivations benefit from step-by-step explanations, factual passages invite reflection on causes and implications, and narrative text contains implicit logical progressions that can be made explicit. In this work, we introduce a comprehensive framework for thinking mid-training, a novel training phase that bridges pretraining and post-training by teaching models to perform general reasoning on pretraining corpora. Our approach consists of three key components. The first is mid-training data thinking augmentation, where we leverage a teacher language model to augment pretraining chunks with interleaved “thoughts”, intermediate reasoning steps inserted at semantically appropriate positions within the original text. This creates a corpus where reasoning is explicitly woven into natural text. The second component is thinking SFT mid-training, in which we perform supervised fine-tuning on the augmented corpus, training a student model to produce both the original content and the inserted thoughts. This “cold-start” phase teaches the model the mechanics of interleaved reasoning. The third component is thinking RL mid-training, where we further refine the model’s reasoning through reinforcement learning. Here, the model must generate useful thoughts that help predict subsequent text, with an LLM judge providing rewards based on the quality of predictions. This encourages thoughts that are genuinely beneficial rather than merely imitative. Following thinking mid-training, we apply standard RL post-training with verifiable rewards on mathematical reasoning tasks. Our experiments demonstrate that thinking mid-training substantially improves the effectiveness of post-training: on Llama-3.1-8B, our full pipeline achieves an average score of 0.3785 across challenging mathem and reasoning benchmarks including GSM8K, MATH-500, AMC23, Olympiad, and GPQA-Diamond compared to 0.1197 for direct RL post-training on the base model. Notably, even the SFT mid-training phase alone yields significant gains, with the RL mid-training phase providing additional improvements particularly on the most challenging competition-level problems. Our contributions can be summarized as follows. First, we identify and address the reasoning gap between pretraining and post-training, proposing thinking mid-training as an intermediate phase that prepares models for reasoning-intensive post-training. Second, we introduce a data augmentation strategy that enriches pretraining corpora with interleaved thoughts, enabling models to learn reasoning patterns from naturally-occurring text. Third, we propose a two-phase mid-training procedure combining supervised learning for reasoning pattern acquisition with reinforcement learning for reasoning quality optimization. Finally, we demonstrate substantial improvements on mathematical reasoning benchmarks, showing that our approach effectively closes the gap between pretraining and post-training. ### 2.1 Method We introduce a multi-step procedure for teaching models to reason throughout mid-training and post-training. #### 2.1.1 Mid-training Data Thinking Augmentation We introduce a data augmentation strategy that enriches pretraining corpora with intermediate “thoughts”. Given a pretraining corpus $\mathcal{D}$, we first partition it into chunks of length $L$: $\mathcal{D}=\{c^{1},c^{2},\ldots,c^{N}\}$, where each chunk $c^{i}$ represents a contiguous segment of text with $|c^{i}|\leq L$ tokens. For each chunk $c^{i}$, we employ an annotator language model $\mathcal{A}$ to generate an augmented version $\tilde{c}^{i}$ that interleaves the original content with generated thoughts: $$ $\tilde{c}^{i}=\mathcal{M}_{\text{teacher}}(c^{i};p_{t})$ $$ where $p_{t}$ represents the prompt ([Figure 19](#A3.F19)) that instructs the teacher model to insert thoughts at semantically appropriate positions within $c^{i}$. The resulting augmented chunk $\tilde{c}^{i}$ takes the form: $\tilde{c}^{i}=[x_{1},\tau_{1},x_{2},\tau_{2},\ldots,x_{K},\tau_{K}]$, where $x_{j}$ represents segments of the original text and $\tau_{j}$ denotes the generated thoughts, such that $\text{concat}(x_{1},\ldots,x_{K})=c^{i}$. The final augmented pretraining corpus is constructed as: $\tilde{\mathcal{D}}=\{\tilde{c}^{1},\tilde{c}^{2},\ldots,\tilde{c}^{N}\}$. #### 2.1.2 Thinking Mid-training We introduce a two-step mid-training phase. The first is a “cold-start” supervised fine-tuning phase which learns how to think on pretraining data. The second is a reinforcement learning phase which learns how to optimally think before predicting the next sequence. ##### Thinking SFT Mid-training We perform supervised fine-tuning (SFT) mid-training on half of the augmented corpus, which we call $\tilde{\mathcal{D}}_{SFT}$ using standard next-token prediction. Given a base model $\mathcal{M}_{\text{0}}$ parameterized by $\theta$, we optimize the following objective: $$ $\mathcal{L}_{\text{SFT}}(\theta)=-\mathbb{E}_{\tilde{c}^{i}\sim\tilde{\mathcal{D}}}\left[\sum_{j=1}^{|\tilde{c}^{i}|}\log P_{\theta}(\tilde{c}^{i}_{j}\mid\tilde{c}^{i}_{