Title: 2604.08477 ArXiv: 2604.08477 SuperNova: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions Title: Content selection saved. Describe the issue below: Description: License: arXiv.org perpetual non-exclusive license arXiv:2604.08477v1 [cs.AI] 09 Apr 2026 SuperNova : Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions Ashima Suvarna, Kendrick Phan, Mehrab Beikzadeh, Hritik Bansal, Saadia Gabriel University of California, Los Angeles github.com/asuvarna31/supernova Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved large language model (LLM) reasoning in formal domains such as mathematics and code. Despite these advancements, LLMs still struggle with general reasoning tasks requiring capabilities such as causal inference and temporal understanding. Extending RLVR to general reasoning is fundamentally constrained by the lack of high-quality, verifiable training data that spans diverse reasoning skills. To address this challenge, we propose SuperNova , a data curation framework for RLVR aimed at enhancing general reasoning. Our key insight is that instruction-tuning datasets containing expert-annotated ground-truth encode rich reasoning patterns that can be systematically adapted for RLVR. To study this, we conduct 100 + 100+ controlled RL experiments to analyze how data design choices impact downstream reasoning performance. In particular, we investigate three key factors: (i) source task selection, (ii) task mixing strategies, and (iii) synthetic interventions for improving data quality. Our analysis reveals that source task selection is non-trivial and has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance. Finally, models trained on SuperNova outperform strong baselines (e.g., Qwen3.5) on challenging reasoning benchmarks including BBEH, Zebralogic, and MMLU-Pro. In particular, training on SuperNova yields relative improvements of up to 52.8 % 52.8\% on BBEH across model sizes, demonstrating the effectiveness of principled data curation for RLVR. Our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Figure 1: SuperNova elicits strong general reasoning. We show that training with our curated SuperNova data leads to consistent pass@k improvements across varying values of k on a challenging benchmark, BBEH-test. We highlight that SuperNova is effective on various models sizes from Qwen3 family. 1 Introduction Large language models (LLMs) have shown remarkable progress in reasoning capabilities for formal domains such as mathematics and code (Guo et al. , 2025 ; Lambert et al. , 2024 ; Guha et al. , 2025 ; Ma et al. , 2025 ; Zeng et al. , 2025 ; Hu et al. , 2025 ; Chen et al. , 2025 ) . However, real-world problem solving requires a broader spectrum of reasoning skills beyond formal domains. For example, it may involve different forms of reasoning, such as determining that a street might be wet because it rained (causal inference), or understanding that an event scheduled "next Friday" cannot conflict with one scheduled "last Tuesday" (temporal reasoning). We refer to this broader set of capabilities as general reasoning—the ability to derive novel conclusions from existing knowledge using skills such as logical deduction, causal reasoning, spatial understanding, and pragmatic inference (Newell et al. , 1972 ; Johnson-Laird, 2010 ; Griffiths, 2020 ) . These capabilities are critical for solving tasks in benchmarks such as Big-Bench Extra Hard (BBEH), which evaluate complex forms of general reasoning. A widely adopted approach for improving reasoning in LLMs is reinforcement learning with verifiable rewards (RLVR). Specifically, RLVR relies on the ability to verify model outputs against a ground-truth final answer (Guo et al. , 2025 ) . The wide availability of human-verified data (e.g., MATH (Hendrycks et al. , 2021 ) , competitions 1 1 1 https://artofproblemsolving.com/ , CodeForces) has led to the rapid scaling of RLVR pipelines for STEM reasoning (Chen et al. , 2025 ; Yu et al. , 2025 ; Akter et al. , 2026 ; Hu et al. , 2025 ) . However, we find that exposure to STEM reasoning (Bhaskar et al. , 2025 ; Huan et al. , 2025 ; Zhou et al. , 2025 ) does not transfer reasoning capabilities to general reasoning tasks. For example, OpenReasoner-7B (Hu et al. , 2025 ) and OpenThinker-7B (Guha et al. , 2025 ) outperform the base model by + 50 % +50\% on challenging math benchmarks such as AIME24 (Zhang and Math-AI, 2024 ) , while reducing performance by − 8 % -8\% on general reasoning tasks in BBEH. Prior work such as General Reasoner (Ma et al. , 2025 ) has attempted to scale RLVR beyond STEM. Specifically, it focuses on deriving domain-specific (e.g., science, business, finance, and history) question–answer pairs from the web. However, this approach suffers from key limitations: (a) expanding to new domains does not necessarily improve the skills required for general reasoning (e.g., models achieve strong MMLU scores but still perform poorly on BBEH), and (b) data sourced from the internet is often difficult to verify due to noise and varying quality. On the other hand, obtaining high-quality human-verified data for teaching general reasoning skills via RLVR is expensive and labor-intensive. To this end, we make a crucial observation: there exists a plethora of high-quality, human-annotated data in resources curated for instruction-following. Specifically, datasets such as SuperNI (Wang et al. , 2022 ) and FLAN (Wei et al. , 2021 ) contain thousands of expert-annotated tasks, including event understanding, question generation, and object counting (Appendix Table LABEL:app_tab:superni ). However, these datasets cannot be directly used for RLVR for several reasons: (a) many open-ended tasks do not allow easy verification, (b) not all tasks are useful for eliciting strong reasoning capabilities, and (c) the principles for curating RLVR data for general reasoning remain underexplored. To address these challenges, we propose SuperNova , a data curation framework for RLVR to advance general reasoning. SuperNova is a multi-stage pipeline for curating high-quality RLVR data to improve downstream general reasoning performance (Figure 2 ). Importantly, we explore several data design decisions through 100 + 100+ compute-matched RL experiments. First, we start with a set of candidate tasks from SuperNI and assess their ability to elicit complex general reasoning across the 23 23 sub-tasks in BBEH. § 3.1 . This involves reformatting several open-ended tasks into easily verifiable formats (e.g., converting them into multiple-choice questions). We find that source task selection has a dramatic impact on downstream general reasoning performance (pass@k). Second, we evaluate two strategies for mixing diverse tasks in the source data. Macro mixing selects a shared set of top-performing tasks across all sub-tasks based on average performance, while micro mixing selects the top-performing tasks separately for each sub-task. Interestingly, we observe that micro mixing consistently outperforms macro mixing, suggesting that different reasoning skills benefit from different source tasks. Third, we examine whether synthetically generated data interventions (e.g., introducing long-context dependencies in questions) improve data quality. Surprisingly, we find that augmenting the original data with these interventions does not improve performance under a fixed training budget. Finally, we combine these insights to construct the SuperNova dataset, comprising 25 25 K RLVR samples that achieve state-of-the-art performance on challenging general reasoning benchmarks. Specifically, we train Qwen3 models of various sizes (0.6B–4B) on SuperNova (Figure LABEL:fig:pull ). We find that SuperNova -4B achieves relative gains of 29.4 % 29.4\% and 42.9 % 42.9\% on pass@1 and pass@8, respectively, on BBEH-test (§ 6 ). These results show that (a) SuperNova elicits strong general reasoning capabilities, and (b) performance improves with increased test-time compute (i.e., from 1 to 8 attempts per problem), highlighting improved exploration during reasoning. Notably, SuperNova -4B outperforms the larger Qwen3-8B model by 8.2 % 8.2\% on pass@8 on general reasoning tasks, demonstrating that SuperNova enables training smaller yet stronger general reasoners. Furthermore, SuperNova models exhibit strong generalization across additional reasoning benchmarks, including BBH (Suzgun et al. , 2023 ) , MMLU-Pro (Wang et al. , 2024 ) , and Zebralogic (Lin et al. , 2025 ) (§ 6 ). In particular, SuperNova -4B achieves a relative improvement of 12.3 % 12.3\% on pass@8 over the baseline model across these benchmarks. Overall, our experiments provide practical guidelines for principled RLVR data curation for training strong general reasoners. Figure 2: SuperNova Framework: In this work, we curate reasoning data from natural instruction to enhance general reasoning capabilities in LLMs. First, we study the impact of task selection on downstream reasoning performance. Then, we explore strategies to mix diverse tasks in source data. Finally, we examine whether synthetic data interventions can enhance data quality and improve downstream reasoning. 2 Preliminaries Reinforcement Learning with Verifiable Rewards (RLVR). RLVR is widely adopted for training LLMs for reasoning in domains that rely on automatically verifiable ground truth such as mathematics and code. Given an input-target pair ( q , t ) (q,t) , RLVR samples G rollouts o i i = 1 G {o_{i}}_{i=1}^{G} from a behavior policy π θ old \pi_{\theta_{\text{old}}} and optimizes the GRPO (Shao et al. , 2024 ) objective: 𝒥 GRPO ​ ( θ ) = 𝔼 ( q , t ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{(q,t)\sim\mathcal{D},\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} [ 1 G ​ ∑ i = 1 G min ⁡ ( ρ i ​ ( θ ) ​ A ^ i , clip ​ ( ρ i ​ ( θ ) , 1 − ϵ , 1 + ϵ ) ​ A ^ i ) ] , \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\min\Big(\rho_{i}(\theta)\,\hat{A}_{i},\;\text{clip}\!\left(\rho_{i}(\theta),1{-}\epsilon,1{+}\epsilon\right)\hat{A}_{i}\Big)\Bigg], (1) where ρ i ​ ( θ ) = π θ ​ ( o i ∣ q ) π θ old ​ ( o i ∣ q ) \rho_{i}(\theta)=\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\text{old}}}(o_{i}\mid q)} is the importance sampling ratio. The group-centered advantage A ^ i \hat{A}_{i} for each output is computed as A ^ i = r i − 1 G ​ ∑ j = 1 G r j \hat{A}_{i}=r_{i}-\frac{1}{G}\sum_{j=1}^{G}r_{j} where r i = r ​ ( o i , q ) r_{i}=r(o_{i},q) , the computed reward. Following Yu et al. ( 2025 ) , we skip the KL penalty to improve training efficiency in our experiments. Task-Specific Instruction Datasets. Instruction-tuning datasets such as SuperNI (Wang et al. , 2022 ) , and Flan-Collection (Wei et al. , 2021 ) are a collection of well-structured, distinct tasks spanning diverse reasoning abilities. These datasets are constructed from high-quality human supervision including task definitions, instructions and ground-truth annotations. We observe that these large instruction-tuning datasets often encode reasoning structures that are not explicitly annotated but can be inferred from the examples and task structure. Consider an instruction dataset D = { D 1 , D 2 , D 3 ​ … ​ D K } D=\{D_{1},D_{2},D_{3}...D_{K}\} comprising K tasks where each subset D k D_{k} is a well-defined task targeting a particular skill. General Reasoning Benchmarks. Evaluation benchmarks that aim to evaluate models on general reasoning, such as BBEH and BBH, can be decomposed into sub-tasks that target a particular skill. This decomposability allows us to systematically assess how task-specific data translates to broad reasoning gains. We adopt BBEH as our validation benchmark as it consists of 23 diverse sub-tasks spanning linguistic, logical and commonsense reasoning (Appendix Table 5 ). Formally, we define our validation set as V = { V 1 , V 2 … . . V N } V=\{V_{1},V_{2}.....V_{N}\} where V i V_{i} denotes the sub-tasks in the benchmark. Problem Setup. In this work, we focus on the curation of high-quality training data to enable strong general reasoning capabilities via reinforcement learning. Given a pool of candidate datasets D = { D 1 , D 2 , … , D K } D=\{D_{1},D_{2},\ldots,D_{K}\} , a model M M , and a training algorithm A A , we seek a subset of tasks S ⊆ D S\subseteq D that maximizes downstream performance after training. Following the data curation formulations proposed for SFT in math reasoning (Guha et al. , 2025 ) and multimodal reasoning (Bansal et al. , 2025 ) , we define our objective as: S ∗ = a ​ r ​ g ​ m ​ a ​ x S ⊆ D ​ Φ ​ ( A ​ ( M , S ) , V ) S^{*}=argmax_{S\subseteq D}\Phi(A(M,S),V) (2) where A ​ ( M , S ) A(M,S) denotes the model after A A to M M on the selected subset S S , V is the validation set and ϕ \phi measures downstream performance on V V . 3 SuperNova We outline our SuperNova framework (Figure 2 ), which consists of multiple stages: (a) task selection, which assesses the impact of task choice (§ 3.1 ); (b) mixing, which identifies the best strategy to mix the diverse tasks (§ 3.2 ); and (c) data interventions, which aim to enhance the quality of our data (§ 3.3 ) 3.1 Task Selection Extracting reasoning data from instructions. The quality of the input and the coverage of reasoning types is critical for determining the reasoning skills imparted to the LLM. For example, a LLM exposed to temporal graphs will excel in downstream temporal understanding tasks (Xiong et al. , 2024 ) . In this work, we leverage instruction-tuning data D D to source diverse tasks D k D_{k} for general reasoning. Since, instructions are formatted for supervised-finetuning they are not directly usable for RLVR as they may incorporate hard to verify ground-truth. Thus, for every instruction p p in D k D_{k} , we reformat the instruction to a verifiable question q q . To further identify the most effective data from D D , we sample 8 rollouts from model M M for each q q and compute the per-question win-rate . Finally, we remove all questions which are too easy for the model (win-rate=1) or too challenging (win-rate=0). Task Ranking. For each task D k D_{k} , we define a task-utility score u k ∈ R u_{k}\in R that indicates how effective D k D_{k} is for RLVR training. Then, we rank the K K tasks according to their utility scores, producing a ranking: u D 1 > u D 2 > u D 3 > ⋯ > u D K ) u_{D_{1}}>u_{D_{2}}>u_{D_{3}}>\cdots>u_{D_{K})} . The task utility scores enable us to select high-quality tasks while downweighting poor and irrelevant tasks. We explore various approaches to compute task-utility: (a) we compute the semantic and lexical similarity between the task questions and the questions from our validation benchmark V V ; (b) we compute the difficulty of the task based on the average win-rate of the task under model M M ; and (c) we train model M M on D k D_{k} and evaluate performance on V V . We adopt approach (c) in our main experiments. 3.2 Mixing After obtaining high-quality tasks, we determine how to combine them to construct an effective training mixture. Mixing strategy is a key design choice in data curation and prior work in LLM reasoning have shown to yield superior datasets by mixing subsets from various sources. Consider the K tasks from § 3.1 and number of tasks to be mixed N ∈ { 1 , 2 , 4 , 8 , 16 } N\in\{1,2,4,8,16\} , we want to determine the optimal value of N under two mixing strategies: • Macro Mixing: Consider the ranking from § 3.1 : u D 1 > u D 2 > u D 3 . . > u D K u_{D_{1}}>u_{D_{2}}>u_{D_{3}}..>u_{D_{K}} where u D k u_{D_{k}} is the macro average of model performance on V B ​ B ​ E ​ H V_{BBEH} . We select the top-ranked N tasks u D 1 > u D 2 > u D 3 > ⋯ > u D N u_{D_{1}}>u_{D_{2}}>u_{D_{3}}>\cdots>u_{D_{N}} for our mixture. • Micro Mixing: Here, we leverage the sub-tasks of our V V and produce a ranking for each sub-task V i V_{i} . Specifically, we define u k ( i ) u_{k}^{(i)} as the performance of model M M trained on D k D_{k} and evaluated on sub-task V i V_{i} , yielding a per-sub-task ranking: u D 1 ( i ) > u D 2 ( i ) > ⋯ > u D K ( i ) u_{D_{1}}^{(i)}>u_{D_{2}}^{(i)}>\cdots>u_{D_{K}}^{(i)} for each V i ∈ V V_{i}\in V . We then select the top-ranked N N tasks per sub-task and take the unique set of selected tasks for our mixture. 3.3 Data Interventions Starting from the best mixture from § 3.2 , we assess whether we can enhance the data quality through targeted data interventions. RLVR datasets primarily focus on the questions since interventions on the target may hinder the verifiability of the answer. Thus, we apply a set of interventions to transform the difficulty of the questions while preserving the target answer. These interventions aim to increase the difficulty of the questions by introducing diverse reasoning types such very long-context dependency, information that prompts model to go against a strong prior or needle in haystack. Let D base = { ( q , t ) } D_{\text{base}}=\{(q,t)\} be the base data with original question-target pairs. We apply an intervention I I that transforms each question while preserving the target, producing D ′ = { ( I ​ ( q ) , t ) } D^{\prime}=\{(I(q),t)\} . We provide the implementation details of applying these interventions in Appendix § G . 4 Experimental Setup Training Data. We use SuperNI (Wang et al. , 2022 ) as our data source. SuperNI consists of 1600 tasks spanning various tasks types such as question answering, question generation and commonsense reasoning. Each task consists of the task description and the instruction-response pair, annotated by experts. For our experiments, we select a candidate pool of 83 tasks. We provide the candidate pool selection strategy in Appendix § B . Training. We train models from Qwen3 Yang et al. ( 2025 ) family (0.6B, 1.7B and 4B) with GRPO (Shao et al. , 2024 ) for all our experiments (§ 2 ). For our data curation experiments, we use Qwen3-0.6B for faster training iterations. All our data curation experiments were run for 250 RL steps. Finally, we train the SuperNova models for 5000 RL steps. We present more details about the training setup in Appendix B . Evaluation. We evaluate our models on various benchmarks that target diverse reasoning capabilities. For our data curation experiments, we choose BBEH-mini as our validation benchmark. BBEH-mini is a curated subset of BBEH (Kazemi et al. , 2025 ) consisting of 460 examples spanning 23 tasks that target diverse reasoning capabilities. We use the remaining BBEH examples, which are not included in BBEH-mini as the unseen test set, BBEH-test. After creation of our SuperNova dataset, we evaluate our models on 4 additional unseen benchmarks including BBH (Suzgun et al. , 2023 ) , Zebralogic (Lin et al. , 2025 ) , MMLU-Pro (Wang et al. , 2024 ) and MATH500‘ (Lightman et al. , 2023 ) . To ensure consistency, we use an identical prompt across all evaluations that encourages the model to think before answering, provided in Appendix B . Evaluation Metric. We adopt pass@k as our evaluation metric, which is well-suited for evaluating RL-trained models (Chen et al. , 2021 ; Yue et al. , 2025 ) . As shown in Appendix § 6 , we find that pass@8 provides 2.5 times greater discriminability than pass@1 ( σ \sigma : 0.76 → \rightarrow 1.92). We therefore utilize pass@8 for our data curation experiments. Baselines Models. We evaluate several models as baselines for our experiments. (1) Qwen3 and Qwen3.5 family : included to measure the gains obtained from training on SuperNova . (2) General-Reasoner-Qwen3-4B (Ma et al. , 2025 ) : an all-domain reasoning model that is closest in motivation to our models. (3) OpenThinker3-7B (Guha et al. , 2025 ) : a strong math reasoning model supervised finetuned on large math corpus. (4) OpenReasoner-Nemotron-7B (Ahmad et al. , 2025 ) : a strong reasoning model. (5) Olmo3-7B-Think (Olmo et al. , 2025 ) : a state-of-art reasoning model. To further compare the quality of SuperNova with other reasoning datasets, we train Qwen3-0.6B under a compute-matched setup using three baseline datasets including Nemotron-CrossThink (Akter et al. , 2026 ) , which targets various domains beyond math; General-Reasoner (Ma et al. , 2025 ) , which curates reasoning data across diverse STEM-focused domains; and DAPO (Yu et al. , 2025 ) , a high-quality math reasoning dataset sourced from competition websites. We provide additional details in Appendix § C . Figure 3: Impact of Task Selection. We train the baseline (Qwen3-0.6B) on each task individually under compute-matched settings. We report relative pass@8 gains on BBEH-mini for each task and highlight the tasks that improve and degrade the baseline. 5 Experiments 5.1 Impact of Task Selection We train Qwen3-0.6B on each task and report model performance on BBEH in Fig. 3 . Our experiments show that task selection has a substantial impact on downstream reasoning performance. Specifically, we observe a 7.6 percentage point (pp) gap between the lowest-performing task ( task213-rocstories , − 9 -9 pp vs. baseline) and the highest-performing task ( task738-perspectrum , + 39 +39 pp vs. baseline). This large spread indicates that tasks do not contribute equally to downstream reasoning. Notably, several tasks degrade performance relative to the baseline, underscoring that not all tasks are beneficial for improving general reasoning under RLVR. Furthermore, we find that tasks involving multi-hop reasoning yield the largest gains over the baseline model (Appendix § F ). Additionally, we explore the efficacy of semantic similarity and lexical similarity between the source and validation tasks as a way to assess task utility (Appendix § H ). However, we observe weak correlation between similarity scores and downstream performance on BBEH. Top 1 Top 2 Top 4 Top 8 Top 16 Micro Mixing pass@1 7.5 8.9 7.5 7.6 7.5 pass@8 18.3 22.8 18.7 18.0 20.2 Macro Mixing pass@1 7.6 8.2 6.6 6.4 7.5 pass@8 21.5 21.7 17.4 17.0 18.3 Table 1: Impact of mixing. We mix questions from tasks following two strategies: micro mixing and macro mixing. We find that micro mixing with top 2 tasks achieves the best performance (bold) . Intervention pass@8 Micro-Top2 22.8 Going Against Prior 22.6 Long-Context 21.3 Inductive Reasoning 20.4 Finding Errors 20.0 Many-hop Reasoning 20.0 Knowledge-intensive Reasoning 19.8 Compositional Understanding 19.6 Learning on the Fly 18.3 Table 2: Impact of interventions. We compare the performance of models trained on datasets transformed using synthetic interventions. We find that the base dataset is superior to all interventions. 5.2 Impact of Task Mixing We present the results of two mixing strategies: Macro Mixing and Micro Mixing in Table 1. Across both strategies, we find that mixing questions from the top two tasks yields the best results, with Micro Mixing achieving the highest pass@8 of 22.8 % 22.8\% .This suggests a trade-off in task diversity: mixing too few tasks limits data diversity, while mixing too many degrades performance. Furthermore, Micro Mixing consistently outperforms Macro Mixing regardless of the number of tasks combined. These results indicate that selecting top-ranked tasks per sub-task (Micro Mixing) better preserves coverage across diverse reasoning skills, whereas selecting tasks based on overall ranking (Macro Mixing) biases the mixture toward a narrower set of abilities. Model BBEH-mini BBEH-test pass@1 pass@8 pass@1 pass@8 Openthinker3-7B 3.4 7.6 4.4 10.1 OpenReasoning-Nemotron-7B 3.5 8.5 4.8 11.0 Olmo-3-7B-Think 6.4 13.0 6.8 14.9 Qwen3-8B 11.9 21.7 14.1 24.2 Qwen3-0.6B 5.7 15.4 5.2 15.2 Qwen3-0.8B 6.6 22.2 7.4 23.8 SuperNova -0.6B 9.8 24.6 9.7 25.0 Qwen3-1.7B 7.6 18.0 7.8 17.7 Qwen3.5-2B 7.5 22.4 8.1 25.8 SuperNova -1.7B 12.4 25.7 11.7 26.7 Qwen3-4B 12.0 23.7 13.6 23.2 General-Reasoner-4B 11.5 30.0 12.0 32.9 SuperNova -4B 14.8 31.7 17.6 33.3 Table 3: Performance of models trained with SuperNova data. We compare the pass@1 and pass@8 of models on BBEH. We find that models trained on SuperNova achieve best-in-class performance across model sizes. SuperNova -4B even beat Qwen3-8B. 5.3 Impact of Data Interventions We apply several data intervention strategies to the best-performing dataset from § 5.2 (Micro-Top2) and report results on BBEH-mini in Table 2. Surprisingly, none of the interventions improve over the original data. While Going Against Prior achieves the highest performance among the interventions ( 22.6 % 22.6\% ), it still falls short of Micro-Top2. This suggests that synthetically generated interventions can degrade data quality, and that improving already high-quality data through such interventions is non-trivial. 6 Training Reasoners with SuperNova SuperNova elicits strong general reasoning. We evaluate the performance of SuperNova models across different scales (0.6B, 1.7B, and 4B) and compare them against several strong reasoning LLMs. Results are reported in Table 3 . Notably, we find that models trained with SuperNova achieve the best pass@1 and pass@8 on BBEH across all model sizes. In particular, SuperNova -1.7B achieves relative gains of 44pp and 3.5pp over Qwen3.5-2B at pass@1 and pass@8 , respectively while SuperNova -4B outperforms of General-Reasoner-4B by 46pp and 1.2pp. Remarkably, SuperNova -4B outperforms Qwen3-8B—a 2 × \times larger model—by 8.2pp on pass@8 , highlighting the effectiveness of SuperNova in training strong general reasoners even at smaller scales. We report per-sub-task pass@8 scores on BBEH-test in Appendix Table 5 . SuperNova beats SOTA reasoning datasets. We compare SuperNova against three state-of-the-art reasoning datasets that target diverse reasoning skills. To ensure a fair comparison of data quality, we perform a compute-matched analysis of all datasets (details in Appendix § C ). Results on BBEH-mini are shown in Figure 4 . We find that SuperNova achieves gains of 42pp on pass@1 and 28pp on pass@8 over the strongest baseline, Nemotron-Crossthink. In contrast, both math reasoning datasets, DAPO and Nemotron-Crossthink (Math) show little to no improvement over the baseline. Overall, under fixed compute, SuperNova delivers substantially better reasoning performance than existing datasets. Model MMLU-Pro BBH Zebralogic MATH500 Average Qwen3-0.6B 55.3 52.4 34.4 71.9 53.5 SuperNova -0.6B 56.2 81.5 49.1 71.4 64.6 Qwen3-1.7B 64.3 80.3 53.3 73.2 67.8 SuperNova -1.7B 61.5 79.7 75.5 84.0 75.2 Qwen3-4B 71.2 84.4 55.6 74.1 71.3 SuperNova -4B 76.0 89.3 77.0 77.9 80.1 Table 4: Performance of SuperNova models on unseen benchmarks . We report pass@8 across four benchmarks that were unseen during data curation. We find that LLMs trained on SuperNova show improved pass@8 across all model sizes. Figure 4: Comparison with Other Reasoning Datasets. We report the relative gains achieved by training Qwen3-0.6B on SuperNova and existing reasoning datasets. SuperNova generalizes to Out-of-Distribution (OOD) Benchmarks We evaluate SuperNova on challenging reasoning benchmarks that are unseen during data curation, as shown in Table 4 . We observe consistent improvements across all benchmarks and model sizes. Notably, SuperNova achieves substantial gains on Zebralogic, where SuperNova -4B outperforms Qwen3-4B by 21pp. This suggests that training on SuperNova enhances logical reasoning capabilities, particularly for constraint satisfaction tasks. We also find that SuperNova -1.7B achieves an average score of 75.2 % 75.2\% , exceeding Qwen3-4B ( 71.3 % 71.3\% ) despite being half its size, highlighting the efficacy of SuperNova . Finally, SuperNova models maintain competitive performance on MATH500, with modest gains, indicating that training on SuperNova does not degrade mathematical reasoning capabilities. Figure 5: (Left) We train Qwen3.5-2B and LLaMA3.2-3B-Instruct with SuperNova and show relatives gains over the baseline model on BBEH-mini. (Right) We show the performance comparison between the baseline model and SuperNova -0.6B by scaling values of k till 128 on BBEH-mini. SuperNova gains are consistent at larger values of k k . We analyze whether the performance gains from SuperNova persist at higher values of k k . As shown in Figure 5 (Right), SuperNova -0.6B maintains consistent gains over Qwen3-0.6B across all values of k k up to 128. This suggests that training on SuperNova expands the model’s exploration space even at large sample sizes, enabling more diverse reasoning behaviors than the baseline. SuperNova shows cross-model generalization. We study how training on SuperNova generalizes across model families (Figure 5 (Left)). In particular, LLaMA3.2-3B-Instruct (Grattafiori et al. , 2024 ) trained on SuperNova achieves gains of 15.8pp over its baseline. We further observe similar improvements on Qwen3.5-2B (Team, 2026 ) , suggesting that data curation insights derived from earlier-generation models (e.g., Qwen3) transfer to newer-generation models. Overall, the benefits of SuperNova generalize across both model families and generations. 7 Conclusion In this work, we propose SuperNova for RLVR data curation to enhance the general reasoning capabilities of LLMs. SuperNova leverages large-scale instruction-tuning datasets to curate RLVR data for general reasoning. Through controlled experiments, we surface several key insights. We show that task selection and micro mixing are critical for training strong general reasoners. Additionally, we find that augmenting data with synthetically generated interventions fails to improve reasoning performance. Additionally, we demonstrate that the gains from SuperNova generalize across model families and challenging benchmarks. While our results demonstrate the effectiveness of SuperNova , several directions remain for future work. First, we evaluate general reasoning on a fixed set of academic benchmarks that may not fully capture real-world problem solving. Second, our experiments are conducted under limited compute, future work may explore how our findings persist with unbounded compute and data. Finally, we believe SuperNova has provided several practical insights that can spur further data curation efforts for RLVR beyond STEM domains. Ethical Concerns Disclosure of LLM use in both research and reviewing. We use ChatGPT and Claude as in our experiments and have provided the relevant prompts. Claude was used in formatting latex tables and code generation for the figures. Finally, we used ChatGPT and Claude to assist with grammar and proof-reading in our paper writing. References W. U. Ahmad, S. Narenthiran, S. Majumdar, A. Ficek, S. Jain, J. Huang, V. Noroozi, and B. Ginsburg (2025) OpenCodeReasoning: Advancing Data Distillation for Competitive Coding . arXiv preprint arXiv:2504.01943 . External Links: 2504.01943 , Link Cited by: §4 . S. N. Akter, S. Prabhumoye, M. Novikov, S. Han, Y. Lin, E. Bakhturina, E. Nyberg, Y. Choi, M. Patwary, M. Shoeybi, et al. (2026) Nemotron-crossthink: scaling self-learning beyond math reasoning . In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 984–1002 . Cited by: §A.1 , §1 , §4 . H. Bansal, D. S. Sachan, K. Chang, A. Grover, G. Ghosh, W. Yih, and R. Pasunuru (2025) Honeybee: data recipes for vision-language reasoners . arXiv preprint arXiv:2510.12225 . Cited by: §2 . A. Bhaskar, X. Ye, and D. Chen (2025) Language models that think, chat better . External Links: 2509.20357 , Link Cited by: §1 . M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code . arXiv preprint arXiv:2107.03374 . Cited by: Appendix E , §4 . Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping (2025) Acereason-nemotron: advancing math and code reasoning through reinforcement learning . arXiv preprint arXiv:2505.16400 . Cited by: §A.2 , §1 , §1 . A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models . arXiv preprint arXiv:2407.21783 . Cited by: §6 . T. L. Griffiths (2020) Understanding human intelligence through human limitations . Trends in Cognitive Sciences 24 ( 11 ), pp. 873–883 . Cited by: §1 . E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, et al. (2025) Openthoughts: data recipes for reasoning models . arXiv preprint arXiv:2506.04178 . Cited by: §A.2 , §1 , §1 , §2 , §4 . D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948 . Cited by: §A.2 , §1 , §1 . D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset . External Links: 2103.03874 , Link Cited by: §1 . J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum (2025) Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model . arXiv preprint arXiv:2503.24290 . Cited by: §A.2 , §1 , §1 . M. Huan, Y. Li, T. Zheng, X. Xu, S. Kim, M. Du, R. Poovendran, G. Neubig, and X. Yue (2025) Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning . External Links: 2507.00432 , Link Cited by: §1 . Hugging Face (2025) Open r1: a fully open reproduction of deepseek-r1 . External Links: Link Cited by: §A.2 . P. N. Johnson-Laird (2010) Mental models and human reasoning . Proceedings of the National Academy of Sciences 107 ( 43 ), pp. 18243–18250 . Cited by: §1 . M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, Y. P. Chen, et al. (2025) Big-bench extra hard . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 26473–26501 . Cited by: Table 6 , Appendix G , §4 . N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training . arXiv preprint arXiv:2411.15124 . Cited by: §1 . H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step . In The twelfth international conference on learning representations , Cited by: §4 . B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi (2025) Zebralogic: on the scaling limits of llms for logical reasoning . arXiv preprint arXiv:2502.01100 . Cited by: §1 , §4 . J. Liu, Y. Fan, Z. Jiang, H. Ding, Y. Hu, C. Zhang, Y. Shi, S. Weng, A. Chen, S. Chen, Y. Huang, M. Zhang, P. Zhao, J. Yan, and J. He (2025) SynLogic: synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond . External Links: 2505.19641 , Link Cited by: §A.1 . Y. Liu, J. Li, and Z. Zheng (2026) RuleReasoner: reinforced rule-based reasoning via domain-aware dynamic sampling . External Links: 2506.08672 , Link Cited by: §A.1 . X. Lu, D. Acuna, J. Jung, J. Hu, D. Zhang, S. Diao, Y. Zou, S. Zhang, B. Cui, M. Liu, H. Kim, P. Ammanabrolu, J. Kautz, Y. Dong, and Y. Choi (2026) Golden goose: a simple trick to synthesize unlimited rlvr tasks from unverifiable internet text . External Links: 2601.22975 , Link Cited by: §A.1 . X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen (2025) General-reasoner: advancing llm reasoning across all domains . arXiv preprint arXiv:2505.14652 . Cited by: §A.1 , §1 , §1 , §4 , §4 . N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. B. Hashimoto (2025) S1: simple test-time scaling . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pp. 20286–20332 . Cited by: §A.2 . A. Newell, H. A. Simon, et al. (1972) Human problem solving . Vol. 104 , Prentice-hall Englewood Cliffs, NJ . Cited by: §1 . T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, et al. (2025) Olmo 3 . arXiv preprint arXiv:2512.13961 . Cited by: §4 . Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . Cited by: §2 , §4 . M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, et al. (2023) Challenging big-bench tasks and whether chain-of-thought can solve them . In Findings of the Association for Computational Linguistics: ACL 2023 , pp. 13003–13051 . Cited by: §1 , §4 . Q. Team (2026) Qwen3. 5: towards native multimodal agents . URL: https://qwen. ai/blog . Cited by: §6 . Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. (2022) Super-naturalinstructions: generalization via declarative instructions on 1600+ nlp tasks . In Proceedings of the 2022 conference on empirical methods in natural language processing , pp. 5085–5109 . Cited by: §A.2 , §1 , §2 , §4 . Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. (2024) Mmlu-pro: a more robust and challenging multi-task language understanding benchmark . Advances in Neural Information Processing Systems 37 , pp. 95266–95290 . Cited by: §1 , §4 . J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2021) Finetuned language models are zero-shot learners . arXiv preprint arXiv:2109.01652 . Cited by: §1 , §2 . S. Xiong, A. Payani, R. Kompella, and F. Fekri (2024) Large language models can learn temporal reasoning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 10452–10470 . Cited by: §3.1 . A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report . arXiv preprint arXiv:2505.09388 . Cited by: §4 . Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Liu (2025) Limo: less is more for reasoning . arXiv preprint arXiv:2502.03387 . Cited by: §A.2 . Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale . arXiv preprint arXiv:2503.14476 . Cited by: §1 , §2 , §4 . Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? . arXiv preprint arXiv:2504.13837 . Cited by: Appendix E , §4 . W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He (2025) Simplerl-zoo: investigating and taming zero reinforcement learning for open base models in the wild . arXiv preprint arXiv:2503.18892 . Cited by: §1 . S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang (2025) Instruction tuning for large language models: a survey . External Links: 2308.10792 , Link Cited by: §A.2 . Y. Zhang and T. Math-AI (2024) American invitational mathematics examination (aime) 2024 . Note: https://huggingface.co/datasets/math-ai/aime24 Cited by: §1 . H. Zhao, H. Wang, Y. Peng, S. Zhao, X. Tian, S. Chen, Y. Ji, and X. Li (2025) 1.4 million open-source distilled reasoning dataset to empower large language model training . arXiv preprint arXiv:2503.19633 . Cited by: §A.2 . R. Zhou, M. Xu, S. Chen, J. Liu, Y. Li, X. Lin, Z. Chen, and J. He (2025) Does learning mathematical problem-solving generalize to broader reasoning? . External Links: 2507.04391 , Link Cited by: §1 . Appendix A Related Work A.1 General-Purpose Reasoning in LLMs Several works have explored expanding the general reasoning capabilities of LLMs. Ma et al. ( 2025 ) constructs a large-scale dataset spanning multiple domains such as history, finance, and physics from web-scraped sources. Akter et al. ( 2026 ) goes beyond mathematics by curating synthetically derived questions from CommonCrawl and open-source QA datasets. Lu et al. ( 2026 ) leverages transformed pretraining data with structured templates and distractors to generate verifiable reasoning data in domains such as cybersecurity. However, these approaches largely rely on internet-sourced data, which can be noisy and of low quality. Other work has focused on rule-based tasks (Liu et al. , 2026 ) and logic puzzles (Liu et al. , 2025 ) . While effective for specialized reasoning, these approaches rely on highly curated logic and rule-based datasets that are challenging to scale and cover a limited range of reasoning types. In contrast, SuperNova leverages instruction-tuning datasets, which are human-annotated and generally of higher quality than raw internet data, enabling broader general reasoning capabilities. A.2 Data Curation for Reasoning High-quality reasoning data is critical for training strong LLM reasoners. Prior work has focused on large-scale datasets for supervised fine-tuning (SFT) (Hugging Face, 2025 ; Zhao et al. , 2025 ) and RLVR (Chen et al. , 2025 ; Hu et al. , 2025 ) , typically by scraping competition websites or distilling knowledge from larger models. On the other hand, (Muennighoff et al. , 2025 ; Ye et al. , 2025 ) demonstrate that carefully curated, high-quality reasoning datasets can yield strong gains even with relatively small datasets. Guha et al. ( 2025 ) systematically studies data design principles for SFT reasoning data at scale through controlled experiments, in a manner similar to SuperNova . However, these efforts primarily focus on reasoning in formal domains using SFT. SFT aims to improve instruction-following by mimicking gold responses (Zhang et al. , 2025 ; Wang et al. , 2022 ) , while RLVR optimizes a sparse, outcome-based reward (Guo et al. , 2025 ) . Moreover, SFT typically requires complete reasoning traces and solutions for training, whereas RLVR only requires the final answer. As a result, SFT-oriented data curation strategies do not directly transfer to RLVR. SuperNova addresses this gap by providing key insights to drive data curation for RLVR with a focus on general reasoning. Appendix B Detailed Experimental Setup B.1 Task Selection We utilize Claude-Opus-4.6, to select a candidate set of 83 tasks from the 1600 tasks of SuperNI. To ensure we can conduct a controlled study with our limited compute and keep our search space tractable, we prompt the LLM with a minimal prompt and do not consider the validation benchmark while preparing this candidate pool. This was done to ensure that the task ranking is done purely on task utility scores. The prompt follows: This is an instruction-following task use to train LLMs. Consider, the given task description and examples. Now assess the suitability of the task for RL training reasoning models. Think step by step and only respond with yes/no. Task ID: {task_id} Task Description: {description} Example Input: {input} Example Output: {output} For reformatting the instruction tuning tasks to verifiable questions, we prompt GPT-5-mini with B.1 . To estimate the quality of the reformatting, we manually inspect 100 samples and find that GPT-5-mini follows the prompt accurately on 100% of the samples while preserving the ground-truth and original task structure. Prompt for Reformatting Instruction-Tuning Dataset ⬇ 1 Role : You are an expert Dataset Engineer specializing in Reinforcement Learning from Human Feedback ( RLHF ) and Verifiable Rewards . 2 3 Objective : Transform a raw task description , input , and output into a structured Problem and Solution pair . This pair must be suitable for RL training where the reward is calculated via exact - match verification . 4 5 Constraints : 6 The Problem : Must incorporate all necessary context from the input and output without giving away the output . 7 The Solution : Must contain only the final answer . No explanations , no " The answer is …", and no punctuation unless it is part of the value . 8 Verifiability : The solution must be uniquely extractable via simple string matching or regex . 9 10 Formatting Logic : 11 Open - ended : Use this if the answer is a unique value ( e . g ., a number , a specific name , or a constant ). 12 MCQ ( Multiple Choice ): Use this if the task is subjective , has multiple valid answers , or involves Yes / No . 13 MCQ Format : Provide options labeled ( A ) through ( J ). If the task is multi - correct , the solution should be a comma - separated list of letters ( e . g ., " A , C "). 14 Output Format : Return a valid JSON object with the keys " formatting logic ", " problem " and " solution ". 15 16 Task Description : { def_task } 17 Input : { example_input } 18 Output : { example_output } Finally, for win-rate filtering we generate 8 samples from Qwen3-0.6B at temperature=0.7 (generation length: 4096) , our baseline model and compute the per-question win-rate across these 8 samples. We filter all questions with a win-rate of 0 (too hard) and a win-rate of 1(too easy). B.2 Training All our experiments were done on 4xH100 gpus. We use the GRPO implementation from TRL 2 2 2 https://github.com/huggingface/trl for our training. All our data curation experiments utilize Qwen3-0.6B with 500 prompts, learning rate of 1e-6, 8 generations per prompt, batch size of 8, decoding temperature of 0.7 and maximum generation length 4096. We run our training for 250 steps (1 epoch). For our large scale experiments, we run 5000 steps (1 epoch) across 10,000 prompts and use a learning rate of 1e-6 for 0.6B models and 4e-6 for 1.7B and 4B models. B.3 Evaluation For our evaluations, we use the following prompt across all benchmarks Think step by step, and when you are ready to provide the final answer, use the prefix "The answer is:" followed by the answer directly, with no formatting and no markup. For instance: "The answer is: 42", or "The answer is: yes", or "The answer is: (a)" For multi-choice questions, provide the letter, e.g. "The answer is: (a) All evaluations were conducted on 1xH100 with a batch size of 8. We use decoding temperature=0.7 with maximum generation length of 4096 across all our experiments. Appendix C Implementation Details of Training Baseline Datasets For fair comparison across dataset quality, we train Qwen3-0.6B on the fixed budget of 250 RL steps across 500 prompts and the same learning rate for all datasets. Since Nemotron-Crossthink, Dapo and General-Reasoner are large-datasets, we report their performance as average pass@8 across three runs trained on three random samples of 500 prompts. Appendix D Additional Results on BBEH We provide the per sub-task pass@8 scores of SuperNova and baseline models on BBEH-test in Table 5 . We observe negligible gains on 7 out of the 23 sub-tasks. We observe that SuperNova is able to improve on tasks like Hyperbaton, Multi-step Arithmetic and Shuffling Objects where the base model has near zero performance. Model Avg. Geo. Bool. Shuf. M.Ar. Zebra Hyp. WoL Dis. Word Brd. NYCC T.Ar. Mov. Caus. Sarc Dyck SuperNova-0.6B 25.0 35.3 56.8 48.8 0 48 5.6 7.1 60.6 14.6 53.1 27.1 18.9 47.1 76.7 44 7.5 Qwen3-0.6B 15.2 0 40.5 2.3 0 20 2.8 19 15.2 2.4 46.9 20.8 16.2 25.5 53.5 50 5 Qwen3.5-0.8B 23.8 5.9 27 20.9 0 2 13.9 23.8 78.8 17.1 71.4 41.7 16.2 37.3 81.4 64 10 SuperNova-1.7B 27.6 61.8 37.8 51.2 0 34 0 38.1 57.6 9.8 67.3 16.7 40.5 62.7 53.5 44 2.5 Qwen3-1.7B 17.7 0 32.4 4.7 0 4 2.8 23.8 45.5 9.8 51 14.6 35.1 31.4 60.5 44 5 Qwen3.5-2B 25.5 11.8 37.8 33.7 3.3 1 19.4 27.4 77.3 32.9 40.8 40.6 23 57.8 79.1 56 8.8 SuperNova-4B 33.5 54.4 56.8 41.9 25.6 33 22.2 47.6 65.2 42.7 71.4 30.2 51.4 65.7 66.3 22 20 Gen.-Reasoner-4B 32.9 55.9 62.2 23.3 6.7 20 8.3 45.2 72.7 24.4 79.6 35.4 37.8 78.4 72.1 38 12.5 Qwen3-4B 23.2 0 5.4 0 2.2 10 0 26.2 51.5 36.6 65.3 25 48.6 64.7 67.4 30 52.5 Qwen3-8B 24.2 2.9 5.4 2.3 0 10 0 31 57.6 36.6 59.2 20.8 67.6 51 79.1 38 42.5 OLMo-3-7B-Think 14.9 0 0 0 0 0 0 2.4 48.5 24.4 6.1 20.8 16.2 78.4 62.8 44 2.5 OpenReasoning-7B 11.1 0 0 0 0 0 0 2.4 60.6 9.8 4.1 29.2 0 19.6 65.1 34 0 OpenThinker3-7B 10.1 0 0 0 0 0 0 0 54.5 17.1 12.2 12.5 8.1 9.8 58.1 28 5 Table 5: BBEH pass@8 results (%) on 16 selected tasks, grouped by model size. 7 near-zero tasks omitted (buggy tables, linguini, object counting, object properties, spatial reasoning, sportqa, temporal sequence). Best in each size class is bolded . Task abbreviations: Geo.=Geometric Shapes, Bool.=Boolean Expressions, Shuf.=Shuffled Objects, M.Ar.=Multistep Arithmetic, Hyp.=Hyperbaton, WoL=Web of Lies, Dis.=Disambiguation QA, Word=Word Sorting, Brd.=Boardgame QA, T.Ar.=Time Arithmetic, Mov.=Movie Recommendation, Caus.=Causal Understanding. Appendix E Pass@k Analysis Following Chen et al. ( 2021 ) and Yue et al. ( 2025 ) , we analyze the pass@k curves of our task-specific models. Across 80+ RL curves, we observe that the spread and distinguishability of model performance increases at k=8, with maximum overlap at k=1. We show the pass@k curves in Figure 6 . Figure 6: Pass@k accuracy of task-specific models across various values of k. Appendix F Task Analysis We prompt an LLM (Claude-Opus-4.6) with the task descriptions from each task and generate coarse category labels. We find that Multi-hop Reasoning and Coreference resolution emerge as the strongest categories, while narrative and surface-formatting tasks (e.g., Story Coherence, Date/Temporal format) consistently underperform (Figure 7 ). However, these aggregate trends obscure variations at the task-level. Despite Textual Entailment & NLI ranking in the middle at the category-level, task738_perspectrum emerges as the top-ranked task by large margin. This highlights that coarse category labels are insufficient for task selection and effective data curation for RL should be driven by fine-grained task utility analysis. Figure 7: (a)We categorize the source tasks based on target reasoning skill and task type. We report Mean Pass@8 across each task category and highlight the categories which degrade baseline (Qwen3-0.6b) model on BBEH-mini. Appendix G Data Interventions Following Kazemi et al. ( 2025 ) , we design the given 7 interventions to improve data quality (Table 6 ) and prompt GPT-5-mini with G . Since, we want to preserve the ground-truth answer, we apply these interventions only to the problem statement. Finally, to ensure the that the final answer is preserved, filter the augmented data with based on win-rate computed again with the augmented problem statements. In our experiments, we combine the original data and the intervened data in a ratio of 1:1. Dimension Description Many-hop reasoning Add information that increases the number of reasoning steps needed to reach the answer. Going against strong prior Add context that creates a misleading prior belief which conflicts with the correct answer, tempting the model to answer incorrectly based on surface-level associations. Learning on the fly Introduce a new rule, definition, or convention within the problem that must be understood and applied to solve it. Long-context Pad the problem with additional (but non-answer-changing) context to increase overall length. Finding errors in reasoning traces Include a flawed reasoning chain within the problem that the model must recognize as incorrect. Inductive reasoning Provide a set of examples that establish a pattern, requiring the model to induce and apply the pattern. Constraint satisfaction Add extra constraints that the model must track, even though they do not affect the final answer. Compositional understanding Fuse an independent sub-problem into the main problem, requiring the model to separate and solve them independently. Knowledge-intensive reasoning Add domain-specific terminology or context that requires specialized knowledge to parse, even though it does not change the answer. Table 6: Following Kazemi et al. ( 2025 ) , we design these interventions to improve the data quality. We provide the interventions and their definitions here. Prompt for Data Intervention ⬇ 1 You are an expert data augmentation assistant . Your task is to take an existing ( problem , answer ) pair from an NLP dataset and inject a distractor into the problem . 2 The distractor must make the problem harder for an AI model to solve , but it must NOT change the ground - truth answer . 3 4 ## DISTRACTOR TYPE 5 You MUST use the following distractor type : 6 **{ distractor_name }**: { distractor_description } 7 8 ## RULES 9 1. ** Answer preservation ( CRITICAL )**: The ground - truth answer MUST remain exactly the same after distractor injection . Do not alter the core reasoning chain . 10 2. ** Naturalness **: The distractor must read naturally within the problem . It should not feel artificially inserted or out of place . 11 3. ** Plausibility **: The distractor should be plausible and contextually relevant enough that a model might be misled by it . 12 4. ** Minimal invasion **: Modify only what is necessary . Do not rewrite the entire problem . Inject the distractor into or around the existing text . 13 5. ** Difficulty calibration **: The distractor should make the problem meaningfully harder , not trivially so . Aim for a difficulty increase that would cause a mid - tier model to fail while a strong model would still succeed . 14 15 ## OUTPUT FORMAT 16 You MUST respond with a valid JSON object and nothing else . No markdown , no explanation outside the JSON . 17 Use the following schema : 18 {{ 19 " original_problem ": "", 20 " original_solution ": "", 21 " augmented_problem ": "", 22 " augmented_solution ": "", 23 24 " distractor_metadata ": {{ 25 " distractor_types_used ": [ 26 {{ 27 " name ": "", 28 " description ": "" 29 }} 30 ], 31 " injected_text_summary ": "", 32 " why_answer_unchanged ": "", 33 " estimated_difficulty_increase ": "" 34 }} 35 }} 36 37 ## IMPORTANT GUIDELINES 38 - Think step by step before generating the output . 39 - First , understand what the problem is asking and why the given answer is correct . 40 - Second , identify which parts of the problem can be augmented without breaking the answer . 41 - Third , apply the specified distractor type as naturally as possible . 42 - Fourth , draft the distractor text . 43 - Fifth , verify that the answer is still correct with the distractor in place . 44 - Only then produce the final JSON output . 45 46 ## FINAL CHECKLIST ( verify before outputting ) 47 - [ ] Is the output valid JSON ? 48 - [ ] Is the answer in augmented_problem identical to the original answer ? 49 - [ ] Does the distractor read naturally in context ? 50 - [ ] Is the specified distractor type used with a clear description ? 51 - [ ] Is the why_answer_unchanged field filled with a logical explanation ? 52 - [ ] Would the augmented problem genuinely be harder for a model to solve ? 53 54 Now , process the following input and return the JSON output : 55 56 Problem : Statement : { problem } 57 Solution : { solution } 58 """ Appendix H Similarity and Task Difficulty We find that semantic similarity and lexical similarity between the tasks and validation benchmark are poor predictors of task utility for RLVR. As shown in Fig. 8 , both measures exhibit weak correlation with model performance on BBEH. While these approaches are attractive because they are cheap, fast to implement and model agnostic, our findings suggest that surface similarity is insufficient for task selection. We also investigated whether task difficulty, measured by the average win-rate of the base model, predicts downstream reasoning performance in Fig. 8 . Similar to surface similarity, we observe only a weak correlation between task difficulty and model performance on BBEH, indicating that task difficulty is also a poor predictor of task utility for RLVR. Overall, our findings underscore that effective task selection relies on controlled, iterative, and compute-matched RL training. Figure 8: (Left) Correlation between semantic similarity and task performance. (Middle) Correlation between lexical similarity and task performance. (Right) Correlation between win rate and task performance. We observe weak correlation for all three approaches with downstream task performance on BBEH-mini. Appendix I Micro Mixing We provide the top tasks ranked per sub-task in Table LABEL:tab:top5_per_bbeh . For Micro-Top1, 16 unique tasks are selected while 31 unique tasks are included in Micro-Top2. Additionally, we show the distribution of reasoning skills as categorized in § F in SuperNova which is scaled from Micro-Top2 and comprises 31 unique tasks. Figure 9: Distribution of reasoning skills in SuperNova . Table 7: Top-5 training tasks per BBEH task. BBEH Task Rank Task ID Training Task movie recommendation 1 task827 copa_commonsense_reasoning 2 task069 abductivenli_classification 3 task212 logic2text_classification 4 task1297 qasc_question_answering 5 task1209 atomic_classification_objectuse word sorting 1 task828 copa_commonsense_cause_effect 2 task1548 wiqa_binary_classification 3 task1385 anli_r1_entailment 4 task835 mathdataset_answer_generation 5 task383 matres_classification object counting 1 task1210 atomic_classification_madeupof 2 task1211 atomic_classification_hassubevent 3 task1155 bard_analogical_reasoning_trash_or_treasure 4 task827 copa_commonsense_reasoning 5 task004 mctaco_answer_generation_event_duration geometric shapes 1 task249 enhanced_wsc_pronoun_disambiguation 2 task1209 atomic_classification_objectuse 3 task1385 anli_r1_entailment 4 task697 mmmlu_answer_generation_formal_logic 5 task717 mmmlu_answer_generation_logical_fallacies nycc 1 task1297 qasc_question_answering 2 task073 commonsenseqa_answer_generation 3 task212 logic2text_classification 4 task213 rocstories_correct_ending_classification 5 task828 copa_commonsense_cause_effect boardgame qa 1 task004 mctaco_answer_generation_event_duration 2 task116 com2sense_commonsense_reasoning 3 task062 bigbench_repeat_copy_logic 4 task1726 mathqa_correct_answer_generation 5 task1387 anli_r3_entailment buggy tables 1 task007 mctaco_answer_generation_transient_stationary 2 task1390 wscfixed_coreference 3 task600 find_the_longest_common_substring_in_two_strings 4 task391 causal_relationship 5 task004 mctaco_answer_generation_event_duration linguini 1 task004 mctaco_answer_generation_event_duration 2 task1209 atomic_classification_objectuse 3 task640 esnli_classification 4 task085 unnatural_addsub_arithmetic 5 task738 perspectrum_classification boolean expressions 1 task850 synthetic_longest_palindrome 2 task600 find_the_longest_common_substring_in_two_strings 3 task1390 wscfixed_coreference 4 task018 mctaco_temporal_reasoning_presence 5 task210 logic2text_structured_text_generation multistep arithmetic 1 task004 mctaco_answer_generation_event_duration 2 task1210 atomic_classification_madeupof 3 task1211 atomic_classification_hassubevent 4 task007 mctaco_answer_generation_transient_stationary 5 task1390 wscfixed_coreference time arithmetic 1 task212 logic2text_classification 2 task835 mathdataset_answer_generation 3 task383 matres_classification 4 task1209 atomic_classification_objectuse 5 task1153 bard_analogical_reasoning_affordance object properties 1 task828 copa_commonsense_cause_effect 2 task004 mctaco_answer_generation_event_duration 3 task1210 atomic_classification_madeupof 4 task1211 atomic_classification_hassubevent 5 task007 mctaco_answer_generation_transient_stationary hyperbaton 1 task249 enhanced_wsc_pronoun_disambiguation 2 task213 rocstories_correct_ending_classification 3 task393 plausible_result_generation 4 task600 find_the_longest_common_substring_in_two_strings 5 task827 copa_commonsense_reasoning sarc triples 1 task210 logic2text_structured_text_generation 2 task640 esnli_classification 3 task970 sherliic_causal_relationship 4 task850 synthetic_longest_palindrome 5 task717 mmmlu_answer_generation_logical_fallacies zebra puzzles 1 task828 copa_commonsense_cause_effect 2 task863 asdiv_multiop_question_answering 3 task210 logic2text_structured_text_generation 4 task1726 mathqa_correct_answer_generation 5 task738 perspectrum_classification spatial reasoning 1 task738 perspectrum_classification 2 task087 new_operator_addsub_arithmetic 3 task019 mctaco_temporal_reasoning_category 4 task080 piqa_answer_generation 5 task697 mmmlu_answer_generation_formal_logic shuffled objects 1 task249 enhanced_wsc_pronoun_disambiguation 2 task018 mctaco_temporal_reasoning_presence 3 task827 copa_commonsense_reasoning 4 task1297 qasc_question_answering 5 task717 mmmlu_answer_generation_logical_fallacies temporal sequence 1 task004 mctaco_answer_generation_event_duration 2 task1210 atomic_classification_madeupof 3 task1211 atomic_classification_hassubevent 4 task007 mctaco_answer_generation_transient_stationary 5 task1390 wscfixed_coreference sportqa 1 task270 csrg_counterfactual_context_generation 2 task210 logic2text_structured_text_generation 3 task062 bigbench_repeat_copy_logic 4 task600 find_the_longest_common_substring_in_two_strings 5 task391 causal_relationship web of lies 1 task1152 bard_analogical_reasoning_causation 2 task828 copa_commonsense_cause_effect 3 task1211 atomic_classification_hassubevent 4 task080 piqa_answer_generation 5 task640 esnli_classification causal understanding 1 task383 matres_classification 2 task004 mctaco_answer_generation_event_duration 3 task1390 wscfixed_coreference 4 task828 copa_commonsense_cause_effect 5 task291 semeval_2020_task4_commonsense_validation disambiguation qa 1 task697 mmmlu_answer_generation_formal_logic 2 task1296 wiki_hop_question_answering 3 task717 mmmlu_answer_generation_logical_fallacies 4 task018 mctaco_temporal_reasoning_presence 5 task065 timetravel_consistent_sentence_classification dyck languages 1 task1390 wscfixed_coreference 2 task1386 anli_r2_entailment 3 task004 mctaco_answer_generation_event_duration 4 task1210 atomic_classification_madeupof 5 task1152 bard_analogical_reasoning_causation Appendix J Additional Details about SuperNI tasks We provide the task descriptions of the 83 tasks that we include in our candidate pool in Table LABEL:app_tab:superni . These tasks are arranged in the order of maximum performance on BBEH-mini. Table 8: Task descriptions. Task Name Summary task738 perspectrum classification Decide whether the given perspective supports or undermines the given claim. task003 mctaco question generation event duration Writing questions that involve commonsense understanding of “event duration”. task717 mmmlu answer generation logical fallacies Answering multiple choice questions on logical fallacies. task249 enhanced wsc pronoun disambiguation Given a sentence and a pronoun, decide which one of the choices the pronoun is referring to. task1385 anli r1 entailment Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. task1296 wiki hop question answering Given a subject, a relation, and a context, find the object with that relation to the subject. task828 copa commonsense cause effect Given a pair of sentences, judge whether the second sentence is the cause or effect of the first one. task073 commonsenseqa answer generation Answer questions based on commonsense knowledge. task018 mctaco temporal reasoning presence Checking the presence of temporal reasoning in a question. task697 mmmlu answer generation formal logic Answering multiple choice questions on formal logic. task827 copa commonsense reasoning Given a premise and two alternatives, select the alternative that more plausibly has a causal relation with the premise. task383 matres classification Given a context and a verb, answer if the given verb can be anchored in time or not. task065 timetravel consistent sentence classification Choosing the option that makes a given short story consistent. task640 esnli classification Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. task1387 anli r3 entailment Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. task863 asdiv multiop question answering Given a mathematical question involving multiple operations, find the most suitable numerical answer. task1209 atomic classification objectuse Given a tuple, determine whether the Head is used for the Tail or not. task212 logic2text classification Given a command, classify the command in one of seven logic types. task750 aqua multiple choice answering Given a mathematical question, find the most suitable numerical answer. task010 mctaco answer generation event ordering Answering questions that involve commonsense understanding of event ordering. task1297 qasc question answering Given two facts and a multiple-choice question, answer the question. task007 mctaco answer generation transient stationary Answering questions that involve commonsense understanding of transient vs. stationary events. task1390 wscfixed coreference Given a context, a pronoun, and a noun, determine if the pronoun in the context refers to the noun or not. task600 find the longest common substring in two strings Given two strings return the longest common substring in those two strings. task080 piqa answer generation Generate a solution to a goal regarding physical knowledge about the world. task1726 mathqa correct answer generation Generate correct answers for math questions. task835 mathdataset answer generation Find the numerical answer for a math word problem. task580 socialiqa answer generation Given a context, a question and three options, provide the correct answer based on the context. task1393 superglue copa text completion Given a premise sentence, two possible options and a question word, choose the best option. task1727 wiqa what is the effect Find the effect of an event on another event, based on an introduced process. task170 hotpotqa answer generation Given a set of context and supporting facts, answer the question asked. task133 winowhy reason plausibility detection Detect if a reason that explains an answer to a pronoun coreference resolution question is correct or not. task004 mctaco answer generation event duration Answering questions that involve commonsense understanding of event duration. task019 mctaco temporal reasoning category Verifying the temporal reasoning category of a given question. task229 arc answer generation hard Given a hard science question, provide the answer based on scientific facts and reasoning. task106 scruples ethical judgment Given two actions choose the one that is considered less ethical. task178 quartz question answering Given a question, select the correct answer from the given options using an explanation. task1152 bard analogical reasoning causation Given an analogy that relates actions with their consequences, give the appropriate consequence of the given action. task090 equation learner algebra Answer the given equation. task850 synthetic longest palindrome Given a string find the longest substring that is a palindrome. task1422 mathqa physics Given a problem on physics and options to choose from, find the correct option that answers the problem. task393 plausible result generation Given a sentence, write another sentence that is a likely result of it. task085 unnatural addsub arithmetic Performing arithmetic with swapped operator symbols. task1529 scitail1.1 classification Determining if there is entailment between hypothesis and premise. task867 mawps multiop question answering Given a mathematical question involving multiple operations, find the most suitable numerical answer. task211 logic2text classification Given a command and corresponding interpretation, classify whether it is the right interpretation or not. task1548 wiqa binary classification Binary classification based on steps in wiqa. task966 ruletaker fact checking based on given context Fact checking based on given context. task935 defeasible nli atomic classification Given a premise, hypothesis and an update, identify whether the update strengthens or weakens the hypothesis. task116 com2sense commonsense reasoning Decide whether a sentence is plausible and matches commonsense. task087 new operator addsub arithmetic Performing arithmetic with newly defined operator symbols. task206 collatz conjecture Given a list of integers, compute the next number in the 3n+1 problem. task970 sherliic causal relationship Determine if A and B share a causal relationship. task086 translated symbol arithmetic Performing arithmetic with translated operator symbols. task270 csrg counterfactual context generation Given a premise, initial context with ending, and new counterfactual ending, generate counterfactual context which supports the new story ending. task392 inverse causal relationship Given two sentences, decide whether the first sentence can be the result of the second one. task105 story cloze-rocstories sentence generation Given four sentences, predict the next coherent sentence. task1507 boolean temporal reasoning Given a statement about date and time values, deduce whether it is true or false. task1404 date conversion Given a date in a particular format, convert it into some other format. task1153 bard analogical reasoning affordance Given an analogy that signifies affordances, give the appropriate affordance of the given action. task069 abductivenli classification Choosing text that completes a story based on given beginning and ending. task062 bigbench repeat copy logic Generating text that follows simple logical operations such as repeat, before, after etc. task1088 array of products Given an integer array, return an array such that its element at each location is equal to the product of elements at every other location in the input array. task190 snli classification Given two sentences choose whether they agree, disagree, or neither with each other. task1333 check validity date ddmmyyyy Given a date in dd/mm/yyyy format, check if it is a valid date or not. task016 mctaco answer generation frequency Answering questions that involve commonsense understanding of event frequency. task1208 atomic classification xreason Given a tuple, determine whether the Tail is the reason for the Head or not. task1386 anli r2 entailment Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. task1516 imppres naturallanguageinference Classify a given premise and hypothesis pair. task199 mnli classification Given 2 sentences, determine if they clearly agree or disagree with each other or if they cannot be answered. task1210 atomic classification madeupof Given a tuple, determine whether the Head is made of the Tail or not. task217 rocstories ordering answer generation Given a five sentence story in shuffled order and the title, put the story in the correct order. task1155 bard analogical reasoning trash or treasure Given an analogy that relates items to whether they are trash or treasure, determine if the given item is trash or treasure. task218 rocstories swap order answer generation Given a five sentence story and the title, determine which two sentences must be swapped so that the story makes complete sense. task1211 atomic classification hassubevent Given a tuple, determine whether the Head includes an event or an action in the Tail or not. task213 rocstories correct ending classification Given the title and the first four sentences of a five sentence story, choose the correct story ending. BETA