--- title: "Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation" authors: ["Ivan Vulić Anna Korhonen", "Language Technology Lab, University of Cambridge", "Collection", "ljvmiranda921/polyglot-teachers", "Code"] url: "https://arxiv.org/abs/2604.11290" sections: 77 estimated_tokens: "41.3k" --- ## Contents - 1 Introduction - 2 Evaluating Language Models as Multilingual Teachers - 2.1 Creating the seed dataset - 2.2 Multilingual Data Quality & Diversity - Synthetic data generation - Data quality and diversity metrics - 2.3 Student Model Performance - 2.4 Computing Polyglot Score - 3 Experiments: Evaluating LMs and PG-Score Generalization - 3.1 Which State-of-the-Art LMs Are Good Multilingual Teachers? - Setup - Teacher Models - Target Languages - Results - 3.2 Generalization of PG-Score Across Different Base Models - Setup - Results - 3.3 Effect of Synthetic Data Generation Method on PG-Score - Setup - Results - 4 Analysis: What Makes a Good Polyglot Teacher? - 4.1 Do stronger models make better teachers? - Setup - Results - 4.2 Which intrinsic metrics determine extrinsic student model performance? - Setup - Results - 5 Discussion: Towards a Recipe for Multilingual Synthetic Data Generation - 6 Related Work - Synthetic Data Generation for Multilingual SFT - Evaluating and Improving the Synthetic Data Pipeline - 7 Conclusion - Limitations - Ethics Statement - Acknowledgments - References - Appendix - Appendix A Multilingual Synthetic Data Generation - Appendix B Seed Dataset Statistics - Appendix C The Polyglot Collection - Appendix D Teacher Model and Target Language Details - Appendix E Experimental Details - E.1 Supervised Finetuning - E.2 Model Evaluation - Appendix F Full Results for Intr. and Extr. Metrics - Percentage Increase Tables - Appendix G Additional Experiments and Ablations - G.1 Effect of Data Scale on Student Model Performance - Setup - Results - G.2 Generalization Across Model Size - Setup - Results - G.3 Effect of Translation Method (Prompting an LM vs. Translation Model) - Setup - Results - G.4 Weighing of Intrinsic and Extrinsic Metrics in PG-Score - Setup - Results - G.5 Effect of language resource levels on PG-Score - Setup - Results - Appendix H Disclosure on the Use of LLMs - Appendix I Multilingual Synthetic Data Recipe: Case Study on Tagalog - I.1 Setup: Recipe Design and Evaluation - Data - Evaluation - I.2 Results: Leaderboard Scores and Ablations - I.3 Analysis: Ablation Experiments - Curation of publicly-available data vs. Synthetic data generation - Using a teacher with a higher PG-Score - Matching teacher and student model families - Increase data scale - Increase model scale - Appendix J Inference Details - Prompt templates - Inference settings ## Abstract Abstract Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score ; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development. ## 1 Introduction Figure: Figure 1: Overview of our method for evaluating language models as multilingual teachers (Polyglot Score). We evaluate teacher models on their synthetic data generation capabilities across three methods: Generate a prompt-response pair given few-shot examples, Translate prompts from English and generate a response, and Respond to a prompt in the target language. The Polyglot Score incorporates both intrinsic data quality metrics and extrinsic student model performance to assess the effectiveness of a teacher model for a target language. Refer to caption: 2604.11290v1/x5.png Supervised finetuning (SFT, Ouyang et al., 2022) has emerged as a standard approach for adapting language models (LMs) to specific target languages (Zhang et al., 2025b; Aryabumi et al., 2024, inter alia). Central to the success of SFT is the availability of high-quality training data, consisting of pairs of user prompts and a corresponding response, which is often scarce for less-resourced languages (Kunchukuttan et al., 2025). Generating prompt-response pairs for these languages demands substantial human effort (Singh et al., 2024; Kapania et al., 2025), creating a bottleneck for language-specific model development. To alleviate the challenge of human effort and data scarcity, synthetic data generation using LMs has gained traction as a promising solution for multilingual LM development (Cahyawijaya et al., 2024; Ng et al., 2025; Martins et al., 2025; Hammoud et al., 2026, inter alia). This approach involves leveraging a typically larger teacher model to generate training examples, which are then used to finetune a smaller student model to replicate the knowledge of the teacher (Kim and Rush, 2016). However, existing works often select teacher models arbitrarily, defaulting to the largest state-of-the-art models that excel on benchmarks (Xu et al., 2025b; Li et al., 2025; Zhang et al., 2025a). This practice is problematic because these models, despite strong performance, may have significant capability gaps in non-English languages, leading to poor-quality synthetic data that propagates the teacher’s weaknesses rather than its strengths. And so we ask: “what makes an effective multilingual teacher for synthetic data generation, and how can we systematically measure it?” In this work, we conduct a comprehensive analysis of 10 LMs across 6 typologically diverse languages on three common synthetic data generation methods: responding to a user query or instruction, translating prompts from English to a target language, and generating prompt-response pairs given in-context examples (§[2.2](#S2.SS2)). To systematically assess teacher model effectiveness, we evaluate LMs using both intrinsic measures of data quality (§[2.2](#S2.SS2), i.e., the diversity of prompts and responses, the perplexity of the base model on the response, and response quality based on a multilingual reward model) and an extrinsic measure of student model performance on multilingual tasks (§[2.3](#S2.SS3), cultural understanding, mathematical reasoning, general chat). We aggregate these measurements into a single metric called Polyglot Score (PG-Score), in order to provide a holistic assessment of a teacher model’s data generation capabilities. Our contributions are as follows: - • We close the evaluation gap by evaluating 10 teacher models, generating over 1.4M SFT examples and finetuning 240 student models from OLMo 3 7B. We find that Gemma 3 27B consistently ranks within the top three highest PG-Score and that the Gemma 3 model family outperforms other families such as Llama 3.1 and IBM Granite (§[3.1](#S3.SS1)). Our PG-Score rankings are consistent across other base model families (Llama 3.1 8B, Qwen 3 8B, Gemma 3 4B, §[3.2](#S3.SS2)). - • We provide analyses and insights on the characteristics of a good multilingual teacher model. Our analyses reveal that model scale and benchmark performance, which are common assumptions of a “strong” model, do not significantly predict teacher effectiveness (§[4.1](#S4.SS1)). Instead, we find that qualities of the generated data, namely prompt diversity and length coupled with fluent and diverse responses, capture over 93.3% of the variance in intrinsic data quality metrics, and their principal components predict student performance with $R^{2}$=0.664 (§[4.2](#S4.SS2)). - • Based on these findings, we recommend a recipe (§[5](#S5)) for generating multilingual synthetic data. For example, we find that matching the model families of the teacher and student is a reliable heuristic for choosing a teacher model (§[3.2](#S3.SS2)), and generating responses to existing prompts or translating from English can yield substantial improvements on less-resourced languages compared to a random mix of data generation methods, though gains vary by teacher model (§[3.3](#S3.SS3)).(^1^11As a supplementary, we show that our recipe improves performance on a held-out language (Tagalog) on a language-specific benchmark ([Appendix I](#A9)).) We hope that this work paves the way for developing inclusive and equitable language technologies through quality and cost-effective data. We release our code, data, and models to drive research in multilingual synthetic data generation. ## 2 Evaluating Language Models as Multilingual Teachers The Polyglot Score ([Figure 1](#S1.F1)) of a teacher model $T$ for a target language $\ell$ is based on the (1) intrinsic quality of the synthetic data generated by the teacher (§[2.2](#S2.SS2)) and the (2) extrinsic performance of a student model $S$ finetuned on this data (§[2.3](#S2.SS3)). ### 2.1 Creating the seed dataset In order to bootstrap the synthetic data generation process, we create a seed dataset $\mathcal{D}_{\text{seed},\ell}$ for each target language $\ell$. We create $\mathcal{D}_{\text{seed},\ell}$ by aggregating publicly available multilingual instruction-tuning datasets, including the Aya Collection (Aryabumi et al., 2024), WildChat 4.8-M (Zhao et al., 2024), EuroBlocks-SFT (Martins et al., 2025), and Magpie-Align (Xu et al., 2025a). In order to simulate scenarios where English prompts are translated into a target language, we also include examples from Tülu 3 SFT (Lambert et al., 2025), Helpsteer3 (chosen responses, Wang et al., 2025), and GSM8K (train split, Cobbe et al., 2021). Detailed seed dataset statistics in [Appendix B](#A2). ### 2.2 Multilingual Data Quality & Diversity #### Synthetic data generation Given a teacher model $T$, target language $\ell$, and a seed dataset for language $\ell$, $\mathcal{D}_{\text{seed},\ell}$, we distill a synthetic dataset $\mathcal{D}_{T,\ell}=\{(x_{i},y_{i})\}_{i=1}^{N}$ consisting of $N$ prompt-response pairs $(x_{i},y_{i})$. We consider three synthetic data generation methods found in literature: - • Generate: we sample $k$ prompt-response pairs from $\mathcal{D}_{\text{seed},\ell}$ as few-shot examples and use $T$ to generate a new pair $(x_{i},y_{i})$ conditioned on these examples. - • Translate: we forward-translate English prompts from $\mathcal{D}_{\text{seed},\ell}$ to the target language $\ell$ to obtain $x_{i}$, and use $T$ to generate the corresponding response $y_{i}$. - • Respond: we take a prompt $x_{i}$ from $\mathcal{D}_{\text{seed},\ell}$ and use $T$ to generate the response $y_{i}$. We provide a brief review of multilingual synthetic data generation methods in §[6](#S6) and a supplementary survey in [Appendix A](#A1). #### Data quality and diversity metrics Synthetic data is valuable when it is both high-quality and diverse (Raventos et al., 2023; Chen et al., 2024; Zhu et al., 2025).(^2^22We use “data quality” to refer to both aspects hereafter.) To estimate the value of $\mathcal{D}_{T,\ell}$, we compute a set of lexical and model-based metrics: - • Diversity of prompts and responses $(d_{x},d_{y})$: a corpus-level statistic that computes the cosine distance of the prompt and response embeddings. In practice, we use Llama-Embed-Nemotron-8B (Babakhin et al., 2025), the top-performing model on the MMTEB leaderboard (Enevoldsen et al., 2025), to embed the texts. - • Perplexity (PPL): the perplexity of a base model on the response $y_{i}$ conditioned on the prompt $x_{i}$, measuring the fluency and naturalness of the generated text. Lower perplexity indicates more coherent and linguistically natural responses. - • Reward score of a multilingual reward model (R): the verbalized score (1-5) of a multilingual reward model based on rubrics relating to fluency, naturalness, and instruction-following. In practice, we prompt M-Prometheus 14B (Pombal et al., 2025) as an LM judge to score the quality of the prompt-response pair ([Figure 13](#A10.F13)). We choose M-Prometheus because of its high performance on human-aligned evaluation benchmarks, suggesting that the reward model aligns well with native speakers. We combine these intrinsic metrics by scaling each metric using z-score normalization and averaging them as shown in [Equation 1](#S2.E1). $$ $\begin{split}\text{Intrinsic}_{T,\ell}&=\frac{1}{|M|}\sum_{m\in M}\text{z-score}(m(\mathcal{D}_{T,\ell}))\\ \text{where }M&=\{d_{x},d_{y},-\log(1+\text{PPL}),R\}\end{split}$ (1) $$ ### 2.3 Student Model Performance We perform supervised finetuning of a base model $S_{\phi}$ on the synthetic dataset $\mathcal{D}_{T,\ell}$ to obtain a student model $S_{T,\ell}$. Then, we evaluate $\mathcal{S}_{T,\ell}$ on a suite of multilingual tasks to assess how well the student has learned from the teacher. These tasks include: - • Cultural and factual understanding (Culture): we evaluate on Global-MMLU Lite (Singh et al., 2025), containing culturally diverse and relevant questions that were localized by native speakers from English (Hendrycks et al., 2021). - • General chat (Chat): we evaluate on M-RewardBench (Gureja et al., 2025) which measures the alignment of models with human preferences in conversational settings. - • Mathematical reasoning (Math): we evaluate on M-GSM (Shi et al., 2023), a multilingual version of the GSM8K dataset (Cobbe et al., 2021) that tests the model’s ability to solve mathematical word problems. Inspired by Kim et al. (2025), we compute the Performance Gap Recovered (PGR) that measures the improvement of $S_{T,\ell}$ over a base model $S_{\phi}$ on a benchmark $b$ relative to a reference model $S_{\text{REF}}$ ([Equation 2](#S2.E2)). $$ $\begin{split}\text{Extrinsic}_{T,\ell}&=\frac{1}{|B|}\sum_{b\in B}\dfrac{\text{score}_{b}(S_{T,\ell})-\text{score}_{b}(S_{\phi})}{\text{score}_{b}(S_{\text{REF}})-\text{score}_{b}(S_{\phi})}\\ \text{where }B&=\{\textsc{Culture},\textsc{Chat},\textsc{Math}\}\end{split}$ (2) $$ **Table 1: Top models with the highest PG-Score (average across six languages). We evaluate teacher models with varying size and model family on 6 typologically-diverse languages. For each language, we highlight the best model in bold and the second-best model with an underline. Detailed results with standard errors are in [Table 13](#A6.T13).** | Teacher Model | Average | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) | | --- | --- | --- | --- | --- | --- | --- | --- | | Gemma 3 27B Inst. | 0.726 | 0.145 | 0.360 | 1.655 | 1.358 | 0.214 | 0.626 | | Aya Expanse 32B | 0.706 | -0.058 | 0.222 | 1.468 | 1.129 | 1.153 | 0.320 | | Gemma 3 12B Inst. | 0.595 | -0.464 | 0.327 | 1.756 | 1.228 | 0.151 | 0.573 | | Command A | 0.546 | -1.360 | 0.114 | 1.673 | 1.102 | 1.063 | 0.683 | | Gemma 3 4B Inst. | 0.469 | -0.488 | 0.330 | 1.644 | 0.929 | -0.105 | 0.504 | | GPT 4o mini | 0.461 | -1.117 | 0.015 | 1.766 | 0.908 | 1.003 | 0.189 | | IBM Granite 4.0 | 0.312 | -0.072 | -0.031 | 1.000 | 0.734 | -0.079 | 0.321 | | IBM Granite Micro | 0.304 | -0.282 | 0.290 | 1.102 | 0.783 | -0.329 | 0.264 | | Llama 3.1 70B Inst. | 0.140 | -0.964 | 0.109 | 1.195 | 0.688 | 0.182 | -0.373 | | Llama 3.1 8B Inst. | -0.356 | -1.693 | -0.974 | 0.891 | 0.182 | 0.322 | -0.863 | ### 2.4 Computing Polyglot Score To provide straightforward comparisons between teacher models, PG-Score reports a single score that combines both extrinsic and intrinsic metrics as shown in [Equation 3](#S2.E3). $$ $\text{{PG-Score}{}}_{T,\ell}=\text{z-score}(\text{Intr.}_{T,\ell}+\text{Extr.}_{T,\ell})\\$ (3) $$ We combine both intrinsic and extrinsic metrics because they capture complementary aspects of teacher quality. Extrinsic metrics alone may overlook the quality of synthetic data that propagates through the ecosystem, while intrinsic metrics alone do not guarantee that the student model achieves strong downstream performance. The resulting PG-Score is z-score normalized, where 0 indicates average teacher effectiveness, and higher scores indicate better synthetic data quality and student performance for that language. We adopt equal weighting as a baseline; we show that teacher rankings are robust to alternative weighting schemes in [Appendix G.4](#A7.SS4). ## 3 Experiments: Evaluating LMs and PG-Score Generalization In this section, we measure the Polyglot Score of state-of-the-art LMs (§[3.1](#S3.SS1)). Then, we test whether our findings are consistent across other base models (§[3.2](#S3.SS2)). Finally, we determine if a certain data generation method is more effective in multilingual settings (§[3.3](#S3.SS3)). We conduct additional experiments and ablations in [Appendix G](#A7). ### 3.1 Which State-of-the-Art LMs Are Good Multilingual Teachers? #### Setup In order to evaluate the effectiveness of different LMs as multilingual teachers, we select 10 state-of-the-art models that vary in scale, architecture, and training data, then evaluate them on 6 typologically diverse languages by generating 10.5k prompt-response pairs for each teacher-language pair where each data generation (§[2.2](#S2.SS2)) method is equally represented. We repeat the data generation process three times with different random seeds to account for variability in LM outputs. Then, we finetune a pretrained OLMo 3 7B model (OLMo Team et al., 2025) on each $\mathcal{D}_{T,\ell}$ to obtain $S_{T,\ell}$. [Appendix E.1](#A5.SS1) describes SFT information. #### Teacher Models We include Llama 3.1 (8B, 70B, Grattafiori et al., 2024), Gemma 3 (4B, 12B, 27B, Gemma Team et al., 2025), Command A (Cohere Team et al., 2025), Aya Expanse 32B (Dang et al., 2024), and IBM Granite (4.0, Micro, Granite Team, IBM, 2025). In addition, we also include GPT 4o mini (OpenAI et al., 2024) as a representative closed-source model. See [Table 7](#A4.T7) in [Appendix D](#A4) for detailed model information. #### Target Languages We select 6 typologically diverse languages: Arabic (ar), Czech (cs), German (de), Spanish (es), Indonesian (id), and Japanese (ja). These languages are chosen due to their variation in resource availability, script, and family. This language choice is also supported by prior work on informed sampling (Ploeger et al., 2026) that considers typological variety of the chosen languages. See [Table 8](#A4.T8) in [Appendix D](#A4) for language statistics. Figure: Figure 2: PG-Score across different base models (average across Arabic, German, and Indonesian). Left: Average PG-Score of each teacher model on students finetuned on three different base models. We highlight the top, second, and third best teacher models for each setting. Right: Heatmap showing Spearman rank correlation $\rho$ of teacher model rankings across base models. We show percentage increases in PG-Score on [Table 14](#A6.T14). Refer to caption: 2604.11290v1/x6.png **Table 2: PG-Score across three data generation methods: Generate, Translate, and Respond (§[2.2](#S2.SS2)). For each data generation method, we generate 10k samples per teacher-language pair and finetune a student model on OLMo 3 7B. We show percentage increases in PG-Score compared to a baseline (equal representation of the three data generation methods) on [Table 15](#A6.T15).** | | Arabic (ar) | German (de) | Indonesian (id) | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Teacher Model | Generate | Translate | Respond | Generate | Translate | Respond | Generate | Translate | Respond | | Gemma 3 27B Inst. | 0.032 | 0.276 | 0.802 | 2.140 | 2.086 | 1.212 | 1.189 | 1.196 | 0.046 | | Aya Expanse 32B | $-$0.276 | 0.148 | $-$1.349 | 1.473 | 1.255 | 1.451 | 0.039 | 0.733 | 1.606 | | Llama 3.1 70B Inst. | $-$0.867 | $-$1.025 | $\mathbf{-}$0.215 | 1.391 | 0.459 | 1.187 | $-$0.146 | 0.089 | 0.155 | #### Results [Table 1](#S2.T1) shows the PG-Score of each teacher model across all target languages. The results suggest the following: - • Gemma 3 27B and Aya Expanse 32B are the most effective teachers. Gemma 3 27B achieves the highest average PG-Score (0.726), followed closely by Aya Expanse 32B (0.706), both outperforming larger models like Llama 3.1 70B Inst. (0.140), suggesting that model scale alone does not determine teacher effectiveness. We also observe that the Gemma 3 family dominates the top ranks, while the Llama 3.1 family underperforms on most languages. - • Smaller LMs can be effective multilingual teachers. Gemma 3 12B (0.595) and 4B (0.469) rank among the top-5 teachers, while the Llama 3.1 70B Inst. (0.140) ranks ninth, suggesting that smaller LMs can match or exceed larger LMs in data generation capabilities. - • Teacher performance varies significantly by language. German and Spanish consistently show the highest scores across all models, while Arabic proves challenging with most teachers yielding negative scores, suggesting that language-specific factors influence teacher effectiveness. We hypothesize that a language’s resource status or presence in pretraining data may contribute to this variability (§[G.5](#A7.SS5)). ### 3.2 Generalization of PG-Score Across Different Base Models #### Setup Instead of using OLMo 3 7B as the base model ($S_{\phi}$) for student finetuning, we use (1) Llama 3.1 8B, (2) Gemma 3 4B PT, and (3) Qwen 3 8B Base (Yang et al., 2025). We recompute $S_{\phi}$-dependent metrics such as perplexity and PGR. To reduce computational costs, we focus on three languages: German (high PG-Score), Indonesian (mid-range), and Arabic (low PG-Score). #### Results [Figure 2](#S3.F2) shows the average PG-Score of each teacher model across different base models while [Table 14](#A6.T14) shows the percentage increase of family-matched teacher-student pairs compared to the OLMo 3 7B (mismatch) baseline. We observe that the best teacher models remain consistent across different student base models, with Gemma 3 27B and Aya Expanse 32B consistently ranking among the top three teachers. Furthermore, the Gemma 3 family continues to outperform other model families. In addition, we find that the model rankings vary slightly depending on the base model used, as Spearman rank correlation ranges from $\rho$=0.57 (moderate) to $\rho$=0.87 (strong). We hypothesize that this variation may be due to differences in architecture and pretraining data between base models. Despite this variation, we observe that teacher-student model family alignment is a reliable heuristic for achieving good PG-Score. For example, Gemma 3 teachers consistently perform well with Gemma 3 student bases, with family-matched pairs achieving at least +20.5% higher PG-Score compared to the worst pair (see [Table 14](#A6.T14)). This finding is interesting but reasonable given that models from the same family likely share similar tokenization schemes, leading to easier transfer from teacher to student. In addition, family-matching is not a hard constraint unlike in other distillation settings (on-policy, Agarwal et al., 2024; Boizard et al., 2025), but it remains a reliable heuristic for teacher selection when the optimal teacher is unknown. For our core experiment, we use OLMo 3 7B as the base model for finetuning to control the effect of model family alignment when evaluating teacher quality. **Table 3: Results from a mixed-effects regression model on PG-Score on an LM’s (a) size and (b) avg. multilingual benchmark performance. The lack of significant correlation suggests that both predictors are not solely sufficient to ensure teacher effectiveness.** | Predictor | $\mathbf{\beta}$ | SE | p | | --- | --- | --- | --- | | $\log\left(\text{Param. Size}\right)$ | 0.053 | 0.080 | 0.507 | | Avg. Multilingual Perf. | 1.387 | 2.204 | 0.529 | ### 3.3 Effect of Synthetic Data Generation Method on PG-Score #### Setup In order to determine if a data generation method is more effective than others, we generate 10k prompt-response pairs for each method in §[2.2](#S2.SS2) and compare the PG-Score of each mix. We recompute intrinsic data quality metrics and finetune OLMo 3 7B to obtain a student model and evaluate the teacher’s PG-Score. We also compare each mix against a baseline consisting of 10k instances with roughly equal number of samples ($\approx$3.3k) from each method. To reduce computational costs, we conduct this experiment on three representative teachers (Gemma 3 27B, Aya Expanse 32B, and Llama 3.1 70B) spanning high to low PG-Score, and three languages (German, Indonesian, Arabic) covering diverse resource levels. #### Results [Table 2](#S3.T2) shows the PG-Score of each data generation (see [Table 15](#A6.T15) for baseline comparisons). We observe that for a high-resource language like German, the Generate method yields the highest PG-Score, while for less-resourced languages like Arabic and Indonesian, the Respond or Translate methods are more effective. We hypothesize that this occurs because the Generate method depends on few-shot examples from the seed dataset, which are typically of higher quality in high-resource languages. Overall, our findings suggest that selecting a data generation method can have an impact on teacher effectiveness. In our core experiment, we sample an equal mix of all three methods (3.5k each) to control their effect when evaluating teacher model quality. ## 4 Analysis: What Makes a Good Polyglot Teacher? We investigate the factors that contribute to effective multilingual teachers. We start by analyzing common assumptions about teacher model performance, such as size and benchmark scores (§[4.1](#S4.SS1)), then determine which intrinsic factors drive student performance (§[4.2](#S4.SS2)). Lastly, we examine language properties that might influence a teacher’s PG-Score (§[G.5](#A7.SS5)). **Table 4: Variance explained by principal components from intrinsic data quality metrics. There are four principal components that explain over 93.3% (cumulative) of the variance.** | PC | Variance Expl. | Cumulative | | --- | --- | --- | | PC 1 | 42.2% | 42.2% | | PC 2 | 22.1% | 64.3% | | PC 3 | 16.5% | 80.8% | | PC 4 | 12.6% | 93.3% | | PC 5 | 3.5% | 96.8% | | PC 6 | 3.2% | 100.0% | ### 4.1 Do stronger models make better teachers? #### Setup In order to determine if there is a relationship between a model’s size or benchmark performance (i.e., common assumptions to assess a model’s “strength”) to its effectiveness as a multilingual teacher, we fit a mixed-effects model regressing PG-Score on (a) parameter size (N=27, 9 models, excluding GPT-4o-mini with unknown size $\times$ 3 trials), and (b) average multilingual benchmark performance on Global-MMLU Lite, M-GSM, and M-RewardBench (N=180, 10 models $\times$ 6 languages $\times$ 3 trials). #### Results [Table 3](#S3.T3) shows the regression results. We observe that neither parameter size nor average multilingual benchmark performance significantly predict PG-Score (p$>$0.05). Specifically, a 1-unit increase in $\log(\text{Param. Size})$ corresponds to a non-significant 0.053 increase in PG-Score. Although this finding confirms the results of Xu et al. (2025b) and Kim et al. (2025) for English-based tasks, we show that “stronger” models do not necessarily make better multilingual teachers. ### 4.2 Which intrinsic metrics determine extrinsic student model performance? Figure: Figure 3: Loading strength of intrinsic metrics on the principal components (PCs). PC1 suggests that good teachers produce diverse and high-quality responses, while PC2 focuses on prompt diversity and length. PC3 and PC4, together, indicates the importance of prompts on student performance. Refer to caption: 2604.11290v1/x7.png Figure: Figure 4: Fit of a linear regression model on the PCs of the intrinsic metrics to predict student performance. Intrinsic metrics, via their PCs, can predict extrinsic student performance ($R^{2}=0.664$ and $\text{RMSE}=0.440$) on multilingual benchmarks (§[2.3](#S2.SS3)). Refer to caption: 2604.11290v1/x8.png #### Setup In order to identify latent factors from the intrinsic metrics that explain student performance, we perform principal component analysis (PCA) on the intrinsic metrics described in §[2.2](#S2.SS2). Then, we fit a regression model to predict extrinsic student performance based on the principal components (PCs) obtained from PCA: we split 180 data points (10 models $\times$ 6 languages $\times$ 3 trials) into 80% train and 20% test, then train a linear regression model with the PCs as the features and the student performance as the target. #### Results [Table 4](#S4.T4) shows how much of the variance is explained by each principal component while [Figure 3](#S4.F3) shows the loading strength of each intrinsic metric on the principal components. We observe that the first four PCs explain over 93.3% of the variance in the intrinsic data quality metrics. Specifically, PC 1 (42.2%) captures characteristics such as lower response perplexity and high distinctiveness, PC2 (22.1%) captures variance in characteristics such as higher prompt diversity and length, whereas PC3 (16.5%) and PC4 (12.6%) capture variance that reinforce trends on prompt length and diversity. In addition, [Figure 4](#S4.F4) shows the fit of a linear model on the test set when the PCs learn to predict student performance. We observe that interactions within the intrinsic metrics can predict extrinsic student performance decently, with $R^{2}=0.664$ and $\text{RMSE}=0.440$. This finding suggests that even with a simple linear model, our chosen intrinsic metrics are predictive of student performance. In practice, these insights can help practitioners select teacher models based on intrinsic metrics alone, which are cheaper to compute than extrinsic student evaluations. ## 5 Discussion: Towards a Recipe for Multilingual Synthetic Data Generation Our results provide actionable insights for selecting and effectively using teacher models in multilingual synthetic data generation. First, we find that model scale does not significantly predict teacher effectiveness: Llama 3.1 70B Instruct, despite being the largest model evaluated, ranks at the bottom half in PG-Score across all student base models we tested (§[3.1](#S3.SS1), §[3.2](#S3.SS2)). Our analyses suggest that what matters instead is the quality of generated data: prompt diversity, response fluency, and length collectively capture over 93% of the variance in intrinsic data quality and predict student performance with $R^{2}$=0.664 (§[4.2](#S4.SS2)), offering practitioners a cheaper alternative to full student training runs for screening teacher candidates. Second, when the optimal teacher is unknown, matching model families offers a reliable heuristic for teacher selection. Gemma teachers paired with Gemma students, and Llama teachers with Llama students, outperform a mismatched baseline by at least 20% ([Figure 2](#S3.F2)). We hypothesize this finding reflects shared tokenization and similar pretraining distributions, though disentangling these factors remains future work. Finally, we find that there are language-dependent considerations for data generation. For high-resource languages like German, where seed data quality is high, the Generate method performs best. For less-resourced languages like Arabic and Indonesian, methods that leverage existing prompts (Respond) or transfer from English (Translate) can yield substantial gains over a uniform mix of methods, though the magnitude varies by teacher ([Table 2](#S3.T2)). For truly low-resource languages, we recommend combining synthetic data generation with targeted data collection. As a supplementary, we demonstrate the applicability of our findings by building a multilingual synthetic data recipe for a held-out language, Tagalog, in [Appendix I](#A9). We show that models trained using our recipe (based on analyses from PG-Score) have better performance on an unseen Filipino-centric benchmark, and that each component of our recommendation (e.g., choose top teacher from [Table 1](#S2.T1), match model families, etc.) resulted in observable performance gains. This suggests that our evaluation protocol is robust that the insights transfer to an unseen language, even when measured with a different set of downstream metrics. ## 6 Related Work #### Synthetic Data Generation for Multilingual SFT In order to offset the high costs of recruiting language experts for data collection, prior works relied on generating synthetic datasets. This effort resulted in large multilingual datasets such as Bactrian-X (Translate, Li et al., 2023), MultiAlpaca (Generate, Wei et al., 2023), and xP3 (Respond, Muennighoff et al., 2023) that were created through various data generation methods. These works have different data generation recipes, and so we provide a brief survey of these works and their recipes in [Appendix A](#A1), then classify them across the three strategies / archetypes (Generate, Translate, Response; [2.2](#S2.SS2)). Building on these prior efforts, we examine the three core strategies for multilingual synthetic data generation, distill them into three strategies, and test each in isolation. This setup enabled us to provide practitioners with empirically-grounded recipe on selecting teacher LMs that we hope to be applicable across any generation method. #### Evaluating and Improving the Synthetic Data Pipeline While prior works have evaluated aspects of the synthetic data pipeline, they typically do so in isolation (i.e., intrinsic $\oplus$ extrinsic) or focus exclusively on English (Zhang et al., 2025a). For instance, Kim et al. (2025) evaluated teacher models solely as a function of extrinsic student performance on English tasks (e.g., reasoning and coding), while Cai et al. (2025)’s OpenDataArena focuses on intrinsic data quality (model-based and heuristic) to score models. Signals of multilingual data quality are often a function of corpus-level diversity (Artetxe and Schwenk, 2019; Enevoldsen et al., 2025; Sam et al., 2025) and generation quality (Pombal et al., 2025; Anugraha et al., 2026) On the other hand, multilingual LMs are typically evaluated on general-knowledge and culture-specific benchmarks (Qin et al., 2025; Gemma Team et al., 2025; Salamanca et al., 2026, inter alia). These practices informed our choice of intrinsic and extrinsic metrics throughout this work. More importantly, PG-Score provides a holistic analysis that combines both intrinsic data quality and extrinsic student downstream performance to evaluate teacher models across various generation methods. ## 7 Conclusion We conduct a comprehensive evaluation of state-of-the-art LMs as multilingual teachers for synthetic data generation by assessing both intrinsic data quality and extrinsic student model performance. We find several properties that contribute to teacher effectiveness outside of model size or benchmark performance, such as prompt-response diversity, fluency, and language representation. Finally, we outline practical recommendations for creating a multilingual synthetic data generation recipe. We hope our findings guide future work on developing inclusive language technologies through high-quality synthetic data. ## Limitations Our work comes with some limitations and open questions left for future work. For example, our language set encompasses six languages. Although we chose these languages carefully based on (1) whether they can be evaluated on publicly-available LM benchmarks and (2) prior theoretical work on principled test language selection (Ploeger et al., 2026), validating our findings across a broader language sample remains important future work. In addition, our Translate data generation method assumes access to English prompts that can be meaningfully translated to target languages. This approach inherits limitations from LM-based techniques such as localizing culture-specific references, introducing translationese artifacts. ## Ethics Statement Synthetic data generation risks amplifying biases present in teacher models. If a teacher model underperforms on certain languages or exhibits cultural biases, these weaknesses propagate to student models trained on its outputs. Our finding that teacher effectiveness correlates with CommonCrawl representation ($\rho=0.886$, based on six languages) suggests that already underrepresented languages may be further disadvantaged in synthetic data pipelines, potentially widening the performance gap between high- and low-resource languages. ## Acknowledgments LJVM and AK acknowledge the support of the UKRI Frontier Grant EP/Y031350/1 (EQUATE). This work was performed using joint resources provided by the Cambridge Service for Data Driven Discovery (CSD3) EP/T022159/1, Isambard AI National AI Research Resource (AIRR) ST/AIRR/I-A-I/1023, and the Microsoft Research Grant. LJVM would also like to thank Songbo Hu, Chen Cecilia Liu, Millicent Ochieng, and Felermino Ali for helpful and productive discussions on the project. ## References - R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem (2024) On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=3zKtaqxLhW) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.2). - S. Ahuja, K. Tanmay, H. H. Chauhan, B. Patra, K. Aggarwal, L. D. Corro, A. Mitra, T. I. Dhamecha, A. H. Awadallah, M. Choudhury, V. Chaudhary, and S. Sitaram (2025) SPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), O. Arviv, M. Clinciu, K. Dhole, R. Dror, S. Gehrmann, E. Habba, I. Itzhak, S. Mille, Y. Perlitz, E. Santus, J. Sedoc, M. Shmueli Scheuer, G. Stanovsky, and O. Tafjord (Eds.), Vienna, Austria and virtual meeting, pp. 927–946. External Links: [Link](https://aclanthology.org/2025.gem-1.73/), ISBN 979-8-89176-261-9 Cited by: [Table 5](#Ax1.T5.1.8.7.1.1.1). - Anthropic (2024) The Claude 3 Model Family: Opus, Sonnet, Haiku. Technical report Anthropic. External Links: [Link](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf) Cited by: [Appendix H](#A8.p1.1). - D. Anugraha, S. Hung, Z. Tang, E. A. Lee, D. T. Wijaya, and G. I. Winata (2026) mR3: Multilingual Rubric-Agnostic Reward Reasoning Models. In The Fourteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=ST0wOB1bdX) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - M. Artetxe and H. Schwenk (2019) Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, pp. 3197–3203. External Links: [Link](https://aclanthology.org/P19-1309/), [Document](https://dx.doi.org/10.18653/v1/P19-1309) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - V. Aryabumi, J. Dang, D. Talupuru, S. Dash, D. Cairuz, H. Lin, B. Venkitesh, M. Smith, J. A. Campos, Y. C. Tan, K. Marchisio, M. Bartolo, S. Ruder, A. Locatelli, J. Kreutzer, N. Frosst, A. Gomez, P. Blunsom, M. Fadaee, A. Üstün, and S. Hooker (2024) Aya 23: Open Weight Releases to Further Multilingual Progress. External Links: 2405.15032, [Link](https://arxiv.org/abs/2405.15032) Cited by: [§1](#S1.p1.1), [§2.1](#S2.SS1.p1.3). - Y. Babakhin, R. Osmulski, R. Ak, G. Moreira, M. Xu, B. Schifferer, B. Liu, and E. Oldridge (2025) Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks. External Links: 2511.07025, [Link](https://arxiv.org/abs/2511.07025) Cited by: [1st item](#S2.I2.i1.p1.1). - N. Boizard, K. E. Haddad, C. Hudelot, and P. Colombo (2025) Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=bwRxXiGO9A) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.2). - S. Cahyawijaya, H. Lovenia, F. Koto, R. Putri, W. Cenggoro, J. Lee, S. Akbar, E. Dave, N. Nuurshadieq, M. Mahendra, R. Putri, B. Wilie, G. Winata, A. Aji, A. Purwarianti, and P. Fung (2024) Cendol: open instruction-tuned generative large language models for Indonesian languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 14899–14914. External Links: [Link](https://aclanthology.org/2024.acl-long.796/), [Document](https://dx.doi.org/10.18653/v1/2024.acl-long.796) Cited by: [Table 5](#Ax1.T5.1.5.4.1.1.1), [§1](#S1.p2.1). - M. Cai, X. Gao, Y. Li, H. Lin, Z. Liu, Z. Pan, Q. Pei, X. Shang, M. Sun, Z. Tang, X. Wang, Z. Zhong, Y. Zhu, D. Lin, C. He, and L. Wu (2025) OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value. External Links: 2512.14051, [Link](https://arxiv.org/abs/2512.14051) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - H. Chen, A. Waheed, X. Li, Y. Wang, J. Wang, B. Raj, and M. I. Abdin (2024) On the Diversity of Synthetic Data and its Impact on Training Large Language Models. External Links: 2410.15226, [Link](https://arxiv.org/abs/2410.15226) Cited by: [§2.2](#S2.SS2.SSS0.Px2.p1.1). - K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training Verifiers to Solve Math Word Problems. External Links: 2110.14168, [Link](https://arxiv.org/abs/2110.14168) Cited by: [3rd item](#S2.I3.i3.p1.1), [§2.1](#S2.SS1.p1.3). - Cohere Team, Aakanksha, A. Ahmadian, M. Ahmed, J. Alammar, M. Alizadeh, Y. Alnumay, S. Althammer, A. Arkhangorodsky, V. Aryabumi, D. Aumiller, R. Avalos, Z. Aviv, S. Bae, S. Baji, A. Barbet, M. Bartolo, B. Bebensee, N. Beladia, W. Beller-Morales, A. Bérard, A. Berneshawi, A. Bialas, P. Blunsom, M. Bobkin, A. Bongale, S. Braun, M. Brunet, S. Cahyawijaya, D. Cairuz, J. A. Campos, C. Cao, K. Cao, R. Castagné, J. Cendrero, L. C. Currie, Y. Chandak, D. Chang, G. Chatziveroglou, H. Chen, C. Cheng, A. Chevalier, J. T. Chiu, E. Cho, E. Choi, E. Choi, T. Chung, V. Cirik, A. Cismaru, P. Clavier, H. Conklin, L. Crawhall-Stein, D. Crouse, A. F. Cruz-Salinas, B. Cyrus, D. D’souza, H. Dalla-Torre, J. Dang, W. Darling, O. D. Domingues, S. Dash, A. Debugne, T. Dehaze, S. Desai, J. Devassy, R. Dholakia, K. Duffy, A. Edalati, A. Eldeib, A. Elkady, S. Elsharkawy, I. Ergün, B. Ermis, M. Fadaee, B. Fan, L. Fayoux, Y. Flet-Berliac, N. Frosst, M. Gallé, W. Galuba, U. Garg, M. Geist, M. G. Azar, E. Gilsenan-McMahon, S. Goldfarb-Tarrant, T. Goldsack, A. Gomez, V. M. Gonzaga, N. Govindarajan, M. Govindassamy, N. Grinsztajn, N. Gritsch, P. Gu, S. Guo, K. Haefeli, R. Hajjar, T. Hawes, J. He, S. Hofstätter, S. Hong, S. Hooker, T. Hosking, S. Howe, E. Hu, R. Huang, H. Jain, R. Jain, N. Jakobi, M. Jenkins, J. Jordan, D. Joshi, J. Jung, T. Kalyanpur, S. R. Kamalakara, J. Kedrzycki, G. Keskin, E. Kim, J. Kim, W. Ko, T. Kocmi, M. Kozakov, W. Kryściński, A. K. Jain, K. K. Teru, S. Land, M. Lasby, O. Lasche, J. Lee, P. Lewis, J. Li, J. Li, H. Lin, A. Locatelli, K. Luong, R. Ma, L. Mach, M. Machado, J. Magbitang, B. M. Lopez, A. Mann, K. Marchisio, O. Markham, A. Matton, A. McKinney, D. McLoughlin, J. Mokry, A. Morisot, A. Moulder, H. Moynehan, M. Mozes, V. Muppalla, L. Murakhovska, H. Nagarajan, A. Nandula, H. Nasir, S. Nehra, J. Netto-Rosen, D. Ohashi, J. Owers-Bardsley, J. Ozuzu, D. Padilla, G. Park, S. Passaglia, J. Pekmez, L. Penstone, A. Piktus, C. Ploeg, A. Poulton, Y. Qi, S. Raghvendra, M. Ramos, E. Ranjan, P. Richemond, C. Robert-Michon, A. Rodriguez, S. Roy, S. Ruder, L. Ruis, L. Rust, A. Sachan, A. Salamanca, K. K. Saravanakumar, I. Satyakam, A. S. Sebag, P. Sen, S. Sepehri, P. Seshadri, Y. Shen, T. Sherborne, S. S. Shi, S. Shivaprasad, V. Shmyhlo, A. Shrinivason, I. Shteinbuk, A. Shukayev, M. Simard, E. Snyder, A. Spataru, V. Spooner, T. Starostina, F. Strub, Y. Su, J. Sun, D. Talupuru, E. Tarassov, E. Tommasone, J. Tracey, B. Trend, E. Tumer, A. Üstün, B. Venkitesh, D. Venuto, P. Verga, M. Voisin, A. Wang, D. Wang, S. Wang, E. Wen, N. White, J. Willman, M. Winkels, C. Xia, J. Xie, M. Xu, B. Yang, T. Yi-Chern, I. Zhang, Z. Zhao, and Z. Zhao (2025) Command A: An Enterprise-Ready Large Language Model. External Links: 2504.00698, [Link](https://arxiv.org/abs/2504.00698) Cited by: [Table 7](#A4.T7.1.5.5.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1). - J. Dang, S. Singh, D. D’souza, A. Ahmadian, A. Salamanca, M. Smith, A. Peppin, S. Hong, M. Govindassamy, T. Zhao, S. Kublik, M. Amer, V. Aryabumi, J. A. Campos, Y. Tan, T. Kocmi, F. Strub, N. Grinsztajn, Y. Flet-Berliac, A. Locatelli, H. Lin, D. Talupuru, B. Venkitesh, D. Cairuz, B. Yang, T. Chung, W. Ko, S. S. Shi, A. Shukayev, S. Bae, A. Piktus, R. Castagné, F. Cruz-Salinas, E. Kim, L. Crawhall-Stein, A. Morisot, S. Roy, P. Blunsom, I. Zhang, A. Gomez, N. Frosst, M. Fadaee, B. Ermis, A. Üstün, and S. Hooker (2024) Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier. External Links: 2412.04261, [Link](https://arxiv.org/abs/2412.04261) Cited by: [Table 7](#A4.T7.1.6.6.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1). - K. Enevoldsen, I. Chung, I. Kerboua, M. Kardos, A. Mathur, D. Stap, J. Gala, W. Siblini, D. Krzemiński, G. I. Winata, S. Sturua, S. Utpala, M. Ciancone, M. Schaeffer, D. Misra, S. Dhakal, J. Rystrøm, R. Solomatin, Ö. V. Çağatan, A. Kundu, M. Bernstorff, S. Xiao, A. Sukhlecha, B. Pahwa, R. Poświata, K. K. GV, S. Ashraf, D. Auras, B. Plüster, J. P. Harries, L. Magne, I. Mohr, D. Zhu, H. Gisserot-Boukhlef, T. Aarsen, J. Kostkan, K. Wojtasik, T. Lee, M. Suppa, C. Zhang, R. Rocca, M. Hamdy, A. Michail, J. Yang, M. Faysse, A. Vatolin, N. Thakur, M. Dey, D. Vasani, P. A. Chitale, S. Tedeschi, N. Tai, A. Snegirev, M. Hendriksen, M. Günther, M. Xia, W. Shi, X. H. Lù, J. Clive, G. K, M. Anna, S. Wehrli, M. Tikhonova, H. S. Panchal, A. Abramov, M. Ostendorff, Z. Liu, S. Clematide, L. J. V. Miranda, A. Fenogenova, G. Song, R. B. Safi, W. Li, A. Borghini, F. Cassano, L. Hansen, S. Hooker, C. Xiao, V. Adlakha, O. Weller, S. Reddy, and N. Muennighoff (2025) MMTEB: Massive Multilingual Text Embedding Benchmark. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=zl3pfz4VCV) Cited by: [1st item](#S2.I2.i1.p1.1), [§6](#S6.SS0.SSS0.Px2.p1.1). - Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot (2025) Gemma 3 Technical Report. External Links: 2503.19786, [Link](https://arxiv.org/abs/2503.19786) Cited by: [Table 7](#A4.T7.1.7.7.1), [Table 7](#A4.T7.1.8.8.1), [Table 7](#A4.T7.1.9.9.1), [§I.3](#A9.SS3.SSS0.Px5.p1.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1), [§6](#S6.SS0.SSS0.Px2.p1.1). - Granite Team, IBM (2025) Granite 4.0 Language Models. Hugging Face. Note: [https://huggingface.co/collections/ibm-granite/granite-40-language-models](https://huggingface.co/collections/ibm-granite/granite-40-language-models)Accessed: 2025-12-08 Cited by: [Table 7](#A4.T7.1.10.10.1), [Table 7](#A4.T7.1.11.11.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1). - A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma (2024) The Llama 3 Herd of Models. External Links: 2407.21783, [Link](https://arxiv.org/abs/2407.21783) Cited by: [Table 7](#A4.T7.1.3.3.1), [Table 7](#A4.T7.1.4.4.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1). - S. Gureja, L. J. V. Miranda, S. B. Islam, R. Maheshwary, D. Sharma, G. T. Winata, N. Lambert, S. Ruder, S. Hooker, and M. Fadaee (2025) M-RewardBench: Evaluating Reward Models in Multilingual Settings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 43–58. External Links: [Link](https://aclanthology.org/2025.acl-long.3/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.3), ISBN 979-8-89176-251-0 Cited by: [§E.2](#A5.SS2.p2.1), [Table 12](#A6.T12), [2nd item](#S2.I3.i2.p1.1). - N. Habib, C. Fourrier, H. Kydlíček, T. Wolf, and L. Tunstall (2023) LightEval: a lightweight framework for LLM evaluation. External Links: [Link](https://github.com/huggingface/lighteval) Cited by: [§E.2](#A5.SS2.p1.1). - H. A. A. K. Hammoud, M. B. Zbib, and B. Ghanem (2026) Hala Technical Report Building Arabic-Centric Instruction & Translation Models at Scale. In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script, M. El-Haj, P. Rayson, M. Jarrar, I. Ezeani, S. Ezzini, S. Ahmadi, A. Haddad Haddad, C. Amol, A. Abdelali, and S. Abudalfa (Eds.), Rabat, Morocco, pp. 236–244. External Links: [Link](https://aclanthology.org/2026.abjadnlp-1.32/), [Document](https://dx.doi.org/10.18653/v1/2026.abjadnlp-1.32) Cited by: [§1](#S1.p2.1). - D. Han, M. Han, and U. Team (2023) Unsloth External Links: [Link](http://github.com/unslothai/unsloth) Cited by: [§E.1](#A5.SS1.p1.1). - D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021) Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=d7KBjmI3GmQ) Cited by: [1st item](#S2.I3.i1.p1.1). - P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury (2020) The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 6282–6293. External Links: [Link](https://aclanthology.org/2020.acl-main.560/), [Document](https://dx.doi.org/10.18653/v1/2020.acl-main.560) Cited by: [Table 8](#A4.T8), [§G.5](#A7.SS5.SSS0.Px1.p1.2), [§G.5](#A7.SS5.SSS0.Px2.p1.4), [Appendix I](#A9.p1.1). - A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov (2016) FastText.zip: Compressing text classification models. External Links: 1612.03651, [Link](https://arxiv.org/abs/1612.03651) Cited by: [footnote 4](#footnote4). - A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (2017) Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, M. Lapata, P. Blunsom, and A. Koller (Eds.), Valencia, Spain, pp. 427–431. External Links: [Link](https://aclanthology.org/E17-2068/) Cited by: [footnote 4](#footnote4). - S. Kapania, S. Ballard, A. Kessler, and J. W. Vaughan (2025) Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’25, New York, NY, USA, pp. 45–60. External Links: ISBN 9798400714825, [Link](https://doi.org/10.1145/3715275.3732005), [Document](https://dx.doi.org/10.1145/3715275.3732005) Cited by: [§1](#S1.p1.1). - J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. External Links: 2001.08361, [Link](https://arxiv.org/abs/2001.08361) Cited by: [§G.1](#A7.SS1.p1.1). - S. Kim, J. Suk, X. Yue, V. Viswanathan, S. Lee, Y. Wang, K. Gashteovski, C. Lawrence, S. Welleck, and G. Neubig (2025) Evaluating language models as synthetic data generators. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 6385–6403. External Links: [Link](https://aclanthology.org/2025.acl-long.320/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.320), ISBN 979-8-89176-251-0 Cited by: [Table 12](#A6.T12), [§2.3](#S2.SS3.p3.4), [§4.1](#S4.SS1.SSS0.Px2.p1.2), [§6](#S6.SS0.SSS0.Px2.p1.1). - Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, pp. 1317–1327. External Links: [Link](https://aclanthology.org/D16-1139/), [Document](https://dx.doi.org/10.18653/v1/D16-1139) Cited by: [§1](#S1.p2.1). - A. Kunchukuttan, R. Dabre, R. Murthy, M. S. U. R. Khan, and T. Jayakumar (2025) Data and Model Centric Approaches for Expansion of Large Language Models to New languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, V. Pyatkin and A. Vlachos (Eds.), Suzhou, China, pp. 12–13. External Links: [Link](https://aclanthology.org/2025.emnlp-tutorials.5/), [Document](https://dx.doi.org/10.18653/v1/2025.emnlp-tutorials.5), ISBN 979-8-89176-336-4 Cited by: [§1](#S1.p1.1). - W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: [Appendix J](#A10.SS0.SSS0.Px2.p1.1). - N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, X. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi (2025) Tulu 3: Pushing Frontiers in Open Language Model Post-Training. In Second Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=i1uGbfHHpH) Cited by: [§2.1](#S2.SS1.p1.3). - H. Li, F. Koto, M. Wu, A. F. Aji, and T. Baldwin (2023) Bactrian-X: Multilingual replicable instruction-following models with low-rank adaptation. External Links: 2305.15011, [Link](https://arxiv.org/abs/2305.15011) Cited by: [Table 5](#Ax1.T5.1.2.1.1.1.1), [§6](#S6.SS0.SSS0.Px1.p1.1). - Y. Li, X. Yue, Z. Xu, F. Jiang, L. Niu, B. Y. Lin, B. Ramasubramanian, and R. Poovendran (2025) Small Models Struggle to Learn from Strong Reasoners. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 25366–25394. External Links: [Link](https://aclanthology.org/2025.findings-acl.1301/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1301), ISBN 979-8-89176-256-5 Cited by: [§1](#S1.p2.1). - R. Marten, T. Vu, C. C. Ji, K. Sharma, S. Pimpalgaonkar, A. Dimakis, and M. Sathiamoorthy (2025) Curator: A Tool for Synthetic Data Creation Note: [https://github.com/bespokelabsai/curator](https://github.com/bespokelabsai/curator) Cited by: [Appendix J](#A10.SS0.SSS0.Px2.p1.1). - P. H. Martins, J. Alves, P. Fernandes, N. M. Guerreiro, R. Rei, A. Farajian, M. Klimaszewski, D. M. Alves, J. Pombal, N. Boizard, M. Faysse, P. Colombo, F. Yvon, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins (2025) EuroLLM-9B: Technical Report. External Links: 2506.04079, [Link](https://arxiv.org/abs/2506.04079) Cited by: [Table 5](#Ax1.T5.1.9.8.1.1.1), [§1](#S1.p2.1), [§2.1](#S2.SS1.p1.3). - P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins (2024) EuroLLM: Multilingual Language Models for Europe. External Links: 2409.16235, [Link](https://arxiv.org/abs/2409.16235) Cited by: [Table 5](#Ax1.T5.1.9.8.1.1.1). - L. J. V. Miranda, E. Aco, C. G. Manuel, J. C. B. Cruz, and J. M. Imperial (2025) FilBench: can LLMs Understand and Generate Filipino?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, pp. 2496–2529. External Links: [Link](https://aclanthology.org/2025.emnlp-main.127/), [Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.127), ISBN 979-8-89176-332-6 Cited by: [§I.1](#A9.SS1.SSS0.Px2.p1.1), [Table 19](#A9.T19). - N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. Le Scao, M. S. Bari, S. Shen, Z. X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel (2023) Crosslingual Generalization through Multitask Finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 15991–16111. External Links: [Link](https://aclanthology.org/2023.acl-long.891/), [Document](https://dx.doi.org/10.18653/v1/2023.acl-long.891) Cited by: [Table 5](#Ax1.T5.1.4.3.1.1.1), [§6](#S6.SS0.SSS0.Px1.p1.1). - R. Ng, T. N. Nguyen, H. Yuli, T. N. Chia, L. W. Yi, W. Q. Leong, X. Yong, J. G. Ngui, Y. Susanto, N. Cheng, H. Rengarajan, P. Limkonchotiwat, A. V. Hulagadri, K. W. Teng, Y. Y. Tong, B. Siow, W. Y. Teo, T. C. Meng, B. Ong, Z. H. Ong, J. R. Montalan, A. Chan, S. Antonyrex, R. Lee, E. Choa, D. O. Tat-Wee, B. J. D. Liu, W. C. Tjhi, E. Cambria, and L. Teo (2025) SEA-LION: Southeast Asian Languages in One Network. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, K. Inui, S. Sakti, H. Wang, D. F. Wong, P. Bhattacharyya, B. Banerjee, A. Ekbal, T. Chakraborty, and D. P. Singh (Eds.), Mumbai, India, pp. 512–526. External Links: [Link](https://aclanthology.org/2025.ijcnlp-long.30/), [Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-long.30), ISBN 979-8-89176-298-5 Cited by: [Table 5](#Ax1.T5.1.10.9.1.1.1), [§1](#S1.p2.1). - NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang (2022) No language left behind: scaling human-centered machine translation. External Links: 2207.04672, [Link](https://arxiv.org/abs/2207.04672) Cited by: [§G.3](#A7.SS3.p1.1). - OLMo Team, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. Le Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025) OLMo 3. Technical report Allen Institute for AI. Note: Technical Report External Links: [Link](https://allenai.org/olmo) Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.2). - OpenAI, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. O’Connell, I. O’Connell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, L. Ouyang, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. O. T. de Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov (2024) GPT-4o System Card. External Links: 2410.21276, [Link](https://arxiv.org/abs/2410.21276) Cited by: [Table 7](#A4.T7.1.2.2.1), [§3.1](#S3.SS1.SSS0.Px2.p1.1). - L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: [Link](https://openreview.net/forum?id=TG8KACxEON) Cited by: [§1](#S1.p1.1). - P. Pengpun, C. Udomcharoenchaikit, W. Buaphet, and P. Limkonchotiwat (2024) Seed-free synthetic data generation framework for instruction-tuning LLMs: a case study in Thai. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), X. Fu and E. Fleisig (Eds.), Bangkok, Thailand, pp. 445–464. External Links: [Link](https://aclanthology.org/2024.acl-srw.50/), ISBN 979-8-89176-097-4 Cited by: [Table 5](#Ax1.T5.1.6.5.1.1.1). - E. Ploeger, W. Poelman, A. H. Høeg-Petersen, A. Schlichtkrull, M. de Lhoneux, and J. Bjerva (2026) A principled framework for evaluating on typologically diverse languages. Computational Linguistics, pp. 1–33. External Links: ISSN 0891-2017, [Document](https://dx.doi.org/10.1162/COLI.a.577), [Link](https://doi.org/10.1162/COLI.a.577), https://direct.mit.edu/coli/article-pdf/doi/10.1162/COLI.a.577/2561978/coli.a.577.pdf Cited by: [§3.1](#S3.SS1.SSS0.Px3.p1.1), [Limitations](#Sx1.p1.1). - J. Pombal, D. Yoon, P. Fernandes, I. Wu, S. Kim, R. Rei, G. Neubig, and A. Martins (2025) M-Prometheus: A Suite of Open Multilingual LLM Judges. In Second Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=Atyk8lnIQQ) Cited by: [Figure 13](#A10.F13), [3rd item](#S2.I2.i3.p1.1), [§6](#S6.SS0.SSS0.Px2.p1.1). - L. Qin, Q. Chen, Y. Zhou, Z. Chen, Y. Li, L. Liao, M. Li, W. Che, and P. S. Yu (2025) A survey of multilingual large language models. Patterns 6 (1), pp. 101118. External Links: ISSN 2666-3899, [Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.patter.2024.101118), [Link](https://www.sciencedirect.com/science/article/pii/S2666389924002903) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - N. P. Rachamalla, A. Konakalla, G. Rajeev, A. Kulkarni, C. Khatri, and S. Agarwal (2025) Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), D. I. Adelani, C. Arnett, D. Ataman, T. A. Chang, H. Gonen, R. Raja, F. Schmidt, D. Stap, and J. Wang (Eds.), Suzhuo, China, pp. 285–321. External Links: [Link](https://aclanthology.org/2025.mrl-main.20/), [Document](https://dx.doi.org/10.18653/v1/2025.mrl-main.20), ISBN 979-8-89176-345-6 Cited by: [Table 5](#Ax1.T5.1.12.11.1.1.1). - C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21 (1). External Links: ISSN 1532-4435 Cited by: [Table 8](#A4.T8), [§G.5](#A7.SS5.SSS0.Px1.p1.2). - A. Raventos, M. Paul, F. Chen, and S. Ganguli (2023) Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=BtAz4a5xDg) Cited by: [§2.2](#S2.SS2.SSS0.Px2.p1.1). - A. R. Salamanca, D. Abagyan, D. D’souza, A. Khairi, D. Mora, S. Dash, V. Aryabumi, S. Rajaee, M. Mofakhami, A. Sahu, T. Euyang, B. Prince, M. Smith, H. Lin, A. Locatelli, S. Hooker, T. Kocmi, A. Gomez, I. Zhang, P. Blunsom, N. Frosst, J. Pineau, B. Ermis, A. Üstün, J. Kreutzer, and M. Fadaee (2026) Tiny Aya: Bridging Scale and Multilingual Depth. External Links: 2603.11510, [Link](https://arxiv.org/abs/2603.11510) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - D. Sam, A. Chakrabarti, A. Rostamizadeh, S. Ramalingam, G. Citovsky, and S. Kumar (2025) Analyzing Similarity Metrics for Data Selection for Language Model Pretraining. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=Idmk7O4sWA) Cited by: [§6](#S6.SS0.SSS0.Px2.p1.1). - M. A. Shafique, K. Mehreen, M. Arham, M. Amjad, S. Butt, and H. Farooq (2025) Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), D. I. Adelani, C. Arnett, D. Ataman, T. A. Chang, H. Gonen, R. Raja, F. Schmidt, D. Stap, and J. Wang (Eds.), Suzhuo, China, pp. 271–284. External Links: [Link](https://aclanthology.org/2025.mrl-main.19/), [Document](https://dx.doi.org/10.18653/v1/2025.mrl-main.19), ISBN 979-8-89176-345-6 Cited by: [Table 5](#Ax1.T5.1.11.10.1.1.1). - F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei (2023) Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=fR3wGCk-IXp) Cited by: [Table 12](#A6.T12), [3rd item](#S2.I3.i3.p1.1). - S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W. Ko, A. Bosselut, A. Oh, A. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker (2025) Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 18761–18799. External Links: [Link](https://aclanthology.org/2025.acl-long.919/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919), ISBN 979-8-89176-251-0 Cited by: [Table 12](#A6.T12), [1st item](#S2.I3.i1.p1.1). - S. Singh, F. Vargus, D. D’souza, B. F. Karlsson, A. Mahendiran, W. Ko, H. Shandilya, J. Patel, D. Mataciunas, L. O’Mahony, M. Zhang, R. Hettiarachchi, J. Wilson, M. Machado, L. Moura, D. Krzemiński, H. Fadaei, I. Ergun, I. Okoh, A. Alaagib, O. Mudannayake, Z. Alyafeai, V. Chien, S. Ruder, S. Guthikonda, E. Alghamdi, S. Gehrmann, N. Muennighoff, M. Bartolo, J. Kreutzer, A. Üstün, M. Fadaee, and S. Hooker (2024) Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 11521–11567. External Links: [Link](https://aclanthology.org/2024.acl-long.620/), [Document](https://dx.doi.org/10.18653/v1/2024.acl-long.620) Cited by: [Table 5](#Ax1.T5.1.7.6.1.1.1), [§1](#S1.p1.1). - B. Upadhayay and V. Behzadan (2024) TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes. In 5th Workshop on practical ML for limited/low resource settings, External Links: [Link](https://openreview.net/forum?id=02MLWBj8HP) Cited by: [Table 18](#A9.T18). - Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 13484–13508. External Links: [Link](https://aclanthology.org/2023.acl-long.754/), [Document](https://dx.doi.org/10.18653/v1/2023.acl-long.754) Cited by: [Table 5](#Ax1.T5.1.3.2.3.1.1). - Z. Wang, J. Zeng, O. Delalleau, D. Egert, E. Evans, H. Shin, F. Soares, Y. Dong, and O. Kuchaiev (2025) HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks. External Links: 2503.04378, [Link](https://arxiv.org/abs/2503.04378) Cited by: [§2.1](#S2.SS1.p1.3). - X. Wei, H. Wei, H. Lin, T. Li, P. Zhang, X. Ren, M. Li, Y. Wan, Z. Cao, B. Xie, T. Hu, S. Li, B. Hui, B. Yu, D. Liu, B. Yang, F. Huang, and J. Xie (2023) PolyLM: An Open Source Polyglot Large Language Model. External Links: 2307.06018, [Link](https://arxiv.org/abs/2307.06018) Cited by: [Table 5](#Ax1.T5.1.3.2.1.1.1), [§6](#S6.SS0.SSS0.Px1.p1.1). - Z. Xu, F. Jiang, L. Niu, Y. Deng, R. Poovendran, Y. Choi, and B. Y. Lin (2025a) Magpie: alignment data synthesis from scratch by prompting aligned LLMs with nothing. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=Pnk7vMbznK) Cited by: [§2.1](#S2.SS1.p1.3). - Z. Xu, F. Jiang, L. Niu, B. Y. Lin, and R. Poovendran (2025b) Stronger models are not always stronger teachers for instruction tuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, pp. 4392–4405. External Links: [Link](https://aclanthology.org/2025.naacl-long.224/), [Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.224), ISBN 979-8-89176-189-6 Cited by: [§1](#S1.p2.1), [§4.1](#S4.SS1.SSS0.Px2.p1.2). - A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 Technical Report. External Links: 2505.09388, [Link](https://arxiv.org/abs/2505.09388) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.2). - H. Zhang, S. Yang, X. Liang, C. Shang, Y. Jiang, C. Tao, J. Xiong, H. K. So, R. Xie, A. X. Chang, and N. Wong (2025a) Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation. External Links: 2510.10925, [Link](https://arxiv.org/abs/2510.10925) Cited by: [§1](#S1.p2.1), [§6](#S6.SS0.SSS0.Px2.p1.1). - S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang (2025b) Instruction Tuning for Large Language Models: A Survey. External Links: 2308.10792, [Link](https://arxiv.org/abs/2308.10792) Cited by: [§1](#S1.p1.1). - W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng (2024) WildChat: 1M ChatGPT Interaction Logs in the Wild. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=Bl8u7ZRlbM) Cited by: [§2.1](#S2.SS1.p1.3). - A. Zhu, P. Asawa, J. Q. Davis, L. Chen, B. Hanin, I. Stoica, J. E. Gonzalez, and M. Zaharia (2025) BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation. External Links: 2502.01697, [Link](https://arxiv.org/abs/2502.01697) Cited by: [§2.2](#S2.SS2.SSS0.Px2.p1.1). ## Appendix **Table 5: Short survey of related work on synthetic data generation for multilingual LMs. For each work, we provide a brief description of their data generation method. We find that most methods fall into one of the three categories described in §[2.2](#S2.SS2.SSS0.Px1), i.e., Generate, Translate, or Respond, which we tested in our experiments.** | Dataset | Language(s) | Generation Method / Description | | --- | --- | --- | | Bactrian-X (Li et al., 2023) | 52 languages - Arabic, Indonesian, Chinese, Malaysian, Tamil, Tagalog, etc. | Translate - used Google Translate API to translate English instructions from Alpaca (52K) and Dolly (15K). | | MultiAlpaca (Wei et al., 2023) | 18 languages - English, Chinese, Russian, Spanish, German, French, etc. | Generate, Translate - used a multilingual self-instruct (Wang et al., 2023) method from English prompt-response pairs to perform translation. | | xP3-MT (Muennighoff et al., 2023) | 46 languages - Arabic, English, Spanish, Hindi, Chinese, Indonesian, etc. | Translate, Respond - used Google Translate API to translate English prompt-response pairs from different sources, in addition to creating template-based prompts where an LM responds to it. | | Cendol (Cahyawijaya et al., 2024) | 18 Indonesian languages - Sundanese, Javanese, Acehnese, Banjarese, Buginese, Gorontalo, etc. | Translate, Respond - curated various prompts from past Indonesian NLP tasks, including translations of Dolly. | | Seed Free Thai (Pengpun et al., 2024) | Thai | Generate - generated synthetic instruction data without seed examples by using Wikipedia contexts. Identifies fluency, diversity, and cultural context as key properties. | | Aya Dataset and Collection (Singh et al., 2024) | 114 languages - Arabic, French, Hindi, Indonesian, Japanese, Spanish, Swahili, Turkish, Yoruba, Filipino, etc. | Translate, Respond - involves a collection of translated prompts from English, and templated prompts. A sizeable portion of the collection includes native-speaker annotations. | | sPhinX (Ahuja et al., 2025) | 51 languages - Afrikaan, Arabic, Bengali, Bulgarian, Burmese, Chinese, Croatian, Czech, etc. | Translate - selectively translates essential portions of multilingual inputs in order to semantically preserve meaning. | | EuroBlocks (Martins et al., 2025, 2024) | 31 languages - English, Chinese, Spanish, Italian, French, German, Portuguese, Dutch, Polish, etc. | Generate, Translate - prompted Llama 3 or an earlier EuroLLM checkpoint with a document, target language, and category, then asking it to generate an instruction. Also involved translating prompt-response pairs. | | SEA-LION Dataset (Ng et al., 2025) | 11 languages - English, Chinese, Indonesian, Vietnamese, Malay, Thai, Burmese, Lao, Filipino, Khmer, and Tamil | Generate, Translate - for the majority of the datasets, samples were first generated into English using Qwen 32B, and then translated into the target language using Gemma 2 27B. | | Urdu-Instruct Dataset (Shafique et al., 2025) | Urdu | Generate - uses a modified Self-Instruct from a pool of culturally relevant prompts. | | Pragyaan (Rachamalla et al., 2025) | 10 Indian languages - Gujarati, Kannada, Marathi, Bengali, Odia, Tamil, Malayalam, Telugu, Punjabi, Hindi, and Sanskrit | Generate, Translate - perform translation using an LM for a subset of data. Used Self-Instruct from a pool of native prompts for another subset of data. | ## Appendix A Multilingual Synthetic Data Generation We present an overview of prior works in [Table 5](#Ax1.T5) that used synthetic data to train multilingual LMs. In general, we find that most data generation methods fall into one of the three categories described in §[2.2](#S2.SS2.SSS0.Px1), i.e., Generate, Translate, or Respond, which we tested in our experiments. Our survey suggests that our choice of data generation methods are grounded in prior work and covers the majority of approaches used in synthetic data generation. ## Appendix B Seed Dataset Statistics [Table 6](#A2.T6) shows the statistics of the seed dataset used for synthetic data generation. **Table 6: Seed dataset statistics. In order to bootstrap our synthetic data generation methods, we use a seed dataset composed of various multilingual instruction-following datasets. We include English samples in order to simulate data generation pipelines where English is translated into a target language. We collect a total of 132,929 seed examples across 7 languages (including English).** | | Language | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | | Source | English (en) | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) | | Aya Dataset | - | - | 5,000 | 241 | 3,854 | 2,786 | 6,259 | | Tülu 3 SFT | 10,000 | - | - | - | - | - | - | | WildChat 4.8M | 10,000 | 4,660 | 1,266 | 5,908 | 5,900 | 7,983 | 602 | | CIDAR | - | 6,000 | - | - | - | - | - | | Cendol v2 | - | - | - | - | - | 3,000 | - | | OpenAssistant 2 | - | 23 | 4 | 2,328 | 8,785 | 3 | 306 | | EuroBlocks SFT | - | - | 3,813 | 12,551 | 15,641 | - | 2,893 | | GSM8k (train) | 7,473 | - | - | - | - | - | - | | Helpsteer3 (chosen) | - | - | - | 462 | 778 | 156 | 534 | | Magpie Pro Filtered | 10,000 | - | - | - | - | - | - | | Total per language | 30,743 | 10,683 | 10,083 | 21,490 | 34,958 | 13,928 | 10,594 | ## Appendix C The Polyglot Collection In order to facilitate future research on multilingual synthetic data generation, we introduce the Polyglot collection, a collection of synthetic datasets and student models generated by the best teacher model across all target languages. The Polyglot collection includes: - • Polyglot-Instructions-Synth: Synthetic datasets for each target language generated by each teacher model using all three data generation methods (§[2.2](#S2.SS2)). - • Polyglot-Gemma-SFT: A set of 8B student models finetuned on each synthetic dataset from the OLMo 3 7B base model using the Gemma 3 27B (highest-scoring model) teacher. We publicly release the Polyglot Collection in HuggingFace.(^3^33[: ljvmiranda921/polyglot-teachers](https://huggingface.co/collections/ljvmiranda921/polyglot-teachers)) ## Appendix D Teacher Model and Target Language Details In this section, we provide additional details about the teacher models and target languages used in our experiments. [Table 7](#A4.T7) summarizes the key characteristics of each teacher model. On the other hand, [Table 8](#A4.T8) provides information about the target languages, including language family, number of speakers, and resource availability. **Table 7: Teacher model details. We evaluate 10 teacher models across different providers, sizes, multilingual capabilities, and licensing terms. Size is reported in billions of parameters (B) where available. # Langs indicates the number of languages the model was trained on or evaluated for.** | Model Name | Provider | Size (B) | # Langs | License | | --- | --- | --- | --- | --- | | GPT-4o mini (OpenAI et al., 2024) | OpenAI | – | 50+ | Proprietary | | Llama 3.1 70B Instruct (Grattafiori et al., 2024) | Meta | 70 | 8 | Llama 3.1 | | Llama 3.1 8B Instruct (Grattafiori et al., 2024) | Meta | 8 | 8 | Llama 3.1 | | Command A (Cohere Team et al., 2025) | Cohere | 104 | 23 | CC-BY-NC-4.0 | | Aya Expanse 32B (Dang et al., 2024) | Cohere | 32 | 23 | CC-BY-NC-4.0 | | Gemma 3 27B Instruct (Gemma Team et al., 2025) | Google | 27 | 100+ | Gemma | | Gemma 3 12B Instruct (Gemma Team et al., 2025) | Google | 12 | 100+ | Gemma | | Gemma 3 4B Instruct (Gemma Team et al., 2025) | Google | 4 | 100+ | Gemma | | IBM Granite 4.0 (Granite Team, IBM, 2025) | IBM | 3 | 116 | Apache 2.0 | | IBM Granite Micro (Granite Team, IBM, 2025) | IBM | 0.4 | 116 | Apache 2.0 | **Table 8: Target language details. We evaluate teacher models across six typologically diverse languages spanning different language families and scripts. Resource availability is based on the classification from Joshi et al. (2020), ranging from 0 (lowest) to 5 (highest). CommonCrawl percentages (Raffel et al., 2020) indicate the proportion of web text available for each language.** | Language | Family | Script | Resource Availability | % in CC | | --- | --- | --- | --- | --- | | Arabic | Afro-Asiatic | Arabic | 5 (High) | 0.65% | | Czech | Indo-European | Latin | 4 (Medium-High) | 0.99% | | German | Indo-European | Latin | 5 (High) | 6.01% | | Spanish | Indo-European | Latin | 5 (High) | 4.37% | | Indonesian | Austronesian | Latin | 3 (Medium) | 0.95% | | Japanese | Japonic | Japanese | 5 (High) | 5.20% | ## Appendix E Experimental Details ### E.1 Supervised Finetuning [Table 9](#A5.T9) summarizes the hyperparameters used for finetuning student models. We train models using the Unsloth framework (Han et al., 2023) using a cluster of Grace Hopper GH200 Superchips. Full finetuning (7B) takes around 1.5 hours (wall clock) for 2 epochs and 2 nodes. **Table 9: Hyperparameters for finetuning a 7B student model from OLMo 3 7B.** | Hyperparameter | Value | Hyperparameter | Value | | --- | --- | --- | --- | | Learning rate | 5e-5 | Batch size | 32 | | Epochs | 2 | Grad. Acum. Steps | 4 | | Max seq. length | 16,384 | Weight decay | 0.001 | | Optimizer | AdamW | Scheduler | Linear | ### E.2 Model Evaluation We used the Lighteval framework (v0.13.1dev0, Habib et al., 2023) for evaluation. [Table 10](#A5.T10) summarizes the benchmarks used for evaluating student models. We decided to use Global-MMLU Lite instead of Global-MMLU becaue the former contains actual native speaker annotations that localized the benchmark into different cultural contexts. **Table 10: Evaluation settings for each benchmark (MCF: Multiple-Choice Formulation).** | Benchmark | Formulation | Metric | N-shots | | --- | --- | --- | --- | | Global-MMLU Lite | MCF | Accuracy | 0 | | M-RewardBench | MCF | Weighted Acc. | 0 | | M-GSM | Generative | Exact-Match | 5 | For Global-MMLU Lite and M-RewardBench, we use the Multiple-Choice Formulation (MCF) with character normalization. In addition, we also follow the corpus-level metric in M-RewardBench which uses a weighted accuracy for each data subset and category (Gureja et al., 2025). For M-GSM, we show 5 few-shot examples from the training set in order for the model to properly generate the answer. We run all evaluation experiments for three trials with different random seeds and report the average and standard deviation. ## Appendix F Full Results for Intr. and Extr. Metrics [Table 11](#A6.T11) shows all the data quality metrics for each teacher model across all languages. [Table 12](#A6.T12) shows the full results of student models finetuned on synthetic datasets generated by each teacher model across all target languages. **Table 11: Full intrinsic evaluation results across all languages. Data quality metrics include the diversity of prompts and responses ($d_{P}$ and $d_{R}$), average perplexity of the student model on the response (PPL), and average reward score based on a multilingual LLM judge (R).** | | Arabic (ar) | Czech (cs) | German (de) | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Model | $d_{x}$ | $d_{y}$ | PPL | R | $d_{x}$ | $d_{y}$ | PPL | R | $d_{x}$ | $d_{y}$ | PPL | R | | GPT 4o mini | 0.704 | 0.869 | 8.40 | 3.516 | 0.643 | 0.862 | 3.18 | 3.716 | 0.732 | 0.889 | 3.65 | 3.810 | | Llama 3.1 70B Inst. | 0.701 | 0.875 | 7.00 | 2.719 | 0.654 | 0.889 | 3.18 | 3.327 | 0.707 | 0.892 | 3.22 | 3.396 | | Llama 3.1 8B Inst. | 0.708 | 0.779 | 6.2e4 | 1.731 | 0.673 | 0.799 | 2.7e4 | 1.908 | 0.738 | 0.873 | 3.6e3 | 2.513 | | Command A | 0.690 | 0.846 | 5.41 | 3.996 | 0.647 | 0.865 | 3.24 | 4.184 | 0.730 | 0.889 | 3.59 | 4.235 | | Aya Expanse 32B | 0.693 | 0.888 | 4.34 | 3.964 | 0.650 | 0.884 | 3.15 | 4.133 | 0.700 | 0.902 | 3.44 | 4.140 | | Gemma 3 27B Inst. | 0.717 | 0.890 | 4.40 | 3.932 | 0.675 | 0.885 | 3.77 | 4.342 | 0.731 | 0.898 | 3.96 | 4.260 | | Gemma 3 12B Inst. | 0.721 | 0.864 | 4.43 | 3.774 | 0.676 | 0.882 | 3.88 | 4.266 | 0.751 | 0.899 | 4.06 | 4.203 | | Gemma 3 4B Inst. | 0.728 | 0.869 | 5.52 | 3.470 | 0.682 | 0.883 | 3.87 | 4.127 | 0.744 | 0.898 | 3.96 | 4.103 | | IBM Granite 4.0 | 0.704 | 0.829 | 1.9e4 | 2.463 | 0.665 | 0.862 | 5.29 | 3.158 | 0.717 | 0.885 | 24.61 | 3.365 | | IBM Granite Micro | 0.741 | 0.863 | 12.45 | 3.033 | 0.713 | 0.874 | 4.61 | 3.568 | 0.726 | 0.892 | 4.59 | 3.704 | **Table 12: Average performance gain recovered (PGR) of a student model across various multilingual benchmarks. Our multilingual evaluation suite includes Global-MMLU Lite (Singh et al., 2025), M-RewardBench (Gureja et al., 2025), and M-GSM (Shi et al., 2023). The PGR computation is based on Kim et al. (2025) and detailed in §[2.3](#S2.SS3) ([Equation 2](#S2.E2)) where $S_{\text{REF}}=\text{OLMo 3 7B Instruct SFT}$ and $S_{\phi}=\text{OLMo 3 1025 7B}$.** | Model | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) | | --- | --- | --- | --- | --- | --- | --- | | GPT 4o mini | -2.086 | 0.538 | 3.098 | 1.395 | 2.025 | 0.099 | | Llama 3.1 70B Inst. | -1.528 | 0.538 | 2.265 | 1.075 | 0.329 | 0.013 | | Llama 3.1 8B Inst. | -0.841 | 0.525 | 2.623 | 0.595 | 1.425 | 0.236 | | Command A | -2.476 | 0.505 | 2.759 | 1.613 | 1.863 | 0.841 | | Aya Expanse 32B | -0.293 | 0.538 | 2.491 | 1.701 | 1.943 | 0.221 | | Gemma 3 27B Inst. | -0.074 | 0.552 | 2.635 | 1.724 | 0.198 | 0.677 | | Gemma 3 12B Inst. | -1.015 | 0.538 | 2.700 | 1.592 | -0.017 | 0.524 | | Gemma 3 4B Inst. | -1.033 | 0.538 | 2.568 | 1.209 | -0.388 | 0.349 | | IBM Granite 4.0 | 1.565 | 0.538 | 2.061 | 1.235 | 0.614 | 0.802 | | IBM Granite Micro | -0.421 | 0.538 | 1.842 | 1.203 | -0.659 | 0.210 | **Table 13: Detailed results from [Table 1](#S2.T1) with standard errors. We compute PG-Score thrice with different synthetically-generated data (each trial uses a different data mix based on a random seed). We report the mean and standard error for each teacher model across all target languages. For each language, we highlight the best model in bold and the second-best model with an underline.** | Model | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) | | --- | --- | --- | --- | --- | --- | --- | | Gemma 3 27B Inst. | 0.145 (0.0121) | 0.360 (0.0004) | 1.655 (0.0141) | 1.358 (0.0141) | 0.214 (0.0167) | 0.626 (0.0124) | | Aya Expanse 32B | -0.058 (0.0116) | 0.222 (0.0004) | 1.468 (0.0134) | 1.129 (0.0123) | 1.153 (0.0124) | 0.320 (0.0111) | | Gemma 3 12B Inst. | -0.464 (0.0119) | 0.327 (0.0004) | 1.756 (0.0137) | 1.228 (0.0140) | 0.151 (0.0126) | 0.573 (0.0142) | | Command A | -1.360 (0.0112) | 0.114 (0.0004) | 1.673 (0.0139) | 1.102 (0.0145) | 1.063 (0.0125) | 0.683 (0.0122) | | Gemma 3 4B Inst. | -0.488 (0.0119) | 0.330 (0.0004) | 1.644 (0.0137) | 0.929 (0.0140) | -0.105 (0.0126) | 0.504 (0.0113) | | GPT 4o mini | -1.117 (0.0117) | 0.015 (0.0004) | 1.766 (0.0136) | 0.908 (0.0149) | 1.003 (0.0125) | 0.189 (0.0117) | | IBM Granite 4.0 | -0.072 (0.0123) | -0.031 (0.0004) | 1.000 (0.0135) | 0.734 (0.0151) | -0.079 (0.0125) | 0.321 (0.0108) | | IBM Granite Micro | -0.282 (0.0121) | 0.290 (0.0004) | 1.102 (0.0139) | 0.783 (0.0133) | -0.329 (0.0126) | 0.264 (0.0121) | | Llama 3.1 70B Inst. | -0.964 (0.0117) | 0.109 (0.0004) | 1.195 (0.0146) | 0.688 (0.0146) | 0.182 (0.0126) | -0.373 (0.0116) | | Llama 3.1 8B Inst. | -1.693 (0.0120) | -0.974 (0.0004) | 0.891 (0.0148) | 0.182 (0.0164) | 0.322 (0.0124) | -0.863 (0.0129) | #### Percentage Increase Tables We provide additional tables from the main experiments in §[3](#S3) and §[4](#S4). [Table 14](#A6.T14) shows the percentage increase in PG-Score when using family-matched teacher-student pairs compared to the OLMo 3 7B baseline (see §[3.2](#S3.SS2)). [Table 15](#A6.T15) shows the percentage increase in PG-Score when using the best data generation method for each teacher-language pair compared to an equal mix baseline (see §[3.3](#S3.SS3)). **Table 14: Percentage increase in PG-Score for family-matched teacher-student pairs. Percentage increase when using family-matched teachers compared to OLMo 3 7B baseline (average across Arabic, German, and Indonesian).** | | Base Model ($S_{\phi}$) | | | --- | --- | --- | | Teacher Model | Gemma 3 4B | Llama 3.1 8B | | Llama 3.1 70B Inst. | +362.3% | +260.1% | | Llama 3.1 8B Inst. | +183.1% | +130.0% | | Gemma 3 27B Inst. | +20.5% | +26.5% | | Gemma 3 12B Inst. | +38.5% | +67.2% | | Gemma 3 4B Inst. | +103.4% | +203.4% | **Table 15: Percentage increase in PG-Score for best data generation method. Percentage increase when using the best-performing data generation method compared to an equal mix baseline of all three methods (Generate, Translate, Respond). For less-resourced languages (Arabic and Indonesian), using Translate or Respond methods yields substantial improvements for most teachers, though gains are teacher-dependent.** | | | Teacher Model ($S_{T,\ell}$) | | | | --- | --- | --- | --- | --- | | Language | Best Method | Gemma 3 27B | Aya Expanse 32B | Llama 3.1 70B | | Arabic (ar) | Respond | +453.1% | +355.2% | +77.7% | | German (de) | Generate | +29.3% | +0.3% | +16.4% | | Indonesian (id) | Translate | +458.9% | +39.3% | $-$14.8% | ## Appendix G Additional Experiments and Ablations In this section, we ablate several aspects of our evaluation protocol that may affect a teacher model’s PG-Score. ### G.1 Effect of Data Scale on Student Model Performance One component of PG-Score is the extrinsic student performance metric (§[2.3](#S2.SS3)) as measured by PGR. Scaling laws suggest that this performance improves with more data (Kaplan et al., 2020). Then, it is possible to inflate PG-Score by simply using more synthetic data. In order to control for this variable, we conduct an experiment to determine how much synthetic data is needed to reliably compute PG-Score. #### Setup We finetune an OLMo 3 7B base model on $n$ SFT instances where $n\in\{\text{1k},\text{5k},\text{10k},\text{25k},\text{50k}\}$. To reduce computational costs, we perform this experiment only on a single teacher model (Gemma 3 27B Instruct) on three target languages that represent diverse scripts and resource availability: Arabic, German, and Indonesian. Similar to the main experiments, we represent each data generation method equally when creating the SFT datasets. Then, we recompute the intrinsic metrics and finetune student models and measure their performance across three benchmarks (§[2.3](#S2.SS3)). Figure: Figure 5: Effect of synthetic data scale on student model performance. Student performance improves with more synthetic data, but gains diminish beyond 10k examples. Refer to caption: 2604.11290v1/x10.png #### Results [Figure 5](#A7.F5) shows the average student model performance as a function of the number of SFT instances. We observe that student performance improves with more synthetic data, but gains diminish beyond 10k examples. This finding suggests that using 10k synthetic examples per language is sufficient to reliably compute PG-Score without inflating the metric by increasing the number of samples. In our experiments, we use 10k synthetic examples per language when computing PG-Score. Specifically, we show that 10k synthetic examples from a strong teacher are sufficient to finetune a student model to achieve reasonable performance across multiple benchmarks. ### G.2 Generalization Across Model Size #### Setup In order to test whether PG-Score generalizes beyond 8B parameter size models, we use an OLMo 32B base model ($S_{\phi}$) and recompute the intrinsic and extrinsic metrics to obtain the PG-Score. To save computational costs, we train student models across three teachers (Gemma 3 27B Instruct, Aya Expanse 32B, Llama 30B Instruct) and all 6 target languages. **Table 16: PG-Score of three teacher models ($S_{\phi}=\text{OLMo 3 32B}$) We show that our findings generalize up to the 32B parameter range on the three teacher models we tested: (1) Gemma 3 27B maintains its position as the most effective teacher, and the (2) language-dependent effects are still apparent with German having the highest PG-Scores across most teachers.** | Teacher Model | Average | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) | | --- | --- | --- | --- | --- | --- | --- | --- | | Gemma 3 27B Inst. | 0.805 | -0.239 | 0.222 | 2.389 | 1.855 | 0.239 | 0.366 | | Aya Expanse 32B | 0.227 | -0.872 | -0.038 | 1.979 | 1.353 | -0.249 | -0.809 | | Llama 3.1 70B Inst. | -0.267 | -1.688 | -0.807 | 0.838 | 1.407 | -1.441 | 0.089 | #### Results [Table 16](#A7.T16) shows the PG-Score scores for three teacher models when using OLMo 3 32B as the student model. We find that Gemma 3 27B Instruct remains the highest-scoring teacher in this comparison, achieving the highest average PG-Score of 0.805 across all languages. This result is consistent with our findings using the 8B student model (§[3](#S3)), demonstrating that the superior data quality generated by Gemma 3 27B generalizes across model scales. Aya Expanse 32B achieves a positive average PG-Score of 0.227, while Llama 3.1 70B Instruct shows a negative average of $-$0.267. Furthermore, the language-dependent effects observed in the 8B experiments remain consistent at 32B scale. German continues to show the highest PG-Score values across all three teachers (2.389 for Gemma, 1.979 for Aya, 0.838 for Llama), suggesting that certain languages benefit more from synthetic data regardless of student model size. Similarly, Spanish exhibits strong performance across all teachers, with PG-Score values ranging from 1.353 to 1.855. In contrast, Arabic shows the most variable results, with Gemma achieving slightly negative scores ($-$0.239) while Aya and Llama show substantially lower performance ($-$0.872 and $-$1.688, respectively). Overall, these findings demonstrate that PG-Score and teacher model rankings generalize to the 32B parameter range. ### G.3 Effect of Translation Method (Prompting an LM vs. Translation Model) An alternative to using an LM for translating texts from English to a target language is via a translation model such as NLLB (NLLB Team et al., 2022). In this section, we examine the effect of the translation method on the PG-Score of teacher models. #### Setup First, we filter and sample 10k English prompt-response pairs from the Tülu 3 SFT dataset.(^4^44Tülu 3 also contains non-English data. We perform English-language filtering using fastText (Joulin et al., 2016, 2017) and the staticvectors library.) Then, using the NLLB model (nllb-200-distilled-600M), we perform two translation methods: (1) NLLB-Translate-then-Respond: translate the prompts to each target language and prompt Gemma 3 27B Instruct to generate a response, and (2) NLLB-Translate-Both: translate both the prompts and responses from English to the target language. We choose the 600M version due to its computational efficiency and popularity among practitioners, as measured by HuggingFace downloads and community likes. We compare these methods against our original Translate method, i.e., prompting Gemma 3 27B Instruct to directly translate the prompt and generate the response in the target language (LM-Translate). Then, we compute the intrinsic data quality metrics and finetune OLMo 3 7B student models on each synthetic dataset to compute PG-Score. #### Results [Figure 6](#A7.F6) shows the PG-Score and average benchmark performance of the student model for each translation method across Arabic, German, and Indonesian. We find that LM-Translate outperforms both NLLB-based approaches, achieving an average PG-SCORE of 1.36 compared to 0.85 for NLLB-Translate-Both and 0.80 for NLLB-Translate-then-Respond. This pattern holds across all three languages, with the largest gap observed for German (2.09 vs 1.26/1.68). Our findings suggest that prompt naturalness, rather than response quality, is a bottleneck in translation-based pipelines: having an LM generate responses to NLLB-translated prompts provides no improvement over pure NLLB translation (0.80 vs 0.85), indicating that translated prompts fail to elicit the same quality of responses as LM-translated prompts. Figure: Figure 6: Effect of translation method on PG-Score. We compare three methods: LM translates prompt EN-to-XX and responds (LM-Translate), NLLB translates prompt EN-to-XX and LM responds (NLLB-Translate-then-Respond), and NLLB translates both prompt and response (NLLB-Translate-Both). Refer to caption: 2604.11290v1/x11.png Figure: Figure 7: Effect of weighing intrinsic and extrinsic metrics in PG-Score. Model rankings remain relatively stable across neighboring weightings of intrinsic and extrinsic metrics. Refer to caption: 2604.11290v1/x12.png ### G.4 Weighing of Intrinsic and Extrinsic Metrics in PG-Score Our PG-Score formulation uses an assumption-free and equal weighing scheme between the intrinsic ($\mathcal{I}$) and extrinsic ($\mathcal{E}$) metrics. In this section, we test whether these two metrics capture (1) complementary aspects of teacher effectiveness and (2) how model rankings differ if one metric is weighted more than the other. #### Setup In order to test whether each metric captures complementary aspects of teacher effectiveness, we compute the Spearman rank correlation ($\rho$) between the intrinsic and extrinsic metrics across all teacher-language pairs (N=60, 10 models $\times$ 6 languages). In addition, in order to test the effect of weighing one metric against the other, we formulate a generalized version of PG-Score: $$ $\begin{split}\text{{PG-Score}{}}_{T,\ell}&=\alpha\mathcal{I}+(1-\alpha)\mathcal{E}\\ &\text{where }0\leq\alpha\leq 1\end{split}$ (4) $$ Note that the experiments in §[3](#S3) and §[4](#S4) assume $\alpha=0.5$. We compute the PG-Score across $\alpha=\{0.00,0.25,0.50,0.75,1.00\}$ and then test the resulting model ranks’ $\rho$ across all pairs of $\alpha$. We perform this experiment on all teacher-language pairs where students are finetuned from the OLMo 3 7B base model (N=30, 10 models $\times$ 6 languages). **Table 17: Inference settings for each teacher model. Generation parameters are based on model provider recommendations from HuggingFace and/or official documentation. The Default row indicates parameters used when model-specific recommendations are unavailable. The “–” symbol indicates the parameter was not specified in the official recommendations.** | | Generation Parameters | | | | | --- | --- | --- | --- | --- | | Model Name | Temperature | Top-p | Top-k | Max Seq Len | | GPT-4o mini | 0.8 | 0.9 | – | 16,384 | | Llama 3.1 70B Instruct | 0.6 | 0.9 | – | 131,072 | | Llama 3.1 8B Instruct | 0.6 | 0.9 | – | 131,072 | | Command A | 0.3 | – | – | 128,000 | | Aya Expanse 32B | 0.3 | – | – | 128,000 | | Gemma 3 27B Instruct | 1.0 | 0.95 | 64 | 8,192 | | Gemma 3 12B Instruct | 1.0 | 0.95 | 64 | 8,192 | | Gemma 3 4B Instruct | 1.0 | 0.95 | 64 | 8,192 | | IBM Granite 4.0 | 0.0 | – | – | 4,096 | | IBM Granite Micro | 0.0 | – | – | 4,096 | | Default | 0.8 | 0.9 | – | – | #### Results Intrinsic and extrinsic metrics show a moderate positive correlation (Spearman $\rho$ = 0.41, p < 0.01), suggesting that data quality metrics are predictive of student performance while capturing complementary information. This finding motivates our combined PG-Score computation. In addition, teacher rankings are stable for nearby weighting schemes ($\rho\geq 0.90$ for adjacent $\alpha$ values) as shown in [Figure 7](#A7.F7). Our finding suggests that model rankings are robust to small changes in the weighing of intrinsic and extrinsic metrics. Our equal weighting ($\alpha=0.5$) balances both perspectives, correlating strongly with extrinsic-focused ($\rho=0.89$) and reasonably with intrinsic-focused ($\rho=0.74$) rankings. ### G.5 Effect of language resource levels on PG-Score #### Setup For each language, we consider the following properties drawn from prior work: CommonCrawl (CC) percentage as a proxy for presence in pretraining data (% in CC, Raffel et al., 2020), and linguistic resource availability (score from 1–5, 5 as high-resource, obtained from the LDC Catalog and the ELRA Map, Joshi et al., 2020). We compute the Spearman rank correlation ($\rho$) between each property and PG-Score across all teacher-language pairs (N=60, 10 models $\times$ 6 languages). #### Results [Figure 8](#A7.F8) shows the relationship between a language’s percentage in CommonCrawl and PG-Score. We observe a suggestive positive trend between CommonCrawl representation and PG-Score ($\rho=$0.886, $p<$0.05). This finding suggests that languages with greater presence in pretraining data enable teacher models to generate higher-quality synthetic data that leads to better student performance. This finding is unsurprising, but it provides empirical evidence of a structural gap that inhibits quality synthetic data generation for long-tail languages. In contrast, we do not find a significant correlation between resource availability and PG-Score ($\rho=$0.372, $p=$0.468). Our findings suggest that teacher model generation quality depends more heavily on pretraining exposure than linguistic resources. Additionally, the data sources from Joshi et al. (2020) do not reflect the current landscape: recent LMs are trained on either publicly-available datasets from HuggingFace or in-house datasets. While our work includes 6 diverse languages, the sample size remains limited; we encourage future work to expand the number of languages to validate these findings. Figure: Figure 8: Relationship between a language’s percentage in CommonCrawl and PG-Score. We observe a suggestive positive trend ($\rho=0.886$, p$<$0.05) between CommonCrawl representation and PG-Score across the six languages tested. Refer to caption: 2604.11290v1/x13.png ## Appendix H Disclosure on the Use of LLMs We used Claude (Anthropic, 2024) to assist with editing, title ideation, and proofreading portions of this work. All scientific claims and interpretations are solely our own. We reviewed and revised all LLM-assisted text. ## Appendix I Multilingual Synthetic Data Recipe: Case Study on Tagalog As an application of our findings and discussion in §[5](#S5), we present a case study on developing a multilingual synthetic data recipe on a held-out language: Tagalog. It is a mid-resource language (Category 3 in Joshi et al. (2020)’s taxonomy) and the standardized form of Filipino, the national language of the Philippines. ### I.1 Setup: Recipe Design and Evaluation #### Data We collect Filipino seed data from various publicly-available SFT datasets such as WildChat 4.8M and the Aya Collection. In addition, we also include English data from the Tülu 3 SFT dataset for the Translate method. [Table 18](#A9.T18) shows the statistics of the seed dataset used for Tagalog synthetic data generation. Then, we implement the following data interventions based on our findings: - • Teacher Model: we use Gemma 3 27B Instruct as the teacher model, as it was the best-performing model across most target languages we evaluated (§[3](#S3)). - • Data Generation Method: we use the Translate and Respond methods, as they were the best-performing methods for mid-resource languages like Indonesian (§[3.3](#S3.SS3)). In addition, we add a small sample of prompt-response pairs synthesized via the Generate method. - • Synthetic Data Scale: we generate 10k synthetic examples using the selected teacher and data generation method, as we found that this scale is sufficient to achieve strong student performance ([Appendix G.1](#A7.SS1)). However, we also test on finetuning a model with 25k synthetic examples to see if more data improves performance. - • Student Base Model: we finetune using the Gemma 3 4B model, as we find that family-matched teacher-student pairs yield higher PG-Score (§[3.2](#S3.SS2)). **Table 18: Tagalog seed dataset statistics. In order to bootstrap the synthetic data generation recipe for Tagalog, we curate a seed dataset containing a mix of Tagalog and English prompts from various sources. Majority of the seed dataset is from the TaCo paper (Upadhayay and Behzadan, 2024).** | Source | Num. Instances | | --- | --- | | TaCo Alpaca | 10,000 | | Aya Collection | 1,241 | | WildChat 4.8M | 997 | | WildChat 1M | 250 | For the purposes of this report, we will designate the model finetuned on Gemma 3 4B using our synthetic recipe as 10K-Polyglot-TL, where “10K” indicates the number of SFT instances used during finetuning. #### Evaluation We evaluate on FilBench (Miranda et al., 2025), a benchmark for LMs that includes Filipino-centric multiple-choice and generative tasks. It measures an LM’s performance across four categories such as classical NLP, cultural knowledge, reading comprehension, and generation, alongside an aggregated FilBench score. We also compare against two data mix baselines: - 1. 10K-Public: we sample 10k Tagalog prompt-response pairs from the seed dataset. This baseline aims to simulate a non-synthetic data approach to training multilingual LMs. - 2. 10K-GPT-4oM: we synthesize 10k instances using an off-the-shelf teacher model (GPT-4o-mini). This baseline simulates a typical data generation approach of choosing a teacher in an ad hoc manner due to its perceived strength (size or benchmark performance) or ease of use. For all methods, we finetune a Gemma 3 4B base model using the same training settings indicated in [Appendix E.1](#A5.SS1). ### I.2 Results: Leaderboard Scores and Ablations Figure: Figure 9: Student model performance on a held-out language (Tagalog) across several synthetic data interventions. Given a held-out language (Tagalog) and an evaluation benchmark (FilBench), we apply data interventions based on our recommendations on creating a multilingual synthetic data recipe (§[5](#S5)). Refer to caption: 2604.11290v1/x14.png **Table 19: Model performance on a held-out language (Tagalog) as evaluated on FilBench (Miranda et al., 2025). We compare our optimal synthetic recipe against baseline approaches and other models in the same parameter range.** | Model | FilBench Score | | --- | --- | | GPT-4o (2024-08-06) | 74.27 | | Gemma 3 27B Inst. | 55.17 | | Gemma 3 12B Inst. | 54.04 | | 25K-Polyglot-TL 4B | 49.73 | | 10K-Polyglot-TL 4B | 49.52 | | Qwen 3 4B | 48.42 | | 10K-GPT-4oM | 47.67 | | Llama 3.1 8B Inst. | 47.38 | | Ministral 8B Inst. | 47.33 | | 10K-Public | 47.24 | | Pangea 7B | 43.98 | | SeaLLMs 3 1.5B | 43.20 | [Table 19](#A9.T19) shows the FilBench score of our optimal synthetic recipe compared to other models in the same parameter range. We find that 10K-Polyglot-TL is competitive against 10K-GPT-4oM (+1.85pp), and has better performance compared to 10K-Public (+2.28pp). These results suggest that (1) synthetic data generation is a viable approach for building less-resource language models, and (2) our finding that selecting strong teacher models based on PG-score is effective, as larger models do not always produce better training data (§[3](#S3)). In addition, comparing 10K-Polyglot-TL to other models in the FilBench leaderboard(^5^55Official FilBench leaderboard: [https://hf.co/spaces/filbench/filbench-leaderboard](https://hf.co/spaces/filbench/filbench-leaderboard)) shows that the former is competitive against Qwen 3 4B and Llama 3.1 8B Instruct. We highlight that our 4B models are competitive against other models with larger parameter sizes, suggesting that a multilingual synthetic data recipe based on our PG-Score findings is data-efficient. We also find that increasing the number of SFT instances (10k to 25k) led to a performance increase of 0.21pp. While we previously found that 10K instances showed diminishing returns (see [Appendix G.1](#A7.SS1)), the continued gains from scaling to 25K instances on FilBench suggest that saturation points may depend on task diversity. FilBench covers a broader range of NLP tasks (e.g., named-entity recognition) compared to our experimental benchmarks in §[3](#S3) and [Appendix G](#A7), indicating that practitioners working with diverse task distributions may benefit from exploring larger synthetic datasets beyond the 10K threshold. ### I.3 Analysis: Ablation Experiments In order to measure the contribution of our findings and recommendations in §[5](#S5), we perform the following ablation experiments as shown in [Figure 9](#A9.F9). Note that the interventions described below are additive. #### Curation of publicly-available data vs. Synthetic data generation We compare student models trained on (1) publicly-available Tagalog SFT data and (2) synthetic SFT instances generated by a GPT-4o teacher (note that these are also the same baselines in [Appendix I.2](#A9.SS2)). We find that the performance of these two baselines are similar ($\Delta=0.5\text{pp}$), suggesting that there is no significant advantage to using a synthetic data pipeline if the teacher model is not optimal. We also hypothesize that some publicly-accessible datasets in Tagalog were semi-synthetic (e.g., TaCO uses a synthetic pipeline akin to the Translate method, but using chain-of-thought to improve the quality of translations), making it difficult to perform a fair comparison. #### Using a teacher with a higher PG-Score We then swap the GPT-4o-mini teacher with Aya Expanse 32B, a teacher with a higher PG-Score based on our main findings (0.461 vs. 0.706, c.f. §[3](#S3), [Table 1](#S2.T1)). We observe a slight performance improvement in this intervention, suggesting that the PG-Score metric is generalizable across an unseen language. #### Matching teacher and student model families One of our key findings and recommendation is to match the model families of the teacher and the student (§[3.2](#S3.SS2)). We use a Gemma 3 Instruct 27B teacher model to match the family of the Gemma 3 4B base model. This intervention yields a substantial performance improvement, demonstrating that family alignment is a reliable heuristic for teacher selection. The improvement from family matching is consistent with our findings that family-matched pairs achieve at least +20.5% higher PG-Score compared to mismatched pairs, likely due to shared tokenization schemes and architectural similarities that facilitate better knowledge transfer from teacher to student. #### Increase data scale We increase the number of synthetic instances from 10k to 25k to assess whether additional data continues to improve performance. We observe a modest gain of 0.21pp, which is smaller than the improvements from teacher model selection and model family matching. This finding aligns with our earlier observation that gains diminish beyond 10k examples ([Appendix G.1](#A7.SS1)), though the continued improvement on FilBench’s diverse task distribution suggests that saturation points may be task-dependent. #### Increase model scale Finally, we explore whether scaling the student model from 4B to 12B (and 27B) parameters provides additional performance gains. We find that the larger student model achieves higher performance, demonstrating that our synthetic data recipe benefits from increased model capacity. This result is consistent with our generalization experiments ([Appendix G.2](#A7.SS2)), where we showed that PG-Score generalizes across different model sizes while maintaining the relative ranking of teacher models. However, we note that the performance of our best models are still behind Gemma 3 27B Instruct and Gemma 3 12B Instruct ([Table 19](#A9.T19)). Given that observation, we still argue that our synthetic pipeline, which uses 25K instances trained only via SFT, can be considered data and resource-efficient compared to the post-training interventions done in Gemma 3, which involved instruction-tuning and reinforcement learning objectives (Gemma Team et al., 2025). ## Appendix J Inference Details #### Prompt templates [Figure 10](#A10.F10) to [Figure 12](#A10.F12) show the prompt templates used for each data generation method. In addition, [Figure 13](#A10.F13) shows the prompt template used for the LLM-as-a-judge method to evaluate text quality. #### Inference settings We use vLLM (Kwon et al., 2023) and Curator (Marten et al., 2025) for inference. For each teacher model, we check whether the model provider recommended best settings for usage. If not, then we set a default configuration (temperature=0.8, top_p=0.9). [Table 17](#A7.T17) summarizes the inference settings we used for each teacher model. Figure: Figure 10: Prompt template for the Generate data generation method. Figure: Figure 11: Prompt template for the Translate data generation method. Figure: Figure 12: Prompt template for the Respond data generation method. Figure: Figure 13: We evaluate text quality of synthesized texts using a multilingual rubric model called M-Prometheus (Pombal et al., 2025). We choose M-Prometheus due to its strong performance on multilingual and human-aligned benchmarks.