Title: 2502.06589 ArXiv: 2502.06589 Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models Through Continual Pre-Training Hephaestus : Improving Fundamental Agent Capabilities of Large Language Models Through Continual Pre-Training Yuchen Zhuang 1 Jingfeng Yang 2 Haoming Jiang 2 Xin Liu 2 Kewei Cheng 2 Sanket Lokegaonkar 2 Yifan Gao 2 Qing Ping 2 Tianyi Liu 2 Binxuan Huang 2 Zheng Li 2 Zhengyang Wang 2 Pei Chen 2 Ruijie Wang 2 Rongzhi Zhang 1 Nasser Zalmout 2 Priyanka Nigam 2 Bing Yin 2 Chao Zhang 1 1 Georgia Institute of Technology 2 Amazon Work done during Yuchen’s internship at Amazon. Correspondence to: Yuchen Zhuang ( yczhuang@gatech.edu ), Jingfeng Yang ( jingfengyangpku@gmail.com ), Chao Zhang ( chaozhang@gatech.edu ). Abstract Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous agents typically rely on complex prompting or extensive fine-tuning, which often fails to introduce new capabilities while preserving strong generalizability. We introduce Hephaestus-Forge , the first large-scale pre-training corpus designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adapting to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories to strengthen intrinsic reasoning. To explore effective training protocols, we investigate scaling laws to identify the optimal recipe in data mixing ratios. By continual pre-training on Hephaestus-Forge , Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks, demonstrating the effectiveness of our pre-training corpus in enhancing fundamental agentic capabilities and generalization of LLMs to new tasks or environments. Hephaestus : Improving Fundamental Agent Capabilities of Large Language Models Through Continual Pre-Training Yuchen Zhuang 1 † † thanks: Work done during Yuchen’s internship at Amazon. Correspondence to: Yuchen Zhuang ( yczhuang@gatech.edu ), Jingfeng Yang ( jingfengyangpku@gmail.com ), Chao Zhang ( chaozhang@gatech.edu ). Jingfeng Yang 2 Haoming Jiang 2 Xin Liu 2 Kewei Cheng 2 Sanket Lokegaonkar 2 Yifan Gao 2 Qing Ping 2 Tianyi Liu 2 Binxuan Huang 2 Zheng Li 2 Zhengyang Wang 2 Pei Chen 2 Ruijie Wang 2 Rongzhi Zhang 1 Nasser Zalmout 2 Priyanka Nigam 2 Bing Yin 2 Chao Zhang 1 1 Georgia Institute of Technology 2 Amazon 1 Introduction Figure 1: Training paradigms of LLM agents. Prompting alone fails to introduce new knowledge and capabilities, while heavy fine-tuning can hinder generalization and degrade performance in non-agent use cases, potentially suppressing the original base model capabilities. Large language models (LLMs) are rapidly evolving beyond traditional natural language processing tasks Ouyang et al. ( 2022 ); Brown et al. ( 2020 ); Achiam et al. ( 2023 ) , demonstrating increasing intelligence and autonomy by exhibiting capabilities in perception, reasoning, planning, and action within complex real-world environments Yao et al. ( 2023 ); Lu et al. ( 2024 ); Sun et al. ( 2024a ) . Through well-crafted prompting or extensive post-training, LLM-based autonomous agents augmented with external tools ( e.g. , APIs) have demonstrated exceptional instruction-following capabilities in a wide range of tasks (Schick et al., 2024 ; Qin et al., 2024 ; Srinivasan et al., 2023 ; Zeng et al., 2023 ) . Despite their remarkable task-specific performance, existing LLM agents often face the following challenges: (1) Overemphasis on instruction fine-tuning while ignoring the pre-training stage. LLMs typically undergo a two-stage training process: pre-training to learn general knowledge and instruction fine-tuning to align to specific tasks and user preferences. The Superficial Alignment Hypothesis (Zhou et al., 2024 ; Gudibande et al., 2024 ; Lin et al., 2024b ) posits that LLMs acquire most of their knowledge during pre-training, which is more important than instruction fine-tuning in terms of obtaining generalizable fundamental capabilities. However, the majority of existing agent frameworks (Figure 1 ) focus on instruction fine-tuning to align with specific patterns or formats, rather than fundamentally enhancing model knowledge or capabilities ( e.g. , API function calling). (2) Scarcity of agent-oriented pre-training data. Agent instructions and trajectories significantly differ from general instructions and responses Zhang et al. ( 2024b ) . Thus, function-calling knowledge is difficult to derive directly from web archives, the primary pre-training data source. This notable lack of agent-specific pre-training corpora constrains LLMs from effectively acquiring new agentic knowledge and capabilities (Table 1 ). (3) Limited generalization across multiple tasks. LLM agents often struggle to generalize to new scenarios ( e.g. , from single to multiple tools) that differ from their original fine-tuning data distributions Qin et al. ( 2024 ) . To address these challenges, we introduce Hephaestus-Forge , a large-scale pre-training corpus specifically designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adaptation to environmental feedback. Specifically, we focus on two primary objectives: (a) improving comprehension of individual function calls , and (b) strengthening intrinsic reasoning capabilities for solving problems requiring multiple function calls. To enhance (a) comprehension of API functions and alignment with their formats, we collect a large-scale dataset of tool documentation tailored for LLM pre-training on API function calls. Given the expanding range of tasks with growing complexity, we incorporate a vast number of function calling trajectories to improve (b) intrinsic reasoning abilities in sequencing API function calls. We then integrate this meticulously curated tool documentation and function-calling data with code (to bolster reasoning capabilities) and text data (to maintain robust text generation capabilities), creating a multi-source , large-scale , and high-quality training corpus, Hephaestus-Forge . Building upon Hephaestus-Forge , we introduce a continual pre-trained open-source LLM, Hephaestus , an LLM with strong agentic and autonomous capabilities across domains, bringing open-source models closer to the capabilities of commercial LLMs. Our empirical evaluations demonstrate that Hephaestus-8B outperforms open-source LLMs at small to medium scales ( e.g. , 9.6 % percent 9.6 9.6\% 9.6 % over LLaMA-3-8B and 17.6 % percent 17.6 17.6\% 17.6 % over Mixtral-8x22B ) and performs comparably to API-based large commercial LLMs ( e.g. , 18.9 % percent 18.9 18.9\% 18.9 % over Claude-3-Haiku and 4.1 % percent 4.1 4.1\% 4.1 % over GPT-3.5-turbo ) across three agent benchmarks. Our large-scale ablation studies further demonstrate the effectiveness of retrieved agent data in scaling up and diversifying the coverage of scenarios in pre-training. Our contributions can be summarized as follows: • We curate Hephaestus-Forge , a large-scale pre-training corpus designed to enhance understanding of API function calls and guide actionable trajectories for LLM agents. Remarkably, through exhaustive scaling law experiments, we discover a pioneering pre-training recipe with an empirically optimal data mix ratio. • We propose Hephaestus , a foundation model that exhibits enhanced fundamental agentic capabilities, including API function calling, intrinsic reasoning and planning, and adaptation to environmental feedback, achieved through continual pre-training on Hephaestus-Forge . • We extensively compare Hephaestus with strong baselines across three agent benchmarks, verifying its enhanced fundamental agentic capabilities and superior generalization derived from Hephaestus-Forge . 2 Related Work Methods Datasets Training Paradigm # PT Data (Tokens) # IFT Data (Samples) # APIs Code Nat. Lang. Action Traj. API Doc. Func. Call Multi. Step Plan Refine Multi. Turn Instruction Finetuning-based LLM Agents for Intrinsic Reasoning FireAct Chen et al. ( 2023 ) FireAct IFT - 2.1K 10 ✗ ✔ ✔ ✗ ✔ ✗ ✔ ✗ ToolAlpaca Tang et al. ( 2023 ) ToolAlpaca IFT - 4.0K 400 ✗ ✔ ✔ ✗ ✔ ✗ ✔ ✗ ToolLLaMA Qin et al. ( 2024 ) ToolBench IFT - 12.7K 16,464 ✗ ✔ ✔ ✗ ✔ ✔ ✔ ✔ AgentEvol (Xi et al., 2024 ) AgentTraj-L IFT - 14.5K 24 ✗ ✔ ✔ ✗ ✔ ✗ ✗ ✔ Lumos Yin et al. ( 2024 ) Lumos IFT - 20.0K 16 ✗ ✔ ✔ ✗ ✔ ✔ ✗ ✔ Agent-FLAN Chen et al. ( 2024b ) Agent-FLAN IFT - 24.7K 20 ✗ ✔ ✔ ✗ ✔ ✔ ✗ ✔ AgentTuning (Zeng et al., 2023 ) AgentInstruct IFT - 35.0K - ✗ ✔ ✔ ✗ ✔ ✗ ✗ ✔ Instruction Finetuning-based LLM Agents for Function Calling NexusRaven (Srinivasan et al., 2023 ) NexusRaven IFT - - 116 ✔ ✔ ✔ ✗ ✔ ✗ ✗ ✗ Gorilla (Patil et al., 2023 ) Gorilla IFT - 16.0K 1,645 ✔ ✗ ✗ ✔ ✔ ✗ ✗ ✗ OpenFunctions-v2 (Patil et al., 2023 ) OpenFunctions-v2 IFT - 65.0K - ✔ ✔ ✗ ✔ ✔ ✗ ✗ ✗ API Pack Guo et al. ( 2024b ) API Pack IFT - 1.1M 11,213 ✔ ✗ ✔ ✗ ✔ ✗ ✗ ✗ LAM (Zhang et al., 2024a ) AgentOhana IFT - 42.6K - ✔ ✔ ✔ ✗ ✔ ✗ ✔ ✔ xLAM (Liu et al., 2024e ) APIGen IFT - 60.0K 3,673 ✔ ✔ ✔ ✗ ✔ ✗ ✔ ✔ Pretraining-based LLM Agents Hephaestus Hephaestus-Forge PT 103B 95.0K 76,537 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Table 1: Summary of existing instruction finetuning-based LLM agents for intrinsic reasoning and function calling, along with their training resources and sample sizes. "PT" and "IFT" denote "Pre-Training" and "Instruction Fine-Tuning", respectively. Prompting-based LLM Agents. Due to the lack of agent-specific pre-training corpus, existing LLM agents rely on either prompt engineering Hsieh et al. ( 2023 ); Lu et al. ( 2024 ); Yao et al. ( 2023 ); Wang et al. ( 2023 ) or instruction fine-tuning Chen et al. ( 2023 ); Zeng et al. ( 2023 ) to understand human instructions, decompose high-level tasks, generate grounded plans, and execute multi-step actions. However, prompting-based methods mainly depend on the capabilities of backbone LLMs (usually commercial LLMs), failing to introduce new knowledge and struggling to generalize to unseen tasks Sun et al. ( 2024a ); Zhuang et al. ( 2024a ) . Instruction Finetuning-based LLM Agents. Considering the extensive diversity of APIs and the complexity of multi-tool instructions, tool learning inherently presents greater challenges than natural language tasks, such as text generation Qin et al. ( 2024 ) . Post-training techniques focus more on instruction following and aligning output with specific formats Patil et al. ( 2023 ); Hao et al. ( 2024 ); Qin et al. ( 2024 ); Schick et al. ( 2024 ) , rather than fundamentally improving model knowledge or capabilities. Moreover, heavy fine-tuning can hinder generalization or even degrade performance in non-agent use cases, potentially suppressing the original base model capabilities Ghosh et al. ( 2024 ) . Pretraining-based LLM Agents. While pre-training serves as an essential alternative, prior works Nijkamp et al. ( 2023 ); Roziere et al. ( 2023 ); Xu et al. ( 2024 ); Patil et al. ( 2023 ) have primarily focused on improving task-specific capabilities ( e.g. , code generation) instead of general-domain LLM agents, due to single-source, uni-type, small-scale, and poor-quality pre-training data. Existing tool documentation data for agent training either lacks diverse real-world APIs Patil et al. ( 2023 ); Tang et al. ( 2023 ) or is constrained to single-tool or single-round tool execution. Furthermore, trajectory data mostly imitate expert behavior or follow function-calling rules with inferior planning and reasoning, failing to fully elicit LLMs’ capabilities and handle complex instructions Qin et al. ( 2024 ) . Given a wide range of candidate API functions, each comprising various function names and parameters available at every planning step, identifying globally optimal solutions and generalizing across tasks remains highly challenging. 3 Preliminaries (a) Hephaestus-Forge (b) Tool Data (c) Retrieved Data (d) t-SNE: Retrieved Data Figure 2: Data composition of (a) the entire Hephaestus-Forge , (b) seed data collection ( § 4.1 ), and (c) retrieved agent data from the open web ( § 4.2 ). A t-SNE visualization (d) depicts seed data ( colorful points, with each color representing different data sources), retrieved data ( black ), and general text ( gray ) within the semantic space, where retrieved data is closer to the selected seed data than to the general text. Detailed data sources are in § A.1 . Problem Formulation. We conceptualize leveraging LLMs as autonomous agents for problem-solving as a planning process. Initially, we augment the LLM agent with access to a pool of candidate API functions, denoted as 𝒜 = { API 0 , API 1 , ⋯ , API m \mathcal{A}=\{\text{API}_{0},\text{API}_{1},\cdots,\text{API}_{m} caligraphic_A = { API start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , API start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , API start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT }, along with a natural language task description g ∈ 𝒢 𝑔 𝒢 g\in\mathcal{G} italic_g ∈ caligraphic_G from the task space 𝒢 𝒢 \mathcal{G} caligraphic_G . The objective of the LLM agent is to translate the task description g 𝑔 g italic_g into an ordered sequence of T g subscript 𝑇 𝑔 T_{g} italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT API function calls p g = { a 0 , ⋯ , a T g } subscript 𝑝 𝑔 subscript 𝑎 0 ⋯ subscript 𝑎 subscript 𝑇 𝑔 p_{g}=\{a_{0},\cdots,a_{T_{g}}\} italic_p start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT = { italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUBSCRIPT } . Specifically, considering the task description g 𝑔 g italic_g as the initial state s 0 subscript 𝑠 0 s_{0} italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , we then sample the plan p g subscript 𝑝 𝑔 p_{g} italic_p start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT by prompting the LLM agent with the API definitions ℐ ℐ \mathcal{I} caligraphic_I and demonstration samples 𝒟 𝒟 \mathcal{D} caligraphic_D as follows: p g ∼ ρ ⁢ ( a 0 , a 1 , ⋯ , a T g | s 0 ; ℐ , 𝒟 ) : 𝒢 × ℐ × 𝒟 → Δ ⁢ ( 𝒜 T g ) : similar-to subscript 𝑝 𝑔 𝜌 subscript 𝑎 0 subscript 𝑎 1 ⋯ conditional subscript 𝑎 subscript 𝑇 𝑔 subscript 𝑠 0 ℐ 𝒟 → 𝒢 ℐ 𝒟 Δ superscript 𝒜 subscript 𝑇 𝑔 p_{g}\sim\rho(a_{0},a_{1},\cdots,a_{T_{g}}|s_{0};\mathcal{I},\mathcal{D}):% \mathcal{G}\times\mathcal{I}\times\mathcal{D}\to\Delta(\mathcal{A}^{T_{g}}) italic_p start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT ∼ italic_ρ ( italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ; caligraphic_I , caligraphic_D ) : caligraphic_G × caligraphic_I × caligraphic_D → roman_Δ ( caligraphic_A start_POSTSUPERSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) , where Δ ⁢ ( ⋅ ) Δ ⋅ \Delta(\cdot) roman_Δ ( ⋅ ) denotes a probability simplex function. The final output is derived after executing the entire plan y ∼ π ⁢ ( y | s 0 , a 1 , a 2 , ⋯ , a T g ) similar-to 𝑦 𝜋 conditional 𝑦 subscript 𝑠 0 subscript 𝑎 1 subscript 𝑎 2 ⋯ subscript 𝑎 subscript 𝑇 𝑔 y\sim\pi(y|s_{0},a_{1},a_{2},\cdots,a_{T_{g}}) italic_y ∼ italic_π ( italic_y | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) , where π ⁢ ( ⋅ ) 𝜋 ⋅ \pi(\cdot) italic_π ( ⋅ ) denotes a plan executor. During this procedure, we focus on three fundamental capabilities of LLM agents: Accurate Function Calling. It involves accurately understanding the API definitions and demonstration samples to generate correct API function calls with corresponding parameters in a given scenario. Specifically, the model should accurately understand the API definitions ℐ ℐ \mathcal{I} caligraphic_I and demonstration samples 𝒟 𝒟 \mathcal{D} caligraphic_D , as well as generate an accurate API function call in the given scenario p ⁢ ( a t | s 0 , a 1 , ⋯ , a t − 1 , ℐ , 𝒟 ) 𝑝 conditional subscript 𝑎 𝑡 subscript 𝑠 0 subscript 𝑎 1 ⋯ subscript 𝑎 𝑡 1 ℐ 𝒟 p(a_{t}|s_{0},a_{1},\cdots,a_{t-1},\mathcal{I},\mathcal{D}) italic_p ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , caligraphic_I , caligraphic_D ) , where a t subscript 𝑎 𝑡 a_{t} italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is the ground-truth API function call with corresponding parameters at t 𝑡 t italic_t -th step. Intrinsic Reasoning and Planning. It refers to the intrinsic reasoning and planning ability to devise a sequence of multiple tool functions as a solution when addressing complex (multi-step) real-world problems. In such cases, LLMs are often required to generate a sequence of API function calls, p ⁢ ( a 1 , a 2 , ⋯ , a T g | s 0 ; ℐ , 𝒟 ) 𝑝 subscript 𝑎 1 subscript 𝑎 2 ⋯ conditional subscript 𝑎 subscript 𝑇 𝑔 subscript 𝑠 0 ℐ 𝒟 p(a_{1},a_{2},\cdots,a_{T_{g}}|s_{0};\mathcal{I},\mathcal{D}) italic_p ( italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ; caligraphic_I , caligraphic_D ) , where { a 1 , a 2 , ⋯ , a T g } subscript 𝑎 1 subscript 𝑎 2 ⋯ subscript 𝑎 subscript 𝑇 𝑔 \{a_{1},a_{2},\cdots,a_{T_{g}}\} { italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , ⋯ , italic_a start_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT end_POSTSUBSCRIPT } constitutes the ground-truth solution plan of length T g subscript 𝑇 𝑔 T_{g} italic_T start_POSTSUBSCRIPT italic_g end_POSTSUBSCRIPT . This process relies on intrinsic reasoning embedded within the model parameters; enhanced reasoning capabilities lead to a solution plan with a higher chance of success. Adaptation with Environment Feedback. It focuses on adapting the current plan or action based on environmental feedback when the environments support interaction with the LLM agent. When such feedback is available, it is crucial for the agent to adjust its actions accordingly: p ⁢ ( a t | s 0 , a 1 , o 1 , a 2 , ⋯ , o t − 1 ; ℐ , 𝒟 ) 𝑝 conditional subscript 𝑎 𝑡 subscript 𝑠 0 subscript 𝑎 1 subscript 𝑜 1 subscript 𝑎 2 ⋯ subscript 𝑜 𝑡 1 ℐ 𝒟 p(a_{t}|s_{0},a_{1},o_{1},a_{2},\cdots,o_{t-1};\mathcal{I},\mathcal{D}) italic_p ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_o start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , ⋯ , italic_o start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ; caligraphic_I , caligraphic_D ) , where o k subscript 𝑜 𝑘 o_{k} italic_o start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT represents the feedback from the environment after the k 𝑘 k italic_k -th action. Incorporating environmental feedback allows the agent to take reflections to refine its plan and improve task performance iteratively. 4 Hephaestus-Forge To scale and diversify the pre-training corpus for LLM agents, we introduce a three-stage construction process for Hephaestus-Forge (see Figure 2 ): (1) Seed Data Collection ( § 4.1 ), where we gather initial high-quality samples; (2) Web Data Retrieval ( § 4.2 ), which expands the seed data by retrieving relevant data from the web; and (3) Data Quality Control ( § 4.3 ), where we ensure the integrity and relevance of the collected data. 4.1 Seed Data Collection For seed data collection, we first traverse available public resources to gather high-quality API documentation and action trajectories, including: (1) Public APIs. High-quality API documentation is collected from over 1 , 400 1 400 1,400 1 , 400 public APIs and official websites, including detailed function definitions and parameter descriptions. (2) Public Repositories. To improve intrinsic reasoning, we integrate action trajectories from over 60 60 60 60 public repositories across diverse domains, such as programming code and web interactions. (3) Code-to-Text Synthesis. Given the limited coverage of curated data, we use LLMs to synthesize additional API documentation from StarCoder-API , generating examples based on code snippets. (4) Simulated Agent Data. We gather simulated action sequences with observational data to facilitate adaptation to environmental feedback. Importantly, we offer step-by-step details of the seed data collection process in § D.1 for reproducibility. 4.2 Web Data Retrieval Given the limited availability of agent-oriented data, we use the high-quality data described in § 4.1 as seed data for further expansion. To enhance agentic capabilities, we retrieve a diverse set of examples from web crawls, focusing on content relevant to API documentation and action trajectories. Our retrieval process involves the following steps: (1) Web Data Corpus Creation. Similar to CommonCrawl Raffel et al. ( 2020 ) and FineWeb (Penedo et al., 2024 ) , we first compile a large-scale web data corpus. (2) Semantic Matching. We utilize COCO-DR Yu et al. ( 2022 ) to encode semantic representations of documents in the seed data and the large-scale web corpus. We then retrieve the top- K 𝐾 K italic_K similar documents by calculating the cosine similarity between the corresponding embeddings. It allows us to identify and retrieve documents from the web corpus that are semantically similar to our seed data, effectively enriching our dataset with relevant and diverse information. (3) Quality Control. To ensure the quality of the retrieved corpus, we perform data pruning to remove semantically redundant content and maintain the diversity of knowledge, preventing overrepresentation of certain topics and ensuring generalization and robustness across domains. 4.3 Data Quality After retrieving semantically relevant data from the web corpus, we obtain a collection of noisy agent data. To ensure the integrity and relevance of our dataset, it is essential to consistently monitor data quality and filter out content that resembles general text rather than agent-specific data. First, we employ Claude-3-Sonnet Anthropic ( 2024 ) as the data annotator to annotate a total of 71 , 473 71 473 71,473 71 , 473 samples from the retrieved data, identifying 37 , 714 37 714 37,714 37 , 714 as agent-relevant and 33 , 767 33 767 33,767 33 , 767 as general text paragraphs. Using the annotated samples, we train a fastText Joulin ( 2016 ) model to effectively recall additional agent-relevant web data. This filtering process then reduces the data volume from approximately 200 200 200 200 B to 80 80 80 80 B tokens, ensuring that the preserved data maintains high relevance and quality. See details in § D.2 . Figure 3: Scaling law of the relationship between agent data mixing ratio ( % percent \% % ) and benchmark loss. Figure 4: Overview of the pre-training (Stages I & II) and instruction fine-tuning (III) framework in Hephaestus . 5 Scaling Laws for Data Composition When designing LLMs, the scaling law Kaplan et al. ( 2020 ); Hoffmann et al. ( 2022 ) is an important predictive tool that can estimate the performance ( e.g. , benchmark loss) of a large-sized target model using a scaling curve fitted over much smaller models (referred to as sampling models). We develop scaling laws to determine the optimal data proportion among agent data, text data, and code data. With the total budget of the data volume fixed, our scaling law experiments show that the effect of agent data ratio x 𝑥 x italic_x on the loss ℒ ℒ \mathcal{L} caligraphic_L of a pre-trained model follows power laws: ℒ = c + k ⁢ x α , ℒ 𝑐 𝑘 superscript 𝑥 𝛼 \displaystyle\mathcal{L}=c+kx^{\alpha}, caligraphic_L = italic_c + italic_k italic_x start_POSTSUPERSCRIPT italic_α end_POSTSUPERSCRIPT , where c 𝑐 c italic_c , k 𝑘 k italic_k , and α 𝛼 \alpha italic_α are parameters to be fitted. By fitting these parameters using a collection of small models, training data, or computational resources, scaling laws can extrapolate to precisely predict the test loss of larger cases over orders of magnitude. Scaling Law Experiments. Concretely, we construct our scaling laws by pre-training models ranging in 45 45 45 45 M to 0.65 0.65 0.65 0.65 B parameters. To simulate the continual pre-training setting, we amplify the target data volume used for training each small model to 50 × 50\times 50 × model parameters. Consequently, the total compute budgets for the scaling law experiments span from 7 × 10 17 7 superscript 10 17 7\times 10^{17} 7 × 10 start_POSTSUPERSCRIPT 17 end_POSTSUPERSCRIPT to 2 × 10 20 2 superscript 10 20 2\times 10^{20} 2 × 10 start_POSTSUPERSCRIPT 20 end_POSTSUPERSCRIPT FLOPs. Regarding data proportions, we begin with the seed agent data and progressively incorporate the retrieved web corpus to increase the agent data ratio. Concurrently, as the agent data ratio increases, we proportionally decrease the volumes of general text and code data to maintain the fixed total data volume. Following Dubey et al. ( 2024 ) , we leverage the benchmark loss of Nexus Srinivasan et al. ( 2023 ) , API-Bank Li et al. ( 2023b ) , API-Bench Patil et al. ( 2023 ) to monitor the agent capabilities, and MMLU Hendrycks et al. ( 2020 ) to monitor the general capabilities of LLMs. Optimal Data Mixing Ratio. Figure 3 illustrates that the optimal mixture of agent data within the entire pre-training corpus is approximately 36%, indicating that the proportion of agent data, text data, and code data should be roughly 1 : 1 : 1 : 1 1 : 1 1:1:1 1 : 1 : 1 . This balanced distribution promotes both specialized agent capabilities and general language understanding, ensuring that the model remains versatile and robust across diverse tasks and domains. Remark. The established scaling laws provide critical insights into the data composition for pre-training LLM agents. By identifying the optimal ratio of agent data, we ensure that the model effectively balances specialized agentic capabilities with general language proficiency. 6 Hephaestus In this section, we propose Hephaestus , a foundation model with enhanced fundamental capabilities of LLM agents. Hephaestus undergoes a two-stage continual pre-training process, followed by instruction fine-tuning (see Figure 4 ): (1) Stage I , continual pre-training stage on the entire Hephaestus-Forge corpus to inject general agent knowledge ( § 6.1 ); (2) Stage II , continual pre-training stage on the high-quality seed set of Hephaestus-Forge to further enhance specific capabilities ( § 6.1 ); and (3) Stage III , instruction fine-tuning to follow general instructions and downstream task requirements ( § 6.2 ). 6.1 Stage I & II: Continual Pre-Training Following Caccia et al. ( 2022 ); Lange et al. ( 2023 ) , we revisit the concept of stability gap , which describes the phenomenon where the performance on old tasks initially drops and then recovers when learning a new task. Specifically, in the continual pre-training of LLMs, if the data distribution shifts too significantly between the initial pre-training and the continual pre-training stages, the model’s capabilities can deteriorate markedly until it assimilates knowledge from the new data distribution Guo et al. ( 2024a ) . To this end, we propose a two-stage continual pre-training framework: Stage I: Injecting General Agent Knowledge. Stage I infuses general agent knowledge, accompanied by commonsense knowledge and code snippets. We pre-train Hephaestus on the entire Hephaestus-Forge , whose data distribution is carefully balanced between general corpus and agent-specific data, facilitating a smooth and gradual integration of agent knowledge. Stage II: Enhancing Agent-Specific Capabilities. Stage II leverages high-quality agent data to further enhance the specific capabilities of an agent LLM, including user interaction, function calling, planning, plan refinement, and coding capabilities. We continually pre-train the model obtained from Stage I on the high-quality seed data in § 4.1 to further align the behavior with agent-specific requirements, ensuring that the specialized functionalities are robustly learned and integrated. Pre-Training Objectives. For both stages, we employ language modeling as the primary pre-training task. The objective is to auto-regressively predict the next token, defined as follows: ℒ PT = − 𝔼 𝐱 ∈ 𝒟 PT ⁢ ∑ i = 1 n p ⁢ ( x i | 𝐱 < i ) , subscript ℒ PT subscript 𝔼 𝐱 subscript 𝒟 PT superscript subscript 𝑖 1 𝑛 𝑝 conditional subscript 𝑥 𝑖 subscript 𝐱 absent 𝑖 \displaystyle\mathcal{L}_{\text{PT}}=-\mathbb{E}_{\mathbf{x}\in\mathcal{D}_{% \text{PT}}}\sum_{i=1}^{n}p(x_{i}|\mathbf{x}_{ Example **API: `easyai.model.base_block.utility.utility_block.ConvBNActivationBlock`** The `ConvBNActivationBlock` is a utility class that combines a convolutional layer, a batch normalization layer, and an activation layer into a single block. This block is commonly used as a building block in various neural network architectures, including the PNASNet architecture demonstrated in the provided code. **Parameters:** - `in_channels` (int): The number of input channels for the convolutional layer. - `out_channels` (int): The number of output channels for the convolutional layer. - `kernel_size` (int or tuple): The size of the convolutional kernel. - `stride` (int or tuple, optional): The stride of the convolutional operation. Default is 1. - `padding` (int or tuple, optional): The padding added to the input tensor before applying the convolution. Default is 0. - `bias` (bool, optional): Whether to include a bias term in the convolutional layer. Default is `False`. - `bnName` (str or `easyai.base_name.block_name.NormalizationType`, optional): The normalization type to be used. Default is `NormalizationType.BatchNormalize2d`. - `activationName` (str or `easyai.base_name.block_name.ActivationType`, optional): The activation function to be used. Default is `ActivationType.ReLU`. **Attributes:** - `conv`: The convolutional layer. - `bn`: The batch normalization layer. - `activation`: The activation layer. **Methods:** - `forward(x)`: Defines the forward pass of the block. It takes an input tensor `x` and applies the convolutional, batch normalization, and activation operations sequentially. - `get_name()`: Returns the name of the block, which is a combination of the class name and a unique index. **Example Usage:** ```python import torch from easyai.model.base_block.utility.utility_block import ConvBNActivationBlock # Create a ConvBNActivationBlock with input channels 3, output channels 64, kernel size 3x3, and stride 1 block = ConvBNActivationBlock(in_channels=3, out_channels=64, kernel_size=3, stride=1) # Generate a random input tensor x = torch.randn(1, 3, 32, 32) # Pass the input tensor through the block output = block(x) ``` In the provided code, the `ConvBNActivationBlock` is used as the first layer of the PNASNet architecture, where it takes the input image data and applies a convolutional operation followed by batch normalization and activation. G.2 Retrieved Data Examples We present two examples of high-quality retrieved data as follows: Example-1 The Cardboard Kitchen : 6 Steps By nicholasniski01 in Craft Cardboard Introduction: The Cardboard Kitchen In this instructable I will show you how to make a Cardboard Kitchen. The Cardboard Kitchen Is almost entirely made out of Cardboard. This Kitchen includes a Stove, Oven, Sink, Dishwasher, Fridge and Microwave. lots of small boxes and a medium size box Step 1: How to Make a Fridge you need to disassemble the medium size box(Get rid of ALL the tape)... Step 2: How to Make a Microwave First get a small box, make a rectangular hole in the box... Step 3: How to Make a Sink First get a small box, cut off the top of the box... Step 4: How to Make a Dishwasher Cut out a square of cardboard for the size of the dishwasher then you color the cardboard black, silver or any other color you would want for the dishwasher... Step 5: How to Make a Stove To make the stove you make a black circle with for lines going out of the circle on an unused section of your big box or \"counter\"... Step 6: How to Make an Oven Get a square the size you want your oven to be... Example-2 manual Prestigio MultiReader 5574 You can create your event and make a plan on your calendar. On the home screen or list menu, tap Calendar. View the calendar On the home screen or list menu, tap Calendar to check the calendar. Tap to change your calendar to Day, Week,Month or Agenda view. Create an event 1. Go to Calendar, select a date. 2. Tap to create a new event. 3. Edit reminder settings. 4. Tap Done to save the event. G.3 Data Quality Filtering Failure Cases We present a failure case of the fastText filter below: Failure Case [TEXT] We’re making it even easier for you to stay connected to 99ROCK wherever you go! Besides tuning in on your radio, you can also stream your favorite station through your computer, smartphone, tablet, and your smart speaker. If you are in or near the Fort Walton Beach-Destin broadcast area, tune your radio to: 99.5 FM Stream 99ROCK at work or home from your computer on one of these web players: Triton Player iHeart Radio TuneIn Listen to 99ROCK on-the-go thru one of these popular streaming apps or thru the 99ROCK mobile app: iOS App Google Play iHeart Radio TuneIn First you need to enable the 99ROCK skill: Say, ``Alexa, enable the ninety-nine rock Skill’’ After you have enabled the Skill, listen to our station just by saying "Alexa, open ninety nine rock" Just say, ``Hey Google, play ninety-nine rock’’ [CATEGORY] Agent In this case, the fastText model incorrectly categorized the text as agent-relevant data. This misclassification likely occurred because fastText relies on gram frequency analysis, and the presence of multiple high-tech terms (e.g., iOS, App, Google Play) in the paragraph may have misled the model. Appendix H Prompt Templates H.1 Prompt Template for Code-to-Text Synthesis Prompt Please use your knowledge to write an API documentation for the given APIs and consider the given code as the example usage. API: {api} Code: {code} API Documentation: H.2 Prompt Template for LLM Annotator in Data Quality Control Prompt Please categorize the given text belong to agent-relevant data or other general text. The definitions are as follows: 1. Agent: Tool documentation text that describes the usage of a tool, software, or API; and action trajectory text that describes a sequence of actions or steps to achieve a goal. 2. General: Other general text that does not belong to the above two categories. Below are some examples: [TEXT] **API: `easyai.model.base_block.utility.utility_block.ConvBNActivationBlock`** The `ConvBNActivationBlock` is a utility class that combines a convolutional layer, a batch normalization layer, and an activation layer into a single block. This block is commonly used as a building block in various neural network architectures, including the PNASNet architecture demonstrated in the provided code.  **Parameters:**  - `in_channels` (int): The number of input channels for the convolutional layer. - `out_channels` (int): The number of output channels for the convolutional layer. - `kernel_size` (int or tuple): The size of the convolutional kernel. - `stride` (int or tuple, optional): The stride of the convolutional operation. Default is 1. - `padding` (int or tuple, optional): The padding added to the input tensor before applying the convolution. Default is 0. - `bias` (bool, optional): Whether to include a bias term in the convolutional layer. Default is `False`. - `bnName` (str or `easyai.base_name.block_name.NormalizationType`, optional): The normalization type to be used. Default is `NormalizationType.BatchNormalize2d`. - `activationName` (str or `easyai.base_name.block_name.ActivationType`, optional): The activation function to be used. Default is `ActivationType.ReLU`. [CATEGORY] Agent [TEXT] I want to deliver a Birthday Gift to my friend in London, UK. Then, I need to book a flight from New York, USA to London, UK on August 1st, 2023 for myself. After arriving in London, I would like to see Dr. Smith for my Migraine. Once my health is in check, I’d like to apply for a Software Engineer job in London. Step 1: Call deliver_package API with package: ’Birthday Gift’ and destination: ’London, UK’ deliver_package(package=Birthday Gift, destination=London, UK) Step 2: Call book_flight API with date: ’2023-08-01’, from: ’New York, USA’ and to: ’London, UK’ book_flight(date=2023-08-01, from=New York, USA, to=London, UK) Step 3: Call see_doctor_online API with disease: ’Migraine’ and doctor: ’Dr. Smith’ see_doctor_online(disease=Migraine, doctor=Dr. Smith) Step 4: Call apply_for_job API with job: ’Software Engineer’ apply_for_job(job=Software Engineer)" [CATEGORY] Agent [TEXT] My New Crock Pot -- Creuzer Leave a comment on My New Crock Pot I went out and got myself a new crock pot. I rather like this one. It has 3 settings, High, Low, and Warm. It is designed to be hauled around even! There are latches on each side of the lid to clip the lid in place. It even came with it’s own spoon that clips into the lid! A really neat feature is that the lid has little tabs so you can set the lid on one of the handles and it won’t go all sliding all over the place. I think this is a winner, I plan on using it a lot for the cooking club I am in. [CATEGORY] General Please categorize the following text into agent-relevant data (Agent) or general text (General). ONLY respond the category name (Agent/General) for each text. If you are unsure, please respond with ’General’. [TEXT] {text} [CATEGORY] Data Source Type Format Tokens (B) URL Link ToolBench Qin et al. ( 2024 ) Traj. Dialog 0.530 https://github.com/OpenBMB/ToolBench AgentInstruct Zeng et al. ( 2023 ) Traj. ReAct 0.002 https://huggingface.co/datasets/THUDM/AgentInstruct Alexa-Arena Gao et al. ( 2024 ) Traj. NL Plan 0.035 https://github.com/amazon-science/alexa-arena/tree/main chat_ego_4d Mu et al. ( 2024 ) Traj. API Seq 0.025 https://github.com/EmbodiedGPT/EgoCOT_Dataset FireAct Chen et al. ( 2023 ) Traj. ReAct 0.002 https://fireact-agent.github.io/ NAT Wang et al. ( 2024c ) Traj. ReAct 0.003 https://github.com/Reason-Wang/NAT ToolAlpaca Tang et al. ( 2023 ) Traj. Plain Text 0.004 https://github.com/tangqiaoyu/ToolAlpaca/tree/main Lumos Yin et al. ( 2024 ) Traj. Dialog 0.109 https://huggingface.co/datasets/ai2lumos/lumos_complex_qa_ground_iterative?row=0 STE Wang et al. ( 2024a ) Traj. Plain Text 0.025 https://github.com/microsoft/simulated-trial-and-error toolbench Xu et al. ( 2023 ) Traj. API Seq 0.010 https://github.com/sambanova/toolbench Gorilla Patil et al. ( 2023 ) Doc. API Seq 0.009 https://gorilla.cs.berkeley.edu/ PublicAPIs Doc. Plain Text 0.008 https://github.com/public-apis/public-apis?tab=readme-ov-file TaskBench Shen et al. ( 2023 ) Traj. NL Plan 0.020 https://github.com/microsoft/JARVIS/tree/main/taskbench RestBench Song et al. ( 2023 ) Traj. API Seq 0.001 https://github.com/Yifan-Song793/RestGPT/tree/main/datasets SayCanPay Hazra et al. ( 2024 ) Traj. NL Plan 0.001 https://github.com/RishiHazra/saycanpay AgentFlan Chen et al. ( 2024b ) Traj. Dialog 0.020 https://github.com/InternLM/Agent-FLAN PlanBench Valmeekam et al. ( 2024 ) Traj. NL Plan 0.001 https://github.com/karthikv792/LLMs-Planning SwftSage Lin et al. ( 2024a ) Traj. NL Plan 0.022 https://github.com/yuchenlin/SwiftSage T-Eval Chen et al. ( 2024a ) Traj. Dialog 0.040 https://github.com/open-compass/T-Eval API-Bank Li et al. ( 2023b ) Traj. API Seq 0.001 https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank JeirchoWorld Ammanabrolu and Riedl ( 2021 ) Traj. NL Plan 0.001 https://github.com/JerichoWorld/JerichoWorld API-Pack Guo et al. ( 2024b ) Traj. API Seq 0.800 https://huggingface.co/datasets/zguo0525/API-Pack/tree/main CodeAct Lv et al. ( 2024 ) Traj. Dialog 0.009 https://huggingface.co/datasets/xingyaoww/code-act UltraTool Huang et al. ( 2024 ) Traj. NL Plan 0.002 https://github.com/JoeYing1019/UltraTool/tree/main Tooleyes Ye et al. ( 2024 ) Doc. JSON 0.001 https://github.com/Junjie-Ye/ToolEyes/tree/main OpenMathInstruct Toshniwal et al. ( 2024 ) Traj. API Seq 0.335 https://huggingface.co/datasets/nvidia/OpenMathInstruct-1 NexasRaven Srinivasan et al. ( 2023 ) Traj. JSON 0.001 https://huggingface.co/Nexusflow Seal-Tools Wu et al. ( 2024 ) Traj. API Seq 0.002 https://github.com/fairyshine/Seal-Tools/tree/master UltraInteract Yuan et al. ( 2024a ) Traj. QA 0.16 https://huggingface.co/datasets/openbmb/UltraInteract_sft?row=0 Python Module Doc. Plain Text 0.001 https://docs.python.org/3.12/ AgentTraj-L Xi et al. ( 2024 ) Traj. Dialog 0.020 https://huggingface.co/datasets/AgentGym/AgentTraj-L MNMs Ma et al. ( 2024 ) Traj. API Seq 0.001 https://huggingface.co/datasets/zixianma/mnms PythonQA-API-Usage Doc. QA 0.003 https://huggingface.co/datasets/RazinAleks/SO-Python_QA-API_USAGE_class APIText Traj. API Seq 0.001 https://huggingface.co/datasets/havens2/apitext StarCoder-APIs Lozhkov et al. ( 2024 ) Traj. Code 6.147 https://huggingface.co/datasets/luna-code/starcoderdata-apis APIs_v2 Traj. API Seq 0.003 https://huggingface.co/datasets/vinilazzari/apis_v2 Ultimate Traj. QA 0.002 https://huggingface.co/datasets/Kris8an/ultimate_apicalls_and_topbot xLAM Zhang et al. ( 2024b ) Doc. QA 0.022 https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k Table 9: Data sources of the seed data in Hephaestus-Forge . Data Source Type Format Tokens (B) URL Link API_doc Doc. Plain Text 0.001 https://huggingface.co/datasets/Prakhar1000/API_Documentation_dataset_alpaanco?row=0 ChatsBug Traj. NL Plan 0.009 https://huggingface.co/datasets/chats-bug/agent_action_plan?row=0 sample_scripts Traj. API Seq 0.002 https://huggingface.co/datasets/prantadi/tokenized_dataset_1024_SampleScripts_deduped_API-ref?row=1 Agent-Trajectories Traj. API Seq 0.001 https://huggingface.co/datasets/Agent-Eval-Refine/Agent-Trajectories/tree/main Agent-Instruct Traj. Dialog 0.056 https://huggingface.co/datasets/sam-mosaic/agent-instruct Agent007 Traj. API Seq 0.001 https://huggingface.co/datasets/DepositorOP/agent007 AgentCode Traj. API Seq 0.010 https://huggingface.co/datasets/AlignmentLab-AI/agentcode syn-web-agent Traj. JSON 0.001 https://huggingface.co/datasets/allyson-ai/synthetic-web-agent syn-llama Traj. Dialog 0.004 https://huggingface.co/datasets/Cyleux/agent-machine-convo-llama-nicholas-2k-gpt4-verified seq-Mind2Web Traj. JSON 1.243 https://huggingface.co/datasets/Izazk/Sequence-of-action-prediction-mind2web syn-gemma Traj. Dialog 0.047 https://huggingface.co/datasets/NickyNicky/function-calling-sharegpt_chatml_gemma_agent LLM Robot Traj. API Seq 0.001 https://huggingface.co/datasets/Aryaduta/llm_robot Verifiers for Code Traj. Plain Text 0.05 https://huggingface.co/datasets/verifiers-for-code/CodeNet-Planner isotonic planner Traj. NL Plan 0.005 https://huggingface.co/datasets/Isotonic/planner_dataset Turing Solutions Traj. NL Plab 0.001 https://huggingface.co/datasets/TuringsSolutions/GlobalFunctionCallingTrainingSetLarge G-PlanET Traj. NL Plan 0.003 https://huggingface.co/datasets/TuringsSolutions/GlobalFunctionCallingTrainingSetLarge Pandas Doc Doc. Plain Text 0.004 https://pandas.pydata.org/ Sugarcrm Doc. Plain Text 0.001 https://huggingface.co/datasets/kaahila/sugarcrm_130_documentation AWS Doc. Plain Text 0.033 https://huggingface.co/datasets/sauravjoshi23/aws-documentation-chunked LangChain Doc. Plain Text 0.005 https://huggingface.co/datasets/jamescalam/langchain-docs-23-06-27 Code Library Doc. Plain Text 0.013 https://huggingface.co/datasets/code-rag-bench/library-documentation PublicAPIs-extend Doc. Plain Text 0.718 https://github.com/public-apis/public-apis?tab=readme-ov-file Torch Doc. Plain Text 0.005 https://pytorch.org/docs/stable/index.html Table 10: Data sources of the seed data in Hephaestus-Forge (Cont’d).