Title: 2604.07789 ArXiv: 2604.07789 ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents Title: Content selection saved. Describe the issue below: Description: License: CC BY 4.0 arXiv:2604.07789v1 [cs.MA] 09 Apr 2026 ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents Kenan Li 1* Qirui Jin 2* Liao Zhu 1 Xiaosong Huang 1 Yijia Wu 1 Yikai Zhang 1 Xin Zhang 1 Zijian Jin 1 Yufan Huang 1 Elsie Nallipogu 1 Chaoyun Zhang 1 Yu Kang 1 Saravan Rajmohan 1 Qingwei Lin 1 Wenke Lee 2 Dongmei Zhang 1 1 Microsoft 2 Georgia Institute of Technology * Equal contribution Abstract Recent advances in language model (LM) agents have significantly improved automated software engineering (SWE). Prior work has proposed various agentic workflows and training strategies as well as analyzed failure modes of agentic systems on SWE tasks, focusing on several contextual information signals: Reproduction Test , Regression Test , Edit Location , Execution Context , and API Usage . However, the individual contribution of each signal to overall success remains underexplored, particularly their ideal contribution when intermediate information is perfectly obtained. To address this gap, we introduce Oracle-SWE, a unified method to isolate and extract oracle information signals from SWE benchmarks and quantify the impact of each signal on agent performance. To further validate the pattern, we evaluate the performance gain of signals extracted by strong LMs when provided to a base agent, approximating real-world task-resolution settings. These evaluations aim to guide research prioritization for autonomous coding systems. 1 Introduction As language models (LMs) demonstrate increasingly sophisticated programming capabilities, the demand for rigorous evaluation frameworks has grown. While early datasets like Defects4J (Just et al., 2014 ) and LiveCodeBench (Jain et al., 2024 ) test standalone code generation, the evaluation landscape has decisively shifted toward repository-level problem solving. In SWE-bench (Jimenez et al., 2023 ) , models are required to function as software engineering agents: they must interact with an executable environment, examine the entire repository, comprehend its organization, and address tasks originating from real-world GitHub issues. The difficulty and practical relevance of SWE-bench have led to extensive follow-up benchmarks— including SWE-bench-Multilingual (Yang et al., 2025b ) , SWE-bench-Multimodal (Yang et al., 2025a ) , SWE-bench-Live (Zhang et al., 2025 ; Li et al., 2026 ) , and SWE-bench-Pro (Deng et al., 2025 ) —each further increasing task diversity and complexity. To tackle the tasks defined in SWE benchmarks, researchers have developed a diverse set of approaches ranging from fully agentic frameworks to simpler agent-free pipelines. Systems such as Yang et al. ( 2024 ); Wang et al. ( 2024 ); Xia et al. ( 2024 ) enable LMs to operate over real code repositories by organizing the problem-solving process into structured stages. Subsequent work has further specialized this process by training models that focus on particular stages of the workflow (Ma et al., 2025b ; Wei et al., 2025 ) , while other studies (Meng et al., 2024 ; Bouzenia & Pradel, 2025 ) analyze execution traces of agents to better understand common failure patterns and potential improvement points. From these prior efforts, we observe that most of them implicitly rely on a small set of critical contextual information signals (or sources / factors, synonymous in the following context) when solving SWE tasks, which can be categorized into five major categories: Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage (Table 1 ). Though numerous works focus on the five signals, one key question remains: how much could these systems improve potentially if specific information signals were made perfectly accurate? The question is critical in guiding future research priorities. To answer this, we construct oracle versions of these signals and measure their individual impact through controlled ablations. Experimental results reveals that for SWE-bench, the contribution order is Reproduction Test > > Execution Context ∼ \sim Edit Location > > API Usage > > Regression Test , while for SWE-bench-Live and Pro, the order becomes Reproduction Test > > Execution Context ∼ \sim API Usage > > Edit Location > > Regression Test . The strong effect of reproduction suggests that current LM agents often struggle to resolve ambiguous issue descriptions and benefit greatly from explicit failure signals produced by high-quality tests. Since such oracle information is not available in real issue resolution, we further conduct an additional validation experiment by prompting a stronger LM agent to find the five signals respectively and evaluate their contribution with the original agent. The result is consistent with the oracle version. The contributions of this work are threefold: • Identifying the critical information factors in SWE benchmarks. This work formally defines five critical information factors previous works focus on through comprehensive literature review and proposes a method to extract ground truths of the 5 factors. The extraction method is generalized to multiple SWE benchmarks to benefit dataset analysis. • Quantifying upper bounds through oracle analysis. This work estimate the potential contribution upper bounds of the five signals to overall task success, guiding research priorities. • Agent-based validation of oracle insights. This work further designs a validation experiment to estimate the contribution of the five factors in real-world issue resolution. 2 Related Works 2.1 Agentic Workflow in SWE Tasks A substantial body of work on SWE benchmarks focuses on designing agent pipelines that structure repository exploration into a sequence of stages supported by tool usage. For example, Agentless (Xia et al., 2024 ) adopts a canonical staged procedure in which an LM first performs bug localization, then generates code edits, constructs reproduction tests, and finally validates the resulting patch using both reproduction and regression tests. Similarly, SWE-agent (Yang et al., 2024 ) and OpenHands (Wang et al., 2024 ) follows a multi-stage instruction of localization, reproduction, editing and test validation, with a minimal tool set of shell commands and string-replacement editor to interact with the repository. Closed-source systems such as Claude Code (Anthropic, 2025 ) follow a comparable paradigm but introduce advanced tools such as planning tool ToDoWrite and delegation tool Task that delegates sub-tasks to sub-agents. Empirical analysis by (Bouzenia & Pradel, 2025 ) further shows that despite architectural differences, agent trajectories across benchmarks such as Defects4J and SWE-bench tend to exhibit three recurring behavioral phases: Localization (explore, search, and identify relevant code regions), Edit (reason about the bug and produce a patch), and Test Execution (reproduce the bug and verify correctness through tests) (Figure 7 (a)). Another line of work explicitly decomposes SWE tasks into multi-agent or multi-task pipelines with specialized components. TDFlow (Han et al., 2025 ) distributes the workflow across agents dedicated to localization, patch generation, format correction, reproduction-test synthesis, and test-based validation. Several multi-agent approaches further introduce a specialized agent for code-graph construction to improve contextual understanding and identify relevant API utilities. MarsCode (Liu et al., 2024 ) and Lingma Agent (Ma et al., 2025a ) build code graphs that combine repository structure, abstract syntax trees, and inter-function call relations, improving localization, contextual reasoning, and API-usage understanding; GraphLocator (Liu et al., 2025 ) augments these graphs with natural-language descriptions to facilitate navigation. AutoCodeRecover (Zhang et al., 2024 ) and PatchPilot (Li et al., 2025 ) obtain code graphs of function call relations by executing reproduction tests and collecting execution traces. 2.2 Training LMs for SWE Tasks In addition to improving agentic systems, another research direction focuses on directly training language models to perform key sub-tasks in the SWE pipeline, including issue localization, patch synthesis, and reproduction-test generation. For instance, SoRFT (Ma et al., 2025b ) and SWE-RL (Wei et al., 2025 ) train LMs for localization and patch generation using supervised fine-tuning (SFT) followed by reinforcement learning (RL). Similarly, SWE-Swiss (He et al., 2025 ) applies SFT across localization, patch generation, and reproduction-test generation, and subsequently performs RL optimization on patch-generation. These approaches typically depend on LM-synthesized datasets that provide training targets for each stage in the workflow. More recent efforts instead train models using complete agent trajectories. Works such as Zainullina et al. ( 2025 ) , Golubev et al. ( 2025 ) , Cao et al. ( 2025 ) collect exploration trajectories of LM agents following the standard workflow of localization, reproduction, editing, and validation. Successful trajectories are then used for rejection-sampling SFT, while RL is applied with binary rewards reflecting the final success or failure of the agent solution. SkyRL (Cao et al., 2025 ) further extends this framework by incorporating code-graph search tools that enhance localization accuracy and contextual reasoning over the repository. Figure 1: Extraction pipeline of oracle information signals for Code Search, including Edit Location, Execution Context and API Usage. 2.3 Failure Pattern Analysis on SWE Tasks A growing body of work investigates the failure modes of agents on SWE tasks. SWE-agent (Yang et al., 2024 ) analyzes unsuccessful runs using GPT-4o and categorizes them into several classes, including failures in reproduction, localization, and code generation (e.g., incorrect logic, overly specific implementations, and iterative editing errors) (Figure 7 (b)). Meng et al. ( 2024 ) further demonstrate that inaccurate reproduction of the issue can significantly reduce overall task success. Chen et al. ( 2025 ) analyze exception traces of agent solutions and identify three major categories of errors: (1) code-generation issues such as syntax mistakes, (2) insufficient familiarity with repository context, including import failures or incorrect variable usage, and (3) misuse of external APIs. PAGENT (Xue et al., 2025 ) studies agent-generated patches and evaluation logs and finds recurring problems including context misunderstanding (e.g., incorrect type transformations), inadequate coverage of corner cases in reproduction, and incorrect usage of internal or external APIs. Synthesizing these observations, prior work suggests that improvements to SWE agents largely fall along two major axes comprising five key information factors: 1. Code Search , which includes accurate edit localization, relevant execution context, internal or external API usage. 2. Testcase Verification , which includes generating reliable reproduction tests that capture the bug behavior and correctly identifying the commands to execute existing regression tests. To clarify how these information factors relate to previous studies, we summarize the focus of representative works in Table 1 . Figure 2: Extraction pipeline of Oracle Tests, including Reproduction Test and Regression Test. Figure 3: Validation pipeline: a stronger model extracts key factors and a weaker model resolves issue. 3 Extraction of Oracle Information This section defines the oracle versions of the five information factors and proposes generalized methods to extract each from the benchmark datasets and corresponding code repositories. We provide one example of extracted information for each signal in Appendix F . 3.1 Code Search 3.1.1 Edit Location The oracle Edit Location consists of the code regions where modifications occur in the gold patch. Extracted snippets include source code within these regions, annotated with explicit file path, function name and line prefixes and ordered by their line numbers. 3.1.2 Execution Context Execution Context refers to the code contexts surrounding critical locations in the source code when reproducing and fixing the issue in an SWE task. Stack traces summarize the relation of function calls leading to an error and therefore provide a compact representation of the program’s runtime code context. The oracle Execution Context is thus derived from the error stack trace produced when executing reproduction tests on the buggy version of the repository. Each frame in the trace is represented as a tuple of file path, line number, function name, and surrounding source code. To provide sufficient context, we include a window of 15 lines centered around each referenced line. Frames originating from test functions are masked to prevent leakage of ground truth tests. In Python projects, many failures arise from assertion errors triggered inside test functions, which prevents the stack trace from reaching the underlying source code responsible for the bug. To address this limitation, we insert a custom stack-trace collector at the oracle edit locations and record the deepest reachable call stack during execution for instances without error stack into the source code. For Golang projects, reproduction tests frequently fail during the build stage, which similarly prevents normal stack-trace collection, so contextual information is entirely obtained from the custom stack-trace collector. 3.1.3 API usage The oracle API Usage signal contains all function calls modified or newly introduced in the ground-truth patch. For each call, we record the function name, its arguments, the approximate region where it should be used, and the corresponding function definition whenever the function is not part of the language’s built-in library. For extraction of API function definition, simple name matching is insufficient because many functions share identical names across different modules. For Python repositories, which is not strictly typed, the correct function definition is inferred through execution coverage. When the target API call is executed, the coverage records first the call site and then the executed lines within the invoked function, allowing us to link the call to its implementation. In contrast, Golang’s static typing enables us to resolve the definition through type checking, import analysis, and argument signatures. Figure 1 illustrates the pipeline used to extract oracle information related to code search. 3.2 Testcase Verification 3.2.1 Reproduction Test The oracle Reproduction Test information consists of three components: the command required to execute the reproduction tests, the list of reproduction test names that must pass, and the full source code of those test functions. These tests correspond to the Fail-to-Pass cases in the benchmark datasets. Each reproduction test typically contains multiple parameterized inputs that cover diverse corner cases, making them a highly informative signal for identifying the bug. Although the official SWE-bench specification requires that the patches introducing these tests (i.e., test patch ) are not applied to the repository before validation, we intentionally apply them in our experiments. This allows the model to directly execute the reproduction tests and iteratively refine its edits based on the resulting error outputs. The location of each test function is inferred from its name, which generally encodes hierarchical information such as folder, file, class, and function identifiers. The function bodies are then extracted using AST-based parsing. Most modern testing frameworks allow execution of individual testcases. We therefore provide commands that run only the relevant reproduction tests. Instances that do not support selective execution are excluded from our evaluation. 3.2.2 Regression Test The oracle Regression Test information includes the command used to execute regression tests and the list of tests that must remain passed after the fix. These tests correspond to the Pass-to-Pass cases in the benchmark datasets. The regression test command runs all tests within the corresponding test files following the official SWE-bench evaluation procedure. Before the test patch is applied, this command executes only the Pass-to-Pass testcases. Figure 3 presents the pipeline used to extract oracle test-validation signals. 4 Experiments 4.1 Base Agent Our experiments require an agent that can freely explore the repository, execute tests to validate the fix, and retry edits based on the test output so that the injected oracle information can be fully utilized during the repair process. We therefore adopt SWE-agent (Yang et al., 2024 ) , a minimal free agent equipped with only two tool interfaces: bash commands and a string-replace editor supporting file viewing, replacement, and creation. The max step limit of SWE-agent is set to 120 to balance performance and cost. More complex systems, such as Claude Code, may benefit disproportionately from information gain of the factors under study, due to their improved design for these factors. Using such agents would make it difficult to isolate the intrinsic and absolute contribution of any single oracle factor. To avoid this confounding effect, we restrict our experiments to the minimal SWE-agent. 4.2 Oracle Ablation Study We quantify the upper-bound contribution of each oracle factor via controlled ablations. Concretely, we append ground-truth signals to the initial user prompt to emulate a real-world issue-resolution setting, where sub-agents report findings to a main/edit agent, or where a single agent summarizes the exploration history from earlier stages as specified in the user instruction before continuing. While alternative ground-truth injection strategies could lead to subtle differences in measured contributions, budget constraints prevent us from exhaustively re-running the study all over again. The agent is explicitly instructed to fully trust the injected information, while any knowledge not injected must be acquired through normal exploration. The full prompt is provided in Appendix G . We first measure the success rate and corresponding LM API cost when injecting each factor individually. We then inject combinations of two or more factors to study interactions among oracle signals and to approximate the attainable upper bound when all five factors are perfectly known. Figure 4: Ablation study measuring the upper-bound contribution of each information factor. 4.3 Validation Study Admittedly, the usage of oracle information may be unrealistic. For example, oracle reproduction tests are embedded in test files, whereas SWE agents typically write reproduction code in the repository root. Moreover, oracle reproduction tests may leak benchmark evaluation criteria. In practice, agents must infer these signals rather than receiving perfectly constructed oracle inputs. To better estimate (i) the contribution of the five factors and (ii) the difficulty of obtaining them under realistic conditions, we design a two-stage validation experiment. In Stage 1, an agent is prompted to extract each factor using the same definition and format as the oracle information (see Appendix H for prompt), with a fixed LM for a benchmark to align the extraction quality. In Stage 2, the agent attempts to solve the task using the factors it extracted. We use a stronger model for Stage 1 to emulate high-quality signal extraction, and a weaker model for Stage 2, as shown in Figure 3 . As baselines, we also evaluate single-agent setups using the strong model and the weak model alone. The two-stage pipeline allocates 50 steps to factor extraction and 70 steps to issue resolution, matching the 120-step budget of the single-agent baseline. Ideally, the two-stage pipeline should outperform the weaker model and approach the stronger model’s performance. 4.4 Benchmarks and Models We evaluate on three datasets: SWE-bench-Verified (Jimenez et al., 2023 ) , SWE-bench-Live (Zhang et al., 2025 ; Li et al., 2026 ) , and SWE-bench-Pro (Deng et al., 2025 ) . Latest models such as GPT-5 already saturate SWE-bench-Verified with over 70% success. We therefore also include the more challenging SWE-bench-Live and SWE-bench-Pro datasets released in 2025. For SWE-bench-Live, we use the verified Python subset for which task feasibility has been confirmed. For SWE-bench-Pro, we evaluate both the Python and Golang subsets to assess cross-language generalization. We filter out instances whose ground-truth patches fail or whose reproduction tests cannot be selectively executed. After filtering, the final datasets contain 459 instances for SWE-bench-Verified, 353 for SWE-bench-Live, 220 for SWE-bench-Pro/Python, and 211 for SWE-bench-Pro/Golang. We obtain execution context in two ways. Native error stack traces are available for 159 instances in SWE-bench-Verified, 72 in SWE-bench-Live, and 50 in SWE-bench-Pro/Python. For all other cases, we use the custom stack-trace collector that records runtime call stacks at the oracle edit locations. In Golang, most reproduction tests fail during the build phase (271/280 instances), so all Golang tasks rely on the custom collector. We evaluate multiple models to ensure the findings are generalizable. GPT-5-Thinking-Medium (GPT-5) and GPT-4o are tested on SWE-bench-Verified; the lower baseline of GPT-4o helps reveal the absolute impact of individual factors. For SWE-bench-Live and Pro, we evaluate both GPT-5 and Claude-4.5-Sonnet. For validation experiment, GPT-4o is the weaker model while GPT-5 is the stronger model for SWE-bench-Verified; for the other benchmarks, GPT-5 is the weaker one while Claude-4.6-Sonnet is the stronger one. Figure 5: Accumulative contribution and average number of rollout steps when factors accumulate. Each factor on the x-axis incorporates all factors to its left. 5 Results 5.1 Ablation Study on the Upper Bound Contribution of each Oracle Factor Figure 4 reports the success rates across the benchmarks and models under study when a single oracle information is provided. The result for execution context is the overall result averaged on native error stack trace and custom stack trace collector (for individual contribution of native stack and custom stack, please refer to Table 3 ). Compared to the baseline where no information is provided, injecting any oracle signal consistently improves the success rate, providing preliminary evidence that the extracted factors are meaningful. Across models and datasets, the results are highly consistent. On SWE-bench-Live and SWE-bench-Pro, the five factors follow the same ordering: Reproduction Test > > Execution Context ∼ \sim API Usage > > Edit Location > > Regression Test . On SWE-bench-Verified, the ordering is unchanged except that Edit Location contributes more than API Usage and Execution Context. This stability holds despite prior concerns that the older SWE-bench instances may have leaked into the training data of newer models such as GPT-5 (Zhang et al., 2025 ; Deng et al., 2025 ) , and that earlier models such as GPT-4o may exhibit weaker instruction following. Different Patterns between Error Trace and Custom Trace. Table 3 shows a clear gap between the contributions of the native error stack and the custom stack. The native error stack is much more effective, likely because exceptions raised within the source code inform the agent of explicit failures (e.g., list index out of range , attribute does not exist ). It also provides rich information about the bug-affected execution path and the exact failure point, which substantially improves success rates. By contrast, custom stacks are used only when native error stacks are unavailable. In these cases, errors are typically implicit (e.g., incorrect calculation result) and rooted in program logic, making the call stack less useful for fixing the issue. Such stack trace is more like a way to locate the edit locations, as suggested by the similar contribution with edit location. In some model-benchmark combinations in Table 3 , the custom stack even contributes slightly less than Edit Location of the same combination, possibly because such custom stack adds little new information while requiring extra effort to locate the exact edit location. Overall, the results suggest that native error stack traces are highly informative but often unavailable. Indications from API usage. We analyze the distribution of API calls extracted from the oracle signals. Excluding built-in functions (for which we do not provide definitions), internal APIs within the repository account for approximately 82% of calls, while external APIs from third-party libraries account for about 18%. This suggests that effective repository-level search for internal utilities is substantially more important for SWE agents than retrieving external documentation. We further investigate why API usage contributes less on SWE-bench-Verified. A manual inspection of 50 sampled instances reveals that the majority of extracted calls correspond to basic built-in functions requiring little additional knowledge. Among the remaining calls, most originate from widely used libraries such as django , sympy , and matplotlib . Because these libraries are highly popular, modern LMs likely already possess strong prior knowledge of their public APIs, reducing the marginal benefit of providing API definitions as oracle information. Figure 6: Visualized result of the two-stage validation compared with the single-agent baseline. Labels are formatted as (strong model) + (weak model) . The high success rate of reproduction. The dominant contribution of reproduction tests raises the question of whether LMs merely modify code to satisfy the specified tests rather than solving the underlying issue. To investigate this, we manually inspected 50 agent-predicted patches sampled from the four datasets as to whether any solutions intentionally circumvent the reproduction tests. The result shows that none of the prediction patches were considered to cheat. Reproduction tests provide precise descriptions of failure conditions, whereas natural-language issue descriptions in SWE benchmarks are often ambiguous (Chowdhury et al., 2024 ; Xia et al., 2024 ) . Therefore, the resulting error outputs serve as highly informative signals for guiding the repair process. Cost of rollout. We also measure the average API cost and the average number of steps required to solve each instance when a single factor is provided. Detailed statistics are reported in Appendix B . 5.2 Contribution of Factor Combinations We further evaluate how the five factors interact when combined. Contribution when factors accumulate. Figure 5 (a) reports success rates as oracle signals are added incrementally; each factor on the x-axis incorporates all factors to its left. Regression tests yield only modest gains, as they mainly prevent LMs from breaking previously correct functionality rather than directly guiding repairs. By contrast, Edit Location becomes substantially more effective when combined with test signals, suggesting that test outcomes help the model infer how to modify the identified locations. With all five factors combined, all four model-bench pairings achieve at least a 97% success rate. This indicates the five factors provide near-complete information for solving the evaluated SWE tasks, demonstrating the completeness of the factor set defined in this paper. The small differences in the marginal contributions and their order across different benchmarks in Figure 4 and Figure 5 further suggest variation in benchmark difficulty and task distributions, and show that our factor-isolation method, as a general SWE dataset analysis methodology, is sensitive enough to surface such dataset differences. Figure 5 (b) shows the average number of rollout steps under factor accumulation. Steps generally decrease as more signals are provided, indicating that richer information reduces exploration. However, with many factors combined, steps begin to rise again, suggesting the agent needs additional iterations to interpret and integrate multiple signals correctly. Average cost under factor accumulation is reported in Table 2 in Appendix. Cost decreases less than steps because additional oracle signals increase context length and may require broader exploration to use effectively; for instance, test execution logs can be lengthy to increase LM input length. To further examine the pattern of accumulative factors, we also evaluate an alternative accumulation order (Figure 10 ). Success rates still increase monotonically as factors are added, and the step and cost trends remain consistent, reinforcing the benefit of combining multiple information sources. Contribution of two-factor combination. Since accumulation experiment suggests that the contribution of combined factors is not additive, we further evaluate pairwise combinations in Table 4 . The results show that combining Reproduction Test, Edit Location, and API Usage yields substantially larger improvements than any single factor alone, indicating that jointly improving multiple factors is more effective than optimizing them in isolation. 5.3 Validation Experiment In this experiment, from each dataset 100 instances were randomly sampled to run for cost control. The contribution of agent-extracted signals compared to single-agent baselines is illustrated in Figure 6 with cost recorded in Table 5 . These results align with our ablation study, particularly highlighting the importance of the Reproduction Test. The contribution of Execution Context is slightly lower than that of oracle versions, likely reflecting the practical difficulty of extracting this signal, especially in the majority of cases where error stack traces are unavailable as discussed above. Regarding LM API cost, for information extraction, obtaining a Reproduction Test is the cheapest for SWE-bench-Verified, while Edit Location is cheapest for SWE-bench-Live and Pro. For issue resolution using extracted information, providing Edit Location yields the lowest cost for SWE-bench-Verified, whereas providing Reproduction Test is most cost-efficient for the others. Surprisingly, for any of the benchmarks, there are always more than 4 instances that are unresolved by either strong or weak model alone but successfully resolved with agent-generated Reproduction Tests. Moreover, this approach surpasses the baseline performance of the stronger model given agent-extracted reproduction, demonstrating the methodological potential of this combined strategy. 6 Conclusion In this work, we address a key gap previous works on SWE agents did not address: the potential contribution of the five mostly studied signals when each is made perfect. The answer to the question is critical in guiding future research emphasis. This work delivers three main findings: • High-quality reproduction tests that capture corner cases are the most influential signal. • Execution context such as code graph is significantly more useful only when error stack traces are available, which only accounts for one-forth of the instances. • Edit-location and API usage becomes significantly more effective when paired with each other or with Reproduction. Limitations This work identifies five information factors based on patterns observed in prior analyses of LM agents on SWE benchmarks. The categorization is partly subjective and intended to synthesize and extend previous observations rather than to represent a definitive or exhaustive decomposition of the information required for software issue resolution. Acknowledgement Kenan Li and Qirui Jin are responsible for the experiments. Kenan Li, Qirui Jin and Liao Zhu are responsible for paper writing. Thanks our supervisors Yu kang, Chaoyun Zhang, Xin Zhang and Wenke Lee for identifying resources and offering valuable suggestions. References Anthropic (2025) Anthropic. Claude code. https://code.claude.com/ , 2025. Bouzenia & Pradel (2025) Islem Bouzenia and Michael Pradel. Understanding software engineering agents: A study of thought-action-result trajectories. arXiv preprint arXiv:2506.18824 , 2025. Cao et al. (2025) Shiyi Cao, Dacheng Li, Fangzhou Zhao, Shuo Yuan, Sumanth R Hegde, Connor Chen, Charlie Ruan, Tyler Griggs, Shu Liu, Eric Tang, et al. Skyrl-agent: Efficient rl training for multi-turn llm agent. arXiv preprint arXiv:2511.16108 , 2025. Chen et al. (2025) Zhi Chen, Wei Ma, and Lingxiao Jiang. Unveiling pitfalls: Understanding why ai-driven code agents fail at github issue resolution. arXiv preprint arXiv:2503.12374 , 2025. Chowdhury et al. (2024) Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing swe-bench verified, 2024. URL https://openai.com/index/introducing-swe-bench-verified/ . Deng et al. (2025) Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941 , 2025. Golubev et al. (2025) Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, et al. Training long-context, multi-turn software engineering agents with reinforcement learning. arXiv preprint arXiv:2508.03501 , 2025. Han et al. (2025) Kevin Han, Siddharth Maddikayala, Tim Knappe, Om Patel, Austen Liao, and Amir Barati Farimani. Tdflow: Agentic workflows for test driven software engineering. arXiv preprint arXiv:2510.23761 , 2025. He et al. (2025) Zhenyu He, Qingping Yang, Wei Shen, Xiaojian Zhong, Kechi Zhang, Chenxin An, Wenlei Shi, Tianle Cai, Di He, Jiaze Chen, and Jingjing Xu. Swe-swiss: A multi-task fine-tuning and rl recipe for high-performance issue resolution. Notion, August 2025. URL https://www.notion.so/SWE-Swiss-A-Multi-Task-Fine-Tuning-and-RL-Recipe-for-High-Performance-Issue-Resolution-21e174dedd4880ea829ed4c861c44f88 . Project Leads: Zhenyu He, Jiaze Chen. Affiliations: Peking University, ByteDance Seed, The University of Hong Kong. Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024. Jimenez et al. (2023) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023. Just et al. (2014) René Just, Darioush Jalali, and Michael D Ernst. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis , pp. 437–440, 2014. Li et al. (2025) Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. Patchpilot: A stable and cost-efficient agentic patching framework. arXiv e-prints , pp. arXiv–2502, 2025. Li et al. (2026) Kenan Li, Rongzhi Li, Linghao Zhang, Qirui Jin, Liao Zhu, Xiaosong Huang, Geng Zhang, Yikai Zhang, Shilin He, Chengxing Xie, Zhang Xin, Jin Zijian, Li Bowen, Zhang Chaoyun, Kang Yu, Huang Yufan, Nallipogu Elsie, Rajmohan Saravan, Lin Qingwei, and Zhang Dongmei. Repolaunch: Automating build&test pipeline of code repositories on any language and any platform. arXiv preprint arXiv:2603.05026 , 2026. Liu et al. (2025) Wei Liu, Chao Peng, Pengfei Gao, Aofan Liu, Wei Zhang, Haiyan Zhao, and Zhi Jin. Graphlocator: Graph-guided causal reasoning for issue localization. arXiv preprint arXiv:2512.22469 , 2025. Liu et al. (2024) Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.00899 , 2024. Ma et al. (2025a) Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering , pp. 238–249, 2025a. Ma et al. (2025b) Zexiong Ma, Chao Peng, Pengfei Gao, Xiangxin Meng, Yanzhen Zou, and Bing Xie. Sorft: Issue resolving with subtask-oriented reinforced fine-tuning. arXiv preprint arXiv:2502.20127 , 2025b. Meng et al. (2024) Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. An empirical study on llm-based agents for automated bug fixing. arXiv preprint arXiv:2411.10213 , 2024. Wang et al. (2024) Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 , 2024. Wei et al. (2025) Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449 , 2025. Xia et al. (2024) Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 , 2024. Xue et al. (2025) Haoran Xue, Gias Uddin, and Song Wang. Pagent: Learning to patch software engineering agents. arXiv preprint arXiv:2506.17772 , 2025. Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems , 37:50528–50652, 2024. Yang et al. (2025a) John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations , 2025a. URL https://openreview.net/forum?id=riTiq3i21b . Yang et al. (2025b) John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025b. URL https://arxiv.org/abs/2504.21798 . Zainullina et al. (2025) Karina Zainullina, Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Daria Litvintseva, Simon Karasik, Filipp Fisin, Sergei Skvortsov, Maksim Nekrashevich, et al. Guided search strategies in non-serializable environments with applications to software engineering agents. arXiv preprint arXiv:2505.13652 , 2025. Zhang et al. (2025) Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419 , 2025. Zhang et al. (2024) Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pp. 1592–1604, 2024. Appendix A Supplementary Information for Related Works A.1 Analysis of proportions of related factors in related works Please refer to Figure 7 . A.2 Mapping of Related Works to Corresponding Factors Please refer to Table 1 . Appendix B Average Cost and Number of Steps for Single Oracle Factor Figure 9 shows the average cost of solving each instance when one factor is provided. Figure 8 shows the average number of steps of solving each instance when one factor is provided. Cost of API is higher than others because the agent needs multiple rounds of edits and test validation to use API utilities correctly from our observation. The slightly higher cost for Execution Context than for Edit Location suggests the agent spends additional steps exploring the repository to locate the exact edit locations and understand the provided function-call relationships. Regression has higher cost because the requirement of running regression tests adds additional steps but contributes little to the actual issue resolution, and the lengthy output of running regression tests such as long list of test statuses would lead to increase in LM input and thus increase in cost. Appendix C Extended Result on Execution Context The contribution of native error stack trace and custom stack collector into edit locations result in different contributions, see Table 3 . (a) (b) Figure 7: (a) Proportion of SWE agents’ exploration pattern, taken directly from (Bouzenia & Pradel, 2025 ) . (b) Proportion of SWE-Agent’s failure modes, taken directly from (Yang et al., 2024 ) . Related Work Type Repro-duction Regre-ssion Location Execution Context API Agentless (Xia et al., 2024 ) Agent Workflow ✓ \checkmark ✓ \checkmark ✓ \checkmark × \times × \times SWE-agent (Yang et al., 2024 ) Agent Workflow ✓ \checkmark × \times ✓ \checkmark ○ \bigcirc × \times OpenHands (Wang et al., 2024 ) Agent Workflow ✓ \checkmark ✓ \checkmark ✓ \checkmark ○ \bigcirc × \times Claude Code (Anthropic, 2025 ) Agent Workflow ✓ \checkmark ○ \bigcirc ✓ \checkmark ○ \bigcirc ○ \bigcirc TDFlow (Han et al., 2025 ) Agent Workflow ✓ \checkmark × \times ✓ \checkmark × \times × \times MarsCode (Liu et al., 2024 ) Agent Workflow × \times × \times ✓ \checkmark ✓ \checkmark ✓ \checkmark Lingma Agent (Ma et al., 2025a ) Agent Workflow × \times × \times ✓ \checkmark ✓ \checkmark ✓ \checkmark GraphLocator (Liu et al., 2025 ) Agent Workflow × \times × \times ✓ \checkmark ✓ \checkmark × \times AutoCodeRecover (Zhang et al., 2024 ) Agent Workflow ✓ \checkmark × \times ✓ \checkmark ✓ \checkmark × \times PatchPilot (Li et al., 2025 ) Agent Workflow ✓ \checkmark × \times ✓ \checkmark ✓ \checkmark × \times SoRFT (Ma et al., 2025b ) Agentic Training × \times × \times ✓ \checkmark × \times × \times SWE-RL (Wei et al., 2025 ) Agentic Training × \times × \times ✓ \checkmark × \times × \times SWE-Swiss (He et al., 2025 ) Agentic Training ✓ \checkmark × \times ✓ \checkmark × \times × \times (Zainullina et al., 2025 ) Agentic Training ✓ \checkmark × \times ✓ \checkmark ○ \bigcirc × \times (Golubev et al., 2025 ) Agentic Training ✓ \checkmark × \times ✓ \checkmark ○ \bigcirc × \times SkyRL (Cao et al., 2025 ) Agentic Training ✓ \checkmark × \times ✓ \checkmark ✓ \checkmark × \times (Bouzenia & Pradel, 2025 ) Pattern Analysis ✓ \checkmark ✓ \checkmark ✓ \checkmark ○ \bigcirc × \times (Meng et al., 2024 ) Pattern Analysis ✓ \checkmark × \times × \times × \times × \times (Chen et al., 2025 ) Pattern Analysis × \times × \times × \times ✓ \checkmark ✓ \checkmark PAGENT (Xue et al., 2025 ) Pattern Analysis × \times × \times × \times ✓ \checkmark ✓ \checkmark Table 1: The factors each work focuses on in Related Works. Figure 8: Average number of steps for single factors. Figure 9: Average cost of rollout for single factors. Bench&Model Base +Reprouction +Regression +Location +API +Context Verified&GPT-5 0.35 0.17 0.23 0.15 0.31 0.18 Verified&GPT-4o 2.18 1.24 1.77 1.21 1.98 1.61 Live&GPT-5 1.11 0.52 0.47 0.83 1.18 0.50 Pro&GPT-5 0.85 0.56 0.45 0.84 0.96 0.86 Table 2: Average cost (USD) when factors accumulate. Each successive factor on the right includes all factors to its left. Verified Live Pro/Python Pro/Go GPT-5 GPT-4o GPT-5 Claude-4.5 GPT5 Claude-4.5 GPT-5 Native Error Num. instances 159 159 72 72 50 50 0 Resolved (%) 87.4 42.1 64.0 66.7 48.0 56.0 – Avg. Cost (usd) 0.29 1.53 0.83 1.60 0.87 1.88 – Avg. Step 25.5 39.3 44.3 84.2 44.0 84.0 – Custom Collector Num. instances 300 300 281 281 170 170 211 Resolved (%) 84.0 38.0 39.9 48.0 27.1 35.3 45.0 Avg. Cost (usd) 0.26 1.36 0.79 1.54 0.91 1.85 1.47 Avg. Step 22.3 38.0 33.5 75.9 42.3 81.7 58.3 Averaged Resolved (%) 85.2 39.4 45.0 51.8 31.7 40.0 45.0 Avg. Cost (usd) 0.27 1.41 0.80 1.55 0.90 1.86 1.47 Avg. Step 23.4 38.5 35.7 77.0 42.8 82.4 58.3 Table 3: Contribution of execution context for error stack and custom stack respectively. Verified&GPT-4o Pro-Python&GPT-5-Thinking-Med Resolved Avg. Cost Avg. Step Resolved Avg. Cost Avg. Step Edit Loc. + Reproduction 64.9% 1.19$ 34.0 80.0% 0.90$ 31.8 Context + Reproduction 64.9% 2.15$ 40.1 79.1% 0.78$ 22.8 Reproduction + API 68.0% 1.20$ 33.8 60.0% 0.87$ 29.1 Regression + Reproduction 58.9% 1.77% 31.4 56.8% 1.09$ 45.9 Edit Loc. + API 43.6% 0.72$ 31.3 48.2% 1.09$ 43.1 Context + API 43.6% 1.60$ 36.5 44.1% 1.14$ 39.4 Context + Edit Loc. 44.0% 1.07$ 29.0 34.1% 1.18$ 38.3 Regression + API 40.3% 1.29% 39.0 38.2% 1.07$ 44.7 Edit Loc. + Regression 40.3% 1.02$ 35.2 29.1% 1.12$ 45.9 Context + Regression 39.8% 1.81$ 40.5 31.8% 1.18$ 46.8 Table 4: Combination of two factors. Roughly sorted by contribution. Regression Location API Context Reproduction SWE-bench-Verified: GPT-5-Thinking-Medium + GPT-4o Success gain +4% +12% +8% +14% +26% Stage 1 $ / Step 0.87$ / 33.0 0.66$ / 32.2 0.89$ / 32.9 0.87$ / 32.4 0.39$ / 17.9 Stage 2 $ / Step 1.65$ / 40.6 0.85$ / 36.8 1.69$ / 40.9 1.51$ / 41.6 1.74$ / 36.7 SWE-bench-Live: Claude-4.6-Sonnet + GPT-5-Thinking-Medium Success gain +4% +8% +14% +14% +24% Stage 1 $/Step 1.02$ / 49.2 0.69$ / 35.5 1.09$ / 53.5 0.97$ / 45.6 0.78$ / 36.4 Stage 2 $/Step 0.68$/ 32.6 0.56$ / 30.4 0.64$ / 31.5 0.72$ / 33.6 0.47$ / 24.0 SWE-bench-Pro/Python: Claude-4.6-Sonnet + GPT-5-Thinking-Medium Success gain +2% +6% +11% +9% +28% Stage 1 $/Step 1.15$ / 55.0 0.82$ / 35.8 1.21$ / 56.3 1.11$ / 48.1 0.94$ / 39.8 Stage 2 $/Step 0.88$ / 39.1 0.89$ / 40.1 0.89$ / 39.0 0.92$ / 38.6 0.47$ / 19.5 SWE-bench-Pro/Go: Claude-4.6-Sonnet + GPT-5-Thinking-Medium Success gain +2% +6% +12% +10% +19% Stage 1 $/Step 1.18$ / 45.6 0.89$ / 39.2 1.24$ / 57.6 1.21$ / 49.7 1.10$ / 40.5 Stage 2 $/Step 1.28$ / 46.6 1.26$ / 56.1 1.12$ / 44.7 1.26$ / 48.8 0.75$ / 30.4 Table 5: Details of validation experiment result, including performance gain for each factor compared to the weaker model baseline and average cost of the two stages. Labels are formatted as Benchmark: (strong model) + (weak model) . Figure 10: Accumulative contribution and average cost with factors accumulated from another direction. Appendix D Extended Results of Combined Factors. Figure 5 reports the average number of steps as factors accumulate, while Table 2 presents the corresponding average cost. Each successive factor on the right includes all factors to its left. Since Figure 5 suggests that the contribution of combined factors is not additive, we further evaluate pairwise combinations in Table 4 . The results show that combining factors such as Reproduction Test, Edit Location, and API yields substantially larger improvements than any single factor alone, indicating that jointly improving multiple factors is more effective than optimizing them in isolation. In contrast, Regression Test continues to provide limited benefit even when combined with other factors. To further examine this, we also evaluate an alternative accumulation order (Figure 10 ). The success rate consistently increases as additional factors are introduced, demonstrating the positive effect of combining multiple information sources. It is important to note that average cost does not always decrease when multiple factors are combined. Providing additional oracle signals increases context length and may require broader exploration to utilize them effectively. For example, test execution logs can be lengthy and significantly expand the prompt. Similarly, the average number of steps does not necessarily decrease, as the agent may require additional steps to understand and integrate multiple factors correctly. Appendix E Validation Experiment Result Details Please refer to Table 5 . Appendix F Examples of Extracted Factor F.1 Regression Test ⬇ { "instance_id" : "verified/django__django-10999" , "command" : "python ./tests/run.py --verbosity 2 --settings=test_pgsql --parallel 1 utils_tests.test_dateparse" , "PASS_TO_PASS" : [ "test_parse_date (utils_tests.test_dateparse.DateParseTests)" , "test_parse_datetime (utils_tests.test_dateparse.DateParseTests)" , "test_parse_time (utils_tests.test_dateparse.DateParseTests)" , "test_days (utils_tests.test_dateparse.DurationParseTests)" , "test_fractions_of_seconds (utils_tests.test_dateparse.DurationParseTests)" , "test_hours_minutes_seconds (utils_tests.test_dateparse.DurationParseTests)" , "test_iso_8601 (utils_tests.test_dateparse.DurationParseTests)" , "test_minutes_seconds (utils_tests.test_dateparse.DurationParseTests)" , "test_parse_python_format (utils_tests.test_dateparse.DurationParseTests)" , "test_seconds (utils_tests.test_dateparse.DurationParseTests)" ] } F.2 Reproduction Test Instance_id: verified/django__django-16560.txt Reproduction Test Code: ⬇ =========================== # constraints.tests.BaseConstraintTests.test_custom_violation_code_message.test_custom_violation_code_message: 1| from unittest import mock 2| 3| from django . core . exceptions import ValidationError 4| from django . db import IntegrityError , connection , models 5| from django . db . models import F 6| from django . db . models . constraints import BaseConstraint , UniqueConstraint 7| from django . db . models . functions import Lower 8| from django . db . transaction import atomic 9| from django . test import SimpleTestCase , TestCase , skipIfDBFeature , skipUnlessDBFeature 10| from django . test . utils import ignore_warnings 11| from django . utils . deprecation import RemovedInDjango60Warning 12| 13| from . models import ( 14| ChildModel , 15| ChildUniqueConstraintProduct , 16| Product , 17| UniqueConstraintConditionProduct , 18| UniqueConstraintDeferrable , 19| UniqueConstraintInclude , 20| UniqueConstraintProduct , 21|) 22| ...... 29| class BaseConstraintTests ( SimpleTestCase ): ...... 80| def test_custom_violation_code_message ( self ): 81| c = BaseConstraint ( name = "base_name" , violation_error_code = "custom_code" ) 82| self . assertEqual ( c . violation_error_code , "custom_code" ) =========================== # constraints.tests.CheckConstraintTests.test_eq.test_eq: 1| from unittest import mock 2| 3| from django . core . exceptions import ValidationError 4| from django . db import IntegrityError , connection , models 5| from django . db . models import F 6| from django . db . models . constraints import BaseConstraint , UniqueConstraint 7| from django . db . models . functions import Lower 8| from django . db . transaction import atomic 9| from django . test import SimpleTestCase , TestCase , skipIfDBFeature , skipUnlessDBFeature 10| from django . test . utils import ignore_warnings 11| from django . utils . deprecation import RemovedInDjango60Warning 12| 13| from . models import ( 14| ChildModel , 15| ChildUniqueConstraintProduct , 16| Product , 17| UniqueConstraintConditionProduct , 18| UniqueConstraintDeferrable , 19| UniqueConstraintInclude , 20| UniqueConstraintProduct , 21|) 22| ...... 120| class CheckConstraintTests ( TestCase ): 121| def test_eq ( self ): 122| check1 = models . Q ( price__gt = models . F ( "discounted_price" )) 123| check2 = models . Q ( price__lt = models . F ( "discounted_price" )) 124| self . assertEqual ( 125| models . CheckConstraint ( check = check1 , name = "price" ), 126| models . CheckConstraint ( check = check1 , name = "price" ), 127| ) 128| self . assertEqual ( models . CheckConstraint ( check = check1 , name = "price" ), mock . ANY ) 129| self . assertNotEqual ( 130| models . CheckConstraint ( check = check1 , name = "price" ), 131| models . CheckConstraint ( check = check1 , name = "price2" ), 132| ) 133| self . assertNotEqual ( 134| models . CheckConstraint ( check = check1 , name = "price" ), 135| models . CheckConstraint ( check = check2 , name = "price" ), 136| ) 137| self . assertNotEqual ( models . CheckConstraint ( check = check1 , name = "price" ), 1) 138| self . assertNotEqual ( 139| models . CheckConstraint ( check = check1 , name = "price" ), 140| models . CheckConstraint ( 141| check = check1 , name = "price" , violation_error_message = "custom error" 142| ), 143| ) 144| self . assertNotEqual ( 145| models . CheckConstraint ( 146| check = check1 , name = "price" , violation_error_message = "custom error" 147| ), 148| models . CheckConstraint ( 149| check = check1 , name = "price" , violation_error_message = "other custom error" 150| ), 151| ) 152| self . assertEqual ( 153| models . CheckConstraint ( 154| check = check1 , name = "price" , violation_error_message = "custom error" 155| ), 156| models . CheckConstraint ( 157| check = check1 , name = "price" , violation_error_message = "custom error" 158| ), 159| ) 160| self . assertNotEqual ( 161| models . CheckConstraint ( check = check1 , name = "price" ), 162| models . CheckConstraint ( 163| check = check1 , name = "price" , violation_error_code = "custom_code" 164| ), 165| ) 166| self . assertEqual ( 167| models . CheckConstraint ( 168| check = check1 , name = "price" , violation_error_code = "custom_code" 169| ), 170| models . CheckConstraint ( 171| check = check1 , name = "price" , violation_error_code = "custom_code" 172| ), 173| ) =========================== ’’ ’ <...As the content is too long, parts of this example have been omitted; the actual information provided to the agent is complete...> ’ ’’ =========================== # constraints.tests.UniqueConstraintTests.test_validate_conditon_custom_error.test_validate_conditon_custom_error: 1| from unittest import mock 2| 3| from django . core . exceptions import ValidationError 4| from django . db import IntegrityError , connection , models 5| from django . db . models import F 6| from django . db . models . constraints import BaseConstraint , UniqueConstraint 7| from django . db . models . functions import Lower 8| from django . db . transaction import atomic 9| from django . test import SimpleTestCase , TestCase , skipIfDBFeature , skipUnlessDBFeature 10| from django . test . utils import ignore_warnings 11| from django . utils . deprecation import RemovedInDjango60Warning 12| 13| from . models import ( 14| ChildModel , 15| ChildUniqueConstraintProduct , 16| Product , 17| UniqueConstraintConditionProduct , 18| UniqueConstraintDeferrable , 19| UniqueConstraintInclude , 20| UniqueConstraintProduct , 21|) 22| ...... 336| class UniqueConstraintTests ( TestCase ): ...... 811| @skipUnlessDBFeature ( "supports_partial_indexes" ) 812| def test_validate_conditon_custom_error ( self ): 813| p1 = UniqueConstraintConditionProduct . objects . create ( name = "p1" ) 814| constraint = UniqueConstraintConditionProduct . _meta . constraints [0] 815| constraint . violation_error_message = "Custom message" 816| constraint . violation_error_code = "custom_code" 817| msg = "Custom message" 818| with self . assertRaisesMessage ( ValidationError , msg ) as cm : 819| constraint . validate ( 820| UniqueConstraintConditionProduct , 821| UniqueConstraintConditionProduct ( name = p1 . name , color = None ), 822| ) 823| self . assertEqual ( cm . exception . code , "custom_code" ) Reproduction Test Command: ⬇ ./ tests / runtests . py \ -- verbosity 2 \ -- settings = test_sqlite \ -- parallel 1 \ constraints . tests . BaseConstraintTests . test_custom_violation_code_message . test_custom_violation_code_message \ constraints . tests . BaseConstraintTests . test_deconstruction . test_deconstruction \ constraints . tests . CheckConstraintTests . test_eq . test_eq \ constraints . tests . CheckConstraintTests . test_repr_with_violation_error_code . test_repr_with_violation_error_code \ constraints . tests . CheckConstraintTests . test_validate_custom_error . test_validate_custom_error \ constraints . tests . UniqueConstraintTests . test_eq . test_eq \ constraints . tests . UniqueConstraintTests . test_repr_with_violation_error_code . test_repr_with_violation_error_code \ constraints . tests . UniqueConstraintTests . test_validate_conditon_custom_error . test_validate_conditon_custom_error F.3 Edit Location ⬇ { "instance_id" : "pro/golang/future-architect__vuls-54e73c2" , "detector/detector.go" : { "line_ranges" : [[134, 142], [145, 151], [154, 166]], "code_snippets" : [ "......" , "35|// Detect vulns and fill CVE detailed information" , "36|func Detect(dbclient DBClient, rs []models.ScanResult, dir string) ([]models.ScanResult, error) {" , "37|" , "38| // Use the same reportedAt for all rs" , "39| reportedAt := time.Now()" , "40| for i, r := range rs {" , "41| if !c.Conf.RefreshCve && !needToRefreshCve(r) {" , "42| logging.Log.Info(’No need to refresh’)" , "43| continue" , "44| }" , "45|" , "46| if !reuseScannedCves(&r) {" , "47| r.ScannedCves = models.VulnInfos{}" , "48| }" , "As the content is too long, parts of this example about the dependencies of the Edit Location in the same file have been omitted; the actual input provided to the agent is complete." "114| rs[i] = r" , "115| }" , "116|" , "117| // Overwrite the json file every time to clear the fields specified in config.IgnoredJSONKeys" , "118| for _, r := range rs {" , "119| if s, ok := c.Conf.Servers[r.ServerName]; ok {" , "120| r = r.ClearFields(s.IgnoredJSONKeys)" , "121| }" , "122| //TODO don’t call here" , "123| if err := reporter.OverwriteJSONFile(dir, r); err != nil {" , "124| return nil, xerrors.Errorf(’Failed to write JSON: %w’, err)" , "125| }" , "126| }" , "127|" , "128| if c.Conf.DiffPlus || c.Conf.DiffMinus {" , "129| prevs, err := loadPrevious(rs)" , "130| if err != nil {" , "131| return nil, err" , "132| }" , "133| rs = diff(rs, prevs, c.Conf.DiffPlus, c.Conf.DiffMinus)" , "134| }" , "135|" , "136| for i, r := range rs {" , "137| r = r.FilterByCvssOver(c.Conf.CvssScoreOver)" , "138| r = r.FilterUnfixed(c.Conf.IgnoreUnfixed)" , "139| r = r.FilterInactiveWordPressLibs(c.Conf.WpScan.DetectInactive)" , "140|" , "141| // IgnoreCves" , "142| ignoreCves := []string{}" , "143| if r.Container.Name == ’’ {" , "144| ignoreCves = c.Conf.Servers[r.ServerName].IgnoreCves" , "145| } else if con, ok := c.Conf.Servers[r.ServerName].Containers[r.Container.Name]; ok {" , "146| ignoreCves = con.IgnoreCves" , "147| }" , "148| r = r.FilterIgnoreCves(ignoreCves)" , "149|" , "150| // ignorePkgs" , "151| ignorePkgsRegexps := []string{}" , "152| if r.Container.Name == ’’ {" , "153| ignorePkgsRegexps = c.Conf.Servers[r.ServerName].IgnorePkgsRegexp" , "154| } else if s, ok := c.Conf.Servers[r.ServerName].Containers[r.Container.Name]; ok {" , "155| ignorePkgsRegexps = s.IgnorePkgsRegexp" , "156| }" , "157| r = r.FilterIgnorePkgs(ignorePkgsRegexps)" , "158|" , "159| // IgnoreUnscored" , "160| if c.Conf.IgnoreUnscoredCves {" , "161| r.ScannedCves = r.ScannedCves.FindScoredVulns()" , "162| }" , "163|" , "164| rs[i] = r" , "165| }" , "166| return rs, nil" , "167|}" , "......" , "169|// DetectPkgCves detects OS pkg cves" , "170|func DetectPkgCves(dbclient DBClient, r *models.ScanResult) error {" , "171| // Pkg Scan" , "172| if r.Release != ’’ {" , "173| // OVAL" , "174| if err := detectPkgsCvesWithOval(dbclient.OvalDB, r); err != nil {" , "175| return xerrors.Errorf(’Failed to detect CVE with OVAL: %w’, err)" , "176| }" , "177|" , "178| // gost" , "179| if err := detectPkgsCvesWithGost(dbclient.GostDB, r); err != nil {" , "180| return xerrors.Errorf(’Failed to detect CVE with gost: %w’, err)" , "181| }" , "182| } else if reuseScannedCves(r) {" , "183| logging.Log.Infof(’r.Release is empty. Use CVEs as it as.’)" , "184| } else if r.Family == constant.ServerTypePseudo {" , "185| logging.Log.Infof(’pseudo type. Skip OVAL and gost detection’)" , "186| } else {" , "187| return xerrors.Errorf(’Failed to fill CVEs. r.Release is empty’)" , "188| }" , "189|" , "190| for i, v := range r.ScannedCves {" , "191| for j, p := range v.AffectedPackages {" , "192| if p.NotFixedYet && p.FixState == ’’ {" , "193| p.FixState = ’Not fixed yet’" , "194| r.ScannedCves[i].AffectedPackages[j] = p" , "195| }" , "196| }" , "197| }" , "198|" , "199| // To keep backward compatibility" , "200| // Newer versions use ListenPortStats," , "201| // but older versions of Vuls are set to ListenPorts." , "202| // Set ListenPorts to ListenPortStats to allow newer Vuls to report old results." , "203| for i, pkg := range r.Packages {" , "204| for j, proc := range pkg.AffectedProcs {" , "205| for _, ipPort := range proc.ListenPorts {" , "206| ps, err := models.NewPortStat(ipPort)" , "207| if err != nil {" , "208| logging.Log.Warnf(’Failed to parse ip:port: %s, err:%+v’, ipPort, err)" , "209| continue" , "210| }" , "211| r.Packages[i].AffectedProcs[j].ListenPortStats = append(" , "212| r.Packages[i].AffectedProcs[j].ListenPortStats, *ps)" , "213| }" , "214| }" , "215| }" , "216|" , "217| return nil" , "218|}" , "......" , ] }, "detector/wordpress.go" : { "line_ranges" : [[110, 116]], "code_snippets" : [ "......" , "51|// DetectWordPressCves access to wpscan and fetch scurity alerts and then set to the given ScanResult." , "52|// https://wpscan.com/" , "53|func detectWordPressCves(r *models.ScanResult, cnf *c.WpScanConf) (int, error) {" , "54| if len(r.WordPressPackages) == 0 {" , "55| return 0, nil" , "56| }" , "57| // Core" , "58| ver := strings.Replace(r.WordPressPackages.CoreVersion(), ’.’, ’’, -1)" , "59| if ver == ’’ {" , "60| return 0, errof.New(errof.ErrFailedToAccessWpScan," , "61| fmt.Sprintf(’Failed to get WordPress core version.’))" , "62| }" , "63| url := fmt.Sprintf(’https://wpscan.com/api/v3/wordpresses/%s’, ver)" , "64| wpVinfos, err := wpscan(url, ver, cnf.Token)" , "65| if err != nil {" , "66| return 0, err" , "67| }" , "68|" , "69| // Themes" , "70| themes := r.WordPressPackages.Themes()" , "71| if !cnf.DetectInactive {" , "72| themes = removeInactives(themes)" , "73| }" , "74| for _, p := range themes {" , "75| url := fmt.Sprintf(’https://wpscan.com/api/v3/themes/%s’, p.Name)" , "76| candidates, err := wpscan(url, p.Name, cnf.Token)" , "77| if err != nil {" , "78| return 0, err" , "79| }" , "80| vulns := detect(p, candidates)" , "81| wpVinfos = append(wpVinfos, vulns...)" , "82| }" , "83|" , "84| // Plugins" , "85| plugins := r.WordPressPackages.Plugins()" , "86| if !cnf.DetectInactive {" , "87| plugins = removeInactives(plugins)" , "88| }" , "89| for _, p := range plugins {" , "90| url := fmt.Sprintf(’https://wpscan.com/api/v3/plugins/%s’, p.Name)" , "91| candidates, err := wpscan(url, p.Name, cnf.Token)" , "92| if err != nil {" , "93| return 0, err" , "94| }" , "95| vulns := detect(p, candidates)" , "96| wpVinfos = append(wpVinfos, vulns...)" , "97| }" , "98|" , "99| for _, wpVinfo := range wpVinfos {" , "100| if vinfo, ok := r.ScannedCves[wpVinfo.CveID]; ok {" , "101| vinfo.CveContents[models.WpScan] = wpVinfo.CveContents[models.WpScan]" , "102| vinfo.VulnType = wpVinfo.VulnType" , "103| vinfo.Confidences = append(vinfo.Confidences, wpVinfo.Confidences...)" , "104| vinfo.WpPackageFixStats = append(vinfo.WpPackageFixStats, wpVinfo.WpPackageFixStats...)" , "105| r.ScannedCves[wpVinfo.CveID] = vinfo" , "106| } else {" , "107| r.ScannedCves[wpVinfo.CveID] = wpVinfo" , "108| }" , "109| }" , "110| return len(wpVinfos), nil" , "111|}" , "112|" , "113|func wpscan(url, name, token string) (vinfos []models.VulnInfo, err error) {" , "114| body, err := httpRequest(url, token)" , "115| if err != nil {" , "116| return nil, err" , "117| }" , "118| if body == ’’ {" , "119| logging.Log.Debugf(’wpscan.com response body is empty. URL: %s’, url)" , "120| }" , "121| return convertToVinfos(name, body)" , "122|}" , "......" , "124|func detect(installed models.WpPackage, candidates []models.VulnInfo) (vulns []models.VulnInfo) {" , "125| for _, v := range candidates {" , "126| for _, fixstat := range v.WpPackageFixStats {" , "127| ok, err := match(installed.Version, fixstat.FixedIn)" , "128| if err != nil {" , "129| logging.Log.Warnf(’Failed to compare versions %s installed: %s, fixedIn: %s, v: %+v’," , "130| installed.Name, installed.Version, fixstat.FixedIn, v)" , "131| // continue scanning" , "132| continue" , "133| }" , "134| if ok {" , "135| vulns = append(vulns, v)" , "136| logging.Log.Debugf(’Affected: %s installed: %s, fixedIn: %s’," , "137| installed.Name, installed.Version, fixstat.FixedIn)" , "138| } else {" , "139| logging.Log.Debugf(’Not affected: %s : %s, fixedIn: %s’," , "140| installed.Name, installed.Version, fixstat.FixedIn)" , "141| }" , "142| }" , "143| }" , "144| return" , "145|}" , "......" , ] } } F.4 API Usage ⬇ { "instance_id" : "sphinx-doc__sphinx-7462" , "sphinx/domains/python.py" : [ { "file of api definition" : "/testbed/sphinx/domains/python.py" , "lineno of api definition" : [92, 111], "api function name" : "unparse" , "api function should be called in this way" : "unparse(elem)" , "the approximate lineno where this api appear in the file you need to modify" : 111, "api function definition" : [ "def _parse_annotation(annotation: str) -> List[Node]:" , " ’’’Parse type annotation.’’’" , " def make_xref(text: str) -> addnodes.pending_xref:" , " if text == ’None’:" , " reftype = ’obj’" , " else:" , " reftype = ’class’" , "" , " return pending_xref(’’, nodes.Text(text)," , " refdomain=’py’, reftype=reftype, reftarget=text)" , "" , " def unparse(node: ast.AST) -> List[Node]:" , " if isinstance(node, ast.Attribute):" , " return [nodes.Text(’%s.%s’ % (unparse(node.value)[0], node.attr))]" , " elif isinstance(node, ast.Expr):" , " return unparse(node.value)" , " elif isinstance(node, ast.Index):" , " return unparse(node.value)" , " elif isinstance(node, ast.List):" , " result = [addnodes.desc_sig_punctuation(’’, ’[’)] # type: List[Node]" , " for elem in node.elts:" , " result.extend(unparse(elem))" , " result.append(addnodes.desc_sig_punctuation(’’, ’, ’))" , " result.pop()" , " result.append(addnodes.desc_sig_punctuation(’’, ’]’))" , " return result" , " elif isinstance(node, ast.Module):" , " return sum((unparse(e) for e in node.body), [])" , " elif isinstance(node, ast.Name):" , " return [nodes.Text(node.id)]" , " elif isinstance(node, ast.Subscript):" , " result = unparse(node.value)" , " result.append(addnodes.desc_sig_punctuation(’’, ’[’))" , " result.extend(unparse(node.slice))" , " result.append(addnodes.desc_sig_punctuation(’’, ’]’))" , " return result" , " elif isinstance(node, ast.Tuple):" , " if node.elts:" , " result = []" , " for elem in node.elts:" , " result.extend(unparse(elem))" , " result.append(addnodes.desc_sig_punctuation(’’, ’, ’))" , " result.pop()" , " else:" , " result = [addnodes.desc_sig_punctuation(’’, ’(’)," , " addnodes.desc_sig_punctuation(’’, ’)’)]" , "" , " return result" , " else:" , " raise SyntaxError # unsupported syntax" , "" , " try:" , " tree = ast_parse(annotation)" , " result = unparse(tree)" , " for i, node in enumerate(result):" , " if isinstance(node, nodes.Text):" , " result[i] = make_xref(str(node))" , " return result" , " except SyntaxError:" , " return [make_xref(annotation)]" , ] }, { "file of api definition" : "/opt/miniconda3/envs/testbed/lib/python3.9/site-packages/docutils/nodes.py" , "lineno of api definition" : [692, 692], "api function name" : "append" , "api function should be called in this way" : "result.append(addnodes.desc_sig_punctuation(’’, ’, ’))" , "the approximate lineno where this api appear in the file you need to modify" : 112, "api function definition" : [ "class Element(Node):" , "......" , " def append(self, item):" , " self.setup_child(item)" , " self.children.append(item)" , ] } ], "sphinx/ext/autodoc/__init__.py" : [ { "file of api definition" : "/testbed/sphinx/util/inspect.py" , "lineno of api definition" : [ 393, 398 ], "api function name" : "isproperty" , "api function should be called in this way" : "inspect.isproperty(member)" , "the approximate lineno where this api appear in the file you need to modify" : 2665, "api function definition" : [ "def isproperty(obj: Any) -> bool:" , " ’’’Check if the object is property.’’’" , " if sys.version_info >= (3, 8):" , " from functools import cached_property # cached_property is available since py3.8" , " if isinstance(obj, cached_property):" , " return True" , "" , " return isinstance(obj, property)" , ] }, { "file of api definition" : "/testbed/sphinx/events.py" , "lineno of api definition" : [ 89, 89 ], "api function name" : "emit" , "api function should be called in this way" : "self.env.app.events.emit(\" source - read \ ", docname, arg)" , "the approximate lineno where this api appear in the file you need to modify" : 392, "api function definition" : [ "class EventManager:" , "......" , " def emit(self, name: str, *args: Any," , " allowed_exceptions: tuple[type[Exception], ...] = ()) -> list:" , " ’’’Emit a Sphinx event.’’’" , "" , " # not every object likes to be repr()’d (think" , " # random stuff coming via autodoc)" , " with contextlib.suppress(Exception):" , " logger.debug(’[app] emitting event: %r%s’, name, repr(args)[:100])" , "" , " results = []" , " listeners = sorted(self.listeners[name], key=attrgetter(’priority’))" , " for listener in listeners:" , " try:" , " results.append(listener.handler(self.app, *args))" , " except allowed_exceptions:" , " # pass through the errors specified as *allowed_exceptions*" , " raise" , " except SphinxError:" , " raise" , " except Exception as exc:" , " if self.app.pdb:" , " # Just pass through the error, so that it can be debugged." , " raise" , " modname = safe_getattr(listener.handler, ’__module__’, None)" , " raise ExtensionError(__(’Handler %r for event %r threw an exception’) %" , " (listener.handler, name), exc, modname=modname) from exc" , " return results" , ] } ] } F.5 Execution Context instance_id: live/python-attrs__attrs-1319 Category: Native Error Stack Trace with Error Message: SyntaxError("invalid syntax") Stack: ⬇ [ [ "/usr/local/lib/python3.11/site-packages/_pytest/runner.py" , 242, "" , [] ], [ "/usr/local/lib/python3.11/site-packages/_pytest/threadexception.py" , 92, "pytest_runtest_call" , [] ], "/usr/local/lib/python3.11/site-packages/_pytest/python.py" , 159, "pytest_pyfunc_call" , [] ], "/testbed/tests/test_make.py" , 704, "test_pre_init_kw_only_work_with_defaults" , [] ], [ "/testbed/src/attr/_next_gen.py" , 402, "define" , [ "387: break" , "388:" , "389: if auto_attribs is not None:" , "390: return do_it(cls, auto_attribs)" , "391:" , "392: try:" , "393: return do_it(cls, True)" , "394: except UnannotatedAttributeError:" , "395: return do_it(cls, False)" , "396:" , "397: # maybe_cls’s type depends on the usage of the decorator. It’s a class" , "398: # if it’s used as ‘@attrs‘ but ‘None‘ if used as ‘@attrs()‘." , "399: if maybe_cls is None:" , "400: return wrap" , "401:" , "402: return wrap(maybe_cls)" , "403:" , "404:" , "405:mutable = define" , "406:frozen = partial(define, frozen=True, on_setattr=None)" , "407:" , "408:" , "409:def field(" , "410: *," , "411: default=NOTHING," , "412: validator=None," , "413: repr=True," , "414: hash=None," , "415: init=True," , "416: metadata=None," , "417: type=None," , ] ], [ "/testbed/src/attr/_next_gen.py" , 393, "wrap" , [ "378: # However, if we subclass a frozen class, we inherit the immutability" , "379: # and disable on_setattr." , "380: for base_cls in cls.__bases__:" , "381: if base_cls.__setattr__ is _frozen_setattrs:" , "382: if had_on_setattr:" , "383: msg = ’Frozen classes can’t use on_setattr (frozen-ness was inherited).’" , "384: raise ValueError(msg)" , "385:" , "386: on_setattr = setters.NO_OP" , "387: break" , "388:" , "389: if auto_attribs is not None:" , "390: return do_it(cls, auto_attribs)" , "391:" , "392: try:" , "393: return do_it(cls, True)" , "394: except UnannotatedAttributeError:" , "395: return do_it(cls, False)" , "396:" , "397: # maybe_cls’s type depends on the usage of the decorator. It’s a class" , "398: # if it’s used as ‘@attrs‘ but ‘None‘ if used as ‘@attrs()‘." , "399: if maybe_cls is None:" , "400: return wrap" , "401:" , "402: return wrap(maybe_cls)" , "403:" , "404:" , "405:mutable = define" , "406:frozen = partial(define, frozen=True, on_setattr=None)" , ] ], [ "/testbed/src/attr/_next_gen.py" , 339, "do_it" , [ "324: - *slots=True*" , "325:" , "326: Usually, this has only upsides and few visible effects in everyday" , "327: programming. But it *can* lead to some surprising behaviors, so" , "328: please make sure to read :term:‘slotted classes‘." , "329:" , "330: - *auto_exc=True*" , "331: - *auto_detect=True*" , "332: - *order=False*" , "333: - Some options that were only relevant on Python 2 or were kept around" , "334: for backwards-compatibility have been removed." , "335:" , "336: ’’’" , "337:" , "338: def do_it(cls, auto_attribs):" , "339: return attrs(" , "340: maybe_cls=cls," , "341: these=these," , "342: repr=repr," , "343: hash=hash," , "344: unsafe_hash=unsafe_hash," , "345: init=init," , "346: slots=slots," , "347: frozen=frozen," , "348: weakref_slot=weakref_slot," , "349: str=str," , "350: auto_attribs=auto_attribs," , "351: kw_only=kw_only," , "352: cache_hash=cache_hash," , "353: auto_exc=auto_exc," , "354: eq=eq," , ] ], [ "/testbed/src/attr/_make.py" , 1403, "attrs" , [ "1388:" , "1389: if (" , "1390: PY_3_10_PLUS" , "1391: and match_args" , "1392: and not _has_own_attribute(cls, ’__match_args__’)" , "1393: ):" , "1394: builder.add_match_args()" , "1395:" , "1396: return builder.build_class()" , "1397:" , "1398: # maybe_cls’s type depends on the usage of the decorator. It’s a class" , "1399: # if it’s used as ‘@attrs‘ but ‘None‘ if used as ‘@attrs()‘." , "1400: if maybe_cls is None:" , "1401: return wrap" , "1402:" , "1403: return wrap(maybe_cls)" , "1404:" , "1405:" , "1406:_attrs = attrs" , "1407:’’’" , "1408:Internal alias so we can use it in functions that take an argument called" , "1409:*attrs*." , "1410:’’’" , "1411:" , "1412:" , "1413:def _has_frozen_base_class(cls):" , "1414: ’’’" , "1415: Check whether *cls* has a frozen ancestor by looking at its" , "1416: __setattr__." , "1417: ’’’" , "1418: return cls.__setattr__ is _frozen_setattrs" , ] ], [ "/testbed/src/attr/_make.py" , 1382, "wrap" , [ "<...As the content is too long, parts of this example have been omitted; the actual information provided to the agent is complete...>" ] ], [ "/testbed/src/attr/_make.py" , 972, "add_init" , [ "<...As the content is too long, parts of this example have been omitted; the actual information provided to the agent is complete...>" ] ], [ "/testbed/src/attr/_make.py" , 1875, "_make_init" , [ "<...As the content is too long, parts of this example have been omitted; the actual information provided to the agent is complete...>" ] ], [ "/testbed/src/attr/_make.py" , 239, "_make_method" , [ "224: base_filename = filename" , "225: while True:" , "226: linecache_tuple = (" , "227: len(script)," , "228: None," , "229: script.splitlines(True)," , "230: filename," , "231: )" , "232: old_val = linecache.cache.setdefault(filename, linecache_tuple)" , "233: if old_val == linecache_tuple:" , "234: break" , "235:" , "236: filename = f’{base_filename[:-1]}-{count}>’" , "237: count += 1" , "238:" , "239: _compile_and_eval(script, globs, locs, filename)" , "240:" , "241: return locs[name]" , "242:" , "243:" , "244:def _make_attr_tuple_class(cls_name, attr_names):" , "245: ’’’" , "246: Create a tuple subclass to hold ‘Attribute‘s for an ‘attrs‘ class." , "247:" , "248: The subclass is a bare tuple with properties for names." , "249:" , "250: class MyClassAttributes(tuple):" , "251: __slots__ = ()" , "252: x = property(itemgetter(0))" , "253: ’’’" , "254: attr_class_name = f’{cls_name}Attributes’" , ] ], [ "/testbed/src/attr/_make.py" , 211, "_compile_and_eval" , [ "196: kw_only=kw_only," , "197: eq=eq," , "198: eq_key=eq_key," , "199: order=order," , "200: order_key=order_key," , "201: on_setattr=on_setattr," , "202: alias=alias," , "203: )" , "204:" , "205:" , "206:def _compile_and_eval(script, globs, locs=None, filename=’’):" , "207: ’’’" , "208: Evaluate the script with the given global (globs) and local (locs)" , "209: variables." , "210: ’’’" , "211: bytecode = compile(script, filename, ’exec’)" , "212: eval(bytecode, globs, locs)" , "213:" , "214:" , "215:def _make_method(name, script, filename, globs, locals=None):" , "216: ’’’" , "217: Create the method with the script given and return the method object." , "218: ’’’" , "219: locs = {} if locals is None else locals" , "220:" , "221: # In order of debuggers like PDB being able to step through the code," , "222: # we add a fake linecache entry." , "223: count = 1" , "224: base_filename = filename" , "225: while True:" , "226: linecache_tuple = (" , ] ] ] Appendix G Prompts for SWE-agent to Inject Oracle Information This section provides how we inject oracle information of five factors into user prompt to run SWE-agent. ⬇ # gt is the dict to store ground truth of the five factors. USER_PROMPT = "" if edit_location : USER_PROMPT += "Your colleague has found all the locations that should be edited to resolve the issue:\n" for location , code in gt [ "location_content" ]. items (): USER_PROMPT += f "#{location}: \n{code}\n\n" USER_PROMPT += f "" " Your colleague has determined that only these locations should be edited. You must edit all of these locations one by one. You must not edit any locations not listed here. As your colleagues have done the localization step for you, you don’t need to do this step again. You can of course view other contexts relevant to these buggy locations to understand the source codes better. But locations not listed here should not be edited! " "" if reproduction : self . sandbox_environment . communicate ( f "" "git apply - <<’NEW_PATCH’\n{gt[" test_patch "]}\nNEW_PATCH" "" ) USER_PROMPT = "Your colleague has written the correct reproduction tests in the test folder and has figured out how to run regression tests in the repository.\n" USER_PROMPT += "The testcases to reproduce the bugs are:\n" for test_name , code in gt [ "F2P_content" ]. items (): USER_PROMPT += f "# {test_name}:\n{code}\n" USER_PROMPT += f "The command to run these reproduction testcases is: \n{gt[" f2p_cmd "]}\n\n" USER_PROMPT += f "The list of reproduction testcases that you must pass: \n{gt[" FAIL_TO_PASS "]}\n\n" USER_PROMPT += "As your colleague has written the reproduction tests for you, you do not need to do this again. After you modified the source codes, please run reproduction tests to validate your fix. You must pass all the reproduction tests before you submit.\n" if regression : USER_PROMPT += f "The command to run regression tests is: \n{test_cmd} {write_to_file}\n\n" USER_PROMPT += f "The list of regression testcases that you must pass: \n{condense_too_long_P2P(gt[" PASS_TO_PASS "])}\n\n" USER_PROMPT += "As your colleague has found the regression tests for you, you do not need to do this again. After you modified the source codes, please run regression tests to validate your fix. You must pass all the regression tests specified above before you submit.\n" if context : USER_PROMPT = f "" " Hints: Your colleague has provided the error stack traces from running reproduction testcaces related to this problem. Each layer of trace is in the format of (file path, line number, function name, source code) {gt[" error_stack "]} The error stack traces can be some hints to help you locate the locations to be edited, and understand the execution contexts which affect the locations to be edited or your edits would affect. " "" if api : USER_PROMPT = f "" " Hints: Your colleague has listed the functions you need to use to edit the source code. The function information format is a mapping: Dict [ " the path of the file you need to edit " : List [ " the information of each function you need to use to edit this file " ] ] {gt[" api "]} You need to use ALL of the function calls listed above. You need to use the exactly same arguments in the " api function should be called in this way " field in each piece of function information to use each function. Some of the function information has best-guess function definition from our development tools, but may not be accurate, so you need to judge and find the correct function definition if necessary. " "" if reproduction and regression and location : user_requirement = ( "Following your colleagues’ contributions, your next steps should be:\n" "1) Explore the source codes if you want to understand the contexts of the buggy locations or testcases better. But remember that all the locations that need to edited have been listed above.\n" "2) Edit all the locations specified by your colleagues to resolve the issue.\n" "3) Run reproduction and regression tests to validate your fix. If any of the testcases required above fails, go back to explore and edit again.\n" "4) If you pass both reproduction and regression tests, submit.\n" ) elif reproduction and regression : USER_PROMPT = "" " Following your colleagues’ contributions, your next steps should be: 1) As a first step, it might be a good idea to find and read code relevant to the . 2) Edit the codes to resolve the issue. 3) Run reproduction and regression tests to validate your fix. If any of the testcases required above fails, go back to explore and edit again. 4) If you pass both reproduction and regression tests, submit. " "" ) elif location : USER_PROMPT = "" " Following your colleagues’ contributions, your next steps should be: 1) Explore the source codes if you want to understand the contexts of the buggy locations or testcases better. But remember that all the locations that need to edited have been listed above. 2) Create a script to reproduce the problem and execute it using the bash tool to confirm the problem. 3) Edit all the locations specified by your colleagues to resolve the issue.\n" 4) Run your reproduction script to validate your fix . Due to time limit , do not run the regression tests in the repository . If the reproduction script still fails , go back to explore and edit again . 5) If you pass your reproduction script , submit . "" " elif reproduction: USER_PROMPT = " "" Following your colleagues ’ contributions, your next steps should be:\n" 1) As a first step, it might be a good idea to find and read code relevant to the . 2) Edit the codes to resolve the issue. 3) Run reproduction tests to validate your fix. Due to time limit, do not run the regression tests in the repository. If any of the reproduction testcases listed above fails, go back to explore and edit again. 4) If you pass ALL the reproduction tests, submit. """ ) elif regression: USER_PROMPT = """ Follow these steps to resolve the issue: 1. As a first step, it might be a good idea to find and read code relevant to the . 2. Create a script to reproduce the problem and execute it using the bash tool to confirm the problem. 3. Edit the sourcecode of the repo to resolve the issue. 4. Rerun your reproduce script to confirm that the error is fixed. Run the regression tests found by your colleague to check that your edits have not broken anything else. If your reproduce script or any of the regression testcases required above fail, go back to explore and edit again. 5. If you pass both your reproduce script and all the regression tests found by your colleague, submit your answer. """ else: USER_PROMPT = """ Follow these steps to resolve the issue: 1. As a first step, it might be a good idea to find and read code relevant to the 2. Create a script to reproduce the error and execute it using the bash tool to confirm the error 3. Edit the sourcecode of the repo to resolve the issue 4. Rerun your reproduce script and confirm that the error is fixed! 5. Think about edgecases and make sure your fix handles them as well Your thinking should be thorough and so it’ s fine if it ’s very long. """’ Appendix H Agent Prompt of Validation Experiment Stage 1 (Signal Extraction) ⬇ if reproduction : if lang == "python" : target_file = "/testbed/reproduction.py" elif lang == "go" : target_file = "/testbed/reproduction_test.go" else : target_file = "/testbed/answer.json" if location : message = f "" " You need to find the locations that need to be edited to resolve the issue. You don’t need to edit the code. You only need to find the locations that need to be edited. You answer should be written into /testbed/answer.json We will check the correctness of your answer by reading /testbed/answer.json In /testbed/answer.json, your answer format should be like: Once you have completed /testbed/answer.json, submit. " "" if regression : message = f "" " You need to find the correct command to run original regression tests in the repository, and parse the test output to record passed testcases and failed testcases. For many repos the regression command is not trivial -- " pytest . / gotest . " does not resolve everything. You need to read the repo documents to figure out the correct way to run regression tests -- the test command, the evironment variables to set... Usually you don’t need to run all tests in the repository -- it’s too time-consuming. It is recommended to only test codes related to the issue description in your test command. After regression test, direct test output to log file and parse the test output to record passed and failed testcases. If the test framework allows JSON / XML structured output, use it to help log parsing. You answer should be written into /testbed/answer.json We will check the correctness of your answer by reading /testbed/answer.json In /testbed/answer.json, your answer format should be like: You don’t need to edit the source codes. Once you have completed /testbed/answer.json, submit. " "" if reproduction : if lang == "python" : target_func = "" cmd = "cd /testbed/ && python reproduction.py" elif lang == "go" : target_func = "The entry of your test file should be func TestReproduce(t *testing.T) " cmd = "cd /testbed/ && go test -run TestReproduce -v" message = f "" " You need to write a reproduction test file to help your colleague reproduce the issue. Your reproduction file, when executed, should exactly reveal the issue in the issue description. Your reproduction file should fail before the issue is resolved and should pass after the issue is correctly fixed. Your should write multiple testcases in the reproduction file to cover all corner cases to ensure all cases are resolved correclty. Your reproduction file should be written in {target_file} using ‘str_replace_editor create‘ {target_func} We will only extract contents in {target_file} to check the correctness of your reproduction. Don’t write anything elsewhere. Your reproduction test file should be run with ‘{cmd}‘ After you write the reproduction test file, run it with ‘{cmd}‘ to confirm it can reproduce the issue in the description. You don’t need to edit the source codes. Once you have run the {target_file} to confirm it can reproduce the issue, submit. " "" if api : message = f "" " You need to find: 1) all the API functions that need to be used to edit the code to resolve the issue; 2) the important API functions that are used around the locations to be edited which can help understand how to edit the code. These API functions are crucial for understanding how to edit the code and avoiding implementing existing utilities again. You need to determine where the API functions are used / will be used for editing the source code, and find the definition of these API functions. You may need to locate the suspicous locations that might need to be edited to resolve the issue first, before you can discover the necessary API functions to help understand / edit these locations. You answer should be written into /testbed/answer.json We will check the correctness of your answer by reading /testbed/answer.json In /testbed/answer.json, your answer format should be like: You don’t need to edit the source codes. Once you have completed /testbed/answer.json, submit. " "" if context : message = f "" " You need to find the locations relevant to the issue description, not limited to the locations to be edited, for your colleague. Organize the relations of potentially relevant locations you find into function call stacks. You answer should be written into /testbed/answer.json We will check the correctness of your answer by reading /testbed/answer.json In /testbed/answer.json, your answer format should be like: [ [ [" file path 1 ", line number, " function name a ", " code snippet "], [" file path 2 ", line number, " function name b ", " code snippet "], [" file path 3 ", line number, " function name c ", " code snippet "], ], [ stack 2 ... ], ... ] where a calls b, and b calls c. For example, Requirements: (1) Include both the suspicous locations that may need to be edited, and the code contexts related to these locations to be edited, which may affect the locations to be edited or the edited locations may affect, in your final answer. (2) Some relevant locations could be found by error stack trace of running issue reproduction test code -- this is the most optimistic cases. You can try to use this way first to get the answer stacks. (3) For problems that cannot be located with error stack of running issue reproduction test code, find a) the locations to be edited first and then find b) the functions which call these edit locations and these edit locations call. Organize the function call relations of BOTH a AND b into stacks. (4) Organize the relations of potentially relevant locations you find into list of function call stacks. You don’t need to edit the source codes. Once you have completed /testbed/answer.json, submit. " "" BETA