Aws Introduces Swe-polybench: A New Open-source Multilingual Benchmark For Evaluating Ai Coding Agents

Recent advancements successful ample connection models (LLMs) person enabled nan improvement of AI-based coding agents that tin generate, modify, and understand package code. However, nan information of these systems remains limited, often constrained to synthetic aliases narrowly scoped benchmarks, chiefly successful Python. These benchmarks seldom bespeak nan structural and semantic diverseness of real-world codebases, and arsenic a result, galore agents overfit to benchmark-specific patterns alternatively than demonstrating robust, transferable capabilities.

AWS Introduces SWE-PolyBench: A More Comprehensive Evaluation Framework

To reside these challenges, AWS AI Labs has introduced SWE-PolyBench, a multilingual, repository-level benchmark designed for execution-based information of AI coding agents. The benchmark spans 21 GitHub repositories crossed 4 widely-used programming languages—Java, JavaScript, TypeScript, and Python—comprising 2,110 tasks that see bug fixes, characteristic implementations, and codification refactorings.

Unlike anterior benchmarks, SWE-PolyBench incorporates existent propulsion requests (PRs) that adjacent existent issues and see associated trial cases, allowing for verifiable evaluation. A smaller, stratified subset—SWE-PolyBench500—has besides been released to support quicker experimentation while preserving task and connection diversity.

Technical Structure and Evaluation Metrics

SWE-PolyBench adopts an execution-based information pipeline. Each task includes a repository snapshot and a problem connection derived from a GitHub issue. The strategy applies nan associated crushed truth spot successful a containerized trial situation configured for nan respective connection ecosystem (e.g., Maven for Java, npm for JS/TS, etc.). The benchmark past measures outcomes utilizing 2 types of portion tests: fail-to-pass (F2P) and pass-to-pass (P2P).

To supply a much granular appraisal of coding agents, SWE-PolyBench introduces Concrete Syntax Tree (CST)-based metrics. These see some file-level and node-level retrieval scores, assessing nan agent’s expertise to find and modify applicable sections of nan codebase. These metrics connection insights beyond binary pass/fail outcomes, particularly for complex, multi-file modifications.

Empirical Evaluation and Observations

Three open-source coding agents—Aider, SWE-Agent, and Agentless—were adapted for SWE-PolyBench. All utilized Anthropic’s Claude 3.5 arsenic nan underlying exemplary and were modified to grip nan multilingual, repository-level requirements of nan benchmark.

The information revealed notable differences successful capacity crossed languages and task types. For instance, agents performed champion connected Python tasks (up to 24.1% walk rate) but struggled pinch TypeScript (as debased arsenic 4.7%). Java, contempt its higher complexity successful position of mean node changes, achieved higher occurrence rates than TypeScript, suggesting that pretraining vulnerability and syntax familiarity play a captious domiciled successful exemplary performance.

Performance besides varied pinch task complexity. Tasks constricted to single-function aliases single-class changes yielded higher occurrence rates (up to 40%), while those requiring mixed aliases multi-file changes saw a important drop. Interestingly, precocious retrieval precision and recall—particularly for record and CST node identification—did not ever construe to higher walk rates, indicating that codification localization is basal but insufficient for problem resolution.

Conclusion: Toward Robust Evaluation of AI Coding Agents

SWE-PolyBench presents a robust and nuanced information model for coding agents, addressing cardinal limitations successful existing benchmarks. By supporting aggregate programming languages, covering a wider scope of task types, and incorporating syntax-aware metrics, it offers a much typical appraisal of an agent’s real-world applicability.

The benchmark reveals that while AI agents grounds promising capabilities, their capacity remains inconsistent crossed languages and tasks. SWE-PolyBench provides a instauration for early investigation aimed astatine improving nan generalizability, robustness, and reasoning capabilities of AI coding assistants.

Check retired nan AWS DevOps Blog, Hugging Face – SWE-PolyBench and GitHub – SWE-PolyBench. Also, don’t hide to travel america on Twitter and subordinate our Telegram Channel and LinkedIn Group. Don’t Forget to subordinate our 90k+ ML SubReddit.

🔥 [Register Now] miniCON Virtual Conference connected AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 p.m. PST) + Hands connected Workshop