Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent
arXiv:2610.39417v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong performance on software engineering (SE) tasks, driving the need for high-quality datasets for reliable training and evaluation. SWE-bench-style datasets (SWE datasets) are widely used to evaluate LLM-based coding agents on real-world software issues, where task clarity and test coverage directly affect whether instances are solvable and verifiable. However, quality issues persist in existing SWE da…