Deep Dive into SWE-bench Verified & Terminal-Bench v2

Evaluating LLMs on single-function code snippets (like HumanEval) is obsolete. Modern agent benchmarking relies on SWE-bench Verified and Terminal-Bench v2 to measure multi-file patch generation and interactive command execution inside isolated sandboxes.

1. SWE-bench Verified: Methodology

SWE-bench Verified filters 500 validated real-world GitHub issues from major repositories (Django, SymPy, Scikit-learn, Sphinx). The evaluation harness executes unit tests inside Docker containers to verify that the agent's generated patch fixes the problem without causing regressions.

2. Terminal-Bench v2: Testing CLI Mechanics

Terminal-Bench v2 evaluates an agent's ability to operate in interactive Linux bash environments. Tasks test system administration, shell scripting, package debugging, and environment configuration.