Terminal-Bench 4.0: Why Agent Benchmarks Need Maintenance, Not Hype

Terminal-Bench 4.0 is one of those releases that looks boring at first, but is actually important if you care about long-horizon agents. It is not mainly saying, "look, model X beat model Y." It is saying something deeper: if the benchmark itself is noisy, broken, saturated, or resource-starved, then the leaderboard is not measuring agent capability cleanly.

That matters because agent benchmarks are not simple multiple-choice exams. They are closer to little software worlds. An agent enters a terminal, reads files, edits code, runs commands, deals with dependencies, and tries to satisfy a verifier. If the environment is flaky, the timeout is wrong, or the task is already solved by every frontier model, the final score can lie to you.

Why It Matters

Terminal-Bench measures whether AI agents can complete tasks through a terminal. That is already closer to real engineering than many static benchmarks, because the agent has to operate inside a live environment instead of only answering a prompt.

But live environments introduce a new problem: you are no longer just evaluating intelligence. You are also evaluating time limits, CPU, memory, container setup, verifier design, task ambiguity, model refusal behavior, and sometimes plain infrastructure noise.

What Changed In 4.0?

The official announcement says Terminal-Bench 4.0 does three big things: calibrates task resources, fixes tasks, and removes saturated tasks. The result is fewer timeouts and fewer agent execution errors than Terminal-Bench 3.0, which should reduce measurement noise.

That is not a cosmetic change. If a task is removed because every current model solves it every time, keeping it around can make the leaderboard less useful. It becomes free points instead of signal.

Resource Calibration

The subtle part of agent evaluation is that failure can come from the model or from the setup around the model. Did the agent fail because it could not reason through the task, or because it ran out of time? Did it fail because it chose the wrong approach, or because the sandbox resources were too tight?

Terminal-Bench 4.0 follows a resource-calibration idea from Anthropic's work on infrastructure noise in agentic coding evals. The basic idea is to run with lots of headroom first, then choose resource settings that do not distort the score. In plain English: make sure the benchmark is testing the agent, not accidentally testing the container.

If an eval result changes because the machine got slightly more CPU or memory, you may be measuring infrastructure, not capability.

Task Quality

The task changes are also a big deal. Terminal-Bench 4.0 removed eight tasks and fixed nineteen. That tells us something important: benchmark tasks are not sacred. They are code. They can have bugs, unclear instructions, bad verifiers, reward hacks, public leaks, or compatibility problems.

This is where agent benchmarks start to look like production systems. You need bug reports. You need changelogs. You need issue triage. You need to know which score changes require rerunning agents and which can be handled by regrading saved artifacts.

Continuous Benchmarks

The most interesting idea behind Terminal-Bench 4.0 is semantic versioning for benchmarks. In normal software, a major version means something broke compatibility, a minor version adds functionality, and a patch version fixes small things. Terminal-Bench is applying a similar idea to benchmark datasets.

This is why the release is 4.0 instead of 3.1. It changed resources and removed tasks, so it is a breaking benchmark update. You cannot fairly compare old trials against the new benchmark without understanding what changed.

The LaunchPulse Angle

The reason I care about this for LaunchPulse-style agent systems is that product-building agents need exactly this kind of evaluation discipline. If an agent is supposed to build a working app, generate a video, recover from errors, or operate over many steps, the benchmark cannot just ask, "did it finish?" It has to capture how it finished, what broke, what was retried, what the environment allowed, and whether the verifier was actually fair.

A good internal benchmark for a product like LaunchPulse would probably need the same principles:

• Version every task and environment like software.
• Separate agent failure from infrastructure failure.
• Track cost, time, retries, and verifier behavior.
• Remove saturated tasks when they stop separating models.
• Add harder tasks from real user failures.
• Keep artifacts so old runs can sometimes be regraded instead of fully rerun.

That is the 10x lesson here: better agents need better measurement loops. If your evals are stale, your product roadmap starts optimizing against yesterday's problems.

How To Run It

Terminal-Bench 4.0 is available through Harbor Hub. The official announcement gives the direct dataset run command:

harbor run -d terminal-bench/terminal-bench@4.0.0

Harbor Hub also shows a fuller example for running Terminal-Bench with a selected agent, model, concurrency, and environment:

uv tool install "harbor[modal,daytona]"

harbor run -d terminal-bench/terminal-bench \
  --agent claude-code \
  --model anthropic/claude-fable-5 \
  --n-concurrent 100 \
  --env modal

The key thing: this is not a tiny local toy benchmark. Harbor Hub notes that Terminal-Bench includes GPU and multi-container tasks, and recommends Modal or Daytona for running it. So if I were experimenting, I would start by reading the task docs and running a small subset before attempting a full leaderboard-style sweep.

My Takeaway

Terminal-Bench 4.0 made me think about benchmarks less like school exams and more like operating systems. They need maintenance. They need security against shortcuts. They need dependency management. They need versioning. They need logs and artifacts.

For long-horizon agents, this is the direction that feels real. The future is not just "bigger model gets bigger score." It is models inside environments, evaluated by living benchmarks that keep changing as the frontier moves.

My simplest summary: Terminal-Bench 4.0 is a benchmark repair release, but benchmark repair might be one of the most important parts of agent progress.

Bibliography