The Harness Is the Verifier

The Harness Is the Verifier

6 min read

The interesting part of GitHub Security Lab’s new fuzzing agent is not that an LLM can write a harness. Models have been generating plausible code for a while.

The interesting part is that the harness is placed inside a measurable loop. The agent has to run AFL++, read coverage, improve the target, triage crashes, and decide when additional work is no longer worth the compute.

That is a much more serious design than asking a model to inspect a repository and list possible bugs.

The agent is not the security system

GitHub Security Lab’s Fuzzing Taskflow accepts a GitHub owner/repo slug. Its documented pipeline identifies entrypoints, analyzes the build, creates fuzz harnesses, runs AFL++, reads coverage, and produces a vulnerability report for each unique crash or bug.

The project is built on the GitHub Security Lab Taskflow Agent, a framework for LLM-driven security automation. GitHub’s engineering writeup describes the division of labor clearly:

  • The agent chooses what to fuzz and which gap to pursue.
  • MCP tools perform operations such as compiling harnesses, running AFL, and reading coverage.
  • SQLite carries state between stages instead of passing everything through the model’s context.

That division matters. The model is making decisions, but it is not the authority that defines whether a crash is real, whether coverage improved, or whether the campaign should stop. Those facts come from tools and persisted state.

The common intuition is that an autonomous security agent needs a smarter prompt. This architecture points in the opposite direction. It needs a control loop that makes unsupported confidence expensive.

Coverage turns prose into a test

The core loop is straightforward:

run AFL++
    ↓
replay the queue against a coverage binary
    ↓
read uncovered branches
    ↓
add a seed, edit the harness, enrich the dictionary, or skip the gap
    ↓
run again

Each harness is built twice. The AFL binary uses edge instrumentation for fuzzing. The coverage binary uses Clang’s profile instrumentation so the queue can be replayed into source line and branch coverage reports.

That is a small implementation detail with a large systems consequence. The agent is not asked whether its last change felt useful. It gets an external measurement from a different execution path.

GitHub describes a staged time budget of 30, 60, 120, 240, 480, and 960 seconds, about 32 minutes per target at the maximum. The loop also uses plateau detection. By default, two consecutive iterations gaining less than one percentage point of absolute line coverage are treated as diminishing returns.

This is what a verifier looks like in practice. It does not make the model correct. It makes the model’s next decision answer to evidence that the model did not invent.

The stop condition is part of the security design

An agent that can always ask for another round is not autonomous. It is an unattended bill.

The Fuzzing Taskflow’s plateau rule creates a bounded control loop. Early rounds are cheap and recover obvious coverage. Later rounds get more time only when the campaign is still producing evidence that the extra work might be useful.

The threshold is not a universal truth. A one percentage point gain may be the wrong stopping rule for a safety-critical parser, a mature codebase, or a target with unusually expensive paths. The value is that the stop condition is explicit, configurable, and visible.

That is a better operational starting point than an agent that runs until a token budget, wall clock, or human operator happens to intervene.

The dangerous part is not hidden

There is an uncomfortable catch in GitHub’s own instructions. The taskflow runs afl-fuzz, clang, and arbitrary build commands selected by the LLM directly on the host. There is no container between the agent and those commands.

GitHub recommends a disposable environment such as a Codespace or throwaway virtual machine, without elevated privileges.

This is the boundary that matters more than the model choice. A prompt injection in a repository can influence the agent’s decisions. If those decisions reach a shell with the user’s permissions, the fuzzing pipeline is also a code execution pipeline.

The safe deployment shape is therefore not:

repository → agent → host shell

It is closer to:

repository
    ↓
short-lived worker with no secrets and no elevated privileges
    ↓
agent decision layer
    ↓
scoped execution tools
    ↓
coverage, crash, and audit artifacts

The exact sandbox can vary. A Codespace, a disposable VM, or a hardened worker may be appropriate in different environments. The invariant is that the repository and the agent do not inherit the operator’s durable trust domain.

MCP is an execution boundary, not a sandbox

The taskflow’s MCP layer is a useful pattern because it separates decisions from execution. It is not, by itself, isolation.

An MCP tool that compiles a harness or starts AFL still has the authority granted to the process behind it. A clean function name does not reduce that authority. The tool server needs its own least-privilege policy, filesystem scope, network policy, resource limits, and audit trail.

This distinction is easy to lose when an agent framework makes tools look tidy. The interface can be declarative while the effect remains arbitrary native code execution.

For operators, the minimum questions are concrete:

  • Which directories can the worker read and write?
  • Can build commands reach the network?
  • Are repository secrets absent from the environment?
  • What happens when the agent asks for a command outside the expected tool contract?
  • Are crashes and generated harnesses retained as evidence?
  • Can the entire worker be destroyed after the run?

If the answer to the last question is no, the worker is probably carrying more state and privilege than a fuzzing campaign needs.

The model is replaceable. The loop is not.

The Fuzzing Taskflow currently defaults to Claude Sonnet 5 in its repository configuration, according to the project’s recent commit. That choice may change as models improve.

The durable design is elsewhere:

  1. Give the model narrow decisions with observable consequences.
  2. Put execution behind tools with explicit scope.
  3. Measure coverage and crash behavior outside the model’s prose.
  4. Persist state so the loop can be inspected and resumed.
  5. Stop on a declared condition instead of trusting enthusiasm.
  6. Run the whole campaign in a disposable, unprivileged boundary.

A stronger model can improve harness quality. It cannot turn an unisolated shell into a safe one. A better prompt can improve triage. It cannot establish that a crash reproduces or that a coverage gain is real.

The agent can fuzz. The harness can find paths. But the verifier is what turns a language model’s suggestion into a security workflow.

That is the line operators should care about: not whether the agent sounds autonomous, but whether every important step has an external check and a boundary that can be thrown away.

Sources

Keep reading

The Pin Was a Label, Not a Check: How Plugin4Shell Walked Past SHA Pinning in Four Coding Agents The Weights Didn't Change The Exploit Claim Is Also a Warning About Agent Verifiers