When an AI agent repeatedly fails at a specific task, the default reaction is often to reach for a better model or fine-tuning.

That can be the wrong first move.

The model is only one part of an agent. The system around it determines what context it receives, which tools it can use, how it interacts with the environment, what it remembers, how it inspects results, and whether one failure becomes a durable improvement.

That surrounding system is the agent harness.

I built and open-sourced Harneloop, an agent-first framework for building and evolving task-specific harnesses through real attempts, artifact-aware evaluation, trace-backed diagnosis, and evidence-gated promotion.

The project is currently a v0.0.2 public alpha. The core lifecycle works, but commands and file formats may still change before a stable release.

Its central idea is simple:

Let the agent try the real task, inspect what it actually produced, trace likely mistakes through the run, test a candidate change to its harness, and adopt that change only when the evidence shows an improvement.

Why ordinary evals are not enough

An evaluation can tell you that an agent failed. It does not necessarily tell the agent how to improve the system that caused the failure.

This becomes obvious when the result is an artifact rather than a short textual answer.

A generated interface can compile while being visually wrong. A 3D scene can render while violating the requested spatial relationships. A document can be structurally valid while failing its intended communication goal. A browser agent can finish without leaving the application in the correct state.

In these cases, a successful command and a plausible transcript are not proof of a successful result. The artifact itself is part of the evidence.

Harneloop records the attempt, its actions, state, logs, and produced artifacts. The operating agent can inspect the result, connect a visible or measurable problem back to the exact run, and reason about where the harness failed it.

That diagnosis might point to an instruction. It might also point to a missing tool, weak retrieval, an unsuitable validator, incomplete environment access, a poor example, or an assumption that was never checked.

The framework does not stop at “the agent failed.” It preserves enough context for the agent to develop and test a concrete improvement.

The harness evolves through candidates, not uncontrolled self-editing

An agent should not freely rewrite its working harness after every observation.

A change that fixes one example can break five others. An evaluator can accidentally be weakened until bad results pass. A useful instruction can disappear because it looked irrelevant during one run. If every experiment immediately becomes production state, trial and error turns into drift.

Harneloop treats a proposed harness change as an isolated candidate. It is a development and versioning layer for agent harnesses, not another agent runtime or evaluation dashboard.

A candidate can modify more than a prompt. It can add or revise:

• agent-facing instructions and principles;
• tools, skills, or MCP integrations;
• retrieval material and examples;
• artifact observers and validators;
• regression cases;
• environment setup and automation;
• any other infrastructure needed by the task.

Related changes can be batched into one coherent candidate, and independent candidates can exist in parallel. Validation scales with risk: a small infrastructure repair may need a focused check, while a behavior-changing harness update may require representative attempts and regression evidence.

The promoted harness remains unchanged while the candidate is tested. Promotion creates a restorable version only after evidence supports the improvement. If the candidate does not help, it can be revised or rejected without damaging the working harness.

A harness unit is a portable development environment

Every task-specific workspace in Harneloop is called a harness unit.

A harness unit contains the current understanding of the target, an operational map of the real environment, reusable harness material, candidates, regression cases, evidence references, and promoted versions.

It is deliberately not a rigid template. The agent can create research notes, scripts, databases, tools, retrieval indexes, custom observers, or new directories when the task requires them. Harneloop protects the integrity-sensitive lifecycle while leaving the agent room to reason.

Harness units also separate portable knowledge from runtime noise. The reusable instructions, contracts, tools, seeds, and promoted state can travel with the unit. Raw traces, caches, secrets, temporary artifacts, and abandoned experiments can remain outside the package.

That makes a successful harness easier to continue, audit, restore, move to another machine, or export into a coding agent or application.

The framework does not pretend to understand every environment

Harneloop cannot magically know which MCP server controls an application, where a render is saved, how a browser flow resets, or what database state proves that a task succeeded.

The operating agent maps those details during onboarding.

The user describes what the harness should help an agent become better at. The agent inspects the actual project and environment, identifies useful evidence, records its assumptions, and asks only for important missing context or permission.

The goal is not to rebuild every capability from scratch. The agent can reuse project-native tools, maintained open-source libraries, skills, datasets, validators, documentation, and prior work. Harneloop asks it to keep those choices inspectable and to justify additions through observed bottlenecks or a clear expected improvement.

This distinction matters because the agent building the harness may itself be under-tooled. A restricted agent might lack a browser, terminal, filesystem access, visual inspection, database access, or a package manager. Harneloop asks it to identify those capability gaps, explain their impact, and use an appropriate permission boundary before expanding access or introducing cost and credentials.

Why evolve the harness before changing model weights?

Several lines of agent research support a harness-first strategy.

SWE-agent showed that a purpose-built agent-computer interface could substantially improve how language models navigated repositories, edited files, and executed tests. Reflexion improved agents through trial-and-error and linguistic feedback stored in episodic memory rather than weight updates. Self-Refine demonstrated broad gains from iterative feedback and refinement without additional training.

More directly, the 2026 Self-Harness paper described agents mining weaknesses from execution traces, proposing minimal harness changes, and accepting them only after regression testing. Across three model families, the paper reported improved held-out pass rates without relying on stronger external agents.

Research comparing knowledge injection approaches also gives reasons not to treat fine-tuning as a universal answer. Fine-Tuning or Retrieval? found retrieval-augmented generation consistently outperformed unsupervised fine-tuning on the knowledge-injection tasks it evaluated. Separate work on fine-tuning new factual knowledge found that models struggled to acquire that knowledge and became more prone to hallucination as it was learned.

These papers do not prove that harnesses always beat fine-tuning. Fine-tuning remains useful for behavior, style, latency, specialized representations, or capabilities that cannot be supplied effectively at inference time. A strong harness and a suitable fine-tune can complement each other.

The practical conclusion is narrower and more useful:

If the bottleneck is context, tools, retrieval, feedback, validation, memory, or environment interaction, test the harness before changing the weights.

Harness changes are easier to inspect, reverse, reuse, and validate. They can also target the exact system boundary causing the failure.

Initial evidence from ViperMesh

Harneloop grew out of a practical problem I encountered while developing ViperMesh, a custom harness for Blender spatial reasoning and scene generation.

The agent could write code and operate Blender, but it repeatedly made spatial mistakes, selected weak tool paths, and failed to use the rendered scene as feedback. Improving isolated prompts was not enough. The complete environment needed an evidence-backed development loop.

The first completed case study compared the ViperMesh harness with the Anthropic x Blender MCP server baseline while keeping the acting model fixed at OpenAI GPT 5.5 High.

Across seven comparable live tasks:

• ViperMesh was faster on 6 of 7 tasks;
• the measured mean speedup was 2.534x;
• preliminary neutral LLM visual evaluation improved by 8.19 points.

One comparable Codex-operated benchmark window recorded 175,158 local acting-agent tokens for ViperMesh versus 1,927,483 for the baseline, a 90.91% reduction. That result is intentionally reported as one scoped token pair rather than a general conclusion across every task.

Harneloop did not generate the Blender scenes. It structured the process used to inspect failures, improve the harness and tool surface, preserve evidence, and verify whether changes helped.

The full ViperMesh case study documents the methodology, results, and limitations.

This is one task family and one environment. It is not a universal benchmark against every fine-tuning method. The next important evidence must come from independent users and different kinds of work.

Getting started

Harneloop is designed primarily for agent-led operation.

Give a capable coding agent the repository link and describe the task you want to improve. The agent can install the framework, inspect the documentation, create a harness unit, map the environment, and ask for missing information when necessary.

A manual CLI and interactive setup are available for users who want direct control.

The current alpha requires Python 3.11 or newer and installs as a user-level tool through uv:

uv tool install git+https://github.com/Ker102/Harneloop.git
harneloop doctor
harneloop setup

Good first use cases have three properties:

1. An agent performs a recurring task with meaningful failures.
2. The outcome can be inspected through artifacts, state, tests, or other evidence.
3. The harness has room to improve through context, tools, feedback, validation, retrieval, or environment integration.

I am looking for early users willing to test Harneloop outside my own environment, including people who find that it does not work for their task. Onboarding failures, expensive loops, weak evidence, and rejected candidates are useful results.

Explore the Harneloop project site, inspect the source on GitHub, read the ViperMesh case study, or bring a difficult agent task and create the next harness unit.