Self-Harness: AI Agents That Rewrite Their Own Rules

By Lazar Milicevic · Published June 28, 2026 · 10 min read

You shipped an agent that worked great in the demo. Then real users hit it and the failure rate climbed to 30%. You've been debugging the prompt for two weeks — tweaking instructions, adding examples, swapping models — and every fix breaks something else. This is the part of agent engineering nobody talks about: the harness is the actual product, and we're all tuning it like it's 2023.

A new line of research is changing that. The idea is straightforward: instead of you hand-editing the harness, the agent rewrites its own rules based on what's failing in production. Early reports from research groups working on self-tuning harnesses claim performance lifts up to 60% on agent benchmarks. That's the kind of delta you can't squeeze out of a better model. Let's break down what a "harness" actually is, why manual tuning has hit a wall, and how self-harnessing works in practice for teams that can't afford a research lab.

What an agent harness actually is

An agent harness is everything that wraps the model: the system prompt, the tool definitions, the routing logic, the memory layer, the retry rules, the output validators, and the orchestration glue between them. The model is a frozen weight file. The harness is the program. When people say "Claude Code is better than X," they're usually praising the harness, not the underlying model.

This distinction matters because it tells you where leverage lives. You can't retrain Claude or GPT — that's a frontier lab problem. But you absolutely can rewrite the harness around them, and for most SMB workflows the harness is where 80% of quality comes from.

A minimal harness for, say, a customer-email triage agent looks like this:

agent:
  model: claude-sonnet-4
  system_prompt: prompts/triage_v14.md
  tools:
    - search_knowledge_base
    - draft_reply
    - escalate_to_human
    - tag_ticket
  routing:
    on_low_confidence: escalate_to_human
    on_billing_keywords: route_to_billing_agent
  validators:
    - reply_under_200_words
    - no_pricing_commitments
  retry:
    max_attempts: 2
    backoff_seconds: 5

Every line here is a tuning knob. The number of knobs grows combinatorially as you add tools, sub-agents, and edge-case rules. Past about 20 knobs, no human can hold the full search space in their head.

Why manual harness tuning hits a wall

Most teams tune harnesses the same way: a customer complaint comes in, you stare at the trace, you edit the prompt, you ship, you wait. This works for the first dozen iterations and then degrades fast. Three reasons:

1. No systematic feedback loop. You see the loud failures. You don't see the silent ones — the ticket that got resolved but with a 30% worse reply than the agent could have produced.
2. Regression blindness. Fixing failure mode A often re-introduces failure mode B you patched three weeks ago. Without an eval suite tied to every edit, the harness drifts.
3. Search space explosion. A modern agent has prompt sections, tool descriptions, routing thresholds, and validator rules. Sweeping by hand means trying maybe 5 variants a week. The space has thousands of meaningful configurations.

Here's the brutal truth: an agent harness in production after six months of human tuning is usually worse than a fresh one tuned for two days by an automated process against a real eval set. Drift is the default state.

How Self-Harness works

Self-harness frameworks (this is now a small but growing area in agent research) flip the loop. Instead of you editing the harness, the agent proposes edits to its own harness based on structured feedback from execution traces.

The core loop has four stages:

1. RUN     -> Execute current harness against task set
2. JUDGE   -> Score outputs against rubric; collect failure traces  
3. PROPOSE -> Agent reads failures + current harness, drafts diff
4. ACCEPT  -> Apply diff IF it beats current on held-out eval

The "accept" step is what separates this from a chaos generator. The new harness has to outperform the old one on a held-out evaluation set before it ships. If it doesn't, the diff is discarded and the loop tries a different proposal.

A stripped-down implementation in Python:

def self_harness_loop(harness, eval_set, holdout_set, max_iters=20):
    best_score = evaluate(harness, holdout_set)
    
    for i in range(max_iters):
        # 1. Run + collect failures
        traces = run_agent(harness, eval_set)
        failures = [t for t in traces if t.score < 0.7]
        
        if not failures:
            break
        
        # 2. Agent proposes harness diff
        proposal = propose_diff(
            current_harness=harness,
            failure_traces=failures[:10],
            rubric=RUBRIC
        )
        
        # 3. Apply + test on holdout
        candidate = apply_diff(harness, proposal)
        new_score = evaluate(candidate, holdout_set)
        
        # 4. Accept only if it wins
        if new_score > best_score:
            harness = candidate
            best_score = new_score
            log_accepted(i, proposal, new_score)
        else:
            log_rejected(i, proposal, new_score)
    
    return harness, best_score

Reported lifts in the 40-60% range come from this loop running for hundreds of iterations against rich eval sets. The reason it works: the proposing model is reading actual failure modes, not guessing what might go wrong. It's the difference between debugging with a stack trace versus debugging from a user's vague description.

What you actually need to run this

Most "self-improving agent" demos skip the boring infrastructure. Here's the real list for a production setup:

The eval set is the bottleneck. If you don't have 50 real examples with clear pass/fail criteria, you have nothing to optimize against. Most teams skip this step and then wonder why their "self-improving" agent is just generating slop.

A concrete starting rubric for a sales-followup agent:

RUBRIC = {
    "addresses_lead_question": (0, 1, "Did reply answer what they asked?"),
    "tone_matches_brand": (0, 1, "Friendly-direct, no formality theater"),
    "no_invented_facts": (0, 2, "Hallucinated pricing/features = -2"),
    "next_step_clear": (0, 1, "Specific CTA, not 'let me know'"),
    "length_appropriate": (0, 1, "Under 150 words unless complex"),
}
# Max score: 6. Failure threshold: < 4.

Scores are coarse on purpose. Fine-grained scoring (1-100) is noise; the model can't reliably distinguish a 73 from a 76. Stick to small integer ranges.

A worked example: tuning an email triage harness

Let's walk through what one iteration looks like for a small B2B SaaS doing customer support triage.

Starting harness (v1): Generic system prompt, four tools, no routing rules, no validators. Baseline eval score: 58/100.

Iteration 3, failure traces flagged:

• 7 cases: agent attempted to answer billing questions instead of routing to billing team
• 4 cases: agent quoted pricing it invented from context
• 3 cases: replies over 400 words on simple questions

Proposed diff:

  system_prompt: |
    You are the first-line support agent for Acme SaaS.
+   You MUST route any message containing pricing, invoice, refund, 
+   or subscription topics to the billing_agent tool. Do not answer
+   billing questions yourself, even if you think you know the answer.
+   
+   Keep replies under 150 words unless the user asked a multi-part
+   technical question. Bullet points over paragraphs.
    
  validators:
+   - no_dollar_amounts_unless_from_kb
+   - reply_under_200_words

Holdout score after applying: 71/100. Accepted.

Iteration 7 failure: Agent now over-routes to billing, sending account-access questions to the billing queue because they mentioned the word "subscription."

Proposed diff: Tighten the routing rule with negative examples. Holdout score: 74. Accepted.

After 20 iterations the harness is 80+ lines longer than v1, scores in the high 80s on holdout, and — critically — every change is a committed diff with a measured lift. You can read the git history and understand why each rule exists. That's something a hand-tuned harness never gives you.

Where self-harness breaks (and how to avoid it)

This is not magic. Three failure modes you'll hit:

1. Judge model collapse. If you use the same model to propose edits and judge outputs, it will learn to game its own rubric. Use a different model family for judging, or use deterministic rule-based checks where possible. A cheap fix: have GPT propose, have Claude judge, or vice versa.

2. Overfitting to the eval set. With small eval sets (< 100 examples), the harness will memorize the test instead of generalizing. Symptoms: eval score climbs to 95+, real-world performance flat or worse. Mitigation: rotate eval examples weekly with fresh production traces, and keep a frozen holdout you never touch.

3. Drift toward verbosity. Proposing models love to add rules. After 50 iterations your system prompt is 4,000 tokens of overlapping instructions that contradict each other. Build a "prune" step into the loop that periodically tries to remove rules and checks if the score holds. If a rule can be deleted without score loss, delete it.

A simple pruning pass:

def prune_harness(harness, holdout_set):
    rules = extract_rules(harness)
    baseline = evaluate(harness, holdout_set)
    
    for rule in rules:
        candidate = remove_rule(harness, rule)
        score = evaluate(candidate, holdout_set)
        
        if score >= baseline - 1:  # tolerance
            harness = candidate
            baseline = score
            log(f"Pruned: {rule.id}")
    
    return harness

Most production self-harness loops alternate: 10 grow iterations, 1 prune pass, repeat. This keeps the harness lean.

When this is worth doing (and when it isn't)

Self-harness is not the right move for every project. Concrete decision rules:

Worth it when:

• You have ≥ 50 real task examples with clear success criteria
• The agent runs ≥ 1,000 times per month (enough volume to justify infra)
• Failure cost is meaningful (lost revenue, bad CX, compliance risk)
• You're past the "is this even possible" stage and into "make it reliable"

Skip it when:

• You're prototyping — manual tuning is faster for the first two weeks
• You can't define a rubric a junior engineer could score against
• The agent runs 10 times a month — just review traces by hand
• Your "agent" is really a single LLM call with a prompt; there's no harness to tune

The honest rule: if you can't write the eval set, you're not ready for self-harness. The eval set is the spec, and most teams discover their spec is undefined the moment they sit down to write it. That discovery alone is worth the exercise.

How BizFlowAI approaches this

We run harness-iteration loops for SMB clients on Claude-based agents — mostly support triage, lead followup, and internal ops workflows. The pattern is consistent: clients come to us with a hand-tuned agent that works "most of the time," we spend the first week building an eval set from their real traffic, and the second week running an iteration loop against it. The 40-60% lift the research reports is roughly what we see on workflows where the baseline harness was tuned by intuition.

What we don't do: hand you a self-improving agent and walk away. The eval set needs upkeep as your business changes, the rubric needs human review, and the prune pass needs a human in the loop. If you want to see what a measured harness lift looks like on a workflow you already run, book a discovery call and bring one real trace file — we'll show you the failure modes a structured loop would catch in the first iteration.

The bigger shift

Frontier model improvements have slowed from "every six months" to "incremental." That changes where leverage lives. For the next two years, the teams that win at AI in production won't be the ones with access to a better base model — everyone has roughly the same access. They'll be the ones whose harnesses iterate faster than their competitors'.

Self-harness is one tool for that. Even if you don't adopt the full automated loop, the discipline it forces — write an eval set, score against a rubric, version every change, measure every lift — is the difference between an agent that ships and an agent that drifts. Start with the eval set. The rest follows.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.