Idea #1
Production agents should run two distinct continuous improvement loops, not one: a real-time memory/escalation loop and a retroactive harness-update loop driven by trace data. These loops are architecturally different and serve different purposes. Loop 1 (memory): the agent recognizes its own uncertainty in a live conversation, surfaces it explicitly rather than hallucinating, and escalates to a human inbox. The human corrects it, and the correction is stored as a lesson in a 'context hub' — both per-user (private, no cross-user leakage) and shared (for patterns that recur across users). The agent retrieves relevant lessons on future similar queries. Loop 2 (harness update): after the fact, the trace data from production runs — especially failures — is analyzed to produce proposed edits to the core instructions and skills (the harness itself), not just episodic memories. These proposals are reviewed and applied by a human. In the demo, a help-desk agent started at 4/14 correct on a test set of real community questions; after running both loops through a realistic set of production interactions, it reached 13/14. The two loops are complementary: loop 1 builds a searchable knowledge base of edge cases; loop 2 generalizes those lessons into the underlying harness so the agent doesn't need to look them up next time.
Idea #2
The correct model for safe autonomous harness improvement is: let the agent surface and propose fixes automatically, but require human approval before any change is applied — treating every harness edit like a pull request. This is the only way to keep the agent's improvement tractable and auditable. Without this constraint, autonomous self-improvement means you lose visibility into what the agent is becoming and have no rollback path when a change makes things worse. The practical implementation shown uses versioned commits in the harness platform (each applied fix is a commit; you can promote versions or roll back), and the human is in the approval loop for every single change — per-user memory updates, shared lessons, and instruction/skill edits alike. The analogy drawn is exact: just as a coding agent proposes a PR rather than force-pushing to main, a production agent proposes a harness change rather than self-patching. This framing resolves the tension between 'agent learns from production' and 'agent goes rogue': the agent does all the analytical work (reading traces, diagnosing root causes, drafting fixes), but the commit only lands when a human clicks approve.
Idea #3
The markdown-based harness pattern from coding agents (Claude Code, Codex, etc.) maps 1:1 onto production/automation agents, and this means every harness engineering technique from the coding-agent world carries over directly. The key insight is that the shift from 'plumbing in code' to 'everything-in-markdown' happened for production agents at the same time it happened for coding agents — so the two are now structurally identical: rules = system prompt, sub-agents = multi-agent delegation, MCP tools = external connections, skills/markdown = workflow definitions replacing hand-coded tool functions, hooks = guardrails for monitoring or blocking at tool-call or conversation boundaries. This matters because it means a single body of knowledge about making coding agents more reliable (how to write rules, how to scope skills, how to structure memories) is directly transferable. The proof offered is an agent.py file of just 30 lines of code that fully defines an agent with guardrails, sub-agents, tools, skills, and instructions — all the substance living in markdown documents that get stitched together by the platform at deploy time, exactly as a CLAUDE.md or agents.md gets picked up automatically by a coding agent.
Source: Let Your AI Agent Do ITS OWN Harness Engineering (Here's How) (Cole Medin)
Text summarized and optimized using Anthropic’s models and reviewed by a human.