Idea #1
Production observability data and agent reasoning operate at different levels of abstraction, and bridging that gap ('prod-to-code') is a prerequisite for accurate agentic diagnosis. Typical production monitoring speaks in endpoints, P90 latencies, CPU, memory, and service-level metrics. Coding agents reason at the function and file level. When you ask an agent 'why is this endpoint slow?' using only service-level metrics, it has to guess the connection between the metric and the code — which introduces assumptions and degrades accuracy. The solution is to instrument at function level and connect each function invocation back to the endpoint, cron job, or event consumer that triggered it, so you can ask 'this endpoint sometimes takes 7 seconds — where exactly is the time spent, which function, which outbound call?' and get a grounded answer. Thundra calls this the 'HUD' (Heads-Up Display): a dataset structured at function granularity, linked to endpoints and to forensic captures triggered only when requests exceed a latency threshold (e.g., P99), so you have concrete slow-request evidence rather than averages. On top of this they added a query language (ClickHouse-based) and a library of 'skills' — pre-built query patterns for common diagnostic questions like 'where did this 500 originate?' or 'what was running during this memory spike compared to baseline?' — because raw querying against columnar data produced high variance in agent results, while named skills produced consistent ones.
[from: From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization – May Walter, Hud — YouTube · AI Engineer · https://youtu.be/JJGbw4ggaFs]
Idea #2
The most dangerous failure mode in agentic workflows is the 'plausible unverified' fix: the agent suggests a change that sounds correct and is even theoretically valid, but when actually applied it doesn't fix the problem. This happens because agents reason over code and static patterns, not over what is actually causing slowness in production right now. The antidote is to close the loop within the workflow itself — don't stop at 'here is a suggested fix'; instead, have the agent apply the fix, rerun the relevant tests, measure the actual impact on the specific flow, and only surface the result to a human once there is runtime evidence that the change worked. This shifts the human's role from reviewing plausible ideas to approving verified outcomes. In this talk, the Thundra team built exactly this: the agent detects a slow endpoint from real production traces, identifies the root cause at function level, proposes a fix, runs it, observes the timing improvement, and only then opens a human-readable summary — so the reviewer is not being asked 'does this look right?' but rather 'do you want to merge something we already know works?'
[from: From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization – May Walter, Hud — YouTube · AI Engineer · https://youtu.be/JJGbw4ggaFs]
Idea #3
The real leverage point for agentic automation is not making existing work faster — it is automating work phases that simply never happen in practice. In performance engineering, the recurring bottleneck is not fixing issues (which is often quick once the cause is known), but the investigation phase: a black box that can take an hour or weeks, with no predictable output, making it nearly impossible to schedule or prioritize. Because that research cost is unpredictable, teams default to ignoring low-grade performance debt until it becomes a crisis, then fix it in emergency mode, then return to ignoring it — a permanently leaky bucket. Scheduling an autonomous agent to run that investigation weekly on real production data fundamentally changes the economics: the cost of investigation drops to near zero (it runs without human time), and the output is a scored, evidence-backed shortlist rather than a blank slate. The speaker's framing of this is precise: 'we're not just doing things faster; we're automating a phase that just did not happen in the day-to-day lives of engineers.' The Thundra workflow runs on a GitHub Actions cron job weekly, uses Claude Code with MCP-connected runtime intelligence, and sends findings to Slack — the key design choice being that it runs without anyone in the loop and delivers to where engineers already live.
[from: From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization – May Walter, Hud — YouTube · AI Engineer · https://youtu.be/JJGbw4ggaFs]
Idea #4
There is a qualitative trust threshold between using an AI assistant interactively and running an autonomous agentic pipeline, and that threshold is much higher than most people expect. When a developer uses an agent in their IDE and it is right 80% of the time, that is fine — the human is present, can immediately spot the 20% failures, and course-corrects in real time. But an autonomous workflow that runs weekly without a human in the loop and surfaces results to a team cannot operate at 80% reliability: below roughly 90%, the noise and wasted review time will erode trust until the workflow is ignored or killed. Crossing that threshold requires investing in the methodology, not just the model — building a playbook that encodes how a senior engineer would approach the problem (what to look for, what counts as a lazy fix vs. a real root cause, what makes a fix 'worth' the reviewer's time), scoring and prioritizing so only high-ROI items surface, and ensuring the fix is runtime-verified before handoff. The speaker explicitly contrasts 'agentic engineering' (autonomous pipelines) with 'coding with an agent' (interactive copilot use) and states that getting to the autonomy level is 'dramatically different' and requires crossing toward 80–90% confidence as a deliberate engineering goal. A related practical implication: rolling out one verified PR at a time rather than flooding reviewers with 80 automated PRs was the key to building team appetite and habit around the workflow.
[from: From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization – May Walter, Hud — YouTube · AI Engineer · https://youtu.be/JJGbw4ggaFs]
Source: From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization – May Walter, Hud (AI Engineer)
Text summarized and optimized using Anthropic’s models and reviewed by a human.