The idea of agents “automating our jobs” has gone around for a while, but most examples stay abstract. Niels Rogge at Hugging Face followed a concrete, reproducible, and honest approach to trace the real line between a workflow and a true agent sits.
The problem that started it
Hundreds of research papers land on arXiv every day. Many release code or model weights, but host those artifacts on Google Drive, Dropbox, or GitHub Releases and not on a central platform with proper metadata and documentation. The manual alternative — opening templated GitHub issues to ask researchers to move their artifacts — is real work, and nowhere near scalable at the volume arXiv produces.
If it is not an agent don’t call it so
The first version of the solution is a deterministic workflow, not an agent. It follows Anthropic’s “Building Effective Agents” post — which recommends starting simple and avoiding frameworks. The pipeline runs as a nightly GitHub Actions cron job, and requires no special infrastructure.
The steps are fixed: find the paper’s GitHub repository, read the README, check whether artifacts already exist on the Hub, open a pull request if the metadata is incomplete, or open a GitHub issue if nothing has been uploaded at all. No branching decisions, no LLM making judgment calls about what to do next. Just a sequence, with LangFuse (an observability platform that logs prompts, outputs, latency, and cost) wired in from the start.
The instinct when building with LLMs is to reach for an autonomous agent because agents feel more powerful. But for needs that follow a predictable script, a workflow is simpler to debug, cheaper to run, and easier to monitor.
The result is hundreds of GitHub issues opened per night, with only two negative replies out of thousands. Researchers responded. PaddleOCR migrated an entire family of models. An issue about small recursive models gathered more than sixty upvotes. Apple and Google DeepMind researchers uploaded artifacts directly.
When flexibility genuinely requires an agent
Follow-up is where the dynamic changes. An initial issue can follow a script; responding to a researcher’s reply requires reading context, understanding their specific situation, and deciding what to do next. That flexibility is where a fixed workflow starts to break down.
The second phase introduces a fully autonomous agent — an LLM running in a loop, where the model repeatedly calls tools and reasons about their outputs until it decides the task is complete. Each GitHub issue gets its own container on a cloud platform for batch processing (Modal), and agents run in parallel across hundreds of issues simultaneously.
The agent’s primary tool is a single command-line interface skill. No elaborate toolset, no complex reasoning. A simpler tool presents fewer ways for an agent to go wrong, and allows an easier evaluation of what it actually produced.
The model powering the agent was initially closed-source; later replaced by Qwen 2.5, an open-weight model through Hugging Face’s own inference providers. Open models, at this point, are viable for production agent work that doesn’t require frontier-level reasoning. The cost difference is real, and the quality gap has narrowed.
What the pattern teaches
The two-phase structure — deterministic workflow first, autonomous agent only where flexibility is genuinely needed — is probably the right default for automating repetitive knowledge work. It keeps the simple cases simple, and reserves the harder machinery for the problems that actually need it.
The other lesson is about evaluation. Agent output quality doesn’t maintain itself. Without active monitoring of what agents are actually producing, quality degrades quietly. Tracing, observability, and deliberate review of outputs are not optional extras added once a system matures; they are part of the design from day one.
As open models continue to close the gap with closed-source alternatives, the economics of this kind of automation keep shifting. The ceiling on what we can build and run at reasonable cost rises. The constraint is no longer model capability — it is the discipline to start simple, instrument well, and earn the agent rather than reaching for it first.
Text summarized and optimized using Anthropic’s models and reviewed by a human.