For a while, the standard advice for keeping up with AI was simply: follow the right people on social media and skim the rest. That worked when the field moved slowly enough for a human to filter it. It does not work anymore.
The volume of AI news, research, and tooling updates that lands every day has grown past what selective attention can handle. Most practitioners have quietly accepted a low-grade anxiety about it — the feeling that something important slipped by, or that a significant chunk of reading time went to noise. The real problem is not that there is too much information. It is that the filtering has not kept up with the production.
The filtering gap
What tends to happen when practitioners build informal systems for staying current is that they accumulate subscriptions: newsletters, RSS feeds, Slack channels, GitHub stars. The input grows; the filtering does not. So the digest gets longer, skimming gets faster, and the signal-to-noise ratio stays roughly the same.
A more principled approach separates the ingestion layer from the filtering layer — and then applies filtering in stages, cheapest steps first.
How a staged pipeline changes the math
A pipeline designed for this problem ingests from a wide set of sources — Reddit, YouTube, RSS feeds, GitHub repositories, Hacker News, arXiv preprint categories, and platform changelogs — running daily. The key is what happens before anything reaches a language model.
The first two filtering stages are deterministic and cost nothing: deduplication (a central database that tracks already-seen items) and code-based rules (dropping items below an engagement threshold or past a freshness cutoff). These stages eliminate most of the volume before any model is involved.
The third stage uses a reranking tool — Jina’s reranker, often called Jev — as a lightweight semantic scorer. It evaluates each remaining item on a small set of criteria: is it on topic, is it something the practitioner would actually care about, and is it a concrete development rather than another toy demo? Items that score well pass through; items that clearly fail are dropped; only genuinely ambiguous items advance to an LLM.
That final LLM step — a small, fast model like Claude Haiku — handles the hard cases and formats the digest. The result, for a daily run across roughly 850 posts, runs under fifty cents. Most of that cost comes from the LLM pass; the reranker costs almost nothing.
The design principle worth keeping is: use the cheapest capable tool at each stage, and never send something to a model that a deterministic rule could have handled.
The profile file as the real interface
One underappreciated piece of this kind of system is the configuration layer. A single JSON file manages which sources are included — RSS feeds, YouTube channels, GitHub repos, arXiv categories — and a separate profile file written in plain language captures the filtering logic: what topics matter, what kinds of content to skip, what has been missed in the past.
That profile file is the part a practitioner will actually maintain over time. Editing it in natural language and pointing an agent at the diff to update the engine is a more durable interface than writing code directly. The criteria for what matters to a given practitioner change faster than the pipeline itself.
Where this is heading
The broader shift here is from passive consumption to active curation infrastructure. A practitioner who has built even a rudimentary version of this kind of pipeline is no longer dependent on platform algorithms or on what happens to surface in their social feeds. They have defined, in explicit terms, what they care about — and they get that, and mostly only that.
The interesting next step is personalisation over time: a profile that updates based on what the practitioner actually reads versus skips, rather than one that requires manual editing to stay accurate. That is a harder problem, but the staged filtering architecture described here is the foundation it would build on.
Text summarized and optimized using Anthropic’s models and reviewed by a human.