Idea #1
Future knowledge leakage is the central validity threat when backtesting any AI agent that uses web search, and it is easy to miss because it manifests as suspiciously good performance rather than an obvious error. The problem: a live search engine returns the web as it exists today, which may already contain explicit confirmation of the outcome you are trying to predict. If your agent is evaluating whether a company was acquired, a search run today might return news articles saying 'Company X was acquired on date Y' — making the agent's 'prediction' trivially correct and your performance metric meaningless. The fix is versioned/snapshotted search: constrain the agent to only see web content that existed before the outcome was announced. This matters not just for financial systems but for any agent evaluation task where ground truth has since been published on the web — coding agents solving GitHub PRs, news-based classification systems, anything with a before/after event boundary. The concrete demonstration: a company-acquisition prediction task ran with live search would trivially 'predict' acquisitions because today's web already records them. With Exa Snapshot, the system can be pinned to 1 day, 7 days, or 30 days before each acquisition date, so only genuinely predictive pre-event signals count.
Idea #2
The holdout set is non-negotiable in backtesting, and skipping it in an evaluation context where developers can see the evaluation set guarantees overfitting — sometimes absurdly so. This is not a theoretical risk; it played out in real time in the workshop: the top-scoring participant's winning 'feature' was literally 'Is this company a company other than [list of the specific company names in the evaluation set]?' — a perfect fit to the visible evaluation set with zero generalization value. The second-place feature included 'or is it named Dominion?' A holdout set — companies the model was never allowed to query during development — would have immediately exposed both of these as useless. The lesson generalizes: any time an evaluation set is visible during the feature-development loop, the developer (or model) will find patterns that are specific to that set. The only structural fix is to keep a portion of ground-truth examples completely off-limits until final scoring. The presenter acknowledged this in real time, calling it 'a feature not a bug' of the demo, but the concrete example makes the principle far more memorable than any abstract warning would.
Idea #3
Web article publish dates are systematically unreliable as historical timestamps and should not be trusted for feature engineering in time-sensitive systems. Sites update publish dates routinely for two reasons: genuine content revisions and SEO, since search engines favor recently updated pages. The implication is that an article nominally published on date T may actually reflect content written weeks or months later, or may have been retroactively updated with information that did not exist at the time of the original publication. For any predictive system that relies on knowing what information was available at time T — financial signals, news-based classifiers, acquisition-prediction features — a publish date is a noisy and gameable proxy. Versioned web indexing sidesteps this entirely by storing page content at the moment of crawl, independent of what the page metadata claims.
Idea #4
Agentic web search can be introduced as individual structured features feeding into a traditional ML model, rather than requiring a full rebuild into an end-to-end agentic system. The pattern: each 'feature' is a natural-language yes/no question answered by an LLM agent that searches the web — 'Has this company recently lost its CEO or appointed interim leadership?', 'Has it been reported to be exploring a sale?' The agent's answer (yes/no/unknown) becomes a column in your feature matrix alongside your existing numeric signals. Because each feature is independent, it can be backtested in isolation against historical versioned search results, letting you measure its marginal predictive contribution before committing it to production. This is low-risk compared to replacing the whole system: you add one agentic feature at a time, validate it against held-out ground truth, and only keep features that demonstrably move the needle. The workshop demonstrated this incremental logic directly — each feature query was run against 10 companies with known outcomes, and the scoring (plus one for true positive, minus one for false positive) let participants immediately see whether combining two features with AND vs. OR logic improved or degraded the result.
Source: Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa (AI Engineer)
Text summarized and optimized using Anthropic’s models and reviewed by a human.