Use a four-stage outcome model
Count searched when the provider performed retrieval, sourced when a consulted URL belongs to the product, mentioned when the answer names it, and recommended only when the answer affirmatively connects it to the requested job. Keep invocation in a separate dataset.
This funnel prevents a large source count from looking like visibility. It also makes regressions legible: a product might remain indexed while losing its recommendation, or gain a mention without evidence support.
Retain enough evidence to reproduce a run
Store the exact prompt, intent class, provider, model, time, response text, searches, opened pages, consulted URLs, citations, and classification reason. Normalize domains for grouping while preserving full URLs.
Web-search APIs can expose queries and source URLs, but provider implementations differ. Report provider coverage explicitly rather than combining partial evidence into a false cross-provider total.
- Use branded checks as entity controls, not discovery wins.
- Use unbranded category and problem prompts for spontaneous retrieval.
- Keep previous days available for comparison and regression review.
Read a positive result conservatively
One explicit recommendation is valuable evidence of a working path, not proof of a general effect. Label the supporting source class and whether another provider confirmed it. Preserve the path so the team can strengthen adjacent intents.
Content changes and later model behavior are correlated observations unless the experiment isolates causality. The honest output is a dated receipt and replication plan, not a guaranteed lift.