Interview Questions › Metrics

Measure AI answers at the top of Google Search

Google shows an AI-generated answer above search results for some queries. How would you measure whether this is good for users, publishers and Google?

Context

The AI answer can reduce clicks to websites. Ads still show below the answer. Answer quality varies by topic.

Clarifying questions to ask first

• Which query types show an answer?
• How do we define a successful search without a click?

How to approach it

1. Three stakeholders with conflicting interests
2. User success beyond clicks (task completion, reformulations)
3. Publisher traffic and ecosystem health
4. Quality and safety metrics
5. Business: ad revenue per query

Framework: Goal of the product -> Users and value exchange -> North Star -> Supporting metrics by funnel stage -> Guardrails and counter-metrics -> Decision or target

Sample answer

• Searches without a quick reformulation or bounce-back
• Outbound clicks per query on answer pages
• Rated accuracy on sampled answers, by topic risk

1. Reformulations fall 8% on queries with an answer
2. Outbound clicks fall 12% on those queries
3. Rated accuracy is 96% overall but 89% on health queries

Primary: query success (no quick reformulation or return), since users come to get answers. Hard guardrails: accuracy by risk tier (stricter on health) and publisher traffic, because the ecosystem feeds the answers. Track ad revenue per query as a business check, not the goal.

What interviewers look for

• Query success rate
• Clicks to publishers
• Answer accuracy