Interview Questions › Metrics

Ship decision: AI channel recaps in Slack

Slack tested daily AI recaps of busy channels. Users who saw recaps read 25% fewer messages but replied the same amount. How do you judge whether recaps are a success and whether to charge for them?

Context

Recaps cost money to generate per channel per day. Slack sells AI as a paid add-on.

Clarifying questions to ask first

• Is this a retention or revenue feature?
• How do we know if someone missed something important?

How to approach it

1. What does success mean: time saved, staying informed, fewer missed items
2. Fewer messages read can be good
3. Cost per recap and willingness to pay
4. Quality guardrails: missed important items

Framework: Goal of the product -> Users and value exchange -> North Star -> Supporting metrics by funnel stage -> Guardrails and counter-metrics -> Decision or target

Sample answer

• Weekly active recap readers
• Replies to important threads within 24h (stayed informed)
• Add-on trial-to-paid conversion

1. Replies to threads where users were mentioned are unchanged
2. Recap open rate drops from 60% in week 1 to 35% by week 4
3. Generation cost is about $0.40 per user per month

Success is staying informed with less reading, and replies held steady, so recaps work for those who keep using them. Charge as part of the AI add-on, focus recaps on the busiest channels to raise week-4 usage, and track missed-mention response time as the quality guardrail.

What interviewers look for

• Weekly recap readers
• Missed-mention rate
• Add-on conversion