Interview Questions › Metrics

Should Duolingo remove "hearts" for paying users only?

In Duolingo, free users lose a heart for each mistake and must wait or practise to refill. How would you measure the impact of hearts on learning and the business?

Context

Unlimited hearts is a key benefit of the paid plan. Some argue hearts frustrate learners and cause churn.

Clarifying questions to ask first

• Is the goal learning, retention or revenue?
• Can we run a holdout without hearts?

How to approach it

1. Learning outcomes vs monetisation vs engagement
2. Metrics for each
3. How to test without harming either
4. Decision framework

Framework: Goal of the product -> Users and value exchange -> North Star -> Supporting metrics by funnel stage -> Guardrails and counter-metrics -> Decision or target

Sample answer

• Day-30 retention of new free users
• Lessons completed per active day
• Free-to-paid conversion within 60 days

1. A 5% holdout without hearts shows +4% day-30 retention
2. Paid conversion fell 11% in the holdout
3. Lessons per day rose 6% but accuracy per lesson fell 3 points

Hearts trade retention for revenue and slightly better accuracy. Use long-term value (retention × conversion) as the decision metric. Explore a softer mechanic for new users in their first week, where churn risk is highest.

What interviewers look for

• Day-30 retention
• Lessons completed
• Paid conversion