Interview Questions › Root Cause Analysis

Engagement dropped 15% after "improved" algo launch

Your ML team deployed an "improved" recommendation algorithm that tested +4% offline engagement, but 5 days post-100% production rollout, total video views and user watch time dropped by 15%. Investigate why the offline model failed in the real world.

Context

The new algorithm prioritized long-term user diversity and creator variety over immediate short-term engagement.

Clarifying questions to ask first

• Did the model over-index on exploration (showing random content) at the expense of proven user preferences?
• How did creator upload frequency respond?

How to approach it

1. Analyze the Feedback Loop Failure: Offline Training Data Bias vs Online Distribution Shift vs Creator Cold-Start Dynamics
2. Segment by User Longevity (New, Casual, Power Users) and Content Modality
3. Evaluate echo chamber mitigation unintended consequences: Did high-affinity content get suppressed too aggressively?
4. Propose model rollback criteria and online-offline evaluation guardrails

Framework: Offline vs Online Evaluation Gap -> Diversity vs Relevance Tradeoff -> Creator Supply Ecosystem Feedback -> Guardrail Metric Architecture

Sample answer

• Exploration penalty alienated top 20% power users who wanted specific niche content
• Model rewarded clickbait thumbnails with low completion rates
• Cold-start recommendation latency caused video playback buffering on slower network connections

1. Analysis reveals: In an attempt to reduce repetitive content, the model penalized top-affinity creator videos by 50%, forcing generic viral videos that drove immediate user session abandonments

Model over-corrected on diversity, starving users of core high-affinity interests; rolled back to hybrid architecture with dynamic exploration exploration caps based on session depth.

What interviewers look for

• Watch Time per User
• Video Completion Rate
• Short-Term Engagement (Likes/Shares) vs Content Diversity Score
• Session Recurrence Rate