Interview Questions › AI Product

Design AI reply drafts for a customer support inbox

Freshdesk agents answer hundreds of tickets a day. Design an AI feature that drafts replies for them. How would you build it and know it works?

Context

Customers are mid-size businesses. Wrong answers to their customers could cause refunds or churn. Agents are measured on handle time and satisfaction.

Clarifying questions to ask first

• Which ticket types first?
• Does the AI have the company's help articles?
• Can drafts ever be sent without an agent?

How to approach it

1. Start from the user problem, not the model: is AI the right tool here?
2. Describe the experience and where the model fits in it
3. Define what "good output" means and how you would evaluate it before and after launch
4. Name the failure modes (wrong answers, bias, privacy, cost) and how you contain them
5. Close with a launch plan and success metrics

Framework: User and problem -> Is AI the right tool? -> Experience design and human-in-the-loop -> Quality bar and evaluation -> Risks, cost and latency -> Staged launch and metrics

Sample answer

• Agent always reviews before sending
• Ground drafts in the knowledge base with citations
• Start with high-volume, low-risk ticket types

1. Offline eval on past tickets rated by senior agents
2. Measure edit distance between draft and sent reply
3. Guardrail: CSAT and reopen rate

Agent-in-the-loop drafts grounded in help articles, launched on two high-volume ticket types. Success: handle time down with CSAT and reopen rate flat or better; acceptance rate and edit distance track quality.

What interviewers look for

• Draft acceptance rate
• Handle time
• CSAT