Keep Making AI Answers Better With Improvements

AI Answers helps your customers get answers quicker, but it works best when it’s guided with the right information. The most effective way to improve results is to regularly review AI conversations and decide whether to make an AI Improvement, set a Guardrail, or update your Docs.

For steps to add an Improvement, take a look at AI Agents in Help Scout: Manage AI Agents: Improvements.

This guide explains what to look for while reviewing AI Answers conversations and how to choose the right next step.

What to Look for When Reviewing AI Conversations

When you review AI conversations, focus on places where the AI response was incomplete, unclear, or incorrect.

A simple way to evaluate a conversation is to ask: “If a customer sent these same questions to us in an email, would our team be able to respond more clearly or completely?”

• If yes, you likely have an opportunity to improve AI Answers.
• If no, the AI may have been limited by missing information from the customer (and there may be nothing to change).

Read the full conversation before you decide what to do. Sometimes the last customer message changes what the AI should have answered earlier.

Ways to Improve AI Answers

There are several ways to influence future AI responses:

1. Add an Improvement to your AI Agent
2. Update your Docs
3. Give your AI Agent additional resources — beef up its knowledge with websites and documents that can help
4. Set up a Guardrail that tells the AI Agent to escalate to your team

In many cases, adding an Improvement is the right first step.

More about adding Improvements and additional resources here: AI Agents in Help Scout: Manage AI Agents

When to Add an Improvement

Improvements are best for fine-tuning AI Answers when the issue is specific, unusual, or not something you want to add to your documentation.

Add an Improvement when:

• The AI response was partly correct, but needs clarification or a more accurate direction.

  Example: The AI suggests a solution that’s close, but misses an important detail or uses the wrong steps.

• Customers ask about features or products you don’t offer.

  You likely don't want to point out all the things your customers might ask about that you don't offer in your Docs. In this case, Improvements help the AI respond clearly without sounding like it’s refusing to help.

• The question is uncommon or highly specific.

  Example: A technical limit question that only comes up occasionally.

• The question requires support involvement.

  If the customer needs account-specific help or a change that requires intervention, add an Improvement that tells the AI to direct the customer to contact Support. It's worth considering making items like this a Guardrail as well! Read more about Guardrails here: AI Agents in Help Scout: Manage AI Agents: Guardrails.

  Examples include billing questions, account-specific investigations, or requests that require a manual action from your team.

When to Update Your Docs Instead

Updating Docs is best when a question is common and broadly relevant.

Consider updating Docs when:

• Many customers ask the same question.
• The answer is useful to most customers, not only a small subset.
• The topic fits naturally into an existing article (or clearly needs a new one).

If you’re deciding between the two, a simple rule is:

• Frequent and broadly useful = update Docs
• Specific, rare, or support-dependent = add an Improvement

Not Every Conversation Needs an Improvement

Some conversations don’t need changes at all.

If your team would need more information to be able to answer this question, don’t add an Improvement. Incomplete questions happen, and adding Improvements that try to guess what the customer meant could make AI Answers less accurate in other situations.

AI Answers will sometimes need clarification for the same reason people do: the customer didn’t provide enough detail to answer safely and correctly.

Test and Iterate

After adding Improvements, head to the Test tab on your AI Agent and ask it some questions that relate to the Improvement you added. You might find that you need to add some additional context to your Improvement to achieve the best answer — you can pop right back over to Knowledge > All Improvements to find the Improvement you added and make those changes, then test again.

See AI Agents in Help Scout: Manage AI Agents: Test for more information on the Test option.