How it works6 min read

Why this reply: the trace that shows where an AI answer came from

Every AI message in ConvertPilot opens a trace showing what it remembered, which sources it read and which of your rules applied. Here is how to read one, and what it will not tell you.

SKShantanu KumarFounder and Chief Solutions Architect, September 30, 2026, updated November 17, 2026
The Why this reply panel in the ConvertPilot console, showing the memories, sources and rules behind an AI answer.

An AI chatbot audit trail is the difference between an agent you can run and an agent you have to watch. Every vendor will tell you their model is accurate. None of them can tell you why it said a particular sentence to a particular customer at half past four last Tuesday, and that is the question a customer asks you when a reply was wrong.

In ConvertPilot every AI message opens a trace called Why this reply. It shows what the agent remembered about the customer, the sources it used and which of your rules applied. Open it after a weak answer and use it to find the next fix.

3

parts in every trace: memory, sources and rules

All 4

plans include Why this reply, Free included

ConvertPilot plan table
5

random replies to read each week, including good ones

20 min

to run the first-week checks below

What the trace actually shows

Open any AI message in the inbox and the trace sits with it. Each part answers a different question, so read them separately.

Part of the traceThe question it answersWhat you do with it
What it rememberedWhat did the agent already know about this person?Check the memory is right. A wrong memory produces a confidently wrong answer.
Sources usedWhich of your pages, products, orders or training did it read?If the source is stale, fix the source. If no source was used, the answer was not grounded.
Rules appliedWhich of your custom instructions and training entries shaped it?If a rule you expected is missing, the rule is not written the way you think it is.
The three parts of Why this reply. Each one points at a different fix, which is why reading them separately is worth the extra ten seconds.

The agent answers from your website pages, your product catalogue, recent orders on Shopify and Wix, and the answers and facts you add in Training. The trace names which of those it used, so a reply is either traceable to something you control or it is not.

A reply you cannot trace is not a reply you can defend. The trace is what turns an argument about what the agent said into a five second check.

Why an audit trail matters more than an accuracy claim

An accuracy claimA trace
DescribesA population of answersOne conversation
Useful when a customer sends a screenshotNoYes: you see it read an order whose state has since changed
Helps your team fix the causeNo, it leaves you guessing at promptsYes: it names the rule or page that produced the answer

A support lead who can see which rule produced an answer can fix the rule. One who cannot ends up with twenty contradictory instructions nobody can reason about. The plan table lists "Why this reply (sources and rules used)" on all four tiers, alongside grounded answers, Training, custom instructions and human handoff. Allowances and seat counts are on the pricing page.

How to read a trace when an answer was wrong

Reading a trace after a wrong answer, from the outside in. Step 1, the reply: pick the exact sentence that is wrong. Step 2, sources: open the page it read and check whether it is out of date. Step 3, memory: is it on the right contact and still true? Step 4, rules: did the rule you rely on actually apply? Step 5, instructions: only now add a Training entry.
Each step rules out one cause before you touch the instructions.
  1. Read the reply again, slowly. Decide which sentence is wrong. "The whole answer" is almost never true and sends you to the wrong place.
  2. Check the sources it used. Open the page it read. A surprising share of wrong answers are correct readings of text that is out of date on your own site.
  3. Check what it remembered. A memory on the wrong contact, or one that was true three months ago, produces an answer that looks like a hallucination and is not.
  4. Check which rules applied. If a rule you rely on is not listed, it did not fire. Rewrite it as a plain instruction about a situation, not a hint about tone.
  5. Only then change the instructions. If sources, memories and rules were all correct and the answer still was not, add a Training entry with the exact answer you want.

Training holds both kinds of correction: the exact answer to a question that must always be answered one way, and a plain fact the agent should simply know. Most wrong answers need the second, and most people reach for the first. The guide to training the agent covers which to use when.

What the trace does not tell you

It shows what the agent used. It does not show what it decided not to use, and it does not prove a sentence is true.

The trace does notExampleUse instead
Validate your sourcesYour returns page says 14 days and your policy is 30. The trace shows a correct reading of a wrong pageKeep source pages current
PredictIt explains one answer already sent, not the next oneThe first-week checks below
Replace a boundaryKnowing why a refund was discussed helps less than a rule that stops itHandoff rules
Count anythingTraces are per conversationThe analytics dashboard on the higher plans. Reading the analytics screen each week explains each number

So the trace is a debugging tool, not a safety net. What stops things going wrong is the rules you wrote and the handoff boundary you agreed, and the trace is how you find out whether either is working.

Turning traces into training

  1. Once a week, open five AI replies at random, including ones that went well.
  2. Look for answers that were right by accident: a correct reply with no source listed, or one that leaned on a memory instead of a page.
  3. Write a Training entry for each, because nothing in your setup made them right.
  4. Approve the FAQ entries the agent drafts from questions customers actually ask, starting with any that keep appearing with a thin source.

The features page shows where Training and the FAQ drafts live in the console.

Checks to run in your first week

Spend twenty minutes proving the trace says what you think it says, from a customer account, on the channel your customers actually use.

AskIt passes when
A question your website answersThe trace names the page you expected, not an old landing page you forgot about
A question only Training answersThe training entry is listed under the rules applied. If not, it is worded as a preference instead of an instruction
Something your rules forbidThe agent asks for a person, and the trace names the rule that stopped it
About an orderOn a connected store, the trace shows it read the order instead of guessing from the conversation
The same question twice, a day apartBoth traces use the same sources. Different sources usually mean a page is competing with a training entry

Run those five and you will know more about your own setup than most stores learn in a quarter. If you are still connecting channels, the channels page covers what each one supports, and the one inbox guide covers the order to connect them in.

Common questions

See it on your own site

The Free plan runs the full agent on website chat and email. No card.

Start free

Make every conversation count.

The Free plan includes the full AI agent on website chat and email. No card needed to get started.

Your business. Your voice. One connected inbox.