An AI chatbot audit trail is the difference between an agent you can run and an agent you have to watch. Every vendor will tell you their model is accurate. None of them can tell you why it said a particular sentence to a particular customer at half past four last Tuesday, and that is the question a customer asks you when a reply was wrong.
In ConvertPilot every AI message opens a trace called Why this reply. It shows what the agent remembered about the customer, the sources it used and which of your rules applied. Open it after a weak answer and use it to find the next fix.
parts in every trace: memory, sources and rules
plans include Why this reply, Free included
random replies to read each week, including good ones
to run the first-week checks below
What the trace actually shows
Open any AI message in the inbox and the trace sits with it. Each part answers a different question, so read them separately.
| Part of the trace | The question it answers | What you do with it |
|---|---|---|
| What it remembered | What did the agent already know about this person? | Check the memory is right. A wrong memory produces a confidently wrong answer. |
| Sources used | Which of your pages, products, orders or training did it read? | If the source is stale, fix the source. If no source was used, the answer was not grounded. |
| Rules applied | Which of your custom instructions and training entries shaped it? | If a rule you expected is missing, the rule is not written the way you think it is. |
The agent answers from your website pages, your product catalogue, recent orders on Shopify and Wix, and the answers and facts you add in Training. The trace names which of those it used, so a reply is either traceable to something you control or it is not.
A reply you cannot trace is not a reply you can defend. The trace is what turns an argument about what the agent said into a five second check.
Why an audit trail matters more than an accuracy claim
| An accuracy claim | A trace | |
|---|---|---|
| Describes | A population of answers | One conversation |
| Useful when a customer sends a screenshot | No | Yes: you see it read an order whose state has since changed |
| Helps your team fix the cause | No, it leaves you guessing at prompts | Yes: it names the rule or page that produced the answer |
A support lead who can see which rule produced an answer can fix the rule. One who cannot ends up with twenty contradictory instructions nobody can reason about. The plan table lists "Why this reply (sources and rules used)" on all four tiers, alongside grounded answers, Training, custom instructions and human handoff. Allowances and seat counts are on the pricing page.
How to read a trace when an answer was wrong

- Read the reply again, slowly. Decide which sentence is wrong. "The whole answer" is almost never true and sends you to the wrong place.
- Check the sources it used. Open the page it read. A surprising share of wrong answers are correct readings of text that is out of date on your own site.
- Check what it remembered. A memory on the wrong contact, or one that was true three months ago, produces an answer that looks like a hallucination and is not.
- Check which rules applied. If a rule you rely on is not listed, it did not fire. Rewrite it as a plain instruction about a situation, not a hint about tone.
- Only then change the instructions. If sources, memories and rules were all correct and the answer still was not, add a Training entry with the exact answer you want.
Training holds both kinds of correction: the exact answer to a question that must always be answered one way, and a plain fact the agent should simply know. Most wrong answers need the second, and most people reach for the first. The guide to training the agent covers which to use when.
What the trace does not tell you
It shows what the agent used. It does not show what it decided not to use, and it does not prove a sentence is true.
| The trace does not | Example | Use instead |
|---|---|---|
| Validate your sources | Your returns page says 14 days and your policy is 30. The trace shows a correct reading of a wrong page | Keep source pages current |
| Predict | It explains one answer already sent, not the next one | The first-week checks below |
| Replace a boundary | Knowing why a refund was discussed helps less than a rule that stops it | Handoff rules |
| Count anything | Traces are per conversation | The analytics dashboard on the higher plans. Reading the analytics screen each week explains each number |
So the trace is a debugging tool, not a safety net. What stops things going wrong is the rules you wrote and the handoff boundary you agreed, and the trace is how you find out whether either is working.
Turning traces into training
- Once a week, open five AI replies at random, including ones that went well.
- Look for answers that were right by accident: a correct reply with no source listed, or one that leaned on a memory instead of a page.
- Write a Training entry for each, because nothing in your setup made them right.
- Approve the FAQ entries the agent drafts from questions customers actually ask, starting with any that keep appearing with a thin source.
The features page shows where Training and the FAQ drafts live in the console.
Checks to run in your first week
Spend twenty minutes proving the trace says what you think it says, from a customer account, on the channel your customers actually use.
| Ask | It passes when |
|---|---|
| A question your website answers | The trace names the page you expected, not an old landing page you forgot about |
| A question only Training answers | The training entry is listed under the rules applied. If not, it is worded as a preference instead of an instruction |
| Something your rules forbid | The agent asks for a person, and the trace names the rule that stopped it |
| About an order | On a connected store, the trace shows it read the order instead of guessing from the conversation |
| The same question twice, a day apart | Both traces use the same sources. Different sources usually mean a page is competing with a training entry |
Run those five and you will know more about your own setup than most stores learn in a quarter. If you are still connecting channels, the channels page covers what each one supports, and the one inbox guide covers the order to connect them in.


