SupaTalkerSuper AI Customer Service Agent
Product updates

New: Reply quality check, a second AI that reviews your AI customer service

A dedicated decision model now checks whether each AI customer service reply is based on your information and marks it Normal or Review. Included in all plans.

Illustration of a chat checked with a magnifying glass and checklist, leading to ratings, a smile and a growth chart

SupaTalker now includes reply quality check, a feature we have been planning for a while: every time the AI customer service sends a reply, the system hands it to a second model to check whether the reply is based on the information you provided on the platform.

In the AI field, similar mechanisms are often called fact-checking or ground truth verification. We apply it as a final check on SupaTalker's customer service replies. The current design is a post-reply check: verification runs after the reply has been sent, rather than holding the reply back before it goes out. The two approaches serve different purposes: the former gives you visibility into reply quality, while the latter acts as a real-time gate.

Why use AI to check AI

Having one AI model review another model's answers has become the mainstream approach as AI technology has matured: AI handles the first round of review, and people step in afterward to confirm the replies that raise concerns.

Jev makes the call

We built test data from real Q&A scenarios on the SupaTalker platform and ran several rounds of evaluation, using models from OpenAI and Claude as a control group and comparing accuracy, false-judgment rate, response time and cost. In the end, we chose the Jev model from TypeSafe AI, which has been drawing attention in the AI community recently. Here is a short introduction:

Jev is a "System One" model: a new kind of frontier model built to make fast, structured decisions on complex data that software can use directly, along with a confidence score. It responds extremely quickly at a very low cost.

In our own tests (including network round-trip time), Jev's median response time was about 0.5 seconds, mostly between 0.4 and 0.7 seconds. Because verification runs after the reply has been sent, visitors never wait for it. Response time only affects how soon the badge appears; it does not slow down the conversation.

The two questions it checks

We ask Jev two questions about every reply. Below, "context" means all the information the AI customer service had when it replied: the knowledge base and product data you provided on the platform, the earlier messages in the conversation, and whatever the system looked up at that moment.

  • grounded (the reply is supported): Can every fact stated in the reply, such as numbers, times, prices, locations, policies, or whether a service or facility is offered, be found in or directly inferred from the context? Saying "this isn't covered in our information", declining, or suggesting the visitor contact support does not count as stating a fact, so a reply that honestly says it doesn't know always scores high.
  • answerable (the context has the answer): Does the context contain enough specific information to answer the visitor's question in this turn? This question looks only at the information, not at what the AI customer service replied. Messages that need no information to respond to, such as greetings or thanks, count as answerable.

The two questions look at different things: answerable looks only at the information, while grounded checks whether the reply stays true to it. Note that grounded judges whether the reply is based on the context, not whether it is right or wrong in an absolute sense.

Putting the two together

Reply is supportedReply is not supported
Context has the answerThe reply follows the information. All goodThe information is correct, but the AI customer service said something beyond it. For example, the information says one cap is one-size-fits-all, but the reply says "all our caps are one size"
Context has no answerThe AI customer service honestly says "this isn't covered". The reply is fine, but there is a gap in your informationThe case that matters most: the context has no answer, but the AI customer service made one up. For example, a visitor asks about airport pickup, the information says nothing about it, but the reply says "we offer free pickup"

Badges in the chat history

In SupaTalker's chat history, a simple badge appears under each AI reply. The badge is based only on the grounded score.

The badge answers one question: "Did this reply say something wrong?" Only grounded can tell you that. A low answerable score doesn't mean the reply is wrong: when the information simply doesn't have the answer, honestly saying so is the correct behavior. If a low answerable score also triggered "Review", every out-of-scope question would be flagged. The answerable score is still stored for later statistical analysis, to tell whether a problem came from missing information or from the AI customer service getting it wrong.

There are two kinds of badges.

The vast majority of replies show a green "Normal" badge, meaning the reply is indeed based on the facts provided on the platform. The following replies also count as normal: saying no relevant information was found; stating a fact that is the opposite of what the visitor expected; and, when a question is unclear and could have different answers, asking back "Do you mean A or B?" to confirm.

In the chat history, a green Normal badge under an AI reply, with the tooltip “Based on provided facts: 60%”
Normal: the reply is based on the information you provided

A yellow "Review" badge means the reply may sit on a blurry boundary, or may not be based on the given facts. The score has to drop quite low before "Review" appears, and the AI customer service itself sticks closely to its rules and the given context, so it only shows up in genuinely unexpected cases. In our tests, you might not see one even across hundreds of conversation turns. When you do see this badge, we recommend that an administrator take a closer look at the reply. The screenshot below is from our internal test data. In this example, a visitor asked which item to recommend for a summer trip, and why. The product information does include this backpack, its single size and its oatmeal color, but "a light, neutral color that's easy to pair with outfits" is the AI customer service's own inference.

In the chat history, a yellow Review badge under an AI reply, with the tooltip “Based on provided facts: 27%”
Review: the reply may not be based on the information you provided. Worth a check

Note: if the connection fails during verification, the reply is marked "Unreviewed". This is rare and does not affect the reply itself.

Known limitations

Real conversations are far more complex than these two questions can cover. The current design is deliberately simple and focused on collecting data for now, so it has blind spots. For example, if the context itself is wrong and the AI customer service answers based on that wrong information, verification will still judge the reply as "supported". In some very blurry cases, differences in the judgment may not come from the model at all.

Another case we can't catch yet: the information clearly has the answer, but the AI customer service replies that it isn't covered. For example, a guesthouse's information says check-in is at 15:00, but the reply says this isn't specified. Both scores will be high and the badge will still show "Normal". Catching this would require a separate check on whether the reply actually answers the question, which the current version doesn't cover.

Compound questions are also hard to judge. When a visitor asks two or three things in one message, the AI customer service has to break the question apart and answer each part, which really tests the model's judgment. Evaluating these replies accurately would mean checking each sub-question and its answer separately; the current verification reviews the whole reply once and does not split it up.

It's worth stressing that no decision model, however capable, can be 100% accurate. In the internal evaluation described above (measured on this platform's test data against our internal standards), Jev's accuracy ranged from 92% to 99%, with a false-judgment rate of about 3–4% (across three types of false-judgment scenarios). A common trade-off with decision models is that accuracy can be very high while the false-judgment rate may rise along with it. Overall, Jev is well suited to this job, and beyond English, its judgment showed no noticeable difference in other languages such as Chinese and Japanese.

What's next

Reply quality check is one part of our broader quality improvement plan. Next, we'll introduce features such as reply quality reports, and bring in frontier models with stronger reasoning to suggest improvements based on data collected over time.

Reply quality check is included in all current SupaTalker plans. See Pricing for details.

CONTACT

Wondering if SupaTalker is right for you?

Book an onboarding consultation and our team will help you find the right way to get started.