Improving a ChatFrom chatbot: conversations, analytics and evaluations
Updated
Reading conversations, fixing wrong answers, analytics, evaluations by a judge from another AI company, Langfuse tracing and published test results.
Can I read what people ask the chatbot?
Yes. Conversations shows every conversation from every channel. You can filter for the ones it could not answer, the answers people rated badly, and the ones passed to your team. Each answer shows its model, its tokens and its cost, and every tool call is in the log.
How do I fix a wrong answer?
Find the conversation, press Fix this under the weak answer, and write the answer you wanted. The chatbot uses it from then on, as a Q&A entry in its knowledge. You can also look at the source the answer came from and fix the page or document there.
Can visitors rate answers?
Yes. Visitors can rate every answer and say what was wrong. Ratings are taken only from the chat itself and are limited per visitor, so nobody can flood them.
What does Analytics show?
The last 7, 30 or 90 days: conversations each day with the share left unanswered, what they cost, and how people rated the answers. When answers are traced with Langfuse, it also shows answers each day, how long they took, and any errors from the model.
What are evaluations?
Evaluate is for measuring quality instead of guessing. You write a set of test questions, each with what a good answer does, and ChatFrom asks your chatbot all of them:
- Answers: a judge model scores each answer from 0 to 10 with a reason. The judge always comes from a different AI company than your chatbot's model, so no model marks its own answers.
- Retrieval: scores how well the chatbot found and used your knowledge.
Run it again after a change to see whether it helped. During a run the chatbot uses none of your tools, so nothing is booked or sent for real.
Do evaluations use my credits?
No. Evaluations run on your own OpenRouter key, paid to OpenRouter at its prices, not from your plan's credits. You add the key on the Evaluate page or under Settings, and it works for all your chatbots.
Can I trace answers in Langfuse?
Yes, optionally. Under Settings, Observability, paste your Langfuse project's address and keys, and every answer is traced there: the question, the passages it read, the model call and its cost. That data then goes to your own account with that service.
What is the Overview page?
The Overview page brings it together: a setup checklist that ticks itself off, your numbers for the week, and anything that needs your attention.
How good are ChatFrom's answers?
ChatFrom tests every change to its engine on questions with known right answers. Latest published results:
- Golden set, 23 September 2026: 12 questions about a made-up lamp shop, run three times and graded by a model from another company: 9.86 out of 10.
- RAGAS, 8 September 2026: 55 questions from the engine's example documents: faithfulness 0.83, context recall 0.92, context precision 0.76, answer relevancy 0.66.
- Scripted Shopify customers, 28 September 2026: 12 customers asking a test store about orders and returns: 12 of 12 handled correctly.
You can run both kinds of test on your own chatbot under Evaluate.