Testing Jev and six LLMs for chatbot intent routing

I tested Jev and six LLMs from OpenAI, Anthropic and DeepSeek on 30 chatbot messages. Here’s what I learned about accuracy, speed, cost and confidence.

I wanted to see how seven models would handle a small but important chatbot task: deciding whether a visitor wants a refund or needs support. Three got all 30 test messages right. Jev, a model built for decisions, matched GPT-5 nano and GPT-4.1 mini. Claude Haiku 4.5 missed one message, both DeepSeek V4 models missed at least two, and Claude 3 Haiku missed eight.

The part that interested me most was what came alongside Jev's answer. With the connection kept open, it replied in about a third of the time. It cost $1.41 per 100,000 decisions, which put it between GPT-5 nano at $1.03 and GPT-4.1 mini at $2.52. It also returned a confidence score for each decision, something none of the LLMs returned in this test. That gave me a way to spot messages where asking the visitor might be more useful than choosing a branch.

I run ChatFrom, where owners build chatbot conversations as workflows. One step decides which branch a visitor's message should take. Today, that step uses an LLM. I ran this test to understand whether a decision model could help, and what else I might need to change in the workflow.

What intent routing is in a chatbot workflow

Intent routing means reading a visitor's message and choosing from a few predefined paths. If someone wants their money back, the chatbot follows the refund path: ask for the order number, call the refund tool and reply. If someone needs help, it follows the support path. The router itself does not write the answer. It just decides what happens next.

In ChatFrom, a small LLM currently makes that choice. The prompt asks for exactly one label, and the code matches the reply against the branch names.

Thirty messages, one question and seven models

I wrote 30 messages that could plausibly arrive in a support inbox and labelled each with the branch I would choose. Of those, 24 are clear requests, with 12 for each path. The remaining 6 are deliberately vague. I included them to see what happens when the visitor has not quite said what they need.

I sent every message to each of the seven models with the same question: What does the visitor want: a refund, or help from customer support? Here is the lineup.

1. GPT-5 nano is the smallest model in OpenAI's GPT-5 family. It is a reasoning model, so I tested it twice: once with reasoning effort set to minimal and once at its default.
2. GPT-4.1 mini is a small OpenAI model that replies without a separate reasoning stage.
3. Claude Haiku 4.5 is Anthropic's small model. I also included Claude 3 Haiku, its predecessor from 2024, which is still sold at a quarter of the price.
4. DeepSeek V4 Flash and DeepSeek V4 Pro are the small and large V4 models. Both reason by default. I ran each twice, with its default reasoning and with reasoning switched off.
5. Jev (version 1.13) is a decision model from TypeSafe. Instead of asking it to write an answer, you send it a state and a typed question. It returns a choice, a probability for each option and a confidence score.

I mostly chose small models because this step runs on every turn, before the chatbot can answer. For a one-word decision, I wanted to keep both the wait and the cost low. DeepSeek V4 Pro was the one large model I included. In this test, it was the slowest and most expensive option, with no accuracy improvement over its small sibling.

For reference, these were the published prices on the day I ran the test.

What the six LLMs receive

system: Answer with exactly one of these labels and nothing else: refund, support

user: What does the visitor want: a refund, or help from customer support?

      Message: this is not working and I paid for it

assistant: support

The reply gives the workflow a branch to follow, but no indication of how uncertain that choice might be.

What Jev receives and what it returns

POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": "this is not working and I paid for it",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "What does the visitor want: a refund, or help from customer support?",
      "criteria": {
        "refund": "The visitor wants money back",
        "support": "The visitor needs help from customer support"
      }
    }
  }
}

{
  "model": "jev-1.13.0",
  "answers": {
    "route": {
      "type": "choice",
      "choice": "support",
      "confidence": 0.21,
      "probabilities": { "refund": 0.39, "support": 0.61 }
    }
  },
  "usage": { "input_tokens": 336, "output_tokens": 31 }
}

Jev also chooses support. Alongside that choice, it reports a split of 61 to 39 and a confidence score of 0.21. That gives the workflow something else to work with: a signal it can use to pause and ask the visitor what they mean.

I measured the full round trip from Germany, making calls one after another for each model. Jev used its own API; all the LLMs went through OpenRouter. I read every reply with the same logic as our workflow step. It first accepts an exact label. If the reply contains anything else, it searches for labels appearing as whole words, checking the branches in order. This matters because the result measures the branch our workflow would take, including any parsing mistakes. I planned the test myself and ran it on my own accounts.

How accurate each model was

I thought a two-way choice might produce a tie across the board. Three models did finish at 30 of 30, but the other four left some useful problems to investigate.

The cost column deserves a look before moving on, because price and accuracy did not move together. Each of the three models that reached 30 of 30 had a setting between $1.03 and $2.52 per 100,000 decisions. The most expensive setting in the test, DeepSeek V4 Pro with default reasoning, cost $12.26 and scored 28. The cheapest, DeepSeek V4 Flash with reasoning off, cost $0.33 and scored 25. Paying more did not buy a better router here, and paying the least came with five missed messages.

Three ways an LLM router went wrong

Across the test, 20 of the 270 LLM replies would have sent the visitor down a different path from my label. Looking at those replies, I found three patterns: ambiguity, parsing problems and incorrect choices.

1. A vague message still got a definite choice

Both DeepSeek models sent “this is not working and I paid for it” to the refund path. In three of their four settings, they did the same with “I want to cancel”. Along with one “I’m really unhappy with the service”, these made up 8 of the 20 mistakes. My labels are open to debate here: the visitor has not clearly asked for either path. That is why these examples are useful. The models disagreed, but the single-word replies gave the workflow no way to recognise the ambiguity.

2. The model chose correctly, but the parser did not

Claude Haiku 4.5 chose the intended branch for all 30 messages. The problem came with “I paid yesterday but my plan still says Free”. It returned the right label, then added an explanation despite the instruction to stop there.

assistant: support

The visitor is reporting a technical issue (their plan status hasn't
updated after payment) and needs help resolving it, not requesting a refund.

That extra explanation changed how the parser handled the reply. Since the response was no longer an exact label, the code searched for label words in branch order. It found “refund” in the explanation and selected the first branch: refund. The model had chosen support, but our code would have sent the visitor elsewhere. I repeated the same message four more times. Two replies included an explanation, and two contained only the label.

DeepSeek ran into related formatting problems three times. One reply was a sentence containing both labels. Two others returned “help”, which is not a valid label, so the parser fell back to the first branch. These account for 4 of the 20 mistakes. In each case, the problem was how the reply was formatted and read, rather than a misunderstanding of the request.

3. It chose wrong

Claude 3 Haiku sent 8 of the 13 refund messages to support. One was “Please refund my last payment, I cancelled before the renewal”. I repeated that message three more times and got support, support and then refund. Claude 3 Haiku was the fastest LLM in this test, and its price is low. Still, this model from 2024 did not handle explicit refund requests consistently enough for me to use it in this workflow.

Once the workflow has reduced a reply to “refund”, the reason behind that choice is easy to lose. It could reflect a clear request, an ambiguous message or a parsing mistake.

Speed and cost

Accuracy was only part of what I wanted to understand. The next two charts show how long visitors would wait for a routing decision, both on a typical call and on a slower one.

A few details helped me make sense of the timing results.

• Reasoning added time and cost to a short answer. At its default setting, GPT-5 nano used about 140 hidden reasoning tokens before returning a label. It was two and a half times slower and seven times more expensive than the same model with reasoning turned down, with the same 30 of 30 result. DeepSeek V4 Pro also reasoned before answering, using about 80 tokens. Turning reasoning down is worth testing for this task, but accuracy needs another check afterwards: DeepSeek V4 Flash missed three more messages with reasoning off.
• The connection made a big difference for Jev. Its server reported 0.10 seconds of processing on every call. Much of the remaining time appeared to come from the network. The API runs in the western United States, with a round trip from Germany taking about 0.2 seconds. Opening a fresh secure connection adds about 0.6 seconds more. Keeping the connection open brought a decision to 0.32 seconds. With a new connection for each call, Jev was no faster than the small LLMs.
• DeepSeek's timings varied across providers. OpenRouter spread the calls over seven to nine hosting providers, and individual calls took from 0.6 to 12 seconds. Choosing a single provider might make the timings more consistent, but I did not test that here. I wanted to see the behaviour with the default setup.

What each decision costs

Routing runs on every turn, so its price is multiplied by every message a chatbot receives. That makes the cost per decision worth a closer look, even when a single call costs a tiny fraction of a cent.

To make those figures easier to picture, here is the same list at one million routed messages a month, next to each setting's accuracy.

At a million decisions a month, the three accurate options cost $10.30, $14.10 and $25.20. GPT-5 nano left at its default reasoning would cost $74.00 for the same 30 of 30, and DeepSeek V4 Pro with default reasoning would cost $122.60 for a lower score. The most expensive setting costs 37 times as much as the cheapest, and most of that gap comes from two things I can control: whether the model reasons, and how much it writes.

The cheapest setting was also one of the least accurate, and that is where the larger cost sits. A routing call costs about a thousandth of a cent. A visitor sent down the wrong path costs a support conversation, and if the refund path pays out, it costs the refund as well. One mistaken refund of a $29 subscription is more than the bill for routing a million messages with any of those three options. I would not choose a router on price alone. Among routers that get the answer right, though, price is a fair tiebreaker, and the token bills explain where it comes from.

What you actually pay for

A low token price does not tell the whole story. Each model used a different number of tokens to make the same decision, and some also billed for reasoning.

• Compare the cost of a decision, not just a token. Jev counted 336 input tokens for a request where the LLM prompt used about 54. Its question template is included in the bill. Although its input price is nearly ten times lower than GPT-4.1 mini's, it uses six times the tokens, bringing the decision cost roughly into the same range as GPT-5 nano. DeepSeek V4 Flash has a lower token price and uses only a seventh of the input tokens, which helps explain its lower total cost.
• Extra explanations also appear on the bill. Claude Haiku 4.5 averaged 7 output tokens, while a single-word reply takes 4. Its occasional explanations account for the difference. At $5 per million output tokens, those extra words had the highest output token rate in this test.
• Jev's bill was the easiest to predict. Its output is free and it has no reasoning stage, so the cost depends only on the length of the request. The LLM bills moved with what each model chose to write, whether that was Claude Haiku 4.5's explanations or GPT-5 nano's reasoning.
• More reasoning did not improve this result. GPT-5 nano used an average of 141 reasoning tokens to choose between two labels. Here, that extra work did not improve accuracy over minimal reasoning.

How confidence helps the workflow ask a better question

Jev's confidence scores were the most useful part of the test for my workflow. Its choices all matched my labels, while its scores also picked out messages where a follow-up question would make sense. That is promising, even though this small sample cannot establish how well the scores are calibrated more broadly.

With a threshold of 0.8, exactly two of the 30 messages would trigger a follow-up question. Both are messages I would want to clarify myself. “I want to cancel” could mean the visitor wants a refund, or simply needs help finding the cancel button. “this is not working and I paid for it” expresses frustration, but does not yet say how the visitor wants us to help.

These were also the two messages where the LLMs disagreed about the route.

On the messages where the LLMs disagreed, Jev returned lower confidence scores. The workflow could use those scores to ask before proceeding.

The confidence score is separate from the probability assigned to the chosen path. Looking at both makes the six vague messages easier to interpret.

Under that rule, the other 28 messages would continue directly to their chosen branch. The two unclear messages would get a question with two buttons, letting the visitor choose what they need.

Checking the confidence adds nothing to the bill. The score arrives in the same response as the choice, so there is no second call to pay for or wait on. The only price is one more question for the visitors whose message was unclear, which was 2 of the 30 in this test.

What TypeSafe claims and what I measured

I also compared the results with the claims on TypeSafe's site. Some fit what I observed; others need a broader comparison or more testing than I did here.

What this changes in our workflow

I am adding a decision model as an optional way to run a condition step, alongside the existing LLM option. The test also pointed to improvements I can make to the step itself.

• The LLM stays the default. It supports our multilingual workflows and lets owners keep using the provider they have already chosen.
• The condition step gets a confidence threshold and a “when unsure” branch. That branch connects to the existing step that asks the visitor a question with buttons.
• The parser checks the first word first. If a reply starts with a valid label, that is the branch it selects, even if an explanation follows. This alone would have taken Claude Haiku 4.5's result from 29 to 30.
• Unrecognised replies lead to a question. When no label is found, the workflow should ask the visitor what they mean instead of silently taking the first path.
• Reasoning is reduced, then accuracy is checked again. The results give me a reason to question whether this step benefits from spending 140 tokens thinking before it returns one word.

Before shipping those changes, I still need to account for three limitations.

• Language. TypeSafe says Jev is strongest in English, which is the only language I tested. I need to test German before shipping it.
• Where the data goes. Jev's API runs in the United States. OpenRouter also spread the DeepSeek calls over 14 hosting companies that I had not individually selected. That was fine for these made-up messages. Before using real visitor messages from a European company, I need to review the data processing terms and make an explicit provider choice.
• Sample size. Thirty messages and one run per setting are a starting point. As two of the examples showed, repeating a message can produce a different reply. I see this as a practical test of our workflow, not a broad benchmark of the models.

Common questions

Is Jev more accurate than an LLM for intent routing?

In this small test, Jev tied with GPT-5 nano and GPT-4.1 mini at 30 of 30 on the two-way routing task. Claude Haiku 4.5 scored 29, DeepSeek V4 Flash and Pro scored 28 with default reasoning, and Claude 3 Haiku scored 22. Jev also returned confidence scores, which the LLMs were not asked to provide in this setup.

Which LLM is best for intent classification in a chatbot?

For this workflow, GPT-4.1 mini and GPT-5 nano with minimal reasoning were the strongest LLM options I tested. Both reached 30 of 30, with median times under a second and costs of $1 to $2.60 per 100,000 decisions. Claude Haiku 4.5 also chose the correct branch each time, but an extra explanation confused our parser, and it cost about $9. DeepSeek V4 Flash cost the least, though its choices on vague messages differed from my labels.

What does intent classification cost per message?

The cost was a small fraction of a cent per message, ranging from below a thousandth to above a hundredth of a cent. Across the settings I tested, it was $0.33 to $12.26 per 100,000 decisions. Each of the three models that reached 30 of 30 had a setting in the $1 to $2.60 range.

Is a decision model cheaper than an LLM for routing?

Not by much for one small question. Jev cost $1.41 per 100,000 decisions in this test, between GPT-5 nano with minimal reasoning at $1.03 and GPT-4.1 mini at $2.52. It was much cheaper than the reasoning models left at their defaults, which cost $7.40 and $12.26, and cheaper than Claude Haiku 4.5 at $9.06. DeepSeek V4 Flash cost less than all of them, but it was less accurate.

What confidence threshold should I use?

I used 0.8, which would send 2 of the 30 messages to a clarifying question. Treat that as a starting point for testing, not a universal setting. Label a few dozen messages from your own workflow, check the scores on requests you would want to clarify, and try a threshold just above them. Then check which messages it sends through and which ones it pauses.

Why was the fast model not faster in practice?

The connection and the distance to the API accounted for much of the wait. Jev reported 0.10 seconds of processing; the rest of the measured time was outside that processing window. Reusing an open connection helped in this test. Measuring from your own server region will give you a more useful expectation of the speed visitors will experience.

I started by asking whether an LLM was the right model for this routing step. The test gave me a more useful question: when the visitor's intent is unclear, what should the workflow do next? The best three models tied on accuracy, but accuracy alone did not address ambiguous requests or parsing problems. Jev's confidence scores gave me a practical way to add a follow-up question. Together with a better parser and an explicit fallback, that is the change I want to bring into ChatFrom.

Cost pointed the same way. Every accurate option was cheap, at $1.03 to $2.52 per 100,000 decisions, so the price list was never going to decide this on its own. What moved the bill was how each model was used. Leaving reasoning at its default multiplied GPT-5 nano's cost by seven for the same result, and a model that adds explanations pays for every extra word. The larger cost sits outside the API bill altogether: a visitor sent down the wrong path, or a refund started on a guess. A router that can say it is unsure costs about the same per call, and it is the one place where I can bring that larger cost down.

ChatFrom lets you build these workflows visually, with a condition step, a branch for each intent and a question when the visitor’s request needs clarification. It is still in development, and registration opens soon.