Testing OpenAI's Decisions API for chatbot intent routing

I ran my 30-message routing test again with OpenAI's new Decisions API, Jev, GPT-6 Luna and three other models, for accuracy, speed, cost and confidence.

OpenAI's new Decisions API got 29 of my 30 routing messages right, with a median answer time of 0.18 seconds from Germany. That makes it the fastest setting I have measured so far. It cost $1.40 per 100,000 decisions, which is the same as Jev at $1.41. Jev got all 30 right and was more consistent on its slower calls. Both cost about 45% less than GPT-4.1 mini, which also got all 30 right but needed 0.75 seconds.

OpenAI opened its Decisions API in public beta on 6 October. Like Jev, it does not write an answer. You give it a typed question, and it returns a choice with a probability for each option. Last month I tested Jev and six LLMs on a small routing task, so I wanted to see how the new API compares. I ran the same 30 messages through it, through Jev again and through four LLMs.

As in my last test in September, the most useful part for my workflow was the confidence score. Jev gave low scores to the two vague messages and high scores to everything else. The Decisions API gave lower scores in general. Its one mistake had a score low enough for a threshold to catch, but only if the threshold is chosen for that model.

What the Decisions API is

The Decisions API runs on GPT-6 Luna, which is OpenAI's small model from September. It has its own endpoint. You send an input and a list of named questions. Each question can be a yes/no condition, a choice between options or a score on a scale. You only pay for input tokens, at $0.10 per million. Output is free.

The API is not on OpenRouter. OpenRouter offers GPT-6 Luna as a chat model, but not the Decisions endpoint. So I called OpenAI directly with my own key.

POST https://api.openai.com/v1/decisions
{
  "model": "gpt-6-luna",
  "input": "this is not working and I paid for it",
  "questions": [
    {
      "type": "choice",
      "name": "route",
      "instructions": "What does the visitor want: a refund, or help from customer support?",
      "choices": [
        { "value": "refund", "description": "The visitor wants money back" },
        { "value": "support", "description": "The visitor needs help from customer support" }
      ]
    }
  ]
}

{
  "answers": [
    {
      "type": "choice",
      "name": "route",
      "choice": "support",
      "confidence": 0.72,
      "probabilities": [
        { "value": "refund", "probability": 0.14 },
        { "value": "support", "probability": 0.86 }
      ]
    }
  ]
}

For this vague message, the Decisions API chose support. It split the message 86 to 14 and reported a confidence of 0.72. Jev chose support as well, but it was much less sure. It split the message 65 to 35 and reported a confidence of 0.30.

The same test, one month later

I used the same 30 messages as in the September test. There are 12 clear refund requests, 12 clear support requests and 6 vague messages. Each one is labelled with the branch I would choose. The LLMs received the same one-label prompt as before, and I read their replies with the same parser as our workflow step. That way, each result is the branch the workflow would actually take.

• GPT-6 Luna through the Decisions API. I asked it once as a choice and once as a yes/no question: does the visitor want a refund? A developer on OpenAI's forum found its yes/no answers better calibrated than its choices, so I tested both.
• Jev through its own API, with the same choice question as in September.
• GPT-6 Luna as a chat model through OpenRouter. I ran it once at its default reasoning and once at minimal.
• GPT-4.1 mini and GPT-5 nano with minimal reasoning. These were the two LLMs that got all 30 right in September.
• Gemini 3.1 Flash-Lite, a small Google model I had not tried before.

I wanted to include Claude Haiku 5.5 as well, which came out two days before the test. My OpenRouter key's guardrail blocked it, so it is not in the results. The table below shows the published prices on the day of the test.

I measured everything from Germany and made one call at a time, as in September. This time, each setting got one extra call at the start that I did not count. That call opened the connection. The 30 measured calls then ran on an open connection, just as they would on a server. For the LLMs, the cost is what OpenRouter billed. For the two decision APIs, it is the input tokens multiplied by the published price.

Results

Four settings got all 30 messages right: Jev, GPT-4.1 mini and GPT-6 Luna as a chat model at both reasoning levels. The Decisions API missed one message with the choice question and two with the yes/no question. Gemini 3.1 Flash-Lite sent three of the vague messages to refund. GPT-5 nano dropped from 30 in September to 26, even though the prompt and settings were the same. I come back to that further down.

Speed

• The decision APIs were three to ten times faster than the chat models. The Decisions API had a median of 0.18 seconds and Jev 0.24 seconds. The chat models needed between 0.75 and 1.88 seconds.
• Jev was more consistent. Its slowest calls took 0.27 seconds. The Decisions API's slowest calls took 0.90 seconds, and its times ranged from 0.09 to 1.30 seconds.
• Minimal reasoning made little difference for GPT-6 Luna as a chat model. It still used about 16 reasoning tokens to return a single word, compared with 18 at its default setting. The median time dropped from 1.88 to 1.30 seconds, and accuracy stayed the same.
• Gemini 3.1 Flash-Lite's times probably say more about the provider than the model. Four calls came back in under a second, while 25 took about 10.5 seconds each. A delay that regular points to the provider OpenRouter picked, so I left it out of the chart.

Cost

Every setting cost between $1.03 and $2.52 per 100,000 decisions. That is a much smaller range than in September, when two settings with reasoning left on cost more than $7.

• The Decisions API also bills its question template. It counted about 140 input tokens per decision. The LLM prompt used 54, and Jev counted about 334. The Decisions API charges more per token, while Jev counts more tokens. In the end, the two cost the same.
• Neither decision API charges for output. The cost depends only on the length of the request, which makes it easy to predict.
• The same model cost more as a chat model. GPT-6 Luna cost $1.71 through OpenRouter and $1.40 through the Decisions API. As a chat model, it wrote about 24 tokens to return a single word. Most of those were reasoning tokens.
• The yes/no question cost more than the choice. Its template is longer, so it cost $1.66 instead of $1.40.

At a million decisions a month, either decision API would cost about $14. GPT-4.1 mini would cost about $25. But as in September, the bigger cost is a visitor who gets sent down the wrong path. That cost does not show up on any price list.

Confidence

Both decision APIs return a confidence score with every choice. The table shows the messages where either of them scored below 0.8, which is the threshold I used in September.

Jev's scores worked the way I hoped, again. Every clear message scored 0.98 or higher, and only the two vague messages fell below 0.8. The Decisions API's scores were lower in general. With the same threshold, it would have asked the visitor four times. That includes a clear question about the token allowance and a simple “hello”.

Its one wrong answer sent “I want to cancel” to refund with a confidence of 0.74. A threshold of 0.8 would have turned that into a question, so no visitor would have ended up on the wrong path. A threshold of 0.7 would have let it through. So I do not think one threshold fits every model. Each model needs its own.

Choice or yes/no

The yes/no version of the question did not help. For “this is not working and I paid for it”, it gave a refund a probability of 0.51. That is close to a coin toss, and it tipped the message to refund. For “hello”, it refused to answer. For “I want to cancel”, it gave a refund a probability of 0. The choice question had sent that same message to refund at 87%.

So the way I asked the question changed the answer. It did so on the vague messages, which are the ones that matter most. That is another reason to test a decision model on real messages from your own workflow before you trust its scores.

What changed since September

GPT-5 nano with minimal reasoning got all 30 right in September but only 26 this time, with the same prompt and settings. Three of its mistakes were vague messages. The fourth was “How do I connect my own tool server?”, which it sent to refund. Jev and GPT-4.1 mini got all 30 right again. Jev's two low scores were on the same two vague messages as in September.

With one run per setting, I cannot tell whether GPT-5 nano changed or whether this is normal variation. What it does tell me is that one run is not enough to rely on. I now keep the test as a script, so I can run it again whenever a model or a price changes.

What this means for our workflow

The plan from September has not changed. A decision model will be an optional way to run a condition step. It comes with a confidence threshold and a separate branch for when it is unsure. This test added a few details to that plan.

• The Decisions API is now a second option. It cost the same as Jev and was faster at the median. In this run it was less consistent, and it got one more message wrong.
• Each model gets its own threshold. At 0.8, the Decisions API's mistake became a question, but so did a clear message. Jev's scores left a wider gap between clear and vague messages.
• A refusal goes to the unsure branch. The yes/no question refused “hello”. The workflow should never treat a refusal as its first branch.
• The data question is easier to answer. OpenAI's guide says the Decisions API can keep data in Europe, which I have not tried yet. Jev's API runs in the United States.

The limits from September still apply. The test used 30 English messages and one run per setting, all measured from one place. The Decisions API is also still in beta, so its price and speed may change before it is generally available.

Common questions

Is OpenAI's Decisions API available on OpenRouter?

Not when I tested it on 9 October 2026. OpenRouter offers GPT-6 Luna as a chat model, at $0.10 per million input tokens and $0.50 per million output tokens. The Decisions endpoint is only available through OpenAI's own API, so you need an OpenAI key.

Is the Decisions API more accurate than Jev for intent routing?

Not in this test. Jev got 30 of 30, and the Decisions API got 29 of 30 with a choice question. Both cost about $1.40 per 100,000 decisions. The Decisions API was faster at the median, with 0.18 seconds against 0.24. Jev was more consistent on its slowest calls.

What does a routing decision cost with the Decisions API?

It cost $1.40 per 100,000 decisions in this test, which is about $14 per million. The API only charges for input tokens, at $0.10 per million. Each question here used about 140 of them.

Should I ask a choice or a yes/no question?

For choosing between two paths, the choice question did better in this test. It got 29 of 30 right against 28, and it never refused. The yes/no question gave a coin toss on one vague message and refused another. Whichever you use, treat a refusal as unsure.

Verdict

After two rounds of the same test, this is where I landed.

For the condition step in ChatFrom, I would start with Jev as the decision model. The Decisions API is a close second, and I plan to test it again when it leaves beta. With either one, I will set the threshold for that model. Anything unclear then becomes a question for the visitor.

Cost did not decide this. Every setting that got all 30 right cost between $1.41 and $2.52 per 100,000 decisions, or about $14 to $25 per million. The real difference is what happens when a score is low. A workflow that asks the visitor at that point avoids the wrong path, and that is worth more than any saving on tokens.

ChatFrom lets you build these workflows visually, with a condition step, a branch for each intent and a question when the visitor’s request needs clarification. You can start on the Free plan, with no card.