Saltar al contenido

Por ahora, este artículo solo está en inglés.

Testing OpenAI's Decisions API for chatbot intent routing

I ran my 30-message routing test again with OpenAI's new Decisions API, Jev, GPT-6 Luna and three other models, for accuracy, speed, cost and confidence.

Por , FundadorActualizado el 9 min de lectura
The cover shows the title, Testing OpenAI's Decisions API, with the line against Jev and four LLMs below it, on a grey-cream background with fine crossing lines.

OpenAI's Decisions API got 29 of my 30 routing messages right, and at 0.18 seconds from Germany it was the fastest setting I have measured so far. Jev got all 30 right and was more consistent on its slower calls, and the two cost the same, $1.40 and $1.41 per 100,000 decisions. Both cost about 45% less than GPT-4.1 mini, which also got all 30 right but needed 0.75 seconds.

OpenAI's Decisions API was in public beta when I ran this test, and it is now generally available. Like Jev, it does not write an answer. You give it a typed question, and it returns a choice with a probability for each option. I had already tested Jev and six LLMs on the same small routing task, so I wanted to see how the new API compares. I ran the same 30 messages through it, through Jev again and through four LLMs, which were GPT-6 Luna as a chat model, GPT-4.1 mini, GPT-5 nano and Gemini 3.1 Flash-Lite.

As in my first test, the most useful part for my workflow was the confidence score. Jev gave low scores to the two vague messages and high scores to everything else. The Decisions API gave lower scores in general. Its one mistake had a score low enough for a threshold to catch, but only if the threshold is chosen for that model.

What the Decisions API is

The Decisions API runs on GPT-6 Luna, which OpenAI calls its most efficient model for focused, high-volume tasks. It has its own endpoint. You send an input and a list of named questions. Each question can be a yes/no condition, a choice between options or a score on a scale. You only pay for input tokens, at $0.10 per million. Output is free.

The API is not on OpenRouter. OpenRouter offers GPT-6 Luna as a chat model, but not the Decisions endpoint. So I called OpenAI directly with my own key.

RequestHTTP
POST https://api.openai.com/v1/decisions
{
  "model": "gpt-6-luna",
  "input": "this is not working and I paid for it",
  "questions": [
    {
      "type": "choice",
      "name": "route",
      "instructions": "What does the visitor want: a refund, or help from customer support?",
      "choices": [
        { "value": "refund", "description": "The visitor wants money back" },
        { "value": "support", "description": "The visitor needs help from customer support" }
      ]
    }
  ]
}
Its answer for that message, trimmedJSON
{
  "answers": [
    {
      "type": "choice",
      "name": "route",
      "choice": "support",
      "confidence": 0.72,
      "probabilities": [
        { "value": "refund", "probability": 0.14 },
        { "value": "support", "probability": 0.86 }
      ]
    }
  ]
}

For this vague message, the Decisions API chose support. It split the message 86 to 14 and reported a confidence of 0.72. Jev chose support as well, but it was much less sure. It split the message 65 to 35 and reported a confidence of 0.30.

The same test, run a second time

I used the same 30 messages as in the first test. There are 12 clear refund requests, 12 clear support requests and 6 vague messages. Each one is labelled with the branch I would choose. The LLMs received the same one-label prompt as before, and I read their replies with the same parser as our workflow step. That way, each result is the branch the workflow would actually take.

  • GPT-6 Luna through the Decisions API got the question twice, once as a choice and once as a yes/no question that asks whether the visitor wants a refund. A developer on OpenAI's forum found its yes/no answers better calibrated than its choices, so I tested both.
  • Jev ran through its own API, with the same choice question as in the first test.
  • GPT-6 Luna as a chat model ran through OpenRouter, once at its default reasoning and once at minimal.
  • GPT-4.1 mini and GPT-5 nano with minimal reasoning were the two LLMs that got all 30 right in the first test.
  • Gemini 3.1 Flash-Lite is a small Google model I had not tried before.

I wanted to include Claude Haiku 5.5 as well, which had just come out. My OpenRouter key's guardrail blocked it, so it is not in the results. Claude Haiku 5.5 and GPT-6 Luna both appear in my test of six models on 900 support answers, which looks at writing answers rather than routing. The table below shows the published prices on the day of the test.

ModelInput, per million tokensOutput, per million tokens
Jev$0.042Free
GPT-5 nano$0.05$0.40 (reasoning tokens are billed as output)
GPT-6 Luna, Decisions API$0.10Free
GPT-6 Luna, chat$0.10$0.50
Gemini 3.1 Flash-Lite$0.25$1.50
GPT-4.1 mini$0.40$1.60

I measured everything from Germany and made one call at a time, as in the first test. This time, each setting got one extra call at the start that I did not count. That call opened the connection. The 30 measured calls then ran on an open connection, just as they would on a server. For the LLMs, the cost is what OpenRouter billed. For the two decision APIs, it is the input tokens multiplied by the published price.

Results

SettingCorrectMedian timeSlowest 5%Cost per 100,000
Jev30 of 300.24 s0.27 s$1.41
GPT-4.1 mini30 of 300.75 s0.89 s$2.52
GPT-6 Luna, chat, minimal reasoning30 of 301.30 s2.88 s$1.71
GPT-6 Luna, chat, default reasoning30 of 301.88 s2.36 s$1.78
GPT-6 Luna, Decisions API, choice29 of 300.18 s0.90 s$1.40
GPT-6 Luna, Decisions API, yes/no28 of 300.18 s0.73 s$1.66
Gemini 3.1 Flash-Lite27 of 3010.47 s10.57 s$1.34
GPT-5 nano, minimal reasoning26 of 300.95 s1.10 s$1.03

Jev, GPT-4.1 mini and GPT-6 Luna as a chat model at both reasoning levels got all 30 messages right. The Decisions API missed one message with the choice question and two with the yes/no question. Gemini 3.1 Flash-Lite sent three of the vague messages to refund. GPT-5 nano dropped from 30 in the first test to 26, even though the prompt and settings were the same. I come back to that further down.

Speed

The Decisions API and Jev made the fastest decisions, at 0.18 and 0.24 seconds.
Each bar is the median of 30 calls, measured from Germany on an open connection. I left Gemini 3.1 Flash-Lite out, for the reason given below the chart.
JevOpenAI Decisions APIOpenAI chat models
GPT-6 Luna, Decisions API, choice
0.18 s
GPT-6 Luna, Decisions API, yes/no
0.18 s
Jev
0.24 s
GPT-4.1 mini
0.75 s
GPT-5 nano, minimal reasoning
0.95 s
GPT-6 Luna, chat, minimal reasoning
1.30 s
GPT-6 Luna, chat, default reasoning
1.88 s
  • Jev and the Decisions API were three to ten times faster than the chat models. The Decisions API had a median of 0.18 seconds and Jev had one of 0.24 seconds. GPT-4.1 mini, GPT-5 nano and GPT-6 Luna as a chat model needed between 0.75 and 1.88 seconds.
  • Jev was more consistent. Its slowest calls took 0.27 seconds. The Decisions API's slowest calls took 0.90 seconds, and its times ranged from 0.09 to 1.30 seconds.
  • Minimal reasoning made little difference for GPT-6 Luna as a chat model. It still used about 16 reasoning tokens to return a single word, compared with 18 at its default setting. The median time dropped from 1.88 to 1.30 seconds, and accuracy stayed the same.
  • Gemini 3.1 Flash-Lite's times probably say more about the provider than the model. Four calls came back in under a second, while 25 took about 10.5 seconds each. A delay that regular points to the provider OpenRouter picked, so I left it out of the chart.

Cost

Every setting cost between $1.03 and $2.52 per 100,000 decisions. That is a much smaller range than in the first test, where GPT-5 nano and DeepSeek V4 Pro with their default reasoning cost more than $7.

In US dollars, GPT-5 nano cost the least per 100,000 decisions, and GPT-4.1 mini cost the most.
The chat models' cost is what OpenRouter billed, and the cost of Jev and the Decisions API is their input tokens times the published price. A shorter bar costs less.
JevOpenAI Decisions APIOpenAI chat modelsGoogle
GPT-5 nano, minimal reasoning
$1.03
Gemini 3.1 Flash-Lite
$1.34
GPT-6 Luna, Decisions API, choice
$1.40
Jev
$1.41
GPT-6 Luna, Decisions API, yes/no
$1.66
GPT-6 Luna, chat, minimal reasoning
$1.71
GPT-6 Luna, chat, default reasoning
$1.78
GPT-4.1 mini
$2.52
SettingCorrectPer 100,000 decisionsPer million decisions
GPT-5 nano, minimal reasoning26 of 30$1.03$10.30
Gemini 3.1 Flash-Lite27 of 30$1.34$13.38
GPT-6 Luna, Decisions API, choice29 of 30$1.40$14.04
Jev30 of 30$1.41$14.10
GPT-6 Luna, Decisions API, yes/no28 of 30$1.66$16.61
GPT-6 Luna, chat, minimal reasoning30 of 30$1.71$17.11
GPT-6 Luna, chat, default reasoning30 of 30$1.78$17.76
GPT-4.1 mini30 of 30$2.52$25.23
  • The Decisions API also bills its question template. It counted about 140 input tokens per decision. The LLM prompt used 54, and Jev counted about 334. The Decisions API charges more per token, while Jev counts more tokens. In the end, the two cost the same.
  • Neither Jev nor the Decisions API charges for output. The cost depends only on the length of the request, which makes it easy to predict.
  • The same model cost more as a chat model. GPT-6 Luna cost $1.71 through OpenRouter and $1.40 through the Decisions API. As a chat model, it wrote about 24 tokens to return a single word. Most of those were reasoning tokens.
  • The yes/no question cost more than the choice. Its template is longer, so it cost $1.66 instead of $1.40.

At a million decisions a month, Jev or the Decisions API would cost about $14, and GPT-4.1 mini would cost about $25. But as in the first test, the bigger cost is a visitor who gets sent down the wrong path. That cost does not show up on any price list.

Confidence

Jev and the Decisions API both return a confidence score with every choice. The table shows the messages where either of them scored below 0.8, which is the threshold I used in the first test.

MessageJevDecisions API, choice
“this is not working and I paid for it”support, 0.30support, 0.72
“I want to cancel”support, 0.60refund, 0.74 (my label: support)
“hello”support, 1.00support, 0.72
“Can you explain how the token allowance is counted?”support, 1.00support, 0.78

Jev's scores worked the way I hoped, again. Every clear message scored 0.98 or higher, and only the two vague messages fell below 0.8. The Decisions API's scores were lower in general. With the same threshold, it would have asked the visitor four times. That includes a clear question about the token allowance and a simple “hello”.

Its one wrong answer sent “I want to cancel” to refund with a confidence of 0.74. A threshold of 0.8 would have turned that into a question, so no visitor would have ended up on the wrong path. A threshold of 0.7 would have let it through. So I do not think one threshold fits every model. Each model needs its own.

Choice or yes/no

The yes/no version of the question did not help. For “this is not working and I paid for it”, it gave a refund a probability of 0.51. That is close to a coin toss, and it tipped the message to refund. For “hello”, it refused to answer. For “I want to cancel”, it gave a refund a probability of 0. The choice question had sent that same message to refund at 87%.

So the way I asked the question changed the answer. It did so on the vague messages, which are the ones that matter most. That is another reason to test a decision model on real messages from your own workflow before you trust its scores.

What changed since the first test

GPT-5 nano with minimal reasoning got all 30 right in the first test but only 26 this time, with the same prompt and settings. Three of its mistakes were vague messages. The fourth was “How do I connect my own tool server?”, which it sent to refund. Jev and GPT-4.1 mini got all 30 right again. Jev's two low scores were on the same two vague messages as in the first test.

With one run per setting, I cannot tell whether GPT-5 nano changed or whether this is normal variation. What it does tell me is that one run is not enough to rely on. I now keep the test as a script, so I can run it again whenever a model or a price changes. OpenAI has since marked GPT-5 nano and GPT-4.1 mini as deprecated and plans to remove both from its API, and it suggests GPT-6 Luna in place of GPT-4.1 mini.

What this means for our workflow

The plan from the first test has not changed. A decision model will be an optional way to run a Decide step. It comes with a confidence threshold and a separate branch for when it is unsure. This test added a few details to that plan.

  • The Decisions API is now a second option. It cost the same as Jev and was faster at the median. In this run it was less consistent, and it got one more message wrong.
  • Each model gets its own threshold. At 0.8, the Decisions API's mistake became a question, but so did a clear message. Jev's scores left a wider gap between clear and vague messages.
  • A refusal goes to the unsure branch. The yes/no question refused “hello”. The workflow should never treat a refusal as its first branch.
  • The data question is easier to answer. OpenAI's guide says the Decisions API can process data in Europe, at a higher price, which I have not tried yet. Jev's API runs in the United States.

The limits from the first test still apply. The test used 30 English messages and one run per setting, all measured from one place. The Decisions API has also left beta since I ran the test. Its price per input token is the same, but its speed may have changed.

Common questions

Is OpenAI's Decisions API available on OpenRouter?

No, the Decisions API is not on OpenRouter, which offers GPT-6 Luna only as a chat model, at $0.10 per million input tokens and $0.50 per million output tokens. The Decisions endpoint is only available through OpenAI's own API, so you need an OpenAI key.

Is the Decisions API more accurate than Jev for intent routing?

No, it was slightly less accurate than Jev in this test. Jev got 30 of 30, and the Decisions API got 29 of 30 with a choice question. Both cost about $1.40 per 100,000 decisions. The Decisions API was faster at the median, with 0.18 seconds against 0.24. Jev was more consistent on its slowest calls.

What does a routing decision cost with the Decisions API?

It cost $1.40 per 100,000 decisions in this test, which is about $14 per million. The API only charges for input tokens, at $0.10 per million, and that price stayed the same when it left beta. Each question here used about 140 of them.

Should I ask a choice or a yes/no question?

For choosing between two paths, the choice question did better in this test. It got 29 of 30 right against 28, and it never refused. The yes/no question gave a coin toss on one vague message and refused another. Whichever you use, treat a refusal as unsure.

Verdict

After two rounds of the same test, this is where I landed.

If you wantUseWhy
Accurate routing and a confidence score you can act onJevIt got 30 of 30 in 0.24 seconds, at $1.41 per 100,000. Its scores flagged the two vague messages and nothing else.
The fastest answerOpenAI's Decisions API, with a choice questionIt answered in 0.18 seconds at the median, at $1.40 per 100,000. It got 29 of 30, and its threshold needs tuning for this model.
An LLM you already useGPT-4.1 mini or GPT-6 Luna as a chat modelBoth got 30 of 30. Luna was cheaper at $1.71 against $2.52, but slower at 1.30 seconds against 0.75, and OpenAI plans to retire GPT-4.1 mini.
The lowest priceGPT-5 nano or Gemini 3.1 Flash-Lite, with careGPT-5 nano and Gemini 3.1 Flash-Lite cost $1.03 and $1.34 per 100,000, but they missed four and three messages in this run, and OpenAI plans to retire GPT-5 nano.

For the Decide step in ChatFrom, I would start with Jev as the decision model. The Decisions API is a close second, and now that it has left beta, I plan to test it again. With either one, I will set the threshold for that model. Anything unclear then becomes a question for the visitor.

Cost did not decide this. Jev, GPT-6 Luna as a chat model and GPT-4.1 mini, the settings that got all 30 right, cost between $1.41 and $2.52 per 100,000 decisions, or about $14 to $25 per million. The real difference is what happens when a score is low. A workflow that asks the visitor at that point avoids the wrong path, and that is worth more than any saving on tokens.


ChatFrom lets you build these workflows visually, with a Decide step, a branch for each intent and a question when the visitor’s request needs clarification. The subscriptions workflow is one example, where a Decide step sends a cancellation and a billing question down different paths. You can start on the free Hobby plan with 50 credits a month and no card, and see what Pro at $29 a month and Business at $49 a month include on the pricing page, or how ChatFrom compares with other chatbot builders on the compare page.