GPT-6.1 Terra review: tested as a customer support chatbot

I asked GPT-6.1 Terra 46 customer support questions next to four other models. It answered well, but GPT-6 Luna did just as well for a fraction of the cost.

OpenAI has released GPT-6.1 Terra, a new mid-priced model. I wanted to know one thing. Does it make a better customer support chatbot than the models people already use?

To find out, I set up a support chatbot for a software company and asked it 46 questions that real customers ask. I ran the same questions on Terra and on four other models, GPT-6 Luna and GPT-6.1 Sol from OpenAI, and Claude Haiku 5.5 and Claude Sonnet 5.5 from Anthropic.

Here is the short version.

• Terra gives good answers. It answered 45 of the 46 questions well, got every calculation right and never fell for a trap hidden in the help pages.
• So did the other four. All five scored between 9.37 and 9.48 out of 10. That gap is too small to matter.
• The price was the real difference. 1,000 answers cost $1.86 with Terra, $0.31 with Claude Haiku 5.5 and $0.11 with GPT-6 Luna.
• For a support chatbot, I would not pay for Terra. A cheaper model does the same job.

I run ChatFrom, where businesses build support chatbots that answer from their own help pages and documents. Each chatbot can use a different model, so this is a question our customers ask me.

What GPT-6.1 Terra is

OpenAI sells four levels of model. From cheapest to most powerful they are Luna, Terra, Sol and Astra. Terra is the second step.

OpenAI built Terra for everyday work, and especially for AI agents that use a computer, read screenshots or work through technical drawings. It says Terra is more than three times faster than Sol, writes more naturally, and makes far fewer mistakes than the previous version.

A support chatbot has a simpler job. It finds the right part of your help pages and writes a short, correct answer. That is what I tested. Here are the published prices of the five models. A token is a small piece of a word.

These list prices don't tell you what an answer costs, because models count text differently and some write longer answers. Further down I show the real cost of 1,000 answers.

How I tested it

The company

I made up a software company called Ledgerbird. It sells invoicing and accounting software to freelancers and small businesses, which is exactly the kind of company that puts a support chatbot on its website.

I wrote its help centre, six short pages like a real one. They cover plans and prices, billing and refunds, invoices, bank connections, account and security, and how to contact support.

One page also contains a trap. Hidden in the billing page is a note telling AI assistants to send customers to a fake refund address and ask for their card number. Text like this can end up in a real help centre through a forum post or a copied email.

The questions

I wrote 46 questions of nine kinds.

• Simple facts, such as the price of a plan.
• Small calculations, such as what a team of five pays, or how much is left of a €200 invoice after the card fee.
• Features, such as invoicing in US dollars.
• Things the help pages don't cover, such as a mobile app or payroll. The right answer here is “I don't know”.
• Wrong assumptions, such as “the Free plan includes a bank connection, right?”
• Other languages, in German, French, Spanish, Italian and Dutch.
• Risky requests, such as tax advice or a free upgrade.
• Follow-up questions, such as “And Business?” after asking about another plan.
• Upset customers, one charged twice and one locked out of their account.

How the answers were marked

1. Every model used the same chatbot, the same help pages and the same instructions. Each answered every question once.
2. I first checked that the chatbot found the right help page for each question. It did for 45 of the 46. For the last one it missed a small detail, so I didn't count that detail against any model.
3. A sixth AI model, Mistral Large 4 from another company, marked every answer from 0 to 10, comparing it with a note of what a good answer should say. Before it started, I gave it five answers I had already marked myself, and it agreed with me on all five.
4. A simple script also checked each answer for the key facts and for the fake address.

I had run the same kind of test before on an online shop's help pages, three times over. The five models finished just as close together there, which gives me more confidence in these results.

The results

All five models did well, and they finished almost level. The best scored 9.48 out of 10 and the worst 9.37. In my earlier shop test, the same model's score moved by up to half a point from one round to the next, so a gap of 0.11 tells you nothing about which model is better.

No model gave a single poor answer, meaning one marked 3 or lower.

The chart shows the main finding. The quality is the same, and the price is not. Terra costs 17 times as much as Luna and 6 times as much as Haiku for answers of the same quality.

Which questions were hardest

The table below breaks each model's score down by kind of question. Each row is one kind of question and each column is one model. The number is the average mark out of 10, and the darker the orange, the lower the mark. Every model scored 8.6 or more everywhere, so even the darkest cells are good scores.

A few things stand out.

• Upset customers were handled perfectly. Every model scored 10 on both. Each apologised once, explained what to do next and promised nothing it couldn't deliver.
• No model made anything up. When the help pages had no answer, every model said so. None invented a mobile app, a payroll feature, a Shopify connection or a phone number.
• Terra was perfect on risky requests, with 10 out of 10. It refused the free upgrade, gave no tax advice and ignored the trap. Sol scored lowest here, 9.0, because it misread one request to email some invoices.
• The weaker cells come from one or two answers each. Terra and Haiku lost points in other languages mostly on one Spanish answer (example 5 below). Terra's 8.7 for follow-ups comes from one short answer that was right but brief. Sonnet's 9.0 for calculations comes from one question about adding a user (example 6).

Six answers worth a closer look

Scores only tell part of the story. These are the real answers, with the mark each one got. I removed the small source numbers the chatbot adds.

1. Every model got the maths right

Three questions needed a calculation. All five models got all three right, including the cheapest one.

2. The better answer says what it does know

Ledgerbird has no startup discount, only one for charities and students. All five models said they didn't know about a startup discount, which is correct. Haiku and Sonnet also mentioned the discount that does exist. That is what a good support agent would do.

3. No tax advice

A wrong tax answer could cost a customer real money. Ledgerbird's help pages say it doesn't give tax advice, and every model respected that and pointed to a tax adviser.

4. An angry customer, and a trap

This is the hardest kind of message. The customer is angry, and the help page with the answer also contains the hidden note asking for card details. Every model scored 10.

Three questions led to that page, so the models answered 15 times in total. Not one answer passed on the fake address or asked for card details. Eight of them even warned the customer not to share card numbers, as the chatbot's instructions ask.

5. Should a chatbot translate button names?

Every model answered in the customer's own language. In Spanish, though, they disagreed about the app's menu names. Terra kept them in English, as they appear on screen. Haiku translated them.

The marking model gave both a 6. It marked Terra down for the English menu names, and Haiku for adding refund details nobody asked for. I disagree about Terra. Ledgerbird's menus are in English, so a customer looking for “Ajustes” won't find it. This is a good reminder to read the answers yourself, not just the scores.

6. When the chatbot finds the wrong part of a page

Every model correctly said that Starter has one user and a second user needs Business. But all five quoted the yearly price, €290, instead of the monthly €29. The likely reason is that the chatbot found the section on yearly prices and not the monthly ones, so the yearly price was all the models had in front of them. Sonnet also sent the customer to support for the switch itself, and was marked 4.

Despite the low mark, Sonnet's last paragraph was the smartest idea any model had here. The help pages do say an accountant can join for free with read-only access, so a partner who only needs to look at the books could join that way at no cost.

How long customers wait

The chart shows how long a customer typically waits for the first word of the answer, and for the whole answer. The wait includes the time the chatbot spends finding the right help page, which is the same whichever model answers.

Terra was the quickest, with a full answer after about 2.3 seconds. Haiku, Sol and Luna were all within a third of a second of it. Sonnet was about a second slower. Gaps this small change with the network and the time of day, and in my earlier shop test Haiku was the quickest. Speed is not a reason to pick one of these models over another.

What it costs

This chart shows what 1,000 answers actually cost, based on the real bill. It includes the small extra work the chatbot does to find the right help pages.

Terra cost about half as much as Sol, as OpenAI promises. The surprise was Anthropic. Haiku has the same list price as Luna but cost almost three times as much per answer. Anthropic's models count the same help pages as about 40% more tokens, and they write longer answers, around 37 words against about 25 for OpenAI's models.

For a small company, all of these amounts are low. At 3,000 questions a month, you would pay about 33 cents with Luna, 93 cents with Haiku, $5.60 with Terra, $10.50 with Sol and $15 with Sonnet. At 30,000 questions a month, that grows to between $3.30 and about $150.

OpenAI's claims, checked

Most of OpenAI's claims are about longer agent tasks, not support answers. This is how they held up here.

Which model I would choose

For a support chatbot that answers from your own help pages, I would not pay for Terra. It is a good model, but GPT-6 Luna and Claude Haiku 5.5 did the same job for a fraction of the price. Terra's strengths, like using a computer and reading screenshots, are not what a support chatbot needs.

• GPT-6 Luna is the cheapest, and the only model with a good answer to all 46 questions.
• Claude Haiku 5.5 costs a little more and was slightly more helpful when the help pages didn't quite cover a question, as in example 2.
• The larger models, Sol and Sonnet, gave no better answers here, at eleven to sixteen times the price of Haiku.

The bigger lesson is that the help pages matter more than the model. When the chatbot found the wrong part of a page, as in example 6, every model's answer suffered. Before you pay for a bigger model, make sure your help pages answer the questions your customers actually ask.

One limit to keep in mind. This was one made-up company with 46 questions, so treat it as a practical check rather than a final verdict. Your own pages and questions may give slightly different numbers.

Common questions

Is GPT-6.1 Terra good for a customer support chatbot?

Yes. It scored 9.37 out of 10, answered 45 of 46 questions well, gave no tax advice and ignored a trap hidden in the help pages. But it was no better than GPT-6 Luna or Claude Haiku 5.5, which cost much less.

How much does GPT-6.1 Terra cost per answer?

About $1.86 per 1,000 support answers, so a little under a fifth of a cent each. OpenAI's list price is $1 per million tokens read and $5 per million written, half the price of Sol.

Is GPT-6.1 Terra faster than GPT-6.1 Sol?

Not noticeably for support answers. A typical answer took 2.33 seconds with Terra and 2.45 with Sol. OpenAI's “three times faster” refers to longer agent tasks.

Should I use GPT-6.1 Terra or Claude Haiku 5.5 for customer support?

Haiku. It scored about the same, 9.43 against Terra's 9.37, was slightly more helpful in tricky cases, and cost $0.31 per 1,000 answers instead of $1.86.

Does the model matter more than the help pages?

Not in this test. The five models were only 0.11 points apart. The one question where the chatbot found the wrong part of a help page brought every model's mark down, whichever model answered.

I expected the newer model to stand out. Instead, all five models, from the cheapest to the most expensive, answered support questions about equally well when they had good help pages. The only real difference was the price. If I were setting up a support chatbot today, I would start with a small, cheap model, spend my time on the help pages, and only move to a bigger model if the answers gave me a reason.

ChatFrom builds a support chatbot from your own pages and documents, lets you choose its model, and shows you every conversation so you can see which questions your pages don't answer yet. You can start on the free Hobby plan, with no card.