Every day brings another System One release: a new TypeSafe AI Jev, ai_decide, OpenAI's Decisions API, Kev or OpenJev, each with a leaderboard of scores. They all do the same job: instead of writing an answer, they pick one from options you define and tell you how sure they are. New to System One models? This video walks through the basics and how to get started with Jev.
This excercise started with a simple question someone asked me: when should we use Jev or Databricks' ai_decide? With OpenAI's new Decisions API, it becomes:
Which hosted decision API should we use?
The headline
On easy calls all three are level. On fine-grained tasks, Jev and OpenAI's Decisions API stay level with each other and well ahead of ai_decide. Add examples to the prompt and Jev pulls ahead on the hardest job.
| Triage: broad intent | Triage: exact case | Next action | |
|---|---|---|---|
| Jev | 0.87 | 0.79 | 0.56 |
| Jev + examples | 0.91 | 0.87 | 0.71 |
| ai_decide | 0.86 | 0.70 | 0.39 |
| ai_decide + examples | 0.91 | 0.73 | 0.43 |
| OpenAI Decisions | 0.88 | 0.79 | 0.57 |
| OpenAI Decisions + examples | 0.92 | 0.86 | 0.63 |
Data: accuracy on 300 unseen test chats (the setup)
Scale: always guessing the most common next action scores 0.21
The setup
The data. The Action-Based Conversations Dataset (ABCD; Chen et al., 2021): about 10,000 role-played customer-support chats for a clothing store, with the company's own rules for what agents should do.
Two jobs:
- Triage: what does the customer want? Pick the broad intent (10 options) and the exact case (54 options).
- Next action: given the chat so far, which of the 30 steps in the company playbook should the agent take next?
Three endpoints:
- Jev, TypeSafe AI's hosted API (version 1.13).
- ai_decide, Databricks' built-in decision function, called from SQL.
- OpenAI Decisions, OpenAI's decision endpoint (GPT-6 Luna).
Two ways of asking (zero shot vs few shot):
- Criteria only: one set of questions written from the company rules, with a short explanation for every option. Every endpoint gets exactly the same questions.
- + examples: the same questions, plus the five most similar past chats with their correct answers, picked from a pool of 500 labelled chats. Part 2 explains how they're picked, and whether fine-tuning a small model does better.
Every endpoint answered the same 300 test chats, which none of them had seen before. The code to rerun it all is on GitHub. I used paired significance tests throughout, so when I say one is behind, the gap is real and not noise.
What's inside each one?
Same kind of API, three quite different things underneath:
| What's inside | |
|---|---|
| Jev | A dedicated, closed decision model, trained for exactly this job |
| ai_decide | Not stated by Databricks. It behaves almost exactly like their open OpenJev model, so probably that (more below) |
| OpenAI Decisions | A general small LLM, GPT-6 Luna, constrained to answer only with your options (how it's described) |
What's behind ai_decide?
ai_decide accepts the same requests as Jev's API, so I expected it to be Jev. To check, I sent the same 150 chats to ai_decide, Jev (both versions 1.13 and 1.14) and Databricks' hosted OpenJev model, a small open decision model, and counted how often each pair gave the same answer (the script):
| Same answer on… | Broad intent | Exact case | Return yes/no |
|---|---|---|---|
| Jev 1.13 vs Jev 1.14 (two versions of one model, for scale) | 98% | 93% | 99% |
| ai_decide vs Jev 1.14 | 96% | 81% | 96% |
| ai_decide vs OpenJev | 96% | 89% | 100% |
ai_decide disagrees with Jev more than two versions of Jev disagree with each other, and it's closest to OpenJev, matching it on every yes/no answer with near-identical probabilities. Databricks' own docs back this up: ai_decide runs on Apache 2.0-licensed models served through Databricks, and Jev's weights are closed. So treat ai_decide as Databricks' OpenJev behind Jev's API, not as Jev itself. Databricks doesn't say which model it runs, so that's my conclusion from the test, not an official fact.
Accuracy
- Easy calls are a tie. All three score about the same on broad intent.
- On the fine-grained calls, ai_decide falls behind, and barely gains from examples.
- Examples help Jev most. That makes Jev with examples the clear winner on next action (0.71 vs OpenAI's 0.63).
Speed and cost
| Jev | ai_decide | OpenAI Decisions | |
|---|---|---|---|
| Price | $0.042 per million input tokens, output free | Databricks compute (DBUs) | $0.10 per million input tokens, output free |
| Per 1,000 calls | $0.06–0.10 ($0.09–0.15 with examples) | about $3 one call at a time; at most about $0.55 as one batch query (mostly start-up cost, so bigger batches are cheaper per call) | $0.11–0.20 ($0.17–0.30 with examples) |
| Speed | about 230 ms per call over the internet | 300 chats in 19 s as one batch query | about 300 ms per call over the internet |
| Your data | Sent to TypeSafe AI (no training on it; zero retention for enterprise) | Stays in your Databricks workspace | Sent to OpenAI |
Which to use
| Use… | When |
|---|---|
| Jev | You want the most accurate hosted option, especially for fine-grained choices, and a hosted API is fine |
| ai_decide | Your data is already in Databricks, the choices are broad, and you can run it as batch SQL. Test it on your fine-grained questions first |
| OpenAI Decisions | You're already on OpenAI. It matches plain Jev for about 1.8× the price, but gains less from examples |
Have a few hundred labelled examples? Adding the most similar ones to each prompt is the biggest single win for Jev, and in part 2 a fine-tuned 2B model catches up with it.
Caveats
- One role-played dataset. Some labels are noisy, nothing reaches 100%, and your own data will differ. Treat this as a template for testing, not a verdict.
- Any tuning bias favours Jev. I checked the questions against Jev's mistakes on separate dev chats, not the others'.
- Speed isn't like-for-like. Jev and OpenAI were called over the internet from the UK, ai_decide through Databricks.
- Connection errors. Some ai_decide calls hit a temporary Databricks network error and were re-run, so every chat has exactly one answer.
The code, the full results and every paired test are on GitHub: decision-model-bench (see RESULTS.md).
We cover all the latest AI updates in detail on our YouTube channel, so you can easily find out more information or know where to put your focus. Subscribe here.
References
Chen, D., Chen, H., Yang, Y., Lin, A. and Yu, Z. (2021) 'Action-Based Conversations Dataset: A corpus for building more in-depth task-oriented dialogue systems', Proceedings of NAACL 2021, pp. 3002–3017. Available at: https://aclanthology.org/2021.naacl-main.239
Topics Covered :
Author
Bev Manz