Open Jev research: a month of chasing coverage at 5% error

TypeSafe AI put Jev into limited early access on 15 September 2026. It is a closed “System One” model: you send it a state and a list of typed questions, and it returns a probability for every option instead of generating text. The open clones already matched its accuracy, but Kev’s README reports that the best of them, Kev-9B, can answer only 0.45 to 0.57 of decisions automatically at a 5% error rate, against 0.70 for Jev. Between 23 September and 12 October I worked on that gap with open models and a single 16 GB RTX 4070 Ti Super, and when the answer for my own use turned out to be “use the models that already exist”, I asked whether decisions of that quality could run on a CPU.

None of the models I built beat the released ones on the metric the series is about. On the same 1,046-question general set, every 4B to 9B model I scored lands between 0.550 and 0.557 coverage at 5% error: Kev-4B, decider-4b, Cloudflare’s Clef-flash, and my llama.cpp port of Kev-4B. Every attempt to get a smaller model there (a raw MoE screen, Kev’s training recipe on Granite-H-350M, distilling Kev-4B into Kev-0.8B) finished at or below Kev-0.8B’s 0.276. Along the way I built a decision router that serves these models to my agent harness, an adversarial panel for shell-command risk, and three small public models that you can try side by side in a demo Space. This post is the hub for the series: what Jev is, the one metric every post uses, how the month went, and a map of the results.

Jev and its open copies

Jev exposes one endpoint, POST /v1/systemone, which takes a state (text or JSON) and a list of questions. There are three question types: Choice picks one of up to 255 options and returns per-option probabilities, Score rates against ordered levels, and Noul returns a yes/no probability. TypeSafe says the model is a transformer trained only on synthetic data and post-trained with “Reinforcement Learning for Calibrated Decisions”, and it has published no paper, weights or parameter count. Its headline figures (193.6 times faster, 444.6 times cheaper) come from TypeSafe’s own workflow evals against the average of two frontier LLMs, while independent tests found about 2.9 times the speed and 12 times lower cost. Archer Hume’s black-box probing suggests the likeliest design: encode the shared state once, run each question as its own branch over that prefix, and read the answer straight off the option logits.

Two open projects speak the same API. Kev, by Jared Palmer, ships at 0.5B, 0.8B, 4B and 9B, and Laya, mostly written by Nandakishor of Convai Innovations (Apache-2.0), ships a server that speaks Jev’s wire format. Two more arrived during the month: decider-4b (Mapika, Apache-2.0) and Cloudflare’s Clef-flash (9B, post-trained from Qwen3.5-9B, Apache-2.0, released 1 October). The landscape post covers how these differ.

The metric: coverage at 5% error

A decision model is useful in an automated pipeline only if you can trust the answers it is sure of and send the rest to something slower. That is selective prediction: the model answers when it is confident and abstains otherwise. Coverage at 5% error measures how much it can answer under that rule.

For one task, take every question, read the model’s confidence as the probability of its top option, and sort the questions from most to least confident. Accept answers from the top of the list. Coverage is the largest fraction of the task you can accept while the error rate among the accepted answers stays at or under 5%. A task where the 60 most confident of 100 answers contain three mistakes covers at least 0.60. Questions with the same confidence are accepted or rejected together, and a question the model failed to answer is never accepted but still counts in the total. This is coverage_at_error in Kev’s kev/metrics.py, which forge’s scorer reuses per task. The series reports the macro figure: coverage computed per task, then averaged over tasks with equal weight.

The metric rewards how well confidence is ordered, and that often disagrees with accuracy. Laya is more accurate than Kev-4B on the emotion task (0.613 against 0.550) yet covers nothing there, because its most confident answers are wrong. A raw Qwen3-4B-Instruct readout answered 72% of the general set correctly and covered 0.091, because its confidence does not separate right answers from wrong ones.

Most numbers in the series come from one instrument, the general set: the development split of Kev’s transfer-v9 evaluation, 1,046 questions over 11 tasks that mix public benchmarks (mmlu, sciq, qnli, emotion, paws) with probes for buried evidence, rule composition and unknowable questions. The bar for the month was about 0.55, where Kev-4B (0.555) and Clef-flash (0.550) sit.

How the month went

The first four days, 23 to 26 September, went into building my own system. After surveying the open clones I chose to build forge, a synthetic decision-data engine and coverage-first evaluation, with Kev’s trainer as the first training backend. forge reproduced Kev’s trainer bit for bit, a leakage audit caught the engine giving answers away through look-alike sentences, and a pilot on long buried documents was registered at about 40 GPU-hours.

From 27 September the work turned toward real decisions. Loom, my agent harness, asked whether a decision model could take calls like command risk and routing off its LLM. That became verdict, a service that answers Loom over the System One API. On 1 October an experiment called K1 found that sending each task to whichever backend covers it best would gain only 1.2 points of coverage over Kev-4B alone, against a 5-point bar, and since a pipeline keeps long context in the LLM and asks the decision model short questions, the long-document pilot was solving a problem nobody had. forge was parked, and Kev and Laya became the models.

On 2 and 3 October I built a command-risk panel from public security datasets, checked a public leaderboard’s ranking against my own panel, and turned verdict into a router once Clef-flash came out.

On 5 October the goal changed to decisions near Clef-flash quality on a CPU ladder, from a Raspberry Pi 5 with 8 GB up to a laptop x86 CPU. A single-pass decision is a prefill, which is compute-bound, so its cost tracks active parameters times tokens and offloading or streaming weights does not make it faster. I screened small bases, trained Kev’s recipe on one, distilled Kev-4B into Kev-0.8B, published the three resulting models, and ported Kev to llama.cpp.

Results map

The posts go up one at a time, so titles without a link are not out yet.

#	Post	Headline result
2	The open Jev landscape	JevBench ranks Kev-4B 37th, but on our panel decider-4b only ties it (0.556 against 0.555 coverage)
3	forge, a synthetic decision-data engine	Replays Kev’s trainer bit for bit (all 314 loss windows and 392 adapter tensors identical); K1’s +1.2 pp against a 5 pp bar parked it
4	The command_risk panel and its source leak	A character n-gram logistic regression scores 0.965 AUROC, against 0.798 for Clef-flash and 0.776 for Kev
5	verdict, a decision router	Clef-flash ties Kev-4B on false alarms at full risky recall (60.5% against 61.8% on test) and now heads every chain
6	Small decision models on a CPU ladder	Kev’s recipe on Granite-H-350M reaches 0.132 coverage against Kev-0.8B’s 0.276
7	Distillation buys accuracy, not coverage	Soft targets add 2.4 pp accuracy over gold labels; coverage stays at 0.250 against a 0.33 kill line
8	Kev on llama.cpp	Kev-4B at Q8 covers 0.557 against 0.555 in torch, at about 4.5 GB instead of 18 GB, with no speed gain

The chart puts every general-set number from the series on one axis. The 4B to 9B models are within 0.007 of each other however they were trained or served, and below 1B nothing I tried, whether a different base, more teacher-labelled data or Kev’s full recipe, moved coverage above the released Kev-0.8B. The distillation result explains part of that: the teacher’s probabilities carried answer knowledge but not a better ordering of confidence, and ordering is what this metric measures.

Jev’s 0.70 was reported on Kev’s benchmark, not on the general set, so none of these posts measures the gap to Jev directly. What the month did establish is that the open 4B to 9B models agree at about 0.55, and that getting a sub-1B model near that number is a confidence-ranking problem the methods I tried did not solve.