The open Jev clones: three ways to read a decision, and a leaderboard that does not transfer

Within a week of TypeSafe AI opening early access to Jev, its closed “System One” decision model, GitHub was full of open look-alikes, and by October an independent board, JevBench, was ranking more than a hundred systems on sealed items. I needed one of these models to answer decisions for Loom, my agent harness, so the question for this post is practical: which open clone should sit behind that endpoint, and can I trust the public ranking to tell me?

The clones split into three mechanism families, and the two I studied most closely, Kev and Laya, sit in different ones. JevBench ranks Kev-4B 37th with a score of 39.9, behind decider-4b at 55.8 and behind a stock Qwen3-4B-Instruct read at its option logits at 54.1. When I scored those models on my own panel, by coverage at 5% error (the share of questions a model can answer on its own while keeping its error on those under 5%; defined in the series overview), decider-4b tied Kev-4B at 0.556 against 0.555, and the stock Qwen readout covered 0.091. The ranking did not transfer, and Kev-4B stayed the pinned model.

What the clones are copying

Jev takes a state (text or JSON) and a list of typed questions, and returns probabilities instead of text. A Choice question picks one of up to 255 options and returns a probability per option, a Score question rates against ordered levels, and a Noul returns a yes/no probability. TypeSafe says it is a transformer trained on synthetic data and post-trained with “Reinforcement Learning for Calibrated Decisions” (RLCD); there is no paper, no weights and no parameter count.

The most useful public guess comes from Archer Hume’s black-box probing of roughly 10,000 API calls. Latency tracks input tokens rather than option count, and content injected into one question does not leak into another, which fits a causal transformer that encodes the state once and runs each question as an isolated branch over that shared prefix (the pattern from Hydragen and DeFT), ending in a softmax over option logits with no text generated. A minority view (djev) holds that Jev is a diffusion model; Hume argues diffusion is not needed to explain what the API does.

Three ways to read a decision

Every open clone has to answer the same two questions: what hidden state does the decision head read, and how does that become a distribution over options? The answers fall into three families.

Pointer head on a causal LLM (Kev by Jared Palmer, and TinyJev). Kev packs one sequence per request: <state> ... <q> instruction <opt> option1 </opt> <opt> option2 </opt> ... <decide>, with one <q> ... <decide> block per question. The delimiters reuse rarely used Qwen special tokens, so no embeddings are resized. A block-causal mask lets each question’s tokens attend to the state and to their own question block only, and position ids restart after the state for every question, so questions share the state but cannot read each other. The head has no vocabulary readout at all: two linear projections map the <decide> hidden state and each option’s </opt> hidden state into a 256-dimensional space, the scaled dot products become one logit per option, and a softmax over options gives the distribution. The base (Qwen3.5 at 0.8B, 4B or 9B) is frozen with a rank-16 LoRA adapter. Qwen3.5 interleaves Gated DeltaNet layers, which are recurrent and cannot honor a block mask, so on those bases each question runs as its own causal row continuing a cached state prefix, and a cache miss costs one forward pass instead of one per question. Calibration is a single post-hoc temperature per checkpoint (2.41 for the Kev-4B I pin).

Mask-marker encoder (Laya by Nandakishor Mukkunnoth of Convai Innovations). Laya is an encoder, ModernBERT-large at 421M parameters. Each question becomes one sequence, [CLS] <qtype> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP], with a [MASK] token inserted before every option. A two-layer transformer head runs over the encoder output, and a small shared scorer turns the hidden state at each [MASK] position into one logit per option; a softmax over those, with a temperature per question type and option-count bucket, gives the probabilities. All of a request’s questions are padded into one batch and answered in one call. Because every option has to fit in a fixed token budget, the option count is capped: on Banking77’s 77 labels each option gets about 3 to 4 tokens. Training uses a strictly proper scoring rule (log score plus spherical score, plus a ranked probability term for ordinal questions) as a policy-gradient reward, which Laya’s documentation calls RLCD.

First-token logit readout (SemIf, formerly OpenJev, and jevfire). These need no training. The options are written as single tokens (letters A, B, C and so on), a stock LLM runs one forward pass, and the decision is a softmax over the logits of those candidate tokens at the first answer position, the same trick as FIRST (arXiv 2406.15657) for listwise reranking. jevfire adds shared-prefix KV in vLLM so many questions over one state stay cheap. Two failure modes recur: ambiguous single-letter tokens (space-prefixed against byte-level) can silently corrupt the readout, and letter-slot variants run out of single tokens at about 26 to 52 options.

Kev and Laya on their own numbers

On sources it never trained on, Kev-9B reaches 0.852 test accuracy against Jev’s 0.857 on the development split, a comparison Kev’s own README flags as uncontrolled. The gap that remains is coverage at 5% error: Kev covers 0.45 to 0.57 against Jev’s 0.70. Kev’s author tried to close it during training (label smoothing, Brier and focal losses), and that round came back negative under its pre-registered gate.

Laya reports beating Jev on its own benchmark (typed-decisions accuracy 0.766 against 0.727, ECE 0.081 against 0.246), but every Jev number in those comparisons is a published third-party figure, so each one is cross-study. One independent test put Laya at 0.590 against Jev’s 0.974, the lowest accuracy of the clones it compared and the fastest at about 30 ms. Laya’s documentation is open about its own losses: Banking77 at 0.425 to 0.492 against Jev’s 0.870, and an English checkpoint that scores 0.000 on Khmer at 0.952 mean confidence.

What JevBench says

JevBench v1.5.4 is the one board that runs many clones on the same items: 106 ranked systems, 1,624 items of which 720 are sealed, on evaluator-owned GPUs. “Intelligence” is a chance-adjusted accuracy, and the composite rank also weights speed and cost.

Rank	System	Base	Intelligence	Choice ECE
1	Cygnet	frozen Gemma-4-12B-it, one-token readout	71.1	0.041
2	Winnow-12B	Gemma-4-12B-it LoRA	74.4	0.085
3	Jev 1.13.0	closed	72.0	0.027
7	decider-4b v2	Qwen3.5-4B-Base	55.8	0.049
21	raw Qwen3-4B-Instruct-2507	stock, option logits	54.1	0.329
37	Kev-4B	LoRA + pointer head	39.9	0.278
93	Laya	ModernBERT-large	0.0	0.149

The top open system is a first-token readout on a frozen 12B model, and a 4B model read the same way with no training at all beats Kev-4B by 14 points, which on the board’s face meant I was serving the wrong model.

Provenance narrowed the candidates before any measurement. Nimble’s pipeline contains a bridge that calls the Jev API although its README says it was not distilled from Jev, JevK5 is reported to include GPT outputs, and several “jev-distill” models train on Jev outputs. decider-4b (by Mapika, Apache-2.0) states a local Qwen3.5-27B teacher and nothing distilled from Jev, which made it the best-placed 4B candidate. The raw Qwen readout was the training-free control.

K2: scoring the board’s picks on my panel

The board’s task mix is not Loom’s: Loom asks short questions, and it needs a model that knows when not to answer more than one that answers most things right. The board’s Kev row also describes a Qwen3-4B base, while I pin jaredpalmer/kev-4b at revision 139fdd9, so I could not tell whether it was the same model. I pre-registered a check, K2, before running anything: score decider-4b and the raw Qwen readout on the same panel Kev-4B had already been scored on, and swap only if a candidate beats Kev-4B by at least 5 points of macro coverage at 5% error with a bootstrap lower bound above zero, on both the full panel and the panel without emotion and sciq (decider’s training mixture touches those two).

The panel is the development split of Kev’s frozen transfer-v9 eval suite: 11 tasks and 1,046 questions. Both new arms ran on my GPU box (a 16 GB RTX 4070 Ti Super) through decider’s own server, the raw Qwen arm through its chat readout layout with no temperature, so that arm measures the stock model plus decider’s prompt and readout. The Kev-4B and Laya rows are reused from the earlier K1 run.

Arm	11-task macro	Difference from Kev-4B (lower bound)	9-task macro	Difference (lower bound)
Kev-4B	0.555		0.549
decider-4b	0.556	+0.1 pp (-9.1)	0.518	-3.2 pp (-13.2)
raw Qwen3-4B	0.091	-46.4 pp (-54.7)	0.000	-54.9 pp (-64.7)
Laya	0.158	-39.7 pp (-46.6)	0.131	-41.8 pp (-50.5)

Nothing came close to the swap rule, so Kev-4B stayed.

The raw Qwen readout answers 72% of the panel correctly and still covers only 0.091, because at temperature 1.0 its confidence does not separate right answers from wrong ones. No threshold keeps its error under 5% on any task except sciq, where every arm covers 1.000. JevBench’s 54.1 rewards accuracy, while my swap rule rewards knowing when to answer.

decider-4b and Kev-4B tie on the macro with opposite strengths:

Task	Kev-4B coverage	decider-4b coverage	Kev-4B accuracy	decider-4b accuracy
unknowable_control	1.000	0.691	0.955	0.827
composition_holdout	1.000	0.750	0.979	0.896
buried	0.375	0.075	0.700	0.675
legacy_holdout	0.800	0.662	0.812	0.738
paws	0.275	0.188	0.787	0.713
qnli	0.825	0.925	0.912	0.925
mmlu	0.350	0.575	0.725	0.725
tweet_offensive	0.312	0.588	0.738	0.787
emotion	0.163	0.463	0.550	0.762
mmlu_pro	0.005	0.205	0.560	0.505
sciq	1.000	1.000	0.975	0.988

decider-4b is better on knowledge and classification tasks (mmlu, mmlu_pro, qnli, tweet_offensive, and emotion, which its training mixture may have seen). Kev-4B is better on buried evidence, held-out rule compositions and above all unknowable_control, the questions where the deciding evidence was removed and answering is the error. That is the task that matters most to Loom, and Kev-4B covers 1.000 of it against decider-4b’s 0.691. decider-4b answered at 48 ms p50 against Kev-4B’s 553 ms, but the two ran on different serving stacks, so that gap mixes model and server.

Running decider took three fixes on the GPU box: its server needs flash-linear-attention and causal-conv1d, so it ran in WSL; its CUDA-graph warm-up did not fit 16 GB, so it ran eager; and the WSL image could not live on a nearly full system drive, because filling it corrupted the distro once.

Where this leaves the pick

The readout family decides what the head reads (option tokens in the vocabulary, a pointer between two hidden states, or a marker in a bidirectional encoder) and with it the option cap and the cost per question. The top open system on JevBench is a first-token readout, yet on my panel a training-free first-token readout covered 0.091 because its confidence could not tell right from wrong. For a caller that has to abstain, what matters is whether the model’s confidence orders right answers above wrong ones, and JevBench’s intelligence axis does not measure that. A per-task mix of Kev-4B and decider-4b looks tempting given the opposite strengths, but the earlier K1 run found only +1.2 points of oracle headroom for per-task selection with Laya as the second model, and this panel was not built to choose between models, so any follow-up needs fresh items. K2 kept Kev-4B as the pinned model in verdict, the decision service Loom calls.