Decision models like Jev are exciting for computational social science (CSS), as they promise zero-shot classification at lower cost and latency than LLMs. But how accurate is it?
A useful new point on the frontier
Jev reached a median macro-F1 of 58.1 on 0-100 scale across the 15 evaluation tasks, compared with 66.9 for Gemini 3.8 Flash, the best-performing LLM by this measure. Macro-F1 gives each class equal weight when averaging F1 scores, so frequent classes do not dominate the result. Jev achieved the highest F1 on the dialect task. On the other 14 evaluation tasks, it scored below the per-task best LLM, with a median deficit of 11.6 points. I’m surprised that the relatively cheap 3.8 Flash took the top LLM spot, instead of Fable 5.1. GPT-6 models weren’t available yet for the benchmark.
Fine-tuning isn’t dead! The fine-tuned RoBERTa-large baseline still beats all tested zero-shot models on half the suite. Labeled training data remains valuable even as zero-shot models improve.
Jev labeled all 7,977 items for just $0.21, cheaper than any tested hosted LLM, with no parsing failures. The paper reports measured inference charges, not the full cost of building and validating an annotation pipeline. Latency was not measured.
Jev scored well on the calibration of its confidence scores, beating 16 of 19 tested LLMs. However, it also showed that Jev was catastrophically overconfident on the empathy classification task. It’s still important to do your own evals on the specific construct.
Overall, Jev is a neat tool that comes into play on easy, high-volume tasks, or in combination with a stronger model that handles cases it’s unsure about. There are also about a dozen open source variants. I’d be excited to see a decision model specifically calibrated for social science tasks and validated in different languages.