- Published on
Build your own decision model
- Authors
- Name
- DeepHorus
Developers can now build decision models by constraining a language model's output to fixed options. Instead of generating tokens sequentially, the model evaluates all allowed answers in one forward pass. Systems like Jev mask other vocabulary items so the model only emits tokens A through E. The highest-probability option becomes the prediction.
Traditional language models need one pass per output token for structured JSON. A typical example requires eleven passes to generate valid output. Decision models assume fixed choices and select quickly in a single pass. By constraining outputs to A, B, C, D, and E, the model cannot choose outside that set. This constraint does not guarantee correctness.
A demonstration used Qwen/Qwen3-1.7B with the question "What color is the sky?" Options were Red, Blue, Green, Purple, and "I don't know." The model predicted B (Blue) with 0.9988 probability. E received 0.0012, while others scored 0.0000. This shows the model can produce a reasonable answer.
To test accuracy, the method ran against a random sample holdout of CommonsenseQA. The model achieved 725 correct out of 1221, an accuracy of 0.5938. Macro F1 was 0.5844. Per-class precision and recall varied; class E had 0.7519 precision but 0.4255 recall. Class A had 0.5733 precision and 0.7197 recall.
These results are notable for a 1.7B parameter model. Output token probabilities are often treated as confidence scores, but without additional training they reflect next-token confidence. They likely do not represent the true probability of a correct answer. Users should therefore be cautious when interpreting high confidence as guaranteed accuracy.
The approach offers a lightweight way to emulate decision-making. It reduces inference passes and provides calibrated-looking probabilities. Developers can implement it with PyTorch and Transformers libraries. Future work involves fine-tuning to improve calibration and accuracy. As open-source models grow more efficient, custom decision models may become common for fast, constrained choices.
