Decision models are having a month. OpenAI and Perplexity sell them by the API call, TypeSafe’s Jev raised $870M to sell them by the enterprise contract, and the open answer is Laya: a 421M parameter model from Convai Innovations that never writes a sentence. It reads your text, answers typed questions (pick from a list, score this, yes or no) in a single forward pass, and generates zero output tokens.
One developer, mizorewww, has now ported Laya to Apple Silicon twice. The MLX port got the attention: 13.4 ms decisions on an M3 Max, no cloud call. The second port is the interesting one. laya-coreml moves the model onto Core ML and the Apple Neural Engine, the chip block that spends most of its life waiting for someone to use it. 1,591 stars in its first month, Apache 2.0, version 0.5.1.
Short version: the Neural Engine is only modestly faster than the GPU here. It uses about a third of the energy per decision, and for a model whose whole job is making millions of tiny decisions, energy is the metric that matters. The fine print is about calibration, and it is spicier than the benchmarks.
The numbers, with receipts
The author publishes raw benchmark data, which already puts him ahead of most launch posts. One short multilingual decision, 91 tokens padded to 96, on an M3 Max, 65,598 calls across alternating blocks:
- Core ML on the Neural Engine (FP16): 4.98 ms median, 0.154 joules per decision
- Compiled MLX on the GPU (FP16): 6.94 ms median, 0.429 joules per decision
- Core ML ANE with W8 palette compression: 4.88 ms, 0.134 joules
So the ANE is 1.39× faster and 2.78× more energy efficient; the W8 variant stretches that to 3.19×. Whole-system power during the runs: about 31 watts against the GPU path’s 61. My favorite sentence in the repo is the honest one: the author was asked for a 10× improvement, did not achieve it, and published the mathematical investigation of why not. There is also a Snake demo sustaining roughly 50 decisions per second with zero deaths, because every serious inference project must eventually play Snake.
The fine print: three things before you trust it
The 5 ms headline is a short-decision number. The ANE bundle caps at 96 tokens total, question, options and state included. Long text falls back to the general CPU/GPU bundle (1,024 tokens), and a full-length request there takes about 91.7 ms. The Neural Engine path is a sprinter. Design for sprints.
The validation proves the port, not the model. The Core ML checkpoints match the upstream model’s answers on 189 of 189 validation questions, with tiny probability drift. That is conversion fidelity. Whether Laya’s decisions are right on your data is a separate question the repo correctly does not claim to answer.
The probabilities need calibration before they mean anything. The upstream model card is unusually blunt: one of its outputs, act_probability, carries no usable signal yet (AUROC 0.30, worse than a coin flip), so gate on confidence instead (0.77). The model also ships over-confident; refitting temperatures per question type cut its calibration error from 0.466 to 0.081. The Core ML port adds its own catch: one shipped temperature bucket, 0.1006, would sharpen the logits until a coin flip reports as near-certainty, so the port clamps it and warns you at load. A decision model whose probabilities are decorative is just a very fast coin. Calibrate on your own data first.
Scenarios: what a 5 ms local decision is actually for
The agent gatekeeper. Most agent tool calls are routine, and a few are the ones from my blast radius audit that keep you up at night. Put Laya in front: every proposed action gets an act-or-escalate decision in 5 ms, on-device, before a frontier model or a human ever sees it. The cheap System 1 decides what deserves an expensive System 2. At cloud prices that routing layer is a per-call tax forever; on the Neural Engine it is a rounding error.
The inbox bouncer. Triage is the demo use case for a reason: route this message (choice), how urgent (score), is this a refund demand or a cancellation threat (yes/no). Running it locally means the text of your support mail never leaves the Mac. Compare yesterday’s topic, NVIDIA’s free endpoint, where the price of free is your prompts in someone else’s logs. For anything with a customer’s words in it, local wins on policy before it wins on speed.
The always-on sentinel. Do the energy math. At 0.154 joules a decision, asking once every ten seconds, all year, costs about 0.13 kWh. A decision every second, all day, is roughly 3.7 watt-hours, a few percent of a laptop battery. That makes continuous classification practical: watch a folder, a feed, a smart home event stream, and only wake the big model or the human when the sentinel says so. On a GPU runtime drawing twice the power for the same answers, you would think twice about leaving it on. On the ANE you stop thinking about it.
The app-native reflex. It is a pip install and a Hugging Face download, no PyTorch, no API key, works offline after the first download. Any Mac app can embed a reflex: sort the screenshot reports, tag the exports, pre-screen the form submissions. Decisions become a library call instead of a cloud dependency.
The part I could not test
Core ML runs on Apple hardware, and my test box is a Linux VM, so this is the mirror image of the OpenShell post: there the software wanted a datacenter, here it wants a Mac and nothing else will do. The numbers above are the author’s. What makes them citable is the method: raw measurements published, failed experiments included (the 4-bit and 6-bit variants flunked his fidelity gate and were never released), and an independent write-up of the MLX sibling that reaches compatible conclusions about the serving path. If you have an M-series Mac, the usage docs get you from install to a first decision in minutes, and you can check every number in this post yourself. That is the correct standard, and almost nobody meets it.
Short version: the decision layer is moving on-device, the Neural Engine is the right silicon for it, and the energy bill is the real headline. Just calibrate the probabilities before you let it decide anything that matters.
