LocalJev brings a Jev-compatible decision API to a local inference setup. Published under GitHub Next, the project connects structured decision requests to a language model through an OpenAI-compatible endpoint. Its documented setup uses oMLX.
For developers, the interesting question is practical: can a local model route a request or assess a piece of text reliably enough to help an application? This guide examines the documentation and published evaluation, then proposes a small test you can adapt. We have not run the models ourselves.
How LocalJev handles decisions
The LocalJev README describes a TypeScript/Bun bridge exposing POST /v1/systemone. It turns state and typed questions into prompts, validates returned JSON, retries malformed output, and normalizes probabilities into Jev-compatible responses.
API compatibility does not establish equivalent decision quality. LocalJev uses model-generated probabilities. OpenJev’s approach reads model logits through specialized backend support. The repository explicitly distinguishes those mechanisms.
Consider a support application that asks which team should receive a message. Returning the expected fields helps the application integrate the answer. Whether the selected team is correct requires a separate evaluation. A response can be perfectly formatted and still send the customer to the wrong queue.
LocalJev setup: what you need first
The documented prerequisites are Bun 1.2+ and a running oMLX server. Install project dependencies with bun install, copy .env.example to .env, configure the upstream key and model, then use bun run start. The default API listens at 127.0.0.1:8080; /ready checks availability. The README also shows TypeSafe SDK configuration.
oMLX targets Apple Silicon and currently lists macOS 15 or later. Check its installation requirements alongside the selected model. The documented Mac route should not be presented as a verified Windows or Linux installation guide.
Before downloading large weights, write down available memory, free storage, the exact checkpoint, and the runner version. Treat these as part of your experiment record. A model name alone is insufficient to reproduce the experience on another machine.
What the LocalJev benchmark actually measured
The September 18 evaluation report describes 1,200 measured requests on an Apple M5 Max with 64 GiB RAM, oMLX 0.6.4, and Bun 1.4.0. These are author-published measurements, not our independent results.
| 4-bit model | Short-input macro accuracy | Median full-decision time |
|---|---|---|
| Gemma 4 E2B | 45.8% | 0.522 seconds |
| Gemma 4 E4B | 63.3% | 0.543 seconds |
| Gemma 4 26B-A4B | 75.0% | 0.675 seconds |
| Qwen3.6-35B-A3B | 76.7% | 0.889 seconds |
| DiffusionGemma 26B-A4B | 74.2% | 1.207 seconds |
Within this run, Qwen had the highest short-input macro accuracy, while Gemma 4 26B-A4B delivered a faster median with similar aggregate accuracy. These figures do not establish a universal winner or a minimum memory requirement. Full-decision latency is not time to first token.
The evaluation methodology uses AG News categorization, BoolQ reading comprehension, and SST-5 sentiment. Each task contains 40 examples; five models run with two input conditions. The longer condition adds 2,048 unrelated background words around the retained target. It is a distraction test, rather than proof of real-world long-document reasoning.
The guide also warns that public datasets may have appeared in training. Its entropy-derived confidence field is not an estimated probability that an answer is correct. A high confidence value therefore needs validation against labeled outcomes before becoming an automation threshold.
A practical LocalJev test for your own workflow
Start with one reversible task: suggesting a support queue. Use three destinations—account access, billing, and technical support—and allow your application to send uncertain cases to a person. The following examples are an editorial test design, not measured LocalJev outputs.
| Example message | Expected handling | What it tests |
|---|---|---|
| “I forgot my password and cannot sign in.” | Account access | A clear routing case |
| “My invoice contains the same charge twice.” | Billing | Payment-specific intent |
| “The app crashes immediately after login.” | Technical support | Distinguishing a crash from an access problem |
| “I cannot sign in, and I also need a refund.” | Human review under your chosen policy | Multiple valid intents |
| “The crash is fixed. My remaining question concerns an invoice.” | Billing | Following the current request |
- Label first: write expected routes before seeing model predictions. Resolve disagreements in your routing policy.
- Vary the wording: include short messages, misspellings, polite complaints, and your actual working languages.
- Measure useful outcomes: record correct routes, invalid responses, retry frequency, median latency, and slow cases.
- Compare a simple baseline: run the same examples through existing routing rules. Extra complexity should earn its place.
- Keep a fresh test set: after improving prompts on development examples, evaluate on messages you did not use for tuning.
For example, an instruction to “route all invoice mentions to billing” may fail when a customer mentions an old invoice while reporting a current outage. Include these mixed messages deliberately. They reveal more about operational usefulness than repeatedly testing obvious examples.
Local processing and deployment boundaries
For a local-only experiment, inspect the complete data path: the configured inference address, application logs, monitoring integrations, and any tools the surrounding agent calls. A local model endpoint does not establish that every component of your application stays offline.
Also separate classification from action. Initially, display the proposed route next to the existing workflow and let a person review disagreements. Only expand automation after you understand the recurring errors and have a clear fallback.
If your agent also needs information from earlier interactions, see our Hindsight memory guide. Memory and decision evaluation address different parts of the application. Our AI tool testing checklist offers a broader framework for recording quality and cost.
Is LocalJev worth evaluating?
LocalJev is worth a focused experiment when you have a concrete routing or scoring task and can supply labeled examples. Use the published results to choose candidates, then judge them on your own error patterns, latency, and hardware. That produces a more useful answer than treating local execution or API compatibility as proof of readiness.
Official sources
- LocalJev repository and setup documentation
- Evaluation design and limitations
- Published benchmark results
- oMLX repository and requirements
Documentation reviewed September 27, 2026. Installation requirements and project behavior can change.