Vision Evals

Overview

@nut-tree/vision-evals is a benchmark harness for the vision models you plug into nut.js through @nut-tree/plugin-ai-sdk. It runs a suite of UI screenshots through the plugin's real matching pipeline, checks whether the model's answers are safe to click, and ranks candidate models against each other. It runs on your machine, against the backend you choose, and never uploads your screens anywhere.

npx @nut-tree/vision-evals run --provider openai-compatible \
  --model mlx-community/Qwen3-VL-8B-Instruct-4bit \
  --base-url http://localhost:1234

The Problem It Solves

With @nut-tree/plugin-ai-sdk, nut.js can locate UI elements from a plain-language description:

await screen.find(contentMatchingDescription("the blue 'Sign in' button"));

That single line hands a screenshot to a vision model and clicks whatever comes back. Which model should be on the other end? Public vision leaderboards do not answer that. They score captioning, document QA or general visual reasoning, and none of that tells you whether a model can put a box on the right button, in the right coordinate system, and stay quiet when the button is not there.

The only meaningful test is to run a model through the exact pipeline that ships: the plugin's system prompt, its structured-output schema, its confidence filtering, its region sanitisation and its coordinate handling. That is what vision-evals does. There are no mocks and no simplified prompts. Every case builds the same match request that nut.js core produces and calls the real vision finder, so the numbers you get predict production behaviour rather than paper scores.

What It Measures

Every case is one question asked of one screenshot. There are two kinds.

Positive cases describe something that is on screen and check whether the model finds it. The pass criterion is deliberately practical: the center of the returned region has to land inside the labelled element, because that center is exactly where nut.js will click. A generous box still passes if its center is on target. A tight box that is one element off fails.

Negative cases describe something that is not on screen, such as an absent dialog, a button that does not exist, or a chart that is not rendered, and check that the model returns nothing. A model that "finds" absent content will click arbitrary coordinates in production. This is the most dangerous failure in UI automation, and vision-evals treats it that way.

What It Can Do

Why It Is Useful

• It answers the question that matters. Not "which model is smartest" but "which model can I let click on my application". Those are different questions, and only the second one has consequences for your automation.
• It catches the failures that hurt. A model that hallucinates UI does not fail loudly. It clicks somewhere plausible and your workflow continues in the wrong state. Negative cases and the hallucination-first ranking make this failure mode visible before it reaches production.
• It makes self-hosting a measured decision. Small local models are cheap, private and fast, and some of them are very good at UI localisation. Others are not, or are only good in the right coordinate convention. vision-evals lets you find the cheapest model that meets your bar instead of defaulting to the largest one.
• It keeps your data on your machine. Screenshots, ground truth and results never leave your environment except as requests to the backend you configure. For a fully local backend, nothing leaves at all, which makes the tool suitable for private application screens.
• It protects you across upgrades. Because the harness runs the plugin's actual pipeline, a score is only meaningful for the plugin version that produced it. Re-qualifying a model after a plugin upgrade is a matter of bumping one version and re-running.
• It turns a model choice into a repeatable artifact. Results are JSONL files, reports render as text, Markdown or JSON, and the exit code is CI-friendly. The decision to adopt a model, and the evidence behind it, live in your repository next to the code that depends on it.

Reading a Report

model              run    cases  pass %  pos pass  recall %  neg pass  neg %   prec %  no-match  miss  halluc  errors  mean IoU  dist px  lat ms  p95 ms
-----------------  -----  -----  ------  --------  --------  --------  ------  ------  --------  ----  ------  ------  --------  -------  ------  ------
openai/gpt-5-mini  run-…  27     96.3%   14/15     93.3%     12/12     100.0%  100.0%  1         0     0       0       0.612     11.3     4210    9800

A few rules of thumb:

• Negative pass rate below 100 % means the model hallucinates UI. Weigh this above every other column.
• Recall below precision means the model refuses more than it errs. Loosening the confidence threshold usually helps.
• Precision below recall means it returns boxes it should not. Tighten the threshold or prefer another model.
• Many misses with a center distance near half or double the expected value points to a coordinate-space problem, not a comprehension problem. Try the other convention.
• A drop in mean IoU between runs of the same model is a regression signal even when pass rates hold.

Getting Started

# Evaluate a local OpenAI-compatible server (vLLM, LM Studio, llama.cpp, ...)
npx @nut-tree/vision-evals run \
  --provider openai-compatible \
  --model <your-model-id> \
  --base-url http://localhost:1234

# Evaluate a hosted provider
OPENAI_API_KEY=... npx @nut-tree/vision-evals run --provider openai

# Compare several backends, with annotated screenshots and a Markdown report
npx @nut-tree/vision-evals compare --config targets.json --screenshots --format markdown

Ready-to-edit compare configs ship in the package under example-configs/: a local quickstart, a coordinate-space A/B, a local multi-model shootout and a hosted OpenAI plus Anthropic example.

Where to Go Next

The package ships its full option reference in its README, guidance on writing discriminating positive and tempting negative cases in fixtures/README.md, and the compare config format in example-configs/README.md.