Evaluations API

Python

The blockether.vis.evals package does all the work on your computer. It writes task folders, runs them with Harbor and reads the Harbor job folders. It needs no gateway.

HTTP

The gateway routes score, diagnose and compare a round summary that you send. They run the same Python code as the SDK, on the gateway. They save nothing: each answer comes from the request body only.

Python

Install the SDK with Python 3.12 or later. It installs Harbor 0.24.0 with it. To run tasks, also configure a provider, as in Evaluations. The SDK downloads Vis when it is not installed.

python -m pip install vis-agent

The examples run as one script in an empty folder. Each example uses the names of the examples before it.

HTTP

Define the vis_api function from Authenticate requests. The examples use it. Each request sends a JSON file with --data @file.json.

The routes need a round summary, and only the Python SDK makes one. The summary comes from the Harbor job folders, which are on the computer that ran Harbor. So you need the Python SDK there, also when another program sends the requests. Keep the summary and the answers yourself. The gateway does not store them.

Python

Each function of blockether.vis.evals gives a typed result. The gateway method sends the same operation to a gateway and gives the answer as a dict. Use it only when the gateway must do the work, for example from a computer without the job folders.

Task	Function	Gateway method
Write a task	write_task(dataset, name, instruction, checks)	None
Run the tasks	run_tasks(path, model)	None
Report a round	round_report(jobs_dir, dataset_dir)	None
List the tasks of a dataset	read_tasks(dataset)	None
Summarize a round	round_summary(jobs_dir, dataset_dir, model=...)	None
Score a round	bench_report(summary)	post_evals_report(body=...)
Find the failure causes	diagnose(summary, labels, limits)	post_evals_diagnose(body=...)
Check the cause labels	cause_check(summary, labels, limits)	post_evals_labels(body=...)
Check other labels	agreement(items, ...)	post_evals_agreement(body=...)
Compare two runs	compare_cases(baseline, candidate)	post_evals_compare(body=...)
Read the leaderboard position	read_leaderboard(package, name, summary)	post_evals_leaderboard(body=...)

HTTP

Each route takes a JSON object and gives a JSON object. No route writes or runs tasks or reads job folders. Do these steps with the Python SDK or the harbor command.

Task	Method and path
Score a round	POST /v1/evals/report
Find the failure causes	POST /v1/evals/diagnose
Check the cause labels	POST /v1/evals/labels
Check other labels	POST /v1/evals/agreement
Compare two runs	POST /v1/evals/compare
Read the leaderboard position	POST /v1/evals/leaderboard

Python

from blockether.vis.evals import write_task

task = write_task(
    "tasks",
    "add-function",
    "In m.py add a new function c that returns 3, placed after b. Keep a and b unchanged.",
    {"want": {"m.py": ["def c(", "return 3", "def a(", "def b("]}},
    files={"m.py": "def a():\n    return 1\n\n\ndef b():\n    return 2\n"},
    agent_timeout_sec=300,
)
print(task.name, task.path, task.agent_timeout_sec)

The call writes the folder tasks/add-function with the files of a task. files maps each seed file path to its text. checks is the content of tests/checks.json. It raises FileExistsError when the task folder exists. Give a list of strings as the instruction to write a task with several turns: one string for each user message.

artifacts lists the workspace files that Harbor copies to the trial folder, for example the record of a mock extension. filesystem_roots maps each filesystem root id to a workspace folder, for example {"library": "library"}. The trial registers each root in vis.yml, and the agent finds it with ls. The e2e/py-project-paths task uses it.

HTTP

No route writes a task, because Harbor reads the task folder on the computer that runs it. Write the files of a task yourself, or run the Python example on that computer:

python3 - <<'EOF'
from blockether.vis.evals import write_task

write_task("tasks", "add-function", "In m.py add a new function c that returns 3.",
           {"want": {"m.py": ["def c(", "return 3"]}},
           files={"m.py": "def a():\n    return 1\n"})
EOF

Python

from blockether.vis.evals import run_tasks

job = run_tasks("tasks", "zai-coding-plan/glm-5.3-flash", jobs_dir="jobs", job_name="first")
print(job)  # jobs/first

path is one task folder or a folder of tasks. The model is a provider/model route. attempts sets the trials for each task, and concurrency sets the trials that run at the same time. Each trial makes paid model calls. The call waits until all trials end.

agent_options gives the options of the Vis agent. {"source": "/path/to/vis"} tests a Vis checkout, and {"native_bin": "/path/to/vis"} tests a raw native image. The call sets HARBOR_TELEMETRY=off, so Harbor sends no usage data.

HTTP

No route runs tasks. Run them with Harbor on the computer that has the task folders:

HARBOR_TELEMETRY=off harbor run --path tasks \
  --agent blockether.vis.evals.agent:VisAgent \
  --env blockether.vis.evals.environment:HostEnvironment \
  --model zai-coding-plan/glm-5.3-flash --jobs-dir jobs --job-name first

Python

from pathlib import Path

from blockether.vis.evals import REPORT_FILE, render_report, round_report

full = round_report("jobs/first", "tasks")
print(render_report(full))
for row in full.repeats:
    print(row.k, row.pass_at_k_percent, row.pass_all_k_percent)
print(Path("jobs/first", REPORT_FILE).exists())  # True, run_tasks wrote it

round_report reads the job folder once and gives a RoundReport:

Field	Content
score	The bench_report of the round.
repeats	One row for each k: k, tasks, pass_at_k_percent and pass_all_k_percent.
diagnosis	The diagnose result, with the label_check of labels.json.
judge	The stage agreement of a judge, or None without judge_stages.json.

round_report raises FileNotFoundError when the folder holds no trial. labels, judge_dir and stage_labels replace the default paths of the label and judge files. write_report(full, path) writes the JSON.

HTTP

No route reads job folders. Run the report command on the computer that has the job folder:

python -m blockether.vis.evals.report jobs/first --tasks tasks --json

Python

import json

from blockether.vis.evals import round_summary

summary = round_summary(job, "tasks", model="zai-coding-plan/glm-5.3-flash")
print(summary["scored_tasks"], summary["solved_tasks"], summary["task_outcomes"])
with open("summary.json", "w") as stream:
    json.dump(summary, stream)

jobs_dir holds one Harbor job folder or one folder for each Harbor job. dataset_dir holds one folder with a task.toml for each task. It is optional, but without it the total counts only the attempted tasks. model is the model name or the provider/model route in the trajectories. An attempt of another model does not count.

HTTP

No route reads job folders, because the gateway can run on another computer. Build the summary with the Python SDK, then send it:

python3 - <<'EOF'
import json
from blockether.vis.evals import round_summary

summary = round_summary("jobs/first", "tasks", model="zai-coding-plan/glm-5.3-flash")
with open("summary.json", "w") as stream:
    json.dump(summary, stream)
EOF
jq '{summary: .}' summary.json > report.json

Python

from blockether.vis.evals import bench_report

report = bench_report(summary, "first")
print(report.pass_rate_percent, report.pass_rate_ci95)
print(report.strict_pass_rate_percent, report.detectable_change_points)

HTTP

vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/report" \
  -H 'content-type: application/json' --data @report.json

Python

from blockether.vis.evals import agent_limits, diagnose

limits = agent_limits("tasks")
diagnosis = diagnose(summary, None, limits)
for finding in diagnosis.findings:
    print(finding.kind, finding.task_count, finding.title)

HTTP

jq '{summary: ., limits: {"add-function": 300}}' summary.json > diagnose.json
vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/diagnose" \
  -H 'content-type: application/json' --data @diagnose.json

Python

from blockether.vis.evals import cause_check, load_labels, save_label

save_label("labels.json", summary, "add-function", "early_stops", "Stopped before the tests.")
labels = load_labels("labels.json")
check = cause_check(summary, labels, limits)
print(check.labelled, check.kappa, check.advice)

save_label saves the cause that a person gave to one failed attempt. The labels file maps each trial name to its cause and note. A new label of the same attempt replaces the old one. A solved attempt raises ValueError.

HTTP

jq -n --slurpfile summary summary.json \
  '{summary: $summary[0], labels: {"add-function__a1b2": {cause: "early_stops"}}}' > labels.json
vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/labels" \
  -H 'content-type: application/json' --data @labels.json

labels maps each trial name to an object with its cause.

Python

from blockether.vis.evals import agreement

check = agreement(
    [("add-function__a1b2", "write", "verify")],
    source="judge",
    noun="judge stages",
    labeller="a person",
    fix="Correct the judge rubric first.",
)
print(check.kappa, check.advice)

Each item is a trial name, the automatic result and the human result.

HTTP

cat > agreement.json <<'EOF'
{"items": [{"name": "add-function__a1b2", "automatic": "write", "human": "verify"}],
 "source": "judge", "noun": "judge stages"}
EOF
vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/agreement" \
  -H 'content-type: application/json' --data @agreement.json

Python

Run the same tasks again after a change, then compare the two jobs:

from blockether.vis.evals import case_results, compare_cases, read_trials

second = run_tasks(
    "tasks",
    "zai-coding-plan/glm-5.3-flash",
    job_name="second",
    agent_options={"source": "/path/to/vis"},
)
comparison = compare_cases(
    case_results(read_trials(job)),
    case_results(read_trials(second)),
    baseline_name="first",
    candidate_name="second",
    margin_points=5,
)
print(comparison.verdict, comparison.difference_points, comparison.difference_ci95)

case_results maps each case to its passed runs and all runs.

HTTP

cat > compare.json <<'EOF'
{"baseline": [{"task": "add-function", "agent": "vis", "model": "zai-coding-plan/glm-5.3-flash", "passed": 0, "runs": 1}],
 "candidate": [{"task": "add-function", "agent": "vis", "model": "zai-coding-plan/glm-5.3-flash", "passed": 1, "runs": 1}],
 "margin_points": 5}
EOF
vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/compare" \
  -H 'content-type: application/json' --data @compare.json

Each case has a task, an agent, a model, the passed runs and all runs.

Python

from blockether.vis.evals import read_leaderboard

board = read_leaderboard("terminal-bench/terminal-bench", "4-0-0", summary, "first")
print(board.title, len(board.rows))
print(board.vis.rank_by_pass_rate, board.vis.rank_range, board.vis.caveats)

HTTP

jq '{package: "terminal-bench/terminal-bench", name: "4-0-0", summary: ., round_name: "first"}' \
  summary.json > leaderboard.json
vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/leaderboard" \
  -H 'content-type: application/json' --data @leaderboard.json

Python

try:
    write_task("tasks", "Add Function", "Add c.", {"want": {"m.py": ["def c("]}})
except ValueError as error:
    print(error)  # task name must be lowercase words joined by hyphens: 'Add Function'

A function raises ValueError for an invalid value and FileNotFoundError for a missing folder. A gateway method raises GatewayError with the HTTP status and the error code of the table.

HTTP

vis_api -X POST "$VIS_GATEWAY_URL/v1/evals/report" \
  -H 'content-type: application/json' --data '{}'