blockether.vis.evals

How well automatic results agree with human labels of the same items.

labelled counts the paired items. agreement_percent is the share of equal pairs, or None without pairs. kappa is Cohen's kappa and alpha is Krippendorff's alpha of the same pairs. Each is rounded to 3 places, or None when it is undefined. disagreements names each item where the two results differ. advice says if you can trust the automatic results.

The benchmark score of one round, with its usage.

pass_rate_percent divides solved tasks by scored tasks, by the latest scored attempt of each task. strict_pass_rate_percent divides them by all dataset tasks, so unscored, unattempted and GPU tasks count as failures. The attempt pass rate counts every scored attempt, with the attempts of one task as one cluster. Each _ci95 value is a 95% Wilson interval in percent. A second round of the same size cannot detect a change below detectable_change_points. attempts_per_task is the most common scored attempt count of a task. When it is more than one, pass_at_k_percent and pass_all_k_percent give the unbiased pass@k and pass^k with that count as k: the chance that at least one or all of k attempts solve a task. Each task uses all its scored attempts. Otherwise they are None. unstable_tasks lists tasks with mixed results. estimated_metered_cost_usd and cost_per_solved_usd use metered API prices. They are not subscription charges.

One case whose pass rate changed between the two sides, as passed/runs.

A paired comparison of the cases that both sides ran.

The rates are the mean pass rates of the paired cases, in percent. difference_ci95 is the Agresti-Min 95% interval of the paired difference, in points. verdict is better or worse only when that interval excludes zero. Without margin_points, every other result is no measurable difference. With it, the position of the interval against the margin gives the verdict:

• equivalent: the interval is inside the margin on both sides.
• not worse: the interval is above the negative margin.
• not better: the interval is below the margin, but reaches the negative margin.
• inconclusive: the interval reaches both sides of the margin.

The verdict uses the 95% interval before rounding. With a margin, a better or worse verdict gets a caveat when the change is, or can be, smaller than the margin.

At this case count, a true change of detectable_change_points or more shows with about 80% power, by a normal approximation. p_value is the two-sided exact McNemar p-value of the changed cases. With repeats, it is the exact sign test of the case changes. routes holds this test for each route. worse and better list the changed cases, largest change first. unpaired counts cases on one side only; the comparison leaves them out. Caveats name unpaired cases, different routes, a change smaller than the margin and a verdict that the exact test does not confirm.

Where the benchmark loses tasks, largest lever first.

estimate projects the effect of full attempts on agent-error tasks. It is an estimate, not a measurement. label_check compares the causes with human labels.

One problem class. share is the fraction of scored tasks it affects.

One host function call in a code block: one Activity row and its place in the run.

operation names the function. arguments, result and error are the text evidence that the engine observed, or None. state is the outcome. A linked call, such as a shell poll, has the id of its parent row in parent_id. step_metrics has the tokens and the cost of the whole step, not of this call.

One Harbor task, read from its task.toml and instruction.md.

name is the task folder name, as in case_results. instruction is the prompt that the agent gets. The timeouts are in seconds, or None when task.toml gives none. gpus is the GPU count that the environment needs. metadata holds the [metadata] table, for example the lang of an e2e scenario.

One Harbor trial, read from its result.json.

task is the Harbor task name. agent is the agent name that the trial reports. model is the model name of the trial configuration, for example openai/gpt-6.1-sol. reward is the reward value of the verifier. It is None when the verifier gave no reward, for example after an infrastructure error. rewards holds all named rewards. error is the exception type of a failed trial. The token counts and cost_usd are None when the agent did not report them. path is the result.json file.

A Harbor Hub leaderboard. vis is None without a local result for its dataset.

One published agent and model result.

accuracy_percent is over all trials of the row. Per-trial values divide the row totals by trials, and cost_per_success_usd divides the total cost by successes. None means that the board does not publish the value.

Tasks whose latest scored attempt ended the same way.

pass@k and pass^k in percent for one k, over the tasks with k attempts.

The statistics of the repeated attempts of a set of tasks.

attempts_per_task is the most common attempt count, the larger one on a tie. attempt_pass_rate_ci95 is the 95% cluster interval of the attempt pass rate over the tasks. A true change of detectable_change or more shows with 80% power. pass_at_k is the mean chance over the tasks that at least one of k attempts solves the task, and pass_all_k that all k attempts solve it. Each task gives the unbiased estimate from all its attempts, so a task with more than k attempts uses all of them. Only tasks with at least k attempts count. Both are None when no task has k attempts, or when each task has one attempt and you give no k. unstable_tasks names the tasks with both solved and failed attempts.

The full report of one round.

score and diagnosis are the bench_report and diagnose results. repeats has one row for each k from 1 to the attempts per task. judge compares the stages of a judge with human stage labels, or is None without a judge folder.

The exact McNemar test of the paired cases of one route.

worse and better count the cases whose pass rate fell or rose. p_value is the two-sided exact p-value. q_value is its Benjamini-Hochberg value over all routes of the comparison.

The counts, the pass rate and the cost of a set of trials.

A case is one task with one agent and one model. pass_rate is passed / scored, and pass_rate_ci95 is its 95% cluster interval over the cases. Both are fractions, or None without a scored trial. errors counts the trials with an exception. cost_usd is the sum of the reported costs, or None when no trial reports a cost. cost_per_pass_usd divides that sum by the passed trials.

How well the failure stages of one judge agree with human stage labels.

stages names the judged stage of each trial: the most common stage of its samples. sample_agreement_percent is the share of trials whose samples all agree, or None with one sample. sample_alpha is Krippendorff's alpha of the samples, corrected for agreement by chance, and sample_advice applies its limits. kappa, alpha and advice compare the judged stages with the labels, with the limits of labels.

A human failure stage of one scenario trial, kept in a stage labels file.

judged_stage is the stage that a judge gave the same trial, or None.

One ATIF step: the user request, or one agent iteration and its code blocks.

message is text for Vis runs. On the final agent step it is the final answer. extra keeps every Vis field of the step, also unknown ones.

The tokens and the cost of one provider call.

prompt_tokens includes cached_tokens. A value is None when the run did not record it. These metrics belong to the whole step, so do not add them up over the function calls of the step.

One task in a finding.

checks is passed/total verifier tests, or empty. budget_used is the share of the task's agent time limit that the agent used.

One code block that the model ran, and its outcome.

output is the observation that the model saw: the standard output and the error text. activity is the Activity receipt of the block, or None when the run did not record one. Each host function call in the block is in function_calls. extra keeps every Vis field of the call, also unknown ones.

One Vis run read from an ATIF trajectory.

The typed fields hold what evaluations check. document keeps the whole input, also the keys that this reader does not know. Child runs are in subagent_trajectories.

The totals of a run: tokens, cost, steps and time.

total_prompt_tokens includes total_cached_tokens. A value is None when the run did not record it.

A human cause for one failed attempt, kept in the labels file of its round.

diagnosed_cause is the automatic attempt_cause of the same attempt.

Where the local Vis benchmark result would stand on the board.

Like the board accuracy, pass_rate_percent counts every scored attempt. strict_pass_rate_percent also counts each unscored, unattempted and GPU task as attempts_per_task failed attempts. Each _ci95 value is a 95% interval with the attempts of one task as one cluster. The ranks insert the rates among the board rows, and each rank range inserts the ends of its interval. Per-attempt values use model attempts. caveats lists the limits of the comparison.

Read the agent time limit in seconds of each task of a dataset directory.

Each task directory holds a task.toml with agent.timeout_sec. A directory that does not exist gives an empty map.

Compare automatic results with human labels.

Each item is (name, automatic, human). source names the automatic side in each disagreement, for example "judge". noun, labeller and fix go to kappa_advice.

Say if Krippendorff's alpha of repeated noun makes them reliable.

Give the diagnose cause of one attempt, or solved.

The causes are dead_tools, agent_errors, agent_timeouts, unscored, early_stops and other. Empty limits use the agent_limits of the summary.

Name how one attempt ended, with the task outcome rules of the round summary.

A Harbor exception or a failed trajectory gives agent_error:<type>.

Score one round from its round_summary data.

Return the Benjamini-Hochberg q-value of each p-value, in the input order.

A q-value is the smallest false discovery rate at which its test is a discovery. A larger p-value never gets a smaller q-value, and each q-value is at most 1.

Count the passed and the scored trials of each case: a task, an agent and a model.

The keys are (task, agent, model). A trial without a reward does not count, so a case without a scored trial is absent.

Compare the diagnose causes with the human labels of the same attempts.

labels maps a trial name to a label with a cause value. Labels of trials that are not in the summary do not count.

Return the 95% interval of the pooled pass rate of clustered units, or None.

Return Cohen's kappa of two raters over the same items, or None when undefined.

Compare the (passed, runs) cases of a baseline and a candidate.

Each key is (task, agent, model). When the sides share no key and each side has one route, the comparison pairs the cases by task and adds a caveat. The names appear in error messages. margin_points is the smallest change in points that matters, above 0 and at most 100. Raise ValueError when no case pairs, or for a margin outside that range.

Return the smallest change of the pass rate that a second run of this size detects.

The test compares two independent runs of the same size, two-sided at the 5% level, with 80% power. The change holds for a rise and for a fall, so the larger of the two applies. A direction without a detectable change before 0 or 1 does not count. The change has steps of 0.001. None means that there are no runs, or that no change from 0 to 1 is detectable.

Find where a round loses tasks, from its round_summary data.

limits gives the agent time limit of each task, from agent_limits. Without limits, it uses the agent_limits of the summary, and without those no attempt counts as an early stop. labels maps a trial name to a human label with a cause value.

Return the run count of clustered results, made smaller by their design effect.

Each unit is (passed, runs) of one case. The size stays between the unit count and the run count. Without any variation, it is the unit count.

Find an attempt by trial name, or the latest scored attempt of a task.

A name with __ is a trial name, such as task__id. Raises LookupError when the summary has no such attempt.

Count the judge results with an error in judge_report.json, or 0 without it.

Read the stage of each judge sample, grouped by trial, from judge_stages.json.

A file that is missing or is not a JSON object gives an empty map.

Say if count labels with this kappa make the automatic noun usable.

labeller names the way to add a label, and fix is the action for a low kappa.

Return Krippendorff's alpha of nominal values, or None when it is undefined.

Each unit holds the values that the raters gave one item. None is a missing value. A unit with fewer than two values does not count. Alpha is 1 for full agreement, 0 for agreement by chance, and below 0 for less agreement than chance. It is undefined without two values in one unit, or when all values are the same.

Give the latest scored trial of each task, by its finish time.

Read one row of a Harbor Hub leaderboard page.

Read the labels file of a round: a map from trial name to its label.

A file that is missing or is not a JSON object gives an empty map.

Read a stage labels file: a map from trial name to its label.

A file that is missing or is not a JSON object gives an empty map.

Read an ATIF trajectory file (UTF-8 JSON) and parse it.

Raises:

• OSError: The file cannot be read.
• ValueError: The file is not JSON or not a valid ATIF trajectory.

Return the most common value of found. A tie gives the value first in order.

Return the two-sided exact McNemar p-value of paired pass and fail results.

worse counts the cases that pass only in the baseline. better counts the cases that pass only in the candidate. Cases with the same result on both sides do not change the value. Without a changed case, the value is 1.

Join the ATIF documents of the turns of one session into one document.

Give the documents in turn order. The steps follow in that order with new step ids from 1. final_metrics adds the numbers of all turns, and subagent_trajectories keeps every child. The other keys come from the last document. extra.turns is the count of turns. The first turn error stays in extra.error, so a failed turn also fails the merged run. An empty list raises ValueError.

Return the Agresti-Min 95% interval of a mean paired change, and its floor.

Each change is the candidate pass rate of one case minus its baseline pass rate. The floor is the smallest true change that this many cases detect with 80% power. The interval stays inside the possible changes, from -1 to 1.

Build a Trajectory from an ATIF document, a decoded JSON object.

The result is immutable and independent of document. With strict=True, an Activity receipt that breaks the Activity contract raises ValueError. With strict=False, that block gets activity=None and the reason in activity_problem, so older files stay readable.

Raises:

• ValueError: The document is not an ATIF trajectory, or an Activity receipt is invalid. The message never contains run content.

Return the unbiased estimate of pass^k from passed solved of runs attempts.

pass^k is the chance that all of k attempts solve the task. The limits of pass_at_k apply.

Return the unbiased estimate of pass@k from passed solved of runs attempts.

pass@k is the chance that at least one of k attempts solves the task. The estimate uses all runs attempts, not only the first k. Raises ValueError unless 0 <= passed <= runs and 1 <= k <= runs.

Read every page of one board, and place the round of summary on it.

package and name select the Harbor Hub board, for example terminal-bench/terminal-bench and 4-0-0.

Sends one unauthenticated request for each page, at most 50. post sends one request and gives the parsed reply; the default uses urllib. vis is None without a summary or when the summary covers another dataset version.

Read the tasks of a Harbor dataset directory, sorted by name.

Each task is a folder with a task.toml. Other folders do not count. A directory that does not exist raises FileNotFoundError. The e2e directory of the Vis repository is such a dataset.

Read one finished trial from its result.json, or return None for other files.

A job result.json, an unfinished trial and a file that is not JSON give None.

Read the finished trials of a Harbor job directory or of a round of jobs.

A trial result.json is one or two levels below directory. The trials come in start order. A directory that does not exist raises FileNotFoundError.

Give the text that the command prints for report.

Give the repeat statistics of the attempts of each task, True when solved.

k is the attempt count of pass@k and pass^k. Without it, k is attempts_per_task. Raises ValueError for a k below 1.

Build the full report of the round in jobs_dir.

dataset_dir and model go to round_summary. labels is the cause labels file, by default LABELS_FILE in jobs_dir. judge_dir is the job folder of a judge. Without it, the report uses jobs_dir when it holds judge stages. stage_labels is the stage labels file, by default STAGE_LABELS_FILE in the judge folder. Raises FileNotFoundError when jobs_dir holds no trial.

Summarize every trial under jobs_dir, as bench_report and diagnose read it.

jobs_dir is one Harbor job folder or a folder of jobs. dataset_dir holds one folder with a task.toml for each task. It gives the total, the GPU tasks, the tasks without an attempt and the agent_limits that diagnose uses. Without it, the total is the attempted tasks. dataset names the dataset. Without it, the name is the source that Harbor wrote in the trials. model is the model name or the provider/model route of the agent. An attempt of another model does not count. With no model, any model counts. Each task counts once, by its latest scored attempt.

Run the tasks in path with the Vis agent on this host and return the job folder.

path is one task folder or a dataset folder of tasks. model is the provider/model route of the agent. Harbor writes the job to jobs_dir/job_name. Without job_name, Harbor names the job with the time. attempts is the number of trials for each task. concurrency is the number of trials that run at the same time. agent_options are the agent options of harbor run --ak, for example {"source": "/path/to/vis"}. The call sets HARBOR_TELEMETRY=off, so Harbor sends no usage data. Each trial makes paid model calls. Call it outside a running event loop. With report, the call then writes the round report vis-report.json in the job folder, as python -m blockether.vis.evals.report does. A report error gives a warning and keeps the job folder.

Return the share of items whose samples all agree, as a percent.

Only items with more than one sample count. Return None without such items.

Return Krippendorff's alpha of the samples of each item, rounded to 3 places.

Each item is one unit and each sample one value. Items can have different sample counts. Return None when alpha is undefined, for example with one sample.

Save the human cause of one failed attempt in the labels file at path.

task is a task name or a trial name, as in find_trial. cause is one of CAUSES. The write replaces the file at once. Raises ValueError for an unknown cause, a note longer than NOTE_LIMIT or a solved attempt.

Save your failure stage of one trial in the stage labels file at path.

stage is one of STAGES. A new label of the same trial replaces the old one, and the write replaces the file at once. Raises ValueError for an unknown stage or a note longer than NOTE_LIMIT.

Return the sha256 of a seal body with sorted keys, without its digest key.

Return the seal of a round: its run records, one row for each trial and a digest.

Each run record holds the commit, the pins, the profiles and the artifact sha256 of one run. The rows hold the task checksums and the result digests. sealed_at defaults to the current UTC time.

Return one seal row for each trial, with the task checksum and the result digest.

result is the path of the trial result.json, relative to root, and result_sha256 is the sha256 of that file.

Return the user messages of instruction, one for each turn.

Each line with only TURN_MARKER ends one message and starts the next. An instruction without that line is one turn and stays unchanged. An empty message raises ValueError.

Compare the judged stages with human stage labels of the same trials.

samples maps a trial name to the stage of each judge sample, as from judged_samples. labels maps a trial name to a label with a stage value, as from load_stage_labels. Labels of trials without a judged stage do not count.

Place the round of data, its summary.json data, among the board rows.

Gives None when the summary covers another dataset or version.

Summarize a set of trials: counts, the pass rate with its interval, and the cost.

Return the problems of one task folder that give a wrong score; empty is valid.

The check reads only files and makes no model calls. It needs an executable tests/test.sh and the folder environment/workspace. It checks tests/checks.json with check_problems and needs each Python file that the checks name. The harness writes .vis/config.yml in the trial workspace, so a seed file there is a problem. Each problem starts with the task name. run_tasks and python -m blockether.vis.evals.check refuse tasks with problems.

List the scored attempts of each task in finish order, True when solved.

List the scored attempts of each task in finish order, True when passed.

Describe one attempt of a task for a finding. limit is its agent time limit.

Return the 95% Wilson interval of a pass rate, or None without runs.

size replaces the run count when the runs are not independent.

Write report as JSON to path and replace the old file at once.

Write the Vis task name in the folder dataset and return it.

name is lowercase words joined by hyphens. instruction is the prompt of the agent. A list of strings gives one user message for each turn of one session; write_task joins them with TURN_MARKER lines. checks is the tests/checks.json object that the Vis grader reads, for example {"want": {"m.py": ["def c("]}}. files maps each seed file path to its text. The paths are relative to environment/workspace/. artifacts lists the workspace files that Harbor copies to artifacts/ in the trial folder after the check, for example the call record of a mock extension. filesystem_roots maps each filesystem root id to a workspace folder; the trial registers them. test_files maps each file name in tests/ to its text, for example the verifier that the verify check names. The agent never sees these files. The task gets tests/test.sh, which runs the grader. An existing task folder, an invalid name, invalid checks or a path outside its folder raise an error.