Agent Assessment and Task Environment HOT

The task is derived from a real career delivery scene, and the rating criteria are set by professional experts to avoid “failure to work in examinations”.

Mission environment Docker is re-established and state can be asserted in scripts, and the results can be compared across teams and horizontally.

Priority is given to building the current directions of SOTA's underperformance (TB 3.0% only 15-26%, Automation Bench 18-30%), opening the model gap.

The same task can be used both for post-training (the standard gold-bearing track) and for evaluation purposes directly.

Single category: 1,500–12 000 mission environments; support delivery by occupation, tool type, difficulty classification.

Docker mirror / Simulate Office Environment / Real Terminal Sandbox three types of scripts that are accompanied by initialization and reset.

Task level target + step level action sequence + status assertion point + rating rubric level four label.

The status of the status of the current list of unsaturated slots is divided into the three tranches of Standard / Hard / Frontier, which is the current file alignment.

Provide model manufacturers and assessment agencies with reversible, anti-pollution high-level rating sets.

The RL rewards the signal by using gold standard tracks and assertions feedback to train real capacity to work.

Horizontal comparison of Agent products with real business process tasks to underpin procurement decisions.

Research and assessment of safety-related capabilities such as RSI, rewarding, ultra vires operations.

Task package (Docker mirror + initial state) + Verify script + standard gold track JSONL + rubric score sheet.

task_id、env_image、goal、toolset、assertions、gold_trajectory、rubric、difficulty、metrics。

Environmental operationality 100%, assured script certainty verification, standard gold track manual review pass rate 98%.

Standard 4-6 weeks; occupational and end-of-life environmental 6-8 weeks; support for rolling amplification.

Commercial use authorization, full de-identification of operational data, and data processing and compliance documentation.

#!/usr/bin/env rash
You know, set-e.
# Claim 1: Service started and monitored 8080
It's all right, it's all right.
# Claim 2: Profile strengthened
"Permit RootLogin no" /etc/ssh/shd config exit 1
# Claim 3: Return test passed
cd /app & /amp; /amp; pytest-q testes/ tail-1 root-q "passed" exit1
"PASS."

Other Organiser
"Task id": "Toolathlon-0213",
"goal": "Check last week's orders and generate reconciliation forms for finance."
"gold security":
"Tool": "db.query", "args": "range": "last week" ,
"Tool": "sheet."
"Tool": "mail.send", "args": "to": "finance@corp.com", "attach": "$2.file"
I don't know.
"fallback with csv export"
"Metrics":
♪ I'm sorry ♪