Variational Automation Benchmark for Agentic Robotics
For robots to operate reliably in commercial and industrial applications, they must persistently perform specific tasks with known variations in object shape and pose. To differentiate these settings from fixed automation, we call this Variational Automation (VA). Recent work on agentic robotics combines the interpretable robot programming of pretrained LLM and VLM coding agents with the open-world adaptability of model-free policies.
We propose a general set of benchmark design criteria emphasising accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency. We instantiate these principles in six manipulation task classes spanning industrial, medical and infrastructure applications: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion and cable routing.
We introduce the Variational Automation Benchmark for Agentic Robotics (VABAR), a new, open, easily replicable, simulation-based manipulation benchmark for systematically evaluating agentic robotics systems. Built fully on open-source tools, VABAR features detailed implementations using the Newton physics engine, standardised task variations, difficulty tiers, success metrics, and a library of 33 generalizable skills with human-authored programmatic reference solutions. We provide initial baseline evaluations using an open agentic manipulation harness, Graph-as-Policy (GaP), and an open version on Isaac Lab Arena.
Five things this work puts on the table.
A manipulation class between fixed automation and unconstrained general-purpose manipulation, in which persistent operational objectives must be achieved across a structured distribution of task instances.
Practical criteria for evaluating agentic systems: accessibility, repeatability, controlled diversity, application relevance, physical realizability, resistance to saturation, and computational efficiency.
Tool sorting, tool hanging, gear meshing, syringe packing, cable insertion and cable routing — with standardised variations, difficulty levels, success metrics, and support for single-arm Franka and bimanual YAM embodiments.
For each task: a photorealistic simulation environment, 33 generalizable skills, and a human-authored reference solution expressed as a computation graph over the same skill library.
Initial baseline evaluations characterising performance as task complexity and instance variation increase, establishing reference results and identifying remaining challenges for agentic manipulation.
A VA task class is a tuple. An instance is a draw from it. Admissibility — not randomness — is what makes the draw meaningful.
An instance τi = {(aj, xj)}j=1..Ni ∈ O names each instantiated asset aj ∈ A and its initial pose xj ∈ SE(3), for all Ni objects and fixtures in the scene. Instances are drawn τi ~ pO under a controlled seed.
Five constraints on admissible variation
The point of the constraint list: O is not defined by unconstrained scene randomization. The variation space and its sampling distribution are designed to represent meaningful changes in the operating conditions of an automation task, where variation is expected rather than anomalous.
A benchmark for agentic robotics has to evaluate task outcomes without prescribing a program representation or agent architecture, so that generated-code, computation-graph, tool-use, learned-policy and hybrid systems can be compared under common conditions. Seven criteria follow.
Support agents that construct, test and revise robot programs from task specifications, with documented robot and tool interfaces and execution feedback for refinement — without prescribing a program representation.
Tasks and variations reflect realistic operating requirements, with results predictive of real-world success, cycle time and intervention needs.
Environments, task specifications, assets, metrics and variation parameters are publicly available and clearly documented, so independent researchers can reproduce and extend the evaluation.
Each task specifies explicit variation axes and difficulty levels rather than relying solely on unconstrained randomization.
Task generation and evaluation use controlled random seeds and standardised success criteria, supporting repeatable experiments and meaningful comparison between methods.
Evaluation stays cheap enough to support repeated experimentation, sensitivity analysis and large-scale benchmarking.
Systematic extensions of variation range and task complexity keep the benchmark informative as systems improve; extensions are versioned to preserve comparability with previously reported results.
Task-level success says whether an execution achieved its objective, not where or why it failed. That matters for agentic robotics, where execution feedback is used to diagnose failures, revise programs and introduce recovery behaviours. VABAR records the evidence needed to relate outcomes to both the program's decisions and the resulting physical interaction.
Fig. 3 — Diagnostic execution flow. Dashed: privileged simulator state. Dotted: physics contact information used for diagnosis.
The four mappings that localise a failure
Object poses and velocities describe the resulting scene configuration; contact records describe how that configuration arose. A failed placement may leave an object in an incorrect pose, while its contact history reveals whether it collided with a fixture, was supported by another object, or became jammed. VABAR records both so that failure attribution is grounded in the physical execution rather than in task-level success or software error messages alone.
Four components hold the suite together: a standardised asset library, representative single- and dual-arm embodiments, a declarative task specification, and multi-physics simulation support.
Asset library
Industrial tools, mechanical components, medical supplies, fixtures, connectors and deformable objects. Assets are normalised in format, scale, appearance and collision geometry to support controlled object and geometric variation across task instances.
Robot embodiment
Franka provides a widely adopted platform for reproducible tabletop manipulation; YAM enables tasks requiring coordinated bimanual interaction, particularly deformable-object manipulation.
Task specification
Tasks declare their objects, initial configurations, variations and success conditions. The Behavior Domain Definition Language is extended to represent both rigid and deformable objects as one specification across the full suite.
Physics
NVIDIA Newton provides unified rigid-body and deformable-body dynamics and their interactions, letting the suite span rigid manipulation, contact-rich assembly and insertion, and deformable tasks such as cable routing.
Each class fixes an operational objective and an application setting, then names the axes along which instances vary. Clips are path-traced replays of recorded episodes, played back faster than real time; success figures are the human-authored reference graph on Newton.
| Task | Level 1 | Level 2 |
|---|---|---|
| 1 · Tool Sorting | Singulated tools; no distractors | Cluttered tools |
| 2 · Tool Hanging | One tool; known fixture pose | Multiple tools and fixture types; randomized poses |
| 3 · Gear Meshing | Assemble two gears | Assemble three gears |
| 4 · Syringe Packing | Singulated targets | Cluttered scenes |
| 5 · Cable Insertion | Fixed, visible port; nearby connector with small misalignment | Randomized port pose; moderate connector misalignment |
| 6 · Cable Routing | 2–3 widely spaced pegs; short route; low turning angle | 4–6 moderately spaced pegs; multiple turns; moderate initial cable variation |
Tasks 1–4 run on the single-arm Franka; tasks 5 and 6 use the bimanual YAM.
Task 01
The robot must place industrial tools into corresponding bins. Instance variation is introduced by changing the tools, their poses, and tool density.
Reference graph (Newton) — L1 47.1% · 421.6 sL2 33.7% · 913.8 s
Task 02
The robot must pick up one or more industrial tools — wrenches, screwdrivers, hammers, scissors — and hang or insert them into designated hooks, clips or slots on a pegboard. Variation comes from tool type, the number of tools, initial tool pose and holder pose.
Reference graph (Newton) — L1 86.7% · 86.1 sL2 81.7% · 158.4 s
Task 03
The task input is gears and axles. The robot must pick up multiple gears, assemble them onto axles so that they properly mesh, and press a button to activate the mechanism. Variation covers the number of gears, gear size and initial gear poses. To ensure physically valid instances, scenes can be initialised from a valid assembled configuration before selected gears are removed and repositioned.
Reference graph (Newton) — L1 100.0% · 165.9 sL2 90.0% · 235.5 s
Task 04
The robot must place syringes into a disposal container while avoiding distractor objects. Instance variation includes the number of objects and the disposal box type.
Reference graph (Newton) — L1 87.0% · 228.8 sL2 63.0% · 472.3 s
Task 05
The robot must pick up a connector, align it with a USB-C port and fully insert it. Instance variation includes the position and orientation of the connector and the port.
Reference graph (Newton) — L1 86.7% · 78.8 sL2 30.0% · 74.2 s
Task 06
The robot must route a flexible cable around a specified ordered sequence of cylindrical fixtures mounted on a planar routing board. The task varies the number, position and spacing of pegs; the ordered target sequence; cable length; initial cable configuration; cable slack and curvature; endpoint position; and the 3D pose of the routing board.
Reference graph (Newton) — L1 46.7% · 116.0 sL2 46.7% · 189.8 s
Every clip below is a flip-book of initial conditions, not a trajectory. Each frame is one independently seeded instance of the same task class, path traced from the scene a policy actually opens on, and held for half a second. Level 1 and Level 2 run the same sampler under different bounds — what changes between the two columns is the width of the admissible region, not the objective.
Task 01
The tools present, their poses, and tool density.
Tool type · clutter · bin occupancy · overhead camera
Task 02
Tool type, the number of tools, initial tool pose, and holder pose.
Tool type · fixture type · object and fixture poses · side camera
Task 03
The number of gears, gear size, and initial gear poses.
Gear count · gear size · initial pose · front camera
Task 04
The number of objects and the disposal box type.
Syringe type · clutter · disposal geometry · agent view camera
Task 05
The position and orientation of both the connector and the port.
Connector pose · port pose · top down camera
Task 06
Peg number, position and spacing; the ordered target sequence; cable length, slack and curvature; and the 3D pose of the board.
Fixture layout · route · cable type · overhead camera
Instances are drawn under a controlled seed, so re-running a document reproduces the same scenes. Counts differ by task because each document declares its own episode count.
VABAR extends the Modular Open Robot Skill Library with reusable procedures for functional-feature manipulation, constrained placement and deformable cable handling. Building on existing perception, grasp generation and collision-aware planning primitives, the extensions provide metric feature localization, source–destination correspondence, in-hand registration and staged feature mating. Cable skills estimate and track ordered 3D centerlines and support anchoring, routing and terminal placement.
Three questions drive the evaluation: how do success rate and cycle time change as the magnitude and dimensionality of task variation increase; what are the failure modes; and can agentic systems reuse skills as task difficulty changes.
Newton. VABAR is developed on the Newton physics engine. Success conditions extend the Behavior Domain Definition Language to represent both rigid and deformable objects as a common specification across the full task suite.
Isaac Lab Arena (ILA). A second implementation runs on NVIDIA Isaac Lab Arena, a framework for composing, executing and analysing policy-evaluation benchmarks with GPU acceleration. ILA supplies environment composition — including object placement from semantic spatial relationships — multiple physics backends including Newton, parallel GPU evaluation, and robustness analysis across environment and task variations. It is executed on a cluster of NVIDIA L40 GPU nodes. The two implementations differ in texture, lighting, assets and robot implementation.
Four evaluated systems
Each task–difficulty combination contains N = 30 independently generated instances per evaluation condition, shared across conditions.
| Task | Diff. | a. GaP (Human–Newton) | b. GaP (Human–ILA) | c. Claude Code | d. GaP | ||||
|---|---|---|---|---|---|---|---|---|---|
| Succ. | Cyc. (s) | Succ. | Cyc. (s) | Succ. | Cyc. (s) | Succ. | Cyc. (s) | ||
| Tool Sorting | L1 | 47.1 | 421.6 | 20.0 | 555.0 | 0.0 | — | 26.7 | 134.2 |
| L2 | 33.7 | 913.8 | — | — | 0.0 | — | 13.3 | 232.3 | |
| Tool Hanging | L1 | 86.7 | 86.1 | 67.5 | 98.3 | 16.7 | 96.2 | 33.3 | 81.5 |
| L2 | 81.7 | 158.4 | 50.0 | 138.3 | 10.0 | 190.8 | 10.0 | 270.9 | |
| Gear Meshing | L1 | 100.0 | 165.9 | 87.5 | 198.9 | 0.0 | — | 70.0 | 156.6 |
| L2 | 90.0 | 235.5 | 80.0 | 381.3 | 0.0 | — | 60.0 | 284.1 | |
| Syringe Packing | L1 | 87.0 | 228.8 | 52.5 | 939.0 | 0.0 | — | 33.3 | 371.1 |
| L2 | 63.0 | 472.3 | 40.0 | 1810.0 | 0.0 | — | 12.1 | 652.2 | |
| Cable Insertion | L1 | 86.7 | 78.8 | 97.5 | 104.9 | 0.0 | — | 36.7 | 79.0 |
| L2 | 30.0 | 74.2 | 17.5 | 118.8 | 0.0 | — | 23.3 | 80.5 | |
| Cable Routing | L1 | 46.7 | 116.0 | 50.0 | 145.9 | 3.3 | 120.7 | 36.7 | 50.8 |
| L2 | 46.7 | 189.8 | 5.0 | 151.5 | 0.0 | — | 13.3 | 120.3 | |
Succ.: success rate (%); bar length encodes the same number. Cyc.: median cycle time in simulated seconds among successful trials; a dash means no successful trial to take a median over. GaP (Human) is a manually specified computation graph executed on the same perception, skill, planning and execution stack as GaP. Each task–difficulty condition contains N = 30 independently generated instances shared across all evaluation conditions, except GaP (Human–ILA), measured on its own instances at N = 40 per variant.
The evaluation unit is set by the dependency structure of each task. For tasks composed of repeated manipulation attempts that are independent by construction — sorting or hanging independently sampled objects — each object-level attempt is a trial, and success rate is the fraction of objects successfully manipulated. This avoids assigning zero success to an otherwise largely successful multi-object execution.
In tasks such as Gear Meshing, later insertions are conditioned on the states produced by earlier ones: inserting one gear changes the geometry and contact configuration the next gear meets. Multiple cable insertions count as separate trials only when each is independently initialised or reset. For sequential, state-dependent tasks the complete instance is the evaluation unit, with subgoal-level success reported as a diagnostic.
Cycle time is the simulated time elapsed from policy start until success, failure or timeout. The table reports the median among successful trials only.
For each unsuccessful execution we record the observed failure outcome and assign a primary failure source when supported by execution evidence. For cascading failures we attribute the execution to the earliest supported failure that explains the subsequent outcome. Failure-source percentages are conditioned on unsuccessful executions.
Aggregate success rates do not say which part of the stack becomes unreliable as variation grows. Unsuccessful GaP executions decompose into perception, grasp and motion-planning failures — one primary source each.
Using the same underlying language model, GaP achieves higher success than Claude Code in 11 of 12 task–difficulty conditions. On Gear Meshing it reaches 70.0% and 60.0% at Levels 1 and 2 where Claude Code records no successful trial. GaP still sits below the human-authored graphs on Newton in every condition, despite sharing the same perception, skill, planning and execution stack — so reliable skill composition and parameterization remain open problems for autonomous policy generation. These comparisons evaluate the combined system; they do not isolate the harness from the skill library.
In low-variation settings, perception and grasp generation are near-perfect across most tasks, leaving motion planning and trajectory execution as the primary failure source. As variation increases the dominant mode branches with the physics of the task: in cluttered scenes and deformable-object manipulation — syringe packing, cable routing — perception and grasping degrade sharply under visual occlusion and unmodelled cable mechanics. In precision-contact tasks such as USB-C insertion, state estimation stays robust while motion planning and fine insertion alignment account for nearly all failures.
Pairing task-level verification with simulator ground-truth inspection identifies the physical conditions behind a failure. Two jammed syringe-packing episodes, t01 and t29, terminate with final verification reporting blocked and 7/8 and 6/8 syringes posted. In both, a syringe straddles the lid plane — needle end inside the cavity, opposite end above the opening — consistent with incomplete passage into the container. The surrounding syringes sit in different configurations, showing execution-induced variation: earlier placements alter the geometry encountered by later insertions.
Despite spanning rigid-object manipulation, contact-rich assembly, precision insertion and deformable-object manipulation, VABAR policies reuse skills heavily. Core robot and perception capabilities — end-effector pose retrieval, SAM3-based segmentation — are shared across all six task classes, and higher-level semantic skills are reused across tasks with different objects but similar manipulation structure: Tool Hanging and Syringe Packing both rely on held-object motion, functional-feature perception, object registration and feature mating. Increasing task diversity does not necessarily require an entirely new set of skills. The central challenge is selecting the right skills, parameterizing them for the current geometry and task state, and composing them into task-specific behaviour.
The jam, recorded
Recovery behaviour, with and without the unjam skill
VABAR is built entirely on open-source tools and runs on a GPU workstation as well as on Isaac Lab Arena. Everything below is released with the paper; links are withheld here because they would identify the authors during double-blind review.
Task environments, instance generator, evaluator and baseline runner, on both the Newton and ILA implementations.
55+ assets normalised in format, scale, appearance and collision geometry.
Perception, grasping, motion planning and execution, and verification — with human-authored reference graphs per task.
The N = 30 instance sets per task–difficulty cell used for every reported number.
Citation
@inproceedings{anonymous2027vabar,
title = {VABAR: A Variational Automation Benchmark
for Agentic Robotics},
author = {Anonymous},
booktitle = {Under review, IEEE International Conference on
Robotics and Automation (ICRA)},
year = {2027},
note = {Anonymous submission}
}