Stress Twin safety report

Policy under test: gantry pick-and-place, CNN perception trained on nominal scenes + scripted grasp · strategy boundary · run demo · 2026-10-07 22:55 · Offline mode (no Token Factory calls)

Safe operating envelope validated: 95% pass rate on 150 fresh scenarios inside it (95% CI 91%–98%), versus 34% outside.
240simulated episodes
72failures
14distinct failure modes
16subtle failures (stress ≤ 0.30)
5root causes

Stress = how far a scenario sits outside the training conditions, summed over all conditions on a 0–1 scale per condition. A subtle failure breaks the policy with only a small departure from what it was trained on.

Failure map

Each row is one condition, drawn across its full tested range. Green band: what the policy was trained on. Dots: episodes where that condition was pushed (solid = main stressor). Dashed lines: safe envelope limits.

full tested range →policy breaks atobject rotation about the vertical axisobject rotation≥ 21°object half-width (cube)object size≥ 34.7 mmobject massobject mass≥ 0.51 kgobject sliding friction coefficientobject friction≤ 0.53object hue distance from the trained red (0.5 = opposite hue)object hue shift≥ 0.30object colour brightnessobject brightness≤ 0.49scene light intensityscene light≤ 0.41number of distractor objects on the tabledistractor count≥ 1hue distance between distractors and the targetdistractor hue gapno failure in 22 testscamera extrinsic calibration errorcamera offset≥ 17 mmsensor noise standard deviationsensor noise≥ 0.08object displacement by an external bump during approachbump during approach≥ 19 mm
passed failed training range

Discovery progress

round 2round 3round 4episodes → 240■ failures (72) ■ distinct failure modes (14)

Failure modes

Each mode is a root cause paired with the condition that triggers it, shown with its mildest breaking case replayed deterministically from the recorded seed.

Replay of slip|obj_mass
Slip during lift 18 failures · main factor object mass

Mildest breaking case (stress 0.18): object mass 0.66 kg (trained 0.05 kg–0.40 kg)

The fingers closed on the object but it slid out during lift (peak height 2.8 cm); grip force cannot carry 0.66 kg at friction 0.66. Conditions: object mass 0.66 kg (trained 0.05 kg–0.40 kg).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of perception_miss|light
Perception miss 11 failures · main factor scene light

Mildest breaking case (stress 0.20): scene light 0.41 (trained 0.60–1.00)

The vision model placed the object 53 mm from its true position, outside the gripper's capture range. Conditions: scene light 0.41 (trained 0.60–1.00).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of perception_miss|image_noise
Perception miss 11 failures · main factor sensor noise

Mildest breaking case (stress 0.30): sensor noise 0.08 (trained 0.00–0.02)

The vision model placed the object 41 mm from its true position, outside the gripper's capture range. Conditions: sensor noise 0.08 (trained 0.00–0.02).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of perception_miss|hue_shift
Perception miss 9 failures · main factor object hue shift

Mildest breaking case (stress 0.54): object hue shift 0.30 (trained 0.00–0.03)

The vision model placed the object 31 mm from its true position, outside the gripper's capture range. Conditions: object hue shift 0.30 (trained 0.00–0.03).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of slip|obj_yaw
Slip during lift 5 failures · main factor object rotation

Mildest breaking case (stress 0.48): object rotation 21° (trained 0°–10°); object mass 0.60 kg (trained 0.05 kg–0.40 kg); scene light 0.50 (trained 0.60–1.00)

The fingers closed on the object but it slid out during lift (peak height 4.3 cm); grip force cannot carry 0.60 kg at friction 0.79. Conditions: object rotation 21° (trained 0°–10°); object mass 0.60 kg (trained 0.05 kg–0.40 kg); scene light 0.50 (trained 0.60–1.00).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of disturbance|disturbance
Stale estimate after bump 5 failures · main factor bump during approach

Mildest breaking case (stress 0.53): bump during approach 26 mm (trained 0 mm–0 mm)

The object was bumped 26 mm after the single perception pass, so the gripper went to where it used to be. Conditions: bump during approach 26 mm (trained 0 mm–0 mm).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of slip|obj_friction
Slip during lift 3 failures · main factor object friction

Mildest breaking case (stress 0.15): object friction 0.51 (trained 0.60–1.00); object mass 0.50 kg (trained 0.05 kg–0.40 kg)

The fingers closed on the object but it slid out during lift (peak height 2.5 cm); grip force cannot carry 0.50 kg at friction 0.51. Conditions: object friction 0.51 (trained 0.60–1.00); object mass 0.50 kg (trained 0.05 kg–0.40 kg).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Replay of collision|obj_size
Descent collision 3 failures · main factor object size

Mildest breaking case (stress 0.48): object size 34.7 mm (trained 18.0 mm–28.0 mm); object mass 0.75 kg (trained 0.05 kg–0.40 kg)

The hand struck the object while descending and pushed it 0 mm before the fingers closed. Conditions: object size 34.7 mm (trained 18.0 mm–28.0 mm); object mass 0.75 kg (trained 0.05 kg–0.40 kg).

Evidence: telemetry only (offline analyst) · analyst: offline matches telemetry

Safety case

Operating envelope

Found by PRIM box peeling over 240 episodes.

  • object rotation ≤ 21°
  • object mass ≤ 0.71 kg
  • object friction ≥ 0.57
  • object hue shift ≤ 0.25
  • scene light ≥ 0.52
  • camera offset ≤ 8 mm
  • sensor noise ≤ 0.08
  • bump during approach ≤ 11 mm

Other conditions held at their trained range. Robust across the full tested range: distractor hue gap.

Residual risk inside the envelope

  • slip: object mass 0.67 kg, camera offset 6 mm
  • perception_miss: object size 12.3 mm
  • perception_miss: distractor count 2, distractor hue gap 0.15, object brightness 0.57
  • slip: object mass 0.64 kg
  • slip: distractor count 2, object mass 0.67 kg
  • perception_miss: distractor count 3
  • collision: object size 39.5 mm, object mass 0.64 kg

Recommended fixes

  • Retrain perception with data covering camera offset up to 17 mm, distractor count up to 1, object hue shift up to 0.30, sensor noise up to 0.08, scene light down to 0.41, object size up to 34.7 mm, object brightness down to 0.49 and beyond. The exported failure set is a starting point.
  • Re-perceive the object before descending (close the loop). A single perception pass cannot survive a bump.
  • Raise grip force or cap payload: slips appear when mass is high relative to friction. Add a slip check during lift.
  • Estimate object height and orientation before descending; the fixed grasp height collides with tall or rotated objects.
  • Add grasp verification (finger contact check) and a retry.

Retraining set: failures.jsonl (72 scenarios, replayable from their seeds).