Bio-Inspired Phase-Aware Skill Graph

Structured motor phases for robust long-horizon manipulation.

MPSG-Control decomposes multimodal robot interaction into physically grounded phases, organizes reusable skills as a perception-conditioned graph, and compiles language instructions into execution-time topology constraints.

Public method diagrams and aggregate results. The manuscript and unfinished publication metadata remain private.

Why phase-aware?

Local motor stability. Global structural flexibility.

Monolithic policies let a local perception or contact error propagate through the remaining trajectory. MPSG-Control inserts an explicit phase abstraction between low-level skills and high-level intent.

Feedback defines progress

Multi-view images, robot state, gripper state, and external torque anchor transitions to physical events rather than elapsed time or appearance alone.

RGB+State+Torque

Structure contains failure

When the observed state disagrees with the active phase, the graph can suppress advancement, revisit a predecessor, or skip work already completed externally.

Nominal↶Rollback⇢Skip

Method

Three modules, one execution substrate.

MPSG-Control separates phase perception, graph-structured execution, and semantic intervention while keeping them connected through explicit phase state.

01 · PPD

Detect the interaction phase

The Multimodal Phase-and-Primitive Detector fuses vision, proprioception, and torque, proposes physically meaningful boundaries, and refines them with MS-TCN.

02 · MPSG

Execute through a skill graph

The Multimodal Perception Skill Graph represents conditioned skills as nodes and nominal, skip, rollback, and recovery transitions as edges.

03 · PPC

Reconfigure with constraints

Promptable Phase Control turns instructions into node masks, edge pruning, pointer updates, and hard ordering constraints without retraining skill policies.

MPSG-Control framework with biological inspiration, multimodal phase detection, skill graph execution, and promptable graph constraints
From feedback-driven motor primitives to multimodal phase perception, graph execution, and prompt-level structural control.

Real-world experiments

Four tasks expose four kinds of brittleness.

The evaluation covers sequential dependencies, contact-rich occlusion, geometric variation, and perceptual state ambiguity.

Four rows of robot task images showing sequential manipulation, charger interaction, block stacking, and switch manipulation
Four real-world, multi-stage manipulation settings.
01
Sequential pick-and-placeTests multi-stage dependencies and order-sensitive execution.
02
Charger interactionTests contact events hidden by severe visual occlusion.
03
Multi-block stackingTests accumulated drift under randomized initial geometry.
04
Switch manipulationTests symmetric states that look identical before and after action.

Execution demos

See the graph state move with the robot.

Selected MPSG visualizations pair the robot stream with predicted skills, path progress, and the active graph node.

Sequential manipulationSkill prediction and path tracking across drawer and object phases.
Block stackingPerception-conditioned progression through a longer skill path.
Charger interactionGraph progress through contact-rich extraction and insertion phases.

Results

Structure improves robustness without giving up control.

Multimodal phase perception sharpens boundaries; graph execution localizes recovery; prompt constraints preserve the requested task order.

73.8%Average disturbed-task success

Across four real-world tasks with runtime intervention.

100%Order compliance

For prompt-driven task reordering with hard graph constraints.

80%Recovery success

While maintaining the requested execution order.

Bar chart comparing vision, vision plus proprioception, and vision plus proprioception plus torque for four segmentation metrics
Vision + proprioception + torque reaches 87.2% frame accuracy and 75.9% mIoU.
Examples of rollback after an external state reset and phase skipping after external task completion
Perception-grounded rollback and skip after runtime intervention.
Prompt-control comparison showing MPSG-Control order compliance, task success, and recovery success
Hard graph constraints resolve the trade-off between strict ordering and recovery.
ConfigurationF1@25F1@50Frame accuracymIoU
Vision9.227.552.7%39.9%
Vision + proprioception31.766.761.7%50.7%
Vision + proprioception + torque75.385.087.2%75.9%

Paper-to-code map

Find the implementation behind each idea.

The repository is research code. This map distinguishes reusable structures from integrated prototypes and experiment scripts.

PPD · Phase detection

phaseact/phase_detector/ms_tcn_demo_*.py

Dual-camera fusion, semantic features, torque-aware change points, MS-TCN training and inference.

Browse phase detector →

MPSG · Integrated execution

phaseact/phase_detector/mpsg_new.py

Graph creation, perception matching, path tracking, recovery behavior, and visualized execution.

Open MPSG controller →

Graph structures

phaseact/skillgraph_control/

Directed graph primitives, visualization, and graph construction from segmented demonstrations.

Browse graph code →

PPC · Prompt constraints

mpsg_new.py control paths

Integrated research functionality for graph constraints; not yet a standalone user-facing package.

Read implementation notes →

Reproduce and extend

Start from the code, with the limits made explicit.

The README documents environment setup, local data layout, actual command-line arguments, evaluation utilities, and generated outputs.

Reproduction entry points

  • Multimodal phase detection with explicit local paths
  • MPSG video visualization and subgraph tracking
  • Frame accuracy, mIoU, boundary F1, edit score, and over-segmentation evaluation
Open quick start

Current research scope

  • Workstation-specific defaults remain in some scripts
  • PPC is integrated rather than separately packaged
  • Recovery depends on demonstrated or validated traces
  • Robot execution requires experiment-specific hardware and policies
See output mapping