Feedback defines progress
Multi-view images, robot state, gripper state, and external torque anchor transitions to physical events rather than elapsed time or appearance alone.
Bio-Inspired Phase-Aware Skill Graph
MPSG-Control decomposes multimodal robot interaction into physically grounded phases, organizes reusable skills as a perception-conditioned graph, and compiles language instructions into execution-time topology constraints.
Public method diagrams and aggregate results. The manuscript and unfinished publication metadata remain private.
Why phase-aware?
Monolithic policies let a local perception or contact error propagate through the remaining trajectory. MPSG-Control inserts an explicit phase abstraction between low-level skills and high-level intent.
Multi-view images, robot state, gripper state, and external torque anchor transitions to physical events rather than elapsed time or appearance alone.
When the observed state disagrees with the active phase, the graph can suppress advancement, revisit a predecessor, or skip work already completed externally.
Method
MPSG-Control separates phase perception, graph-structured execution, and semantic intervention while keeping them connected through explicit phase state.
The Multimodal Phase-and-Primitive Detector fuses vision, proprioception, and torque, proposes physically meaningful boundaries, and refines them with MS-TCN.
The Multimodal Perception Skill Graph represents conditioned skills as nodes and nominal, skip, rollback, and recovery transitions as edges.
Promptable Phase Control turns instructions into node masks, edge pruning, pointer updates, and hard ordering constraints without retraining skill policies.
Real-world experiments
The evaluation covers sequential dependencies, contact-rich occlusion, geometric variation, and perceptual state ambiguity.
Execution demos
Selected MPSG visualizations pair the robot stream with predicted skills, path progress, and the active graph node.
Results
Multimodal phase perception sharpens boundaries; graph execution localizes recovery; prompt constraints preserve the requested task order.
Across four real-world tasks with runtime intervention.
For prompt-driven task reordering with hard graph constraints.
While maintaining the requested execution order.
| Configuration | F1@25 | F1@50 | Frame accuracy | mIoU |
|---|---|---|---|---|
| Vision | 9.2 | 27.5 | 52.7% | 39.9% |
| Vision + proprioception | 31.7 | 66.7 | 61.7% | 50.7% |
| Vision + proprioception + torque | 75.3 | 85.0 | 87.2% | 75.9% |
Paper-to-code map
The repository is research code. This map distinguishes reusable structures from integrated prototypes and experiment scripts.
phaseact/phase_detector/ms_tcn_demo_*.py
Dual-camera fusion, semantic features, torque-aware change points, MS-TCN training and inference.
Browse phase detector →phaseact/phase_detector/mpsg_new.py
Graph creation, perception matching, path tracking, recovery behavior, and visualized execution.
Open MPSG controller →phaseact/skillgraph_control/
Directed graph primitives, visualization, and graph construction from segmented demonstrations.
Browse graph code →mpsg_new.py control paths
Integrated research functionality for graph constraints; not yet a standalone user-facing package.
Read implementation notes →Reproduce and extend
The README documents environment setup, local data layout, actual command-line arguments, evaluation utilities, and generated outputs.