Whole-Body Control of a Dexterous Hand–Robot System
Designed a kinematic-twin teleoperation system for whole-body control of a dexterous hand–arm robot. Direct joint mapping enables intuitive operation, while a real-time MuJoCo digital twin supports visualization and validation of teleoperation commands.
To extend the reachable workspace and enable intuitive whole-body teleoperation of the DeltaHand–Franka system, We developed a kinematic-twin-based teleoperation framework. The robotic platform consists of a sensorized DeltaHand mounted on the end effector of a Franka robotic arm. The DeltaHand is a multi-fingered, non-anthropomorphic robotic hand equipped with tactile sensing.
The teleoperation interface is built around a TeleHand–GELLO system that serves as a kinematic twin of the DeltaHand–Franka robot. During teleoperation, the operator can manipulate the TeleHand to simultaneously control both the arm and hand motions of the robot. Motion of the TeleHand determines the end-effector pose of the Franka arm, while the TeleHand finger joints directly command the corresponding joints of the DeltaHand. By leveraging a kinematic-twin design, the system enables direct joint-to-joint mapping, resulting in intuitive and low-latency control.
To facilitate visualization and verification of the teleoperation signals, I also created virtual counterparts of both the TeleHand–GELLO interface and the DeltaHand–Franka system in MuJoCo. These simulated models receive the teleoperator joint outputs as commands, providing real-time visualization of the commanded robot configuration and serving as a digital twin of the physical teleoperation system.