- Published on
Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G
- Authors
- Name
- DeepHorus
New 5G Sensing System Enables Full-Body, Finger-Level Avatar Control for Immersive XR
A novel approach to Extended Reality (XR) interaction, leveraging the dual purpose of 5G millimeter-wave signals, promises to deliver truly immersive and responsive avatar control. Researchers have proposed a multimodal sensing architecture that uses integrated sensing and communication (ISAC) from 5G networks for full-body pose estimation, combined with lightweight sensors for detailed finger movement. This system addresses critical limitations of current camera and controller-based methods, which struggle with full-body tracking, require clear lines of sight, and obstruct natural hand use.
The core innovation lies in repurposing the very 5G signals that deliver high-bandwidth, low-latency content to a head-mounted display (HMD). Instead of using these millimeter-wave (mmWave) signals solely for data transmission, the system employs them as a radar-like sensing tool. By analyzing standard beam management procedures from the 5G New Radio (NR) standard, the network can derive a user's coarse body-level gestures and poses. This process, known as Power-per-Beam-Pair (PPBP) sensing, allows for real-time avatar movement that mirrors the user's physical actions without requiring additional complex hardware.
For the fine-grained control that makes virtual interaction feel natural, the architecture incorporates surface electromyography (sEMG) sensors. These lightweight, wearable sensors are placed on the forearm to capture the electrical activity of muscles, translating subtle muscle twitches into precise finger and hand gestures. This dual approach creates a complete framework: the 5G network handles large-scale body movements, while the sEMG sensors manage the intricate dexterity of the hands and fingers.
In evaluations detailed in a paper posted to arXiv on June 22, 2026, the researchers demonstrated the system's effectiveness. The PPBP-based body-level sensing achieved an average accuracy of 82.2% (±5.9%) when tested on users not seen during the training phase. For the finger-level interactions, the team showed that sEMG signals carry strong discriminative information, yielding consistent and promising performance across various movement settings. This combination mitigates the weaknesses of each individual modality.
The implications for the XR industry are significant. Current solutions like handheld controllers limit user freedom, while camera-based tracking can be obstructed by objects or poor lighting. By utilizing existing 5G infrastructure and minimal wearable sensors, this architecture could dramatically lower barriers to creating truly interactive virtual and augmented worlds. It paves the way for applications ranging from immersive remote collaboration and teleoperation to advanced gaming and virtual fitness, where natural, full-body interaction is essential.
The research team, including Nabeel Nisar Bhat and Jeroen Famaey, highlights that this work moves beyond the conventional view of 5G as merely a data pipe. "Our architecture reframes the network itself as a distributed sensor," the paper suggests, enabling a symbiotic relationship between communication and environmental perception. This paradigm is seen as a crucial stepping stone toward the advanced capabilities anticipated for 6G networks.
Looking ahead, the outlook focuses on refinement and real-world testing. Future work will involve optimizing the algorithms for even higher accuracy and lower latency, ensuring seamless synchronization between the 5G-derived body pose and the sEMG-derived finger gestures. As the technology matures, this integrated sensing and communication model could become a foundational element for next-generation XR experiences, making virtual worlds more responsive and accessible than ever before.
