Zhuang Chang

Zhuang Chang

Cross-Reality Agent; Biosensing; HCI

LLM-based Conversational Virtual Agents in VR for Traumatic Brain Injury Rehabilitation

This project builds upon the Pre-scripted agents for traumatic brain injury (TBI) project, which focuses on developing a Virtual Reality (VR) system aimed at rehabilitating individuals with cognitive fatigue resulting from TBI. The previous work culminates in the creation of a virtual café, a simulated environment where TBI patients relearn essential life skills in a realistic setting. The effectiveness of this VR tool is validated through a successful clinical trial involving 11 TBI clients and 13 clinicians. The trial demonstrates that clinicians use the VR system to observe patients during therapy sessions, moving beyond reliance on client feedback or hypothetical scenarios. Additionally, the VR system proves beneficial for a variety of therapeutic interventions and integrates seamlessly into clinical practice.

Feedback from the trial highlights several areas for improvement to enhance clinical uptake and patient outcomes. The current project addresses these key changes:

  • Portability and Ease of Setup: To increase clinical adoption of the VR system, it becomes more user-friendly and transportable. This project develops a stand-alone VR application that operates on self-contained VR headsets, eliminating the need for complex setups and making it easier to use in different clinical settings.

  • Enhanced Interaction with AI-Driven Characters: Feedback from the initial trials indicates that scripted interactions with virtual characters (playing the roles of friends and café staff) feel unnatural, limiting the transfer of skills from the VR environment to real life. To address this, the project integrates AI-driven virtual characters capable of engaging in natural conversations. This enhancement creates a more immersive and effective therapeutic experience.

  • Customization Tools for Clinicians: To tailor the VR experience to individual patient needs, the project develops tools that allow clinicians to customize therapy sessions. This customization enables more targeted and effective rehabilitation interventions.

The existing TBI VR application is restructured into a modular format, enhancing customization for a broader spectrum of treatments. To demonstrate this versatility, the project includes an additional experience: a virtual café scenario.

The conversational system, designed in collaboration with Iwi partners (IUE Ltd, He Waka Tapu), undergoes validation through user testing with therapists and clients from the Laura Ferguson Brain Injury Trust (LFBIT), continuing the co-design process.

In summary, this project refines and enhances the existing VR rehabilitation system by making it more portable, improving the naturalness of interactions through AI, and providing clinicians with tools to customize therapy. These advancements ensure the VR system is more widely adopted and effective in rehabilitating individuals with TBI.