Pre-Scripted Virtual Agents in VR for Traumatic Brain Injury Rehabilitation

This project develops new technology for the existing rehabilitation industry – a knowledge-intensive industry. Rehabilitation systems are traditionally customized for patients, relying on highly trained clinicians. To enhance this, we will develop a new type of virtual agent, the Empathic Virtual Character (EVC), and use it for cognitive rehabilitation. Our EVC will combine Virtual Reality (VR), Artificial Intelligence (AI), physiological sensing, and Empathic Computing to identify and respond to patient needs in therapeutic VR environments.

Over 36,000 NZers experience Traumatic Brain Injury (TBI) each year, many of whom are young males of Māori and Pacific ethnicities. The cost to ACC is over $83 million/year, second only to stroke in terms of impact on employment and income [1][2]. VR has been shown to be effective for rehabilitation of TBI patients, but current systems require the patient to travel to a therapist’s location and the VR application cannot adapt to patient needs, making it less effective than it could be.

We will explore how EVCs can be developed that use a range of physiological sensors integrated in VR displays to respond to the patient’s emotional and cognitive state and adjust the VR therapy. The main hypothesis is that VR therapy with EVCs will be more effective than current VR approaches because the therapy can be adapted to individual needs. We will also explore how EVCs can be used in collaboration with a remote therapist in the VR environment, facilitating remote treatment.

This builds on our previous work exploring VR environments for TBI rehabilitation, focusing on cognitive fatigue in a social setting [3]. That research found it will be necessary to create non-scripted conversations between the patient and more realistic adaptive characters, and for the VR experience to respond more effectively to patient progress by gradually becoming more complex.

This project will extend traditional TBI treatment by enabling it to occur in the patient’s home with a remote therapist and by adapting to the needs of the individual patients. This will transform existing cognitive rehabilitation approaches by enabling care to be provided at any location and any time suitable to the patient. Using EVCs for treatment will also have great potential in other healthcare areas, such as counseling, or stroke rehabilitation. Thus, our project will lead to the rehabilitation industry, and broader healthcare industry, having tools which enable clinicians to provide more effective services to a wider range of patients.

[1] Anon. (2017). Traumatic Brain Injury Strategy and Action Plan (2017–2021). ACC7819. Available at: www.acc.co.nz.

[2] Dixon, S. (2015). The Employment and Income Effects of Eight Chronic and Acute Health Conditions. New Zealand Treasury Working Paper 15/15. Wellington: New Zealand Government.

[3] Nunnerley, J., King, M., Hodge, K., Hopkins, P., Stockwell, R., Thorne, N., & Gozdzikowska, K. (2022). Co-design of a therapeutic virtual reality tool to increase awareness and self-management of cognitive fatigue after traumatic brain injury. Disability and Rehabilitation: Assistive Technology, 1–7.