14th Speech in Noise Workshop, 12-13 January 2023, Split, Croatia 14th Speech in Noise Workshop, 12-13 January 2023, Split, Croatia

P04Session 2 (Friday 13 January 2023, 09:00-11:00)
Video database for intelligibility testing in Virtual Reality

Mark Huckvale, Gaston Hilkhuysen, Tim Green
Speech, Hearing and Phonetic Sciences, University College London, UK

Background: The immersive experience allowed by VR headsets holds much promise for hearing assessment within more realistic listening environments than are usually found in audiological tests - ones that include visual information, spatialised sound and everyday settings [1]. However, realising the promise of VR-based assessment requires both new audiovisual materials compatible with the headsets and software tools for composition and running of tests. Existing resources, such as the Facebook audio-visual corpus [2] are still rather limited in scope. Here we describe a new corpus of audiovisual materials designed for intelligibility testing in VR.

Methods: British English matrix sentences [3] were recorded in our anechoic chamber against a green screen, using a 360º camera and a high-quality condenser microphone. These sentences contain 5 slots with 10 choices in each slot. Different sets of 200 sentences were recorded by 5 male and 5 female speakers sitting at a table. The individual speakers have been cropped from the videos and the green screen and table replaced by a transparent background to allow compositing of speakers in new 360° scenes. The individual matrix sentences have been extracted as 11s videos together with level-normalised monophonic audio. We have developed scripts that build multi-speaker videos from these elements in a format that can be played on VR Headsets such as the Oculus 2. We have also developed a Unity application for the headsets, which will play the 360° composite videos and create spatialised sound sources for the speakers together with background noise stored in ambisonic format.

Conclusions: We have created a new resource for running intelligibility tests within a VR environment which will allow the assessment of listener performance in more realistic listening settings including how listeners react to multiple speakers, spatialised sound, visual cues, and changes in head and gaze direction. We anticipate that the finished database will be made freely available, together with scripts for compositing new videos and software for testing listeners in VR.


  • [1] Stecker, C. Using Virtual Reality to assess auditory performance, Hearing Journal, June 2019.
  • [2] Kishline, L., Colburn, S., Robinson, P. A multimedia speech corpus for audio visual research in virtual reality. Journal of the Acoustical Society of America 148, 492-495 (2020) https://doi.org/10.1121/10.0001670
  • [3] Kollmeier, B., Warzybok, A., Hochmuth, S., Zokoll, M., Uslar, V., Brand, T., Wagener, K. The multilingual matrix test: Principles, applications, and comparison across languages: A review. International Journal of Audiology, 54 (2015) sup2, 3-16.

Last modified 2023-01-06 23:41:06