Interactive Sonification for Health and Energy using ChucK and Unity
Published in Conference on Sonification of Health and Environmental Data, 2022
Soniļ¬cation can provide valuable insights about data but most existing approaches are not designed to be controlled by the user in an interactive fashion. Interactions enable the designer of the soniļ¬cation to more rapidly experiment with sound design and allow the soniļ¬cation to be modiļ¬ed in real-time by interacting with various control parameters. In this paper, we describe two case studies of interactive soniļ¬cation that utilize publicly available datasets that have been described recently in the International Conference on Auditory Display (ICAD). They are from the health and energy domains: electroencephalogram (EEG) alpha wave data and air pollutant data consisting of nitrogen dioxide, sulfur dioxide, carbon monoxide, and ozone. We show how these sonļ¬cations can be re-created to support interaction utilizing a general interactive soniļ¬cation framework built using ChucK, Unity, and Chunity. In addition to supporting typical soniļ¬cation methods that are common in existing soniļ¬cation toolkits, our framework introduces novel methods such as supporting discrete events, interleaved playback of multiple data streams for comparison, and using frequency modulation (FM) synthesis in terms of one data attribute modulating another. We also describe how these new functionalities can be used to improve the soniļ¬cation experience of the two datasets we have investigated.
Recommended citation: Zhao, Y., & Tzanetakis, G. (2022). "Interactive Sonification for Health and Energy using ChucK and Unity." Proceedings of the Conference on Sonification of Health and Environmental Data. 57-62. (SoniHED, paper presentation). link