CISL Undergraduate Research Assistants Huang Zou and Rose Pisacano recording example data of non-native Mandarin speaker pronunciation.
Three CISL Collaborators — Jonas Braasch, School of Architecture, CISL; Mei Si, Cognitive Science Department, Games and Simulation Arts and Sciences (GSAS); and Yalun Zhou, Department of Communication and Media, Games and Simulation Arts and Sciences (GSAS), Mandarin Project — have been awarded a 2017 Rensselaer KIP grant to investigate new virtual reality methods to teach students pitch-related abilities including learning pitch contours for tone languages (Chinese) and correct intonation for musical instruments in conservatory-style education environments.
One of the major problems in acoustic learning is that an erroneous production of a given task often produces an erroneous internal template. For example, if a pitch on a musical wind instrument is performed too low for a specific note, the acoustic event of this note will likely become an involuntary template. The same holds for the production of pitch-based words in tone languages. As soon as a non-native speaker makes an utterance, this utterance becomes an acoustically reinforced template. Often language teachers produce the to-be-spoken word with an accurate pitch contour for the student, but it is often not easy for the student to translate the suggested changes to his or her own voice. Consequently, we propose a technological learning environment in which the signal of the student is analyzed and corrected using digital signal processing tools based on granular synthesis. This way the student can hear his or her own voice with the correct pitch contour to establish a positive template. The same technique will be used for orchestral instruments so that students can hear their own performance with correct intonation. A synthetic character will be used for system communication with the student so the student can learn to produce the correct pitch contour.
Within this project, we will also develop an immersive in situ environment, where music performance student can learn to play accurately, in various simulated concert venues providing the natural feedback they would receive in the physical environments.