The Rensselaer Mandarin Project enables a group of foreign language students to improve functional understanding, pronunciation and vocabulary in Mandarin Chinese through authentic speaking situations in a virtual visit to China. Students use speech, gestures, and combinations thereof to navigate an immersive, mixed reality, stylized realism game experience through interaction with AI agents, immersive technologies, and game mechanics. The environment was developed in a black box theater equipped with a human-scale 360◦ panoramic screen (140h, 200r), arrays of markerless motion tracking sensors, and speakers for spatial audio.
Reference
"The Rensselaer Mandarin Project — A Cognitive and Immersive Language Learning Environment,"
Proceedings of the AAAI Conference on Artificial Intelligence - Demonstration Track, vol. 33, pp. 9845-9846. 2019.
Bibtex
@inproceedings{allen2019rensselaer,
title={The Rensselaer Mandarin Project—A Cognitive and Immersive Language Learning Environment},
author={allen, david and Divekar, Rahul R and Drozdal, Jaimie and Balagyozyan, Lilit and Zheng, Shuyue and Song, Ziyi and Zou, Huang and Tyler, Jeramey and Mou, Xiangyang and Zhao, Rui and others},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={33},
pages={9845--9846},
year={2019}
}