Manipulating Screen Elements in an Immersive Environment with a Wrist-Mounted Device and Free Body Movement

Large display environments supporting fluid user interactions are commonly supported in living lab environments such as situation rooms and industrial lobby spaces. However, conventional input mechanisms such as mouse and keyboard are poorly suited to interacting with such spaces due to their potentially huge spatial extent. In this context, we propose a novel technique for manipulating screen elements in large display environments. The method combines a wrist-mounted Leap Motion controller with user tracking from an array of ceiling-mounted Kinects. The overhead Kinects provide tracking coordinates in a global coordinate system along with the body orientation of the user, creating a dynamic and directed interaction screen space. The Leap Motion controller determines the location of the hand, and interprets it in the context of a large, wrap-around screen. We demonstrate the system in the CRAIVE-Lab, a large immersive environment (10m x 12m floor plan, with a 4.3m high, 40m long, 360 degree display wall).