Why situations room
When considering human-computer interaction, whether in an office with laptops, in a car with a mobile phone, or on a couch with a tablet, an individual user is most often the sole focus of that interaction. However, many situations involve groups of people working together, especially in critical situations involving complex problems. These people often have diverse backgrounds and domain expertise. We can imagine scenarios in corporate board rooms where groups of executives are tasked with allocating capital, or in health centers where teams of doctors determine patient treatment plans.
In most collaborative situations, people communicate using their personal devices and they may share information on a large screen or multiple screens to disseminate information to the group. However, the ability of the group to directly interact with one another’s information is limited by restricted access to personal devices. When additional information is required and not available, that information must be researched and gathered, analysis must be performed and reports must be created for the next meeting.
Given these limited affordances, decisions in these situations are too often made without full knowledge of or insight to pertinent data. Such issues must be resolved to significantly improve the quality and productivity of group decision making. Even with increasingly intelligent computers and data-analytical algorithms, there is inherent difficulty in linking them directly into the group’s wisdom in the context of discussions or decision making in real-time.
We aim to create a Situations Room that allows increasingly powerful computers to interact naturally and directly with group discussions and augment group intelligence in real-time:
-
Summarize meetings automatically;
-
Bring real-time analysis of context information to the discussion when person-directed decision making is required;
-
Facilitate discussion when people have biases or misunderstand one another;
Interject into discussions as appropriate; -
Generate new hypothesis;
-
Use analogies to create links between seemingly uncorrelated datasets.
Scientific breakthroughs and technical innovations must continue in computer science, cognitive science, and engineering to achieve this vision of the cognitive and immersive Situations Room.
The Interaction Loop for Human-Scale Group-Computers Interaction
The Situations Room serves as a bridge between groups of individuals working on a task, and the supercomputing services (our Society of Cognitive Agents) those individuals use to make critical decisions. The Situations Room integrates immersive environments—to allow human-scale interaction—and cognitive computing—a digital analysis of the context of group activities, or “mind-of-room.” As an initial step towards understand group dynamics, the interaction between humans and the Situations Room is approached as a long-term multi-round conversation loop with multiple interaction modalities:
Area 1
Recognize interpersonal communication, identify conversations between humans and computers, and organize this information as input for Area 2.
The idea is to utilize advanced cameras, microphones, and sensors as the eyes and ears the Situations Room. Those elements would capture humans locations, movements, head position, eye focus, and body movements, speech, physical objects, digital objects, and voice in the room. At this level, cognitive computing technologies are applied to room input channels to recognize human speech, gestures, attention, emotion, grouping, and interpersonal interactions.
Area 2
Interpret and understand group activity and its context, then find relevant information from outside the room which might facilitate the discussion.
This step uses the mind of the Situations Room. Cognitive computing technologies are applied to interpret and understand the meaning of human activity, based on input from Area 1. The room will be able to address the following questions within the discussion:
- What is the focus of the discussion? What are its topics and knowledge domains?
- What types of discussion activities are taking place? Validation, exploration, decisions, or task prioritizations?
- What are the interpersonal dynamics between discussion participants? What are their relationships and trust levels?
Based on information derived from the people in the room, cognitive computing will be leveraged behind the scenes to determine additional context not explicitly included in the discussion taking place between participants in the room. With both the context from the conversation in the room and the external context brought to the conversation, cognitive systems will be able to reason and learn from both contexts, find suggestions to present to the participants, and facilitate productive discussions.
Area 3
Present and communicate the analysis and cognitive viewpoints to discussion participants through creative multimodal narratives.
Humans must understand what computational processes are gathered by the computers to analyze context information. What is in the data and why did the computers derive its conclusion(s)? The most memorable way for humans to explain such insight to others is through storytelling, often in a multimodal fashion, with creative analogies, summaries, and references.
Research in creative interactive multimodal storytelling will help create multimodal narratives to translate insights and discovery from large amounts of data into advanced visualization and sonification in an immersive environment and allow interaction by humans to further explore the information and thus enable human-in-the-loop analysis.