Focus Questions
References
PART III COMMON MODALITY COMBINATIONS
Chapter 9 Gaze-Informed Multimodal Interaction
Pernilla Qvarfordt
9.2 Eye Movements and Eye Tracking Data Analysis
9.3 Eye Movements in Relation to Other Modalities
9.4 Gaze in Multimodal Interaction and Systems
9.5 Conclusion and Outlook
Focus Questions
References
Chapter 10 Multimodal Speech and Pen Interfaces
Philip R. Cohen, Sharon Oviatt
10.2 Empirical Research on Multimodal Speech and Pen Interaction
10.3 Design Prototyping and Data Collection
10.4 Flow of Signal and Information Processing
10.5 Distributed Architectural Components
10.6 Multimodal Fusion and Semantic Integration Architectures
10.7 Multimodal Speech and Pen Systems
10.8 Conclusion and Future Directions
Focus Questions
References
Chapter 11 Multimodal Gesture Recognition
Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, Petros Maragos
11.1 Introduction
11.2 Multimodal Communication and Gestures
11.3 Recognizing Speech and Gestures
11.4 A System in Detail
11.5 Conclusions and Outlook
Focus Questions
References
Chapter 12 Audio and Visual Modality Combination in Speech Processing Applications
Gerasimos Potamianos, Etienne Marcheret, Youssef Mroueh, Vaibhava Goel, Alexandros Koumbaroulis, Argyrios Vartholomaios, Spyridon Thermos
12.1 Introduction
12.2 Bimodality in Perception and Production of Human Speech
12.3 AVASR Applications and Resources
12.4 The Visual Front-End
12.5 Audio-Visual Fusion Models and Experimental Results
12.6 Other Audio-Visual Speech Applications
12.7 Conclusions and Outlook
Focus Questions
References
PART IV MULTIDISCIPLINARY CHALLENGE TOPIC: PERSPECTIVES ON LEARNING WITH MULTIMODAL TECHNOLOGY
Chapter 13 Perspectives on Learning with Multimodal Technology
Karin H. James, James Lester, Dan Schwartz, Katherine M. Cheng, Sharon Oviatt
13.1 Perspectives from Neuroscience and Human-Centered Interfaces
13.2 Perspectives from Artificial Intelligence and Adaptive Computation
13.3 The Enablers: New Techniques and Models
13.4 Opening Up New Research Horizons
13.5 Conclusion
References
Preface
The content of this handbook would be most appropriate for graduate students, and of primary interest to students studying computer science and information technology, human–computer interfaces, mobile and ubiquitous interfaces, and related multidisciplinary majors. When teaching graduate classes with this book, whether in quarter or semester classes, we recommend initially requiring that students spend two weeks reading the introductory textbook, The Paradigm Shift to Multimodality in Contemporary Interfaces (Morgan Claypool, Human-Centered Interfaces Synthesis Series, 2015). This textbook is suitable for upper-division undergraduate and graduate students. With this orientation, a graduate class providing an overview of multimodal-multisensor interfaces then could select chapters from the handbook distributed across topics in the different sections.
As an example, in a 10-week quarter course the remaining 8 weeks might be allocated to reading select chapters on: (1) theory, user modeling, common modality combinations (2 weeks); (2) prototyping and software tools, signal processing and architectures (2 weeks); (3) language and dialogue processing (1 week); (4) detection of emotional and cognitive state (2 weeks); and (5) commercialization, future trends, and societal issues (1 week). In a more extended 16-week semester class, we would recommend spending an additional week reading and discussing chapters on each of these five topic areas, as well as an additional week on the introductory textbook, The Paradigm Shift to Multimodality in Contemporary Interfaces. As an alternative, in a semester course in which students will be conducting a project in one target area (e.g., designing multimodal dialogue systems for in-vehicle use), some or all of the additional
