to move beyond human factors and physical considerations to consider the social and cognitive effects of alternative designs. These opportunities create a new era in interaction design that includes such things as designing gestures and a stronger focus on physical and digital affordances and metaphors. This book is a starting point for understanding the significance of this transition and is a harbinger for future interaction designs in which large body movements are the basis for interaction. Not only is embodied interaction creating new modalities for interaction, it is also redefining the focus of good interaction design by moving away from efficiency and productivity as the basis for interaction design toward the inclusion of creativity and social interaction in the goals for new designs.
Mary Lou Maher and Lina Lee
January 2017
Acknowledgments
This book is a reflection on our collaboration with many colleagues whose views and ideas are tightly woven within our understanding of tangible and gesture interaction. After many years of collaboration, it is hard to untangle our thoughts from the engaging discussions about tangible and gesture interaction. We acknowledge our colleagues here as an integral part of our ability to produce this book. Our understanding of tangible interaction design and our design examples, Tangible Keyboard and Tangible Models, were influenced by Tim Clausner, Mijeong Kim, Alberto Gonzalez, and Kaz Grace. Our understanding of gesture interaction design and our design examples, walk-up-and-use information display and the willful marionette, were influenced by collaboration with our artists in residence, Lilla LoCurto and Bill Outcault, and our colleagues Kaz Grace and Mohammad Mahzoon. Lilla and Bill are the artisits that imagined, created, and built the willful marionette.
Completing this book was possible only with the patience of our families. Our special thanks go to the babies born during this writing project and the many sleepless nights we experienced. We particularly acknowledge the birth of Jessica, Kei, Iris, Matthew, and Erin during the time we were writing this book.
The tangible interaction research reported in this book was partially supported by NSF Grant IIS-1218160: HCC: Small: Designing Tangible Computing for Creativity. The artist-in-residence program that enabled our contribution to the willful marionette was supported by the College of Computing and Informatics and the College of Arts and Architecture at UNC Charlotte. We acknowledge the support of the College of Computing and Informatics at UNC Charlotte for funding graduate students and equipment in the InDe Lab.
Mary Lou Maher and Lina Lee
January 2017
CHAPTER 1
Introduction
In this book we explore the design issues for embodied interaction design, and specifically for tangible and gesture interaction. This book describes engaging ways of interacting with tangible and gesture-based interactive systems through four different examples as vehicles for exploring the design issues and methods relevant to embodied interaction design. In recent years, new interaction styles have emerged. Researchers in human-computer interaction (HCI) have explored an embodied interaction that seeks to explain bodily action, human experiences, and physicality in the context of interaction with computational technology (Antle et al., 2009, 2011; Klemmer et al., 2006). Dourish (2004) set out a theoretical foundation of embodiment. The concept of embodiment in tangible user interfaces (TUIs) describes how physical objects may be used simultaneously as input and output for computational processes. Similarly, in gesture-based interaction the concept of embodiment recognizes and takes advantage of the fact that humans have bodies, and people can use those bodies when interacting with technology in the same ways they use them in the natural physical world (Antle et al., 2011). This is an attractive approach to interaction design because it relates to our previous experience and makes it easier to learn new systems.
The success of interaction design depends on providing appropriate methods for the task at hand that improve discoverability and learnability. Designers should consider the user’s mental model based on previous experience when defining how the user can interact with the system, and then give the user clues about expected behavior before they take action. Giving feedback to the users is important to make it clear how the user completes an interaction. Since interaction is a conversation between the user(s) and the system, interaction design for gesture input methods and real-time feedback to the user(s) should be very carefully considered. Well-timed, appropriate feedback helps users to notice and understand that the system is interactive, to learn how to interact with it, and to be motivated to interact with it. Ideally, feedback communicates to the user that the system status has altered as the user intended (Dillon, 2003).
As we move toward embodied interaction, we maintain these basic principles of good interaction design: the user initiates interaction with some form of action, and the system responds or alters as the user intended. However, the trend for embodied interaction is the design of very broad and varied ways in which the user is expected to act to initiate interaction, and the iterative action-response needs to be discovered and learned. For example, laptop and touchscreen interactions are ubiquitous enough that there are established guidelines and design patterns that designers adhere to (Norman, 1983). These patterns and guidelines cause users to have certain expectations of how a system might work even before they begin interacting with it. However, embodied interaction is relatively new and does not have as coherent a set of consistent design patterns for interaction. Therefore, we transition from an expectation for consistent and effective interaction design using keyboard, mouse, and display, toward novel interactive systems in which the user explores and learns which actions lead to expected changes in the state of the system. We propose that HCI is a cognitive process in which the user mental model is the basis for their exploration and use of the interactive system. Users decide how to interact on the basis of expectation and prior experience, and the affordances of the specific interactive system modify the user mental model.
As Dourish (2001) says, when users approach an embodied interactive system, they must construct a new understanding of how it works on the basis of their physical exploration. Different people may have unique experiences and expectations, which will affect the way in which they initially explore a system and, ultimately, the mental model they construct of how the system works (Dillon, 2003). Embodied interaction has been used to describe the interactions of users with a wide range of interactive technologies, including tangible and gesture-based interfaces.
We posit that good tangible and gesture interaction design depends on an understanding of the cognitive issues associated with these modalities. We organize these issues into four categories: embodiment, affordances, metaphor, and epistemic actions. These four categories can be used as clues that the designer can give the user to aid the user in understanding how the interactive system is to be operated. If these concepts are integrated into the design process, the user’s mental model and knowledge can be activated and extended as they try to use and understand the interactive system. While these cognitive issues require further exploration and empirical validation (Antle et al., 2011), we present specific projects that explore various aspects of embodied HCI.
1.1 EMBODIMENT
Interaction through tangible and gesture-based systems is intrinsically embodied, and therefore decisions about which embodied actions can be recognized by the interactive system are part of the design process. Human gestures are expressive body motions involving physical movements of the fingers, hands, arms, head, face, or body that may convey meaningful information or be performed to interact with the environment (Dan and Mohod, 2014). Designing embodied interaction is not just about designing computing ability, but is also about designing the human experience and anticipated human behavior.
Research has shown that gestures play an integral role in human cognition. Psychologists and cognitive
