having a variety of tangible objects and physical arrangements may aid problem solving while interacting with a tangible interactive system. Fitzmaurice (1996) discussed the concepts of pragmatic and epistemic actions to provide the underlying theoretical support for workable graspable user interfaces (UIs). Pragmatic action refers to performatory motor activity that directs the user toward to the final goal. Epistemic action refers to exploratory motor activity that may uncover hidden information that would otherwise require a great deal of mental computation.
Kim and Maher (2007) found an increase of epistemic actions in a design task while using a tangible UI, and through a protocol analysis, were able to also observe an increase in the cognitive processes typically associated with creative design. The projects in this book build on that result to design tangible interfaces based on physical objects that offer more opportunities for epistemic (i.e., exploratory) actions than pragmatic (i.e., performatory) actions. Exploration through epistemic actions enables a better perception of the environment and supports learning more about the properties of the objects. When designing gesture-based interaction, the process of discovering the interaction model can be leveraged by encouraging and responding to epistemic actions.
1.5 THIS BOOK
In this book we present tangible and gesture interaction design with an underlying assumption that embodiment, affordances, metaphor, and epistemic actions are critical cognitive issues that can influence the quality of the design. If the interaction design is not well conceived with respect to these cognitive issues, users suffer from frustration, discomfort, stress, and fatigue. Applying appropriate design methods is crucial and should help bridge the differences between the designer’s view of the system and user’s mental model. It is important to conduct user research to know how to incorporate the insights from users’ experiences into the design. In this book, various user research and design methods such as gesture elicitation, protocol analysis, heuristic evaluation, prototyping, body-storming, role-playing, personas, and image boards are described to show how designers understand the potential user mental models of the interactive system. We describe these methods in the context of their use in the four design examples: Tangible Keyboard, Tangible Models, walk-up-and-use information display, and the willful marionette.
This book can provide HCI practitioners and researchers with new principles for better designs and new ideas for research in embodied interaction. For HCI practitioners, the book describes specific design projects and the methods used during design and evaluation. These methods are specific to designing for tangible and gesture interaction. The description of these methods will help practitioners understand how these methods are applied, and, when appropriate, how these methods are uniquely suited to tangible and gesture interaction. For the HCI researcher, the book identifies the cognitive and design research issues that are raised when designing for tangible and gesture interaction. Many of the methods described in the design projects are also applicable in a research context.
The organization of this book is as follows: Chapter 2 presents the concepts and significance of tangible interaction design. In Chapter 3, we present a description of our experience in designing the Tangible Keyboard and Tangible Models. Gesture interaction design is presented in terms of the technology and significance in Chapter 4. We follow this with a description of our experience in designing the walk-up-and-use information display and the willful marionette in Chapter 5. In Chapter 6, we conclude with our understanding of the research challenges in designing for embodied interaction design.
CHAPTER 2
Tangible Interaction Design
2.1 WHAT IS A TANGIBLE INTERACTION?
Tangible User Interfaces (TUIs) have emerged as an interface and interaction style that links the digital and physical worlds (Ullmer and Ishii, 2000; Shaer and Hornecker, 2010). An early definition of tangible interaction was introduced by Ishii and Ullmer (1997) as an extension of the idea of graspable user interfaces (UIs): they argued that tangible interaction allows users to grasp and manipulate bits by coupling digital information with physical objects and architectural surfaces.
Figure 2.1: Graspable object. Based on Fitzmaurice (1996, p. 4).
TUIs employ physical objects with a direct correlation to digital objects as an alternative to traditional computer input and output devices for control (e.g., mouse) and display (e.g., screen) (Fishkin, 2004). A person uses their hands to manipulate one or more physical objects via gestures and actions such as pointing, clicking, holding, and grasping. A computer system detects the movement, changes its state, and provides feedback (Petridis et al., 2006). TUIs are designed to build on our experience and skills from interacting with the non-digital world (Ishii and Ullmer, 1997; Shaer and Jacob, 2009). TUIs offer the possibility of natural interfaces that are intuitive and enjoyable to use as well as easy to learn (Shaer, 2008). TUIs have the potential to enhance learning and problem solving by changing the way people interact with and leverage digital information (Shaer and Jacob, 2009). Current research in tangible interaction includes understanding the design and cognitive implications of TUIs, developing new technologies that further bridge the digital and the physical, and guiding TUI design with knowledge gained from empirical studies.
The goal of this chapter is to provide an overview and general framework for the design of tangible interaction, including consideration of the role of gesture and the impact on cognition. We believe that TUIs have an impact on cognition because they provide affordances that encourage and facilitate specific gestures and actions, making some cognitive activities easier. TUIs change the way we interact with digital information via physical affordances that are distinctly different from pointing and keyboard/mouse interaction. This chapter explores physical manipulation as an interaction design space. TUIs trigger various gestures and have potential for exploring information through novel forms of interacting and discovering. The chapter presents the concepts and design issues of TUIs through two examples: the Tangible Keyboard and Tangible Models. They exemplify two approaches to integrating TUIs with traditional interaction design: the Tangible Keyboard design examines the metaphor of a keyboard where each key is a tangible object; Tangible Models design examines physical interaction with 3D objects as proxies for 3D digital models.
2.1.1 TANGIBLE KEYBOARD
Figure 2.2: Pattern Maker application on the Tangible Keyboard. Tangible Keyboard video available on: https://drive.google.com/file/d/0B4S5ptYjjjuGbFEyX2ljUk90LVU/view?usp=sharing.
Tangible Keyboard is a tangible computing platform that adopts a keyboard metaphor in developing tangible devices for touch screen tablets. The tangible interaction design has a focus on supporting composition tasks and the potential for enhancing creative cognition through spatial arrangement. The Tangible Keyboard design provides separate interaction spaces for composition tasks: the whole composition is displayed on the tablet, and the individual pieces of the composition are manipulated on tangible interactive objects. Individual elements are displayed on tangible interactive objects (inspired by Sifteo cubes™), and these smaller displays are manipulated to create a composition on a larger touch display tablet (Merrill et al., 2012). A variety of different gestures and actions on the tangible objects serve
