explored the role of gesture and thought for several decades. McNeil (1992, 2008) explains that gesture and thought are tightly connected, and he also establishes a categorization of gestures and their role in human cognition and communication. There is evidence that gesturing aids thinking. Several studies have shown that learning to count is facilitated by touching physical objects (Efron 1941; Kessell and Tversky 2006; Cook et al., 2008). Kessell and Tversky (2006) show that when people are solving and explaining spatial insight problems, gesturing facilitates finding solutions. Goldin-Meadow and Beilock (2010) summarize findings as “gesture influences thought by linking it to action, (p. 669)” and “producing gesture changes thought (p. 670)” and can “create new knowledge (p. 671).” These studies show that gesture, while originally associated with communication, is also related to thinking. Embodied interaction design creates an environment that is activated by gesture and actions on objects and therefore induces cognitive effects that traditional user interaction does not.
One challenge to embodied interaction is that while it is built upon natural actions, it still requires some level of discovery, especially when it is a public display. Tangible and gesture-based interaction designers consider both the integration of technology and its effects on human experience. The major consideration that has emerged to influence tangible design is the physical embodiment of computing. Interaction design is not just screen-based digital interaction anymore. Tangible interaction designers should think about physical, graspable objects that give cues for understanding and provide the basis for interaction. Gesture interaction designers should think about how various human movements can be recognized and interpreted in the context of changing the state and response of the computational system. Interactive platforms can be interpreted as spaces to act and move in, and they effectively determine interaction patterns.
Dourish (2004) explores the physicality of embodied interaction and its affect on moving human computer interaction toward more social environments. He describes an approach to embodiment grounded in phenomenology, and claims that any understanding we have of the world is the result of some initial physical exploration. Embodied interaction is about establishing meaning and it is through embodied interaction that we develop an understanding of the meaning of the system. As the user constructs their mental model, they are influenced by the phenomena they are experiencing at that moment as well as their prior experiences and understanding of how technology works.
In this book, we take a cognitive view of embodied interaction design: Discovering the interaction model relies on pre-existing mental models derived from physical experiences, and executing intentional physical movements during interaction has an effect on cognition. We demonstrate and elaborate on this view of embodiment through four projects; where we describe the gestures that enable interaction, the design methods, and the usability issues for each project.
1.2 AFFORDANCE
The concept of affordance was introduced to the HCI community by Norman (1988) and Gibson (1982). According to Norman (1988), an affordance is the design aspect of an object that allows people to know how to use it and that gives a clue to its function and use. Norman discusses the concept of affordance as properties of an object that allow specific actions such as a handle affords holding and turning, a button affords pressing and make it its own function clear. Tangible interaction design is arguably more influenced by physical affordances than by visual or gesture interaction design.
TUIs change the way we interact with digital information, with physical affordances that are distinctly different from pointing and keyboard/mouse interaction. According to Wang et al. (2002), there are two advantages to tangible interaction; first, it allows direct, naïve manipulability and intuitive understanding; and second, the sense of touch provides an additional dimension. The tactile feedback afforded by TUIs is consistent with the early empiricist argument that kinesthetic information provides us with the ability to construct a spatial map of objects that we touch (Lederman and Klatzky, 1993; Loomis and Lederman, 1986). Fitzmaurice (Fitzmaurice, 1996; Fitzmaurice and Buxton, 1997) demonstrated that having multiple graspable interactive devices encourages two-handed interaction that calls upon everyday coordination skills. Leganchuk et al. (1998) explored the potential benefits of such two-handed input through experimental tasks to find that bimanual manipulation may bring two types of advantages to HCI: manual and cognitive. The two-handed interaction doubles the freedom simultaneously available to the user and reduces the cognitive load of the input performance.
The potential affordances of the TUIs, such as manipulability and physical arrangements, may reduce cognitive load associated with spatial reasoning, thus resulting in enhanced spatial cognition and creative cognition. Brereton and McGarry (2000) studied the role of objects in supporting design thinking as a precursor to designing tangible interaction; they found that design thinking is dependent on gesturing with objects, and recommend that the design of tangible devices consider a tradeoff between exploiting the ambiguous and varied affordances of specific physical objects. The affordances of design tools facilitate specific aspects of designing. As we move away from the traditional WIMP (Windows, Icons, Menus, and Pointer) interaction, we encounter new kinds of affordances in interactive digital design tools (Burlamaqui and Dong, 2015). Tangible interaction design takes advantage of natural physical affordances (Ishii and Ullmer, 1997) by exploiting the knowledge that people already have from their experience with nondigital objects to design novel forms of interacting and discovering. In this book, we focus on the affordances of the interaction that can be sensed by the interactive devices. Well-designed objects make it clear how they work just by looking at them. The successful design of embodied interaction systems does not ignore the affordances of the physical and visual aspects of the system.
1.3 METAPHOR
While affordances of physical objects are closely related to our experience with their physical properties, the properties of tangible interaction objects have both physical and digital relationships. In contrast to physical objects, on-screen objects are clusters of pixels without a physical dimension. A common way to create the appearance of physical affordances to on screen objects is the use of metaphor in designing interface elements (Szabó, 1995). By creating a visual reference on screen to familiar physical objects, the on-screen objects take on some of the affordances of the metaphorical
object (Mohnkern, 1997).
The use of a metaphor during design makes familiar that which is unknown or unfamiliar by connecting it with the user’s previous experience (Dillon, 2003). The most well-known is the “desktop metaphor” used in current operating systems. Another common example of metaphor is the trash can. You can grab a file with the mouse to take it above the trash can and release it. A designer can use the shape, the size, the color, the weight, and the texture of the object to invoke any number of metaphorical links (Fishkin, 2004).
Metaphors are an important concept for embodied interaction. An interaction model based on embodied metaphors effectively implements a mapping between action and output that is consistent with the metaphorical object. Through design, we can map human behaviors and bodily experiences onto abstract concepts in interactive environments (Bakker et al., 2012). Metaphor gives users a known model for an unknown system. Metaphor can help ease the transition to a new situation, so it is good for creating systems that will be used primarily by novices, like public displays. For embodied interaction design, in which there are few standards and fewer user manuals, the role of metaphor in design may be critical in creating easily discovered and learnable interactive systems.
1.4 EPISTEMIC ACTIONS
Epistemic action is exploratory motor activity aimed at uncovering information that is hard to compute mentally. Kirsh and Maglio (1994) distinguish between epistemic and pragmatic actions. A pragmatic action is the action needed to actually perform the task. Epistemic actions are actions that help the person explore the task and guide them to the solution. As such, epistemic actions enable the person to use physical objects and their environment to aid their cognition (Kirsh and Maglio, 1994; van den Hoven and Mazalek, 2011).
