different curriculum materials or computer interfaces. Educational researchers then focus on evidence-based redesign of these materials and tools to decrease students’ extraneous cognitive load, so their learning progress can be enhanced.
Numerous learning studies have shown that a multimodal presentation format supports students’ learning more successfully than does unimodal presentation. For example, presentation of educational information that includes diagrams and audiotapes improves students’ ability to solve geometry problems, compared with visual-only presentation of comparable information content [Mousavi et al. 1995]. When using the multimodal format, larger performance advantages have been demonstrated on more difficult tasks, compared with simpler ones [Tindall-Ford et al. 1997]. These performance advantages of a multimodal presentation format have been replicated in different content domains, with different types of instructional materials (e.g., computer-based multimedia animations), and using different dependent measures [Mayer and Moreno 1998, Tindall-Ford et al. 1997]. These research findings based on educational activities are consistent with the general literature on multimodal processing advantages.
In recent years, Cognitive Load theory has been applied more broadly to computer interface design [Oviatt 2006]. It has supported the development of multimodal interfaces for education, and adaptive interface design tailored to a learner’s level of domain knowledge. Empirical studies have demonstrated that flexible multimodal interfaces are effective partly because they support students’ ability to self-manage their own working memory in a way that reduces cognitive load [Oviatt et al. 2004a]. For example, students prefer to interact unimodally when working on easy problems. However, they will upshift to interacting multimodally on harder problems in order to distribute processing, minimize cognitive load, and improve their performance [Oviatt et al. 2004a].
One implication of these research findings is that flexible multimodal interfaces are especially well suited for applications like education, which typically involve higher levels of load associated with mastering new content. In fact, all applications that require extended thinking and reasoning potentially could be improved by implementing a flexible and expressively powerful multimodal interface. In addition, Working Memory theory has direct implications for designing well-integrated multimodal interfaces that can combine complementary modalities to process specific types of content with minimal interference effects.
1.3 Activity Theory, Embodied Cognition, and Multisensory-Multimodal Facilitation of Cognition
Activity Theory is a meta-theory with numerous branches. It makes the central claim that activity and consciousness are dynamically interrelated. Vygotskian Activity theory states that physical and communicative activity play a major role in mediating, guiding, and refining mental activities [Luria 1961, Vygotsky 1962, 1978, 1987].
In Vygotsky’s view, the most powerful tools for semiotic meditation are symbolic representational ones such as language. Vygotsky was especially interested in speech, which he believed serves dual purposes: (1) social communication and (2) self-regulation during physical and mental activities. He described self-regulatory language, also known as “self talk” or “private speech,” as a think-aloud process in which individuals verbalize poorly understood aspects of difficult tasks to assist in guiding their thought [Berk 1994, Duncan and Cheyne 2002, Luria 1961]. In fact, during human-computer interaction the highest rates of self talk occur during more difficult tasks [Xiao et al. 2003]. For example, when using a multimodal interface during a map task, people typically have the most difficulty with relative directional information. They may subvocalize, “East, no, west of …” when thinking about where to place a landmark on a digital map. As a map task increases in difficulty, users’ self talk progressively increases, which has been shown to improve their performance [Xiao et al. 2003].
Since Vygotsky’s original work on the role of speech in self-regulation, further research has confirmed that activity in all communication modalities mediates thought, and plays a self-regulatory role in improving performance [Luria 1961, Vygotsky 1962, 1987]. As tasks become more difficult, speech, gesture, and writing all increase in frequency, reducing cognitive load and improving performance [Comblain 1994, Goldin-Meadow et al. 2001, Oviatt et al. 2007, Xiao et al. 2003]. For example, manual gesturing reduces cognitive load and improves memory during math tasks, with increased benefit on more difficult tasks [Goldin-Meadow et al. 2001]. When writing, students also diagram more as math problems became harder, which can improve correct solutions by 30–40% [Oviatt 2006, 2007]. In summary, research across modalities is compatible with Vygotsky’s theoretical view that communicative activity mediates thought and improves performance [Luria 1961, Vygotsky 1962].
Activity theory is well supported by neuroscience results on activity- and experience-dependent neural plasticity. Activity-dependent plasticity adapts the brain according to the frequency of an activity. Activities have a profound impact on human brain structure and processing, including changes in the number and strength of synapses, dendritic branching, myelination, the presence of neurotransmitters, and changes in cell responsivity, which are associated with learning and memory [Markham and Greenough 2004, Sale et al. 2009]. Recent neuroscience data indicate that physical activity can generate change within minutes in neocortical dendritic spine growth, and the extent of dendritic spine remodeling correlates with success of learning [Yang et al. 2009]. Other research has shown that the experience of using a tool can change the properties of multisensory neurons involved in their control [Ishibashi et al. 2004].
A major theme uncovered by neuroscience research related to Activity theory is the following:
• Neural adaptations are most responsive to direct physical activity, rather than passive viewing or vicarious experience [Ferchmin and Bennett 1975].
One major implication of this finding is that the design of computer input tools is particularly consequential for eliciting actions that directly stimulate cognition. This contrasts with the predominant engineering focus on developing system output capabilities.
In addition, neuroscience findings emphasize the following:
• Physical activity that involves novel or complex actions is most effective at stimulating synaptogenesis, or neural adaptations compatible with learning and memory. In contrast, familiar and simple actions do not have the same impact [Black et al. 1990, Kleim et al. 1997].
A further theme revealed by neuroscience research, which focuses specifically on activity theory and multimodality, is:
• Multisensory and multimodal activity involve more total neural activity across a range of modalities, more intense bursts of neural activity, more widely distributed activity across the brain’s neurological substrates, and longer distance connections.
Since multimodal interfaces elicit more extensive neural activity across many dimensions, compared with unimodal interfaces, they can have a greater impact on stimulating cognition. In particular, they produce deeper and more elaborated learning, improve long-term memory, and result in higher performance levels during human-computer interaction [Oviatt 2013].
Embodied Cognition theory, which is related to Activity theory and Situated Cognition theory, asserts that thought is directly shaped by actions in context as part of an action-perception loop [Beilock et al. 2008, Shapiro 2014, Varela et al. 1991]. For example, specific gestures or hand movements during problem solving can facilitate an understanding of proportional equivalence and other mathematical concepts [Goldin-Meadow and Beilock 2010, Howison et al. 2011]. Representations and meaning are created and interpreted within activity, rather than being stored as past knowledge structures. More specifically, representation involves activating neural processes that recreate a related action-perception experiential loop. Two key findings in the embodied cognition literature are that:
• The action-perception loop is based on multisensory perceptual and multimodal motor neural circuits in the brain [Nakamura et al. 2012].
•
