The field’s understanding of how to design well-integrated multimodal interfaces remains rudimentary, in particular focused on what modalities to include in a system. What other more fine-grained questions should be asked regarding how to design a well-integrated system? And how could future human-computer interaction research and theory be organized to refine our understanding of this topic?
References
T. Anastasio and P. Patton. 2004. Analysis and modeling of multisensory enhancement in the deep superior colliculus. In G. Calvert, C. Spence, and B. Stein, editors. The Handbook of Multisensory Processing. pp. 265–283. MIT Press, Cambridge, MA. 25
M. Anderson and C. Green. 2001. Suppressing unwanted memories by executive control. Nature, 410:366–369. DOI: 10.1038/35066572. 32
A. Baddeley. 1986. Working Memory. Oxford University Press, New York. 30, 32
A. Baddeley. 2003. Working memory: Looking back and looking forward. Nature Reviews., 4:829–839. DOI: 10.1038/nrn1201. 30
A. D. Baddeley and G.J. Hitch. 1974. Working memory. In G. H. Bower, editor. The Psychology of Learning and Motivation: Advances in Research and Theory, vol. 8, pp. 47–89. Academic, New York. 30
S. L. Beilock, I. M. Lyons, A. Mattarella-Micke, H. C. Nusbaum, and S. L. Small. 2008. Sports experience changes the neural processing of action language. In Proceedings of the National Academy of Sciences, vol. 105, pages, 13269–13273. DOI: 10.1073/pnas.0803424105. 35
L. E. Berk. 1994. Why children talk to themselves. Scientific American, 71(5):78–83. 33
V. Berninger, R. Abbott, A. Augsburger, and N. Garcia. 2009. Comparison of pen and keyboard transcription modes in children with and without learning disabilities. Learning Disability Quarterly 32:123–141. DOI: 10.2307/27740364. 35
L. Bernstein and C. Benoit. 1996. For speech perception by humans or machines, three senses are better than one. In Proceedings of the International Conference on Spoken Language Processing, vol. 3, pages 1477–1480.DOI: 10.1.1.16.5972. 25
P. Bertelson and B. deGelder. 2004. The psychology of multimodal perception. In C. Spence and J. Driver, editors. Crossmodal Space and Crossmodal Attention, pp. 141–177. Oxford University Press, Oxford, UK. DOI: 10.1093/acprof:oso/9780198524861.003.0007. 24
J. Black K. Isaacs, B. Anderson, A. Alcantara, and W. Greenough. 1990. Learning causes synaptogenesis, whereas motor activity causes angiogenesis in cerebellar cortex of adult rats. In Proceedings of the National Academy of Sciences, vol. 87, 5568–5572. 34
A. S. Bregman. 1990. Auditory Scene Analysis. MIT Press, Cambridge, MA. 21
J. Burgoon, L. Stern, and L. Dillman. 1995. Interpersonal Adaptation: Dyadic Interaction Patterns. Cambridge University Press, Cambridge, UK. 37
G. Calvert, C. Spence, and B. E. Stein, editors. 2004. The Handbook of Multisensory Processing. MIT Press, Cambridge, MA. 20, 21, 25
P. R. Cohen, M. Dalrymple, D. B. Moran, F. C. N. Pereira, J. W. Sullivan, R. A. Gargan, J. L. Schlossberg, and S. W. Tyler. 1989. Synergistic use of direct manipulation and natural language. In Proceedings of the Conference on Human Factors in Computing Systems [CHI’89], pp. 227–234. ACM Press, New York. Reprinted in M. T. Maybury and W. Wahlster, editors. 1998. Readings in Intelligent User Interfaces, pp. 29–37. Morgan Kaufmann, San Francisco. DOI: 10.1145/67450.67494. 29
A. Comblain. 1994. Working memory in Down’s Syndrome: Training the rehearsal strategy. Down’s Syndrome Research and Practice, 2(3):123–126. DOI: 10.3104/reports.42. 33
K. Daffner and M. Searl. 2008. The dysexecutive syndromes. In G. Goldenberg and B. Miller, editors. Handbook of Clinical Neurology, vol. 88, ch. 12, pp. 249–267. Elsevier B.V. DOI: 10.1016/S0072-9752(07)88012-2. 31
M. D’Esposito. 2008. Working memory, The dysexecutive syndrome. In G. Goldenberg and B. Miller, edtiors. Handbook of Clinical Neurology, vol. 88, ch. 11, pp. 237–248. Elsevier B.V. 31
N. F. Dixon and L. Spitz. 1980. The detection of auditory visual desynchrony. Perception, 9:719–721. 24
R. M. Duncan and J. A. Cheyne. 2002. Private speech in young adults: Task difficulty, self-regulation, and psychological predication. Cognitive Development, 16:889–906. DOI: 10.1016/S0885-2014(01)00069-7. 33
M. Ernst and M. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415:429–433. DOI: 10.1038/415429a. 25, 36, 37
M. Ernst and H. Bulthoff. 2004. Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4):162–169. DOI: 10.1016/j.tics.2004.02.002. 36
N. Fay, S. Garrod, L. Roberts, and N. Swoboda. 2010. The interactive evolution of human communication systems. Cognitive Science, 34:351–386. DOI: 10.1111/j.1551-6709.2009.01090.x. 37
P. Ferchmin and E. Bennett. 1975. Direct contact with enriched environment is required to alter cerebral weight in rats. Journal of Comparative and Physiological Psychology, 88:360–367. 34
W. Gaver. 1991. Technology affordances. In Proceedings of the CHI Conference, pp. 79–84. ACM Press, New York. DOI: 10.1145/108844.108856. 38, 39
J. Gibson. 1977. The theory of affordances. In R. Shaw and J. Bransford, editors. Perceiving, Acting and Knowing. vol. 3, pp. 67–82. Erlbaum, Hillsdale, NJ. 38
J. Gibson. 1979. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston. 38
H. Giles, A. Mulac, J. Bradac, and P. Johnson. 1987. Speech accommodation theory: The first decade and beyond. In M. L. McLaughlin, editor. Communication Yearbook 10, pp. 13–48. Sage Publications, London. DOI: 10.1080/23808985.1987.11678638. 37
S. Goldin-Meadow. 2003. The Resilience of Language: What Gesture Creation in Deaf Children Can Tell Us About How Children Learn Language. Psychology Press, New York. 38
S. Goldin-Meadow and S. Beilock. 2010. Action’s influence on thought: The case of gesture. Perspectives on Psychological Science, 5(6):664–674. DOI: 10.1177/1745691610388764. 35
S. Goldin-Meadow, H. Nusbaum, S. J. Kelly, and S. Wagner. 2001. Explaining math: Gesturing lightens the load. Psychological Science, 12(6):516–522. 33
J. Greeno. 1994. Gibson’s affordances. Psychological Review, 101(2):336–342. 39
J. Hayes and V. Berninger. 2010. Relationships between idea generation and transcription: How the act of writing shapes what children write. In C. Bazerman, R. Krut, K. Lunsford, S. McLeod, S. Null, P. Rogers, and A. Stansell, editors. Traditions of Writing Research, pp. 116–180. Routledge, New York. 35
M. Howison, D. Trninic, D. Reinholz, and D. Abrahamson. 2011. The mathematical imagery trainer: From embodied interaction to conceptual learning. In Proceedings of the CHI Conference, pp. 1989–1998. ACM Press, New York. DOI: 10.1145/1978942.1979230. 35
X. Huang and S. Oviatt. 2005. Toward adaptive information fusion in multimodal systems. In Second Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms [MIML’05]. Springer-Verlag, Edinburgh, UK. 28
X.
