3.1).
Table 3.1 A set of scenarios are used throughout to illustrate some possible multimodal interaction goals (Section 3.2.1), and roles that a haptic component might take within it (Section 3.2.3).
When provided or offered, information is continuously available. It can be accessed at the user’s will, or offered as an ambient stream where the user may consume or ignore it. It might be functional, e.g., indicating the time remaining on a clock or progress toward a goal on a wearable display [S1], or adding dexterity-enabling sensory layers to remote surgery context [S4]. It could enrich an experience (watching a haptically augmented movie [S3]). An interface might escalate an ambient information display channel to notify level (transitioning to a higher salience and discrete medium) when it becomes crucial.
In notify, information is pushed to the user when it becomes of interest, or ready. Notifications can vary in salience, including sub-attentional; but a conceptual differentiation from provided information is that it is event-based, rather than continually available.
A guiding display supports user movement and action, in real or virtual space or processes. Guiding can be continuous, e.g., steering assistance [Forsyth and MacLean 2006]; or periodic or occasional, e.g. when a wearable exercise device gives pace feedback [Karuei and MacLean 2014] [S1]. There are many other types of guiding interfaces, such as software wizards that take a user through steps of a complex configuration task, but these may not be as well suited to haptic participation. Guidance can be attentionally dominant or backgrounded, especially once well learned, as when the view of the road and traffic ahead nonconciously influences one’s speed control of a car.
3.2.2 Parameters of a Multimodal Interaction
The larger goals of a multimodal interaction expose design parameters that will define how an interaction can play out, and are a step toward setting its requirements. All modalities can potentially be called upon for these design elements; some may work better than others in a given situation, and redundancy may be called for. We detail some of the interaction parameters that may need to be resolved.
The manner of access may be push (the user is notified that information is available) or pull (the user queries for the information). Query can entail degrees of information availability: waiting (already displayed—just needs to be looked at or touched); ready to display upon request; or in need of fetching, with some time lag—perhaps even with a notification when it does arrive.
The interaction’s information origin may be endogenous or exogenous. Origin is key to a user’s conceptual understanding of what a display element means, and relevant to how it should be portrayed to the user. Data to be conveyed might be sourced endogenously from the primary user, whether voluntarily or through sensing (e.g., current running pace or effort; one’s personal emotive state, which you wish to share with someone else; time elapsed since you last stood up). Or it might come from outside, exogenously: time for a meeting to start, a target that has been met, a notification of an externally derived event, a feature available in media being felt in a virtual environment or identified by an automatic algorithm in media that is being perused.
A signal in any modality may occur once, recur occasionally or periodically, or appear continuously—from an information stream or channel being monitored, or from discrete events. Consider how an interface could provide, notify, or guide with differing degrees and types of recurrence.
Information may be supplied in the user’s attentional foreground or background. Interface attentional demand is a spectrum, from signals that target a users’ full attention to ambient presentation [Weiser and Brown 1996, MacLean 2009], and a focus of considerable current study [Roda 2011]. As with other modalities, haptic sensations can be designed to fall almost anywhere in that spectrum. Information parameters that justify varying salience include urgency (time criticality), importance, or the user’s context. Guiding information (often continuous) may be designed for conscious or non-conscious use, or both. Mechanisms to modulate a user’s attentional demand include perceptual salience of a given signal element, and recruiting additional perceptual modalities to reinforce (amplify) a percept.
3.2.3 Roles of a Haptic Signal within a Multimodal Team
The haptic signal can play several roles as part of a “team” of sensory channels involved in a multimodal design (Figure 3.1).
A haptic signal can work with other senses to provide reinforcing information about the same percept, or complementary information about a separate one. An example of a reinforcing multimodal display is when an automobile driver is informed of an upcoming turn with a visual map display showing the turn approaching, an auditory voice (“In one kilometer, turn left on Elm Drive”), and vibration of the left side of the seat or steering wheel as the turn approaches.
Alternatively, a visual map might show a bird’s-eye overview, while the vibration gives graded information about how far away the turn is. In this case, the visual and haptic information complement one another—even though they are both related to the navigation task, each gives a different part of the overall picture.
When modalities give redundant (reinforcing) information, one may be primary—the one which users will be lost without, even if they are aided by a secondary one. Often the benefit of the secondary modality is to differ in quality or timing of information display. In the previous driving example, the driver might have a general sense of location in mind, and need just a little nudge to distinguish which of several choices is the correct turn—here, the low-detail, easy-to-absorb haptic tap is just right, and having to look away at a detailed map is overkill.
Figure 3.1 Roles a haptic element can take in a multimodal interaction.
A given modality’s information must be coordinated in temporality and sequencing with respect to others. Information in a multimodal display can be presented at varying levels of detail at different times depending on need. For instance, the holistic display system can be considered as a state machine, or alternatively as detail that fades in and out. In these different states (or levels of detail zoom), modalities may play different roles.
One approach is frontline notification to backup detail. A haptic signal can present an easily processed initial notification with low information density; then the user can follow up to query a visual modality for more detail at a better time. In [S1], a smartwatch vibrates to notify the user of a message (haptics as frontline modality) then audio/visual information is displayed when the user looks at the watch, and possibly queries it for additional detail.
Alternatively, action can be followed by confirmation. A user’s interaction with a device can be actively confirmed, either immediately (button feedback) or later (message sent). In [S2], a user presses a button (visual interaction first), and receives a followup vibration for confirmation.
3.2.4 What a Haptic Component can Contribute Within an Interaction
We have outlined some possible structures of interaction models, general types of roles that haptic elements might play within them, and some design parameters that must be resolved. Here we will look at how the
