The three contexts of device interaction

The history of computing has seen many different ways of interacting with devices and new ones are invented frequently. This post groups these interactions into three contexts and then discusses what they mean for the future of device interaction.

The contexts are:

1. Mobile: You carry the device with you, it is less than an arm’s length away from you. The archetypal mobile device is the cell phone.
2. Stationary: The device sits somewhere stationary and is about an arm’s length away from you (sometimes more, sometimes less). The archetypal stationary device is the desktop computer.
3. Remote: The device is out of reach, you would have to get up and/or walk a little to touch it. The archetypal remote device is the television.

For each context, a typical mode of interaction has developed.

• Mobile: Direct interaction. Touch has become very popular and is usually complemented by buttons.
• Stationary: Indirect interaction. Keyboards are used for text entry, a pointing device such as a mouse to interact with graphical user interface elements. A recent innovation is to use trackpads with gestures (mouse gestures have been around for a while, but gestures make more sense with a touch surface).
• Remote: Indirect interaction. Remote controls (which can be considered simplified keyboards) are the norm. A recent innovation is to use body gestures, as popularized by Microsoft’s Kinect.

Level of detail:

• Low level of detail: Mobile and remote devices usually show few details and have simplified user interfaces.
• High level of detail: Stationary devices provide a high level of detail and sophisticated user interfaces.

Hybrids:

• Tablets such as the iPad are a device that shares characteristics with typical mobile devices (touch user interface) and typical stationary devices (high level of detail).
• Playing games by connecting the iPad to a TV and using the iPhone as a controller.
• A projector from Sanyo stands 2ft (about 60cm) away from a wall and functions as a whiteboard via infrared pens (whose “writing” is registered by built-in front-facing cameras, leading to virtual ink being projected). A remote device (a projector) that is partially used in a stationary manner.

The future:

• Switching contexts on tablets: Using a Bluetooth keyboard with an iPad switches contexts. Accordingly, touch becomes impractical, because moving your hand from the keyboard to the iPad is tiresome. A trackpad on the keyboard would help, but then the whole user interface metaphor has to change, too. One is effectively turning the iPad into a notebook. That this is so easy, hardware-wise is testament to the versatility of tablets. Further evidence is the next iPad probably having a proximity sensor which would allow it to wake up when the case’s cover is opened and to go to sleep when it is closed.
• Switching contexts on desktop computers: Using a desktop computer with a remote control switches contexts. Well, any time you are away from your desktop computer, you switch contexts. Then controlling it remotely should be possible. Currently, remote controls are used for this purpose. Soon, Kinect-style body gestures might fulfill that duty.
• Dual and triple-context apps: If a devices hosts a set of applications and the user switches context, then the applications should adapt accordingly. It is not yet clear what the best way for doing so is. Switching a tablet from mobile to stationary might mean that a cursor should be shown for a pointing device. Switching a desktop computer from stationary to remote has been prototyped several times (Microsoft’s Media Center, Apple’s Frontrow, Google TV). As a remote device, I think a computer (be it a desktop computer, a media PC, a smart TV, a console) will mainly be used for shared experiences: Playing a game together, sharing photos, sharing a movie, playing music or music videos, etc. Google seems to think that a remote device has to offer (almost) the complete desktop experience. Gruber argues against it and I agree.
• Linking devices: With more devices specialized for different contexts floating around in households, linking them becomes interesting. The scenario above where a TV-connected iPad uses the iPhone as a controller is a good example.