NUI = No Use of Intermediaries?

As designers and developers of new products, we strive to make people’s interactions with our products more natural and easy. Every once in a great while we reach a major turning point in which a new form of interaction is introduced and we find ourselves at the beginning or bottom of a new learning curve. Well, we have definitely arrived at a new turning point, again. New technologies like Xbox Kinect have brought gesture and speech input to the masses. We refer to these forms of gesture and speech input as more natural, because each of us has been gesturing and speaking almost our entire lives and we don’t expect to have to re-learn them to interact with products. We want the products to have natural user interfaces so we can have natural user interactions (NUI). We assume and expect this to be a good thing as we equate natural with easy. However, there is a potential trap, thinking that the use of gesture and/or speech will ‘automagically’ ensure an easy and great experience. Certainly, the potential is there, but there are some assumptions and common mistakes; I will focus on one.

We use our bodies and voices everyday and when we use them directly, without anything between us and the target of our communication or interaction, everything can feel natural and easy. For example, talking to a friend face-to-face feels more natural than over the phone – an intermediary device between me and my friend. However, this concept is often misunderstood. The absence of intermediaries (e.g., devices like phones, keyboards, or video game controllers) or the use of one’s own body/voice only is not sufficient to make interactions more natural and easy. Let’s examine this more closely.

Devices

There are countless examples of tasks we perform with the use of devices and tools. Of course, as is the case with all tools, there is a cost we pay climbing the learning curve to improve our efficiency and effectiveness with a tool. The steeper the learning curve the less natural we consider the device or tool. Since almost all devices and tools require some training and learning, should we abandon them for not being natural? Of course not. Many tasks are simply more easily accomplished with the right tool or device. However, to make learning it and using it worthwhile, the value must exceed the cost. The value of these tools is in their ability to extend our abilities, and to do more with them than we could without them. Let’s stay with the Xbox example and look at its controller. Once a player has mastered the Xbox controller for a particular game, the player no longer pays attention to the mechanics of how to operate the controller, and instead has “become one” with the controller. At this point, the input interface has become invisible and the player only pays attention to the game and what must be done. From this point onward, the game experience is fully enjoyed and the longer the game is played the more the value exceeds any initial learning costs. This is why so many gamers play one game for so long; they have paid the cost and want to reap the value and rewards for as long as possible. This is also why so many games within a given genre tend to converge or standardize controller usage. Players can generalize and leverage controller (device / tool) learnings from one game to the next, reducing costs while getting to enjoyment faster.

Not all controllers are necessarily the right tool, but a steering wheel is an example of a controller that just feels right. When designed well, the steering wheel can make driving in a racing game feel more natural. Additionally, there are many situations in which just having a prop or something in your hand feels right or better. Children often prefer to play some games, even make-believe ones, holding something. A stick, for example, makes for an excellent imaginary magical wand or knight’s sword.

Software “Devices”

So far, when discussing devices like game controllers, I have referred only to hardware devices. However, software objects and controls can also create an intermediary layer of control between users and the target of their interaction. For example, a user’s hand moves a hardware mouse to control an on-screen cursor used to manipulate an on-screen scroll-bar to scroll a page of content. In that example, the user was operating one hardware and two software intermediary controls just to scroll a page of content! Ideally, users would simply reach out and move the page of content directly and naturally, much as they would do in the physical world. In the digital world, we see examples of direct interaction with content on our new generation of smart phones; touch a list and drag your finger up and down to slide the list itself up or down.

Body and Voice

Do not mistakenly assume that by only requiring the use of body movements and/or voice that interactions will automatically be easy and more natural. We can consider one’s own body and voice as already familiar and natural tools, but this only gives designers and developers greater potential to create a more natural interaction. Players come to a game where they are already “one with the controller”, their body. When games can take advantage of this life experience and knowledge to reduce costs, players get to playing and enjoying faster. Players actually run in place to effect running in a game, jump to jump in game, etc.

For a game console dashboard, game menus, or more traditional UIs , however, many of the tasks and activities simply do not involve running, jumping, hitting, etc. Activities are more administrative, like navigating lists. For these UIs, creating gestures and a speech system that feel natural for scrolling, paging, selecting, etc., can be extremely challenging (more on this in a future post).

Bottom Line

When an activity or action is more natural, we consider it easy (i.e., low cost). However, performing a natural action to accomplish a task does not guarantee accomplishing that task will be natural or easy. For example, while walking or running may be natural actions and activities, asking players to run in place as a way to scroll a list is not natural. It’s putting something natural to use in the wrong way, which does not make the interaction natural nor the task easy. Our goal is to help users accomplish tasks easily. That does not happen automatically just because we have people use their bodies and/or their voices. To make interactions more natural, we must enable users to move and/or talk in ways that are natural to the task or activity at hand. Sometimes, this may mean that we don’t force fit NUI alone onto tasks that would otherwise be easier to perform with the aid of the right device or tool. We push to define and create more natural interactions knowing that in many cases the addition or use of a device or tool may make interactions less natural, but it’s okay if it ultimately helps us accomplish tasks easier. I guess you could say it’s like taking one step back to take two steps forward.

Related Reading

• Avoiding NUIcide
• Interact Issue 8 from Blueprint Usability contains a great little piece on “Cuelessness”
• The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places by Byron Reeves and Clifford Nass (thanks to Blueprint Usability for this recommendation)