Avoiding NUIcide

The early 1990s witnessed a major turning point in the evolution of user interfaces. We transitioned from command-line interfaces, like DOS, to graphical user interfaces (GUI), like Microsoft Windows. Now, we are at another major turning point as we evolve from GUI to natural user interactions (NUI). I consider myself lucky to be able to say I was an active participant in the commercial birth of GUI and now again with NUI. In this article I talk about a few key lessons for 3D air gestures that will help you avoid ‘NUIcide’.

Micro & Macro-Usability

The excitement and adoption of GUI was extraordinary. While it offered a huge step forward in simplifying human-computer interaction for the masses, application producers made a common mistake. By using GUI’s standard building blocks and objects like menus, icons, text boxes, buttons, drop-down list boxes, etc., they thought their applications would automatically be usable. After all, people quickly and easily learned how to use these standard objects; so some text input here and a button press there and presto, GUI = Great User Interface! However, though people could look at a screen and understand how to mechanically interact with the individual UI objects, they had no idea how to accomplish their larger goals. This brings us to two concepts I’d like to introduce:

• Micro-usability: the user’s capabilities are kept in mind as discreet object-level interactions are designed to be familiar and easy (e.g., activating a button).
• Macro-usability: the user’s goals are kept in mind as tasks are designed to be easily accomplished (e.g., purchasing an item).

A combination of micro and macro-usability is essential, but when solutions are designed with only micro-usability in mind then, paraphrasing a co-worker of mine, you’re ‘designing for GUI, instead of designing for people.’ We’re still at the early stages of our evolution toward NUI, but is history about to repeat itself? Are solutions being designed for NUI, instead of designed for people?

Design for People

A common NUI mistake is related to the forms of user input: gesture (and speech). The assumption is that because we are all already experts at gesturing, then NUI applications using gesture are once again automatically easy-to-use (i.e., micro-usability). After all, there’s no need to teach you how to move your body or individual body parts. But, the technology is not advanced enough yet to enable us to move however we want whenever we want. When that time comes, interactions will be much more natural. Until then, people still need to learn the system’s limited vocabulary. In other words, how it wants you to move, and when to do it.

Today’s nascent solutions are akin to your friend with a yellow belt in Karate who says he’s awesome and can deflect any attack! However, before you throw a punch to test him, he backs you up and shows you the one type of punch you should do slowly so he can block it! Your friend and many of our initial NUI systems suffer from yellow-belt syndrome. Consequently, we cannot assume micro-usability is taken care of and we need to put a lot of energy into designing the low-level mechanical interactions for NUI. Carrying past lessons forward, we know that even when we get micro-usability right, we still need macro-usability to ensure we are designing for people and their tasks/goals, not NUI (suggested reading: Red Route Usability).

Context and Mindset

Many of today’s consumer solutions and devices span multiple domains. For example, on my phone I can be going through my email in one moment and playing a game the next. Even though I may be using the same device or system in both cases, my expectations are different in each case. To see how this affects our design and use of air gestures, let’s look at a user’s mindset when playing an Xbox Kinect game compared to something more administrative like scrolling a list:

In a table-tennis game, I may have to swing my arm as if swinging a real ping-pong paddle. Performing that swing gesture in the right way at the right time is a challenge. In the context of a game my mindset is about fun, and I view challenge as part of the game’s fun. If I get the gesture wrong, I say to myself “right, I’ll try it with more of an up-swing next time; I know I can do better and improve my score. This is fun.”

In the Xbox dashboard, I may have to swing my arm to scroll a list or go to the next page of content. Performing that swing-like gesture in the right way is a challenge. In the context of the dashboard my mindset is about efficiency and utility, and I view challenge as frustration. If I get the gesture wrong, I say to myself “right, that didn’t work; there’s probably a better way. This is frustrating.”

Understanding the difference in user mindset will greatly inform and drive your NUI design decisions. While the point here may seem obvious, there are other similar factors to consider that should help you think more carefully about whether NUI should be used and how for different domains. For example:

• Privacy: NUI as a form of interaction may not necessarily be private. A small gesture on a phone’s touchscreen is innocuous, but a large body movement, like swinging an arm, is more visible.
• Appearance: The context of use may drive concerns around how you appear to others. Gestures that may make me appear silly while playing a game with friends in my living room is part of having a good time. However, gestures that make me appear silly while presenting to co-workers at my office is a non-starter.
• Attention: Some activities will demand and get a person’s full attention. For example, most immersive Kinect games can assume that once you’re playing, you are focused and your gestures are intentional and directed at game play. Other activities, however, cannot make the same assumption, or doing so would be risky. For example, users may multi-task while browsing a list of movies. Full-focus vs. split-focus situations will call for a different set of design considerations in terms of how users engage, disengage, and express intent to interact.
• Fatigue: In the context of a Kinect game, gesturing may be fatiguing but the whole point of the game is to play in a truly physical manner; a little sweat is expected and part of the fun. In other situations like browsing a list of movies, fatigue from gesturing is unexpected, and if it happens it will lead to abandonment.

Design Principles

As designers we create and use a short list of design principles. Sometimes, we even stack rank or prioritize them, acknowledging that conflicts arise and tradeoffs may be necessary. Because we’re talking design, it’s easy to imagine principles like “magic”, “visceral”, “effervescent”, or other cool and creative titles on the list, even at the top of the list. For NUI, however, there is one principle that may surprise you, and not only does it have to be on the list but should be at the top of the list: Reliability (dependable and predictably yielding the same results on repeated trials). When it comes to NUI, especially during this yellow-belt time period, reliable interactions trump all other design principles. NUI interactions, no matter how cool, quickly deteriorate into frustration and abandonment when they only work once in a while.

If you achieve reliable NUI interactions, are you done? Reliable is necessary but not be sufficient, alone it’s unlikely to lead to user delight. Although initially captivated, people will get over the novelty of doing something with an air gesture and become practical again. Hence, you still have to put NUI to use in a compelling way (suggested reading: Beyond Task Completion: Flow in Design).

Retrofitting

In a hurried quest to quickly adopt the latest thing and ride the novelty wave, we often consider retrofitting an old system with the new thing. In this case, we’re talking about layering NUI on top of an existing UI and selling it as a NUI system. This is a classic example of “designing for NUI instead of for people”.

We’ve seen examples of this when applications designed for keyboard and mouse were transplanted onto touchscreens. It’s quite frustrating trying to use a finger to interact with a UI object designed for a mouse pointer. The simple lesson here is don’t force fit a new form of user input onto a solution designed and optimized for a different form of input. Just because you could, doesn’t mean you should. Sounds like common sense, but it happens more often than you think. A new design created for NUI will be more successful because it will address both micro and macro usability…

• Micro-usability: UI objects are optimized for a particular form of input as affordances, interactions, and feedback will be different.
• Macro-usability: New forms of input and interaction can enable entirely different ways of accomplishing goals. For a particular form of input, designing to its strengths and capabilities may lead to different steps and processes to accomplish a task.

When Xbox Kinect launched, the most successful Kinect games were designed from scratch. Of course, there were many ambitious ideas, but the designers did a great job playing to the strengths and capabilities of the input technology (e.g., Kinect Sports and Dance Central). They kept both micro and macro-usability in mind and took full advantage of all key factors within the context of gaming.

For those of you considering or using NUI for non-gaming applications, I hope this basic information will provide a better and more informed starting point.