On Avatars and Galaxies: The Uncanny Valley Extended

April 18, 2011

I just happened upon this blog post by Jean-Claude Heudin…


…where he takes the concept of the uncanny valley and applies it to the more general notion of representation of complexity. Very interesting observation.

Jean-Claude sez: “if a very small detail can transform an empathic avatar into a monster, it can also substantially decrease the benefits of a model and its representation“.

As I point out in my book, the uncanny valley concept applies to expression and movement as well as appearances. I have long suspected that the uncanny valley might apply to other things as well. It might have a more fundamental explanation. Jean-Claude appears to be thinking in this larger, more general realm.

One conclusion is this: human cultural evolution requires us to manage more and more complexity. We cannot do this given the tools that genetic evolution has given us….unless we design the right representations that filter complex phenomena in the right way for us to understand them, and act upon them.

Building engaging virtual worlds and designing data visualizations to explain colliding galaxies are both constrained by one thing: delivering some subset of the phenomenon – well-chosen and well-crafted – for the human mind to grasp…sympathetically.

Using Kinect to Puppeteer my Avatar? My Arms are Getting Tired Just Thinking About It

March 24, 2011

Wagner James Au pointed me to his New World Notes blog post about Microsoft’s Kinect – hooked up to Second Life. It highlights a video made by Thai Phan of ICT showing a way to control SL avatars using Kinect.

The Kinect offers huge potential for revolutionizing user interaction design. Watch this video and you will agree. But I would warn readers against the knee-jerk conclusion that the ultimate goal of gestural interfaces is to allow us to just be ourselves. Let me explain.

Multi-Level Puppeteering

Either because of the clunky interfaces to the Second Life avatar, or because Thai is smart about virtual body language messaging (or both), he has rigged up the system to recognize gestures – emblems if you will. These are interpreted into semantic units that trigger common avatar animations. One nice procedural piece he added was the ability to start and stop animations, holding them in place, and then stopping them, based on the user’s movements.

Wagner James Au points out the powerful effect and utility that would come about if we had a more direct-manipulation approach, whereby all the body’s motions are mapped onto the avatar. He makes an open call for “different variations of Kinect-to-SL interaction, experimenting with the most natural body-to-avatar UI”. He cites the avatar puppeteering work I did while I was at Linden Lab. He also cites the continuing thread among residents hoping to have puppeteering revived. We are all of similar minds as to the power of direct-manipulation avatar animation in SL.

But, as my book points out, direct gestural interfaces are not for everyone, and … not all the time! Also, some people have physical disabilities, and so they cannot “be themselves” gesturally. They have no choice but to use virtual body language to control their avatar expressions.

And for people like Stephen Hawking, the Kinect is useless. Personally, I would LOVE it if someone could invent a virtual body language interface as an extension to Hawking’s speech synthesizer, to drive a Hawking avatar. Wouldn’t it be cool to witness the excitement in Hawking’s whole being while describing the Big Bang?

Throttling the Gestural Pipeline

But, getting back to the subject of those of us who are able to move our bodies…

The question I have is…WHEN is whole-body gestural input a good thing, and WHEN is it unnecessary and cumbersome? Or moot?

Here’s a prediction: eventually we will have Kinect-like devices installed everywhere – in our homes, our business offices, even our cars. Public environments will be installed with the equivalent of the Vicon motion capture studio. Natural body language will be continually sucked into multiple ubiquitous computer input devices. They will watch our every move.

With the likelihood of large screens or augmented reality displays showing our avatars among remote users, we will be able to have our motions mapped onto our avatars.

Or not.

And that’s the point: we will want to be able to control when, and to what degree, our movements get broadcast to the cloud. Ultimately, WE need to have the ability to turn on or off the distribution of our direct-manipulation body language. The Design Challenge is how to provide intuitive controls.

The Homuncular Kinection

I have a crew of puppeteers in my brain: bodymaps, homunculi, controllers in my prefrontal cortex, mirror neurons, and other neural structures. So that my arms don’t get tired from having to do all my avatar’s gesturing, my neural puppeteers are activated when I want to evoke mediated body language. I do it in many ways, and across many media (including text: eg, emoticons).

Stephen Hawking also has humuncular puppeteers in his brain.

Ultimately, I want several layers of control permitting me to provide small-motion substitutions for large motions, or complex bodily expressions. Virtual Body Language will permit an infinite variety of ways for me to control my avatar…including wiggling my index finger to make my avatar nod “yes”.

THIS is where the magic will happen for the future of Kinect interfaces for avatar puppeteering.

Watson’s Avatar: Just Abstract Art?

February 17, 2011

This video describes the visual design of the “avatar” for Watson – the Jeopardy-playing AI that recently debuted on the Jeopardy show.

This is a lovely example of generative art. Fun to watch as it swirls and swarms and shimmers. But I do not think it is a masterpiece of avatar design – or even information design in general. The most successful information design, in my opinion, employs natural affordances – the property of expressing the function or true state of an animal or thing. Natural affordances are the product of millions of years of evolution. Graphical user interfaces, no matter how clever and pretty, rarely come close to offering the multimodal stimuli that allow a farmer to read the light of the sky to predict rain, or for a spouse to sense the sincerity of her partner’s words by watching his head motions and changes in gaze.

Watson’s avatar, like many other attempts at visualizing emotion, intent, or states of human communication, uses arbitrary visual effects. They may look cool, but they do not express anything very deep.

…although Ebroodle thinks there is something pretty deep going on with Watson, as in… world domination.

Despite my criticism, I do commend Joshua Davis, the artist who developed the avatar. It is difficult to design non-human visual representations of human expression and communication. But it is a worthy effort, considering the rampant uncanny valley effect that has infected so much virtual human craft for so long, caused by artists (or non-artists) trying to put a literal human face on something artificial.

What Was Watson Thinking?

Watson’s avatar takes the form of a sphere with a swarm of particles that swirl around it. The particles migrate up to the top when Watson is confident about its answer, and to the bottom when it is unsure. Four different colors are used to indicate levels of confidence. Green means very confident. Sounds pretty arbitrary. I’ve never been a fan of color for indicating emotion or states of mind – it is overused, and ultimately arbitrary. Too many other visual affordances are underutilized (such as styles of motion).

Contradiction Between Visual and Audible Realism

Here’s something to ponder: Watson’s avatar is very abstract and arty. But Watsons voice is realistic … and kinda droll. I think Watson’s less-than-perfect speech creates a sonic uncanny valley effect. Does the abstraction of Watsons visual face help this problem, or make it more noticeable?

Is the uncanny valley effect aggravated when there is a discrepancy between visual and audible realism? I can say with more confidence that the same is true when visual realism is not met with complimentary behavioral realism (as I discuss in my book).

Am I saying that Watson should have a realistic human face – to match its voice? Not at all! That would be a disaster. But this doesn’t mean that its maker can craft abstract shapes and motions with reckless abandon. Indeed, the perception of shapes and colors changing over time – accompanied by sound – is the basis for all body language interpretation – it penetrates deep into the ancient communicative energy of planet Earth. Body language is the primary communication channel of humans as well as all animals. Understanding these ancient affordances is a good way to become good at information design.

Hmm – I just got an image of Max Headroom in my mind. Max Headroom had an electronic stutter in his voice as well as in his visual manifestation. Audio and video were complimentary. It would be kinda fun to see Max as Watson’s avatar.

What do you think, my dear Watson?


February 3, 2011

When my wife lay in a hospital bed for several weeks with a burst appendix, I spent a lot of time by her side. I was horrified by the cacophony in the recovery ward. How can anyone expect to heal when they are surrounded by a jungle of machines beeping incessantly? And what about the hospital staff? I decided that the nurses could just as easily be responding to the sound of bird calls. These machines could be playing the sounds of songbirds instead of emitting the sonic equivalent of a finger repeatedly poking you in the eye. Not only would bird calls make for a more pleasant soundscape in the hospital, but different bird calls could be used for different meanings.

Below is a poem I wrote many years ago which expresses my feelings about the way machines talk to us.


I came to the conclusion the other day
That our machines have something to say
Our cars, our phones, our computer screens,
Our ovens and our bank machines

They’re learning how to speak to us
Yes, times are changing and so they must
How could we ever get along
If they didn’t tell us when something’s wrong?

“Your seatbelt’s off”
“Don’t forget your card”
“Don’t click me there”
“Don’t press too hard”

But the builders of these technologies
Have yet to give them personalities
I think our machines might benefit, you see,
By having a bigger vocabulary!

There’s another gripe I need to share
A concern for which you may not care
There’s pollution in my neighborhood
I’d love to end it if I could

You hear that incessant beep beep beep?
It woke me from my morning sleep
Six blocks east on Main and First
A van just went into reverse

The device which generates this din
was built to reach the ears within
a ten block radius of this city
that all may know the van’s velocity

Folks living in a future world
May wince and twitch at what they once heard
Recalling the voice of our technology
This low-resolution cacaphony


Beep beep beep beep!

This is the third post of my new blog. I expect to have more things to say about body language in user interface design – not just regarding avatars in virtual worlds. I think the subject of virtual body language spans across all kinds of technology.  I’d love to hear your thoughts!


Body Language and Web Site Design

January 19, 2011

“Body language” and “web site design” are two phrases we don’t often see in the same sentence. But I believe body language literacy is desperately needed in the world of web design. Let me explain.

But first, I need to get something off my chest….

I am cosmically frustrated, annoyed and cynical about the current state of web design. Sending a message to my doctor REALLY COULD be easy and intuitive. Booking an airline ticket REALLY DOESN’T have to involve sighing and grumbling and scrolling through absolute nonsense in search of the right button to click – if there is one.

Here, Tommy Oddo points out How a Bad User Experience Hurts a Healthy Brand.

Too many companies are skimping on genuine customer service and putting up web sites as if to say, “talk to the hand”.

The “hand” is not doing its job. A generation of socially-illiterate, visually-illiterate, and literally illiterate geeks are determining the way we interact with each other, how we shop, how we manage our finances….just about everything we do these days. The situation is dire, and I believe that far too many people don’t care or don’t notice. They are like the proverbial frogs in the pot of water, slowly being brought to boil: they don’t realize the negative effect it is having on the quality of their lives – that is, until their insides explode.

If you don’t think that the following web site (the main page for GoDaddy) is a miserable mess…

…then you should take a little stroll. Breathe deep. Feel, listen, and watch the natural world for a spell. Nature speaks to us in a language simple and pure, yet amazingly complex and expressive. Nature can be our best teacher in gaining clarity in these situations. I trust that you agree with me at least a little bit. If you don’t agree with me, then you are a frog.

Thank God we don’t see much of this kind of thing anymore…

Getting to my Point

So, what does body language have to do with making better web pages? Lots. Not only are web sites graphical compositions that rely on nonverbal communication (like all forms of graphic design), they are also interactive. Web sites are conversational agents. They are clicked-through, scrolled-through, talked to, and read-from. Body language is the Music that carries human communication through time. Good interaction design relies on this soundtrack.


“Affordance” – another one of those great idea-words that is sometimes diluted from over-use, causing people to think it is something less than it really is. Donald Norman reminds us to remember the original meaning:

The word “affordance” was originally invented by the perceptual psychologist J. J. Gibson (1977, 1979) to refer to the actionable properties between the world and an actor (a person or animal). To Gibson, affordances are a relationship. They are a part of nature: they do not have to be visible, known, or desirable. Some affordances are yet to be discovered. Some are dangerous. I suspect that none of us know all the affordances of even everyday objects.

This is a subtle and important concept. And when Donald Norman says that we may not know all the affordances of everyday objects, I think he would agree that web sites are everyday objects.

What Joe Sez

Joe Gillespie addresses this question of body language in web site design:

Many pages I see communicate ineptness, even from large companies who should know better. The pages may be technically perfect, but they have been produced by people without any visual communications skills. They are like the karaoke artist who can’t sing. They are like the person talking to you with their head bowed down, they don’t know that they are sending the wrong signals, they are not in control of their communication.

Then there are the sites like the glitzy wine label, trying to fool people that they are something that they are not. They are trying too hard, shouting every trick in the book at the top of their voices – animation, sound, flashing, blinking waving their hands in the air. They are making noise, not communicating, and wasting a lot of bandwidth for all their efforts.

Okay, do you think this web page is nicely designed? …. Well, that’s what I thought when I saw the screenshot…until I visited the site. Just go there, and you’ll see what I mean. http://www.ohioairquality.org/ Roll around on the blue buttons and watch your retinas curdle.



Communicating with a Body and Getting Good At It

Some people are really good at body language. They know how to make themselves clear. They know how to use their voice, eye contact, and gesture to get the message across, or to show you that they understand with you are saying. What happens when we are remote and have to rely on computers for communication? Online communication is in a crisis. Without our real bodies involved, we have to use other means to generate meaning, understanding, and context. Web sites – increasingly the medium through which we do so many things – are terribly poor – on average – at delivering the all-important nonverbal channel of natural language.

I will conclude with FOUR rules of thumb for designing good web site body language:

1. Make Eye Contact

Did you know that a web page can look at you? Have you ever wondered where you are supposed to “go” in a web site in order to do what you want to do? A real body has a head and eyes – and we use this focal-point to direct attention and to quickly make a connection. (Hands are also used). Too many web sites are cluttered with a mangle of widgets without focus. At any given time, there are usually only a few things the user needs to do.  A smartly-designed web site knows how to make a good guess as to what the user may be looking for – and to pull the eye and clicking-cursor to a natural focus of attention. Cross-acknowledgement – and maintaining focus – are important in natural body language… as well as in interaction design.

2. Only Animate What Needs Attention

All I have to say is that animated ads make me crazy – like, when I land on a web site that has Las Vegas pixel-guns shooting at my eyeballs from several directions, I start making tunnels with my hands and looking through the hole. I’m not kidding.

3.  Keep Interactive Context

If you are a web designer who codes, look for ways to hold in the web site’s short-term memory the recent activities between the user and the web site. Even if you can’t code, there are still ways to find flow with the user’s actions (as extended through time). Imagine that you are simulating a natural conversation between two humans. You don’t want the user to be lost in an eternally cluttered, unchanging, undifferentiated wall of choices. This is subtle, hard to define, and best explored on a case-by-case basis. But I think it’s important and I suspect we’ll be hearing more about this over time.

4. Stop Geeking out!

If you are a web designer, go for a walk! Listen to the birds. Build a tower of blocks with a child. Hang out with a dog and exchange some canine body language. Feel the interactive energy that flows effortlessly in the natural world. The secret to good interaction design is not found in cutting-edge technology. It’s found in the bodies of creatures that have been interacting on Earth for several billions of years, and building lifestyles around each other’s affordances. The human body is the original user-interface. We have much to learn from watching how it works.

Do you Have any Fave Good Body-Language Web Sites?

I want to write a blog post that lists bad body language web sites and good body language web sites…and critique them using these four rules-of-thumb. I have a few in mind. Give me some of your own examples!

/nods /waves