High Fidelity: Body Language through “Telekinesics”

Human communication demonstrates the usual punctuated equilibria of any natural evolutionary system. From hand gestures to grunts to telephones to email and beyond, human communication has not only evolved, but splintered off into many modalities and degrees of asynchrony.

I recently had the great fortune to join a company that is working on the next great surge in human communication: High Fidelity, Inc. This company is bringing together several new technologies to make this happen.

So, what is the newest evolutionary surge in human communication? I would describe it using a term from Virtual Body Language (page 22):

Telekinesics is a word invented to denote…”the study of all emerging nonverbal practices across the internet, by adding the prefix, tele to Birdwhistell’s, term kinesics. It could easily be confused with “telekinesis”: the ability to cause movement at a distance through the mind alone (the words differ by only one letter). But hey, these two phenomena are not so different anyway, so a slip of the tongue wouldn’t be such a bad thing. Telekinesics may be defined as “the science of body language as conducted over remote distances via some medium, including the internet”. 

And now it’s not just science, but practice: body language is ready to go online…in realtime.

And when I say “realtime” – I mean, pretty damn fast, compared to most things that zip (or try to zip) across the internet. And when we’re talking about subtle head nods, changes in eye contact, fluctuations in your voice, and shoulder shrugs, fast is not just a nicety, it is a necessity – for clear communication using a body.

Here’s Ryan Downe showing an early stage of avatar head movement using Google Glass.

Philip Rosedale, the founder of High Fidelity, often talks about how cool it would be for my avatar to walk up to your avatar and give it a little shoulder-shove, or a fist-bump, or an elbow-nudge, or a hug…and for your avatar to respond with a slight – but noticeable – movement.

It would appear that human touch (or at least the visual/audible representation of human touch) is on the verge of becoming a reality – through telekinesics. Of all the modalities and senses that we use to communicate, touch is the most primal: we share it with the oldest microorganisms.

When touch is manifest on the internet, along with highly-crafted virtual environments, maybe, just maybe, we will have reached that stage in human evolution when we can have a meaningful, intimate exchange – even if one person is in Shanghai and the other is in Chicago.

And that means people can stop having to fly around the world and burning fossil fuels in order to have 2-hour-long business meetings. And that means reducing our carbon footprint. And that means we might have a better chance of not pissing-off Mother Earth to the degree that she has a spontaneous fever and shrugs us off like pesky fleas.

Which would really suck.

So…keep an eye on what we’re doing at High Fidelity, and get ready for the next evolutionary step in human communication. It just might be necessary for our survival.

Screensharing: Don’t Look at Me

Imagine discussing a project you are doing with a small group: a web site, a drawing, a contraption you are building; whatever. You would not expect the people to be looking at your face the whole time. Much of the time you will all be gazing around at different parts of the project. You may be pointing your fingers around, using terms like “this”, “that”, “here” and “there”.

When people have their focus on something separate from their own bodies, that thing becomes an extension of their bodies. Bodymind is not bound by skin. And collaborating, communicating bodyminds meld on an object of common interest.

TeleKinesics

The internet is dispersing our workspaces globally, and the same is happening to our bodies.

The anthropologist, Ray Birdwhistell coined the term “kinesics“, referring to the interpretation, science, or study of body language.

I invented a word: “telekinesics”. I define it as, “the science of body language as conducted over remote distances via some medium, including the internet” (ref)

My primary interest is the creation of body langage using remote manifestations of ourselves, such as with avatars and other visual-interactive forms. I don’t consider video conferencing as a form of virtual body language, because it is essentially a re-creation of one’s literal appearances and sounds. It is an extension of telephony.

But it is virtual in one sense: it is remote from your real body.

Video conferencing, and applications like Skype are extremely useful. I use Skype all the time to chat with friends or colleagues. Seeing my collaborator’s face helps tremendously to fill-in the missing nonverbal signals in telephony. But if the subject of conversation is a project we are working on, then “face-time”, is not helpful. We need to enter into, and embody, the space of our collaboration.

Screen Sharing

This is why screen sharing is so useful. Screen sharing happens when you flip a switch on your Skype (or whatever) application that changes the output signal from your camera to your computer screen. Your mouse cursor becomes a tiny Vanna White – annotating, referencing, directing people’s gazes.

Michael Braun, in the blog post: Screen Sharing for Face Time, says that seeing your chat partner is not always helpful, while screen sharing “has been shown to increase productivity. When remote participants had access to a shared workspace (for example, seeing the same spreadsheet or computer program), then their productivity improved. This is not especially surprising to anyone who has tried to give someone computer help over the phone. Not being able to see that person’s screen can be maddening, because the person needing help has to describe everything and the person giving help has to reconstruct the problem in her mind.”

Many software applications include cute features like collaborative drawing spaces, intended for co-collaborators to co-create, co-communicate, and to to co-mess up each other’s co-work. The interaction design (from what I’ve seen) is generally awkward. But more to the point: we don’t yet have a good sense of how people can and should interact in such collaborative virtual spaces. The technology is still frothing like tadpole eggs.

Some proponents of gestural theory believe that one reason speech emerged out of gestural communication was because it freed up the “talking hands” so that they could do physical work – so our mouths started to do the talking. Result: we can put our hands to work, look at our work, and talk about it, all at the same time.

Screen sharing may be a natural evolutionary trend – a continuing thread to this ancient  activity – as manifested in the virtual world of internet communications.

 

 

Virtual Sentience Requires a Gaze

(This blog post is re-published from an earlier blog of mine called “avatar puppetry” – the nonverbal internet.  I originally wrote it in September of 2009. I’ll be phasing out that earlier blog, so I’m migrating a few of those earlier posts here before I trash it).

———————

I was speaking with my colleague Michael Nixon at the School of Interactive Art and Technology. We were talking about body language in non-human animated characters. He commented that before you can imbue a virtual character with apparent sentience, it has to have the ability to GAZE – in other words, look at something. In other words, it has a head with eyes. Or maybe just a head. Or… a “head”.

Here’s the thing about gaze: it pokes out of the local (“lonely”) coordinate system of the character and into the global (“social”) coordinate system of the world and other sentient beings. Gaze is the psychic vector that connects a character with the world. The character “places it’s gaze upon the world”. Luxo Jr is a great example of imbuing an otherwise inanimate object with sentience (and lots of personality besides) by using body language such as gaze.

I have observed something missing in video conferencing. Gaze. Notice in this set of four images how the video chat participants cannot make eye-contact with each other. This is because they are not sharing the same physical 3D space. Nor are they sharing the same virtual 3D space!

Gaze is one of the most powerful communicative elements of natural language, along with the musicality of speech, and of course facial and bodily gesture. This is especially true among groups of young single people in which hormones are flying, and flirtation, coyness, and jealousy create a symphony of psychic vectors…

At There.com, I designed the initial avatar gaze system. With the help of Chuck Clanton, I created an “intimacam”, which aimed perpendicular to the consensual gaze of the avatars, and zoomed-in closer when the avatar heads came closer to each other.

The greatest animators have known about the power of gaze for as long as the craft has existed. This highly-social component of body language has a mathematical manifestation in the virtual spaces of cartoons, computer games, and virtual worlds. And it is one of the many elements that will become refined and codified and included into the virtual body language of the internet.

Human communication is migrating over to the internet – the geo-cortex of posthumanity. Text is leading the way. Body language has some catching up to do. Brian Rotman has some interesting things to say along these lines in his book, Becoming Beside Ourselves.

We can learn a lot from Pixar animators, as well as psychologists and actors, as we develop virtual worlds and collaborative workspaces.

————–

In response to my earlier post, Laban-for-animators expert Leslie Bishko made this comment:

“My .2c – breath promotes the illusion of sentience, gaze promotes the illusion of interaction and relationship!”

Avatar Gaze Breaks Fourth Wall

I’ve done a lot of pontificating about avatar gaze in virtual worlds (eye contact, and all the emotional and sometimes strange effects that this causes). I’m writing a paper called Virtual Gaze, soon to be published in a book by  ETC Press. While checking on other references to avatar gaze, I came across this blog post by Ironyca about the avatars in Blue Mars. Here’s a picture from that blog:

That avatar is looking at ME! Creepy.

Wagner James Au comments on this effect in a blog post. He quotes the engineer who added this feature (Koji Nagashima):

“On a cinematic project,” Koji explains, “all animators carefully make animation for eyes. But in our world, the program needs to take care of that.” Eye animation in a virtual world or MMO is challenging because the avatar’s position or the user’s camera changes so often. “That’s very interesting for me,” Koji says.

Okay guys. It’s interesting, I agree. And it’s a cute trick. But it makes no sense to me. Have you thought about the media effects? Do you understand how this creeps out users? In cinema, there is this concept of Breaking the Fourth Wall, which refers to a fictional character acknowledging the reader/viewer/audience, and acknowledging the fact that he or she is fictional. The fourth wall, in this case, is the computer screen. So, what is the reason you broke this wall?

I developed avatar gaze code for There.com. This is described in my chapter, The Three Dimensional Music of Gaze. Gaze is a powerful form of body language, and virtual worlds, by and large, are still lacking in this form of expressive connectivity. The silent act of shooting a bit of eye contact on someone’s avatar can be a signal to start a conversation. It can show attraction, and it can show WHO YOU ARE TALKING TO (one bit of affordance that is often missing in virtual worlds). It can also create romantic effects, as shown in these pictures from There.com (the left image shows a prototype I developed with Chuck Clanton for chat props).

The Look

In real life, my wife only needs to make one quick glance at me, and I know that it is time to take out the trash. Or…depending on the timing, or the situation…it may mean something else entirely: something which is more fun than taking out the trash. This simple bit of body language is powerful indeed. Virtual gaze can enliven virtual worlds – infusing silent communicative energy between avatar faces. Virtual gaze gives virtual worlds increased validation as a communication medium of embodiment.

Conclusion

As far as Koji’s trick of having avatars break the fourth wall, I’d like to hear what you think. How does this effect the Blue Mars experience? Ironyca’s critique says it better than I could:

“At least she kept looking at me! I have always thought of an avatar as a virtual representation of me in cyberspace, but perhaps Blue Mars disagrees. To me, the identity immersion was completely broken by the fact my avatar liked to look at me (i.e. gaze at the “camera”), and sometimes her posing even looked flirtatious. I was highly disturbed by the fact, that my idea of her and me being the same was thrown overboard, when she continously decided to turn her head and smile at me. If she is looking at ME, I can’t be HER.”

Body Language and Web Site Design

“Body language” and “web site design” are two phrases we don’t often see in the same sentence. But I believe body language literacy is desperately needed in the world of web design. Let me explain.

But first, I need to get something off my chest….

I am cosmically frustrated, annoyed and cynical about the current state of web design. Sending a message to my doctor REALLY COULD be easy and intuitive. Booking an airline ticket REALLY DOESN’T have to involve sighing and grumbling and scrolling through absolute nonsense in search of the right button to click – if there is one.

Here, Tommy Oddo points out How a Bad User Experience Hurts a Healthy Brand.

Too many companies are skimping on genuine customer service and putting up web sites as if to say, “talk to the hand”.

The “hand” is not doing its job. A generation of socially-illiterate, visually-illiterate, and literally illiterate geeks are determining the way we interact with each other, how we shop, how we manage our finances….just about everything we do these days. The situation is dire, and I believe that far too many people don’t care or don’t notice. They are like the proverbial frogs in the pot of water, slowly being brought to boil: they don’t realize the negative effect it is having on the quality of their lives – that is, until their insides explode.

If you don’t think that the following web site (the main page for GoDaddy) is a miserable mess…

…then you should take a little stroll. Breathe deep. Feel, listen, and watch the natural world for a spell. Nature speaks to us in a language simple and pure, yet amazingly complex and expressive. Nature can be our best teacher in gaining clarity in these situations. I trust that you agree with me at least a little bit. If you don’t agree with me, then you are a frog.

Thank God we don’t see much of this kind of thing anymore…

Getting to my Point

So, what does body language have to do with making better web pages? Lots. Not only are web sites graphical compositions that rely on nonverbal communication (like all forms of graphic design), they are also interactive. Web sites are conversational agents. They are clicked-through, scrolled-through, talked to, and read-from. Body language is the Music that carries human communication through time. Good interaction design relies on this soundtrack.

Affordance

“Affordance” – another one of those great idea-words that is sometimes diluted from over-use, causing people to think it is something less than it really is. Donald Norman reminds us to remember the original meaning:

The word “affordance” was originally invented by the perceptual psychologist J. J. Gibson (1977, 1979) to refer to the actionable properties between the world and an actor (a person or animal). To Gibson, affordances are a relationship. They are a part of nature: they do not have to be visible, known, or desirable. Some affordances are yet to be discovered. Some are dangerous. I suspect that none of us know all the affordances of even everyday objects.

This is a subtle and important concept. And when Donald Norman says that we may not know all the affordances of everyday objects, I think he would agree that web sites are everyday objects.

What Joe Sez

Joe Gillespie addresses this question of body language in web site design:

Many pages I see communicate ineptness, even from large companies who should know better. The pages may be technically perfect, but they have been produced by people without any visual communications skills. They are like the karaoke artist who can’t sing. They are like the person talking to you with their head bowed down, they don’t know that they are sending the wrong signals, they are not in control of their communication.

Then there are the sites like the glitzy wine label, trying to fool people that they are something that they are not. They are trying too hard, shouting every trick in the book at the top of their voices – animation, sound, flashing, blinking waving their hands in the air. They are making noise, not communicating, and wasting a lot of bandwidth for all their efforts.

Okay, do you think this web page is nicely designed? …. Well, that’s what I thought when I saw the screenshot…until I visited the site. Just go there, and you’ll see what I mean. http://www.ohioairquality.org/ Roll around on the blue buttons and watch your retinas curdle.

.

.

Communicating with a Body and Getting Good At It

Some people are really good at body language. They know how to make themselves clear. They know how to use their voice, eye contact, and gesture to get the message across, or to show you that they understand with you are saying. What happens when we are remote and have to rely on computers for communication? Online communication is in a crisis. Without our real bodies involved, we have to use other means to generate meaning, understanding, and context. Web sites – increasingly the medium through which we do so many things – are terribly poor – on average – at delivering the all-important nonverbal channel of natural language.

I will conclude with FOUR rules of thumb for designing good web site body language:

1. Make Eye Contact

Did you know that a web page can look at you? Have you ever wondered where you are supposed to “go” in a web site in order to do what you want to do? A real body has a head and eyes – and we use this focal-point to direct attention and to quickly make a connection. (Hands are also used). Too many web sites are cluttered with a mangle of widgets without focus. At any given time, there are usually only a few things the user needs to do.  A smartly-designed web site knows how to make a good guess as to what the user may be looking for – and to pull the eye and clicking-cursor to a natural focus of attention. Cross-acknowledgement – and maintaining focus – are important in natural body language… as well as in interaction design.

2. Only Animate What Needs Attention

All I have to say is that animated ads make me crazy – like, when I land on a web site that has Las Vegas pixel-guns shooting at my eyeballs from several directions, I start making tunnels with my hands and looking through the hole. I’m not kidding.

3.  Keep Interactive Context

If you are a web designer who codes, look for ways to hold in the web site’s short-term memory the recent activities between the user and the web site. Even if you can’t code, there are still ways to find flow with the user’s actions (as extended through time). Imagine that you are simulating a natural conversation between two humans. You don’t want the user to be lost in an eternally cluttered, unchanging, undifferentiated wall of choices. This is subtle, hard to define, and best explored on a case-by-case basis. But I think it’s important and I suspect we’ll be hearing more about this over time.

4. Stop Geeking out!

If you are a web designer, go for a walk! Listen to the birds. Build a tower of blocks with a child. Hang out with a dog and exchange some canine body language. Feel the interactive energy that flows effortlessly in the natural world. The secret to good interaction design is not found in cutting-edge technology. It’s found in the bodies of creatures that have been interacting on Earth for several billions of years, and building lifestyles around each other’s affordances. The human body is the original user-interface. We have much to learn from watching how it works.

Do you Have any Fave Good Body-Language Web Sites?

I want to write a blog post that lists bad body language web sites and good body language web sites…and critique them using these four rules-of-thumb. I have a few in mind. Give me some of your own examples!

/nods /waves

-jeffrey