When people have their focus on something separate from their own bodies, that thing becomes an extension of their bodies. Bodymind is not bound by skin. And collaborating, communicating bodyminds meld on an object of common interest.

TeleKinesics

The internet is dispersing our workspaces globally, and the same is happening to our bodies.

The anthropologist, Ray Birdwhistell coined the term “kinesics“, referring to the interpretation, science, or study of body language.

I invented a word: “telekinesics”. I define it as, “the science of body language as conducted over remote distances via some medium, including the internet” (ref)

My primary interest is the creation of body langage using remote manifestations of ourselves, such as with avatars and other visual-interactive forms. I don’t consider video conferencing as a form of virtual body language, because it is essentially a re-creation of one’s literal appearances and sounds. It is an extension of telephony.

But it is virtual in one sense: it is remote from your real body.

Video conferencing, and applications like Skype are extremely useful. I use Skype all the time to chat with friends or colleagues. Seeing my collaborator’s face helps tremendously to fill-in the missing nonverbal signals in telephony. But if the subject of conversation is a project we are working on, then “face-time”, is not helpful. We need to enter into, and embody, the space of our collaboration.

Screen Sharing

This is why screen sharing is so useful. Screen sharing happens when you flip a switch on your Skype (or whatever) application that changes the output signal from your camera to your computer screen. Your mouse cursor becomes a tiny Vanna White – annotating, referencing, directing people’s gazes.

Michael Braun, in the blog post: Screen Sharing for Face Time, says that seeing your chat partner is not always helpful, while screen sharing “has been shown to increase productivity. When remote participants had access to a shared workspace (for example, seeing the same spreadsheet or computer program), then their productivity improved. This is not especially surprising to anyone who has tried to give someone computer help over the phone. Not being able to see that person’s screen can be maddening, because the person needing help has to describe everything and the person giving help has to reconstruct the problem in her mind.”

Many software applications include cute features like collaborative drawing spaces, intended for co-collaborators to co-create, co-communicate, and to to co-mess up each other’s co-work. The interaction design (from what I’ve seen) is generally awkward. But more to the point: we don’t yet have a good sense of how people can and should interact in such collaborative virtual spaces. The technology is still frothing like tadpole eggs.

Some proponents of gestural theory believe that one reason speech emerged out of gestural communication was because it freed up the “talking hands” so that they could do physical work – so our mouths started to do the talking. Result: we can put our hands to work, look at our work, and talk about it, all at the same time.

Screen sharing may be a natural evolutionary trend – a continuing thread to this ancient  activity – as manifested in the virtual world of internet communications.

Steve makes some interesting observations. But, like so many attempts at teaching us “how to read” body language, Steve makes several assumptions that miss the highly contextual, and highly tenuous nature of interpreting emotion via email.

In fact, email is often used by people as a way to avoid emotion or intimacy. It’s an example of asynchronous communication: an email message could take an arbitrary amount of time to compose, and it could be sent at an arbitrary time after writing it. Thus, email is not a reliable medium for reading one’s emotions. It’s hard to lie with your body. It’s much easier to lie with a virtual body. With email, you don’t even have a body.

Damn That Send Button

Actually, I wish I could say that I have always used email in a premeditative, calculated way. I have been guilty of sending email messages in the heat of an emotional moment. A few too many of those emails have lead me to believe that the SEND button should be kept in a locked box in a governmental facility. And the box should have a big sign that says, Are You Sure?

People often make the mistake of assuming that a given communication medium provides a transparent channel for human expression. Oddly enough: email can bring out certain negative qualities in people who may not be negative in normal face-to-face encounters.

People don’t take into account the McLuhan effect, and assume the message is determined only by the communicators. Steve says this about flame mail:

“You probably don’t need me to tell you this, but when you receive what we affectionately call flame mail – where someone lets loose on you in a big, ugly way – that’s aggressive behavior. In other words, they’re acting out like a child throwing a temper tantrum and it’s not about you, it’s about them. I know it’s tempting to think it’s just a misunderstanding, but ask yourself, why did they assume the worst?”

But it’s not just “about them”. It’s also about the medium – an awkward, body-language-challenged medium.

Also, people can feel “safe” behind the email wall (meaning they know they won’t get punched in the face – at least not immediately).  There’s something about the medium that can cause people to flame – EVEN if they are not normally flame-throwers. Jaron Lanier in You Are Not a Gadget gives a good explanation for how and why this phenomenon occurs. Read the book, even if you don’t always take Jaron seriously. He is brave and bold, and he challenges many assumptions about internet culture.

Everyone has stories about email messages they wish they had never written, or email messages they wish they had never read.

It’s wise to understand how media mediates our interactions with each other. That is an important kind of literacy: a literacy of understanding media effects.

———————

I was speaking with my colleague Michael Nixon at the School of Interactive Art and Technology. We were talking about body language in non-human animated characters. He commented that before you can imbue a virtual character with apparent sentience, it has to have the ability to GAZE – in other words, look at something. In other words, it has a head with eyes. Or maybe just a head. Or… a “head”.

Here’s the thing about gaze: it pokes out of the local (“lonely”) coordinate system of the character and into the global (“social”) coordinate system of the world and other sentient beings. Gaze is the psychic vector that connects a character with the world. The character “places it’s gaze upon the world”. Luxo Jr is a great example of imbuing an otherwise inanimate object with sentience (and lots of personality besides) by using body language such as gaze.

I have observed something missing in video conferencing. Gaze. Notice in this set of four images how the video chat participants cannot make eye-contact with each other. This is because they are not sharing the same physical 3D space. Nor are they sharing the same virtual 3D space!

Gaze is one of the most powerful communicative elements of natural language, along with the musicality of speech, and of course facial and bodily gesture. This is especially true among groups of young single people in which hormones are flying, and flirtation, coyness, and jealousy create a symphony of psychic vectors…

At There.com, I designed the initial avatar gaze system. With the help of Chuck Clanton, I created an “intimacam”, which aimed perpendicular to the consensual gaze of the avatars, and zoomed-in closer when the avatar heads came closer to each other.

The greatest animators have known about the power of gaze for as long as the craft has existed. This highly-social component of body language has a mathematical manifestation in the virtual spaces of cartoons, computer games, and virtual worlds. And it is one of the many elements that will become refined and codified and included into the virtual body language of the internet.

Human communication is migrating over to the internet – the geo-cortex of posthumanity. Text is leading the way. Body language has some catching up to do. Brian Rotman has some interesting things to say along these lines in his book, Becoming Beside Ourselves.

We can learn a lot from Pixar animators, as well as psychologists and actors, as we develop virtual worlds and collaborative workspaces.

————–

In response to my earlier post, Laban-for-animators expert Leslie Bishko made this comment:

“My .2c – breath promotes the illusion of sentience, gaze promotes the illusion of interaction and relationship!”

I am a gmail user, but more recently I have considered switching.

Every so often, I notice a new gmail feature. Google is usually kind enough to let me know that a new feature has been introduced, such as offering me the option to try the “new look”, although after I say “no thank you” which I always do, I keep getting notifications to try the “new look”, even though I had already said “no thank you” to the “new look”. Thanks Google, but please STOP TELLING ME ABOUT YOUR “NEW LOOK”.

And then there is the little yellow “Important” symbol that one day magically appeared next to some of my messages. When I roll over the symbol I see the text, “Important mainly because of the people in the conversation”.

Yo Google: how ’bout if I decide what’s important.

One person in the Google forums complained about gmail tagging her message as: “Important mainly because of the words in the message”. She says, “Can we stop with the idiotic messages from Google, as if our paternalistic uncle was looking out for us?”

Consider

But that’s not what I want to talk about: I want to talk about a feature which is the ultimate example of Google developers trying to be oh so clever but just coming across as stupid. I’m talking about the text that appears when I’m composing an email to someone, which says, “Consider including: John, Rebecca…” And so on.

Peter Thomas, one of the many bloggers who has complained about this ridiculous feature, summarizes it:

“When you type an e-mail, Gmail comes up with a list of people that you may like to also copy it to. Let’s pause and just think about this. You are writing an e-mail, generally the first thing that you do is to type in the address of the person (or people) you are writing to. Gmail has a useful feature that scans your previous mails, so typing “Pe” will bring up “Peter Thomas” as an option. So far so good….

…but then, gmail offers a list of people that you may consider including as recipients of your email, based on simple association. Hello? What if I am emailing a colleague to complain about the boss? I certainly don’t want to include the boss, and it scares me that his name is sitting up there, a mouse-click away from disaster. Or what if I am plotting a surprise birthday party for Beth? Including Beth is specifically NOT what I want to do.

And…what if the person is DEAD?

I found this on the Google forums:

“I deleted my dead friend as a contact which was traumatic enough, but having google STILL suggesting I include her when there’s honestly nothing I’d like better than to be able to include her BECAUSE SHE’S DEAD.  How do I make this stop?!?!?!“

Note to Google:

Please get out of the business of reading our minds. You suck at it.

Peter Thomas concludes: “This “feature” is bad enough to have merited me writing to Google asking them to remove it, or at least make it optional. Their support forums are full of people saying the same. It will be interesting to see whether or not they listen.”

Do a search for “consider including”, and you’ll come across several people railing against this act of stupidity from Google. My blog post is not original. Yet I feel compelled to add another voice to the chorus.

Do I have any conclusions or insights? Not really, other than my opinion that any good thing can turn bad when it gets too big and too powerful. Google is generally a good thing. But I think Google is getting too big and too powerful. And I am getting smaller and less powerful, in relative terms. I want to be completely in charge of how I communicate with my friends and colleagues.

The fact that Google is brimming with young, clever, cocky geeks does not make for an agreeable form of world domination.

IMHO.

“Talk to the Virtual Hands: Body Language of Both Speaker and Listener Affects Success in Virtual Reality Communication Game“

Researchers at the Max Planck Institute found that “…virtual communication usually lacks the body gestures so common in face-to-face interactions”.

Usually?

The researchers found that …”the lack of gestural information from both speaker and listener limits successful communication in virtual environments.”

That’s quite an insight.

They also found that “participants move much less in a virtual environment than they do in the “real world.”

Remarkable.

My smiles were his tail wags; his tail wags were my smiles. Because of neuroplasticity, the ability for our brains to adapt and adjust, we were able to fuse semiotically across species lines.

This communication across species lines is analogous to people and software interpreting signals across the internet. We have invented new forms of punctuation to make up for a lack of physical expression in emails and text chats. I would say the same is true for 3D games and virtual worlds. But avatars, no matter how awesome-looking, are terribly clunky as instruments for realtime expression.

Meanwhile, new forms of punctuation have been invented: small, packaged symbols. They are quick to create, and they travel efficiently across the internet. Smileys and emoticons have more currency and emotional leverage than avatars, because they live in typographical soil: an ecosystem that is still much more established and pervasive than virtual worlds. Perhaps text will continue to become more electric, dynamic, intelligent, and integrated with graphical interfaces, such that smileys will evolve into avatars.

The internet is accelerating our posthuman evolution. We will come to have a deeper understanding of our animal cousins – because the primal affordances of the biosphere will be better-understood. Wha? you might say.  Jaron Lanier has already been talking about this kind of stuff for a long time – this idea that (with virtual reality) we will be able to “become” lobsters or snakes or cloud-sized creatures. I mention Jaron in a previous post, and the ways in which our bodymaps adjust to posthuman communication.

It’s not just about imagination: it’s about communicating and having a form of body language that is compatible with the internet. More and more of our communication is migrating to the internet. And since living languages evolve (including body languages) the new ecology of the internet will fertilize new forms of gesture, sound, moving text, and other dynamical forms.

What does this have to do with tails and brains?

Me and Higgs had established a body language bond. New kinds of body language bonds are emerging as we interact through the internet. Our brains are adapting.

Micha Cardenas became a Dragon in Second Life for 365 hours straight. What happened to her brain? I can imagine that people who spend large portions of their lives as Furries with animated tails have dreams of expressing with their tails and ears, like the Na’vi. These ideas are covered more thoroughly in The Tail Wagging the Brain.

Speaking Dolphin

Researchers from Aberdeen University and the Polytechnic University of Catalonia found that dolphins use discrete units of body language as they swim together near the surface of water. They observed efficiency in these signals, similar to what occurs in frequently-used words in human verbal language.

As human natural language goes online, and as our body language gets processed, data-compressed, and alphabetized for efficient traversal over the internet, we may start to see more patterns of our embodied language that resemble those created by dolphins, and many other social species besides. The background communicative buzz of the biosphere may start to make more sense in the process of whittling our own communicative energy down to its essential features, and being able to analyze it digitally. With a universal body language alphabet, we might someday be able to animate our skin like cephalopods, or speak “dolphin”, using our tails, as we lope across the virtual waves.

You may have had the following experience: you are exchanging texts in an online chat with a friend. There is a long period of no response after you send a text. Did you annoy your friend? Maybe your friend has gone to the bathroom? Is your friend still thinking about what you said? One problem that ensues is cross-dialog: during the silent period, you may change the subject by issuing a new text, but unknowingly, your friend had been writing some text as a response to your last text on the previous topic. You get that text, and – relieved that you didn’t annoy your friend – you quickly switch to the previous topic. Meanwhile, your friend has just begun to respond to your text on the new topic. The conversation bifurcates – simply due to a lack of nonverbal signaling.

Like frogs in boiling water, most of us are not aware that our bodies are slowly dissolving as we engage increasingly in text-based communication, which is often asynchronous (or at least running at lower than conversation-rates). My theory: new forms of body language are emerging in the absence of our real bodies. Smart design of visual/interactive interfaces can adapt to this natural evolution. I don’t see it as a choice. It’s simply a part of our evolution – our adaptability.

Jill Chivers, in the blog, “I’m Listening – the Power and Magic of Listening in Everyday Lives“, makes a great case for reaching for the phone when repeated email pings are not getting through to someone, or for going face-to-face, when phone calls are left unanswered.

Call her old-fashioned, call her a Luddite. But she is simply suggesting that we all need to stay connected in ways that maximize our body language. It’s not an anti-technology stance. In fact, I would argue that we need more technology and smarter technology – just that it has to be the kind of technology that manifests embodiment over the internet – in whatever forms it takes. Without bodies, virtual or otherwise, and without the synchrony of realtime bodies, voices, and some stream of co-presence, we tend to fragment into text-like pieces.

Some people like deconstructing themselves into textual fragments. Sometimes I like it – I can hide behind my well-crafted words. But I don’t like the fact that I like it. I don’t want to like it anymore than I do. I would prefer to like connecting with people more in realtime, like I used to – before the world was wired.

Finally, here’s a relevant piece by Si Dawson

http://sidawson.org/2010/06/talking-by-text-sucks-how.html

That avatar is looking at ME! Creepy.

Wagner James Au comments on this effect in a blog post. He quotes the engineer who added this feature (Koji Nagashima):

“On a cinematic project,” Koji explains, “all animators carefully make animation for eyes. But in our world, the program needs to take care of that.” Eye animation in a virtual world or MMO is challenging because the avatar’s position or the user’s camera changes so often. “That’s very interesting for me,” Koji says.

Okay guys. It’s interesting, I agree. And it’s a cute trick. But it makes no sense to me. Have you thought about the media effects? Do you understand how this creeps out users? In cinema, there is this concept of Breaking the Fourth Wall, which refers to a fictional character acknowledging the reader/viewer/audience, and acknowledging the fact that he or she is fictional. The fourth wall, in this case, is the computer screen. So, what is the reason you broke this wall?

I developed avatar gaze code for There.com. This is described in my chapter, The Three Dimensional Music of Gaze. Gaze is a powerful form of body language, and virtual worlds, by and large, are still lacking in this form of expressive connectivity. The silent act of shooting a bit of eye contact on someone’s avatar can be a signal to start a conversation. It can show attraction, and it can show WHO YOU ARE TALKING TO (one bit of affordance that is often missing in virtual worlds). It can also create romantic effects, as shown in these pictures from There.com (the left image shows a prototype I developed with Chuck Clanton for chat props).

The Look

In real life, my wife only needs to make one quick glance at me, and I know that it is time to take out the trash. Or…depending on the timing, or the situation…it may mean something else entirely: something which is more fun than taking out the trash. This simple bit of body language is powerful indeed. Virtual gaze can enliven virtual worlds – infusing silent communicative energy between avatar faces. Virtual gaze gives virtual worlds increased validation as a communication medium of embodiment.

Conclusion

As far as Koji’s trick of having avatars break the fourth wall, I’d like to hear what you think. How does this effect the Blue Mars experience? Ironyca’s critique says it better than I could:

“At least she kept looking at me! I have always thought of an avatar as a virtual representation of me in cyberspace, but perhaps Blue Mars disagrees. To me, the identity immersion was completely broken by the fact my avatar liked to look at me (i.e. gaze at the “camera”), and sometimes her posing even looked flirtatious. I was highly disturbed by the fact, that my idea of her and me being the same was thrown overboard, when she continously decided to turn her head and smile at me. If she is looking at ME, I can’t be HER.”

Aside from the micro-gestures used on specific widgets for touch screens (like the Apple iPad/iPhone “slide to unlock” gesture), there are some pretty crazy ones. Ready for this?

Shaking your mobile device
Annalee sez, “Ever get pissed off at your phone and shake it up and down until it reboots? You could be benefitting from a patent infringement! Intellectual Ventures owns a very broad patent on moving your mobile device around (basically, shaking your phone). If any company dares to create a product which relies on somebody shaking a mobile device to reboot, they’d better cough up some cash to license this gesture from Intellectual Ventures.”

Other examples include: Making hand gestures to move icons around on your phone, Flicking your pen at something, and Moving 3D objects in a virtual environment. 

Many of these have never been challenged in court. But here’s a thought: multi-touch screens, virtual worlds, avatars, gestural interfaces, etc., are creating a rich environment for human visual language to grow in new directions, in big ways.

Our body language vocabulary has always been very rich, but now we have technologies that will allow us to discover and invent entirely new expressions. Will these patent-hungry companies try to gobble up as many potential physical expressions as possible, thereby owning the space of future gestural expression before it has time to mature and settle into the scope of Natural Language?

The answer is not easy because the technology must evolve along with its use.

Here’s one more example that Annalee gives:

Two-handed motions
“Here’s a seriously weird one. A company called GestureTek owns several patents on optical systems for sensing gestures, but the most bizarre is their insanely broad patent on a device that tracks two-handed motions. If you want to make a device that detects two-handed movement (charmingly called “bimanual movements” in the patent), you should be paying a licensing fee to GestureTek every time you do jazz hands.”

Jaron Lanier believes that humans of the future will be truly “multihomuncular”. Why? Because with virtual reality, and new flavors of virtual worlds and augmented reality, we will be able to inhabit avatars that do not necessarily look like us. Jaron’s favorite example is the cuttlefish – a mollusk that is able to animate its skin in order to express itself. But he goes further than this. In virtual reality, you might be able to wiggle your toes to make the clouds move. If you do this kind of thing enough times, your homunculi will start to morph to accommodate this part of the environment as an extension of your body. And as far as the brain is concerned, there is no clear difference between a virtual body and a real one.

At Microsoft, Lanier is working on a project called Somatic Computing. His team is exploring the Kinect as a way to translate real body movement into “computational gestures”.

This video shows Lanier being interviewed by a very non-Jaron-Lanier-looking person. At the end he explains how “being” a molecule can help you achieve deep intuition for what it is like to dock with another molecule. This has radical implications for education.

Personally, I find most uses of virtual worlds for education to be droll. There are still universities constructing virtual classrooms with chairs, podiums, and chalk boards (yawn).

A teacher should be able to demonstrate the chemistry of water by turning into a water molecule. Not only that, the teacher should be able to control all the students’ virtual cameras to swoop into the scene. Even better: all the students should be able to turn themselves into hydrophylic molecules and dance with the teacher. Not trying to get kinky here. Just sayin’.

The current popular understanding of virtual worlds is quite unimaginative. And its true potential to expand minds and offer shared experiences has barely been tapped. Lanier has a following and the publicity to maybe help shift the conversation back to what it was in those early hippy days when he and his friends were trying this stuff out for the first time.

Prepare for mind expansion. Without acid.