I’m here at the Museo Nacional Centro de Arte Reina Sofia in Madrid to take part in a large museum show called “Máquinas y Almas: Arte digital” (“Machines and Souls: Digital Art”), which, as one might expect, includes a lot of contemporary techie work. The exhibition title presumes an opposition, and I suspect that many of the show’s participants might disagree with this implied duality since much of the work here demonstrates a symbiotic relationship between artist and machine. Just as Descartes once separated the mind and body, we now separate machines — which we see as extensions of our body and senses — from something we refer to as our souls.
What is a soul? Some suggest it’s what survives death, joining the other evanescent and matterless essences when our “machines,” our bodies, cease to function. Some claim it has a weight, that the body mysteriously loses 21 grams at the moment of our passing. In this view, it is not DNA but the soul that contains the true self, our sacred and immutable identities.
When a limb is amputated, is part of the soul lost? According to convention, no, it is not. When a lobotomy is performed or a person has a stroke is part of the soul “lost” or diminished? Presumably not — even a person in a vegetative state still has a soul, though we don’t know exactly where it is. Where do we draw the line? Would a brain kept alive but apart from its body still have a soul? Who cares?
I remember a comic book I read as an adolescent, Binky Brown Meets The Holy Virgin Mary, by Justin Green.
It’s an irreverent look at a young man’s tortured Catholic upbringing. In one section, after being inundated with Church dogma regarding the soul, young Binky tries to imagine what it looks like. Where is this thing they keep referring to, and what form does it take? Binky imagines his own soul is a small, amorphous lump located somewhere in the chest, tarnished and dirtied by his accumulated sins. In other narratives, evil scientists and Voodoo priests know how to locate and remove the soul, or to control it, while normal doctors haven’t a clue to its existence or where it hides.
What about machines? Will there come a time when machines have the characteristics of a sentient, self-aware being? Raymond Kurzweil believes that a day will soon come when machines and living beings will merge. He predicts that we will augment ourselves with machines to such an extent that our sense of self will encompass the silicon, the circuitry, and the chunks of titanium — we and the machines will become one, what he calls The Singularity.
During some intensely emotional portions of my recent past, I would dream that I either lost my laptop, or that the screen suddenly went blank, and I would feel traumatized and distraught. The laptop, it seems, was a stand-in for myself. Others may toss and turn when they dream of losing their Blackberries, or their houses burning down, or their cars mysteriously dying, but for me it was my computer.
Our identities can be tied to certain machines, though not just any type. A power drill? Probably not, but a luxury watch, a sexy car, a camera (for a DP, or photographer, maybe?), a typewriter (for a writer stuck in the last century?), and a computer for those like me who use it to store their personal information and to interface with others. The tools we use can represent — at least symbolically — our very identities: a Western gunfighter sees his weapon as a totem, a chef lovingly sharpens his knives, and the suburban guy washes his new car in the driveway for all to see. Though, it seems some of these machines might be standing in for a different organ, and not the soul. Oh, well.
I don’t think what I’ve addressed thus far really engages the supposed theme of the exhibition; many works seem to address the uncanny, the creepy, and the vaguely lifelike, though there are some exceptions. For instance, Theo Jansen’s marvelous giant walking sculptures are somewhat uncanny. Their insect-like movements make them appear to be alive since the artist didn’t rely on wheels or other common mechanical devices. A piece by the Japanese artist Sachiko Kodama creates weird shapes with iron filings suspended in oil, which, when subjected to massive magnetic forces, makes even stranger shapes, seeming to morph organically from one shape to another. These squirming shapes appear disturbingly alive.
A series of video projections resembles tiny deep-sea creatures or protoplasmic life, pulsing, slithering, and interacting with one another. The piece by Ben Rubin and Mark Hansen gathers blog texts, then filters and speaks them aloud, evoking a visible/audible manifestation of the hive mind, and makes me wonder if our the essence of our being is now scattered, dispersed into the ether, across the internet, and living in the computers of a billion people across the world.
Other pieces move away from the uncanny, addressing different ideas. Natalie Jerimenjenko’s hydroponic plants act as lungs to cleanse the air outside the museum. Antonio Mutanda’s map shows human connectivity around the globe, as does the work of some of the other artists.
Julio, the singing robot made in collaboration with David Hanson’s lab, fits in mainly with the creepy uncanny side of the show. Julio is old-school creepy — he resembles a person, uses lifelike motions, and — yikes! — smiles and looks around, mumbles to himself, and then bursts into song. He recalls a Frankenstein monster, although, instead of being outwardly and obviously scary, he’s quasi-friendly looking and bursting with emotion. I hope the sense of realism together with the singing make him doubly creepy. How can a machine be feeling what’s expressed in the songs?
I could have designed Julio to say “human” things like, “Hi there!” or “Wussup?” I thought of having him say more disturbing things like “Touch me,” “I’m so horny,” or “Come closer.” But instead I opted for singing.
Like many animals, humans sing for pleasure, for sex, for attention, to express pain, to relieve angst and to join and participate in a social group. All of these urges seem, if not uniquely human, at least not at all machine like. To see machines mimic these aspects of human life, is to watch some part of our imagined souls being appropriated.
While machines can mimic aspects of human, animal and biological processes, they still lack souls, or whatever it is that leaves us sentient, independent beings. Machines, even computers, are for the most part still modeled on digital, binary and logical thought processes, clutching the legacy of Descartes and the Enlightenment. For machines to truly simulate human beings, they will need to reason with their hearts, their emotions, as we and other animals do. We may like to think that cool logic guides, buffers, and tames our hot emotions, but many now believe that the amygdala and other emotional areas of the brain do most of the “thinking.” It seems that much of our thought process is unconscious, based on impulse, gut feeling, and instinct — and no less wise because of it. This is what’s absent in these machines.
For me, an important part of this show is about this lacuna, this missing part. Witnessing a machine approach being human — and for it to be almost believable, but not quite — can be a creepy and unsettling experience. [See Julio the Uncanny essay]
What would a machine that “feels” be like? I think it would have something like acquired instincts — a lot of them — coupled with a constellation of fears, desires, and loves. The machine would experience desire, hate, friendship, and inexorable ties to other objects and living beings. It would learn the art of deceit, whom to trust, and test the limits of its relationships. It would perform favors and help out, but might expect something in return. What a handful!
In order to think like a human, or like an animal, the machine would need the faults, foibles, and self-interests characteristic of living things. Of course, it would have mirror neurons too — or some equivalent capacity — allowing it to predict what people, animals and other sentient machines were feeling and how they might behave or react next.
We can guess the potential scenarios to follow. We’ve seen them in a hundred sci-fi movies in which the machines take over. In some of these, the machines are even arguably “better” than the people (and certainly better than some of the actors). They make more informed decisions, behave more altruistically, and are marginally less fucked up. In some dramas, the machines adopt the ultimate human foible, or strength, depending on your point of view — religion. They should be showing Battlestar Galactia at this exhibition.
Does this stuff belong in an art museum? Shouldn’t it be in the science museum, some might ask? Is it even science? Some would say that it’s engineering and innovative design, not true science, and certainly not art. I’d argue that their essentially useless content and attention to metaphor in these works suggest that they can only be shown alongside art — thereafter, let the deciders decide.
Participating artists:
Antoni Abad, 1956, ES David Byrne, 1952, UK & David Hanson, US, robot Daniel Canogar, 1964, ES Vuk Cosic, 1966, RS Evru / Zush, 1946, ES Harun Farocki, 1944, CZ, football Paul Friedlander, lights Pierre Huyghe 1962, FR, anime girl Theo Jansen 1948, NL, insect machines Natalie Jeremijenko, 1966, AU, recycling Sachiko Kodama magnet forms Rafael Lozano-Hemmer, 1967, MX Chico MacMurtrie John Maeda, 1966, US Antoni Muntadas, 1942, ES, map Daniel Rozin, 1961 Ben Rubin & Mark Hansen 1964, US, blog feed
About a year ago, I was approached by some Spanish curators to participate in a show scheduled to open at the Museo de Arte Reina Sofia in Madrid at the end of this month. I was told the show would be called “Machines and Souls: Digital Art,” so I suggested I work with David Hanson (of Hanson Robotics) to make a quasi-realistic singing robot. Animatronics date at least as far back as Disney’s Lincoln robot delivering part of the Gettysburg address, although Abraham’s delivery all but ignored any emotional fervor. Having seen some of Hanson’s work at Wired Magazine’s Nextfest—and having heard about it for years before that—I thought it might be time to attempt a collaboration. I immediately thought the robot should perform an action with a weird emotional resonance, like singing. An impassioned speech, laughter, or tears, would have worked just as well, but I had an inkling I could write a short passionate song (in both English and Spanish) for Julio the robot to croon.
Here is a piece I wrote for the museum:
Julio The Uncanny
I had heard of David Hanson and his robot creations—the Einstein head and especially the robot of Phillip K. Dick that responded to questions from the public—but I had never seen one. At an event called Nextfest in New York, I saw a prototype of Jules, a fairly lifelike-looking head of an ordinary young man, slightly androgynous, who could change his facial expressions to simulate emotions. Jules could frown, smile and look slightly skeptical. In addition, Jules could make eye contact—of a sort. Hanson had rigged a system by which the thing could lock on to someone standing in front of it and then, within limits, follow them as they changed position. The robot thus appeared to be looking at you—which in a way it was. There were other robots at this event, but Hanson’s seemed to me to be on the verge of something—something both truly uncanny and something that might cause us to question what is seeing, what are emotions and what is communication and conversation.
Hanson and I chatted briefly and later, when the organizers of “Máquinas y Almas" asked me if I had something that might be appropriate for the exhibit I immediately thought of contacting Hanson to see if he could program Jules to appear to sing if I recorded a vocal for him. By sing I don’t just mean open and close the mouth—that wouldn’t fool anyone. What we call singing is not just the vibrating of the vocal chords and the mouth moving to create the proper syllables and timbres; it’s also tied to a host of emotions that play across the muscles and tissues of the face and neck. The movements of these muscles, the facial expressions, give us clues as to what the singer is feeling, what the singer intends to communicate and what the song means.
Of course, a large percentage of this meaning is in the song itself—the sound and lyrics—but one of the reasons we enjoy a live performance is that we are given (visual) bonus features, clues that give us additional information about the singer, the song and about our immediate situation.
Hanson said such a thing was possible, so I suggested that we make the presentation very simple: just a young man, dressed in a nondescript, ordinary manner, standing in a room, singing to himself. Hanson suggested that this new robot be christened Julio.
Part of the enjoyment of seeing the various robots at Nextfest was experiencing a taste of the uncanny. The idea of the uncanny was proposed by Ernst Jentsch in 1906. He refers to the uncanny as something uncertain or undecidable which therefore makes us uncomfortable. [Freud disagreed—or elaborated on this]. He calls it un-heimlich, the un-home-like. His idea is that our psychological concept of home implies familiarity and comfort, a sense of ease, and, according to him, any concept we hold also implies the existence of its polar opposite—the un-home-like, the unusual, the unknown, the strange.
I love where this is going. It brings to mind an image of someone sitting in a comfortable chair, maybe with friends, and maybe they’re having drinks—and at the same time Jentsch posits that layered over or under this image is the profoundly creepy, the deeply strange and disturbing. We’re in the land of David Lynch and Hitchcock. ET landing in the familiar U.S. suburbs could be viewed this way, or the various living dead and vampire movies.
More recently Japanese roboticist Masahiro Mori proposed the existence of something called the uncanny valley. This “valley” is an area of emotional uncertainty and often revulsion experienced by an observer when a robot or computer animation (for example) approaches being human, is almost believable, but not quite.
He suggests that our emotional empathy with animations and robots increases as they get closer and closer to being human (or animal)—but then, at a certain point, they fall into the valley, and our empathy turns to disgust. In his view they switch from being a cute thing approaching humanity to a bad or faulty version of humanity. It is at this point that we see them as not merely slightly strange, but as a human with serious problems. If the creation can succeed in being a little bit better as a believable creature the feeling of revulsion disappears. For some viewers, recent films like Beowulf fall into this valley, while others find the almost humans acceptable.
Mori further suggests that this reaction might be innate—that it might be linked to our biological reactions to people who are physically or mentally ill—or to corpses. Evolution would have ingrained this reaction as a way of weeding out sick people from the social group. Hanson and others dispute the scientific veracity of the uncanny valley, but I think no one can doubt the strange and weird emotions that well up when confronted by one of these entities.
Knowing that singing elicits an emotional reaction from a listener and observer, I sense that encountering Julio might push some very odd buttons. I remember that my first encounter with Hanson’s robot made me rethink what it means to see, to look. We think of seeing and looking as something optical, something the eyes do. But actually seeing something, and recognizing it, is a lot more than that—it is the act of “naming” the thing the eyes are locking on to. It involves other meta brain functions that often have nothing to do with optics or the muscles controlling the eye. If seeing were just the visual and eye-muscle behavior, then isn’t that the same as what Jules does? And then isn’t singing, and displaying the attendant emotions, the same as what Julio does?
We tend to think that our emotions live inside—in our “hearts” and minds—and that the cues given by facial expressions, for example, are simply more or less involuntary reflections of these “true” inner feelings.
I think it’s more complicated and more confusing than that. I sense that sometimes the effect can produce the cause. Smiling can make you happy. Scowling can make you aggressive and angry. Actors experience this all the time—many of them act by learning to produce the visual effects of an emotion—the posture and facial expressions, the tone of voice—rather than always having to summon up an interior emotion in order to self-generate those affects. (That’s method acting, another whole topic)
So—if an entity displays the correct facial expressions, sounds and gestures, who’s to say it’s not “experiencing” the emotions? I personally still think it’s not, as emotions trigger hormonal release, breathing changes and a whole range of other physiological changes, and they’re not simply limited to what we see. They imply future action or inaction—they imply a set of behaviors guided and steered by the emotions we only see displayed in the face and voice. I love that Julio might make us confront some of these issues—that his emotional singing might give us a discomforting pause.
A video and other images of the robot can be found on the project's webpage, here.
It’s still a work in progress—the movements will be more “natural,” as will his hair. But this definitely demonstrates the creepiness factor at work!
In my opinion, this is just the beginning. When gas tops $10 a gallon THEN perhaps we’ll see the US public demand that our government seek alternative energy sources and demand that everything in our lives be more energy efficient. That day is not so very far away. (And this might actually be a good thing if we consider our carbon footprints). In a recent New York Times review of two books addressing the Middle East, Dexter Filkins points out that in 2003 a barrel of oil was $30 a gallon. It’s now what? $138? At this rate, gas will pass the $10 mark in a couple of years.
For years the economic experts and financial masters of the universe told us that oil prices wouldn’t significantly mess with the economy. Think again. Those guys were so insulated in their high paying jobs with corner offices, that they had no reason or will to see otherwise. Denial is a wonderful thing. Of course, every business that involves transporting people and goods from one place to another will be impacted by the rising price of oil: commuters (naturally), parts for manufacturing, food trucked across the continent, all of those Netflix and Amazon packages (obviously), and even regular mail will be affected.
Paint, computer and television screens, mobile phones, light bulbs, cushions, paper, mattresses, car seats, carpets, steering wheels and polyesters are all made with materials derived from refined oil and natural gas. They’re all going to cost twice as much. If oil was $30 a few years ago and now it’s more than four times as much, well, there will be a lag and some belt tightening, but eventually the costs will be passed on.
This sounds depressing and ominous, but I sense that one’s quality of life is not actually tied to these goods, and is certainly not tied to the automobile and all the roads and services it’s demanded for almost a century. US commuters, for example, will probably blame the Arab world, but will also look for alternate solutions, like decent public transportation, and living closer to their place of work, which is also a good thing. In fact, after some very nasty times we could — if we don’t let our anger and pain get the better of us — emerge with a better quality of life than what we have now.
In the early 1990’s, Giacomo Rizzolatti proposed the existence of mirror neurons. (He recently published Mirrors in the Brain, his book on the topic.) Simply explained, when we see another person performing an action or reacting in a certain way, and we feel like we are experiencing some of what they are feeling, it’s because — at least to some extent — the identical set of neurons fires in our brains.
These are often called “empathy neurons,” since they allow us to sense and share in someone else’s feelings. Not only do we “read” another person’s emotional state — his or her words, tone, gestures, and facial expressions — we actually take on some it. Getting inside your mate or rival’s head for example, is clearly an advantage, helping us to understand, empathize and predict what might happen next in a situation. We feel for others because we become the others. Obviously, the neurons are not perfect mirrors, or we’d all be feeling way too much.
Actors have learned to mimic the signs and indicators of emotional states that we then read and mirror. Part of the pleasure of watching a good performance must be that we experience, in a limited way, everything the actor does. For instance, if someone is fidgeting while confessing to an indiscretion, we (unconsciously) mentally mirror those body movements and emotional signs, and somehow that helps us understand that other person better. We understand them because, to some extent, we put ourselves in their shoes.
This effect extends to muscle movement. When we watch a dancer, or a basketball player, the part of our brain that controls these same muscles begins to fire. Obviously, years spent watching your favorite team doesn’t mean one can perform as well as the players — our brains do not absorb what they spent years training to accomplish — but some part of their experience is imparted, or mirrored, in the viewer. No wonder we like watching shows and sports! We are, in a neurological way, having the same experience as those in front of us.
I suspect this extends to watching movies in which the actors perform impossible feats thanks to CG trickery — our brains simulate the ecstatic experience of leaping over the bad guys and driving like a maniac. This might also explain some of the appeal of porn, though I suspect there are some other, more earthy biological circuits being triggered as well. But at the very least, when men see the endless hydraulic pumping in porn some of their neurons are “pumping” as well.
What about video games? I somehow doubt that our brains mirror the hopping and leaping of Mario, but in the more realistic action games, I suspect the lines get crossed, and the mirror neurons fire as we mow down zombies or shoot rival drug dealers.
As a sometime performer I wonder how much of what I do on stage is mirrored. Scientists claim that physical movement is commonly mirrored, but what about all the other stuff that we enjoy? When I sing, does everyone in the audience “sing” too? Mentally at least, it appears they do. The simulated and recreated feelings in a song performance are not just imparted through some intellectual understanding; instead, the audience members have the same feelings that I have, their vocal neurons firing with mine and yes, we become one hive mind. Sounds creepy, but that might be where the pleasure lies.
I wonder if music, if recordings without the visual component, transmit a portion of that too? Is it possible that the multitude of mental coordinates involved in our reaction to music (it’s way more than simply our ears) triggers some of these empathy neurons upon simply hearing a piece of music (especially vocal music)? Seeing a performance might create the strongest reaction, but perhaps other cultural streams carry some potent triggers as well. Maybe looking at pictures, photographs especially, do a similar thing? Pictures of loved ones are particularly potent: looking at a snapshot of a close friend or family member, I can read their anger, calm, sadness, happiness or deceitfulness. I experience that almost unconsciously, but I am conscious that I think I am “feeling” them — I translate these feelings as my love for them, or my closeness at least. No wonder people feel they “know” celebrities! They have, in some small way, all been Britney, Brad or Angelina.
