Untitled Document

Multi-sensory Interactive Environments for Art

by Janice Whaley, March 2006

‘Dark Matter’ sculpture
by Paul Friedlander

Over the years, I have become increasingly interested in multi-sensory art, and more specifically, the sound of muscle movement due to my experience with a muscle disease called Polymiositis which I was diagnosed with as a teenager. In the initial testing for determining the cause of my illness, I was tested for various diseases based on the symptoms. Electromyography (EMG), a test that produces an audible and visual record of the muscle being tested, was among the first tests performed. Although I was unaware of how this would change my perception at the time, my experience of EMG testing has proved to have a lasting effect

on how I related to music in a physical sense as well as aural. At the time, I could only relate to the test as a painful and traumatic incident, but years later I realize my experience of hearing and seeing my muscles move simultaneously resulted in life-changing effects on me personally and artistically.

The act of simultaneously hearing and feeling my muscle movement was profound for me because it was multi-sensory in a way that I had never experienced before. I felt the pain of the needles, I felt my muscles moving, I saw it on the display. My muscles were affecting things I never thought possible. In this moment, I became aware of how alive and literally electrical my body is. Since this experience, I have often listened to the sound of muscles by pressing my ear to a muscle such as my shoulder and moving my arm in various ways to hear the sounds produced. Today, multi-sensory technology exists that would allow me as an artist, not a medical professional, to incorporate the sound of physical movement in my art.

In the medical world, an Electromyograph test consists of needles connected to wires, electrodes for signal generating and processing, and/or highly responsive, flat microphones taped to the skin. An electronic signal is sent out through an electrode into the muscle. When a needle is inserted into muscle tissue, it picks up the signal, performs a mathematic response to the signal and sends the calculated electrical impulses into a machine. This machine interprets the calculation and displays the frequency and/or spectral analysis of the muscle with visual sound waves on a screen and audible sound through speakers. Because these microphones are so sensitive, the machine uses band-pass filters to target a specific range of the

audio spectrum. EMG technicians are trained to interpret the meaning of various muscles sounds, in the same way a musician learns ear training techniques of recognizing pitch and interval relationships between notes. In a static state, the muscles produce no sound or slight “clicks and pops,” but by moving the needle around inside the muscle tissue, or contracting the muscle, it responds with a muffled sound of thunder which changes to a louder, more intense, groaning sound that becomes higher in pitch as the tissue is distressed or damaged. The EMG technician evaluates what the muscles are “saying” in response to his needle movements. To put it simply, sick muscles sound different than healthy muscles. Although we are unaware, there is a world of sound happening inside of us every moment of every day. EMG Technician works in a multi-sensory environment that remains silent and invisible to most of us. Humans understand life based on what the five senses tell the brain- but what if we could experience something new with our senses? What could we feel, understand, and communicate if we could experience synesthesia - the rewiring of the five senses. What could we experience with less than five senses?

Integrating all senses into art and making it interactive through electronics is a relatively new technological field of study, but the concept of full sensory interaction is something that has intrigued people for as long as we have had five senses. There is a strong attraction to the idea of being able to understand something with more than just one sense at a time. Rulers and leaders throughout time have desired art, food, clothes, perfumes, architecture, and music to please the senses. Humans, especially now more than ever, want the ability to be part of the art and interact in a way that creates a deeper personal meaning in this fast-paced, technological world. Early Western artistic examples of this can be found in modern art Happenings of the 1950-60s that created interactive installations as a “stage” where the participants could become part of a live art experience. Participatory awareness performance art such as The Performance Group, who removed the stage and performed in the audience, and Living Theater who wanted the audience to participate in the performances and get politically involved with their specific agendas.

Electronic multi-sensory virtual reality simulators of the past 20 years have advanced at a rapid pace. Video game developers are making increasingly more realistic and complex worlds for electronic interactions to take place. Video game controllers are set up to vibrate along with certain actions as a way of helping gamers “feel” the game in a more realistic way. Virtual reality gear seeks to give us

multisensory imaginary worlds to experience without living the comfort of our own homes. Some virtual reality headgear is being advertised to dentists as a way of making their patients more comfortable. The word “smell-o-vision” comes up often on cooking television programs as a half-hearted joke, but in fact, two movies have made their attempt made to simulate the multi-sensory audio experience. Mike Todd, Jr. invested all his money in Smell-o-Vision and made one movie called Sent of a Mystery. Perhaps it was ahead of it’s time or maybe people have aversions to certain smells, but the concept did not catch on. Years later, the technique inspired John Waters used scratch and sniff cards in the film Polyester, dubbing the technique “oderama.”

Flight simulators are now used as a safe and effective way of training pilots how to prepare for dangerous flying situations that would normally not be possible to train for. Most of us can relate to this from Disneyland’s flight simulated ride Star Tours which moves the simulator to match what is projected onto the screen. All of these multi-sensory interactions allow us to understand a particular concept in a more tangible and personal way. These interactions are also attempting to transport us to alternate realities so that we can experience things that would otherwise not be possible.

Besides its artistic or entertainment value, multi-sensory technology has a lesser known albeit important value

to the general public. The forerunners in sound movement technology are companies who develop movement and sound sensors for sound therapy. Many schools and medical research centers are using this technology as educational, assessment, and rehabilitation tools. There are many companies who develop gear for use with

small children and also for anyone with learning and/or physical disabilities. The ability to create changes in sound as well as lighting, without needing to use specific muscle coordination or intellectual intent opens up a new world of non-judgmental expression and basic cause and effect skills, as well as promotes movement to strengthen muscles and coordination. The creators and organizations who champion this new technology understand that learning is not limited to the scholastic method of reading books or memorizing terms. Effective learning takes place through playing and stimulating the senses in ways that are memorable and meaningful. It is empowering to give a person input, options, and control over sensory decisions. Learning incorporates any of the five senses and is not limited to the standardized methods, timeframes, or rules of society.

There are many forms of non-invasive surface sensors that work similarly to needles in Electromyograms to detect electrical muscle impulses. The signal is weaker and more difficult to read accurately because the sensors are on the skin and not inside the muscle fiber, however they work well in as motion sensors. Sound Beam is the

leading producer of movement-responsive sensor devices based on electrical impulses. Their webpage describes the Sound Beam as “The Invisible Expanding Keyboard in Space.” The company has both stand-alone and computer intergraded models for use in various settings such as physical therapy, public schools, and artistic settings. The flat sensors are embedded in a cloth and Velcro band which can be strapped to a body part or static object. The senor reader can be placed at a distance of up to 50 meters away. The sensors can be used to trigger a sampler to play a sound wave such as music, words, or any other sounds. The sensors can also be used to trigger non-aural things such as lighting colors and intensity, vibrations, or smells.

Max/MSP Object-based programming language:

Another technology which can incorporate movement-responsive sensors is the object based programming language Max/MSP by Cycling74. This software allows the programmer to build any kind of software to read and respond to midi sensors. These sensors, rather than reading electrical impulses of muscles, detect physical vibrations, physical movement or location in space (x & y grid). With this software it is possible to build your own SoundBeam interface and have even more control over the way sound is generated. The drawback to Max/MSP is the amount of time it takes to learn the programming language, however, as this technology becomes more widely accepted, free programs, called “patches” are turning up on the internet. Some midi-sensor products, such as I-CubeX, include Max/MSP patches for their products which you can download and modify to meet your needs.

Multi-sensory art can bridge gaps in the five senses or redirect artistic concepts to entirely different senses. By doing so, art is not limited to only those who can see visuals, hear sound, play instruments, paint, move to dance steps, etc. With this technology, art can be made by people on their own terms and experienced in ways that are meaningful to them regardless of society’s assumptions about how art is created and experienced. At its heart, multi-sensory technology for artistic purposes is about inclusiveness and finding new ways to communicate and learn about ourselves and each other.