Kinetic Synaesthesia: Experiencing Dance in Multimedia Scenography’s
The contrasting kinetic values between dancer and projected moving image in multimedia scenographies provide the viewer with a particular type of synaesthetic experience. It results from the interaction of kinesthesis proper to each medium, the dancing body and the moving image. The extensive use of projected moving images in performing arts is part of a cultural trend that priviledges a fundamental yet little understood aspect of aesthetic experience. Kinetic synaesthesia is a transdisciplinary concept formulated in light of psychological, physiological, and phenomenological accounts of both synaesthesia and kinesthesis.
dance, video projections, moving images, kinesthesis, synaesthesia, kinetic synaesthesia
The use of projected moving images in staged performances has become commonplace in a very short period of time. The spectacular effects it provides are now largely taken for granted, and the more subtle ones are not much discussed either. Projected moving images radically transform the stage. It acquires instant kinetic value by incorporating this newer form of representation within itself.
The projected image can be perceived either as a visual ‘mise en abyme’, a dynamic background (or foreground when a scrim is used), a novel light source or even as an actor, depending on the given artistic means, uses, and contexts. Whether the images are live or prerecorded, whether they have iconic (figurative or abstract) value or not, whether they are used as decor, as part of a narrative, or as representations, and whatever space they occupy on or around the set, what matters most for my purpose is that they are moving images. They must therefore be understood not merely as light projections, or as light images, limitless in form and content, but as dynamic projections.
These projected moving images can provide a wide range of kinesthetic effects depending on the arrangement of successive movement variables (speed, acceleration and direction). Some authors have dealt with such effects as provided by cinema, namely Gene Youngblood who, in his Expanded Cinema (1970), adresses both kinesthesis and synaesthesia.
Essentially, we are dealing with vectors whose values and coordinates vary constantly. But as part of a scenographic totality, fragmented or fusional, the vectors of the moving images are combined with those of the moving performer. All those vectors are effective in terms of kinesthesis, regardless of the “world” (virtual or tangible) they belong to.
The concept of kinetic synaesthesia has evolved out of my attempt to adumbrate the “movement element” from the encounter of dancing bodies and projected moving images. The research is motivated by an interest in movement as a fundamental element of aesthetic perception (as are sounds and colours), intrinsic to dance but also to cinema, animation, kinetic art and performing arts.
Len Lye evoked his discovery of kinetic art in these few words: “All of a sudden it hit me. If there was such a thing as composing music, there could be such a thing as composing motion.” Russian avant-garde painter and composer Mikhail Matyushin conducted experiments at his Visiology Center (Zorved) to demonstrate that expanding visual sensitivity from retinian optical centers would enable the discovery of “new organic substance and rhythm in the apprehension of space.”
The word synaesthesia was coined to refer to a neurological condition where the stimulation of one sensory channel elicits an impression in another, as in “coloured hearing,” for example, which is the most common form of synaesthesia. It is now easy to find both scientific and literary references to the concept of synaesthesia even if its definition remains elusive, since it is a highly subjective sensorial experience.
From a scientific perspective, the causes of synaesthesia are to be found in biogenetics (developmental synaesthesia), in neurology (pathological synaesthesia), or in chemistry (synaesthesia from intoxications). Developmental synesthetes, of which there are few, experience synaesthesia frequently throughout their lives. The research into synaesthesia involves the determination of neuro anatomical variations, since for the synesthete to experience visual sensations when channels other than visual are stimulated implies particular neural connections.
Merleau-Ponty considered synaesthesia to be the priviledged mode of perception, until it conflicts with habits acquired from dominant scientific views pertaining to sensorial mechanisms.
Synaesthetic perception is the rule, and we are unaware of it only because scientific knowledge shifts the centre of gravity of experience, so that we have unlearned how to see, hear, and generally speaking, feel, in order to deduce, from our bodily organization and the world as the physicist conceives it, what we are to see, hear and feel.
About ten years before the publication of Merleau-Ponty’sPhenomenology of Perception, the Austrian-American psychoanalyst Paul Schilder wrote:
“But we should not forget that every sensation is generally synaesthetic. This means that there does not exist any primary isolation between the different senses. The isolation is secondary. We perceive and we may with some difficulty decide that one part of the perception is based upon optic impressions. The synaesthesia, therefore, is the normal situation. The isolated sensation is the product of an analysis. In the scheme of the body tactile-kinaesthetic and optic impulses can only be separated from each other by artificial methods. What we have studied is the change in the unity of the postural model of the body by change in the sensation of the tactile and optic sphere. The nervous system acts as a unit according to the total situation. The unit of perception is the object which presents itself through the senses and through all the senses. Perception is synaesthetic. There is no question that the object ‘body’ presents itself to all senses.”
What different authors mean by synaesthesia depends a lot on the distinction between amodal and cross modal perception. Synaesthesia can be thought of as preceding, bypassing, or underlying sensory differentiation.
In Daniel Stern’s psycho-dynamic model, amodal perception is understood to precede modal perception in the neonate’s development. For Stern, the cross modal relies on the possibility of the amodal. Richard Cytowic presents synaesthesia as the manifestation of a stage in the maturation of normal cognitive process, implying that we are all fundamentally synesthetes. For him, synaesthesia is not a higher cognitive function, since the link between stimulation and synaesthetic sensation is located at a lower level of the neural axis.
3. Synaesthesia and Art
Artists seem more interested in giving their artworks a synaesthetic dimension than expressing personal synaesthetic experiences. Kandinsky, Schoenberg, Eisenstein, Listz and Rimsky-Korsakov were likely synesthetes but not of the biogenetic type as David Hockney is reputed to be, or as Vladimir Nabokov and Olivier Messiaen were. Baudelaire’s transitory bouts of synaesthesia were likely induced by hashish. According to Rimbaud, his famous Sonnet des voyelles — which became an anthem to synaesthesia — was not an expression of a synaesthetic experience, but a fancy of his intellect.
Undoubtedly art is a priviledged domain for synaesthesia, yet combining scientific and artistic approaches to synaesthesia is not unprecedented. Gertrud Stein, Itten’s assistant at the Bauhaus, sought to reconcile romantic conceptions of synaesthesia with contemporary psychological knowledge. She noticed relationships between body movements and sounds and believed that movement was influenced by the perception of sounds and of colours following set rules. Even though synaesthesia is often experienced as some form of correspondance between sight and hearing, body movement can also be involved.
Kandinsky, who conceived a synaesthetic abstract play for the stage, Der gelbe Klang (The Yellow Sound), was likely influenced by Grunow’s ideas. Kandinsky relates having experienced a synaesthesic episode while attending a performance of Wagner’sLohengrin; of having seen ‘in spirit’ all of the colours of the opera before his eyes.
Yet synaesthesia does not necessarely imply a harmonious correspondance between sensations provided by a diversity of sensory channels; Kandinsky, Schoenberg and Scriabin all interested in synaesthetic dissonance. As Chrétien van Campen observes,
“After Schoenberg published the atonal theory of dissonant harmony, Kandinsky wanted to make use of these principles in painting and theatre. In his theatre piece Der gelbe Klang (The Yellow Sound), he experimented with the opposion of three types of movement: visual movement (film), musical movement and physical movement (dance).”
Kinesthesis is generally considered as part of somesthesia, the sensory system of conscious bodily perception associated with skin senses, proprioception, and perception of the internal organs. Kinesthesis allows us to feel the position and movements of our limbs and body, which is an extremely complicated process and about which much remains to be discovered.
The sensation of movement is provided by the interplay of cutaneous and deeper internal receptors. Muscular, tendinous, articulary, and vestibular receptors respond to variations in length and tension of muscles, variations of tension in tendons, changes in joint angles, and head motions. All of these informations contribute to making us aware of the position and movement of the body and its parts. There are at least three different types of kinesthetic neurons located in body joints, and the vestibular system is itself composed of two subsystems.
Of course, perceiving one’s own movements is different from perceiving someone else’s. It can be argued that sympathetic communication through body movement is what makes the appreciation of dance possible. Dance can be thought of as a spatial extension of the body that reaches out and touches other bodies, just as the voice can be thought of as an aural extension. When we view dance or moving images, we can experience kinesthesis without being ourselves in motion by virtue of sympathetic communication, that is to say that kinesthesis is not only a somesthetic sense, but one that allows a form of corporeal communication or psycho-kinesthetic influence. This phenomenon is an important element of eurhythmics, a method of rhythmic education invented by Emile Jaques-Dalcroze, which had considerable bearing on avant-garde dance and theatre (notably on Adolphe Appia, Vaslav Nijinsky, and Mary Wigman).
5. Kinetic Synaesthesia
Sight and touch may very well be interconnected in such a way as to make synaesthesia possible, since tactile movement perception can stimulate the visual cortex of a blind person.Kinetic synaesthesia refers to visual and proprioceptive interconnections, and to how visually perceived movement vectors can be experienced as kinesthesis.
The movement vectors resulting from the combination of dance and moving images in multimedia scenographies are fundamentally different from those of each of these media taken separately. The effect they provide jointly has a synaesthetic component, since it integrates in a perceptual synthesis or gestalt the tangible movements of the dance and the virtual movements of an image. Kinetic synaesthesia differs from kinesthesis in that it derives from dynamic tensions between figure and ground.
The phenomena of the perception of structure, studied by the Danish psychologist Gunnar Johansson, give some insight into kinetic synaesthesia from a perceptual psychological perspective. “Examples of situations in which movement creates a perception of structure […] (b) perceiving movement of objects relative to one another — in which our perception of one object’s movement is influenced by its proximity to other moving objects. Johannson has explained the effects in terms of perceptual vector analysis.”
The analysis of perceptual vectors seems promising for the study of kinetic synaesthesia from a psychophysiological perspective. From a scientific perspective, kinetic synaesthesia is a difficult proposition to ascertain. It can perhaps be dealt with more meaningfully in phenomenological and cognitive terms. What matters most, in order to provide a kinetic synaesthesia effect, is the combination of movement that occupies a large area of the visual field — the projected images — in relation to the moving bodies on which the gaze is directed.
The distinction between figure and ground, both in movement in multimedia dance, is a particularity that has allowed to bring out the concept of kinetic synaesthesia. Central and peripheral visual perceptions are distinct, and their relative values (more or less contrasted in terms of speed, direction, and brightness) create certain tensions within the “gestalt,” tensions that are always shifting. They differ as such from the visual tensions perceived in static compositions, where the shapes are still and dynamism belongs to their perception.
The spectators’ perspective is also an important factor, since kinesthetic effects vary considerably depending on one’s proximity and angle of vision. This is a far less vital factor in a cinema context, where all there is to be seen is on one flat screen, where the movement vectors all come from the same source, and where there is no parallax.
It is difficult to describe multimedia dance because we are dealing with a combination of two artforms that develop concurrently through space and time. It is difficult to speak about dance and cinema individually, and more so of the combined effect of moving bodies and moving images. The fact that we might be dealing with non-narrative and abstract propositions, where there is no plot, progression, or story further complicates our task. The dancing and the images together create a space that is not merely additive but always shifting, as dynamic tensions arise between its diverse and moving elements.
6. Merce Cunningham’s Biped
Merce Cunningham’s Biped (1999) is a piece that combines large scale projected moving images and dance. “The projections consist of hand-drawn figures animated by motion-captureY.Others are more abstract. The images are projected against a huge transparent scrim covering the front of the stage, then reflect off thin white strips at the back. The live dancers perform behind and among the projections.”Contrary to the dance, the projections are not continuous. They appear and disappear throughout the performance, which is very much in the Fluxus style or Fluxus flow of apparently unpredictable and unrelated occurences, where causality, dramatic progression, and other Aristotelean notions are irrelevant.
As is typical of Merce Cunningham’s choreographies, the body is trained and used in a way that evokes ballet (i.e., priviledges verticality, technical virtuosity, a codified movement vocabulary) while the traditional and usual dance composition principles are ignored or deconstructed. Cunningham is interested in juxtaposition of elements through chance. In Biped the space belonging to the dancers and the space created by the projected images are distinct, even if their juxtaposition is crucial to the emergent properties of their combined effect, in other words to kinetic synaesthesia. Dancers, for example, shine through evanescent projected light images, which can somehow be likened to aurora borealis or holographic projections, since the scrim is itself invisible. The respective dynamic values of the dance and of the projected images are clearly discernible, even though the contrast between the images and the dancers isn’t of the usual figure/background type. In Biped, the dancers are immersed in the projections.
Even though we are concerned here with visual art forms, the effects they provide jointly are not reducible to sight, since the visual stimulation triggers a somesthetic experience related to the dynamic vectors at play. In other words, when combined in multimedia presentations, these art forms provide kinesthetic effects of a particular nature, which can be likened to some form of synaesthesia. It is through kinesthetic impressions, as much as visual ones, that the dancing body and projected moving images are interacting.
In what follows, I describe certain brief key moments in Bipedthat provide certain relationships between the dance and the projected images that are favourable to the experience of kinetic synaesthesia. The experience itself is difficult to describe because an adequate vocabulary is yet to be established, and because the elements giving rise to it are in a constant state of flux across different spaces.
A couple of dancers hop forward side by side as circles of blue light, or large dots, appear and move smoothly towards stage right. A larger circle moves slowly upwards near center stage, as two opposing groups of dancers enter from each wing, and move towards each other, pausing after each step. The projections fade away and reappear, they are brighter and visible on the front scrim as well as at the back of the stage, creating a sense of weightless space, while five partnering couples, spread out on the stage, slowly dance “on the spot.” The whole stage seems anchored by these figures, until they suddenly exit without maintaining their unison.
This interaction is due to overlapping or collaged heterogenous spaces and objects (3D and 2D, tangible and visual, weighty and weightless). Whatever the ontological status of these spaces and objects, physical, virtual or phenomenological, they possess dynamic qualities, their actions and interactions being simultaneously perceived. The combination of movement that occupies a large area of the visual field — the projected images, in relation to the moving bodies on which the gaze is directed, implies simultaneous perception of different scales of movement: monumental and human. The contrast of scales is an important factor of kinetic synaesthesia, especially since it already suggests dynamism.
A lone dancer, center stage, dances on the spot, then travels along a circular path back to her starting point, where she dances briefly before exiting. While she dances, groups of intersecting straight blue lines, like virtual sticks, move steadily towards the audience, along a slightly sinuous path. These lines move without stopping and without bending, whereas the dancer moves in complex rhythmic patterns in a series of full body gestures that suddenly explode after a brief pause.
In this very short passage, the spectator’s visual attention is constantly distracted, or scattered in a seemingly haphazard way. The outflowing geometrical line groupings exert a strong kinesthetic sensation, as if one is being pulled into the space, drawn into a tunnel and towards the vanishing point out of which the line patterns emerge. This large scale movement — the lines extend beyond the scrim as they advance — contrasts with the dancers’ proportions and stance, she appears unperturbed by the outflow of lines that move back of her, through her, and past her.
In another section, two dancers go through a series of varied and interlocking postures, almost touching, as narrow projected vertical bands (pillar like) drift stage right. In between those virtual pillars appears a fast moving hand drawn dancing figure. Its movements are highly realistic, thanks to motion capture technology, even though they are conveyed by a few lines only, like taut sinews. This figure swiftly moves through the virtual space, appearing from behind a pillar, disappearing behind another. It seems to move on the exact same surface as the live dancers, grounded on the same stage.
Despite its size, double or quadruple that of the live dancers, this drawn figure communicates human-like kinesthetic sensations in that the illusions of weight and of muscular and joint activity are conveyed. This figure fills a much larger part of the visual field than the dancers do, and it also possesses dynamic chacracteristics particular to monumental images, which dwarf us.
The viewer is thus affected by three concurrent kinaesthetic sources: the moving projected image, the dancing body, and the “perceptual area” between these two (interdimensional space, where 2D and 3D merge dynamically). The perception of each of the two first elements (dancing body and moving image), far from depriving them of their respective autonomy, provides greater complexity. The third element in this threesome of dynamic vectors is the relationship of the dance to the image: kinetic synaesthesia.
The experience of kinetic synaesthesia in a multimedia dance performance is enhanced by virtue of the clarity of the relationships between the movements of the bodies and that of the images, so that movement vector interactions are perceived. When attention is dispersed by complex and competing stimuli, the experience is less likely to be resolved in terms of vectors, which is not a problem with Biped. The interaction of moving bodies and moving images in performing arts can only be fully experienced in live format, video captations flatten the space and squeeze it in a small picture frame. Not only are the finer details lost, but also the effects dependent on the immersive properties of large scale images.
Kinetic synaesthesia focuses the synaesthesia already present in kinesthesis, which is a complex sensory system that involves many receptors whose cross modal connections vary subjectively. Not all individuals, for example, experience visual impressions from tactile stimulation. Kinesthetic sensibility and synaesthesia both refer to sensory impressions of an eminently subjective nature. Describing artistic forms that give rise to their effects is difficult, since they are in a constant state of flux, temporal and spatial.
To be sure, these newer forms of performing arts and multimedia scenographies have developed extensively and quickly because they build on decades of exposure to moving images, in increasingly varied contexts. I hope to have given an indication of the magnitude of the gap between theory and practice in this field. My primary objective has been to draw attention to kinetic synaesthesia as something worthy of further investigation, given that multimedia performing arts create contexts geared towards the stimulation of both kinesthesis and synaesthesia.
 J.-C. Marcadé, “Post-Face: La Victoire sur le soleil ou le merveilleux futuriste comme nouvelle sensibilité” in A. Kroutchonykh and M. Matiouchine, La Victoire sur le Soleil, translated from the Russian by V. and J.-C. Marcadé (Lausanne : Éditions L’Age d’Homme, Collection Théâtre années vingt, 1976), pp. 65-97, ref. on p. 78.
 Maurice Merleau-Ponty, Phenomenology of Perception, Colin Smith, trans. (London: Routledge & Kegan Paul, 1962), p. 229.
 Paul Schilder, The Image and Appearance of the Human Body: Studies in the Constructive Energies of the Psyche (New York: International Universities Press, 1950), pp. 38-39.
 Daniel Stern, The Interpersonnal World of the Infant (New York: Basic Books, 1985), Chap. III “The Sense of an Emergent Self,” pp. 37-68.
 Richard E. Cytowic, “Synaesthesia: phenomenology and neuropsychology,” in Synaesthesia: Classic and Contemporary Readings, pp. 17-42.
 “A dynamic element is equal to a specific element, deduced from the synthetic, the totality, which is of a lower or higher degree, a certain color, and is a form, size, material on the one hand, and a quality, scale and weight on the other. The kinetic form of such a balanced force represents an angle, a right, acute, or obtuse, of a curvilinear type, and, as an entity completed on all sides, yields geometric shapes. They are physical forms of life that enter the consciousness and that in the living being spiritually form themselves into phantasies (beings and objects) according to their emphasis on form or material. And they are also actively felt in that they lead man to actions, creations, and productions. Matter is sensed as entity, as force and energy, and further, as a condition.” Gertrud Grunow, “The Creation of Living Form through Color, Form and Sound” in Hans M. Wingler, The Bauhaus Weimar, Dessau, Berlin, Chicago, trans. Wolfgang Jabs and Basil Gilbert, (Cambridge: MIT Press, 1978), p. 70.
 Chrétien van Campen, “Artistic and Psychological Experiments with Synesthesia,” Leonardo, 32, 1 (1999), pp. 9-14, ref. on p. 10.
 “Humans appear to have a special ability for recognizing human motion, which has been hypothesized to rely on the recruitment of brain structures associated with the control of movement. Experiments in the early 1970s showed how little information is needed for the human brain to recognize human motion […] It has also been demonstrated that, when shown images of two body positions, people will connect the two positions with a virtual movement describing an anatomically feasible path. This suggests that knowledge as to what constitutes a feasible movement, which one would expect to reside in the motor system, somehow enters the perceptual system. In addition it has been shown that watching a photo of an object or human in motion activates motor areas in the brain. Based on these and other findings I have argued that watching dance submerges the brain in motor imagery, the motor correlate of visual imagery. Motor imagery is the kinesthetic feeling of movement without actually moving.” Ivar Hagendoorn, Dance, Perception, Aesthetic Experience & the Brain, 2003 webstite version,www.ivarhagendoorn.com/research/perception.html
 Annselm Vinje Morpurgo, “Toward a Scientific Explication of Synesthetic Phenomena Through Kinesthetic Stimulation,”Congress on Research in Dance, vol. X (1979) (Dance Research Collage), pp. 31-37. Here are excerpts from a few papers dealing with sense cross modality:
“The Visual Ego” :
“The covariation between movements of parts of the body, both active and passive, and changes of occlusion in the optic array constitutes one kind of “visual kinesthesis”. (What gives rise to the perception of motion? Psychol. Rev.). All such changes in the optic array are seen as movements of the body or its parts not as motions of objects. Similarly, locomotion is seen as such, and the transformations in the optic array or the “apparent motions ” in the visual field that samples it are not perceived as motions in the environment. The rule seems to be that any optical change that is always a concomitant of a bodily movement will eventually be seen as having a kinesthetic quality and as pertaining to the self instead of to the world, even if at first the optical change is taken to be the motion of an object or the behavior of another animal.” From Purple Peril of J. J. Gibson (unpublished manuscripts, not intended for publication) J. J. Gibson, Cornell University; January 1970.www.huwi.org/gibson/ego.php
“Tactile-Visual Links in Exogenous Spatial Attention under Different Postures: Convergent Evidence from Psychophysics and ERPs” :
“. . . results demonstrate that crossmodal links in spatial attention can influence sensory brain responses as early as the N1, and that these links operate in a spatial frame-of-reference that can remap between the modalities across changes in posture.” Steffan Kennett, Martin Eimer, Charles Spence and Jon Driver. Journal of Cognitive Neuroscience Vol. 13 No. 4, May 2001, pp. 462 – 478.
“Visual Enhancing of Tactile Perception in the Posterior Parietal Cortex” :
“. . . results provide evidence for visual-to-tactile perceptual modulation and demonstrate effects of illusory vision of touch on touch perception through a long-lasting modulatory process in the posterior parietal cortex.” Tony Ro, Ruth Wallace, Judith Hagedorn, Alessandro Farnà and Elizabeth Pienkos. Journal of Cognitive Neuroscience Vol. 16, No. 1, January-February 2004, pp. 24 – 30.
“Hearing Cheats Touch, but Less in Congenitally Blind Than in Sighted Individuals” :
“. . . the present data are in accordance with a modality-appropriateness account suggesting that interference by a task-irrelevant modality is reduced if processing accuracy of the task-relevant modality is high.” Kirsten Hötting and Brigitte Röder.Psychological Science Vol. 15, No. 1, January 2004, p. 60.
 “Configurations in Event Perception” (1950), “Visual Motion Perception” (1975), “Visual Space Perception through Motion” (1982), “About Event Perception” (1985).
 Bruce E. Goldstein, Sensation and Perception (Belmont, California: Wadsworth, 1989), p. 314.
A documentary titled, Merce Cunningham: A Lifetime of Dance(2000), contains several sequences of Biped. I also recommend visiting:
 “Cunningham is known for subverting the foreground and background of the stage space in favor of a more complex and democratic visual field. In Biped, his dancers interact within a visual space enveloped by virtual 3-D ‘dancers’ performing choreographic sequences that, in true Cunningham fashion, are ordered by chance operations — in other words, via a toss of the dice. In this manner, Cunningham eliminates character, dramatic through-line, intention and psychology in favor of pure movement. According to Kaiser, a serendipitous moment occurred when the projections were united with the live dancers and music for the first time at Biped‘s premiere in Berkeley last year. In an essay about the collaboration, he writes, ‘By a miracle of chance operations, one of the first dancers on stage (Jeannie Steele) was haloed in a projection of her own motion-capture — “as if I were dancing inside myself,” she said afterwards.'” Sara Wolf, L.A. Weekly, April 14-20, 2000 (www.laweekly.com/ink/00/21/dance-wolf.php).