MULTIMODALITY

Research Paper & SOCIETY

Multimodality & Society
2021, Vol. 1(1) 97–118
The semiotics of ª The Author(s) 2021
Article reuse guidelines:
sagepub.com/journals-permissions
movement and mobility DOI: 10.1177/2634979521992733
journals.sagepub.com/home/mas

Theo van Leeuwen

University of Southern Denmark, Denmark

Abstract
The paper presents a framework for the distinctive feature analysis of movement and
mobility in texts, performances and semiotic artefacts, showing its applicability to the
analysis of meaning-making in dance, music, animated and live action film and video, and
product design. Emphasis is placed on the role of movement and mobility in identity
design. Identity design is realized by the style in which movements are performed and can
be analysed in terms of the gradable distinctive features present in any movement –
direction, expansiveness, velocity, force, angularity, fluidity, directedness and regularity.
The paper includes a historical dimension, focusing on the development of movement
and mobility as semiotic resources, and argues for the pioneering role of modernist
artists in this development.

Keywords
Distinctive features, identity design, animation, kinetic sculpture, mobility design, move-
ment, style

Introduction
From the beginning of the 20th century, movement has been actively developed as a
semiotic mode in its own right. It first manifested itself in the work of avantgarde artists,
but was soon picked up, first in the mass media, and, today, in video and animation
software, available, with different degrees of sophistication, to any computer user.
As early as 1912, the Futurist sculptor and writer Umberto Boccioni described
machine movement as a potential new form of artistic expression (quoted in Kepes,
1965: 810):

Corresponding author:
Theo van Leeuwen, University of Southern Denmark, Denmark.
Email: leeuwen@sdu.dk
98 Multimodality & Society 1(1)

We cannot forget that the tick-tock and the moving hands of the clock, the in-and-out of a
piston in a cylinder, the opening and closing of two cogwheels with the continual appear-
ance and disappearance of their square cogs, the fury of a flywheel or the turbine of a
propeller, are all plastic and pictorial elements of which a Futurist work in sculpture should
take account.

In 1920, Naum Gabo and Antoine Pevsner, in their Realist Manifesto, introduced the
term ‘kinetic art’ (Gabo & Pevsner, quoted in Kepes, 1965: 81)

We renounce the thousand-year-old delusion in art that held that static rhythms are the only
elements of the plastic and pictorial arts. We affirm in these arts a new element, the kinetic
rhythms, as the basic form of our perception of real time.

In the same period, Duchamp mounted a bicycle wheel on a stool and called it Mobile,
Meyerhold developed his biomechanics, a method of acting based on movement and
contrasting with Stanislavski’s focus on ‘inner’ inspiration, and avantgarde artists every-
where embraced the moving image, playing a key role in developing its semiotic
potential.
Entirely new this was not, of course. For several hundreds of years, people had been
fascinated with movement, using clockworks to create walking dolls, animated chess
players, dolls playing a miniature piano or writing a few words, and more, and devel-
oping the forerunners of the cinema, from flip charts to the more complex ‘kineto-
scopes’, ‘praxinoscopes’, etc. that were popular throughout the 19th century, as toys
and as tools for studies of movement by Muybridge and others. But it was only in the
20th century, the age of the motor car and the movies, that movement integrated, not only
with art, but also with the design of everyday objects, and even of texts, where the screen,
the medium of the moving image, began to merge with the page, the medium of the static
word. Historian of typography Beatrice Warde recounted her amazement at seeing an
animation by Norman McLaren projected onto the gigantic Animated Electric Screen in
Times Square, New York (quoted in Bellantoni and Woolman, 2000: 5): ‘After forty
centuries of the necessarily static alphabet, I saw what its members could do in the fourth
dimension of time, “flux,” movement’. Today, ‘kinetic typography’, first developed in
animation films by Len Lye and Norman McLaren and now accessible to every computer
user, is hailed as a new form of writing, able to convey a speaker’s tone of voice as well
as “qualities of character, and affective (emotional) qualities of texts” (Forlizzi et al.,
2003: 377).
Lewis Mumford (1934) has described the long incubation periods of inventions that
eventually become part of everyday life as periods of ‘cultural preparation’ in which they
exist, seemingly marginal, as artistic movements, philosophical ideas, popular play-
things, or tools for relatively marginal specialist endeavours. But he also noted that such
‘cultural preparations’, and the crucial inventions to which they eventually lead, are
driven by broader social and cultural quests, in this case the pursuit of dynamization,
of ever greater mobility and ever greater speed, which continues to be a major source of
further social and cultural change.
van Leeuwen 99

Tracing the historical development of semiotic resources in this way is an important

part of establishing their cultural context, and hence of moving beyond description
towards understanding how and why they have developed the way they have.
In this paper I discuss two aspects of movement design, starting with the dynamic
representation of movement in dance, music, film, video, and computer animation, and
then moving to mobility, that is, to the design principles underpinning the resources for
creating meaning with movement which, today, are built into many everyday objects and
even into ubiquitous text-production software such as PowerPoint. Below I briefly
introduce each.

The meaning of movement

The approach to the semiotics of movement I introduce here applies to live action
(recorded or otherwise) as well as to animated films and videos. The main differences
between live action and animation films and videos are, firstly, that in animation move-
ments are not recorded but created by artists or software designers, secondly, that
animation can make inert objects come to life, and thirdly, that animation not only
represents actions and events, but also dynamizes identities and attributes, making
objects or letter forms change colour or shape in front of our eyes, or, for instance,
morphing letter forms into images or images into letter forms. Live film and video can do
this only to a limited extent, for instance through time lapse photography.
My approach is based on the principles of provenance and experiential meaning
potential, which I have discussed in detail elsewhere (e.g. Kress and Van Leeuwen,
2001). In the case of provenance, individual signifiers are associated with ‘where they
come from’, and meaning is made on this basis, that is, by reference to ideas and values
associated with that context. Needless to say, such associations can differ according to
context, although an increasingly globalized popular culture will make many of them
widely understood, creating a kind of lexicon of semiotic clichés. In the case of
experiential meaning potential, meaning potential derives from the material qualities
of signifiers, and from our experience of, and with, these material qualities. Following
Jakobson and Halle’s (1956) account of the material qualities of speech, I refer to these
material qualities as distinctive features. To explain their meaning-making potential
with an example, Johannessen and Van Leeuwen (2018) have described the wide use of
irregularity in typography in this way. From our experience of handwriting, they argue,
we know what can make handwriting irregular – lack of skill, as in the writing of young
children, inadequate tools and materials, unwillingness or lack of interest in producing
neat handwriting, mental and physical states such as intoxication or physical handicaps
affecting the ability to move the hand, and so on. This creates a wide meaning poten-
tial. It can express the playfulness of a young child, the rebelliousness of a punk
fanzine, the authenticity of handwriting in advertisements for mass-produced luxury
goods, and so on.
When represented movements are naturalistic, whether in live action or animation, we
readily recognize that they represent, for instance, walking or running or swimming. But
styles of human movements (and, in animation, also the movement of things) can also
convey meanings, and these usually are identity meanings – style is essentially identity
100 Multimodality & Society 1(1)

design. There are many styles of walking, for instance – strolling, striding, strutting,
lumbering, shuffling, to name just a few. Some of these have recognizable provenances,
military marching steps for instance, or the characteristic walks of star performers, such
as Mick Jagger’s strut or Michael Jackson’s ‘moonwalk’, others do not. But even move-
ments with specific provenances can also be understood on the basis of embodied
experience. Our body knows how much energy they would require and with what
situations, activities or moods they can be associated. This is so even in the case of the
movement of objects we have never seen. As I write this paper, the corona virus pan-
demic still dominates the news, and the cyclorama style backgrounds of many TV news
studios show large corona viruses, sometimes still, sometimes moving. In the news of the
Dutch national broadcaster NOS, large corona viruses float in a blue space, seemingly
weightless and slowly moving, mostly towards the right and slightly upwards. Direction
of movement is here a key source of meaning – the corona viruses slowly and steadily
progress, and the trend is upwards. In the news of the Australian ABC, the viruses are
large and pink and slowly and menacingly hover in place, while little bubbles float
upwards. Here we are kept in suspense as to what the viruses will do, whether they will
move, and if so, where. In each case, different qualities, different ‘identity traits’ of the
viruses are foregrounded.
Scholars of dance discuss the relation between everyday movements and the see-
mingly abstract movements of dance. Sheets-Johnstone (2013), for instance, sees move-
ment in everyday life as symptomatically expressive of feeling, and movement in dance
as symbolically expressive. ‘Linear design’ and ‘linear pattern’ are then among the
qualities of movement that carry such symbolic meanings – the term ‘qualities’ corre-
sponds to what, below, I will refer to as ‘distinctive features’ (following e.g. Kress and
Van Leeuwen, 2002; Van Leeuwen, 2006). As Han explains (2021), in a dissertation
about the relation between movement and music: ‘Linear design refers to the line created
by the body as a whole or by any individual body parts alone or in combination, e.g.
twisted, angular, diagonal, vertical. Linear pattern refers to the linear paths created by the
body as the result of the direction of movement, diagonal, zig-zag, circular’. There are
parallels to be explored with the movement of melodies in music, where ‘direction of
movement’ is also a key feature. Tagg (2000: 5) describes and exemplifies 10 common
‘contours’ – rising, falling, ‘tumbling strain’, V-shaped, centric, terraced (falling), ter-
raced (rising), oscillatory, arched and wavy. All of these could easily be translated into
body movements or the movements of animated objects. Tagg also notes that some
melodic contours make meaning on the basis of provenance, for instance the semiquaver
triplets of Irish traditional music, or certain ‘onbeat figures’ in ‘popular notions of
Hispanic music’. He describes these formal features, but, as the corona virus example
shows, and as we will see in more detail below, they all carry meaning potentials.
To give a further example, in the well-known Intel ‘vector logo’, a likeness of which
is reproduced in Figure 1, a beam of flashing white light makes a swooshing orbit and
then disappears into the distance of a dark blue background which dissolves into the logo
itself, that is, the word ‘Intel’, surrounded by the light beam, now in blue against a white
background, and morphed into two static curves which increase in size and thickness
towards the front. At the point where they almost meet, the point where, so to speak, a
spark might jump across them, the words ‘leap ahead’ appear, letter by letter. Meanwhile
van Leeuwen 101

Figure 1. A likeness of the Intel logo (illustration by Toby van Leeuwen).

the soundtrack accompanies the initial light flash with an ‘audio sparkle’ which first
becomes louder, then softer again, as if describing an orbit, and, as the words ‘leap
ahead’ appear, a four-note melody can be heard, spelling ‘Intel Inside’. The logo thus
creates a dynamic, energetic identity, entering with a burst of energy, then ‘leaping
ahead’. The short melody blends electronic sounds with a xylophone, a marimba and
a ‘secret recipe of instruments’, so combining the sounds of technological perfection
with sounds that seek to evoke affective resonance, in order to create ‘a logo with
emotional content, in the same way a film score works’ (Jackson, 2003: 128).

The meaning of mobility

The meaning potential of mobility derives from resources for meaning making that are
built into the design of objects, whether these are everyday objects or works of art, and
whether they are physical objects or computer software. In earlier work (Van Leeuwen
and Caldas-Coulthard, 2004), Caldas-Coulthard and I compared a Sindy doll and an
Action Man in the same price class and found some striking differences, not only in their
use of plastic, but also in their kinetic design. Action Man’s head could only move
sideways, and not up and down. Sindy’s head, on the other hand, could move freely
in every direction, allowing ‘head cants’, the coy poses with the head cocked sideways
which Goffman had commented on in his study of gendered poses in advertisements
(1976: 47) and which, more than 40 years later, can still be seen in many fashion
photographs. Again, Action Man’s legs could spread, allowing him to sit the way many
men sit, with the legs spread wide, but Sindy’s legs could not. She could however bend
her knees (in fact she had three degrees of knee bend) so that she could adopt the ‘body
cants’ and ‘bashful knee bends’ also described by Goffman (1976: 45–46). This Action
Man could not do. In addition, Action Man could be made to stand on his feet while
Sindy could not, and Action Man’s hands could hold objects while Sindy’s could not. In
short, kinetic design expressed the gender identities Goffman had so ably analysed:
in contrast to men, women are kinetically represented as literally and figuratively unable
to support themselves, which can then ‘be read as an acceptance of subordination, and an
expression of ingratiation, submissiveness and appeasement’ (1976: 46). Men, mean-
while, are represented as oriented towards action, seeking to occupy the maximum
amount of space, and standing erect, in no need of support. These meanings are never
102 Multimodality & Society 1(1)

Figure 2. Tizio light (left) and Anglepoise light (right) (illustrations by Toby van Leeuwen).

made explicit but communicated as children (or adults) actively explore the movements
which the dolls’ kinetic design makes possible.
The kinetic design of everyday objects also involves identity design. The Tizio light
designed by Richard Sappor (see Figure 2) can move only vertically, only tilt up and
down. It is designed to illuminate a single, small object, the book read in bed for instance,
and to assist single-minded concentration. The now ubiquitous Anglepoise light (see
Figure 2) designed by George Cawardine in 1932 is still a desk light – ‘Focus brings
learnings, experience and knowledge’, the ‘Anglepoise manifesto’ declares (Anglepoise,
2020). But its spring-loaded arm and its shade can move in any direction, so bringing
different objects and activities in focus, and directing the shade upwards can cause light
to be reflected from the ceiling so as to illuminate a larger area.
Although the design of these lights is functionally motivated, they may not be used for
the functions for which they were designed. Functional design features may come to
signify identity traits such as ‘single-mindedness of concentration’ or ‘multiplicity and
versatility’. The same applies to the use of spotlights on rails fixed to the ceiling in
domestic settings. In film and video studios such lighting is functional, as lighting set ups
must be changed frequently. In home settings their kinetic capacity may be underused or
not used at all. It then becomes symbolic, either on the basis of provenance, signifying
some kind of allegiance to, or alignment with, the world of film and television, and/or on
the basis of experiential metaphor, signifying a taste for change, flexibility, versatility. In
short, while kinetic design is usually functionally motivated, whether for the purpose of
recognizably representing specific movements or effectively facilitating specific activ-
ities, it also leaves room for style, and hence for identity design.
van Leeuwen 103

Figure 3. A still from Help stop the spread. Reproduced with kind permission from the Depart-
ment of Health, Commonwealth of Australia.

The distinctive features of movement

In this section I discuss the representation of movement in multimodal texts of various
kinds, drawing first on pioneering work by Gisela Leão (2012), who studied movement
as representation and then on equally pioneering work by Joshua Han (2021), who
studies movement as style. Movements are of course always carried by the elements
that move, whether these are images, words, or abstract shapes (or parts of images, words
or abstract shapes). They must be the movements of something, or someone. To
exemplify the distinctive features of movement I will use a short video issued by the
Australian Government, (https://www.health.gov.au/resources/coronavirus-covid-19-
video-help-stop-the-spread), and transcribed below – in this video corona viruses are
the primary moving elements. Figure 3 gives an impression of the style of the video.
Note that the transitions between the shots are not cuts or dissolves – people and things
appear and disappear as if being drawn or erased in front of our eyes by an invisible hand.

1. CLOSE SHOT of boy coughing. Pink viruses escape from his mouth, growing, and
coalescing into a large cloud which eventually fills the whole screen.

VOICE OVER
Viruses spread when you cough and sneeze.
The cloud moves up, releasing small pink droplets which fall on a table. A hand
moves into the shot and touches the table, which now is covered in pink dots.
104 Multimodality & Society 1(1)

VOICE OVER
And the tiny droplets land on surfaces others may touch.
As the hand moves left, out of shot, the drawing of the table is erased.

2. LONG SHOT. A drawing appears of a group of five people, one of them holding
hands with a young child. Pink viruses fly in and hover around the people.

VOICE OVER
You can help reduce . . .
The viruses turn into white crystals as the drawing is erased.

3. MEDIUM CLOSE SHOT. The drawing of a boy appears. He coughs up purple

viruses, this time in his arm.

VOICE OVER
. . . the risk by coughing or sneezing into your arm . . .

The drawing is erased

4. MEDIUM CLOSE SHOT. The drawing of a girl appears. She coughs into a white
tissue which then colours pink.

VOICE OVER
. . . or a tissue.

The drawing of the girl is erased

5. FULL SHOT of the drawing of a bin, with purple tissue falling in, dropped by an
invisible hand.

VOICE OVER
Bin the tissue.

The drawing of the bin is erased.

6. FULL SHOT. The drawing of a tap appears. Camera tilts down with the water that
comes out of the tap, to end up in shot of hands washing, with pink viruses
disappearing as a result.
van Leeuwen 105

VOICE OVER
Wash your hands with soap and water.

7. MEDIUM CLOSE SHOT. The drawing of a boy with a thermometer in his mouth
appears.

VOICE OVER
And if . . .

8. FULL SHOT thermometer. Camera zooms in. A dialogue balloon appears at end of
thermometer (‘37.5þ’)

VOICE OVER
. . . you’re sick . . .

9. CLOSE SHOT. A hand holding a mobile phone, moves up, then down again.

VOICE OVER
. . . seek medical advice. Together . . .

10. LONG SHOT (AS SHOT 2) The drawing of group of five people appears, viruses
and white spots hovering around them.

VOICE OVER
. . . we can help stop the spread . . .
The drawing is erased

11. CLOSE SHOT. The drawing of six hands making ‘thumbs up’ gesture appears. As
the drawing appears, the viruses from the previous shot change into pink ‘plus’
signs, the dots into smaller, white ‘plus’ signs

VOICE OVER
. . . and stay healthy.

12. TITLE SHOT reading ‘Visit Health Gov Au’, with the Australian Government’s
coat of arms in the centre and some pink viruses at the bottom of the shot. A hand
moves in, wiping out the viruses with a cloth.
106 Multimodality & Society 1(1)

To represent people or things (including abstract things and letterforms) as engaged in

action, they must be displaced, change their position on the page or screen, as in the
Dutch news programme where corona viruses slowly move to the right and upwards.
Such movements can also be ‘in place’ (He and Van Leeuwen, 2019), as in the Aus-
tralian news programme, where the viruses mostly hover in place. Such ‘in place’
movements are increasingly used in the minimal, looped animations of still images and
words that can now be so easily produced with various softwares.
As detailed in Leão (2012), actions can be represented as transactional, in which case
they are shown to somehow affect another element, or as non-transactional, in which
case they do not affect another element. In shot 1 of the above transcript, the droplets
affect the table (they ‘infect’ the table, or rather, they make it into a source of infection)
and the hand of the boy affects the table by touching it. The movements in shot 9 are non-
transactional. Here the boy’s gesture is not directed at anyone or anything. The fact that
he is contacting a doctor is only conveyed verbally, by the word ‘Doctor’ on the mobile
phone and the words ‘medical advice’ in the voice over.
However, the distinctions between displacement and movement-in-place, and
between transactional and non-transactional, are essentially functional distinctions, sys-
temic distinctions which fulfil more or less the same representational functions as the
system of transitivity in language and the systems of narrative and conceptual visual
representation described by Kress and Van Leeuwen (2006). Identity design, on the other
hand, relies on the how of these basic movements, on their style, and hence on the
gradable distinctive features necessarily present in any movement. Drawing specifically
on the work of Han (2021), the most important of these are direction, expansiveness,
velocity, force, angularity, fluidity, directedness and regularity.

Direction
Directionality involves the horizontal, left-right dimension as well as the vertical, up-
down dimension. Our experience tells us that moving upwards involves effort and
energy, while moving downwards allows a decrease in effort, relaxation. This is again
a matter of degree, with, in between the two extremes, horizontality, flatness. In context
this meaning potential is activated in specific ways. In the Dutch news programme, for
instance, many of the corona viruses move up, and in shot 1 of the transcript they also
rise, becoming a menacing cloud. In both cases the pandemic is, literally and figura-
tively, ‘on the rise’. The movement in shot 10 is also upwards, showing the boy as
actively taking control of his health. As Lakoff and Johnson (1982) have shown, ‘up’
and ‘down’ are key sources of metaphor, and the same applies to their dynamic counter-
parts. ‘Rising’ can be ‘aspiring’, ‘making one’s way up’, ‘climbing the ladder’, for
instance; ‘going down’ can be ‘lowering oneself’ or ‘sinking’ – and these are only some
of the many possibilities.
Horizontal movements on pages or screens relate to the direction of writing. In
spoken language, sentences (and longer stretches of speech) begin with the ‘Given’,
with something that has already been mentioned or is assumed to be known by the
listener and then move to the ‘New’, the information the speaker wants to impart (Halli-
day, 1985: 277). When we move from speech to writing, ‘before’ and ‘after’ become
van Leeuwen 107

‘left’ and ‘right’, at least in Western writing. This is so also in musical phrases and in the
performance units of acting (Van Leeuwen, 2005: 202). Moving from left to right
therefore suggests some form of goal-directedness or future-orientation, moving from
right to left its opposite.

Expansiveness
The same kinds of action can extend over a larger or smaller amount of space. We can
walk with large strides or measured steps, jump with energetic leaps or skip from one
foot to the other, wave our arms around or restrain our gestures. And such degrees of
expansiveness can be associated with different kinds of activity and different moods
and emotions. A comparison with music can again be made. As described by Cooke
(1959: 109)

Medieval and early Renaissance music tend to move in stepwise progression at a normal
medium pitch, befitting man’s humble subjection to the deity, but with the growth of human
self-realization, music drama [ . . . ] began to introduce more and more liberty of pitch
movement to express the rhetoric of human passion

In this example, expansiveness characterizes the style of an era, but expansiveness

can also characterize the movement style of an individual, a social group, or a nation, and
both expansiveness and constraint can have positive as well as negative overtones.
Exuberance may be seen as domineering or as impressive, for instance, constraint as
showing admirable moderation or as shy and timid.

Velocity
The meaning potential of velocity derives from our physical knowledge of what slows us
down and what speeds us up (age or fatigue, for instance) and our cultural knowledge of
occasions which require slow movement – solemn processions, for instance, or funerals.
But slowness can also be pleasurable and relaxing – leisurely strolling through a park, for
instance, or taking time over a job that needs care and precision. Members of the ‘Slow
Movement’ see slowness as a ‘life affirming’ way of ‘connecting to spirituality’ (Slow
Living, 2020). In the corona virus video, the slow movement of the viruses becomes
menacing and suspenseful, as it delays the feared impact of the viruses. Speed, on the
other hand, suggests energy, as in the animated Intel logo (Figure 1). It is highly valued
in many circumstances, but too much speed can overwhelm and confuse, making it
impossible to keep up with things.

Force
Movements with the same directionality, expansiveness and/or speed may have different
degrees of force, ranging from maximally forceful to maximally gentle, from fortissimo
to pianissimo, one might say. Like loudness in sound, force can suggest vigour or power
as well as anger. The opening of the Intel Logo (Figure 1) has force, as do the stamping
rituals of military drills, or forceful blows in a fight. But force can also express a positive
108 Multimodality & Society 1(1)

intensity of belief and commitment. The opposite of force, too, can have positive or
negative overtones – it can be weak and timorous, or gentle and tender, for instance.

Angularity and fluidity

Like graphic shapes, movements can be angular or curved, regardless of whether they
are, for instance, movements of the whole of a body or parts of it. Our experience of our
natural and cultural environment tells us that curved forms and curved movements
dominate the natural world and rectilinear forms and movements the world created by
humans. The curved trajectory of the flash of light in the Intel logo (Figure 1), therefore
also suggests a natural movement, akin to the orbits of planets and moons.
Closely related is the issue of fluidity, the contrast between ‘connection’ and ‘dis-
connection’, between long, unbroken, smooth movements and movements that consist of
distinct short steps. Like staccato in music, disconnection can be lively, energetic and
determined, but also disjointed and mechanical, movement as a succession of still
frames, as in the experiments of Muybridge. And like legato in music, connectedness
can be smooth and sensual, but also imprecise, slurring things together. It all depends on
the context. Some well-known ballets of the late 19th century and early 20th century,
such as Coppélia (1870) and Petrouchka (1911) revived the age-old story of Pygmalion,
in which a sculptor’s creation comes to life (Austin, 2016). Such ballets contrast the
angular, mechanical movements of automata with more fluid and natural human move-
ment. Today, machine-like forms of dance and music have entered popular culture, e.g.
the ‘Dancing Machine’ which Michael Jackson performed with the Jackson Five in 1973,
and the jerky, mechanical movements of the ‘robot dancing’ of the 1980s.

Directedness
Displacement necessarily occurs in specific directions, but not all displacements take the
shortest route. Movements may turn and twist, zigzag, move stepwise, and more. Embo-
died experience can tell us what kinds of things can cause indirection. We may zigzag to
avoid obstacles, for instance. But indirection may also be symbolic – and aesthetically
pleasing. Like the trills, mordents and turns of baroque music, flourishes in dancing
intersperse displacement with ‘movements-in-place’. In the minuet, as described by
Sachs (1937: 407), dancers moved ‘with little dainty steps and glides, to the right and
to the left, forward and backward, in quarter turns, approaching and retreating hand in
hand, searching and evading, now side by side, now facing, now gliding past one
another’. Such movements were aesthetically pleasing as well as meaningful: the minuet
was a couple dance, enacting a stylized courtship ritual, but the couples acknowledged
the other dancers and the spectators in the room with little bows in their direction. This
would no longer happen in the waltz, the minuet’s 19th century successor, in which
couples had eyes only for each other, as if there were no other dancers in the ballroom at
all – the 19th century was a time of increasing separation between private and public life.
van Leeuwen 109

Regularity
Movements can be regular or irregular, rhythmically organized and periodically pat-
terned or meandering, wavering, teetering, oscillating irregularly. The meaning potential
of irregular movement stems from the same experiences as the meaning potential of
irregular shapes – physical conditions such as intoxication or infirmity, uncertainty and
confusion, or a refusal of control and discipline. But irregularity may also celebrate
human spontaneity and unpredictability, for instance in forms of dancing and music that
contrast the regularity of the mechanical (or, today, the electronic) with a human touch.
In such cases, regular movements may be regarded as repetitive and mechanical, but in
other cases they may seem well-proportioned and elegant.
On the ABC news, the movements of the quivering corona virus are irregular, and in
shot 2 of the corona virus video, the viruses hover irregularly around the people. This
makes the viruses unpredictable and hence dangerous – we do not know where they may
move next. Complex movements of this kind could of course also be interpreted with
reference to creatures or objects that we know to move this way, for instance mosquitoes.
In a 2003 New York exhibition, David Byrne, who is not only a musician but also an
author and artist, used PowerPoint as an art medium in a number of videos (cf Van
Leeuwen et al., 2013). In one these, ‘Architectures of Comparison’, arrows wander
slowly and aimlessly across the screen, accompanied by the sound of a dreamy waltz
(Byrne’s interpretation of the love duet of Un di Felice, from Verdi’s La Traviata).
Arrows, in PowerPoint, function to point at important content elements or make mean-
ingful connections between content elements. But here the arrows meander without a
sense of direction and without affecting anything. As Byrne has commented: ‘Goal-
oriented behaviour is like sleep-walking. It is easy and purposeful, but what is its
purpose? Its purpose is itself’ (Byrne, 2003: 7)
Figure 4 summarizes the distinctive features of movement styles. They all play a role
in every instance of movement, and they are all graded features, clines running, for
instance, from maximally fast to maximally slow.

Dynamization
So far, I have discussed the distinctive features of movements that realize actions and
events. But animation can also dynamize the attributes and identities of the elements
seen on the page or screen. Elements, be they images, texts (or parts of images or texts)
or abstract graphic shapes, can, in front of our eyes, change shape, colour, or texture (at
least insofar as texture can be visually represented). Change of shape can either mean a
change in attributes such as size or boldness, or it can change the identity of an element
entirely, changing an image into a letter, for instance, or a frog into a prince, or, as can
now be easily done with a programme called ‘Monsterfly’, changing yourself into a
zombie, werewolf or vampire.
Changes of shape can dynamize the distinctive features of shape (cf Van Leeuwen,
2006), separately or in various combinations: curvature/angularity, regularity, weight,
size, connectivity, sloping, expansion, density and orientation. Changes of colour, simi-
larly, can dynamize the distinctive features of colour (cf Van Leeuwen, 2011) – value,
110 Multimodality & Society 1(1)

Figure 4. The distinctive features of movement.

saturation, purity, transparency, luminosity, luminescence, lustre and temperature. In

shot 1 of the corona virus video, the ‘cloud’ of viruses grows in size. In shot 12, the
viruses change identity and transform into plus signs, which, along with the thumbs up
gesture of the hands in that shot, suggests that the preventive measures covered by the
video will have a positive outcome, although it is somewhat strange that this
van Leeuwen 111

Figure 5. Still from Girl Effect ‘The Clock is Ticking’. Reproduced with kind permission from Girl
Effect.

transformation happens both to the still active pink viruses and to the white dots that
represent suppressed viruses.
The following excerpt comes from a 3-minute animation titled The Girl effect – The
Clock is Ticking. It is almost entirely based on changes of shape. As a clock shows the
years of her life ticking away, we see a girl growing up in poverty, marrying at age 14,
having her first child at age 15, and ending up selling herself and risking HIV to feed her
family. The clock then goes back and restarts with a positive story in which the girl has
an education that allows her to earn a good living and have children when she is ready.
Figure 5 gives an impression of the style of the video.
FULL SHOT DETAIL OF CLOCK. At ‘1’ the clock stops moving and fades out. The
number one remains in the centre of the image. A dot appears to the right of the ‘1’ and
grows in size. The number ‘1’ then changes into ‘2’ and the dot continues to grow in size,
taking the shape of a small baby, sprouting a single hair and issuing tears. As ‘2’ changes
into ‘3’, and ‘3’ into ‘4’, the dot grows further and becomes a young girl with pigtails. As
‘4’ changes into ‘5’ and ‘5’ into ‘6’, she grows still further, acquiring a skirt and legs. As
‘6’ changes into ‘7’, she acquires arms. As ‘7’ changes into ‘8’, she makes a few dance
steps. As ‘8’ changes into ‘9’ and ‘9’ into ‘10’, she grows further. Then, as ‘10’ changes
into ‘11’, she moves towards the left, merges with the number 11, and becomes a black
pillar. The pillar then changes into the number ‘12’, the girl emerges from it and moves
to a position at the left of the number. Two white lines appear on her body, outlining
budding breasts.
112 Multimodality & Society 1(1)

This short segment is entirely told in terms of changes of shape. Although we briefly
see the hands of the clock move, after that only the numbers remain, morphing into each
other, and although the girl makes a few brief dance steps and later moves to the other
side of the screen, her story, too, is told only by her changing shape. When she becomes
12 years old, her identity changes: she merges with the number that indicates her age
(‘11’) to momentarily turn into a black monolith, which perhaps symbolizes the fatal
transition to a damaged life.
Even the very existence or non-existence of elements can be dynamized: elements can
appear or disappear, whether gradually or instantaneously. The cinema early on devel-
oped dissolves and fades (and many variants that have not stood the test of time) to
provide gradual transitions between shots or scenes. In animation films and videos, not
only whole shots can appear or disappear, also parts of shots. In The Girl Effect, for
instance, the dot that will grow into a girl appears out of nowhere in the middle of a
continuing shot, and in the corona virus video people and things appear and disappear as
if created by an invisible hand. Such transitions can be compared to Halliday’s ‘exis-
tential processes’ (1985: 130). They signify ‘coming into being’ or ‘moving out of
being’. They dynamize the idea of ‘being’.
The ‘animation schemes’ offered by PowerPoint are for the most part transitions of
this kind (‘entrances’ and ‘exits’), and they can also provide salience, as movement
draws the viewer’s attention more than any of the other factors that can create salience.
The PowerPoint animation system is therefore in the first place functional. It provides
the textual functions of framing and salience (Kress and Van Leeuwen, 2006). But the
different ‘entrance’, ‘exit’ and ‘emphasis’ options PowerPoint offers are subcategorized
as ‘basic’, ‘subtle’, ‘moderate’ and ‘exciting’, and these kinds of meaning are realized by
the distinctive features discussed in this paper – direction, velocity, angularity and
directedness, and so on, as also indicated by the names of the animation effects – exciting
entrance effects, for instance, include ‘boomerang’, ‘float’, ‘spiral in’ and ‘swivel’,
exciting emphasis effects ‘wave’, ‘blast’, ‘blink’, and more.

Mobility and identity design

Moving now to resources for creating mobility, these can also realize identity meanings,
albeit in a more interactive manner. The wheeled desk chair designs its user as someone
who is ‘mobile’ in some way, perhaps by being busy, moving from one activity to
another, attending to a range of different things. In one University where I worked, the
chairs in new lecture theatres were made to swivel, so that students could turn away from
the lecturer, and interact with each other. Such differences facilitate preferred identities –
the multi-tasker, the active learner who learns through team projects rather than from a
teacher.
In the case of toys, such subject positions become imaginary. lf, in a toy car, only the
driver’s door and the steering wheel can be moved, the user becomes an imaginary
driver. If the bonnet can also be opened, revealing an engine, the user can also become
an imaginary mechanic. Such toys have built-in assumptions about the user’s needs and
interests and may provide obstacles for other uses. An important part of learning consists
van Leeuwen 113

precisely in encountering these obstacles and their implicit messages. What you cannot,
or not easily, do with objects is as important as what you can do.
‘Rigid’ objects can of course also be made to move. An inert wooden figurine can be
made to walk, jump, dance, fly and much more. I can unproblematically balance my
kitchen chair on its back legs to lean back or move it from one position to another. In fact,
inert objects may allow more uses than kinetically designed objects, which inevitably
favour certain uses or positions.
In the world of toys, many objects allow mobility. Toy telephones, toy computers,
even books for young children may all have wheels. In such cases mobility becomes a
metaphor. It teaches children that telephones, computers and books ‘get you some-
where’, help you to ‘be on the move’.

Articulation and flexibility

There are two basic ways in which objects can be ‘non-rigid’. They can be articulated, in
which case they will have joints or hinges, as with the limbs of Sindy and Action Man, or
they can be flexible, made out of materials that can be bent into different positions or
moulded into different configurations, as with Sindy’s hair, which, unlike Action Man’s
rigid cap of short hair, can be styled in different ways. Flexibility is a of course matter of
degree. Sindy’s face is also flexible. Unlike Action Man’s face, it is made of soft plastic,
whereas her body is made of hard plastic. But this does not make it as flexible as the face
of a ragdoll or a muppet-style soft glove puppet.
Articulation and flexibility define what we can do with an object or what we can make
an object do, In the case of representational toys, they represent how objects, animals or
people can move. As we have seen in the case of Sindy and Action Man, such repre-
sentations are not necessarily faithful to physical reality. In one plastic toy for babies,
intended to be suspended across the cradle, in reach of baby’s arms and feet, a number of
objects can rotate around a plastic pipe – a kind of paddle-wheel, something resembling a
heavily profiled tyre, and, in the centre, a yellow plastic dog, The ‘technical’ objects, the
‘paddlewheel’ and the ‘tyre’, rotate easily, without friction. The dog, on the other hand,
rotates stiffly, by means of a noisy ratchet mechanism. An abstract quality (and a
technological bias) is demonstrated here: technology is more easily controlled and
moves more freely than living beings.
The difference between articulation and flexibility is important. Articulation is neces-
sarily based on a principled approach to what should and what should not be able to
move and how, as in the case of the ‘body language’ underlying the articulation of Sindy
and Action Man, or the kinds of work which the design of a chair can support. Flexibility
not only provides users with more tactile satisfaction, as it necessarily involves a degree
of softness, it also allows more creative freedom. Soft toys such as teddy bears, can be
cuddled. They have affective value. But most importantly, they have a relatively open-
ended kinetic potential, although the transformations flexibility allows will often be
fleeting, as the materials will bounce back in their original shape as soon as the user
withdraws his or her hands. This provides another lesson: tactile, affective and creative
experiences are fleeting, leaving no traces, while the more deliberate transformation of
objects through a system of articulation does. There are of course flexible materials
114 Multimodality & Society 1(1)

which do allow more permanent transformations, for instance rubber, but their use in
toys has remained relatively marginal.

(De)constructability
Deconstructability provides a ‘hands on’ lesson in what things are made of and how they
fit together. Even when toys are not designed for deconstruction, children may want to
take them apart, not from some kind of destructive urge, but from a desire to learn.
Deconstructable toys teach analysis – although children may also ‘deconstruct’ in a
silent protest against turning play into work, as documented by Ito (2009) in her study
of children playing SimCity in an after school workshop.
Any object can be analysed in different ways and according to different principles. In
the past, and to a degree still today, construction toys such as Meccano or Lego taught
children that the world is built up out of a small number of in themselves meaningless
units (‘atoms’ or ‘molecules’ one might say). Larger, meaningful units could be con-
structed from these basic units. The basic units themselves connote the dominant con-
struction methods of the time, steel girders in the case of Meccano, bricks, in modernist
colours, in the case of Lego. Such toys position children as imaginary engineers or
designers. More recently, construction toys often come in do-it-yourself kits which
contain just the right amount and kind of blocks for one and only one larger item, for
instance a Ninjago warship, or a Star Wars spacecraft with helmeted occupants. This
makes users consumers rather than builders or designers and withholds a whole dimen-
sion of analysis and construction from them. Children can of course work against such
restrictions. In one of the videos we made as part of a project on toys, a 7-year-old
mounted the rotor blade of a Lego helicopter on a kind of pedestal and put it in the garden
of a house, a garden otherwise filled with plastic miniature ready-made trees, parasols
and garden furniture, etc. It would keep the garden cool, he explained.

Sources of kinetic energy

Mobile and articulated toys may either be manually driven or powered by some other
form of artificial energy, a clockwork mechanism, for instance, or an electric motor.
Toys may also be driven by natural power, by wind, water, or gravity. Few contemporary
toys make use of this, so that children perhaps do not get enough chances to learn how to
work with these forms of power, despite the fact that many traditional toys such as kites
and spin tops owe their enduring fascination precisely to the way in which they allow the
exploration of natural sources of energy.
The powering of mobile objects can be ‘instigatory’ or ‘continuous’. In the former
case the user sets the object in motion or exposes it to a natural source of energy, but does
not control its movements throughout the duration of the movement, as in the case of
clockwork-driven cars or paper aeroplanes thrown and then left to the currents of the air.
In the case of continuous control, control is exercised throughout the duration of the
movement. Hand-driven instigatory control requires skill and co-ordination, especially
when the effects of natural forces such as the wind are to be predicted and inculcated in
the movement.
van Leeuwen 115

Figure 6. The system of kinetic design.

The powering of objects can also be ‘direct’ or ‘remote’. The continuous control of
artificial energy is always remote, leading to different forms of human-machine inter-
faces: rotating knobs, steering wheels, joysticks, the computer mouse, the touch screen,
the drawing pad – and to the subtle skills of fingertip control which, today, are of
increasing importance in contemporary society as more and more activities are powered
by remote control, from street sweeping to brain surgery.
Figure 6 summarizes the types of mobility discussed in this section.

Kinetic sculpture as semiotic research

I started this paper by stressing the role of artists in developing the resources of mobility
design, and will end by discussing the work of the Swiss sculptor Jean Tinguely (cf Van
Leeuwen, 2016), who, during his long career, has explored every single one of the
dimensions of mobility design I have discussed in this article.
His earliest works were abstract paintings inspired by Malevich, Kandinsky and
others (e.g. ‘Meta-Malevich’, 1956) in which the abstract shapes were articulated by
means of pulleys linked by a rubber belt to a 78 rpm gramophone motor hidden behind
the painting. Later he experimented with other energy sources, using electromotors and
even petrol engines to drive the gears, cams, cranks and levers that produced the move-
ments. In many of these works the articulated movements were repetitive. In Inferno
116 Multimodality & Society 1(1)

(1984), for instance, a metal beam repeatedly tried to lift itself into an upright position,
but never managed to do so, in an endless Sisyphus-like toil, while other, smaller parts of
the same sculpture energetically repeated the same pointless movements.
Tinguely also devised many techniques for producing irregularity, unpredictability of
movement -differently sized wheels, elastic rubber belts, the use of several motors, etc.
His Jealousy 1 (1960), for instance, represented jealousy by the nervous jiggling of
wooden beads suspended from a moving metal bar.
In later work, he introduced flexible materials such as feathers, dusters, rags and
clothes. As described by Hulten (1975: 204), Maranar (1961) was a hanging sculpture
in which:

nine levers hung from camshafts which were hidden by an intermediate ceiling. When the
machine was switched on, the levers tore at a tangled heap of junk and rags, a dirty night-
dress, an artificial leg wearing a red sock, a washing-up bowl, coffee-tins and lengths of
film. After the machine had been running for a certain time, this collection of junk suddenly
exploded into violent, spastic jerks.

Although many of Tinguely’s sculptures are immobile, he used mobility almost from
the beginning, mostly by putting his sculptures on wheels, but also by using rails,
conveyor belts and so on. Auto-Mobile (1954) was a small sculpture which could move
through the room with a small electromotor. In Klamauk (1979) Tinguely used a tractor
to support a contraption of cogwheels, the motions of which activated hammers and
metal sticks banging on bells and cymbals in a happy fairground cacophony.
And while many of Tinguely’s sculptures were driven by motors, he also explored
manually and naturally powered motion. His Prayer Mills (1954) were wire sculptures
with interlocking wheels, some vertical, some horizontal, which viewers could move
with a crank handle. His Cyclograveur (1960) was a stationary bicycle with a saddle for
the viewer to sit on and pedals to activate a rusty and complex set of interlocking
cogwheels, which ultimately drove a drum, a cymbal, and a rusty toy car.
In many of these sculptures, viewers could instigate the action manually (or with their
feet), as in Prayer Mills and Cyclograveur. In other cases, they could switch on a motor.
In Dissecting Machine (1965) viewers activated drills and saws which attacked the
dismembered parts of a life-like window dummy, so diminishing, as one reviewer put
it, the ‘gap between the vicarious enjoyment of cruelty and the act itself’ (Herald
Tribune, 28 May 1971). In other cases, movement was ‘autokinetic’, so that the machine
appeared to have a life of its own. Velocity also played a role – animal skulls rocked
gently, discs rotated furiously, expressing anger and frustration.
Throughout his career, Tinguely developed an arsenal of resources for mobility
design and explored their meaning potentials, initially primarily in relation to the
machine itself – its mindless repetition, its wastefulness (one of his works endlessly
smashed bottles). In later work, he broadened his vocabulary, using not only machine
parts but also plastic toys, clothes, feathers, animal skulls, etc. animating them with the
movement resources he had developed, to create a wide range of genres and contexts
from altar pieces and portraits to children’s playgrounds and theatrical sets.
van Leeuwen 117

In short, Tinguely (along with Calder) pioneered the semiotic mode of mobility I have
tried to describe here, and he did so a long time before I began to discover the importance
of mobility design in my research on toys (Van Leeuwen and Caldas-Coulthard, 2004)
and on kinetic typography (Van Leeuwen and Djonov, 2015). More broadly, 20th cen-
tury modernist artists such as Tinguely have played a fundamental role in cultural
semiosis, the creation of new forms of meaning-making, and in the renaissance of multi-
modality. Today’s semioticians (and software designers) stand on their shoulders, and
would do well to recognize that.

Acknowledgement
The drawings in this paper are by Toby van Leeuwen.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship,
and/or publication of this article.

Funding
The author(s) received no financial support for the research, authorship, and/or publication of this
article.

References
Anglepoise (2020) Available at: https://www.anglepoise.com/manifesto/the-long-view (accessed
3 February 2021).
Austin L (2016) Elaborations of the Machine: The Automata Ballets. Available at: https://moder
nismmodernity.org/articles/elaborations-machine-automata-ballets (accessed 3 February
2021).
Bellantoni J and Woolman M (2000) Type in Motion – Innovations in Digital Graphics. London:
Thames and Hudson.
Byrne D (2003) Envisioning Emotional Epistemological Information. New York, NY: Steidl Pace-
MacGill.
Cooke D (1959) The Language of Music. Oxford: Clarendon.
Forlizzi J, Lee J and Hudson S (1997) The kinedit system: affective messages using dynamic texts.
In: Proceedings of the CHI 2003 conference on human factors in computer systems, Ft. Lauder-
dale, FL, USA, April 2003, pp. 377–384.
Goffman E (1976) Gender Advertisements. London: Macmillan.
Halliday MAK (1985) An Introduction to Functional Grammar. London: Arnold.
Han J (2021) A Social Semiotic Account of Music-Movement Correspondences. Unpublished PhD
thesis, University of New South Wales.
He Y and Van Leeuwen T (2019) Animation and the remediation of school physics – a social
semiotic approach. Social Semiotics 30(5): 665–684.
Hulten P (1975) Jean Tinguely: Meta. Boston, MA: New York Graphic Society.
Ito M (2009) Engineering Play – A Cultural History of Children’s Software. Cambridge, Mass:
MIT Press.
Jackson DM (2003) Sonic Branding – An Introduction. London: Palgrave Macmillan.
Jakobson R and Halle M (1956) Fundamentals of Language. The Hague: Mouton.
118 Multimodality & Society 1(1)

Johannessen CM and Van Leeuwen T (2018) (Ir)Regularity. In: Johannessen CM and van
Leeuwen T (eds) The Materiality of Writing – A Trace-Making Perspective. London: Routledge,
pp. 175–192.
Kepes G (1965) The Nature and Art of Motion. New York, NY: George Braziller.
Kress G and van Leeuwen T (2001) Multimodal Discourse – The Modes and Media of Contem-
porary Communication. London: Arnold.
Kress G and van Leeuwen T (2002) Colour as a semiotic mode: notes for a grammar of colour.
Visual Communication 1(3): 343–368.
Kress G and Van Leeuwen T (2006) Reading Images – The Grammar of Visual Design, 2nd edn.
London: Routledge.
Lakoff G and Johnson M (1982) Metaphors We Live By. Chicago, IL: University of Chicago Press.
Leão G (2012) Movement in Film Titles: An Analytical Approach, Unpublished PhD, University of
Technology, Sydney.
Mumford L (1934) Technics and Civilization. New York, NY: Harcourt, Brace and World.
Sachs C (1937) World History of the Dance. New York, NL: W.W. Norton & Co.
Sheets-Johnstone M (2013) The Primacy of Movement. Amsterdam: John Benjamins.
Slow Living (2020) Available at: https://vanillapapers.net/2019/02/20/slow-living-blog-quote
(accessed 3 February 2021).
Tagg P (2000) Melody and Accompaniment. Available at: https://tagg.org/articles/melodaccUS.pdf
(accessed 3 February 2021).
Van Leeuwen T (2005) Introducing Social Semiotics. London: Routledge.
Van Leeuwen T (2006) Towards a semiotics of typography. Information Design Journal 14(2):
139–155.
Van Leeuwen T (2011) The Language of Colour – An Introduction. London: Routledge.
Van Leeuwen T (2016) Creativity in the fourth dimension – the grammar of movement according
to Jean Tinguely. In: Jones R (ed.) The Routledge Handbook of Language and Creativity.
London: Routledge, pp. 336–352.
Van Leeuwen T and Caldas-Coulthard CR (2004) The semiotics of kinetic design. In: Banks D (ed.)
Text and Texture: Systemic-Functional Viewpoints on the Nature and Structure of Text. Paris:
L’Harmattan, pp. 355–380.
Van Leeuwen T and Djonov E (2015) Notes towards a semiotics of kinetic typography. Social
Semiotics 25(2): 244–253.
Van Leeuwen T, Djonov E and O’Halloran K (2013) ‘David Byrne really does love PowerPoint’:
art as research on semiotics and semiotic technology. Social Semiotics 3(3): 409–423.

Author biography
Theo van Leeuwen is Professor of Language and Communication at the University of Southern
Denmark. He has published widely in the area of visual communication, multimodality and critical
discourse analysis. His most recent book is the 3rd edition of Reading Images – The Grammar of
Visual Design (with Gunther Kress). His new book Multimodality and Identity will appear in
July 2021.