multimodal instructions. a descriptive linguistic analysis - tekom
TRANSCRIPT
Multimodal instructions.
A descriptive linguistic analysis
Dr. Birgitta Meex
Dr. Cornelia Wermuth
tcworld 2013
6.- 8.11.2013
Background
tcworld 2013
Dr. B
irgitta
Meex
• Assistant Professor of German and Applied Linguistics
• University of Leuven
• Language and Communication
• Master Multilingual Communication
• Member of the research group Multimodality, Interaction and Discourse @ Leuven
• Certified Technical Communicator tekom
• President IC tekom Belgium
• President advisory board European Academic Colloquium on TC
• Secretary tekom Europe
Dr.
Corn
elia
Werm
uth
• Assistant Professor of German and Applied Linguistics
• University of Leuven
• Language and Communication
• Master Translation Studies (Medical Translation)
• Member of the research group Multimodality, Interaction and Discourse @ Leuven
Background
tcworld 2013
Research group
Multimodality, Interaction and Discourse @ Leuven (MIDI)
www.arts.kuleuven.be/ling/midi
• Relation between language and other semiotic modes in multimodal institutional genres
• Focus on technical-medical discourse
Agenda
• Relevant concepts:
o Modern information products
o The concept of multimodality
o The concept of discourse
• Research questions
• Methodology
• Analysis and results
• Practical implications
• Summary
tcworld 2013
Modern information products
• Emergence of new technologies
• Integration of text, image, and sound
• Continuum text – layout - image – sound ↔ simulation - interactivity -
virtual reality (Van Bart & Steehouder 2008)
• Inherently and increasingly multimodal
tcworld 2013
The concept of multimodality
• Multimodality: “The phenomenon in texts and communicative events
whereby a variety of ‘semiotic modes’ (means of expression) are
integrated into a unified whole” (Van Leeuwen & Kress 2011: 107)
• Semiotic modes: “the material resources we use in multimodal
communication […] for purposes of communication and expression”
• Semiotic resources: physiological and technical
tcworld 2013
Semiotic resources: Physiological
tcworld 2013
Semiotic resources: Technical
tcworld 2013
The concept of multimodality
• Terminological confusion!
• Two perspectives on multimodality (Pauwels 2012: 250)
o Input modes: Senses (visual, auditory, tactile, gustatory, olfactory)
o Output modes: Media/devices
• Intermodal relations:
o Interplay of semiotic modes (→ communicative act)
o Effects on the user (→ perceptual processing)
o From linear to non-linear (→ presentation format)
tcworld 2013
The concept of multimodality
tcworld 2013
The concept of discourse
tcworld 2013
• “Socially constructed knowledges about some aspect of reality” (van
Leeuwen & Kress: 113)
• Realized in different genres (design)
• Realized in different (combinations of) modes (materialization)
Case study
Multimodal instructions
A descriptive linguistic analysis of instructive discourse across different
semiotic modes
tcworld 2013
Objectives
tcworld 2013
• Better insight into similarities / differences / interaction between the
specific modes in instructive discourse
• Better understanding of their common properties and communicative
potential (van Leeuwen & Kress 2011)
• Practical implications for technical communication: Which presentation
format to choose for which purpose and communicative situation?
Research questions
tcworld 2013
• Which modes are involved in technical instructions?
• What can be observed with respect to modal density?
• Which intermodal relations can be identified?
• What can be observed with respect to the modal configuration?
• How is coherence established in nonlinear multimodal communication
forms?
Methodology
• Qualitative discourse analysis
o Empirical: data-driven
o Multimodal: text, image, sound
o Descriptive and interpretative
• Corpus of 15 German movie instructions (< YouTube):
o 5 cooking instructions
o 5 medical patient instructions
o 5 technical user instructions
o + their respective textual counterparts
tcworld 2013
Analysis and results
tcworld 2013
• Semiotic resources
• Intermodal relations
• Modal density
• Discourse types
• Modal configurations and communicative purposes
Semiotic resources
tcworld 2013
• Verbal
• Paraverbal
• Nonverbal
Verbal • Central semiotic code: logocentric view
• The symbolic nature of language: a
semantic structure is related to a
phonological structure that symbolizes it
• Both written and spoken language
• Fulfilling all semantic, syntactic, and
pragmatic functions:
o Sender-viewer references: pronouns and forms
of address
o Deictic markers
o Qualitative / evaluative markers
o Mood (declarative, interrogative, imperative)
o Rhetorical elements
o Vocabulary
o Register
o Foreign and loan words
The prominent status of
language
tcworld 2013
Peripheral (sub)modes related to language
Paraverbal
The use of non-linguistic modes
Paprikas sind sehr gesúnd!
• Tone of voice
• Intonation
• Word stress
• Accent
• Pauses
• Volume
• Articulation
• Resonance
• Timbre
tcworld 2013
Peripheral (sub)modes related to language
Nonverbal
The use of non-linguistic modes
o Visual
• Moving images
• Color, clothing, logo
• Facial expressions, eye-contact,
gaze, laughing
• Different camera perspectives
o Audio
• Music (volume, pitch, rhythm)
• Sound (vocal and non-vocal;
realistic and designed)
• Voice-over, voice-off
o Gestural
• (Pointing) hand gestures
• Body posture
o Spatial
• Motion within scenery
tcworld 2013
Intermodal relations
tcworld 2013
• Meaning is constructed by an interplay of two or more modes:
o Indexical function (e.g. intonation, deictic devices (so, this), pointing
hand gestures)
o Iconic function (e.g. shaking head, forbidden sign, ganz ganz hoch,
pictograms, logos, depictive gestures)
o Symbolic function
• Linguistic, visual, auditory, and gestural interplay
• High modal density!
Intermodal relations: Key information/actions
tcworld 2013
• Moving images
• Actions of the (virtual) person
• Spoken text (speaker, co-speaker, voice-off)
• Grammatical constructions (e.g. imperative, infinitive, 3rd person present tense)
• Word stress (vocal sound)
• Real & virtual sounds
• Facial expression (actor)
• Written text
• Color highlights
• Animations
• Fly-ins
• Written enumeration
• Zoom-in/-out camera movements
Intermodal relations: Examples
tcworld 2013
• Example of verbal-visual relation with metaphorical character:
• “Nasenspray richtig anwenden”
• Typography: simple letters without serifs in blue metaphorically stand
for cleanliness, confidence, truth, precision of the verbally described
instruction
Intermodal relations: Examples
tcworld 2013
• Example of verbal-visual-auditive relation with metonymic character
“Vortex Inhalationsgerät”
• The visual depiction of a mountain crest under a blue sky
• Simultaneous verbal reference Freiheit ist Atmen
• Calm soothing piano music
Intermodal density: Examples
tcworld 2013
• Several modes to signal excellence
• Gestern hat unser Koch, unser Spitzenkoch, ähm, Attila Hildmann, ja ein
vegetarisches Gericht gekocht, Spaghetti Bolognese, das richtig lecker war,
ausgezeichnet.
o Verbal mode: lexical and pragmatic intensifiers
o Paraverbal mode: full word stress, mouth open wide
o Nonverbal mode:
→ Eyes closed and head slightly bent downwards to the right
→ Accompanying hand gesture: okay sign
Intermodal density: Examples
tcworld 2013
• Example: Video-Rezept: McFish – schnell und einfach
• Presentation of ingredients with several simultaneous modes:
o Visual mode: close-up of ingredients
o Verbal-auditive mode: oral enumeration of ingredients
o Gestural mode: each ingredient is pointed at by the cook’s hand
Types of discourse
tcworld 2013
• Narrative
• Instructive
• Commercial
→ Hybrid character of movie instructions
Narrative discourse
tcworld 2013
• Storytelling: narrator = protagonist
• Significant events, personal experience, opinions, evaluations, and
knowledge
• Agent-oriented
• First or third person pronoun forms
• Present or past tense indicative
• Camera perspective: Medium close-up or close-up
Instructive discourse
tcworld 2013
• Visual demonstration how something is done
• Direct address of the viewer
• Second person pronouns du/Sie or first person pronouns ich/wir
• Use of the imperative and the infinitive
• Short sentences
• Procedural character: a series of chronological steps leading to a goal
• Camera perspective: Close-up
• Focus on the person’s hand and the contingent actions performed
• Accompanying real or virtual sounds
Commercial discourse
tcworld 2013
• Promote the product, company or brand
• Corporate name, logo, and claim
• Linguistic means: rhetorical devices, melioration (cf. Schröder 2013)
• Music
• Prominent placement
Modal configurations
Communicative purposes
Motivation
Persuasion
Related aspects
Coherence
Voice (POV)
Target audience
Sender-viewer relationship
tcworld 2013
Expert footing
Motivation: To view the movie instruction
tcworld 2013
• Task is linguistically referred to as “easy/easier” and/or “fast”
Examples: Ganz einfach – ganz schnell; in nur 15 Sekunden
• Task is linguistically referred to as “complex”
Example: Die Anwendung eines Nasensprays ist einfach. Da kann man nichts
falsch machen. Oder doch?
• Task is introduced by a warning
Example: Fallen, in die man tappen kann, wenn man regelmäßig Medikamente
einnimmt
• Task is staged as an experience & engagement by means of emotional
triggers and attention spots: pleasure, curiosity, humor, irony, …
Example: Njamnjamnjam
Motivation: To become aware of the potential benefit(s)
tcworld 2013
• Different possibilities or advantages associated with the task/end result
are highlighted:
Examples:
Ich nehme das Sandwich nämlich immer mit, wenn ich unterwegs bin.
Written list with five reasons why one should opt for Roto-windows:
o Montagefreundlichkeit
o Produktqualität
o Energieeffizienz
o Lieferfähigkeit
o Partnerschaft (+ small logo visualizing each of them)
Motivation: To perform the task yourself
tcworld 2013
• Ready-to-use setting
• Example: nasal spray 3D-animation
• Shift from narrative ich to du/Sie or inclusive wir (divided agent role)
• Shift in camera focus from medium close-up to close-up
Persuasion: Maximal involvement of the user
tcworld 2013
• Identification offer: real person ↔ virtual person
• Eye-contact
• Frontal camera view at eye-level
• Camera perspectives: medium-shots and close-ups
• Pleasant setting and exciting music
• Use of jokes & sayings
• Hand movements demonstrating love for product
• Melioration:
o Wir machen nämlich ein superleckeres Sandwich, gutes Olivenöl, schön
würzig, richtig dünne Scheiben, mit reichlich Pfeffer, richtig schön kross
geworden, unser schönes Vollkornbaguette, die sind wirklich
hauchdünn.
• Fictive dialogues with the audience
o Rhetorical questions: Warum ist dieses Vorgehen sinnvoll?
Persuasion: Associative relations
tcworld 2013
• The use of verbal and visual metaphors and metonymy:
o Bildlich dargestellt ist der Wirkstoff wie in einer Seifenblase
gefangen.
o Die Enzyme muss man sich vorstellen praktisch wie eine
Bahnschranke, vor der dann der Wirkstoff stehen bleibt wie eine
Autoschlange. (verbal metaphors)
o Running and ticking clock stands for time (visual metonymy)
o Informative and factual function of blue in medical and technical
instructions: confidence, intelligence, truth, precision (visual
metaphor) (cf. Förster 2003)
Expert footing
The speaker positions himself as an expert. How?
• Storytelling: exhibiting
experience, sharing knowledge,
answering questions
• Technical vocabulary
• Explicit linguistic and/or
visual reference to the written
version
• Explicit linguistic and/or
visual reference to expert
• Setting/location
Paprikas haben übrigens mehr Vitamin
C als Zitronen, sind also wirklich sehr
gesund.
die Nasenscheidewand, das
sogenannte Septum
Bitte lesen Sie in jedem Fall auch die
Gebrauchsanweisung.
Unser Experte für die vegane Küche
Pharmacist / pharmacologist/ technician
Pharmacy / laboratory / assembly shop
tcworld 2013
Expert footing
The speaker positions himself as an expert. How?
• Attributed evidentials: Awards
• Unattributed evidentials: Self-
reported
• Typography: Clothing
Ausgezeichnet als vegetarisches
Kochbuch des Jahres vom
Vegetarierbund
Ich glaub, ich kann sehr gut beurteilen,
ob das ein guter Crêpe ist.
cook casual clothes
pharmacist doctor’s overall
technician boiler suit
tcworld 2013
Coherence: Beginning
tcworld 2013
• Advance organizers
o Music, jingle
o Company / logo / claim / mascot
o Informative title specifying the content
o Establishing shot
o Personal address: Salutation phrases
o Goal of the action is shown
o Ready-to-use setting
Coherence: Middle part
tcworld 2013
• Structuring elements in non-linear presentation formats
o Appearance of (virtual) person, mascot
o Temporal markers: structure subsequent instructions indexically (nun, jetzt)
or iconically (bevor, an-, abschließend)
o Stepwise instructions
o Dialogue format: Question-answer structure
o Preliminary summaries of key messages
o Warnings
o Repair scenarios
o Sometimes numbered lists as written text
o Multimodally marked shifts between discourse types
• Jingles
• Camera perspectives
o Background music: decorative function
Coherence: End
tcworld 2013
• Concluding elements
o Achieved goal of the action is stated (Fertig!) or shown
o Summary
o Music, jingle (higher volume)
o Company / logo / claim
o Personal address: closing phrases, eye-contact, subtle smiling
o Asking for feedback
Voice (POV)
tcworld 2013
• The protagonist = the dominant POV
• Sometimes also a co-speaker and/or an antagonist
• A real vs. virtual person
• First person vs. third person voice-off
• Example: Ich habe ein tolles Rezept für euch. Wir machen nämlich ein
Baguette […], wir belegen ein […] superleckeres Sandwich.
• Objective vs. (half) subjective camera positions
Target audience
tcworld 2013
• Construction of implied audience(s) and personae: age, gender, social
status, educational background, …
• Linked to embedded purposes & expectations:
o Looking for quick information on the correct handling / installation /
maintenance
o Evaluation
o Entertainment
Target audience
tcworld 2013
• Explicit vs. implicit address
o Verbally:
• An installer in blue desk coat addressing the viewing audience with
Liebe Kollegen
• Und wenn ihr zum Beispiel in der Schule seid und was richtig Leckeres,
Gesundes essen wolllt.
o Visually:
• An older lady figuring as an actor in a medical instruction video
• Female person with baby as an actor in a medical instruction video
Sender-viewer relationship and distance
• Formal (Sie) style
• Ich-Agens
• Standard language
• German words
• Full sentences
• Absence of eye-contact (e.g.
naked animated figure)
• Objective: total shot view
• Formally dressed
• Informal (du) style
• Shared agens role
• Regional language: moin, moin
• Foreign words, anglicisms
• Elliptic style: e.g. einen kleinen
Schluck Wasser rein
• Direct eye–contact
• Subjective: close-up frontal view
• Casually dressed
tcworld 2013
Practical implications
tcworld 2013
• New technologies: internet, tablets, smartphones
→ possibility to broadcast / consume video everywhere
• Today’s user is lazy, emotional, and always in an emergency
→ hates to read documentation
→ loves to watch videos (success of YouTube)
Practical implications
tcworld 2013
• Are semiotically rich audio-visual presentation formats more user-
friendly than their traditional counterparts with a focus on written
instructions?
• YES!
Practical implications
tcworld 2013
• Video instructions and animations:
o Information with a low cognitive load and a high emotional appeal
• Combination of moving images and spoken words is an optimal form of
multimedia usage (Mayer 2005 in Van der Meij & Van der Meij 2013)
• Several information channels (multimodal)
• Complementary (instead of repeated or isolated ) information
o Both educational and entertaining: excellent user experience!
o Translation of complex information into easy-to-understand formats
o Improved communication and better learning effects through
extensive viewing
o Useful and preferable extension of traditional documentation
• To introduce new (features of) products
• To illustrate “advanced technique[s] that [are] difficult to describe in written form”
(Samuels 2013)
Practical implications
tcworld 2013
• Integrated TD
o Printed instructions
• Part of the product
• Comprehensive: complete & accurate information
• Task-oriented modular & functional design
• Liability reasons!
o Corresponding audio-visual performance
• Short
• Informative title
• Task-oriented, procedural information, clear instructions, brief reflection pauses
• A good story and a good narrator
• Visual attention cues
• Information experience and content marketing
• Fitting the communicative purpose at hand!
The role of the technical communicator?
tcworld 2013
• The technical communicator is the ideal actor!
(p.c. Holger Thater)
• He/she knows about the product, task, communicative purposes,
company, and most importantly the target audience.
Summarizing…
tcworld 2013
• Movie instructions are inherently multimodal.
• They create an excellent user experience, at least if they fit the
communicative purposes at hand.
• They break down technical information into highly accessible series of
interrelated semiotic modes.
• The different modes are convertible (mode overlapping).
• The modes complement the authoritative written counterparts.
• The main benefit and appeal of multimodal texts resides in the mutual
enforcement of modes in semantic, formal and functional terms.
References
• Anstey, Michèle & Bull, Geoff (2010). Helping teachers to explore multimodal texts.
Curriculum and Leadership Journal 8 (16). Accessed via
http://www.curriculum.edu.au/leader/helping_teachers_to_explore_multimodal_texts,315
22.html?issueID=12141.
• Bucher, Hans-Jürgen (2013). Multimodalität – ein universelles Merkmal der
Medienkommunikation: Zum Verhältnis von Medienangebot und Medienrezeption. In:
Hans-Jürgen Bucher and Peter Schumacher (eds.), Interaktionale Rezeptionsforschung.
Theorie und Methode der Blickaufzeichnung in der Medienforschung. Wiesbaden:
Springer, 51-82.
• Carey, Jewitt (2009). The Routledge Handbook of Multimodal Analysis. London:
Routledge.
• Förster, Hans-Peter (2003). Corporate Wording. Das Strategiebuch . Für Entscheider
und Verantwortliche in der Unternehmenskommunikation. Frankfurt a.M.: Frankfurter
Allgemeine Buch.
• Norrick, Neil R. (2011). Conversational recipe telling. Journal of Pragmatics 43: 2740-
2761.
tcworld 2013
References
• Pauwels, Luc (2012). A Multimodal Framework for Analyzing Websites as Cultural
Expressions. Journal of Computer-Mediated Communication 17: 247-265.
• Samuels, Jacquie (2013). Technical Writer Tips & Tricks: Video Tutorials.
http://techwhirl.com/technical-writer-tips-tricks-video-tutorials/
• Schröder, Tilman (2013). Marketingstrategien auf Unternehmenswebsites im
internationalen Vergleich. Tübingen: Narr.
• Stöckl, Hartmut (2006). Zeichen, Text und Sinn – Theorie und Praxis der multimodalen
Textanalyse. In: Eva Martha Eckkrammer & Gudrun Held (eds.), Textsemiotik. Studien
zu multimodalen Medientexten (= Sprache im Kontext), Frankfurt am Main: Peter Lang.
11-26.
• Van Bart, Peter & Steehouder, Michaël (2008). Basisboek Technische Communicatie.
Assen: Van Gorcum.
• Van der Meij, Hans & Van der Meij, Jan (2013). Leren via YouTube. Het ontwerpen van
instructievideo’s voor softwaretraining. Tekstblad 3: 6-1.
• Van Leeuwen, Theo & Kress, Gunther (2011). Discourse Semiotics. In: Teun Van Dijk
(ed.), Discourse Studies. A multidisciplinary Introduction, 107-125. London: Sage.
tcworld 2013
