Multimodal Epistemologies: Towards an Integrated Framework

No Comments

EDITOR: Arianna  Maiorani
EDITOR: Christine  Christie
TITLE: Multimodal Epistemologies
SUBTITLE: Towards an Integrated Framework
SERIES TITLE: Routledge Studies in Multimodality
PUBLISHER: Routledge (Taylor and Francis)
YEAR: 2014

REVIEWER: Kunkun Zhang, Macquarie University

Review’s Editors: Malgorzata Cavar and Sara Couture

SUMMARY

“Multimodal Epistemologies”, edited by Arianna Maiorani and Christine
Christie, is a new volume in the “Routledge Studies in Multimodality” series
edited by Kay L. O’Halloran. The book aims to create “a new, comprehensive,
more flexible and adaptable epistemology” (p. 1) for discourse analysts and
scholars in communication studies to respond to fresh challenges of new media
and technologies. This volume explores diverse frontier issues and presents
new analytical frameworks and methods in the study of multimodal discourse,
that is, discourse that combines more than one semiotic mode (including but
not limited to written language, speech, gesture, image, colour, music, sound,
and typography) to make meaning. It is divided into three sections (16
chapters) apart from an introduction and a conclusion. The three main sections
are based on “three ways of interpreting multimodality as an approach to
knowledge: (1) multimodality as a semiotic perspective; (2) multimodality as a
tool for cultural research; and (3) multimodality as a way to analyse
contemporary narrative processes” (pp. 1-2).

Section 1, “Multimodality as a Semiotic Perspective,” presents the first six
chapters, in which different approaches and methods are drawn upon to
investigate multimodal semiotic discourse. In Chapter 1, “An Eye-Tracking
Account of Reference Points, Cognitive Affordance and Multimodal Metaphors,”
Luna Bergh and Tanya Beelders measure 21 participants’ movement of eye and
duration of fixation when they view 30 print advertisements to test their
perception of reference points in multimodal texts, illustrating the
importance of conceptual archetypes, reference points, mental spaces, and
blending in viewers’ understanding of multimodal metaphors. Chapter 2,
“Demotivators as Deprecating and Phatic Multimodal Communicative Acts” by
Krzysztof Ozga integrates multimodal, semantic and pragmatic analyses to
explore demotivators. A demotivator is a combination of a picture or photo
with a caption that tends to ironically comment on the picture or photo. The
author analyses the socio-communicative functions of demotivators and the
visual-verbal relations in demotivators. In Chapter 3, “Legitimation in
Multimodal Material Ensembles,” Giulio Pagani combines the multimodal and
material analysis framework with the critical discourse analysis of
legitimation (Van Leeuwen, 2007) to see how Norwich Bus Station is represented
and legitimized, considering the social functions of the Station. The analysis
covers the material items and forms of the building itself as well as the
documentary texts recording the process of its construction. The congruity
theory, which approaches the meaning of a text in terms of its pragmatic
functions, is adapted in Chapter 4, “A Pragma-Semiotic Analysis of
Advertisements as Multimodal Texts: A Case Study” by Sabrina Mazzali-Lurati
and Chiara Pollaroli, to do a multimodal analysis of a Nike Air print
advertisement. The paper investigates the rhetorical strategies this
advertisement uses to persuade consumers and analyses the language, images and
layout of the advertisement. Chapter 5, “Analysing Pictures: A
Systemic-Functional Semiotic Model for Drawing” by Howard Riley, takes
Halliday’s systemic functional approach and analyses how different resources
for making meaning are chosen in drawing and fine arts to represent things,
communicate with viewers and organize compositional elements. In Chapter 6
“Multimodal Advertisement as a Genre within a Historical Context,” Sonja Starc
merges the text pattern theory (Hoey, 2001), social semiotic visual grammar
(Kress and Van Leeuwen, 2006) and systemic functional Appraisal Theory (Martin
and White, 2005) to probe into the advertisements in the early 20th century
Slovene newspapers. The focus is on the interpersonal function of the
advertisements, and their text structures as culturally shaped patterns.

Section 2, “Multimodality as a Tool for Cultural Research”, features five
articles that discuss various cultural issues. In Chapter 7, “A Multimodal
Analysis of the Metonymic Indexing of Power Relations in Novel and Film,”
Christine Christie draws upon the Relevance Theory (Sperber and Wilson, 1995)
and pragmatic theorisation of indexicality to do a multimodal analysis of
power relations between characters in Kazuo Ishiguro’s fiction ‘The Remains of
the Day’ and the film adapted from it. The author particularly illustrates the
role of “banter,” a phenomenon of mock impoliteness that involves using acts
of impoliteness to index intimacy and equality in communication. Chapter 8,
“Re-Bombing in Memento: Traumata of Coventry, Belgrade and Dresden in
Multimodal Collective Memory” by Jan Krasni, looks into the public memories of
the bombing of three cities in different countries during the Second World War
through multimodal analysis of websites in memory of these bombing events,
combining the frameworks of Key Visuals (Kramer and Ludes, 2010) and
Multimodal Discourse Analysis (Kress and Van Leeuwen, 2006). According to this
article, collective remembrance serves as a common ground for identity
construction in a given community. Chapter 9, “Argumentation, Persuasion and
Manipulation on Revisionist Websites: A Multimodal Rhetorical Analysis” by
Michael Rinn, takes a rhetorical approach to explore the multimodal
revisionist websites that try to deny the Holocaust and restore the Nazi
ideology. These websites use such strategies as argumentative manipulation,
polyphonic communication (using pseudo-diversity of opinions to stir up
controversy), and effective infotainment to deny the Holocaust and change the
audience’s worldviews. Chapter 10, “A Corpus Approach to Semantic
Transformations in Multisemiotic Texts” by Aleksandar Trklja, investigates the
semantic changes of 500 modified versions of the American recruitment poster
“I want you for U.S. Army” in terms of experiential, interpersonal and textual
meanings. The author tries to provide a new approach for analyzing meaning
transformations in multimodal texts. Chapter 11, “Multimodality and
Illustrations: A Comparative Study of the English and Italian Illustrated
First Editions of The Jungle Book by Rudyard Kipling” is written by Monica
Turci. Using Kress and Van Leeuwen’s (2006) model of multimodal analysis, the
author identifies the similarities and differences between two illustrations
separately from the “Multimodal Source Text” (the English illustrated edition)
and the “Multimodal Target Text” (the Italian illustrated edition) in the
aspects of image composition, ideational meaning, and visual-verbal
relationship. Then she connects the domestication of illustrations in the
target text to historical and political contexts in Italy of the time when the
translated text was published.

The final five chapters in Section 3, “Multimodality as a Way to Analyse
Contemporary Narrative Processes,” study the narrative processes as well as
textual cohesion and coherence in multimodal discourse. Using English-Italian
parallel filmic dialogues as a corpus, Maria Freddi and Chiara Malagori in
Chapter 12, “Discourse Markers in Audiovisual Translation,” investigate how
pragmatic markers, including “well”, “so”, “now”, “you know/see”,
“look/listen” and “I mean”, in English films are translated into Italian and
what roles these pragmatic markers play. The contribution finds that
translation by omission is a regular pattern in dealing with pragmatic markers
in film dubbing. In Chapter 13, “Filmic Narrative Sequences as Multimodal
Environments: A New Perspective on the Effects of Dubbing,” Arianna Maiorani
takes systemic functional and Multimodal Discourse Analysis approaches to make
a corpus-based comparative study of English filmic dialogues and their Italian
dubbed versions, focusing on the space in film as a semiotic dimension.
Transitivity patterns and perspective shifts are analysed and compared in the
source text and target text to see whether the Italian audience is in fact
watching a different sequence. In Chapter 14, “Multimodal Analysis of the
Textual Function in Children’s Face-to-Face Classroom Interaction,” Roberta
Taylor investigates the pupil-to-pupil conversations in a Year Five class in a
British primary school. The pupils make use of different modes such as speech,
gesture, and noise to make cohesive and coherent meaning through the
strategies of repetition, omission, reference, intertextuality and so on.
Chapter 15, “The Contribution of Language to Multimodal Storytelling in
Commercials” by Sabine Wahl, is a multimodal investigation of a “mini-drama”
television commercial, in which the product (a Mercedes-Benz engine) is
promoted through a story. It first presents how the story is told without
written language or speech but using the modes such as image, music, and sound
effects. Then the author analyses the role played by the on-screen language.
The language does not contribute to the storytelling but is indispensable for
the advertisement, as it identifies the product by its name, for example.
Finally taking a filmic extract as example, Janina Wildfeuer in Chapter 16,
“Coherence in Film: Analysing the Logical Form of Multimodal Discourse,”
analyses the process of filmic comprehension and interpretation based on
textual qualities and structure on the one hand and knowledge of the world and
film conventions on the other hand. The interpretation process is illustrated
combining the formal, logical analytical framework (Asher and Lascarides,
2003) and multimodal film analysis approach (e.g. Bateman and Schmidt, 2011).

EVALUATION

This volume enriches the approaches and analytical methods in the study of
multimodality and multimodal communication by presenting studies with both
practical analysis and theoretical considerations. These chapters draw upon a
variety of theoretical approaches (semiotics, discourse analysis, semantics,
pragmatics, rhetoric, corpus linguistics, systemic functional linguistics,
cognitive linguistics, film studies, and so on) to explore multimodal
discourse in different areas (advertisements, websites, social media, films,
architecture, drawing, literature, translation, face-to-face interaction, and
so forth). Although the book is divided into three parts, the titles of the
sections do not cover all the theoretical aspects of these chapters. For
instance,  Chapter 1 is included in the section on “Multimodality as a
Semiotic Perspective”, yet the experiment and analysis in the chapter adopt a
cognitive rather than semiotic perspective.

Indeed the book covers a wide range of fields of research. On the one hand, it
enriches the studies of multimodality from the perspective of social
semiotics, which is one significant theoretical foundation for multimodal
analysis (Van Leeuwen, 2005; Jewitt, 2009; Kress, 2010). On the other hand,
the book promotes the application of multimodal analysis in several new
fields. Firstly, it continues the tradition of the multimodal approach to
literature and narrative, which could be traced back to Page (2010). Secondly,
it features the multimodal analysis of translation, including both traditional
literary translation (Chapter 11), and audiovisual translation and film
dubbing (Chapter 12 and Chapter 13). As audiovisual discourse involves
multiple semiotic modes such as written language, speech, image, colour,
sound, and music, it can be said that the translation of audiovisual “texts”
provides a ready-made sample for cross-cultural multimodal discourse analysis.

This volume is innovative not only in terms of the research fields that it
covers, but also in terms of integrated theoretical frameworks. The
combination of multimodal analysis with critical discourse or cultural studies
is displayed in Chapter 3, Chapter 6, and Section 2 “Multimodality as a Tool
for Cultural Research” of this book. This theoretical integration responds to
Van Leeuwen’s (2013) call for the “critical analysis of multimodal discourse”
and Djonov and Zhao’s (2014) critical first step in establishing “critical
multimodal discourse analysis” as a distinctive branch of discourse analysis.
Particularly Chapter 3, “Legitimation in Multimodal Material Ensembles”, meets
the trend of a multimodal legitimation (e.g. Van Leeuwen, 2014).

Another theoretical innovation of this volume lies in the editors’ effort in
combining multimodality and pragmatics. This tendency is obviously
demonstrated in Chapter 2, Chapter 4, Chapter 7 and Chapter 12. Christie
(Chapter 7) remarks that the chapter also aims to “initiate a debate about
what pragmatic approaches in general can bring to the analysis of multimodal
texts” (p. 111). It seems, however, that this goal of combining pragmatics in
general and multimodality has not been reached so far. Nevertheless, the
debate promoted in this volume could prove beneficial to the combination of
the two approaches.

In conclusion, with theoretical and methodological innovations, this edited
volume offers its reader a range of recent studies in multimodal discourses of
diverse kinds. These studies renew our knowledge, or in the editors’ term,
“epistemologies” in multimodality and the rules played by multimodality in
contemporary communication.

REFERENCES

Asher, N. and Lascarides, A. (2003). Logics of conversation. Cambridge:
Cambridge University Press.

Bateman, J. A. and Schmidt, K-H. (2011). Multimodal film analysis: How films
mean. London and New York: Routledge.

Djonov, E., and Zhao, S. (Eds.). (2014). Critical multimodal studies of
popular discourse. New York: Routledge.

Hoey, M. (2001). Textual interaction: An introduction to written discourse.
London and New York: Routledge.

Jewitt, C. (Ed.). (2009). The Routledge handbook of multimodal analysis.
London and New York: Routledge.

Martin, J. R. and White, P. R. R. (2005). The language of evaluation:
Appraisal in English. Basingstoke and New York: Palgrave MacMillan.

Kramer, S. and Ludes, P. (2010). Networks of culture: The world language of
key visuals. Berlin: Li.

Kress, G. (2010). Multimodality: A social semiotic approach to contemporary
communication. London: Routledge.

Kress, G. and Van Leeuwen, T. (2006[1996]). Reading images: The grammar of
visual design. London and New York: Routledge.

Page, R. (Ed.). (2010). New perspectives on narrative and multimodality. New
York: Routledge.

Sperber, D. and Wilson, D. (1995[1986]). Relevance: Communication and
cognition. Oxford: Blackwell.

Van Leeuwen, T. (2005). Introducing social semiotics. New York: Routledge.

Van Leeuwen, T. (2007). Legitimation in discourse and communication. Discourse
and Communication 1(1): 91-112.

Van Leeuwen, T. (2013). Critical analysis of multimodal discourse. In C.
Chapelle (Ed.), Encyclopedia of applied linguistics (pp. 1-5). Oxford:
Wiley-Blackwell.

Van Leeuwen, T. (2014). Multimodality and legitimation. Plenary speech
delivered at the 2014 Australian Systemic Functional Linguistics Association
Conference, University of New South Wales, Sydney.

ABOUT THE REVIEWER

Kunkun Zhang is a PhD candidate at Macquarie University, Sydney, Australia.
His research interests include multimodality, (multimodal) discourse analysis,
(social) semiotics, literary stylistics, and systemic functional linguistics.
Currently he has been engaged in exploring children’s literature across media,
particularly the relations between narrative, media, multimodality, and
literacies.

Get Adobe Flash player