Language Awareness

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rmla20

Multimodality in translation: a look into EFL and

JSL classrooms

Eiko Gyogi & Vivian Lee

To cite this article: Eiko Gyogi & Vivian Lee (2022) Multimodality in translation: a look into EFL
and JSL classrooms, Language Awareness, 31:2, 232-249, DOI: 10.1080/09658416.2021.2023557

To link to this article: https://doi.org/10.1080/09658416.2021.2023557

Published online: 27 Apr 2022.

Submit your article to this journal

Article views: 94

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rmla20
Language Awareness
2022, VOL. 31, NO. 2, 232–249
https://doi.org/10.1080/09658416.2021.2023557

Multimodality in translation: a look into EFL and JSL

classrooms
Eiko Gyogia,b and Vivian Leec
a
Akita International University, Akita, Japan; bOkinawa University, Okinawa, Japan; cHankuk University of
Foreign Studies, Seoul, South Korea

ABSTRACT ARTICLE HISTORY

There has been an increased interest in multimodality in the field of Received 19 March 2019
both language pedagogy and translation studies. To our knowledge, Accepted 21 December 2021
however, only a few empirical studies have investigated students’ KEYWORDS
perceptions and learning through translation classrooms focusing on Multimodality; Korean;
multimodality. This paper reports on multimodal translation tasks Japanese; translation in
implemented in two different classrooms: a Korean-to-English transla- language teaching
tion classroom in Korea (English as a foreign language (EFL) classroom)
and a Japanese-English translation classroom in Japan (Japanese as a
second language (JSL) classroom). In both classrooms, the students
were assigned a task translating comics. In addition to the text itself,
students also discussed other semiotics that appear in the comics,
including images, font size, and colour, and made translation decisions
accordingly. Based on the analysis of students’ learning journals, this
paper discusses a number of pedagogical benefits and challenges
derived from a multimodal translation classroom. In doing so, it high-
lights the importance of raising students’ awareness of different semi-
otics that construct text.

Introduction
Recent years have seen an increased interest in the role that non-verbal semiotics play in
written and spoken texts. Increased studies in the fields of both language pedagogy (Cope
& Kalantzis, 2015; Royce, 2002; The New London Group, 1996) and translation studies
(Damaskinidis, 2016; Kußmaul, 2005; Lee, 2013; Li, 2019; Risku & Pircher, 2008; Torresi, 2008)
emphasise language is but one of the resources to make meaning and also the importance
of taking into account different non-verbal signs that contribute to the construction of the
meanings of the text. However, there are still an insufficient number of empirical studies
that look at students’ perceptions. Students’ perceptions are important in order to under-
stand whether and how explicit teaching of multimodality can raise students’ awareness of
multimodality. Thus, this paper attempts to explicitly draw students’ attention to multimo-
dality in two different contexts: a Korean-to-English translation classroom in Korea (English
as a foreign language (EFL) classroom) and a Japanese-English translation classroom in Japan
(Japanese as a second language (JSL) classroom). Based on the findings, it will argue the

CONTACT Eiko Gyogi [email protected]

© 2022 Informa UK Limited, trading as Taylor & Francis Group
 Language Awareness 233

pedagogical benefits and challenges of applying a social semiotic approach in the classroom.
The ‘translation classrooms’ focused on in this study refers to classrooms which incorporate
translation activities. In this study, the term ‘mode’ means the semiotic channel to be used
in making a text and ‘multimodal text’ refers to any text of which ‘meanings are realized
through more than one semiotic mode’ (Kress & van Leeuwen, 2006, p. 177).

A social semiotic approach

There has been an increased attention to multimodality, which is evident in related research
and literature. Kress (2000, p. 337) argues that it is no longer possible to ‘understand language
and its uses without understanding the effect of all modes of communication that are copre-
sent in any text’. Meaning is communicated in various forms other than languages, such as
pictorial images, gesture, posture, gaze, and colour (Kress & van Leeuwen, 2001, 2006). All
communication is multimodal, and any communication necessitates the mobilisation of
multiple modes. A multimodal approach means ‘looking at language as it is nestled and
embedded within a wider social semiotic rather than a decision to “side-line” language’
(Jewitt, 2008, p. 357). Dividing a text into its several dimensions (such as verbal, pictorial and
graphic aspects) is an artificial procedure as all components contribute to the construction
of meaning (Torresi, 2008). Jewitt et al. (2016, p. 3) gives three key premises for multimodality
as follows:

1. Meaning is made with different semiotic resources, each offering distinct potentialities
and limitations.
2. Meaning making involves the production of multimodal wholes.
3. If we want to study meaning, we need to attend to all semiotic resources being used
to make a complete whole.

Although there are different disciplines and fields for approaching multimodality (see
Jewitt et al., 2016 for details), the social semiotics approach (Kress, 2009) emphasizes three
important principles of sign-making. First, signs are motivated conjunctions of form and
meaning. Signs are never arbitrary, and there is a motivation on the part of the sign maker
and the context in which the sign is produced (Kress, 2009). Thus, secondly, there is always
agency of the sign maker. The sign maker refers not only to the producer, but also the inter-
preter of a sign who is viewed as engaging in a remaking of a sign. Thirdly, these sign makers
use culturally available semiotic resources in a given situation. Each resource offers different
potentials for meaning making, and when making signs, a person brings together and con-
nects available resources that are most appropriate to express meanings they wish to create.
A social semiotic approach views the process of text production as central to meaning
making, and it attempts to understand and describe how people choose semiotic resources
in order to achieve particular social purposes (Kress, 2009). These semiotic resources are
shaped through their social usage, and it is impossible to separate these resources from its
context of use. Social semiotic analysis involves detailed analysis of artifacts, which is viewed
as a residue of a sign maker’s motivation. It analyses artifacts based on information on the
context in which they were produced. Rather than being limited to visual texts, social semi-
otic research expanded its scope to other modes as well, including sound, multimedia, space,
and architecture (Kress & van Leeuwen, 2001).
234 E. GYOGI AND V. LEE

Multimodality in the language and translation classrooms

Research on multimodality has influenced language learning practices in many important
ways. The New London group introduces the concept of multiliteracies, where they advocate
the need to broaden the understanding of literacy to include a multiplicity of discourses,
including the multimodal nature of the text.
A growing number of studies attempt to advance the discussion of multimodality in the
language classroom (Early et al., 2015; Lotherington & Jenson, 2011; Royce, 2002; Stein, 2000).
For example, Royce (2002) argues the importance of developing pedagogical resources that
enable students to understand what non-verbal modes are trying to say and how they
interrelate with linguistic aspects of the meaning. Royce gives methodological suggestions
for reading, writing, speaking and listening, and learning vocabulary, respectively. For reading
activities, he suggests asking questions about visuals, focusing on what information the
visuals are intended to convey to the viewers. Yi and Angay-Crowder (2016) have examined
how multimodal literacy practices can be integrated into teacher education, revealing a
number of challenges such as epistemological issues in what are considered as necessary
knowledge and some teachers’ resistance to multimodal practices.
Increased studies have applied the pedagogy of multiliteracies in different language
learning contexts (Amgott, 2020; Cloonan, 2010; Cope & Kalantzis, 2015; Kumagai et al., 2015;
Mills, 2010; Schmerbeck & Lucht, 2017). Recognising the importance of an array of resources
including gesture, sounds, images, textures, and silences, Stein (2000) introduces a mini-re-
search project implemented to undergraduate ESL students. For this project, each pair of
students presented photographs they took in the home or the workplace in the form of an
exhibition poster with written captions and prepared a written explanation on A4 paper. In
Amgott’s study (2020), 12 American undergraduates studying French recorded their reflec-
tions in the form of video blogs (vlogs) and Flipgrid videos during their six-week French
language and culture study abroad programme. The analysis of students’ reflections shows
how these activities helped students to leverage multiple modes to express their linguistic,
cultural, and emotional experiences.
In translation pedagogy, the term ‘translation’ is defined as linguistic-only practice and
translation classrooms tend to focus on linguistic aspects of the text (Torresi, 2008). Texts
that necessitate students to examine non-linguistic aspects (such as multimedia translation)
are treated as highly specialized ones (Torresi, 2008). However, most texts are multimodal,
rather than mono-modal and, as with research in the field of language pedagogy, there has
been a growing interest in the role of the non-textual elements in translation studies
(Damaskinidis, 2016; Kußmaul, 2005; Lee, 2013; Li, 2019; Risku & Pircher, 2008; Torresi, 2008).
Focusing on the translation of advertising, Torresi (2008, p. 70) argues that ‘a translator who
is aware of the importance of nonverbal elements, and the resource they represent for trans-
lation, proves a more reliable team-worker and produces better target texts’. Prieto Velasco
et al. (2007) show an example of the translation classroom where students translated scien-
tific multimodal texts. They were asked to think aloud and record their description of the
denotative and connotative aspects of visual materials at home and record it. Tercedor
Sánchez et al. (2009) also report on several pedagogical activities that can be introduced to
enhance students’ awareness of the importance of visual aspects of the text. For example,
in order to understand the image-text interface, students were given questions to analyse
the text such as ‘What aspect of a text/concept does the image bring about?’ and ‘does the
 Language Awareness 235

image rely on the text and vice versa?’ (Tercedor Sánchez et al., 2009). These studies show
the importance of other modes in the meaning-making process and also propose some
informative classroom activities.

Comics
In particular, researchers recognise manga or comics as a pedagogical tool to increase stu-
dents’ awareness of multimodality (Huang & Archer, 2006; Schwartz & Rubinstein-Ávila, 2006).
Since manga is commonly associated with Japanese comic books or graphic novels, this
paper uses the term ‘comics’ to include a wide range of visual and narrative work, including
Japanese manga, comic books and web cartoons. In comics, the visual images play a central
role and the verbal text is in a complementary role while they interact with each other. The
salience of visual images will help students draw attention to not only the verbal text, but
also the various meaning-making resources that create the text and how they interact with
each other to convey the communicative goals. Murakami and Bryce (2009) consider manga
as an educational medium, and point out that manga has the effect of making difficult topics
easy to understand due to the hybridity of the manga text, which consists of both visual and
linguistic texts. Schwartz and Rubinstein-Ávila (2006) highlight how manga require a unique
multimodal form of reading and suggest the potential of manga in developing students’
analytical and critical readings of visual texts. Based on their multimodal analysis of manga,
Huang and Archer (2006) highlight how manga can be used in the classroom to understand
how different social groups use semiotic resources to create culturally-specific ways of con-
veying meaning.
While existing research on multimodality and comics offer precious insight into how
multimodal teaching activities can be integrated in the classrooms, to our knowledge there
are few empirical studies that evaluate the effectiveness of such activities from the per-
spective of students. Students’ perceptions are considered important to understand
whether and how pedagogical interventions lead to raising students’ awareness of
multimodality.
In order to fill this gap, this study examined students’ reflections after classes which intro-
duced multimodal aspects of text, with the following research questions:

1. What are students’ perceptions and reflections towards the multimodal nature of
comics in two classroom contexts?
2. Does explicit attention to the multimodal nature of comics used for translation prac-
tice in the classroom help students reflect upon such features?
3. What mode categories attract students’ attention the most in translation
assignments?

Furthermore, in order to understand how such pedagogical intervention is effective across

contexts, this study will draw upon data from two classrooms in different contexts. These
classrooms were chosen as these are contexts that the researchers were familiar with,
enabling us to have an in-depth understanding of the data. By introducing two case studies,
it aims at understanding the similarities that can be identified in two contexts in order to
give pedagogical implications of teaching multimodality from multiple perspectives.
236 E. GYOGI AND V. LEE

Methodology
Summary of participants
The current research is qualitative and classroom-based, and the data was collected from
two different classrooms: a Korean-to-English translation classroom at a university in Seoul
and a Japanese-English translation classroom at a university in Akita, Japan. Seventeen (aged
20–24 years, average age = 21.5, SD = 1.1) and 20 undergraduate (aged 19–22 years, average
age = 20.6, SD = 1.1) participated in a class in Seoul and that in Akita, respectively. For the
class in Seoul, the students were all L1 users of Korean. Their level of English was intermediate
and above. For the class in Akita, the students were mostly exchange students with different
backgrounds. They had intermediate level of Japanese proficiency.
Both classrooms were similar in terms of the students’ proficiency level (i.e. intermedi-
ate-level in English (Seoul) or Japanese (Akita)) and the number of students. However, they
were different in terms of the students’ backgrounds: While the participants in class in
Seoul were regular students, those in the one in Akita were short-term exchange students
from different universities.
The translation classroom in Seoul was a class from a regular 16-week long Korean-to-
English translation course which focused on translation practice, and typical activities over
the semester included translation practice, peer editing and review of translation work, as
well as discussions about one’s own translation or that of peers. In Akita, the class was an
extracurricular 5-week long Japanese-to-English/English-to-Japanese translation course, and
typical activities included translation practice and discussions about one’s own translation
or that of peers. Both classrooms used comics as source text material for translation.

Summary of classes
Table 1 shows a summary of both classrooms.
In both classrooms, the students were given the prompts to consider non-verbal aspects
of the text when they translated the text in class. Drawing on a social semiotic approach,
these prompts were developed to make students explicitly think about their agency as a
sign maker of the text. Through questions such as ‘which font will you use?’, the students
were asked to reflect upon and discuss how they make use of different potential for different
modes when making meanings. The students were also given opportunities to discuss this
feature during class. To our knowledge, all students in both classrooms did not take any
multimodal-focused classrooms before class.

Class in Seoul
The class was a regular course for students majoring in translation and interpreting. The
semester was 16-weeks in duration, and class contact time was 2 hours per week. Typically,
three weeks were spent on each new source text/topic. The current study looked at the
source text students worked on during Weeks 5–8 of the semester.
The source text for the Seoul classroom was from a web comic series called Pojatoon,
about an owner and her pet cats. The students were asked to translate a part of this, and
had the following translation brief provided:
 Language Awareness 237

Table 1. Summary of class in Seoul and Akita.

Class in Seoul Class in Akita
Text for translation Web comic Manga & web comic
Class Regular 16-week long translation class Extra-curricular 5-week long translation
class
Duration 2 hours per week (Weeks 5-8) 75 minutes per week (2nd week)
Task (assignment) You have been asked to translate the web comic You read Gyagu manga biyori. You’ve
for an overseas web comic site, which has decided to translate a part of it and
web comics from various different cultures. introduce it on your own web comic
site (after obtaining copyright
permission).
Discussion questions • How will you arrange the layout of the • Which font type/size would you use?
translated text? • Will you use vertical or horizontal
• What font/font colour/font size will you use? writing?
• Will you keep the font in the same place or • Will you write speech bubbles in the
place it somewhere else? Why? same place or would you change it?
• Will you delete all the original text or leave it
and make the text bilingual? Why?
Data collection Learning journals included in students’ portfolio Learning journals submitted after class

You have been asked to translate the web comic for an overseas web comic site, which has web
comics from various different cultures.

The class started with a pre-translation discussion about the translation brief and text
type. The students were also given the prompts to enable awareness of multimodal factors.

• How will you arrange the layout of the translated text?

• What font/font colour/font size will you use?
• Will you keep the font in the same place or place it somewhere else? Why?
• Will you delete all the original text or leave it and make the text bilingual? Why?

The chapter the students were given for translation was about the cats’ love of eating.
The following are some examples from the source text used for the class:
(1) ayu pobi manimeokossda

Oh Pobi you’ve already eaten a lot.

ije keuman meokja. kkeut.

Let’s stop now. That’s it.

meogi gonggeubisiwonchangeona

If the food supply (?) is mediocre

Seongeanchmyeon…

or not satisfying…
This chapter of the web comic focuses on how much the cats, especially one cat named
Pobi, loves eating. The above exchange is from a clip where Pobi is not happy with the food
supply and is requesting more from the owner.
238 E. GYOGI AND V. LEE

(2) nyaang

meow

deojwo

Give me more

ppali

Hurry
This clip shows Pobi looking at the owner with huge, begging eyes in a bid to receive
more food.
While the total source text spanned 21 pages, students were assigned half of the text to
translate, with Group 1 translating pages 1–12 and Group 2 pages 13–22. The webcomic
was given to students in PDF format in its original form. All the students had access to the
whole PDF file, even though they only needed to translate half of the text. Following the
translation, they then held a post-translation discussion to discuss their translations with
peers. They were then to bring their completed first drafts to class the following week, when
student presentations took place. Feedback was given on translations during this session
as well as the third week, which was dedicated to peer editing and feedback and editing for
a final draft which was to be submitted as a part of their graded homework assignment.

Class in Akita
With the exception of one student, all students were on short-term exchange programmes
from 18 different universities, including 8 students from the USA, 4 students from Taiwan,
2 students from Russia, 2 students from Australia and 1 student each from Romania, Estonia
and New Zealand. English is not the strongest language for half of the students. At the time
of study, all students except one took one of the three intermediate-level Japanese class
four times a week for 4.5 hours.
The class is a part of the project that examines the use of translation in the language
classroom. The students attended a series of the five translation classrooms outside the
regular class, and this session was the second session. The sessions were not counted towards
the grades of their regular classes.
The class started with the introduction of language style, or role language, used in
manga. After that, students translated a one-page web-comic from English to Japanese as
an in-class activity, with the following translation brief:
You made a web comic website to increase web comic fans. You decided to translate interesting
comic.

In the class, students were asked to discuss the questions related to non-verbal aspects
of the text, together with other questions related to the speech styles of the characters:

• Which font type/size will you use?

• Will you use vertical or horizontal writing?
• Will you write speech bubbles in the same place or will you change it?
 Language Awareness 239

After discussing this, the students tentatively translated the comic and discussed it in
pairs and then in the whole class. After that, the students were given a copy of a chapter of
Gyagu manga biyori (Good Day for Gag Manga) (Masuda, 2004) and were asked to translate
a part of this manga as an assignment. This is a comic manga and known for its hyperactive,
random and nonsensical style. The students were asked to translate two panels of this manga
with the following translation brief:
You have read Gyagu manga biyori. You’ve decided to translate a part of it and introduce it on
your own web comic site (after obtaining copyright permission).

In this manga, Nakanishi and his junior, and an Indian guide, Govinda, were lost in Amazon.
This was a scene where one of Nakanishi’s junior was caught by a big snake.
(3) 1st panel

Govinda: Jiizasu! Hayaaku tasuke nai to shinde shimaaaaai maaaasu.

Jesus! He will die if we don’t help him.

Nakanishi: Dooshite ima goro nihongo heta ni naru n dayo!

Why did you Japanese become bad now?

2nd panel

Nakanishi: Shikashi kora erai koccha de…

But this is a problem…

Majika de miru daija te kowasugi ya de. Te e dase hen wa. Akima hen wa.

The big snake around the corner is too scary. I can’t touch it. I cannot do anything.

Junior: Dooshite kyuuni kansaiben nan su ka. Sempai hayaku tasuketee.

Why do you suddenly speak in Kansai dialect? Senior, please help me quickly!

In the first scene, Govinda suddenly speaks in broken Japanese, which was shown by his
use of katakana, one of three characters used in the Japanese writing system, which is often
used to transcribe foreign words. The senior’s utterance was shown in bold letters in a dif-
ferent font in order to show his emphasis. In the second panel, Nakanishi suddenly switches
to Kansai dialect. The junior, who was caught by the snake, cries for help. His urgency was
illustrated with the use of different font in bold. The change of the style created a comical
effect to the scene. The students were given the original manga with blank bubbles as a
template in a Word file, and were asked to submit this template with their translations. The
teacher did not provide any specifications as to font, layouts, etc. to be used.
Feedback was given at the beginning of the following session. As this class was an
extra-curricular class, students’ assignments were not not assessed and not counted towards
their grades in the regular courses.

Data collection and analysis

We used the learning journal which students submitted after class as data. A learning
journal (or also called learning diary or log) is ‘an accumulation of material that is mainly
240 E. GYOGI AND V. LEE

based on the writer’s processes of reflection’ (Moon, 1999, p. 4). It has been used in both
the translation classrooms (Fox, 2000; V. Lee, 2015) and language teaching classrooms
(Cross, 2010; V. Lee, 2018; Pearson-Evans, 2006; Simard, 2004) for both education and
research purposes. While it has limitations in regard to their reliability as a source of data
(i.e. it assumes that students write their thoughts fully and honestly), it nonetheless plays
a useful role in helping the teacher and researcher to look into students’ perspectives
(McKay, 2009).
For the Seoul classroom, the learning journal was included in students’ portfolios, which
was a semester-long assignment they kept and which contained images or scans of their
in-class work and final translation drafts. There was no requirement for word count for entries.
A new entry was added and the portfolio was submitted after each topic/text type was
finalised. Three weeks were typically spent on one topic/text, which consisted of first draft,
peer review and editing of a second draft, and then finally self-editing and producing of a
final draft.
For the Akita classroom, the learning journal was submitted after each class. In the learning
journal, the students were asked to write a commentary on their translation (i.e. how they
translated the assignment and the reasons behind their choice) and what they thought
about the class in English, Japanese, or mixture of both. Although the class discussion
included questions on multimodality, the teacher did not ask students to specifically mention
or comment on the multi-modal nature of the text in their learning journal. There was no
requirement for the length of the journal, and the average length of the learning journal
was about half an A4 page.
Students’ entries were coded, using NVivo 12, based on the following different mode
categories proposed by Cope and Kalantzis (2009, p. 362), consisting of linguistic (i.e. written
and oral language), visual (e.g. image, sculpture, and craft), audio (e.g. music, ambient
sounds, noises, and alerts), tactile (e.g. touch, smell, and taste), gestural (e.g. movements
of the hands and arms and expressions of the face), and spatial (e.g. proximity, spacing, and
layout). After coding students’ description based on which aspect of ‘modes’ they were
talking about, the researcher checked each code again to examine the consistency of the
coding criteria.

Findings
This section will discuss the findings to Research Question 1, that is, students’ perceptions
and reflections towards the multimodal nature of comics in two classroom contexts. Overall,
17 journal entries (average word count = 416, SD = 85.0) were collected for the Seoul class-
room and 20 journal entries (average word count = 456, SD = 152.1) for the Akita classroom.
For both classrooms, findings were grouped into the categories Linguistic, Visual and Spatial.
No gestural, audio, and tactile dimensions were found, due to the nature of the text and task
(i.e. translating a web comic or comic manga).

Class in Seoul
Students in the Seoul classroom showed considerations relating to all three categories;
such considerations will be discussed in each relevant section below.
 Language Awareness 241

Linguistic
For the coding related to the node ‘Linguistic’, eight students in the Seoul classroom mention
linguistic features of the web comic explicitly in the journal part of their portfolios. The
considerations include dialect and onomatopoeia use, formality, tone and word choice.
For example, Student 1 mentions the following in his entry:
(4) “Every time Ma starts frying, Poby turns super-excited” got changed into “the smell of mum’s
fritter always makes Poby super-excited,” since I thought the second version suits better with
the ST “튀김냄 새.” However, after peer-editing in class, I decided to change “fritter” into “deep-
fries” because it is more accurate.

The excerpt shows how word choice was a factor in the student’s decision-making process.
The student changes the choice of word for the target text following peer-editing, as a result
of discussion and feedback with classmates.
Student 2 mentions the following:
(5) I translated “스윽” into “peek,” since I couldn’t find a perfectly matching word for it in English,
but I received a feedback that it can be omitted. This is because it is obvious that she is peeking
when people look at the illustration.

In the original web comic text, a mimetic word was used. In the target text, the student
decides to not translate this as she could not find a good equivalent in the target text lan-
guage. The student also considers how the illustration shows the cat peeking, and as such
the mimetic word translation is not necessary.
The same student also considers dialect and its translation:
(6) I translated “오셨어예” into “you’re back,” since I learned in class that it is not necessary to
translate dialects. This is because if I use certain English dialect for the TT, the tone of it might
change too much. Moreover, since “오셨어예” is just a way of expressing acknowledgement
that he’s back, I didn’t translate it into a greeting (for instance, good evening).

The student thinks about how the use of a dialect may alter the original tone of the source
text too much, and opts to not use a dialect for the translation.
Student 3 considers the word repetition on the source text, and how to translate this to
convey a similar feeling in the target text:
(7) The subtitle “자꾸자꾸 먹고싶어요” contains its own feelings. There is a special intention of
writer to repeat the word “자꾸” rather than just saying it “자꾸 먹고싶어요.” It is to make readers
feel intimate and cute towards the web comic. At first, I translated this as “Wanna keep eating”
but I thought this is not enough. But by listening to group presentation, I found a good alter-
native that is “can’t stop eating, again and again.” I believe this one is better because by repeat-
ing certain word (again) as the source text does, it delivers similar effect of it.

Data from the linguistic nodes demonstrate students’ considerations relating to lan-
guage-based aspects, such as word choice, nuance, dialect and onomatopoeia.

Visual
Eight out of 17 students mention considerations relating to ‘visual’ aspects. These include
font size, font colour and font type. Also, one student commented on her decision to keep
the original Korean text next to the English translation.
242 E. GYOGI AND V. LEE

This is an excerpt from Student 4’s data:

(8) I want to use ‘comic sans MS’ or ‘bradley hand ITC’ fonts because they look most similar to
the original text font and it’s commonly used in web comics.

As the excerpt shows, this student wanted to choose a font style which looked the most
similar to the one in the ST. This shows an awareness of the visual aspects of font and the
effects it can create.
Student 5 mentions the following:
(9) So, I generally maintain the form and if there is a need for extra explanations for culturally
different elements, I add some comments at the bottom of the space with different font colour.

This excerpt shows how the student is aware of the visual aspects of font, such as the use
of a different font colour for cultural term explanations. Her excerpt shows how she is aware
of the effect different colour font use can have visually for the reader.
As the data excerpts show, explicit attention to the multimodal nature of the texts in
the classroom seems to have a direct effect on students. Students explicitly mention
certain multimodal aspects, such as how they will arrange font and also font style
and colour.

Spatial
For the ‘spatial’ code, seven students in the Seoul classroom mention considerations relating
to layout, text length and especially text length for speech bubbles.
Student 6 comments that:
(10) Layouts and fonts are very important because there are speech bubbles and generally,
English texts are longer than Korean ones.

The speech bubbles and layout encourage this student to think about the difference in
text length, and how to fit the longer English text into the ST.
Student 7 mentions the following about layout in the target text:
(11) I save both Korean and English as there might be foreign readers who have interest in
Korea. I believe that people who have interest in Korea are also included the target readers,
therefore, I saved the language as bilingual. I got hint by the webtoon named ‘Penguin
loves Mev’ in NAVER. It also made the language as bilingual. However, unlike ‘Penguin loves
Mev’, I positioned the whole dialogues in English to stay next to original text to make it
easier to read.

Student 7 decides to keep the text bilingual, as she thinks it would benefit readers who
are interested in Korea (or Korean). She saw this type of text arrangement in another web
comic, which inspired her to make the decision. Student 7 also mentions how she will layout
the text, and decides to insert the English font close to the original Korean text for ease of
reading.
The excerpts in this section also seem to demonstrate how explicit attention to the
multimodal nature of the texts encouraged students to make decisions regarding spatial
aspects, such as speech bubble space and text length and positioning of the target text
translation.
 Language Awareness 243

Class in Akita
Linguistic
Similar to the Seoul classroom, all three categories, linguistic, visual, and spatial were
observed in the class in Akita. However, although the class discussion drew specific attention
to multimodality, their entries were predominantly on linguistic codes, and the students
who explicitly mentioned about other modes (in this case, visual and spatial) were relatively
small in number: eight students and four students each.
For the node ‘Linguistic’, all students in the Akita classroom mention the linguistic features
of the text at least once. Their entries include language used in comics, translation of accent,
foreigner talk, set expressions that are not easily translatable, and humour.
For example, Student 1 mentions the following:

(12) This was actually a lot harder than I expected. I wasn’t sure how to translate ‘bad Japanese’
to English nor how to translate Kansai dialect. Which dialect would correspond to the Kansai
dialect? Should I even try to find a dialect to match it or should I just translate it as I please?
These were thoughts that went through my head as I tried to translate this manga.

Student 2 decided to translate the Kansai accent to a U.S. Southern accent, considering
its distinctive and well-known nature. However, she reflects on this as follows:
(13) When translating, I think it’s important to be aware of what stereotypes you’re bring into
the scene, especially because you don’t want to seem insensitive or offend people. Also when
you’re trying to translate dialects you really have to know the full meaning of the source text to
be able to successfully express it in the other language. I found this part quite difficult.

Student 3 said that there is no dialect equivalent to the Kansai dialect in America, and
decided instead to focus on humour conveyed by dialect in the following way:
(14) These two panels are meant to be incredibly funny in Japanese, but if they’re translated
literally, their humor is lost in the translation. Therefore, if translated literally, the new audience
will be unable to enjoy the media and the translator has failed their purpose. Instead, I chose
to localize the manga, placing more importance on the audience enjoying the text instead of
the audience reading the actual message.

She decided to emphasise religion and prayer, and translated the Kansai dialects part as
‘Dear Lord in Heaven, please do not let this giant snake kill my friend. I am unable to help
him so I pray for his soul to be welcomed into your arms. Amen’.
As their excerpt shows, translation is a challenging task where the students constantly
have to make decisions on each individual choice of word. Since the change of speech styles
produced a humorous effect in this comic, students tried to think about how to convey
this humour.

Visual
Eight out of 20 students in the Akita classroom mention visual aspects of the comics. Their
entries include font size, capitalization and how to match image and language in comics. In
the source text, on the top of different speech styles, two different fonts were used in the
page in question.
244 E. GYOGI AND V. LEE

Students 4 used two different fonts: Comic Sans MS and italicized Times New Roman. She
uses italicized Times New Roman when the protagonists switched their normal speech style
to either the foreigner talk or Kansai dialect.
(15) Another problem was the font. I ended up deciding on using two different fonts to contrast
the different dialects that were used

Her excerpt shows her awareness of how fonts can contribute to the meaning-making
process. Five out of 20 students also changed the font, depending on the tone or speech
style of the protagonists. Furthermore, five students used bold, capitalization or italicized
letters to show urgency. For example, Student 14 translated the junior’s call for help as ‘WHY
you are suddenly speaking Japanese English? HELP MEeeee!!!!!’, capitalizing ‘Why’ and ‘help
me’ for emphasis.
Student 5 also is aware of the typical style of comics, not only in terms of language style
but the use of capitalization.
(16) I decided to write the text for the translation in uppercase, since this is typical for comic
books and I thought it was a little jarring for the shouting parts to be suddenly in uppercase
[…] That exercise made me think about how important font is

Her excerpt shows she also takes into account the norms of the target text in terms of
the visual aspects of the letters.
Student 6 mentions the importance of matching images and language in manga:
(17) Manga is a product of art. Manga artists use images to express their thoughts [to readers]
especially children. So the feeling conveyed through the images should match what characters
say in their speech.

In summary, these students showed awareness of the visual aspects when translating comics.

Spatial
For the Akita classroom, four students out of 20 students mention the place of speech bub-
bles and vertical or horizontal writing.
For example, Student 7 comments on the text length and shows awareness of the differ-
ence in textual characteristics of the Japanese and English languages. While recognizing it
is not a common practice, she spelled ‘Jesus’ vertically, in order to fit her translation to the
speech bubble. However, she reflected on it as follows:
(18) The use of font, positioning, and bolding can also create word emphasis, particularly in
something like a comic or manga. Getting the same feeling into English can be difficult. Some
translated manga have to edit the text bubbles to fit horizontal text for example

Student 3 also mentions the following:

(19) I know of many of the difficulties that comes with font, language size, and speech bubbles.
It was nice to learn it in a formal class setting though.

She was aware of the importance of multiple modes that contribute to meanings (includ-
ing spatial aspects such as speech bubbles), and appreciated the fact it was brought up in
class for discussion.
 Language Awareness 245

These students considered the spatial aspects of manga, considering where they would
place speech bubbles and the use of vertical and horizontal writing.

Discussion
Based on the findings, this section will discuss Research Questions 2 and 3 and suggest
pedagogical implications for incorporating multimodality in translation classrooms.
Regarding Research Question 2, which asks ‘Does explicit attention to the multimodal nature
of comics help students reflect upon such features?’, despite a difference in contexts, this
study shows that reflection of multimodality can be seen in both the Seoul and Akita class-
rooms. Students in both classes considered font type, text, colour and also the option of
keeping the text bilingual. In addition, students also contemplated the layout of the target
text and differences in language characteristics, such as speech bubble text length.
The findings highlight benefits and challenges in incorporating multimodality in trans-
lation classrooms. As the data show, students explicitly commented on the discussion topics
in class. Such results indicate that explicit classroom instructions in the form of prompts
played a role in encouraging students to reflect on multimodal aspects even after class.
This type of intervention (i.e. the form of prompts) is relatively easy to implement and can
be applicable to different contexts. Further, while previous studies report some teachers’
resistance to multimodal practices (Yi & Angay-Crowder, 2016), comics can be used as an
entry point for persuading the importance of multimodality, both for teachers and students.
Due to the visual and entertaining nature of comics, it is an appropriate classroom material
source to use for exploring multimodal aspects from both the student and teacher
perspectives.
However, this study also shows a number of challenges in changing students’ perceptions.
Regarding Research Question 3, that is, what mode categories attract students’ attention
the most, while multimodal aspects were considered in the current study, students still
showed a tendency to concentrate more on comic specific linguistic features such as the
use of dialect, onomatopoeia, tone, manga-specific language, formality and word choice.
In both classrooms, those who mentioned spatial and visual categories were seven out of
17 students (Seoul) and eight out of 20 students (Akita) respectively. Thus, this accounted
for less than half of the students. Furthermore, although multiple modes are considered,
some students (such as Student 3’s second excerpt (excerpt 19) from the Akita classroom)
limited their reflection to the importance of multiple modes only and did not give any com-
ments on how these modes contribute to the meaning-making process or how these factors
interrelate with each other.
The lack of entries on visual or spatial codes does not necessarily mean a lack of students’
consideration on multimodality. The learning journal has limitations in its underlying
assumption that students write what they think honestly and fully (V. Lee & Gyogi, 2016).
However, the results suggest that students are perhaps accustomed to focusing on linguistic
aspects of the text when they translate, possibly because of their previous experiences in
translation and/or language classrooms. At least, it can be said that the students either fail
to take into account other modes or implicitly or explicitly think that these other modes are
not worth mentioning in the learning journal.
246 E. GYOGI AND V. LEE

The findings also suggest that one class intervention may not be sufficient, at least for
some students, to reconceptualize the term ‘translation’ from a text-focused practice to one
that focuses on how multiple modes interact and contribute to the meaning-making process.
The teacher will need to draw constant attention to the multimodal nature of the text in
class in order to enable students to consider its importance. The teacher might also need to
draw more explicit attention to multimodality, in addition to in-class prompts. For example,
the teacher can provide a checklist that includes items about multimodality when assigning
homework. Furthermore, the teacher has to provide feedback that can prompt the students
to further think about the effects and interrelatedness of these multiple modes. As an exam-
ple, the teacher could compare students’ work in two different fonts or layouts, and ask them
how such a difference creates different meanings and how effective they are in conveying
the message.
Here is an example of a checklist which could be provided to students to check for items
related to multimodality in a translation classroom:

• How will you arrange the layout of the translated text?

• What font will you use?
• What font colour will you use?
• What font size will you use?
• Will you keep the font in the same place or place it somewhere else? Why?
• Will you delete all the original text or leave it and make the text bilingual? Why?

This study adds to a growing body of studies that aim to incorporate multimodality in
language or translation classrooms (Early et al., 2015; Lotherington & Jenson, 2011; Prieto
Velasco et al., 2007; Royce, 2002; Stein, 2000; Tercedor Sánchez et al., 2009). In particular, this
study looked at students’ perceptions, which is relatively an under-investigated area of
research, to examine the effectiveness of pedagogical interventions in two different contexts.
The findings highlight the importance and necessity of investigating students’ interaction,
perception and reflection of class translation tasks. As described in this study, multimodal
teaching activities may not necessarily result in raising students’ awareness. In order to eval-
uate the effectiveness of multimodal teaching activities, more studies looking at students’
interaction, perception, and reflection of their learning experiences would be necessary.

Conclusion
This paper looked at an attempt to raise students’ awareness of multimodality with a focus
on comics in two different classrooms: a Korean-to-English translation classroom in Korea
and a Japanese-English translation classroom in Japan. The findings show that the approach
can indeed encourage and elicit reflection on multimodal aspects in both classrooms.
Students in both classes considered multimodal aspects such as font type, text, colour and
also the option of keeping the text bilingual. In addition, students also contemplated the
layout of the target text and differences in language characteristics, such as speech bubble
text length. Although this study focused on EFL and JSL classrooms in South Korea and
Japan, the approach and method can be applied across diverse levels and settings.
 Language Awareness 247

As with other qualitative studies, this study has limitations as to the generalizability of
the results. Also, as mentioned above, the learning journal has issues of students’ honesty
and candidness. Despite these limitations, it is hoped this study shows that reflection of
multimodality can be observed in different classrooms, and serves as a piece of evidence of
the potential and usefulness of the pedagogical method outlined in the study.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
The data collection and analyses of the Japanese language classroom was supported by the
Japan Society for the Promotion of Science (JSPS) KAKENHI under Grant Numbers [17K13486]
and [20K13079].

Notes on contributors
Eiko Gyogi is currently an assistant professor at Akita International University in Akita, Japan and
will start as an associate professor at Okinawa University, Okinawa, Japan in April 2022. She
teaches various undergraduate Japanese language courses. She obtained a PhD in Linguistics
at SOAS, University of London. Her research interests include translation in language teaching,
pragmatics, and heritage language education.
Vivian Lee is an associate professor at Hankuk University of Foreign Studies in Seoul, South
Korea. She teaches various undergraduate and postgraduate English language and Korean-
English translation and interpreting courses. She obtained her PhD in Linguistics at SOAS,
University of London. Her research interests include translation, intercultural communication
and language pedagogy.

ORCID
Eiko Gyogi http://orcid.org/0000-0003-3325-7104

References
Amgott, N. (2020). L2 multimodal composing abroad: Remixing languages, cultures, and identities. L2
Journal, 12(3), 1–22. https://doi.org/10.5070/L20047045
Cloonan, A. (2010). Multiliteracies, multimodality and teacher professional learning. Common Ground.
Cope, B., & Kalantzis, M. (2009). A grammar of multimodality. The International Journal of Learning,
16(2), 361–425.
Cope, B., & Kalantzis, M. (Eds.). (2015). A pedagogy of multiliteracies: Learning by design. Palgrave
Macmillan.
Cross, J. (2010). Raising L2 listeners’ metacognitive awareness: A sociocultural theory perspective.
Language Awareness, 19(4), 281–297. https://doi.org/10.1080/09658416.2010.519033
Damaskinidis, G. (2016). The visual aspect of translation training in multimodal texts. Meta, 61(2),
299–319. https://doi.org/10.7202/1037761ar
Early, M., Kendrick, M., & Potts, D. (2015). Multimodality: Out from the margins of English language
teaching. TESOL Quarterly, 49(3), 447–460. https://doi.org/10.1002/tesq.246
248 E. GYOGI AND V. LEE

Fox, O. (2000). The use of translation diaries in a process-oriented translation teaching methodology.
In C. Schäffner & B. Adab (Eds.), Developing translation competence (pp. 115–130). John Benjamins.
Huang, C., & Archer, A. (2006). Uncovering the multimodal manga and the implications for pedagogy.
In B. T. Williams & A. A. Zenger (Eds.), New media literacies and participatory popular culture across
borders (pp. 44–60). Routledge.
Jewitt, C. (2008). Multimodal discourses across the curriculum. In N. H. Hornberger (Ed.), Encyclopedia
of language and education (pp. 357–367). Springer. https://doi.org/10.1007/978-0-387-30424-3
Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing multimodality. Routledge.
Kress, G. (2000). Multimodality: Challenges to thinking about language. TESOL Quarterly, 34(2), 337–340.
https://doi.org/10.2307/3587959
Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. Routledge.
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary com-
munication. Arnold.
Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). Routledge.
Kumagai, Y., Lopez-Sanchez, A., & Wu, S. (Eds.). (2015). Multiliteracies in world language education.
Routledge.
Kußmaul, P. (2005). Translation through visualization. Meta, 50(2), 378–391. https://doi.org/
10.7202/010943ar
Lee, T.-K. (2013). Performing multimodality: Literary translation, intersemioticity and technology.
Perspectives, 21(2), 241–256. https://doi.org/10.1080/0907676X.2012.693107
Lee, V. (2015). A model for using the reflective learning journal in the postgraduate translation
practice classroom. Perspectives, 23(3), 489–505. https://doi.org/10.1080/0907676X.
2014.948884
Lee, V. (2018). L1 and L2 writing: The learning journal for pedagogy. The Asian Journal of Applied
Linguistics, 5(1), 36–45.
Lee, V., & Gyogi, E. (2016). The reflective learning journal in the classroom. In P. Clements, A. Krause, &
H. Brown (Eds.), Focus on the learner (pp. 270–276). JALT.
Li, S. (2019). A corpus-based multimodal approach to the translation of restaurant menus. Perspectives,
27(1), 1–19. https://doi.org/10.1080/0907676X.2018.1483408
Lotherington, H., & Jenson, J. (2011). Teaching multimodal and digital literacy in L2 settings: New
literacies, new basics, new pedagogies. Annual Review of Applied Linguistics, 31, 226–246.
https://doi.org/10.1017/S0267190511000110
Masuda, K. (2004). Gyagu Manga Biyori 5 Masuda Kōsuke Gekijō [Good days for gag manga 5 Kosuke
Masuda Theater]. Shueisha.
McKay, S. L. (2009). Introspective techniques. In J. Heigham & R. A. Croker (Eds.), Qualitative research in
applied linguistics: A practical introduction (pp. 220–241). Palgrave Macmillan.
Mills, K. A. (2010). The multiliteracies classroom. Multilingual Matters.
Moon, J. A. (1999). Learning journals: A handbook for academics, students and professional develop-
ment. RoutledgeFalmer.
Murakami, S., & Bryce, M. (2009). Manga as an educational medium. The International Journal of the
Humanities: Annual Review, 7(10), 47–56. https://doi.org/10.18848/1447-9508/CGP/v07i10/42761
Pearson-Evans, A. (2006). Recording the journey: Diaries of Irish students in Japan. In M. Byram & A.
Feng (Eds.), Living and studying abroad: Research and practice research and practice (pp. 38–63).
Multilingual Matters.
Prieto Velasco, J. A., Tercedor Sánchez, M. I., & López Rodríguez, C. I. (2007). Using multimedia materi-
als in the teaching of scientific and technical translation. Linguistica Antverpiensia, 6, 115–134.
Risku, H., & Pircher, R. (2008). Visual aspects of intercultural technical communication: A cognitive
scientific and semiotic point of view. Meta, 53(1), 154–166. https://doi.org/10.7202/017980ar
Royce, T. (2002). Multimodality in the TESOL classroom: Exploring visual-verbal synergy. TESOL
Quarterly, 36(2), 191–205. https://doi.org/10.2307/3588330
Schmerbeck, N., & Lucht, F. (2017). Creating meaning through multimodality: Multiliteracies assess-
ment and photo projects for online portfolios. Die Unterrichtspraxis/Teaching German, 50(1), 32–44.
https://doi.org/10.1111/tger.12020
 Language Awareness 249

Schwartz, A., & Rubinstein-Ávila, E. (2006). Understanding the manga hype: Uncovering the multimo-
dality of comic-book literacies. Journal of Adolescent & Adult Literacy, 50(1), 40–49. https://doi.
org/10.1598/JAAL.50.1.5
Simard, D. (2004). Using diaries to promote metalinguistic reflection among elementary school
students. Language Awareness, 13(1), 34–48. https://doi.org/10.1080/09658410408667084
Stein, P. (2000). Rethinking resources in the ESL classroom: Rethinking resources: Multimodal pedago-
gies in the ESL classroom. TESOL Quarterly, 34(2), 333–336. https://doi.org/10.2307/3587958
Tercedor Sánchez, M., Alarcón Navío, E., Prieto Velasco, J. A., & López Rodríguez, C. I. (2009). Images as
part of technical translation courses: Implications and applications. The Journal of Specialised
Translation, 11, 143–168.
The New London Group. (1996). A Pedagogy of multiliteracies: Designing social futures. Harvard
Educational Review, 66(1), 60–93. https://doi.org/10.17763/haer.66.1.17370n67v22j160u
Torresi, I. (2008). Advertising: A case for intersemiotic translation advertising. Meta, 53(1), 62–75.
https://doi.org/10.7202/017974ar
Yi, Y., & Angay-Crowder, T. (2016). Multimodal pedagogies for teacher education in TESOL. TESOL
Quarterly, 50(4), 988–998. https://doi.org/10.1002/tesq.326