DRAFT 17/11/10

A cross-disciplinary approach to the analysis of language learners' online

conversations

Abstract

Online conversations are now ubiquitous, and language learning researchers are one community
with a particular need for a better understanding of data collected from such exchanges. This paper
addresses the lack of formalised methodology for analysing research data created in
technologically-mediated conversations. First we show the importance of conversations in language
learning and establish that appropriate methodologies for investigating them have as yet emerged
neither from the field of computer-assisted language learning nor from conversation analysis (CA).
We analyse three brief text and voice-based conversations among language learners engaged in
online tutorials via a multimodal platform. From this we conclude that the analysis and
interpretation of such exchanges can be improved by a crossdisciplinary approach which consists of
augmenting constructs drawn from CA with selected constructs from social semiotics.

Keywords

Multimodality, computer-mediated communication, second language acquisition, social semiotics,

1
Introduction

This paper reports on research into interactions undertaken in technology-enhanced environments

by learners seeking to enhance their oral and written fluency in their second or foreign language

(L2). The aim of the study is to establish methodological principles to address a gap in the analysis

of online multimodal data. The paper, which focuses on such phenomena as turn-taking and face-

saving in an online course, takes Conversational Analysis (CA) as its theoretical starting point but

adopts a multimodal perspective that appears to shed greater light on the meaning-making processes

involved in interactive digital environments.

To this aim, we will start by establishing the need for an understanding of what learners are

doing when they converse via multimodal electronic systems. We will then discuss the applicability

of CA to multimodal conversations online. Finally, drawing from communication theory and

multimodal social semiotics, we will identify three frameworks which may provide guiding

principles for a methodology with the potential to answer questions raised by our data.

The role played by online conversation in language teaching and learning

Conversations are seen as beneficial to L2 learners for three reasons. Firstly they have been part of

the communicative model of language teaching for half a century, as ‘conversation classes’ in

which learners, in the safety of the classroom, are invited to experience the ‘pressure of

conversation’ (Cook, 1996 [1991]: 61) that it is assumed they will face when called upon to talk to

native speakers in the target country. Secondly, work carried out since the mid-80s within Second

Language Acquisition theory has established that, providing they are structured so as to require

participants to negotiate meaning, conversations promote socio-cognitive progress (Gass

2
andVaronis 1995; Long 1998; Gass, Mackeyand Pica 1998; Chapelle 2004). Thirdly, for

researchers such as Belz andThorne (2005) and O’Dowd (2006), who have investigated

intercultural aspects of telecollaborative transnational online projects, conversations are the means

whereby learners from different cultures construct their knowledge of the conversational strategies

of the target culture while developing their skills as interactants.

To attain these learning outcomes, research has shown that conversations have to satisfy

certain criteria. They need to be part of a constraining task (for example a group of learners might

be asked to address a problem then reach a negotiated consensus). However, to help learners cope

with the spontaneity of speech, conversations also have to offer them sufficient time and freedom to

develop new topics or to attend to unexpected conversational moves made by their discourse

partners. Tutorial software can offer such opportunities: it may allow learners to negotiate in groups

online (through the use of grouping tools and ‘breakaway rooms’), to respond fast (by textchat or

audio) or more reflectively (with written documents co-created during the interaction). Such

software may also allow the sharing of visual objects or sound files, that act as stimuli for talk.

Multimodal voice-enabled platforms are thus of great interest to language learning researchers.

Some preliminary thinking about how to approach the study of conversations in these

environments was nourished by reflections on our earlier piece of work (Author, 2006: 260),

focused on a course involving both written text (asynchronous and synchronous) and recorded

speech (asynchronous only). We had observed that prescribing the use of specific communication

channels for particular discursive goals was ineffective, because users re-appropriated the

environment in their own way, successfully developing conversations in phases of the educational

task, and in ‘spaces’ of the environment, that were not originally dedicated to this purpose. For

example, although we had structured the asynchronous forum into two separate ‘threads’, one for

task-completion and one for social interaction, users exchanged messages about both topics

3
everywhere in the environment, making no attempt to select the prescribed thread. Building on this

experience, we based the design of the course from which the data below is extracted on the

hypothesis that observations of learners’ unconstrained conversational behaviours across the

environment’s modalities would provide us with opportunities to understand how they achieved

their discourse aims. We also expected that learners would reach some of these aims by creative re-

assignment of the tools’ functions, or catachresis as Rabardel (1996) terms it.

Central to our project was a focus on multimodality, defined for the purpose of this study as

the co-availability of several modalities, this term itself being understood as the material through

which the conditions for interaction are created when humans use an online platform. For example

any technological feature seen in its relationship with a semiotic resource is a modality, so that a

semiotic resource may be associated with a single modality (as when spoken language is associated

solely with the output of an audio channel), or with several modalities (as when written language is

associated both with user-generated messages (real-time and asynchronous) as well as with system-

generated ones.

Analysing conversations in technology-mediated environments

One approach to research into online conversations is computer-mediated discourse analysis.

Introduced by Herring in 2004, it has been influential within the computer-assisted language

learning research community but its application is limited to text-based exchanges. To account for

synchronous voice-enabled interaction, CA appears to be a more promising methodological

approach. Drawing on Goffman’s (1967) work on everyday mother-tongue face-to-face

interactions, Sacks, Schegloff and Jefferson (1974) used CA to analyse everyday conversations in a

technologically-mediated setting, the telephone. Hutchby (2001) further extended the field of

application of CA to Internet-mediated (including video-enhanced) everyday interaction. Mondada

has used CA to explore technology-mediated professional conversations, for example telesurgery

4
involving surgical teams using both mother tongue and L2 (Mondada, 2003). There has been some

debate about the usefulness of CA in researching face-to-face L2 conversations, with Wagner

(1996) querying it and Seedhouse (1998, 2004) convincingly countering his views. Garcia and

Jacobs (1999) demonstrated CA’s appropriateness for analysing online L2 conversations in a paper

on turn-taking in written chat, as long as the research instruments included videos of the screen of

each learner – which today we would call ‘dynamic screen capture’ – in order that hesitations and

self-corrections could be evidenced in users’ messages prior to their being sent. Also focused on

textchat, Payne and Ross (2005) used elements of CA to argue that there is a relationship between

written chat proficiency and oral proficiency in L2. But to date there has been no use of CA for

investigations that integrate the multimodality of online settings and human interactivity in a L2

learning project, with the exception of our own work Author (2004) and Author and Another

(forthcoming), where we draw selectively from CA to account for aspects of learning on

audiographic platforms and desk-videoconferencing environments respectively.

The nature of the research gap: CA and synchronous multimodality online

Among the tenets of CA, two are pertinent to the data to be introduced shortly. Like all CA

principles, they can account for conversational material both when participants abide by them and

equally when they breach them. In a simplified form they can be summed up as follows: (a) turns of

speech alternate and interlink, since the basic principle of ‘conditional relevance’ asserts that given

a question, you may expect an answer; given an apology, expect an acknowledgement; given a

topic, expect that it will be pursued; (b) face-saving is a major preoccupation in any conversation,

because conversants are in a relationship of perpetually converging and diverging interests with

their fellow-conversants - you may try to save your own face or to protect others’, and it is also

possible to threaten one’s own or others’ faces for strategic reasons.

5
 In the computer-assisted multimodal communication context under study, discourse partners

use many semiotic systems, including linguistic (written and spoken), iconic and symbolic systems.

They may use these systems in rapid succession (type then draw), quasi-simultaneously (speak

while hitting the ‘send’ button to post a written message) or they may choose among systems to

make meaning in particular ways. For example, a user may close a conversation by typing ‘Bye for

now’, by clicking a specific button, or by announcing their withdrawal orally. Different semiotic

systems are also involved in responding to such a move. For example, if users of a synchronous

environment signal their impending departure orally, their discourse partners will receive aural

input, made up of linguistic and non-linguistic information, e.g. a phone heard ringing in the

departing person’s home will suggest reasons for their terminating the conversation. Or a user may

type a valedictory message into a box and send it. In this case, their message will be displayed on

the shared screens and will remain a part of the ongoing conversation even after they have

disappeared from the environment. Or again users might avail themselves of the system’s

telepresence indicators, e.g. icons of different colours signalling that users are ‘offline’, ‘online’,

‘on standby’ etc, or display symbolic changes to the objects on the screen, such as the greying-out

of a name, or fading-out of a photo. Choosing to leave the conversation by activating such a tool is

a strategy likely to influence the direction of the conversation in a way that is distinct from the

effect produced by disconnecting altogether (in which case the name or image would vanish), or

from the effect of leaving all connections ostensibly active but walking away from the computer for

the rest of the session.

Thus in environments with multiple meaning-making devices, the combinations and

interlinking of semiotic systems and effects can become extremely complex, and there is scope for

observing conditional relevance and face-saving at work, through studying the users’ conventional

as well as catachrestic use of the tools. In the rest of this paper we seek theoretical support when

attempting to grasp this complex reality, by studying three brief conversations carried out through

6
the Lyceum environment as part of an English-for-Special-Purposes project. The next section briefly

describes the system and the pedagogical setting, and then introduces the three data sets.

The Copéas project: environment, population and learning context

Lyceum is a synchronous audiographic groupware system used at the University of XXXX between

1995 and 2009, designed to facilitate social learning in distance tutorials. Figure 1 shows the

characteristic tools and spaces of this tutorial platform: a facility for creating ‘rooms’ for small-

group work, a space for textual and graphic displays (whiteboards or other shared documents), a

space representing the participants’ conversational status (speaking, listening quietly or seeking

permission to take a turn), a space for text-chatting and icons for indicating agreement or

disagreement non-vocally and non-textually (voting or ‘Tick’ icons).

[INSERT Fig 1. Screenshot from a Lyceum session]

Fig.1. Screenshot from a Lyceum session

While technically the University’s servers may accommodate hundreds of connections at a

time, the logistics of turn-taking on this platform is such that the optimal number of participants is

10 to 15 per tutorial room. However the subdividing of plenary groups into separate sub-groups of 3

to 5 students, and back again to plenary spaces, is easy and almost instantaneous. Within each room,

different tools and shared spaces can be displayed and collaboratively used. A turn-taking

management system based on clickable icons is another feature of such platforms that has relevance

for our study. As shown in Figure 1, a raised hand icon symbolises an intention to take a turn, while

a loudhailer or a microphone icon indicates who is speaking. However, there is no technical

obstacle to ignoring the turn-taking tool altogether (although in practice no more than three people

are recommended to speak at once, because of aural overload on listeners).

7
 The extracts discussed below originate from a project concerned with teacher-learner

communication in an audio-synchronous environment. The project, known as Copéas, was a

partnership between the University of XXXX and the University of YYYY. Sixteen French-

speaking students studying for a Professional Masters in Open and Distance Teaching (ODT)

worked in two groups of eight, connecting from their homes in various parts of France. Each group

had an English native speaker tutor connecting from the United Kingdom. The groups met during

10 sessions of over an hour each. The course had a dual objective, linguistic and professional,

which was the development of competences in ODT through spoken and written English. The less

proficient group (false beginners with wide variations in knowledge) provided the extracts

discussed below.

The Lyceum environment and the Copéas project are presented in detail in Vetter and

Chanier (2006). The environment supports semiotic resources for constructing discourse in

interaction, such as natural language in its written and spoken forms, as well as visual resources

such as icons, images, colours and shapes.

The data: three multimodal conversations

The three extracts occur within the last seven minute of the last session in the course. Both Tables 1

and 2 give simplified versions of a tabular transcription of the screen videos. Reading the top row of

Table 1 from left to right, the transcripts show: participants’ initials (User), the speech turns (T,

characterising all inputs, rather than simply those in ‘speech’), chronological sequences (Time) and

data collected from the four modalities (in the four rightmost columns).

Table 1

Re-interpreting sequentiality (Extract 1)

8
[INSERT TABLE 1 EXTRACT 1]

When the first extract begins, the learners are engaged in an evaluation-type conversation.

Three students, H, C and A are communicating orally negotiating an agreed heading, summing up

what they think they mostly learned during the course. The heading then needs to be typed on the

shared document screen by one of the group. In Extract 1 (Table 1), H twice tries to pronounce the

phrase technical vocabulary (at T1 and T7) but stutters and hesitates each time. As C attempts to

settle the answer by typing vocabulary learning in the shared document (T3) and eliciting H’s

approval of this formulation (T5), H holds to his original wording by writing it in the chat window

(T6). This results in C modifying (T8) what he had typed on the shared document, which H finally

approves (T9) by clicking the Tick icon. Arrows show topic-maintenance by H across modalities.

To compensate for his articulatory problems in English, H deploys an alternative conversational

strategy that takes advantage of  the environment’s multimodality. His strategy cannot be made

visible through conversation analysis conducted in its classical form since CA relies on the

sequentiality principle, and the conversation, if read vertically down the second column of Table 1,

has no sequencing that could be sensibly interpreted. Sequentiality is in fact present, but can only be

detected by analysing the four rightmost columns.

Table 2

Re-interpreting the face-saving principle (Extract 2)

[INSERT TABLE 2 EXTRACT 2]

Just before the start of Extract 2, the tutor had launched the conversation by asking the

members of the group (A, P, G and C) to introduce themselves orally. The extract in Table 2 shows

the conversation proceeding orally, with some input typed into the text chat window (highlighted in

grey-blue), while two of the learners (A and P) carry out a completely different conversation in text

9
chat (highlighted in green), concerned with an auditory difficulty. In the recording which was used

for the transcription, A speaks quite audibly but, possibly due to her home setup or for server-

related reasons, P does not hear her. So A and P run a conversation in parallel with the main

conversation initiated by the tutor. They use chat to construct a dialogue on a different theme,

without apparently disturbing the ‘main’ conversation which is proceeding in the audio modality.

Thus, their attitude maybe seen as non-transgressive in terms of saving the tutor’s face. In a non-

virtual classroom, such a move would probably have been seen as face-threatening for the tutor,

who may have taken steps to stop it, had it persisted for more than a few seconds. The face-saving

principle helps explain A and P’s conversational moves. However, the example shows how this

principle needs to be re-interpreted for this online multimodal communication scenario.

In the last example, Figure 2, we see how learner C uses the four modalities at his disposal:

audio, text-chat, Tick icon and shared document.

[INSERT FIGURE 2]

Fig.2 Use of modalities by participants in Extract 3

The chart in Figure 2 highlights discrepancies in the ways each participant (H, C and A) uses

these modalities: the horizontal axis shows the modalities available while the vertical axis gives the

number of speech turns per individual. Learner C’s inputs are mainly via the audio channel (45

audio speech turns, 4  text-chat turns). Of the group, C has the highest level of proficiency, as

evidenced by his score in the pre-experiment test and by his ease with target structures such as

‘would you repeat?’, ‘we must choose’, ‘we can’t answer’, ‘do you want to add something?’, ‘don’t

you think so?’ ‘are you OK with what I write?’ etc. Content analysis of his input shows that in over

half his spoken turns (28 out of 44) he is asking for others’ opinions. Yet all his text-chat inputs are

language accuracy checks. Additionally, C makes much more use of the shared document than do

his two colleagues. In data collected from the parts of the session that precede and follow Extract 3,

10
90% of C’s turns in the shared document are directly preceded by a turn elsewhere (mostly in the

audio channel) seeking confirmation of the group’s approval of what he has written.

We hypothesize that C uses the text-chat modality when he is checking the accuracy of the

lexicogrammatical forms he is using rather than moving through the conversational agenda of the

class. He prefers to progress through the class agenda via the shared document modality, but only

after obtaining consensus from his peers orally. This pattern of use could relate to two different yet

converging factors: C may be specialising particular discourse aims to particular screen elements,

and his representation of himself as a communicator may also play a role in which of these he uses.

For example, in relation to the elements in the screen layout, analysts of his conversational

strategies would need to assess the degree of salience of the text-chat window tucked away

unobtrusively (and indicated in the bottom right-hand corner of Figure 1 by an arrow pointing to

‘chat’) in contrast to the intrusiveness of the sound of his voice coming through each group

member’s headset. As for self-representation, we may posit that face-saving issues are involved: for

example C might be prepared to give an image of himself as a confident language speaker in the

audio channel while specialising the chat window for more risk-taking face-threatening activities,

such as asking for help with English forms. A similar explanation might be offered for his use of the

shared document: the visual organisation of the screen when that document is uploaded to it (see

Figure 3 further down), the central disposition of that document and its status as the ‘official’ record

of the group’s collaboratively negotiated view may explain both his self-appointed guardianship

of it and his diffidence in committing material to it, unsupported by his peers.

Discussion

In this section and on the basis of the findings from the extracts presented earlier we suggest that

while CA remains a useful approach to the understanding of sense-making in real-time online

multimodal settings, it needs to be rearticulated. Our suggestion is that such reframing can be done

11
in the light of three theoretical frameworks drawing from affordance theory as developed by Gibson

(1979), multimodal social semiotics and geosemiotics respectively. Our reading of the data showed

that two of the principles of traditional CA, sequentiality (Extract 1) and face management (Extract

2) could be used to characterise conversational moves within electronically-mediated multimodal

conversations, albeit redefined, because when more than one modality was available, both

sequentiality and face management operated across modalities. Hutchby’s (2001) work is, we

suggest, relevant in this respect.

1. CA re-interpreted in the light of communicative affordances

Hutchby (2001) reminds us through the example of the early history of telephone communication

that technology is in a reciprocal relationship with its users. The telephone was originally sold as an

instrument for transacting business or for exchanging practical information. Early marketing

stressed this functional use to the extent of advising subscribers to wait until late at night if they

really needed to use the machine for the purpose of personal chat. However users were not

persuaded and appropriated the telephone as a medium for family or other intimate conversations,

forcing the telephone companies to review their marketing strategies.

[...] while designers may be said to have some control over the features they design into an

artefact, and while they may have some idea about the range of uses to which the artefact

should be put, they have little control over the artefact’s communicative affordances – over

the range of things it turns out to enable people to do (Hutchby, 2001: 123).

We witnessed this mechanism at work in the project extracts, where learners appropriated

modalities in diverse ways. For example the text-chat was re-appropriated in the contexts of

different conversations in order to support control over content (Extract 1), to provide technical

assistance (Extract 2) and to enact face management (Extract 3). We also observed that individual

participants engaged with different communicative affordances to satisfy identical communicative

12
needs. For example legitimation of self as a turn-taker (and therefore control over the conversation)

was achieved via moves from audio to text-chat then to the Tick icon by H (in Extract 1), but for C

(in Extract 3) by toggling between the audio channel and the shared document. The construct of

communicative affordance – as described by Hutchby in the last line of the quotation above - also

helps to understand the different ways in which human actors in different technological settings

solve a single problem such as (to stay with the example of the telephone) answering a landline call,

a mobile phone or a computer bleep from an Internet-telephony system. The same conversational

problem is involved in each case (how to respond to a conversational invitation from a remote

caller), but different discursive solutions are appropriate. For instance, a person called on a landline

without user ID will in most ordinary circumstances pick up the handset and initiate their side of the

conversation by uttering a conventional query (e.g. hello, allô, pronto etc.) in a rising tone followed

by a pause, in the expectation that the caller will identify themselves. With user ID, or in Internet

telephony where the caller’s name and sometimes photograph appears on screen, the person called

is likely to respond instead with a greeting usually followed by the caller’s name. A pause, if

provided by the person called, will not be heard by the caller as an invitation to identify him/herself;

instead it will require interpretation and negotiation so that the conversation may progress.

Although a close textual analysis of our learners’ input is not the focus of the present

methodological study, we follow Hutchby in maintaining that:

The detailed ways in which people have taken up the affordance for verbal intimacy across

distance and shaped it for communicative ends can only be revealed once we move beyond

looking at the cultural meanings or representations of the telephone as a ‘technological

artefact’ and observe telephone talk itself (Hutchby, 2001: 31).

In this view CA continues to be central to understanding conversations in electronically-

mediated multimodal settings. However only by paying attention to communicative affordances will

we be able to do the necessary work of re-interpreting its principles in terms of their manifestations

and discursive adaptations across modalities.

13
2. The influence of design on discourse in multimodal environments

The materiality of the environment impacts on the dynamics of conversations. In Extract 2, two

conversations were progressing in parallel. It is not accidental that the tutor-initiated one was

carried out orally while the students’ exchange about sound levels was conducted in text-chat: all

we need to do to become persuaded of this is to mentally invert the modal choices and imagine that

the tutor led his tutorial via postings in the text-chat while students talked about other topics in the

audio channel. It is unlikely that the group would accept such a position for the tutor, and we draw

from multimodal social semiotics to help explain why.

Kress and van Leeuwen’s (2001) work on the semiotics of multimodal pages in newspapers

and books identifies the four dimensions of structuring for such artefacts: discourse, production,

dissemination and design. While we believe that all four are relevant to understanding meaning-

making in electronic learning environments (see for example a discussion of the impact of

dissemination on learner progress in Author, 2004: 523-524), for the purpose of the current

discussion we concentrate on one of them, the dimension of design. The designers of the software

that our learners used can appositely be compared with the architect in Kress and van Leeuwen’s

explanation of the relationship between design and discourse:

An architect, for instance, designs (but does not build) a house or a block of apartments.

The discourse provides a certain view of how houses are lived in the way they do, and

arguments which critique or defend this way of life. The design of the house then

conceptualises how to give shape to this discourse in the form of a house, or a type of

apartment (Kress and van Leeuwen, 2001:6).

14
 The relevance of this architectural example will become clear as we look at two very

different designs of environments that have been used for language learning, Lyceum and Traveler,

through comments made by learners. Lyceum (Figure 3), as we have seen, is an academic tool

designed to look like a university campus building, which is a traditional design choice for

electronic learning environments as pointed out by Bayne (2008). Traveler (Figure 4) is an avatar-

based system with a ‘fantasy’ feel inherited from the world of games.

[INSERT FIGURE 3]

Fig.3: The design of a Lyceum screen, with whiteboard displaying learners’ work

[INSERT FIGURE 4]

Fig.4.The design of a Traveler screen

Just as the architect has provided a shape for the cultural discourses of human habitats, so

the designers of electronic environments and virtual worlds conceptualise the form that the

environments take on the screen into interpretable signs. For example here are responses from two

of our users (for Lyceum) and two of Örnberg Berglund’s (2005) users (for Traveler), upon being

asked about their feelings of ‘presence’ when online:

 Lyceum user 1: ‘Quand le prof rentre dans la salle, cela ne dérange pas. Je sais pas comment

l’expliquer’. (When the teacher enters the room, it’s not intrusive. I don’t know how to

explain it).

15
  Lyceum user 2: ‘Le style du prof joue, mais le fait qu’il est invisible, il ne peut pas s’imposer

de la même façon qu’en présentiel’. (The teacher’s style is a factor but the fact that he’s

invisible, he can’t impose himself in the same way as in face-to-face).

 Traveler user 1: ‘It took me to another world and was a real adrenaline buzz. It was on my

screen and I was conscious of it always, but I was definitely virtually gone from my usual

habitat’.

 Traveler user 2: ‘I’m always immersed.[…] It doesn’t matter that the environment is

artificial. […] I think of the place as real’.

Whereas these two Traveler users produce a discourse of emotions and escapism, the

discourse of Lyceum users reflects school-like representations of a particular type: teacher-led

classes. We make two comments here. Firstly, although these differences in perception may not be

surprising given the strongly contrasted visual identities of Figures 3 and 4, the question is whether

two groups using these environments for achieving the same language-learning objectives might

have very different types of conversations in each environment. The second observation relates to

designers and the uses made of their designs. Lyceum’s design was underpinned by a democratic

and participative pedagogical posture: ‘We have imposed minimal technical constraints on ‘floor

control’: anyone can speak anytime’ (Buckingham-Shum, Marshall, Brier and Evans, 2001:4). Yet

the users’ comments show a preoccupation with teacher control. It is likely that this is part of their

pre-existing non-virtual educational culture, in which case the question can be asked: to what extent

and in what ways can the design features of interactive learning environments transform the users’

representations of self? The answer to this question is another determiner of sense-making in these

environments. Finally, regarding the device which introduced this section, i.e. the proposal that the

Copéas tutor could conduct core tutorial business in the chat box while the students conversed

orally, evidence from Lyceum users’ perceptions supports the view that the system’s design

16
provides a shape for the cultural discourses of traditional teacher-centered classrooms. But based on

multimodal social semiotics’ understanding of design, there is no in-principle reason why other

types of design could not work to support other cultural discourses, producing distinct types of

conversations.

3. The role of the body in relation to space in multimodal environments

While Scollon and Scollon (2003) acknowledge the importance of Kress and van Leeuwen’s design

dimension, they re-interpret and re-inforce it in order to account for interaction. We have found

their insights useful for understanding the way that our users perform conversations by choosing

among the different spaces offered to them within the interface. Scollon et al stress the spatial

dimension of social semiotic resources, presenting a framework which they call ‘geosemiotics’:

Geosemiotics is the study of meaning systems by which language is located in the material

world. This includes not just the location of words on the page you are reading now but also

the location of the book in your hands and your location as you stand or sit reading this

(Scollon et al, 2003: x-xi).

The authors structure geosemiotics into three sub-sets: the interaction order, visual semiotics

(on which we will not elaborate here, as this concept comes close to Kress and van Leeuwen’s

notion of design mentioned earlier) and space semiotics. The understanding of space that is of

interest to us in our study of learners using a virtual environment on a computer is predicated on

each of these three sub-sets.

The interaction order provides a construct for understanding how individuals perceive the

interactional value of the space they choose to use. In their description of the interaction order, the

authors include perceptual spaces and interpersonal distances. Dominant perceptual spaces are

17
visual and auditory (‘less noticed’ ones are olfactory, thermal and tactile). The addition of the

construct of interpersonal distances – as a scale of values inspired by Hall’s (1969) work on

proxemics – allows geosemiotics to ask questions about the relationship between space, sound,

embodiment and socialisation. For example, the auditory space which I perceive and my perceived

intimacy or distance with the individual vocalising the sound that I am hearing, together create the

semiotic resource through which I embody meanings. Applying this framework to Copéas

participants, in particular to the parallel conversation mechanism in Extract 2 and to the multimodal

preferences of the learner in Extract 3, the question becomes: how do they co-construct

interpersonal values (intimate, personal, social, public) into conversations which proceed

simultaneously through visual spaces of varying salience and through an auditory space defined by

the spatially and tactilely intimate device of an earpiece or headset?

Space semiotics, in Scollon et al’s words, is the most fundamental part of geosemiotics,

because it asks ‘Where in the world is the sign or image located?’ and because it aims to account for

‘any aspect of the meaning that is predicated on the placement of the sign in the material world’

(2003: 146, our italics). The distinction between well-rehearsed debates within visual semiotics on

the subversive placement of images for artistic purposes (e.g. Warhol soup cans), and the focus of

space semiotics, is that the latter is looking at the material world as a whole, and not simply at

materials used for the bearing of signs, such as paper, canvas or brick. In terms of multimodal

electronic environments, space semiotics provides the basis for asking questions such as: how do

users decode and encode meanings in a material situation involving their computer and its various

peripherals (keyboard, mouse or keypad, webcam) as well as other stimuli around them (possibly

another computer, a video screen, a person physically present, who is talking, writing or drawing,

etc.)?

Conclusion

18
Two considerations should be offered in conclusion, one reflective and the other more concerned

with application. Firstly, due to the complexity of the phenomena under study and the

pluridisciplinary nature of the analysis, we have only touched on principles for a future

methodology. A clear direction for further work, then, is to complement and operationalise these

principles. Secondly, we are aware of some use of social semiotics for researching some online

environments, such as the semiotics of multiplayer immersion games (Lemke, 2006), and of social

tagging (Huang and Chuang, 2009), but not yet for investigating the use of the multimodal

communication tools at the core of this study. Similarly, in the educational field, although Jewitt,

and Kress (2003) have used social semiotics in their work as part of a research area sometimes

called ‘multimedia semiotics’ to lay the foundation for an understanding of learning via multimodal

texts and via educational software (Jewitt 2004), neither on-screen interaction nor issues related to

synchronicity of communication are part of such work to date.

As a response to both the reflective and the application dimensions just mentioned, we see a

useful research agenda emerging: to test out, with a large volume of learner interaction data

collected from interactive multimodal technologies (different platforms, social networks, mobile

devices) in settings both educational and non-educational, the methodological claims made in this

paper according to which such data can be better understood through the synergistic use of

conversation analysis, multimodal social semiotics and geosemiotics.

Acknowledgement

We thank Steve DiPaola (www.dipaola.org) for permission to reproduce a Traveler screen.

19
References

Author, (2004) ‘Oral Conversations Online: Redefining Oral Competence in Synchronous

Environments’,  ReCALL 16 (2), 520-538.

Author, (2006) ‘Interactive Task Design: Metachat and the Whole Learner’, in M. del Pilar García

Mayo. (ed) Investigating Tasks in Formal Language Learning. Clevedon: Multilingual

Matters, 242-264.

Author and Another, (forthcoming) ‘The multimodal mediation of semiotic resources in an online

conversation’, in  Develotte, C. Kern, R. and Author (eds) Décrire la communication en

ligne: le face-à-face distanciel, Lyon : ENS Editions.

Belz, J. and Thorne S.(eds.) (2005). Internet-Mediated Intercultural Foreign Language Education.

Boston: Thomson Heinle.

Bayne, S. (2008) ‘Higher Education as a Visual Practice: Seeing Through the Virtual Learning

Environment’, Teaching in Higher Education, 13(4): 395-410.

Buckingham-Shum, S., Marshall, S. and Evans, T. (2001) ‘Lyceum: Internet Voice Groupware for

Distance Learning’, in Proceedings of the 1st First European Conference on Computer-

Supported Collaborative Learning. Maastricht: The Netherlands.

Chapelle, C. (2004). ‘Technology and Second Language Learning: Expanding Methods and

Agendas’, System, 32: 593-601.

Cook, V. (1996 [1991]). Second Language Teaching and Language Learning. London: Arnold.

Garcia, A. and Jacobs, J. (1999). ‘The Eyes of the Beholder: Understanding the Turn-taking System

in Quasi-Synchronous Computer-Mediated Communication’, Research on Language and

Social Interaction, 32 (4): 337-367.

Gass, S., Mackey, A. and Pica, T.(1998). ‘The Role of Input and Interaction in Second Language

Acquisition’, Modern Language Journal, 82: 299-307.

Gass, S. and Varonis, E. (1994). ‘Input, Interaction, and Second Language Production’, Studies in

Second Language Acquisition, 16 (3): 283-302.

20
Gibson, J. J. (1979). The Ecological Approach to Visual Perceptions. Boston: Haughton Mifflin.

Goffman, E. (1967) Interaction Rituals: Essays on Face-to-face Behavior, New-York: Random

House.

Hall, E. (1969). The Hidden Dimension. New York: Doubleday.

Herring, S. (2004). ‘Computer-Mediated Discourse Analysis: An Approach to Researching Online

Behavior’, in S. Barab, R. Kling and J. Gray. (eds), Designing for Virtual Communities in the

Service of Learning. New York: Cambridge University Press, pp. 338-376.

Huang A.W. and Chuang, T.R. (2009) ‘Social Tagging, Online Communication, and Peircean

Semiotics: A Conceptual Framework’, Journal of Information Science, 35 (3); 340-357.

Hutchby, I. (2001). Conversation and Technology: From the Telephone to the Internet. Cambridge:

Polity.

Jewitt, C. and Kress, G. (2003). Multimodal Literacy. Bern: Peter Lang Publishing.

Jewitt, C. (2004). ‘Multimodality and New Communication Technologies’, in P. LeVine and

R.Scollon. (eds), Discourse and Technology: Multimodal Discourse Analysis. Washington

DC: Georgetown University Press, 184-195.

Kress, G. and Van Leeuwen, T. (2001). Multimodal Discourse: the Modes and Media of

Contemporary Communication. London: Arnold.

Lemke, J. (2006). ‘Towards Critical Multimedia Literacy’, in M.McKenna, L.Labbo , R. Kieffer

and D. Reinking. (eds). International Handbook of Literacy and Technology Volume III.

Mahwah, New Jersey: Laurence Erlbaum Associates.

Long, M. (1983). ‘Linguistic and Conversational Adjustments to Non-Native Speakers’, Studies in

Second Language Acquisition. 5 (2): 177-193.

Mondada, L. (2003). ‘Describing Surgical Gestures: the View from Researcher’s and Surgeons’

Video Recordings’, in Proceedings of the First International Gesture Conference, Austin, July

2002.

21
O'Dowd, R. (2006). Telecollaboration and the Development of Intercultural Communicative

Competence. Berlin, Langenscheidt.

Örnberg Berglund, T.(2005) ‘Multimodality in a Three-dimensional Voice Chat’, in Proceedings of

the 3rd Conference Multimodal Communication, Papers in Theoretical Linguistics. Göteborg:

Göteborg University.

Payne, J. S. and Ross, B. M. (2005) ‘Synchronous CMC, Working memory, and L2 Oral

Proficiency Development’, Language Learning and Technology, 9 (3): 35-54.

Rabardel, P. (1995) Les Hommes et les Technologies, Approche Cognitive des Instruments

Contemporains, Paris: Armand Colin.

Sacks, H., Schegloff, E. and Jefferson, G. (1974). ‘A Simplex Systematics for the Organisation of

Turn-taking for Conversation’, Language, 50: 696-735.

Scollon, R. and Scollon, S. W. (2003). Discourses in Place: Language in the Material World.

London and New York: Routledge.

Seedhouse, P. (1998) ‘CA and the Analysis of Foreign language Interaction: a Reply to Wagner’,

Journal of Pragmatics 30: 85-102.

Seedhouse, P. (2004) The Interactional Architecture of the Language Classroom: A Conversation

Analysis Perspective. Oxford: Blackwell.

van Leeuwen, T. and Jewitt, C. (2001) A Handbook of Visual Analysis. London: Sage.

Vetter, A. and Chanier, T. (2006). ‘Supporting Oral Production for Professional Purposes, in

Synchronous Communication with Heterogenous Learners’, ReCALL 18 (1): 5-23.

Wagner, J. (1996) ‘Language Acquisition through Foreign Language Interaction: A Critical Review

of Studies in Second Language Acquisition’, Journal of Pragmatics 26: 215-35.

Online reference

Traveler http://dipaola.org. Accessed 23.10.2010.

22