“ChatGPT is the Companion, Not Enemies”- EFL Learners’ Perceptions and Experiences in Using ChatGPT for Feedback in Writing
“ChatGPT is the Companion, Not Enemies”- EFL Learners’ Perceptions and Experiences in Using ChatGPT for Feedback in Writing
A R T I C L E I N F O A B S T R A C T
Keywords: The present research was to bridge a research gap in comprehensively understanding students’ perceptions and
ChatGPT experiences in utilizing ChatGPT for their writing process in English as a foreign language (EFL) context. The
EFL writing participants were 45 EFL learners in Macau. The study aimed to explore the potential impact of ChatGPT on
Process writing instruction
writing, as well as their perceptions and experiences of ChatGPT in generating feedback for writing. A mixed-
AI assistance
Digital education
methods approach was employed, with quantitative data collected from a questionnaire and qualitative in-
sights drawn from interviews conducted after a semester-long writing course. The findings supported the sig-
nificant positive effects of AI assistance on writing, including writing motivation, self-efficacy, engagement, and
collaborative writing tendency. The qualitative data obtained from the interviews partially corroborated the
quantitative findings, although some participants expressed concerns about the use of ChatGPT. Relevant im-
plications and limitations were proposed based on the findings.
1. Introduction systems have faced criticism for their inability to assess the deeper, more
nuanced aspects of students’ writing.
Feedback is crucial to EFL writing instruction. However, providing Recently, the advancement of artificial intelligence (AI) technology,
effective feedback can be a significant challenge for teachers, often particularly with the introduction of ChatGPT, has become a significant
leading to an overwhelming workload (McMartin-Miller, 2014). Deliv- topic in language education (see Kohnke and Zou’s special issue on
ering personalized and constructive feedback to a diverse group of ChatGPT in language education). The creation and deployment of AI are
learners can be both time-consuming and mentally exhausting (Teng & transforming our lives and communities in numerous ways (Borenstein
Ma, 2024). Teachers frequently experience fatigue from the repetitive & Howard, 2021), and higher education is no exception. ChatGPT, a
task of correcting student essays (Hyland, 1990) and EFL learners’ large language model (LLM), is a chatbot capable of generating re-
self-efficacy beliefs play an important role in giving and receiving sponses to a wide range of prompts in a form of languages, instructions,
feedback for writing (Teng & Teng, 2024). As a result, researchers are questions. It can engage in sustained, human-like interactions, under-
increasingly exploring the potential of using machine-generated feed- standing and generating text in a conversational manner. This capability
back for writing. allows it to simulate chat-based conversations with humans, making it
To address the challenges associated with teacher feedback, alter- applicable in various contexts and useful for providing feedback on
native methods have been employed. Generally, there are two main writing (Huang, 2023). The release of ChatGPT underscores the urgent
types of supplementary feedback: peer feedback and machine feedback need for school-age learners to foster an awareness of digital compe-
(Guo & Wang, 2024). Peer feedback involves classmates reviewing each tencies. These tools are being increasingly suggested as valuable re-
other’s texts and offering comments and suggestions. Machine feedback, sources for enhancing writing instruction and feedback in educational
on the other hand, utilizes software or tools to automatically generate settings.
feedback aimed at improving writing. While peer feedback fosters Teachers and researchers in applied linguistics are currently navi-
collaboration and critical thinking among students, machine feedback gating the integration of ChatGPT technologies in the classroom,
offers notable advantages in terms of efficiency and engagement. For uncovering both its advantages and challenges (e.g., Kohnke et al.,
instance, automated writing evaluation (AWE) systems can enhance EFL 2023). One significant concern is the ethical use of such technologies in
students’ engagement in writing (Zhang, 2020). However, current AWE academic writing, particularly regarding risks to academic integrity
https://doi.org/10.1016/j.caeai.2024.100270
Received 30 May 2024; Received in revised form 9 July 2024; Accepted 16 July 2024
Available online 26 July 2024
2666-920X/© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
(Teng, 2023). Despite these concerns, AI technology remains a valuable support students in effectively processing and utilizing feedback to
tool for writing assistance (Godwin-Jones, 2022). For instance, ChatGPT enhance their writing skills.
has been identified as a practical and user-friendly tool for automated To address these concerns and motivate EFL students to provide
essay scoring (Mizumoto & Eguchi, 2023) and for assessing writing ac- more valuable and constructive feedback, the emergence of AI has been
curacy (Mizumoto et al., 2024). Additionally, students in Japanese EFL proposed as a solution. Pre-trained large language models (LLMs) have
contexts have shown a strong preference for using AI technology for demonstrated outstanding performance across various tasks. ChatGPT-
editing and proofreading tasks (Allen & Mizumoto, 2024). By incorpo- 4o (OpenAI, 2024), the latest LLM trained on an extensive corpus of
rating guidance, teachers can help learners develop the skills necessary text data, has garnered significant attention within a month of its
to use ChatGPT effectively and critically. This approach can help release. Unlike its predecessors—GPT-1, GPT-2, GPT-3, GPT-4—this
maximize the potential benefits of ChatGPT for language learners, version employs reinforcement learning from human feedback to
enhancing their writing capabilities and overall learning experience. enhance the objectivity of its language modeling. ChatGPT-4o is reputed
However, there remains a significant gap in understanding the to generate more accurate and less toxic human-like responses, making
practical implementation and effectiveness of ChatGPT within foreign it a versatile tool for various applications, including different writing
language writing classrooms, particularly from the perspective of EFL contexts and genres.
learners (see Javier & Moorhouse, 2023). While previous studies have However, due to ethical concerns, some universities have prohibited
provided insights into EFL students’ writing strategy use under the their students from using ChatGPT for classroom activities, coursework,
ChatGPT context (Allen & Mizumoto, 2024), and learners’ interest and and assessments. While ChatGPT offers ample opportunities for writing
familiarity with ChatGPT (Javier & Moorhouse, 2023), limited attention innovation, its use has sparked widespread concerns in public media.
has been paid to examining EFL learners’ actual use and perceptions of For instance, Noam Chomsky referred to ChatGPT as high-tech plagia-
ChatGPT in generating feedback for writing. The limited research in this rism (Stewart, 2023), and Weismann (2023) argued that it poses a novel
specific area highlights the urgency and importance of further investi- threat to human intelligence and academic integrity. According to
gation to inform best practices and pedagogical approaches in EFL Kostka and Toncelli (2023), there is a fear that students will no longer
writing instruction. As AI technologies are incorporated into writing write their own assignments, as ChatGPT introduces new risks for
pedagogy, the dynamics of writing and feedback generation may cheating and raises heightened concerns about academic integrity. In
change. This includes the collaboration between teachers and students, response to these concerns, some institutions, such as the University of
among students themselves, and between students and machines. Stu- Hong Kong, initially banned the use of ChatGPT when it was first
dents may find themselves better equipped to decide how to collaborate released. Teng (2024), after conducting a systematic review of academic
with AI-enabled tools for feedback generation, figuring out the differ- articles published on ChatGPT for EFL writing from January 2023
ences between the roles of EFL teachers and AI technology. To date, few, through June 2024, acknowledges the potential of ChatGPT in
if any, studies have assessed EFL learners’ voices and experiences with enhancing EFL writing. However, Teng (2023) emphasizes the need for
AI-generated feedback (Guo et al., 2022). This research gap may stem measures to mitigate the impact of ChatGPT on academic integrity and
from the fact that generative AI has not been fully incorporated into the calls for a new reflection on approaches to address these ethical
curriculum. Considering the emergence of advanced AI-powered tech- concerns.
nologies like ChatGPT-4o, it is time to reconsider the proposition. The Despite these bans, the trend of using ChatGPT in education con-
present study, employing a mixed-methods approach of surveys and tinues to grow. Researchers are actively discussing the implications of
interviews, aims to explore students’ perspectives on AI over a ChatGPT for teacher feedback (Guo & Wang, 2024). Given ChatGPT’s
semester-long writing course. It is important to note that the goal of this ability to understand language patterns (Rudolph et al., 2023), its po-
research is not to evaluate AI’s potential to replace human teachers, but tential to assist with writing-related tasks, such as correcting gram-
rather to examine its possible role in supporting learners and teachers by matical errors and rephrasing sentences, appears promising. The debate
generating feedback for writing. on whether ChatGPT should be banned or embraced by educators and
students is ongoing, and empirical studies on AI in EFL writing are likely
2. Literature review to continue. However, to the best of my knowledge, there is limited
empirical evidence demonstrating ChatGPT’s potential to support stu-
2.1. ChatGPT and feedback dent writers in providing feedback. Therefore, the present study ex-
plores how EFL students might collaborate with ChatGPT to generate
In the realm of English language teaching, peer feedback has become feedback for writing and investigates their perceptions and experiences
a common practice in writing classrooms (Rollinson, 2005). This during this process.
approach involves students reading and evaluating each other’s drafts
(Storch, 2002), which is beneficial for both the feedback providers and 2.2. AI-assisted writing
recipients (Teng & Ma, 2024). Writing teachers, particularly those
handling large classes, often face challenges in delivering timely and Advanced large language models (LLMs), such as ChatGPT, have
comprehensive written feedback. The volume of personalized feedback gained widespread usage in various educational settings. Kasneci et al.
that addresses the specific needs of each student’s writing is typically (2023) examined the potential of incorporating ChatGPT into education
limited (Er et al., 2021). Additionally, students may find that teacher and noted the continuous advancements of large language models in
feedback on content is overly general, rendering it less helpful (Zacha- pushing natural language processing. Kohnke et al. (2023) also high-
rias, 2007). In such contexts, peer feedback emerges as a practical so- lighted the potential and challenges of using ChatGPT in language
lution, aiding teachers in overcoming these issues. learning, emphasizing the need for teachers and learners to use this
While peer feedback can offer numerous advantages, it also has chatbot ethically and effectively to support language instruction.
certain drawbacks, such as issues with quality, focus, and a lack of in- Specific research on ChatGPT for writing assessment has been con-
depth feedback on meaning (Guo et al., 2022). For instance, compared ducted. Mizumoto and Eguchi (2023) explored the integration of
to teacher feedback, feedback from peers may be of lower quality, as ChatGPT into L2 writing settings, particularly in AES. Analyzing data
students may possess similar levels of writing proficiency (Ruegg, 2015). from 12,100 essays, they found that AES using GPT demonstrated
EFL students, who are untrained in providing feedback, tend to notable accuracy and reliability, supporting human evaluations. They
concentrate excessively on surface-level aspects like grammar, vocabu- also suggested that leveraging linguistic features could further enhance
lary, and punctuation, rather than on more substantive issues such as scoring accuracy. Based on these findings, they argued that ChatGPT has
content and organization (Rahimi, 2013). Therefore, it is essential to the capability to provide automated corrective feedback, making it a
2
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
3
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
Table 1
Sample prompts for writing and proofreading.
Sample prompts
1. Scan this text and highlight any grammatical errors you find.
2. Review this paragraph for proper punctuation usage.
3. Verify that the verb tenses remain consistent in this narrative.
4. Evaluate the sentence structure in this technical manual for clarity and simplicity.
5. Adjust the language and style to suit a formal report intended for board members.
6. Adjust this article’s language to make it accessible to a broader audience without technical expertise
7. Identify and correct spelling mistakes in this document targeted at an international audience.
8. Review the flow and transition between sections in this proposal to enhance logical progression.
9. List any differences or improved points for my writing in a table
Wang, 2023; Teng & Yang, 2023), the survey focused on four factors: These interviews provided additional qualitative data to complement
motivation for writing, writing engagement, self-efficacy for writing, the questionnaire results, offering a more comprehensive understanding
and collaborative writing tendency. Each factor comprised 5 items, of the students’ experiences with ChatGPT.
making a total of 20 items (see appendix). Participants were required to
respond to each statement using a five-point Likert scale, ranging from 1
(strongly disagree) to 5 (strongly agree). The Cronbach’s value for the 3.4. Procedures
questionnaire was 0.86, indicating satisfactory reliability.
As mentioned above, the course lasted for 15 weeks. Students were
3.3.2. Semi-structured interviews administered the questionnaire when the course came to an end. They
To gain a deeper understanding of learners’ perceptions and feelings were asked to reflect on their experiences throughout the semester. The
toward using ChatGPT, semi-structured interviews were conducted. purpose of this reflection was to gather comprehensive feedback on their
Eight students from the AI group voluntarily participated in these in- writing development and their use of ChatGPT for editing and
terviews. The interviews aimed to explore their experiences and proofreading.
thoughts in more detail. There were five interview questions, which are Overall, students spent approximately 10 min completing each
listed below. questionnaire. This brief yet focused time frame allowed them to
thoughtfully consider and respond to each item without feeling rushed.
1. How did you feel about using ChatGPT for editing and proofreading The questionnaire responses provided valuable quantitative data on
your writing? various aspects of their writing experiences, including motivation,
2. What were the advantages and disadvantages of using ChatGPT in engagement, self-efficacy, and collaborative tendencies.
your writing process? In addition to the questionnaire, semi-structured interviews were
3. How did ChatGPT impact your motivation and engagement in performed with eight students from the AI group. Each interview lasted
writing activities? about 10 min, during which the students shared more detailed and
4. Did you encounter any challenges while using ChatGPT? If so, how personal insights into their experiences with ChatGPT. These interviews
did you overcome them? Did you become more confident in writing? aimed to capture qualitative data that could enrich the quantitative
5. How do you think using ChatGPT influenced your overall writing findings from the questionnaires.
skills and self-efficacy? What about teacher feedback? Do you want Both the questionnaires and the interviews were conducted in Chi-
to co-work with your partners in future writing? nese to ensure that all participants could fully understand the re-
quirements. Using their native language also ensured their ability to
4
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
The original plan was to use mixed effects modeling for the survey Group − 0.317b − 0.0667 0.0214 − 0.144
(-2.58) (-0.55) (0.18) (-1.01)
data. In a mixed effects model, both fixed effects and random effects are
Time 0.105 0.181 0.0286 0.00952
considered simultaneously. The fixed effects are used to explain the (0.83) (1.45) (0.24) (0.06)
overall variation in the sample, while the random effects are used to GroupaTime 1.062c 0.427b 0.463c 0.882c
explain the differences between individuals or groups. However, after (6.11) (2.50) (2.82) (4.39)
_cons 3.467c 3.467c 3.562c 3.486c
implementing the mixed model, the results were similar to those ob-
(constant
tained using linear regression, and the random effects were not signifi- term)
cant. This means the differences between individuals or groups were not (38.63) (39.29) (42.07) (33.57)
statistically significant. Therefore, the decision was made to use least
N 90 90 90 90
squares regression instead. F 34.45 10.74 9.617 16.96
Stata 18.0 was used to conduct ordinary least squares regression on r2 0.546 0.273 0.251 0.372
the mean scores of the four variables. The assumptions for running linear r2_a 0.530 0.247 0.225 0.350
(adjusted
regression were met. The Q-Q plots of the residuals and the Kolmogorov-
R2 )
Smirnov test showed that the residuals (the differences between p 1.01e-14 0.00000456 0.0000153 9.69e-09
observed and predicted values) were normally distributed. All data an-
t statistics in parentheses.
alyses and visualizations were conducted using Stata 18.0. The data and a
p < 0.1.
the code used have been made accessible on OSF (https://osf.io/eqad9/) b
p < 0.05.
to ensure open sciences. c
p < 0.01.
The analysis of the interview data was based on theme coding. I
attempted to look for similar themes from the participants. The possi-
had a stronger impact in enhancing these key writing-related outcomes.
bility of reflecting the survey results was also considered, as the main
Skewness and Kurtosis supported the normal distribution of the data,
purpose was to support the quantitative findings. All participants in the
indicating that the data distributions were generally well-behaved with
interview sessions checked their transcripts and coding to ensure the
no major deviations from normality, further supporting the validity of
findings reflect their own feelings.
the findings.
Fig. 2 showcases the importance of AI on motivation, self-efficacy,
4. Results
collaborative writing tendency, and engagement.
Fig. 2 showcases the importance of AI on motivation, self-efficacy,
4.1. Question 1 The impact of ChatGPT on writing
collaborative writing tendency, and engagement. According to Fig. 2,
the visual depiction of these differential effects across the key outcome
The first research question explored the impact of ChatGPT on
variables provides clear evidence of the advantages of the AI-assisted
writing, focusing on motivation for writing, writing engagement, self-
approach over the traditional writing practice alone. The AI-assisted
efficacy for writing, and collaborative writing tendency. Table 2 pre-
writing intervention was particularly effective in boosting the partici-
sents the descriptive statistics of the four factors examined in the survey
pants’ motivation for writing and their overall engagement with the
for the classroom writing and AI assistance group.
writing process, while also enhancing their self-efficacy and tendency
According to Table 3, the descriptive statistics reveal promising re-
for collaborative writing to a somewhat lesser but still meaningful
sults for the AI-assisted group. While the writing group showed modest
degree.
increases in their mean scores from pre-test to post-test across the
The interaction between time and group can be detected in Fig. 3.
measured constructs of motivation for writing, self-efficacy for writing,
The interaction plot between time and group can be detected in
collaborative writing tendency, and writing engagement, the AI-assisted
Fig. 3. The interaction plot shown in Fig. 3 clearly indicates a significant
group demonstrated more substantial improvements, with larger gains
interaction effect between time (pre-test vs. post-test) and group (AI-
in their mean scores. This suggests the AI-assisted writing intervention
Table 2
Descriptive statistics of the survey for the classroom writing and AI assistance group.
n M SD min max range skew kurtosis
Motivation for writing (pre-writing group) 21 3.467 0.376 2.8 4.2 1.4 0.054 − 1.110
Motivation for writing (post-writing group) 21 3.571 0.354 2.8 4 1.2 − 0.569 − 0.988
Self efficacy for writing (pre-writing group) 21 3.467 0.431 2.6 4 1.4 − 0.098 − 1.243
Self efficacy for writing (post-writing group) 21 3.648 0.334 3 4.2 1.2 − 0.114 − 1.235
Collaborative writing tendency (pre-writing group) 21 3.562 0.326 3 4 1 − 0.169 − 1.262
Collaborative writing tendency (post-writing group) 21 3.590 0.360 3 4 1 − 0.665 − 1.210
Writing engagement (pre-writing group) 21 3.486 0.393 2.8 4 1.2 − 0.012 − 1.430
Writing engagement (post-writing group) 21 3.495 0.383 2.6 4 1.4 − 0.455 − 0.711
Motivation for writing (pre-AI group) 24 3.150 0.522 2 4 2 − 0.343 − 0.828
Motivation for writing (post-AI group) 24 4.317 0.358 3.8 5 1.2 0.783 − 0.718
Self efficacy for writing (pre-AI group) 24 3.400 0.334 3 4 1 0.485 − 1.041
Self efficacy for writing (post-AI group) 24 4.008 0.492 3 5 2 − 0.046 − 0.210
Collaborative writing tendency (pre-AI group) 24 3.583 0.363 3 4 1 − 0.258 − 1.434
Collaborative writing tendency (post-AI group) 24 4.075 0.475 3 5 2 − 0.262 0.580
Writing engagement (pre-AI group) 24 3.342 0.579 2 4 2 − 0.925 0.045
Writing engagement (post-AI group) 24 4.233 0.499 3 5 2 0.092 − 0.153
5
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
Fig. 3. Interaction effects between time and group for the different survey factors.
assisted vs. writing group). This finding suggests that the effect of the AI- scores.
based intervention was significantly moderated by the testing time Table 4 demonstrates a significant effect of the AI group compared to
point. the writing group when examining the change scores (post-test minus
As shown in Table 3, the interaction effect between time and group pre-test) across the four key writing-related factors. These findings are
was significant. When we used ordinary least squares regression in Stata further supported by the visual representation in Fig. 4, which clearly
18.0 to model the mean scores of the four variables, the group effect was illustrates the more pronounced improvements for the AI-assisted par-
only significant for motivation, but not for self-efficacy, collaborative ticipants relative to the writing group.
writing tendency, and engagement. However, the time and group Results in Table 4 and Fig. 4, when using post-test scores subtracting
interaction effect was significant for all four factors. Considering the the pre-test scores, showed the effects of group membership (AI treat-
interaction effect between time and group, this interaction effect was ment vs. control) on four different outcome variables: motivation for
most pronounced. Specifically, the AI experimental group’s post-test writing, self-efficacy for writing, collaborative writing tendency, and
scores were significantly higher than the control group’s pre-test and writing engagement. The group coefficient of 1.062 is positive and
post-test scores, as well as the experimental group’s pre-test scores. highly significant (p < 0.01), indicating that the treatment group had
There was no significant difference between the control group’s pre-test significantly higher motivation for writing compared to the control
and post-test scores, or between the experimental group’s pre-test group, after controlling for other factors. The group coefficient of 0.427
6
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
Fig. 4. Increased gains in the four factors between the two groups.
7
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
with the revision process compared to feedback provided by the students may prioritize grammar and language correctness over the
teachers” because she liked “playing with different prompts, like cor- substance and content of their compositions, and they called for a fact
recting mistakes, listing differences in writing, and word choice”. that “teachers are not replaceable in instructing content and ideas,
Furthermore, other participants also reported positive experiences sharing experiences in writing”.
related to AI in writing. Some students mentioned that utilizing AI tools
created “an environment of continuous learning and improvement”. 5. Discussion
They appreciated the instant and personalized feedback offered by the
AI system, which allowed them to “make real-time adjustments to their The discussion was organized around the research questions,
writing and witness progress”. The strengthened engagement in the comparing the findings with previous studies and referencing relevant
writing course may facilitate them to explore more in their writing. theoretical frameworks.
8
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
self-efficacy, engagement, motivation, and collaborative writing ten- structural corrections, including creativity, coherence, and argumenta-
dencies, thereby contributing to a more effective and supportive tion. Future studies can cover those aspects.
learning environment. Despite these limitations, the present study addresses a significant
gap in understanding EFL learners’ perspectives by demonstrating the
5.2. Students’ perceptions and experiences of AI-assisted proofreading practical and guided implementation of AI tools in EFL writing classes.
The findings suggest implications for EFL writing teachers to integrate
Despite some concerns raised by scholars regarding the use of ChatGPT into their teaching practices. Teachers should enhance writing
ChatGPT for maintaining academic integrity (Javier & Moorhouse, motivation and engagement by incorporating AI feedback cycles and
2023; Kohnke et al., 2023; Teng, 2023), overall, learners expressed gamifying writing tasks. This approach can boost students’ confidence
positive perceptions and experiences when utilizing AI assistance for and self-efficacy through immediate, constructive feedback and self-
writing. In line with Allen and Mizumoto (2024), learners found assessment opportunities. Collaborative writing tendencies can also be
ChatGPT’s feedback to be beneficial and directly applicable to their fostered by using ChatGPT in group projects and peer review sessions,
writing, confirming its effectiveness as a tool for proofreading and where students refine their drafts with AI-generated feedback before
editing. The majority of participants appreciated the accuracy, reli- engaging in peer discussions. Teachers play an active role in AI-assisted
ability, and ease of integrating ChatGPT’s corrections into their work. courses, supplementing ChatGPT’s feedback with personalized, empa-
They also found specific prompts, such as summarizing corrections, thetic guidance to create a supportive learning environment. A student-
helpful in identifying grammar, word choice, spelling, and structural centric approach, with personalized learning paths and customizable
errors (Barrot, 2023; Huang, 2023). feedback, can address individual writing challenges more effectively. By
However, EFL learners also acknowledged certain limitations in adopting these strategies, teachers can harness ChatGPT’s potential to
ChatGPT’s feedback features. They noted its overly formal nature, lack enhance students’ writing skills while fostering a balanced and ethical
of human touch, difficulty in understanding and comprehending re- learning environment.
sponses, provision of irrelevant comments, and challenges in identifying
specific writing problems. When comparing ChatGPT’s feedback with Ethical approval
that of teachers, participants had mixed feelings. On one hand, they
perceived teachers’ feedback as superficial, while considering ChatGPT The study was approved by an ethical committee with ID: RP–02/
to be more powerful in its feedback capabilities. On the other hand, they 2023. Informed consent was obtained from all participants, and their
expressed dissatisfaction with ChatGPT’s inability to provide the human privacy rights were strictly observed.
touch that teachers offer, including sharing their own feelings and ex-
periences in writing. These observations highlight that while ChatGPT CRediT authorship contribution statement
appears to be effective in generating feedback for writing, it cannot fully
replace the role of teachers. It is important for teachers to evaluate and Mark Feng Teng: Writing – review & editing, Writing – original
supplement machine-generated feedback, even if the machine, like draft, Visualization, Supervision, Software, Resources, Methodology,
ChatGPT, is highly capable. Teachers should be an integral part of AI- Investigation, Funding acquisition, Formal analysis, Data curation,
assisted writing courses, where ChatGPT serves as “a companion, not Conceptualization.
enemies”.
Declaration of competing interest
6. Concluding remarks
The authors declare that they have no known competing financial
The findings provide valuable insights into students’ preferences and interests or personal relationships that could have appeared to influence
experiences regarding the use of AI tools for enhancing their written the work reported in this paper.
assignments through guided editing and proofreading prompts. The
results indicate that ChatGPT was the preferred choice among students
Acknowledgment
due to its ability to provide specific, timely, and easily implementable
feedback. Moreover, the survey data supports the positive impact of AI
ChatGPT has been used for proofreading the language in this article.
assistance on writing motivation, self-efficacy, engagement, and
However, it should be noted that ChatGPT was not involved in gener-
collaborative writing tendency.
ating any ideas. Any errors, if present, are solely mine.
However, there were still some limitations. Firstly, the research was
conducted in a specific context with a limited number of participants,
Appendix A. Supplementary data
which restricts the generalizability of the findings. Secondly, the study
only focused on two assignments within a single course, which may have
Supplementary data to this article can be found online at https://doi.
influenced the overall outcomes. Thirdly, the reliance on self-reported
org/10.1016/j.caeai.2024.100270.
data through questionnaires and interviews may introduce bias, as
participants’ responses could be influenced by social desirability or
References
other factors. In addition, the questionnaire could benefit from addi-
tional items to capture a broader range of students’ experiences and Bandura, A. (1997). Self-efficacy: The exercise of control. W.H. Freeman.
preferences. The interview sample size was small, consisting of only Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials.
eight participants from the ChatGPT group. To gain a more compre- Assessing Writing, 57, Article 100745. https://doi.org/10.1016/j.asw.2023.100745
Borenstein, J., & Howard, A. (2021). Emerging challenges in AI and the need for AI ethics
hensive understanding, further analysis should be conducted to examine education. AI and Ethics, 1, 61–65. https://doi.org/10.1007/s43681-020-00002-7
the types of feedback students received from both ChatGPT and another Collie, R. J., Martin, A. J., & Curwood, J. S. (2016). Multidimensional motivation and
writing class, covering aspects such as grammar, vocabulary, and sen- engagement for writing: Construct validation with a sample of boys. Educational
Psychology, 36(4), 771–791.
tence structure. Fourth, there was no comparison made with other AI
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and
tools or traditional methods of feedback. Such comparison in future mixed methods approaches (5th ed.). London: Sage.
studies provide a more comprehensive understanding of ChatGPT’s Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human
unique advantages or disadvantages. Finally, Feedback types examined behavior. Springer.
Er, E., Dimitriadis, Y., & Gašević, D. (2021). Collaborative peer feedback and learning
in the present study were limited. Learners might benefit from exam- analytics: Theory-oriented design for supporting class-wide interventions. Assessment
ining a wider variety of feedback types beyond grammatical and & Evaluation in Higher Education, 46(2), 169–190.
9
M.F. Teng Computers and Education: Arti cial Intelligence 7 (2024) 100270
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of writing. Thinking Skills and Creativity, 52, Article 101524. https://doi.org/10.1016/j.
the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. tsc.2024.101524
https://doi.org/10.3102/00346543074001059 Stewart, J. (2023). Noam Chomsky says ChatGPT is a form of "high-tech plagiarism". My
Ghafouri, M., Hassaskhah, J., & Mahdavi-Zafarghandi, A. (2024). From virtual assistant Modern Met. https://mymodernmet.com/noam-chomsky-chatgpt/#:~:text=Chat
to writing mentor: Exploring the impact of a ChatGPT-based writing instruction GPT/20is/20basically/20high/2Dtech,to/20avoid/20doing/20-the/20work.
protocol on EFL teachers’ self-efficacy and learners’ writing skill. Language Teaching Storch, N. (2002). Patterns of interaction in ESL pair work. Language Learning, 52,
Research. https://doi.org/10.1177/1362168824123976 119–158.
Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and Teng, M. F. (2023). Scientific writing, reviewing, and editing for open-access TESOL
instructed language learning. Language, Learning and Technology, 26, 5–24, 10125/ journals: The role of ChatGPT. International Journal of TESOL Studies, 5, 87–91.
73474. https://doi.org/10.58304/ijts.20230107
Guo, K., Chen, X., & Qiao, S. (2022). Exploring a collaborative approach to peer feedback Teng, M. F. (2024). A systematic review of ChatGPT for English as a foreign language
in EFL Writing: How do students participate? RELC Journal. https://doi.org/ writing: Opportunities, challenges, and recommendations. International Journal of
10.1177/00336882221143192 TESOL Studies, 6(3), 36–57. https://doi.org/10.58304/ijts.20240304
Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potential Teng, M. F., & Ma, M. (2024). Assessing metacognition-based student feedback literacy
to support teacher feedback in EFL writing. Education and Information Technologies, for academic writing. Assessing Writing, 59, Article 100811. https://doi.org/
29, 8435–8463. https://doi.org/10.1007/s10639-023-12146-0 10.1016/j.asw.2024.100811
Huang, J. (2023). Engineering ChatGPT prompts for EFL writing classes. International Teng, M. F., & Teng, L. S. (2024). Validating the multi-dimensional structure of self-
Journal of TESOL Studies, 5(4), 73–79. https://doi.org/10.58304/ijts.20230405 efficacy beliefs in peer feedback for L2 writing: A bifactor-exploratory structural
Hyland, K. (1990). Providing productive feedback. ELT Journal, 44(4), 279–285. https:// equation modeling approach. Research Methods in Applied Linguistics, 3(3), 100136.
doi.org/10.1093/elt/44.4.279 https://doi.org/10.1016/j.rmal.2024.100136
Javier, D. R. C., & Moorhouse, B. L. (2023). Developing secondary school English Teng, M. F., & Wang, C. (2023). Assessing academic writing self-efficacy belief and
language learners’ productive and critical use of ChatGPT. TESOL Journal. https:// writing performance in a foreign language context. Foreign Language Annals, 56(1),
doi.org/10.1002/tesj.755 144–169. https://doi.org/10.1111/flan.12638
Kasneci, E., et al. (2023). ChatGPT for good? On opportunities and challenges of large Teng, M. F., & Yang, Z. (2023). Metacognition, motivation, self-efficacy belief, and
language models for education. Learning and Individual Differences, 103, Article English learning achievement in online learning: Longitudinal mediation modeling
102274. https://doi.org/10.1016/j.lindif.2023.102274 approach. Innovation in Language Learning and Teaching, 17(4), 778–794. https://doi.
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and org/10.1080/17501229.2022.2144327
learning. RELC Journal. https://doi.org/10.1177/00336882231162868 Teng, M. F., & Zhan, Y. (2023). Assessing self-regulated writing strategies, self-efficacy,
Kostka, I., & Toncelli, R. (2023). Exploring applications of ChatGPT to English language task complexity, and performance in English academic writing. Assessing Writing, 57,
teaching: Opportunities, challenges, and recommendations. TESL-EJ, 27(3). https:// Article 100728. https://doi.org/10.1016/j.asw.2023.100728
doi.org/10.55593/ej.27107int Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.
Ling, G., Elliot, N., Burstein, J. C., McCaffrey, D. F., MacArthur, C. A., & Holtzman, S. Harvard University Press.
(2021). Writing motivation: A validation study of self-judgment and performance. Weissman, J. (2023). ChatGPT is a plague upon education (opinion). Inside Higher Ed.
Assessing Writing, 48, Article 100509. https://www.insidehighered.com/views/2023/02/09/chatgpt-plague-uponeducatio
McMartin-Miller, C. (2014). How much feedback is enough?: Instructor practices and n-opinion.
student attitudes toward error treatment in second language writing. Assessing Zacharias, N. T. (2007). Teacher and student attitudes toward teacher feedback. RELC
Writing, 19, 24–35. https://doi.org/10.1016/j.asw.2013.11.003 Journal, 38(1), 38–52.
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language Zhang, Z. V. (2020). Engaging with automated writing evaluation (AWE) feedback on L2
model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), writing: Student perceptions and revisions. Assessing Writing, 43, Article 100439.
Article 100050. https://doi.org/10.1016/j.rmal.2023.100050 https://doi.org/10.1016/j.asw.2019.100439
Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of Allen, T., & Mizumoto, A. (2024). ChatGPT over my friends: Japanese EFL learners’
ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in preferences for editing and proofreading strategies. RELC Journal. https://doi.org/
Applied Linguistics. https://doi.org/10.1016/j.rmal.2024.100116 10.1177/00336882241262533
OpenAI. (2024). ChatGPT: Optimizing language models for dialogue. https://openai.
com/blog/chatgpt/.
Mark Feng Teng, Ph.D., is Associate Professor at Macao Polytechnic University. He was the
Rahimi, M. (2013). Is training student reviewers worth its while? A study of how training
recipient of the 2017 Best Paper Award from the Hong Kong Association for Applied
influences the quality of students’ feedback and writing. Language Teaching Research,
Linguistics (HAAL), 2023 Best Paper Award in social sciences from Education Ministry in
17, 67–89.
China. His research portfolio mainly focuses on computer-assisted L2 vocabulary acqui-
Rollinson, P. (2005). Using peer feedback in the ESL writing class. ELT Journal, 59(1),
sition and L2 writing. His publications have appeared in international journals, including
23–30.
Applied Linguistics, TESOL Quarterly, Language Teaching Research, System, Applied Linguistics
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional
Review, Computer Assisted Language Learning, Computers & Education, Foreign Language
assessments in higher education? Journal of Applied Learning and Teaching, 6(1).
Annals, and IRAL, among others. His recent monographs were published by Routledge,
https://doi.org/10.37074/jalt.2023.6.1.9
Springer, and Bloomsbury. He also edited and co-edited special issues for international
Ruegg, R. (2015). Differences in the uptake of peer and teacher feedback. RELC Journal,
journals, including Journal of Writing Research, Studies in Second Language Learning and
46(2), 131–145.
Teaching, and TESOL Journal, etc. He serves as editor-in-chief for International Journal of
Shen, X., & Teng, M. F. (2024). Three-wave cross-lagged model on the correlations
TESOL Studies (IJTS). E-mail: [email protected]
between critical thinking skills, self-directed learning competency and AI-assisted
10