Determining the Efficacy Written Feedback
Ryan L. Wiley, Russell T. Rodrigo
Stamford International University, Bangkok, Thailand
ryan.wiley@stamford.edu, Russell.rodrigo@stamford.edu
Abstract
There is a wide body of literature showing that quality teacher feedback is directly related to academic success. What is not so clearly defined, however, is what exactly constitutes “quality” feedback, particularly when considering written corrective feedback strategies in ESL environments. Also unclear is whether or not established forms of feedback are equally as effective when applied in different cultural contexts. The aim of this study was to add to the existing body of knowledge associated with the efficacy of written corrective feedback and to examine its effectiveness with consideration of the Thai cultural context within which it was delivered. The findings of the study showed that indirect, coded, unfocused written corrective feedback was effective at improving the writing accuracy of Thai university students. It was also discovered that 52% of the 2718 writing errors marked for correction were semantic mistakes. The results of this study should help both foreign and local teachers choose written corrective feedback strategies when teaching in Thai higher education learning environments. It also calls for further research in regards to the possible connection between written semantic mistakes, L1 interference, and electronic translators.
1. Introduction
It is well proven in the literature that effective feedback contributes to successful student achievement (Davis, S. E., & Dargusch, J. M., 2015). More than simple scores on formative evaluations, feedback includes adherence to accepted social practices and a responsibility for shared responsibilities between students and teachers (Davis, S. E., & Dargusch, J. M., 2015). As stated, the importance of feedback is well established. What feedback processes and procedures are most effective, however, is highly debated and not fully understood. With the established correlation between feedback and educational success, it is imperative that the most effective way of providing feedback is better understood.
The effectiveness of providing corrective feedback to improve the writing accuracy of second language learners has been widely debated in the literature. Most notably, Truscott (1996, 1999, 2007) argued that providing written corrective feedback was not only ineffective, but, in fact, detrimental to second language learners working to improve writing accuracy. He went so far as to recommend abandoning the technique altogether (Truscott, 1996).
Ferris (1999) subsequently argued that the Truscott’s analysis, of the studies by Kepner (1991), Semke (1984) and Sheppard (1992), which he used to support his argument, was overreaching and premature (Ferris, 1999). In his response to Ferris, Truscott (1999) accepted that the case should not be closed and that further research into what specific corrective feedback strategies were effective was warranted. Since these early debates, much new research has examined not just the overall effectiveness of corrective feedback, but many of the different strategies and techniques used to provide it.
2. Literature Review
Early research completed by Robb et al. (1986), Fathman & Whalley (1990), Ashwell (2000), and Ferris & Roberts (2001) successfully proved that written corrective feedback, in various forms, was an effective way to improve writing accuracy (Storch, N., 2010). These early studies, however, were criticized for examining accuracy performance only in revised texts. As they were performed, these early studies did not prove that actual learning had taken place, only that students could use written corrective feedback to edit and improve second drafts.
To address this problem, Truscott & Hsu (2008) completed a study that examined the accuracy of students’ writing not just on the rewrite of a first work, but also on the accuracy of the first draft of a subsequent work. The analyzed work was a narrative text. Half of the class received indirect feedback in the form of underlined mistakes, and the other half of the class received no feedback at all (Truscott, Hsu, 2008). As expected, the students who received feedback were successful at decreasing the number of errors they made in their final drafts (Truscott, Hsu, 2008). In a new assignment, completed one week later, however, the subjects error rates were nearly identical, showing that the decreased error rate in the revision of the previous assignment was not evidence that the indirect feedback improved learners writing abilities over time (Truscott, Hsu, 2008). This was in direct contradiction to the results of a previous study by Bitchener et al. (2005) that showed direct, explicit feedback and direct, explicit feedback with teacher/student meetings were both effective methods to improve writing accuracy over time. From these contradictory examples, it was clear that the question was not whether or not written corrective feedback was effective, but exactly what types of feedback are effective.
To continue narrowing the focus of corrective feedback research, Kassim & Ng (2014) produced a study testing the results of focused and unfocused feedback. Focused corrective feedback occurs when teachers or researchers provide corrective feedback to a limited number of writing errors, for example simple past tense, prepositions, and articles (Ellis et al., 2008; Ellis, 2009, Lee 2009). Unfocused feedback, on the other hand, occurs when all types of grammar mistakes are marked for feedback. (Ellis et al., 2008; Ellis, 2009; Lee 2009). Kassim & Ng’s (2014) study compared the effects of subjects receiving no feedback, focused feedback, and unfocused feedback. The study showed a significant difference between the group who received no feedback and the groups who received either focused or unfocused feedback (Kassim & Ng, 2014). Both groups that received feedback showed higher post feedback grammar accuracy when compared to the group that received no feedback. However, the study did not show a significant difference between the groups who received focused and unfocused feedback (Kassim & Ng, 2014), providing conflicting data to Ellis’ beliefs that focused feedback would enhance learning (Ellis, 2009).
Aghajanloo, et al., (2016) narrowed the focus of research even further by combining and comparing combinations of direct, indirect, focused, and unfocused corrective feedback. Specifically, they compared the effectiveness of focused direct, unfocused direct, focused indirect, and unfocused indirect corrective feedback (Aghajanloo, et al., 2016). Their study included a pre-test administered before the term and a post-test administered at the completion of the term (18 teaching sessions). The research showed that all combinations of corrective feedback significantly improved the writing accuracy of their subjects. It also showed that unfocused direct feedback was the most effective technique (Aghajanloo, et al., 2016). This point is far from agreed upon, however, with many, such as Bitchner & Knoch, 2008; Ferris, 1995: and Lalande, 1982, arguing that the process of solving one’s own errors, inherent in indirect feedback, is beneficial to the learning process.
Significance of the Study
While past research has thoroughly examined the effectiveness of different strategies for delivering corrective feedback, it is still not clear which methods are most effective. In addition, the greater body of literature focuses on the effectiveness of individual or combined feedback strategies with little consideration of the cultural context within which they were delivered. In this study, the researchers aim to add to the body of corrective feedback knowledge while also focusing on the types and frequencies of errors made, considering them in light of their cultural context, Thai university students. In order to do this, a 12-week study was administered to Thai university students in a B1 CEFR level English writing class. The data provided from this study will be helpful for future comparative culture studies and for local educational professionals searching for the most effective ways to deliver written corrective feedback.
Research Questions
- To what extent does indirect, coded, unfocused written corrective feedback improve writing accuracy for B1 CEFR level Thai ESL students?
- Which syntactic, semantic, and morphological errors do B1 CEFR level Thai ESL students commit most often?
Definitions of Terms
- Written corrective feedback is feedback written on essay assignments and returned to students for the purpose error correction
- Direct feedback is feedback that identifies mistakes and shows students exactly how to correct them
- Indirect feedback is feedback that identifies mistakes but does not show students how to correct them.
- Coded feedback is indirect feedback that uses codes to identify mistakes (ie. “s/v” identifies a subject/verb agreement mistake).
- Focused feedback is feedback that covers a limited number or errors (ie. only direct articles, subject/verb agreement, and preposition mistakes will be noted)
- Unfocused feedback is feedback that identifies all mistakes in a written work
- Syntactic errors are errors in sentence structure (ie. fragments, run-on sentences, and comma splices)
- Semantic errors are errors in word choice (ie. I ride my car to work)
- Morphological errors are errors in word formation (ie. Run is healthy.)
3. Methods
3.1 Learning Environment and Participants
This study occurred in a bilingual university in Bangkok, Thailand. In this program, students complete the first half of their degrees in their native language, Thai. At the same time, they are enrolled in an intensive English program. At the completion of this intensive English program, participants switch to studying their undergraduate courses purely in English. The 16 participants in this study were all native Thai speakers enrolled in the abovementioned bilingual undergraduate program. They were 2nd year university students, and their English course was a 12-week B1 CEFR level Reading and Writing class. The researcher was the instructor for this course.
3.2 Design
6 works were analyzed in the study, 2 written homework assignments that each had 1st and 2nd drafts and 2 in-class writing assessments, a mid-term and final examination. Table 1 below displays the errors analyzed for each work.
Table 1 Writing Errors Marked for Feedback
| Syntactic | Morphological | Semantic |
| Fragments Run-on sentences Comma splices Word order | Noun/verb form Noun/adjective form Subject/verb agreement Verb tense Singular/plural Verb/gerund | Articles Prepositions Extra words Wrong word |
Data Collection
3.21 Writing Tasks 1 & 2
Writing task 1 was a 4-paragraph cause and effect essay. The participants first received classroom instruction in regards to the purpose and structure of the writing genre. After that, in class, participants completed brainstorming and mind mapping activities and created outlines for their written works. The written works were then completed as homework assignments. Upon completion, participants handed the assignments in to the researcher, who then provided indirect, coded, unfocused feedback using an error correction symbol sheet (appendix 1).
The types and frequencies of syntactic, semantic, and morphological errors made were classified and documented and the assignments were handed back to the participants. The participants then used the feedback to re-write the assignments. They were handed them back to the researcher 1 week later. This 2nd draft was used as writing task 2, and, again, the types and frequencies of syntactic, semantic, and morphological errors were classified and recorded. At this stage, a final grade was given for the assignments.
3.22 Writing Task 3
Writing task 3 was an in-class, hand-written mid-term examination. On this exam, students were tasked with writing the same type of cause and effect essay they produced for writing task 1. A previously unannounced topic was assigned on exam day. During the exam, students were not allowed to use any electronic devices or other syntactic, semantic, and morphological aids or references. The exams were collected, and the types and frequencies of syntactic, semantic, and morphological errors made were classified and documented. A grade was given without the benefit of a re-write.
3.23 Writing Tasks 4 & 5
Writing task 4 was a compare and contrast essay, and the process of learning, preparing, producing, receiving feedback, and rewriting happened in the same way as it did for writing task 1 (see 3.21 writing task 1 above). The prewriting tasks were done in class with the help of the instructor. The production of the piece was completed as homework and then handed in to the researcher. The types and frequencies of syntactic, semantic, and morphological errors made were again classified and documented. Indirect, coded, unfocused written corrective was given. The essays were rewritten as writing task 5 and handed back to the instructor. The types and frequencies of syntactic, semantic, and morphological errors made were again classified and documented.
3.24 Writing task 6
Writing task 6 was an in-class, hand-written final examination. As with writing task 3, the mid-term examination, the final examination required students to produce the same type of essay they had just learnt, in this case, the abovementioned compare and contrast essay that was writing task 4. Again, they were not allowed to use syntactic, semantic, and morphological aids or references, and final grades were given without the benefit of feedback and a re-write.
4. Findings
Analysis involved the types and frequencies of errors made, measured using two quantitate variables: total error frequency by error type and comparative chronological accuracy performance. To ascertain the total error frequency by error type, 2718 errors originating from 96 written works by 16 students were identified, categorized and documented. The total errors by error type were then calculated as a percentage of the total number of errors (see table 1).
To find chronological accuracy performance, changes in writing accuracy in written works were compared. The number of errors made in assignments completed later in the course were taken as a percentage of the number of errors made in assignments completed earlier in the course. Where errors are calculated, A = assignment, D = draft, MT = mid-term exam and FT = final exam. The calculations were displayed as either positive or negative changes in accuracy.
Table 2 Error Rate Calculations
| Calculation 1 (table 3): (A1D2—A1D1) / A1D2 Calculation 2 (table 3): (A2D2 – A2D1) / A2D2 Calculation 3 (table 4): (A2D1 – A1D1) / A2D1 Calculation 4 (table 4): (A2D2 – A1D2) / A2D2 Calculation 5 (table 5): [(A2D1+A2D2) – (A1D1+A1D2)] / (A2D1+A2D2) Calculation 6 (table 6): (FE-ME) / FE |
Research Question 1
This study’s primary research question was, “To what extent does indirect, coded, unfocused written coded feedback improve writing accuracy for CEFR B1 level Thai ESL students?” This question was answered by making six comparisons and calculations. The first comparisons were between the 1st and 2nd drafts of assignment 1 and the 1st and 2nd drafts of assignment 2.
Table 3 Draft Comparisons
| Assignment 1 Draft 1 | Assignment 1 Draft 2 | accuracy improvement | Assignment 2 Draft 1 | Assignment 2 Draft 2 | accuracy improvement | |
| Writing Error Totals | 534 | 301 | 44% | 561 | 218 | 61% |
There were 534 combined errors made in the 1st drafts assignment 1 and 301 combined errors made in the 2nd drafts of assignment 1. The difference was 233 errors between drafts, a 44% improvement in accuracy between drafts 1 and 2 of assignment 1. In the 1st drafts of assignment 2 there was a combined total of 561 errors. In the 2nd drafts of assignment 2, there was a combined total of 218 errors, a 61% improvement in writing accuracy. The chart also shows a 44% accuracy improvement between drafts in assignment 1 and a 61% improvement in accuracy between drafts in assignment 2, a 17% improvement in participants’ abilities to improve accuracy from one draft to the next using indirect unfocused coded feedback. Next, accuracy improvement from the first drafts of assignment 1 to assignment 2 was measured.
Table 4 Assignment Comparisons
| Assignment 1 Draft 1 | Assignment 2 Draft 1 | accuracy improvement | Assignment 1 Draft 2 | Assignment 2 Draft 2 | accuracy improvement | |
| Writing Error Totals | 534 | 561 | -5% | 301 | 218 | 28% |
In the 1st draft of assignment 1, 534 errors were made. In the 1st draft of assignment 2, 561 errors were made, a 5% decrease in writing accuracy. There were 301 errors in the 2nd drafts of assignment 1 and 218 errors in the 2nd drafts of assignment 2, a 28% increase in writing accuracy between second drafts. The researchers next calculated the accuracy improvement of both drafts of assignments 1 and 2.
Table 5 Total Mistakes by Assignment
| Assignment 1 Draft 1 & 2 | Assignment 2 Drafts 1 & 2 | Accuracy Improvement | |
| Writing Error Totals | 835 | 779 | 7% |
The combined number of errors made in drafts 1 and 2 of assignment 1 was 835, and the combined number of errors in drafts 1 and 2 of assignment 2 was 779. There were 56 fewer total mistakes made in assignment 2, a 7% increase in overall writing accuracy. Next, the mid-term and final examinations were compared.
Table 6 Mid-Term & Final Exams
| Mid-Term Exam | Final Exam | Accuracy Improvement | |
| Writing Error Totals | 625 | 479 | 23% |
There were a total of 625 writing errors on the mid-term exam and 479 errors made on the final exam, a 23% increase in writing accuracy from the mid-term to the final exam.
Research Question 2
This study’s secondary research questions was, “Which syntactic, semantic, and morphological errors do CEFR B1 level Thai ESL students commit most often?
Table 7 Errors and Percentages
| Error Types | Total Number of Errors | Error Percentages |
| wrong word | 498 | 18% |
| singular / plural | 451 | 17% |
| articles | 418 | 15% |
| prepositions | 260 | 10% |
| extra word | 232 | 9% |
| subject/verb agreement | 145 | 5% |
| tense | 143 | 5% |
| gerund / verb | 141 | 5% |
| fragment | 118 | 4% |
| run on sentence | 97 | 4% |
| word order | 73 | 3% |
| noun / adjective (WF) | 59 | 2% |
| comma splice | 54 | 2% |
| noun / verb (WF) | 29 | 1% |
As table 5 (above) shows, the semantic mistake of using the wrong word accounted for 18% of total errors made. The morphological error, improper usage of the singular and plural forms, was the second most frequent error (17%), and another semantic mistake, the inappropriate use of articles, was the third most common error at 15%. Those three categories accounted for 50% of the 2718 errors made.
Table 8 Errors by Category
| Total Number of Errors | Error Percentages | |
| Semantic | 1408 | 52% |
| Morphological | 823 | 30% |
| Syntactic | 487 | 18% |
52% of the errors made were semantic, 30% were morphological, and 18% were syntactic.
5. Discussions
Early research completed by Robb et al. (1986), Fathman & Whalley (1999), Ashwell (2000), and Ferris & Roberts (2001), showed that written corrective feedback could be used to improve writing accuracy from 1st to 2nd drafts of single assignments (Storch, N., 2010). In this study, there was a 44% improvement between draft 1 and draft 2 on assignment 1 and a 61% increase in writing accuracy from draft 1 to draft 2 of assignment number 2. These results support the data provided by Robb et al. (1986), Fathman & Whalley (1999), Ashwell (2000), and Ferris & Roberts (2001), and others, and suggests that written indirect, coded, unfocused written corrective feedback is an effective way for Thai students to increase writing accuracy from draft to draft on individual assignments. Furthermore, the jump from a 44% draft-to-draft accuracy improvement on assignment 1 to a 61% draft- to-draft accuracy improvement on assignment 2 suggests that legitimate transfer took place, resulting in an increased ability for Thai students to use indirect, coded, unfocused feedback to improve writing accuracy over time.
Truscott & Hsu’s 2008 study showed that using written corrective feedback for two assignments written one week apart did not improve students’ writing accuracy. Bitchner et al. (2005) completed a study that showing that written corrective feedback was an effective way to increase writing accuracy from assignment to assignment. This study also produced conflicting results. The first draft of assignment 2 had 7% more errors than the first draft of assignment 1, suggesting that the indirect, coded, unfocused, written feedback was not an effective way of improving writing accuracy over time. In contrast, however, there was a 23% increase in accuracy between the week 6 mid term exam and the week 12 final examination, suggesting that the indirect, coded, unfocused written feedback was an effective way of improving writing accuracy. The latter evidence is especially compelling given that fact that both exams were given in controlled environments with no help from electronic devices.
In regards to our secondary research question, “Which syntactic, semantic, and morphological errors do CEFR B1 level Thai ESL students commit most often” Camilleri (2004) found, in an examination of 100 ESL essays, that the main cause of writing errors was direct translation from native languages into target languages. In this study, the most common error was using the wrong word. Notably, many of the wrong words that were used had no relation or similarity in meaning to the word that should have been used, suggesting that students were using their own knowledge or electronic devices to directly translate Thai words in to English, seeming to confirm the finding of Camilleri (2004). In addition, many of the syntactic errors showed little relation to proper English syntax. It did not appear to the researcher that students were trying and failing to use English syntax they had learnt, but rather that they were translating directly from Thai, even at the syntax level, again supporting conclusions drawn by Camilleri (2008)
6. Recommendations
A wide body of literature has proven that written corrective feedback is an effective way to increase writing accuracy. This study, while having showed a decrease in accuracy in one writing comparison, found evidence in 5 others that using indirect, coded, unfocused written corrective feedback was an effective way to improve the writing accuracy of Thai university ESL students at the B1 CEFR level. In light of that, teachers in this marketplace should continue or begin using these feedback methods.
In addition, the effectiveness of other individual and combinations of written corrective feedback types need to be used and studied in Thai cultural contexts. We know that feedback helps; we do not know exactly which type or combination of types provide the best results for Thai ESL students. To further explore this, the researcher suggests that academics in the Thai ESL field continue to replicate studies completed in other cultural settings in the Thai demographic.
Finally, while natural L1 interference, as described by Camilleri (2008), does affect the writing accuracy of second language learners, this researcher suspects that the problem is being compounded by electronic devices, both in that they discourage students from learning the new vocabulary needed to express themselves and in that they provide direct translations of words that are oftentimes incorrect, which very well may account for the fact that using the wrong word was the most common mistake made in the 96 written works analyzed in this study. Further research in regards to the usefulness of electronic translators in university environments is needed.
7. References
Aghajanloo, K., Mobini, F., & Khosravi, R. (2016). The Effect of Teachers’ Written Corrective Feedback (WCF) Types on Intermediate EFL Learners’ Writing Performance. Advances in Language and Literary Studies, 7(3), 28-37. Retrieved from http://www.journals.aiac.org.au/index.php/alls/article/view/2261
Ashwell, T. (2000). Patterns of teacher response to student writing in a multiple-draft composition classroom: Is content feedback followed by form feedback the best method? Journal of Second Language Writing, 9 (3): 227- 257.
Bitchener, J. Young, S. & Cameron, D. (2005). The effect of different types of corrective feedback on ESL student writing. Journal of Second Language Writing, 9, 227–58.
Bitchener, J., & Knoch, U. (2008). The value of written corrective feedback for migrant and international students. Language Teaching Research Journal, 12(3), 409-431
Camilleri, G. (2004). Negative Transfer in Maltese Students’ Writing in English. Journal of Maltese Education Research, 2(1), 3-12.
Davis, S. E., & Dargusch, J. M. (2015). Feedback, Iterative Processing and Academic Trust – Teacher Education Students’ Perceptions of Assessment Feedback. Australian Journal of Teacher Education, 40(1). http://dx.doi.org/10.14221/ajte.2015v40n1.10
Ellis, R. Sheen, Y. Murakami, M. & Takashima, H. (2008). The effects of focused and unfocused written corrective feedback in an English as foreign language context. System. 36: 353–371.
Fathman, A., & Whalley, E. (1990). Teacher response to student writing: Focus on form versus content. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 178–190). Cambridge: Cambridge University Press.
Ferris, D. R. (1995a). Can advanced ESL students be taught to correct their most serious and frequent errors? CATESOL Journal, 8, 41-62.
Ferris, Dana. (1999). The case for grammar correction in L2 writing classes: A response to Truscott (1996). Journal of Second Language Writing. 8. 1-11. 10.1016/S1060-3743(99)80110-6.
Ferris, D. R., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161–184.
Kassim, A. & Ng, L.L. (2014). The Roles of Collaborative Dialogue in Enhancing Written Corrective Feedback Efficacy. Malaysian Journal of ELT Research, Vol. 10(1), pp. 16-30.
Kepner, C. G. (1991). An Experiment in the Relationship of Types of Written Feedback to the Development of Second-Language Writing Skills. The Modern Language Journal,75(3), 305. doi:10.2307/328724
Lalande, J. F. (1982). Reducing composition errors: an experiment. Modern Language Journal, 66, 140-149.
Lee, Soon Yeun, “The Efficacy of Dynamic Written Corrective Feedback on Intermediate-high ESL Learners’ Writing Accuracy” (2009). All Theses and Dissertations. 2304.
https://scholarsarchive.byu.edu/etd/2304
Storch, N. (2010). Critical Feedback on Written Corrective Feedback Research. International Journal of English Studies, 10(2), 29-46. https://doi.org/10.6018/ijes/2010/2/119181
Semke Ph.D, Harriet. (1984). Effects of the Red Pen. Foreign Language Annals. 17. 195 – 202. 10.1111/j.1944-9720.1984.tb01727.x.
Robb, T., Ross, S., & Shortreed, I. (1986). Salience of feedback on error and its effect on EFL writing quality. TESOL Quarterly, 20, 83–93.
Truscott, J. (1996). The Case Against Grammar Correction in L2 Writing Classes. Language Learning,46(2), 327-369. doi:10.1111/j.1467-1770.1996.tb01238.x
Truscott, J. (1999). The case for ‘‘the case for grammar correction in L2 writing classes’’: A response to Ferris. Journal of Second Language Writing, 8, 111–122.
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of Second Language Writing,16(4), 255-272. doi:10.1016/j.jslw.2007.06.003
Truscott, J., & Hsu, A. Y. (2008). Error correction, revision, and learning. Journal of Second Language Writing, 17, 292–305.