Rater Cognition and Errors - A Corpus-based Approach to Validating Writing Assessment

The goal of the research was to investigate the possibility of establishing a meaningful link between rater judgments made when rating writing performances and the instances of errors occurring in the same writing samples. This is of special interest given the strong philosophical preference in the contemporary writing assessment paradigm for the judgment process to focus predominantly on positive features. By investigating whether or not such a link can be shown to exist in an L2 rating context not explicitly focused on errors, the study aimed first to obtain a better understanding of the rating process itself. This knowledge provides a methodological contribution on the basis of which validity arguments can be made. Auxiliary findings relating to the use of the results in determining writing task difficulty in language teaching, and in language testing more broadly, were also explored.

The goal was achieved through the examination of 60 writing samples generated by the Fachprüfung 3 high-stakes exam in the English Department of the University of Klagenfurt (Austria). The writing performances at the core of the study were collected between 2015 and 2017 and rated, following a procedure unrelated in any overt manner to error counting, on the basis of an analytic rating scale informed by CEFR, consisting of four dimensions, Task Achievement, Textual Competence, Grammar, and Vocabulary, as well as aggregated and agreed-upon Total ratings. The rating was carried out by a team of trained raters, all of whom were practicing teachers with teaching experience at university level. This study focused on errors found in those sample texts, with the aim of ascertaining to what extent error analysis and error counts predict the ratings these writing samples were independently awarded.

The findings indicate a high level of correlation between the ratings awarded and the error counts. The results are corroborated by theoretical findings from the field of rater cognition, related to the nature of the rating process and from the field of second language acquisition (SLA), related also to the role errors play in terms of language competence. The local validity of the findings, together with the strong theoretical underpinnings from corroborating research, suggest that the results lend themselves to a more global interpretation and to consideration in wider arguments regarding validation.

project lead: Nikola Dobric

period: 2016-2021