blog




  • Essay / Differences in evaluator behavior - 1063

    Differences in evaluator behavior are among the factors responsible for variability in the decision-making process (DMP) during evaluations. The interference of the rating style or experience of the rater determines the validity and reliability of the rating and the rater themselves. Identifying and measuring factors related to rater inconsistency in the DMP is necessary to avoid factors underlying variability in the decision-making process. Several studies have identified raters' skill level, experiences, and tasks as factors. The purpose of this article is to critically review two articles that contribute to describing rater behaviors related to the factors studied. Barkaoui (2010) “Variability in ESL Essay Assessment Processes: The Role of Rating Scale and Rater Experience” identifies the effects of rating scales and experiences on rater behavior through a think-aloud protocol. While Baker (2012) “Individual differences in evaluator decision-making style: an exploratory mixed methods study” defines and addresses decision-making style (DMS) as a factor related to the decision-making process when writing the evaluation rating. The following summary and critical assessment of these articles provides explicitly what the reviewers' decisions imply and the strengths and weaknesses of these two articles. experiences in writing assessment (p.54).12 The ESL essays were evaluated holistically and analytically by 25 raters, both novice and experienced (Barkaoui, 2010, p.56). The results show that rating scales have a greater impact on rating processes, especially in analytical ratings (Barkaoui, 2010, p.56). ...... middle of paper ...... in rater training or rater profiling as well. As a final suggestion, the two studies discussed in this article successfully identify additional factors in the evaluator decision-making process and will be of great use to those interested in the area of ​​evaluator evaluations and review consistency. ratings. Works Cited Baker, BA (2012). Individual differences in evaluators' decision-making style: An exploratory mixed-methods study. Language Assessment Quarterly, 9(3), 225-248. Retrieved from http://search.proquest.com.ezp.lib.unimelb.edu.au/docview/1364738673?accountid=12372Barkaoui, K. (2010). Variability in ESL essay evaluation processes: The role of rating scale and evaluator experience. Language Assessment Quarterly, 7(1), 54-74. Retrieved from http://search.proquest.com.ezp.lib.unimelb.edu.au/docview/744444316?accountid=12372