Publication: Building an Initial Validity Argument for Binary and Analytic Rating Scales for an EFL Classroom Writing Assessment: Evidence from Many-Facets Rasch Measurement
Submitted Date
Received Date
Accepted Date
Issued Date
2022
Copyright Date
Announcement No.
Application No.
Patent No.
Valid Date
Resource Type
Edition
Resource Version
Language
en
File Type
No. of Pages/File Size
ISBN
ISSN
1513-5934 (Print), 2651-1479 (Online)
eISSN
DOI
Scopus ID
WOS ID
Pubmed ID
arXiv ID
item.page.harrt.identifier.callno
Other identifier(s)
Journal Title
rEFLections Journal
Volume
29
Issue
3
Edition
Start Page
675
End Page
699
Access Rights
Access Status
Rights
Rights Holder(s)
Physical Location
Bibliographic Citation
Research Projects
Organizational Units
Authors
Journal Issue
Title
Building an Initial Validity Argument for Binary and Analytic Rating Scales for an EFL Classroom Writing Assessment: Evidence from Many-Facets Rasch Measurement
Alternative Title(s)
Author(s)
Author’s Affiliation
Author's E-mail
Editor(s)
Editor’s Affiliation
Corresponding person(s)
Creator(s)
Compiler
Advisor(s)
Illustrator(s)
Applicant(s)
Inventor(s)
Issuer
Assignee
Other Contributor(s)
Series
Has Part
Abstract
Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative assessment and for paragraph writing assessment in a Thai EFL university classroom context. Specifically, this study applied an argument-based validation approach to build an initial validity argument for the rating scales with emphasis on the evaluation, generalization, and explanation inferences, and employed a many-facets Rasch measurement (MFRM) approach to investigate the psychometric functionalities of the rating scales which served as the initial validity evidence for the rating scales. Three trained teacher raters applied the rating scales to rate the same set of 51 opinion paragraphs written by English-major students. The rating scores were analysed following the MFRM psychometrics. Overall, the MFRM results revealed that (1) the rating scales largely generated accurate writing scores, supporting the valuation inference, (2) the raters were self-consistent in applying the rating scales, contributing to the generalization inference, (3) the rating scales sufficiently captured the defined writing construct, substantiating the explanation inference, and (4) the binary rating scale showed more desirable psychometric properties than the analytic rating scale. The present findings confirm the appropriate functioning and reasonable validity argument of the rating scales and highlight the greater potential of the binary rating scale to mitigate rater inconsistency and cognitive load in a formative classroom assessment.