Automated Scoring in the Era of Artificial Intelligence: An Empirical Study with Turkish Essays
Automated scoring (AS) has gained significant attention as a tool to enhance the efficiency and reliability of assessment processes. Yet, its application in low-resource languages, such as Turkish, remains limited. This study addresses this gap by empirically evaluating AS for Turkish using a zero-shot approach with a rubric powered by OpenAI’s GPT-4o. A dataset of 590 essays written by learners of Turkish as a second language was scored by professional human raters and an artificial intelligence (AI) model integrated via a custom-built interface. The scoring rubric, grounded in the Common European Framework of Reference for Languages, assess
Resource information
- Authors
- AYDIN, Burak, TARIK KIŞLA, Elmas, Nursel Tan, Bulut, Okan
- Research area
- Artificial intelligence
- Research topics
- Education, Educational Assessment, Evaluation, and Research, Language and Literacy Education
- Publication date
- 2025
- DOI
- 10.17605/osf.io/qbck7
- Upstream last modified
- Mar 2, 2026
- Last record update
- Nov 5, 2026
About this resource
Automated scoring (AS) has gained significant attention as a tool to enhance the efficiency and reliability of assessment processes. Yet, its application in low-resource languages, such as Turkish, remains limited. This study addresses this gap by empirically evaluating AS for Turkish using a zero-shot approach with a rubric powered by OpenAI’s GPT-4o. A dataset of 590 essays written by learners of Turkish as a second language was scored by professional human raters and an artificial intelligence (AI) model integrated via a custom-built interface. The scoring rubric, grounded in the Common European Framework of Reference for Languages, assessed six dimensions of writing quality. Results revealed a strong alignment between human and AI scores with Pearson correlation of 0.73, and an overlap measure of 83.5%. Analysis of rater effects showed minimal influence on score discrepancies, though factors such as experience and gender exhibited modest effects. These findings demonstrate the potential of AI-driven scoring in Turkish, offering valuable insights for broader implementation in underrepresented languages. The limitations of the study, such as reliance on single-rater scores and focusing on a specific task type, highlight avenues for future research to expand and refine the proposed approach.