Article Sidebar

Download
PDF
Statistic
Read Counter : 0 Download : 0

Main Article Content

Abstract

Rater-mediated scoring is central to performance assessment, yet high reliability alone does not show how dependable scores were produced. This study examines the quality-control architecture and documented inter-rater agreement of a locally developed Aviation English oral-aural performance assessment used with 34 fourth-semester Air Traffic Control students in Indonesian vocational higher education. The assessment represented six analytic dimensions: pronunciation, structure, vocabulary, fluency, comprehension, and interaction. Two raters coded R1 and R2 completed a six-hour calibration session on 25 June 2026 using 12 anchor recordings; the calibration ICC was .91 against a prespecified criterion of .80. Main scoring used blinded, randomized recordings, analytic descriptors, anchor-based drift checks, and an adjudication rule for material disagreement. The verified report documents 408 participant-time-dimension rating units and an overall ICC(2,1)=.982, 95% CI [.979, .985], with a mean absolute rater difference of 1.39 points on the 0-100 analysis scale. Results indicate very high pooled agreement but do not establish dimension-specific reliability, test-form equivalence, official ICAO proficiency levels, or operational licensing competence. Rater dependability in safety-sensitive language assessment is therefore an evidence chain linking construct definition, descriptors, calibration, blinded scoring, monitoring, and bounded interpretation rather than a single coefficient.

Keywords

Aviation English inter-rater reliability performance assessment rater calibration blind scoring

Article Details

References

  1. Alderson, J. C. (2009). Air safety, language assessment policy, and policy implementation: The case of aviation English. Annual Review of Applied Linguistics, 29, 168–187. https://doi.org/10.1017/S0267190509090138
  2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
  3. Assassi, T., & Ghodbane, T. (2023). The recurring issue of aviation English test validity: Echoes from test-takers and assessors of the English for Aviation Language Testing System in Algeria. English Studies at NBU, 9(2), 239–269. https://doi.org/10.33919/esnbu.23.2.6
  4. Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford University Press.
  5. Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
  6. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284
  7. Davis, L. (2016). The influence of training and experience on rater performance in scoring spoken language. Language Testing, 33(1), 117–135. https://doi.org/10.1177/0265532215582282
  8. Douglas, D. (2000). Assessing languages for specific purposes. Cambridge University Press.
  9. Eckes, T. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.
  10. Estival, D., Farris, C., & Molesworth, B. (2016). Aviation English: A lingua franca for pilots and air traffic controllers. Routledge.
  11. Farris, C., Trofimovich, P., Segalowitz, N., & Gatbonton, E. (2008). Air traffic communication in a second language: Implications of cognitive factors for training and assessment. TESOL Quarterly, 42(3), 397–410. https://doi.org/10.1002/j.1545-7249.2008.tb00138.x
  12. Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. https://doi.org/10.1177/1948550617693063
  13. Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2012). Observer bias in randomised clinical trials with binary outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. BMJ, 344, e1119. https://doi.org/10.1136/bmj.e1119
  14. International Civil Aviation Organization. (2009). Guidelines for Aviation English training programmes (Circular 323-AN/185). ICAO.
  15. International Civil Aviation Organization. (2010). Manual on the implementation of ICAO language proficiency requirements (Doc 9835, 2nd ed.). ICAO.
  16. International Civil Aviation Organization. (2022). Annex 1 to the Convention on International Civil Aviation: Personnel licensing (14th ed.). ICAO.
  17. International Civil Aviation Organization. (2023). Annex 10 to the Convention on International Civil Aviation: Aeronautical telecommunications, Volume II—Communication procedures including those with PANS status. ICAO.
  18. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
  19. Kim, H. (2018). What constitutes professional communication in aviation: Is language proficiency enough for testing purposes? Language Testing, 35(3), 403–426. https://doi.org/10.1177/0265532218758127
  20. Kim, H., & Elder, C. (2009). Understanding aviation English as a lingua franca: Perceptions of Korean aviation personnel. Australian Review of Applied Linguistics, 32(3), 23.1–23.17. https://doi.org/10.2104/aral0923
  21. Kim, Y.-h. (2023). The challenges of radiotelephony communication and effective training approaches: A study of Korean pilots and air traffic controllers. English for Specific Purposes, 72, 26–39. https://doi.org/10.1016/j.esp.2023.07.001
  22. Knoch, U. (2014). Using subject specialists to validate an ESP rating scale: The case of the International Civil Aviation Organization (ICAO) rating scale. English for Specific Purposes, 33, 77–86. https://doi.org/10.1016/j.esp.2013.08.002
  23. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
  24. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(1), 54–71. https://doi.org/10.1177/026553229501200104
  25. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30
  26. McNamara, T. F. (1996). Measuring second language performance. Longman.
  27. McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
  28. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education/Macmillan.
  29. Moder, C. L., & Halleck, G. B. (2009). Planes, politics and oral proficiency: Testing international air traffic controllers. Australian Review of Applied Linguistics, 32(3), 25.1–25.16. https://doi.org/10.2104/aral0925
  30. Munro, M. J., & Derwing, T. M. (1995). Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 45(1), 73–97. https://doi.org/10.1111/j.1467-1770.1995.tb00963.x
  31. Neittaanmäki, R., & Lamprianou, I. (2024). All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency. Language Testing, 41(3), 606–626. https://doi.org/10.1177/02655322241239362
  32. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
  33. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
  34. Treadaway, M., & Read, J. (2024). Setting standards for a diagnostic test of Aviation English for student pilots. Language Testing, 41(3), 557–583. https://doi.org/10.1177/02655322231224051
  35. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15(2), 263–287. https://doi.org/10.1177/026553229801500205
  36. Wind, S. A., & Peterson, M. E. (2018). A systematic review of methods for evaluating rating quality in language assessment. Language Testing, 35(2), 161–192. https://doi.org/10.1177/0265532216686999
  37. Yin, Y., Yu, G., & Huang, D. (2024). Implementing the ICAO Language Proficiency Requirements: The localization of Aviation English testing. Applied Linguistics, 45(4), 699–717. https://doi.org/10.1093/applin/amad057