Using OpenAI GPT to Generate Reading Comprehension Items

SAYIN A., Gierl M.

Educational Measurement: Issues and Practice, cilt.43, sa.1, ss.5-18, 2024 (SSCI) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 43 Sayı: 1
  • Basım Tarihi: 2024
  • Doi Numarası: 10.1111/emip.12590
  • Dergi Adı: Educational Measurement: Issues and Practice
  • Derginin Tarandığı İndeksler: Social Sciences Citation Index (SSCI), Scopus, Academic Search Premier, EBSCO Education Source, Education Abstracts, Educational research abstracts (ERA), ERIC (Education Resources Information Center), Psycinfo
  • Sayfa Sayıları: ss.5-18
  • Anahtar Kelimeler: automatic item generation, item development, reading comprehension
  • Gazi Üniversitesi Adresli: Evet

Özet

The purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template-based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT-3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject-matter experts. The generated items were also administered to a sample of 1,607 Grade-8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.