The Comparision of Reliability of Pisa Mathematical Literacy Items' Competency Needs Scores Obtained With Comptency Scheme Based on Generalizability Theory And Classical Test Theory

---

Thesis Type: Postgraduate

Institution Of The Thesis: Gazi University, Eğitim Bilimleri Enstitüsü, Eğitim Bilimleri Enstitüsü, Turkey

Approval Date: 2016

Thesis Language: Turkish

Student: FATMA KIZILTOPRAK

Supervisor: MEHTAP ÇAKAN

Open Archive Collection: AVESIS Open Access Collection

Abstract:

The purpose of this study is to investigate the criterion and construct validity and reliability of the scores obtained by using Competency Scheme, which is improved for determining PISA mathematical literacy items' competency needs according to PISA mathematical literacy framework, based on generalizability theory and classical test theory. With this purpose initially the adaptation study of competency scheme was conducted. When experts' views in this context was examined, it was seen that the translated form was compatible with the language, culture and mathematical education jargon. All of the published PISA mathematical literacy items were scored at two different times with a month of interval by five independent raters who were PhD candidates in mathematical education field. With the purpose of determining the validity of scores, criterion validity and structural validity were examined. In the context of criterion validity, PISA mathematical literacy items' empirical difficulty scores were chosen as the predicted variable. Classical Test Theory and Generalizability Theory were utilized with the purpose of determining the reliability of scores. In CTT framework, intra-rater and inter rater reliability were examined. In the framework of G theory, completely crossed designs via univariate and multivariate G and D studies were conducted. According to results, scores which were obtained via Competency Scheme explain %74,7 of the empirical difficulty scores' variability significantly. Besides, the conducted factor analysis confirmed the theoretical model. In the framework of CTT, it was seen that correlation coefficients which were calculated for intra-rater reliability differed from each other in terms of raters and categories and there is also a high reliability among the raters in the context of inter-rater reliability. At the end of the univariate and multivariate G and D studies, it was seen that the reliability coefficients which were predicted for scores obtained from five raters at two different times are higher than the acceptable levels. Furthermore, G and Phi coefficients which were estimated by multivariate model G and D studies are higher than G and Phi coefficients which were estimated by univariate G and D studies. It was found that in CTT and G theory frameworks the reliability levels are compatible with each other.