Investigation of Differential Item and Step Functioning Procedures in Polytomus Items

Abstract

This study aimed to compare differential item functioning (DIF) and differential step function (DSF) detection methods in polytomous items under various conditions. In this context, the study examined Kazakhstan and Turkey data obtained from the ICT Familiarity Questionnaire in PISA 2018. Mantel test, Liu-Agresti statistics, Cox β, and poly-SIBTEST methods were used for polytomous DIF analysis while Adjacent Category Logistic Regression Model and Cumulative Category Log Odds Ratio methods were used for DSF analysis. This study was carried out by using “differential category combining, focus group sample size, focus group: reference group sample ratio and DIF/DSF detection method”. SAS and R software were utilized in the creation of conditions; SIBTEST was used for poly-SIBTEST analysis and DIFAS programs were used for the other methods. Analyses demonstrated that the number of items with large DIF was higher in the small sample according to the polytomous DIF detecting methods. Likewise, the number of steps with large DSF is higher in large samples according to the DSF methods. However, it was found that the methods give more consistent results in large samples. During the steps, the DIF value was lower in the items containing DSF with the opposite sign; therefore, not performing DSF analysis on an item with no DIF may yield erroneous results. Although the differential category combining conditions created within the scope of the research did not have a systematic effect on the results, it was suggested to examine this situation in future studies, considering that the frequency of marking the combined categories differentiated the results.