Examination of Parameter Estimation Using Recursive Bayesian Analysis in Simulated Item Response Theory Applications by Robert Hendrick

For the past several years, high-stakes testing has been the predominant indicator used to assess students' academic ability. School systems, teachers, parents, and students are dependent upon the accuracy of academic ability estimates designated, θs, by item response theory (IRT) computer programs. In this study, the accuracy of 3 parameter logistic (3PL) IRT estimates of academic ability were obtained from the BILOG-MG and WinBUGS computer programs which were employed to compare the use of non-informative and informative priors in θ estimation. The rationale for comparing the output of these two computer programs is that the underlying statistical theory employed in these two computer programs is different, and there may be a notable difference in the accuracy of θ estimation when an informative prior is used by WinBUGS in analyzing skewed populations. In particular, the θ parameter estimates of BILOG-MG using traditional IRT analysis with non-informative priors in each situation and the θ parameter estimates of WinBUGS using Recursive Bayesian Analysis (RBA) with informative priors are compared to the true simulated θ value using Root Mean Square Errors (RMSEs). To make this comparison, Monte Carlo computer simulation is used across three occasions within three conditions giving nine comparison situations. For the priors and data generated, results show similar θ estimation accuracy for a normally distributed latent trait (RMSE = 0.35), a more accurate θ estimation process using RBA compared to traditional analysis (RMSEs of 0.36 compared to 0.76) when using latent trait distributions skewed in a similar direction, and less accurate θ estimation using RBA compared to traditional analysis (RMSEs of 1.48 compared to 0.80) when using extremely skewed negative then positive distributions in a longitudinal setting. Implications for further research include extensions to other IRT models, developing prior elicitation equations, and applying Bayesian informative prior elicitation in BILOG-MG.