Document Type

Conference Proceeding

Publication Date



Background and Context. There is a constant, demonstrated need for valid and reliable assessments in computing education research. While there exist assessments at a course-based level (e.g., CS1, Data Structures, Discrete math, etc.), instructors and researchers would also like concept-based subscales that are more fine-grained. However, assessments designed and validated at the course level need additional work to determine whether they can reliably and validly measure individual concepts.

Objectives. In this paper, we explore the content and factor structure of an existing CS1 assessment, the Second CS1 (SCS1) assessment, which consists of nine CS1 concepts and three question types (definitional, code tracing, and code completion). We investigate the underlying structure of the assessment in terms of these concepts and question types to determine whether the assessment could be separated into subscales.

Method. We used a mixed-methods approach to answer our research question. We first investigated the designer-intended subscales of concepts and question types using confirmatory factor analysis, constructing multiple models using data from 547 students.We then qualitatively coded the assessment using an established framework to better understand our findings. Finally, we used a combination of exploratory and confirmatory factor analysis to take a data-driven approach to understanding the underlying factor structure.

Findings. We argue that SCS1 cannot discretely measure student knowledge in terms of the concepts or question types. However, more work is needed before the assessment can be split into reliable, valid, and useful subscales.

Implications. We discuss the future work needed to create a CS1 assessment to fit the needs of the computer science education community; namely, the need to revise and expand SCS1 to more appropriately measure its intended constructs and to be used outside of controlled experiments.


Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). © 2023 Copyright held by the owner/author(s).


Creative Commons License

Creative Commons Attribution-NonCommercial 4.0 International License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License