Title
Toward CS1 Content Subscales: A Mixed-Methods Analysis of an Introductory Computing Assessment
Author ORCID Identifier
Miranda Parker https://orcid.org/0000-0003-0066-2664
Matt Davidson: https://orcid.org/0000-0001-6848-2876
Yvonne Kao: https://orcid.org/0000-0002-4116-3856
Lauren Margulieux: https://orcid.org/0000-0002-8800-2398
Jan Vahrenhold: https://orcid.org/0000-0001-8708-4814
Document Type
Conference Proceeding
Publication Date
11-2023
Abstract
Background and Context. There is a constant, demonstrated need for valid and reliable assessments in computing education research. While there exist assessments at a course-based level (e.g., CS1, Data Structures, Discrete math, etc.), instructors and researchers would also like concept-based subscales that are more fine-grained. However, assessments designed and validated at the course level need additional work to determine whether they can reliably and validly measure individual concepts.
Objectives. In this paper, we explore the content and factor structure of an existing CS1 assessment, the Second CS1 (SCS1) assessment, which consists of nine CS1 concepts and three question types (definitional, code tracing, and code completion). We investigate the underlying structure of the assessment in terms of these concepts and question types to determine whether the assessment could be separated into subscales.
Method. We used a mixed-methods approach to answer our research question. We first investigated the designer-intended subscales of concepts and question types using confirmatory factor analysis, constructing multiple models using data from 547 students.We then qualitatively coded the assessment using an established framework to better understand our findings. Finally, we used a combination of exploratory and confirmatory factor analysis to take a data-driven approach to understanding the underlying factor structure.
Findings. We argue that SCS1 cannot discretely measure student knowledge in terms of the concepts or question types. However, more work is needed before the assessment can be split into reliable, valid, and useful subscales.
Implications. We discuss the future work needed to create a CS1 assessment to fit the needs of the computer science education community; namely, the need to revise and expand SCS1 to more appropriately measure its intended constructs and to be used outside of controlled experiments.
DOI
https://doi.org/10.1145/3631802.3631828
Recommended Citation
Miranda C. Parker, Matt J. Davidson, Yvonne S. Kao, Lauren E. Margulieux, Zachary R. Tidler, and Jan Vahrenhold. 2023. Toward CS1 Content Subscales: A Mixed-Methods Analysis of an Introductory Computing Assessment. In 23rd Koli Calling International Conference on Computing Education Research (Koli Calling ’23), November 13–18, 2023, Koli, Finland. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/3631802.3631828
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Comments
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). © 2023 Copyright held by the owner/author(s).