This study examined the measurement quality of an instructor-developed classroom assessment instrument (i.e., a final written test) used to assess linguistic knowledge covered in an undergraduate elementary Chinese course at a U.S. university. Participants were 222 learners enrolled in the Chinese course from Fall 2011 to Fall 2013. Analyses were performed on a subset of 64 binary-scored (0/1) test items. The 64 items showed acceptable overall test reliability, test discrimination, and Rasch model fit. Meanwhile, insufficient number of difficult items, below-threshold discriminatory power of certain items, and existence of measurement redundancy were also found. Strategies for improving the measurement quality of the test were discussed.