Item Development Process

A key part of this project’s work involves determining how CKT science items can be assembled into a valid and reliable assessment instrument to measure elementary teacher’s CKT proficiency in one science area: matter and its interactions. In our work, we have been using the principles of evidence-centered design (Mislevy & Risconscente, 2006) along with a process closely modeled after the item development work on the Measures of Effective Teaching (MET) Project (Phelps, Weren, Croft, & Gitomer, 2014) and an earlier NSF research project (Mikeska, Phelps, & Croft, 2017) to develop this CKT matter assessment and the individual CKT Matter Items. Our CKT assessment development process is illustrated in the figure below.

Item Development Process

As mentioned in the ‘CKT’ tab of this website, in earlier work we collaborated with a national advisory committee to create a science-specific CKT assessment framework – what we currently refer to as the ‘Work of Teaching Science’ (WOTS) framework – to detail the science-specific teaching practices that are most critical for beginning elementary science teachers (Mikeska, Kurzum, Steinberg, & Xu, 2018). The full WOTS framework  includes 27 science teaching practices (3 to 6 practices per instructional tool category) that research has indicated or hypothesized are critical for novice elementary science teachers to know how to do well to be effective practitioners. On this project, we have been using the WOTS framework to guide the development of an instrument to assess elementary pre-service teachers’ CKT about matter and its interactions. In addition, to better focus our assessment development efforts, we reviewed the performance expectations in the area of matter and its interactions at the elementary level and identified five key topic areas within matter and its interactions (properties of matter and their measurements; changes in matter; conservation of matter; model of matter; and materials) that serve as important instructional foci from kindergarten to fifth grade.

References

Mikeska, J.N., Kurzum, C., Steinberg, J., & Xu, J. (2018). Assessing elementary science teachers’ content knowledge for teaching science for the ETS Educator Series: Pilot results. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12207

Mikeska, J.N., Phelps, G., & Croft, A. (2017). Practice-based measures of elementary science teachers’ content knowledge for teaching: Initial item development and validity evidence. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12168

Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered design: Layers, concepts, and terminology. In Downing, S., & Haladyna, T. Mahway (Eds.), Handbook of test development. New Jersey: Erlbaum. Phelps, G., Weren, B., Croft, A., & Gitomer, D. (2014). Developing content knowledge for teaching assessments for the Measures of Effective Teaching study. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12031

Phelps, G., Weren, B., Croft, A., & Gitomer, D. (2014). Developing content knowledge for teaching assessments for the Measures of Effective Teaching study. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12031