Item Development Overview
On this project, we have been using the ‘Work of Teaching Science’ (WOTS) framework (Mikeska, Kurzum, Steinberg, & Xu, 2018) to guide the development of our CKT assessment instrument about matter and its interactions. The full WOTS framework:
- focuses on seven instructional tools (e.g., scientific models and representations; scientific investigations; etc.) that elementary science teachers interact with.
- includes 27 science teaching practices that research has indicated or hypothesized are critical for beginning elementary science teachers to know how to do well to be effective practitioners.
- is organized into two to six science teaching practices per instructional tool category (e.g., the scientific explanations tool category includes two science teaching practices — one about critiquing student generated explanations and another about selecting explanations of scientific phenomena).
To create each CKT assessment item, we used content-specific resources, such as research studies, educative curriculum guides and materials, and practitioner literature and knowledge, to generate instructional scenarios that involve elementary science teachers in leveraging their subject matter knowledge as they engage in these specific science teaching practices. This work involved identifying the content challenges elementary teachers face when teaching science within each of five key topic areas within matter and its interactions:
- properties of matter and their measurements;
- changes in matter;
- conservation of matter;
- model of matter; and
These challenges include the work teachers do to identify and select instructional activities, science phenomena, demonstrations, and models used within each of these topic areas. The instructional scenarios were intended to directly map onto the science teaching practices described in the WOTS framework.
Each CKT matter item was aligned to one of the five specific topics (e.g., changes in matter) and to a specific science teaching practice within one of the seven WOTS instructional tool categories (e.g., scientific models).
Item Development Steps
A key part of this project’s work involves determining how CKT assessment items can be assembled into a valid and reliable assessment instrument to measure pre-service elementary teachers’ CKT proficiency in one science area: matter and its interactions. To develop this CKT assessment and the individual CKT items about matter and its interactions, we used:
- the principles of evidence-centered design (Mislevy & Risconscente, 2006) and
- a process closely modeled after the item development work on the Measures of Effective Teaching (MET) Project (Phelps, Weren, Croft, & Gitomer, 2014) and an earlier NSF research project (Mikeska, Phelps, & Croft, 2017).
Our CKT assessment development process includes the following steps:
Mikeska, J.N., Kurzum, C., Steinberg, J., & Xu, J. (2018). Assessing elementary science teachers’ content knowledge for teaching science for the ETS Educator Series: Pilot results. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12207
Mikeska, J.N., Phelps, G., & Croft, A. (2017). Practice-based measures of elementary scienceteachers’ content knowledge for teaching: Initial item development and validity evidence. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12168
Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered design: Layers, concepts, and terminology. In Downing, S., & Haladyna, T. Mahway (Eds.), Handbook of test development. New Jersey: Erlbaum.
Phelps, G., Weren, B., Croft, A., & Gitomer, D. (2014). Developing content knowledge for teaching assessments for the Measures of Effective Teaching study. ETS Research Report Series. Princeton, NJ: Educational Testing Service. doi: https://doi.org/10.1002/ets2.12031