Measuring What Counts: Developing Scales of Connectedness, Empowerment, and Meaning in the Non-Cognitive Domain
Across the fields of K-12 and postsecondary education, there is an increasing focus on developing the whole person and thus the impact of non-cognitive psychological factors on promoting students’ abilities to learn, be successful, and contribute to the larger society in meaningful ways. Continued interest in evaluation of college readiness programs such as GEAR UP, Upward Bound, and AVID which foster development of both academic and non-cognitive skills to promote achievement and college readiness among at-risk youth, has motivated all stakeholders to better understand the non-cognitive domain. We describe a qualitative and quantitative approach to constructing new measures of student engagement for San Jose State University’s GEAR UP program. Examining a sample of 7-12 grade student responses in three California public schools, we found a sufficient degree of internal structure validity evidence (APA, AERA, NCME, 2014) to support limited, diagnostic uses of the survey instrument. Using a construct mapping framework (Wilson, 2005), our initial IRT examination of the data found that the unidimensional Partial Credit Model fits the data very well and results in high reliability (.91). Further multi-dimensional IRT analyses of a sub-sample of the data (n=472) provided evidence for three distinct constructs--connectedness, empowerment, meaning. Gender DIF analyses also support the instrument's intended use. We share potential threats to validity related to Likert-scales used to measure student non-cognitive capacities, particularly with low-income, culturally and linguistically diverse middle and high school students. Discussion includes 2014 Testing Standards and the evidentiary base required to validate particular uses of non-cognitive measures, including program evaluation and State accountability more broadly.
Brent Duckor is an Associate Professor at the Department of Teacher Education in the Lurie College of Education at San Jose State University. Dr. Duckor’s current research interests focus on three areas: teachers’ understanding and use of formative assessment in the K-12 classroom; validation of teacher licensure exams and certification in state, national, and international contexts; and measuring non-cognitive outcomes for program evaluation and school improvement to better serve historically disadvantaged, low-income youth. He has served as a consultant to SJSU’s GEAR UP program and is a member of the College and Career Readiness Evaluation Consortium, National Council for Community and Education Partnerships (NCCEP).
Joshua Sussman is a PhD Candidate in the School Psychology program at the University of California, Berkeley and a practicing school psychologist. He has participated in BEAR center research focused on developing an integrated learning progression for scientific practices. His dissertation research, supported by an IES pre-doctoral fellowship in the Research in Cognition and Mathematics Education (RCME) program, investigates measurement problems in applied research related to the use of standardized tests for evaluating new instructional interventions in science and mathematics.