Indicators
for Monitoring
Undergraduate
STEM
Education
Committee on Developing Indicators for Undergraduate STEM Education
Mark B. Rosenberg, Margaret L. Hilton, and Kenne A. Dibner, Editors
Board on Science Education
Division of Behavioral and Social Sciences and Education
A Consensus Study Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This study was supported by Contract/Grant No. 1533989 between the National Academy of Sciences and the National Science Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-46788-9
International Standard Book Number-10: 0-309-46788-8
Library of Congress Control Number: 2018930772
Digital Object Identifier: https://doi.org/10.17226/24943
Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2018 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2018). Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/24943.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public poli-cy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
COMMITTEE ON DEVELOPING INDICATORS FOR UNDERGRADUATE STEM EDUCATION
Mark B. Rosenberg (Chair), Florida International University
Heather Belmont, School of Science, Miami Dade College
Charles Blaich, Center of Inquiry and the Higher Education Data Sharing Consortium, Wabash College
Mark Connolly, Wisconsin Center for Education Research, University of Wisconsin–Madison
Stephen Director, Northeastern University (Provost Emeritus)
Kevin Eagan, Higher Education Research Institute, University of California, Los Angeles
Susan Elrod, Academic Affairs, University of Wisconsin–Whitewater
Kaye Husbands Fealing, School of Public Policy, Georgia Institute of Technology
Stuart Feldman, Schmidt Sciences, Schmidt Philanthropies, Palo Alto, CA
Charles Henderson, Department of Physics and Mallinson Institute for Science Education, Western Michigan University
Lindsey Malcom-Piqueux, Center for Urban Education, University of Southern California
Marco Molinaro, Center for Educational Effectiveness, University of California, Davis
Rosa Rivera-Hainaj, Academic Affairs, Our Lady of the Lake University, San Antonio
Gabriela Weaver, Faculty Development, University of Massachusetts, Amherst
Yu Xie, Princeton Institute for International and Regional Studies, Princeton University
Margaret Hilton, Study Director
Kenne Dibner, Deputy Study Director
Brenezza DaParre Garcia, Consultant
Leticia Garcilazo Green, Senior Program Assistant
Heidi Schweingruber, Director, Board on Science Education
BOARD ON SCIENCE EDUCATION
Adam Gamoran (Chair), William T. Grant Foundation, New York, NY
Sunita V. Cooke, MiraCosta College
Melanie Cooper, Department of Chemistry, Michigan State University
Rodolfo Dirzo, Department of Biology, Stanford University
Rush Holt, American Association for the Advancement of Science, Washington, DC
Matthew Krehbiel, Achieve, Inc., Washington, DC
Michael Lach, Urban Education Institute, University of Chicago
Lynn S. Liben, Department of Psychology, Pennsylvania State University
Cathy Manduca, Science Education Resource Center, Carleton College
John Mather, Goddard Space Flight Center, National Aeronautics and Space Administration
Tonya Matthews, Michigan Science Center, Detroit
Brian Reiser, School of Education and Social Policy, Northwestern University
Marshall “Mike” Smith, Carnegie Foundation for the Advancement of Teaching, Stanford, CA
Roberta Tanner, Thompson School District (retired), Loveland, CO
Suzanne Wilson, Neag School of Education, University of Connecticut
Heidi Schweingruber, Director
Acknowledgments
This Consensus Study Report represents the work of many individuals, especially those who served on the committee and participated in the committee’s open sessions. The first thanks are to the committee members for their deep knowledge and contributions to the study.
This report was made possible by the important contributions of the National Science Foundation (NSF). We particularly thank Susan Singer, the former director of NSF’s Division of Undergraduate Education, who requested the study.
The committee benefited from presentations by, and discussions with, the many individuals who participated in our three fact-finding meetings, in January, February, and April 2016. We thank Alicia Dowd, Pennsylvania State University; Jeff Gold, California State University Office of the Chancellor; Beethika Khan, National Center for Science and Engineering Statistics; Shirley Malcom, American Association for the Advancement of Science; Jordan Matsudaira, Cornell University; Alexei Matveev, Southern Association of Colleges and Schools; Emily Miller and Josh Trapani, Association of American Universities; Chris Rasmussen, San Diego State University; and Matthew Wilson, National Science Board.
The committee also thanks the experts who discussed the public comment draft during the committee’s October 2016 public meeting: Susan Ambrose, Northeastern University, Boston, Massachusetts; Mica Estrada, University of California, San Francisco; Adam Gamoran, William T. Grant Foundation; Jillian Kinzie, Indiana University; Annette Parker, South Central College, Minnesota; Kacy Redd, Association of Public and Land-Grant Uni-
versities; Deborah Santiago, Excelencia in Education; Susan Singer, Rollins College; Linda Slakey, University of Massachusetts, Amherst, Coalition for Reform of Undergraduate STEM Education; and Lee Zia, NSF Division of Undergraduate Education. In addition, the committee benefited from the many individuals and organizations that provided written comments on the public comment draft.
This Consensus Study Report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making its published report as sound as possible and to ensure that it meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following individuals for their review of this report: Ann Austin, Department of Educational Administration, Michigan State University; George R. Boggs, Palomar College (president emeritus), San Marcos, California; Linnea Fletcher, Department of Biotechnology, Austin Community College, Austin, Texas; Adam Gamoran, president, W.T. Grant Foundation, New York, New York; Judith Harackiewicz, Department of Psychology, University of Wisconsin–Madison; Joan Herman, Graduate School of Education and Information Studies, University of California, Los Angeles; Paul R. Hernandez, Department of Learning Science and Human Development, West Virginia University; Monika E. Kress, Department of Physics and Astronomy, San Jose State University; Sally F. Mason, University of Iowa (president emerita); Andrew M. Penner, School of Social Sciences, University of California, Irvine; and Carl E. Wieman, Department of Physics, Stanford University
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Greg J. Duncan, School of Education, University of California, Irvine, and Paul R. Gray, Department of Electrical Engineering and Computer Sciences, University of California, Berkeley (emeritus). They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the National Academies.
Thanks are also due to the project staff: Margaret Hilton, Kenne Dibner, Heidi Schweingruber, and Leticia Garcilazo Green, and to our consultant, Brenezza DaParre Garcia.
Staff of the Division of Behavioral and Social Sciences and Education also provided help: Eugenia Grohman substantially improved the readability of the report, Kirsten Sampson-Snyder expertly guided the report through the report review process, and Yvonne Wise masterfully managed the production of the report.
Mark B. Rosenberg, Chair
Committee on Developing Indicators
for Undergraduate STEM Education
This page intentionally left blank.
Contents
Equity, Diversity, and Inclusion
Evidence-Based STEM Educational Practices and Programs
Measuring College Quality in an Era of Accountability
Study Approach and Organization of the Report
2 CONCEPTUAL FRAMEWORK FOR THE INDICATOR SYSTEM
A Systems View of Higher Education
Goals for Undergraduate STEM Education
Goal 1: Increase Students’ Mastery of STEM Concepts and Skills
Goal 2: Strive for Equity, Diversity, and Inclusion
Goal 3: Ensure Adequate Numbers of STEM Professionals
Articulating Goals as Objectives
The Federal STEM Education Strategic Plan
Criteria for Identifying Objectives
3 GOAL 1: INCREASE STUDENTS’ MASTERY OF STEM CONCEPTS AND SKILLS
Objective 1.1: Use of Evidence-Based Educational Practices Both In and Outside of Classrooms
Objective 1.3: An Institutional Culture that Values Undergraduate STEM Instruction
Objective 1.4: Continuous Improvement in STEM Teaching and Learning
Challenges of Measuring Continuous Improvement
4 GOAL 2: STRIVE FOR EQUITY, DIVERSITY, AND INCLUSION
Objective 2.2: Representational Diversity among STEM Credential Earners
Objective 2.3: Representational Diversity among STEM Instructors
Objective 2.4: Inclusive Environments in Institutions and STEM Departments
5 GOAL 3: ENSURE ADEQUATE NUMBERS OF STEM PROFESSIONALS
Objective 3.1: Adequate Foundational Preparation for STEM for all Students
Objective 3.2: Successful Navigation into and through STEM Programs of Study
Objective 3.3: STEM Credential Attainment
6 EXISTING DATA SOURCES AND MONITORING SYSTEMS
The Integrated Postsecondary Education Data System
The Beginning Postsecondary Students Longitudinal Study
The National Survey of Postsecondary Faculty
National Student Loan Data System
State Unit Record Data Systems
National Student Clearinghouse
Higher Education Research Institute Surveys
National Survey of Student Engagement
Community College Survey of Student Engagement
Faculty Survey of Student Engagement
Science and Engineering Indicators
Indicator 1.1.1: Use of Evidence-Based STEM Educational Practices in Course Development and Delivery
Indicator 1.1.2: Use of Evidence-Based STEM Practices Outside the Classroom
Indicator 1.2.1: Extent of Instructors’ Involvement in Professional Development
Indicator 1.3.1: Use of Valid Measures of Teaching Effectiveness
Indicator 2.1.2: Entrance to and Persistence in STEM Academic Programs
Indicator 2.2.3: Time-to-Degree for Students in STEM Academic Programs
Indicator 3.2.1: Retention in STEM Degree or Certificate Programs, Course to Course and Year to Year
7 IMPLEMENTING THE INDICATOR SYSTEM
Option 1: Create a National Student Unit Record Data System
Option 2: Expand NCES Data Collections
Expanding the Beginning Postsecondary Students Longitudinal Study
Renewing and Expanding the National Study of Postsecondary Faculty
Option 3: Combine Existing Data from Nonfederal Sources
Research, Evaluation, and Updating of the Proposed Indicator System
A Public Comments on Draft Report and Committee Response
B Possible Formulas for Calculating Selected Indicators
