• Open access
  • Published: 25 November 2019

Developing student 21 st Century skills in selected exemplary inclusive STEM high schools

  • Stephanie M. Stehle   ORCID: orcid.org/0000-0003-4017-186X 1 &
  • Erin E. Peters-Burton 1  

International Journal of STEM Education volume  6 , Article number:  39 ( 2019 ) Cite this article

117k Accesses

86 Citations

Metrics details

There is a need to arm students with noncognitive, or 21 st Century, skills to prepare them for a more STEM-based job market. As STEM schools are created in a response to this call to action, research is needed to better understand how exemplary STEM schools successfully accomplish this goal. This conversion mixed method study analyzed student work samples and teacher lesson plans from seven exemplary inclusive STEM high schools to better understand at what level teachers at these schools are engaging and developing student 21 st Century skills.

We found of the 67 lesson plans collected at the inclusive STEM high schools, 50 included instruction on 21 st Century skills. Most of these lesson plans designed instruction for 21 st Century skills at an introductory level. Few lesson plans encouraged multiple 21 st Century skills and addressed higher levels of those skills. Although there was not a significant difference between levels of 21 st Century skills by grade level, there was an overall trend of higher levels of 21 st Century skills demonstrated in lesson plans designed for grades 11 and 12. We also found that lesson plans that lasted three or more days had higher levels of 21 st Century skills.

Conclusions

These findings suggest that inclusive STEM high schools provide environments that support the development of 21 st Century skills. Yet, more can be done in the area of teacher professional development to improve instruction of high levels of 21 st Century skills.

Introduction

School-aged students in the USA are underperforming, particularly in science, technology, engineering, and mathematics (STEM) subjects. National Assessment of Educational Progress (U.S. Department of Education, 2015a ) scores show that in science, only 34% of 8th graders are performing at or above proficiency and 12th grade students at or above proficient US students drop to 22%. Similarly, mathematics scores show 33% of 8th graders and 22% of 12th graders were at or above proficiency (U.S. Department of Education, 2015a ). Additionally, the US mathematics scores for the Programme for International Student Assessment (PISA) for 2015 were lower than the scores for 2009 and 2012 (Organisation for Economic Co-operation and Development; OECD, 2018 ). US students not only underachieve in mathematics and science, but are also not engaging successfully in engineering and technology. At the secondary level, there are relatively few students in the USA that take engineering (2%) and computer science (5.7%) (National Science Board, 2016 ). The NAEP technology and engineering literacy (TEL) assessment found that for technology and engineering literacy, only 43% of 8th graders were at or above the proficiency level (U.S. Department of Education, 2015b ). This consistent trend of underperformance has focused many national, state, and local efforts to improve student experiences in integrated STEM subjects (cf. President’s Council of Advisors on Science and Technology, 2010 ; Texas Education Association ( n.d. ) for school-aged students and beyond.

The efforts for improvement in STEM teaching in K-12 environments have yielded a slight increase in the enrollment of STEM majors recently (National Science Board, 2016 ). However, roughly half of students who declare a STEM major when entering college either switch majors or drop out of college (National Science Board, 2016 ). One approach to helping students persist in undergraduate education is a stronger foundation in content knowledge, academic skills, and noncognitive skills (Farrington et al., 2012 ). Academic skills, including analysis and problem solving skills, allow students to engage with content knowledge at higher levels of cognition. Noncognitive skills, including study skills, time management, and self-management, assist students in optimizing their ability to gain content knowledge and use their academic skills to solve problems. Students who possess these skills have high-quality academic behaviors, characterized by a pursuit of academic goals despite any setbacks (Farrington et al., 2012 ).

Because academic skills, noncognitive skills, and content knowledge have fluid definitions and may not be directly observable, for the purposes of this study we used 21 st Century skills consisting of knowledge construction, real-world problem solving, skilled communication, collaboration, use of information and communication technology for learning, and self-regulation (Partnership for 21 st Century Learning, 2016 ). Graduates who possess 21 st Century skills are sought out by employers (National Research Council, 2013 ). In the environment of rapid advancements in technology and globalization, employees need to be flexible and perpetual learners in order to keep up with new developments (Bybee, 2013 ; Johnson, Peters-Burton, & Moore, 2016 ). There is a need to ensure that students who graduate the K-12 system are adept in 21 st Century skills so that they can be successful in this new workforce landscape (Bybee, 2013 ).

Not only do 21 st Century skills help students be successful in all areas of formal school, these skills are also necessary for a person to adapt and thrive in an ever changing world (Partnership for 21 st Century Learning, 2016 ). One movement embracing the need for the development of student 21 st Century skills is the proliferation of inclusive STEM high schools (ISHSs), schools that serve all students regardless of prior academic achievement (LaForce et al., 2016 ; Lynch et al., 2018 ). ISHSs promote student research experiences by using inquiry-based curricular models to scaffold independent learning and encourage personal responsibility (Tofel-Grehl & Callahan, 2014 ). The goal for ISHSs to facilitate this type of student-centered learning is to build students’ 21 st Century skills such as adaptability, communication, problem solving, critical thinking, collaboration, and self-management (Bybee, 2013 ; Johnson et al., 2016 ; LaForce et al., 2016 ). Although there has been some evidence that not all ISHSs are advantageous in offering STEM opportunities (Eisenhart et al., 2015 ), there is an accumulation of evidence that ISHSs can increase college and career readiness for students from groups who are typically underrepresented in STEM careers (Erdogan & Stuessy, 2015 ; Means, Wang, Viki, Peters, & Lynch, 2016 ). As the number of inclusive STEM schools continue to increase across the USA, there is a need to understand the ways these schools successfully engage students in 21 st Century skills. The purpose of this paper is to systematically analyze teacher-constructed lessons and student work from seven exemplar ISHSs in order to better understand how teachers are engaging and developing student 21 st Century skills.

Specifically, this study looked at the extent to which teachers at these exemplar ISHSs ask students to practice the 21 st Century skills and at the level of student performance of the following categories: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation (SRI International, n.d. -a; SRI International, n.d. -b). An examination of the lesson plans and student work products at exemplar ISHSs provides insight into effective development of student 21 st Century skills in a variety of contexts.

Conceptual framework

In an attempt to clearly define the skills, content knowledge and literacies that students would need to be successful in their future endeavors, the Partnership for 21 st Century Learning (P21; 2016) created a framework that includes (a) life and career skills; (b) learning and innovation skills; (c) information, media, and technology skills; and (d) key subjects (Partnership for 21 st Century Learning, 2016 ). The first three parts of the framework, (a) life and career skills, (b) learning and innovation skills, and (c) information, media, and technology skills, describe proficiencies or literacies students should develop and can be integrated and developed in any academic lesson. The fourth piece, key subjects, suggests 21 st Century interdisciplinary themes or content to engage students in authentic study (Partnership for 21 st Century Learning, 2016 ).

Due to the need to build 21 st Century skills, this study focused on the teaching and learning of (a) learning and innovation skills; (b) information, media, and technology skills; and (c) life and career skills at exemplar ISHSs. In order to operationalize and measure the three categories, we searched for instruments that measured the learning of 21 st Century skills. Microsoft, in collaboration with SRI Education, developed two rubrics that are designed to assess the extent to which 21 st Century skills are present in lessons and the extent to which students demonstrate the skills from these lessons (SRI International, n.d. -a; SRI International, n.d. -b). The 21 st Century Learning Design Learning Activity Rubric examined the proficiency of teacher lesson plans for the development of 21 st Century skills while the 21 st Century Learning Design Student Work Rubric examined the level of competency for each 21 st Century skill. Although the rubrics did not align exactly with the P21 Framework, we felt that there was enough alignment with the categories that the rubrics would be useful in measuring the extent to which lessons in ISHSs taught 21 st Century skills and the extent to which students demonstrated these skills. The rubrics had the same categories for lesson assessment and student work assessment: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation in teacher lesson plans and student work samples (SRI International, n.d. -a; SRI International, n.d. -b). Table 1 shows how the categories assessed in the two rubrics align with the categories in the P21 Framework. Further, as we reviewed the literature on these categories, a model of their relationship emerged. Our literature review discusses the individual categories followed by the conceptual model of how these categories work together in 21 st Century skill development.

  • Knowledge construction

Knowledge construction occurs when students create new knowledge themselves rather than reproducing or consuming information (Prettyman, Ward, Jauk, & Awad, 2012 ; Shear, Novais, Means, Gallagher, & Langworthy, 2010 ). When students participate in knowledge construction rather than reproduction, they build a deeper understanding of the content. Learning environments that are designed for knowledge construction promote self-regulated and self-directed learners as well as building grit (Carpenter & Pease, 2013 ).

Although knowledge construction helps students to build deep understandings and skills to be self-directed and resilient learners, many students are unfamiliar with this approach to learning and frequently need scaffolding to take on joint responsibility of learning (Carpenter & Pease, 2013 ; Peters, 2010 ). When transitioning to a more student-centered learning environment that supports knowledge construction, the teacher becomes more of a facilitator rather than a lecturer (McCabe & O’Connor, 2014 ). A student-centered learning environment encourages students to shift from a paradigm of expecting one convergent answer and toward deeper meaning-making when learning (Peters, 2010 ). Knowledge construction anchors the development of 21 st Century skills because students need to be able to have background knowledge in order to perform the skills in an authentic context.

  • Real-world problem solving

Sometimes called project-based learning (Warin, Talbi, Kolski, & Hoogstoel, 2016 ), real-world problem solving is characterized by students working to solve problems that have no current solution and where the students can implement their own approach (Shear et al., 2010 ). When solving a real-world problem, students work to identify the problem, propose a solution for a specific client, test the solution, and share their ideas (Prettyman et al., 2012 ; Warin et al., 2016 ). The design aspect of the process encourages students to be creative and learn from failures (Carroll, 2015 ). When using real-world problem solving, students develop knowledge in a meaningful way (White & Frederiksen, 1998 ), must regulate their cognition and behavior in a way to reach their goals (Brown, Bransford, Ferrara, & Campione, 1983 ; Flavell, 1987 ), and gain experience defending their choices through evidence and effective communication skills (Voss & Post, 1988 ).

Teachers can develop real-world problem solving skills in their students by modeling inquiry after research actual scientist are involved in, using databases with real-life data, and evaluating evidence from current events (Chinn & Malhortra, 2002 ). Designing real-world problem scenarios for the classroom provide a framework by which students can engage in 21 st Century learning and can help to encourage a more positive attitude towards STEM careers (Williams & Mangan, 2016 ). Together, knowledge construction and real-world problem solving create the foundation from which students can engage in self-regulation, collaboration, and communication.

  • Self-regulation

Self-regulation is a key 21 st Century skill for independent learners. Students who are self-regulated plan their approach to problem solving, monitor their progress, and reflect on their work given feedback (Shear et al., 2010 ; Zimmerman, 2000 ). During the self-regulation process, a student motivates himself or herself to control impulses in order to efficiently solve problems (Carpenter & Pease, 2013 ; English & Kitsantas, 2013 ). Fortunately, these skills are teachable; however, students need time to accomplish regulatory tasks and guidance for the key processes of reflection and revision (Zimmerman, 2000 ). Therefore, long-term projects give a more appropriate time frame than short-term projects to hone these regulatory skills.

Students have different levels of self-regulation (English & Kitsantas, 2013 ) and teachers may need to integrate strategies and ways of monitoring students into lessons (Bell & Pape, 2014 ; English & Kitsantas, 2013 ). Incorporating self-regulated learning strategies helps students to stay engaged and deal with any adversity that may come up in the process (Boekaerts, 2016 ; Peters & Kitsantas, 2010 ). A tangible way teachers can support student self-regulation is by using Zimmerman’s ( 1998 ) four-stage model of self-regulated learning support: modeling, emulation, self-control, and self-regulation (Peters, 2010 ). First, teachers explicitly model the target learning strategy that the student should acquire, pointing out key processes (modeling). Second, teachers can provide students with verbal or written support for the key processes of the learning strategy while the student attempts to emulate the modeling from the teacher (emulation). Once students can roughly emulate the learning strategy, the teacher can fade support and have the student try to do the learning strategy on their own (self-control). After students attempt it on their own, the teacher provides feedback to the student to help them improve their attempted learning strategy (self-regulation). When a student can successfully perform the learning strategy on their own, they have become self-regulated in that aspect of their learning. Students who have mastered self-regulated learning have the ability to be proactive in knowledge building and in problem solving, which are characteristics that STEM industry employers value.

  • Collaboration

Collaboration occurs when students take on roles and interact with one another in groups while working to produce a product (Shear et al., 2010 ). Collaborative interactions include taking on leadership roles, making decisions, building trust, communicating, reflecting, and managing conflicts (Carpenter & Pease, 2013 ). Students who collaborate solve problems at higher levels than students who work individually because students respond to feedback and questions to create solutions that better fit the problem (Care, Scoular, & Griffin, 2016 ). Collaboration is an important skill to enhance knowledge building and problem solving. Conversations among peers can support student self-regulated learning through modeling of verbalized thinking.

  • Skilled communication

“Even the most brilliant scientific discovery, if not communicated widely and accurately, is of little value” (McNutt, 2013 , p. 13). For the purpose of this paper, skilled communication is defined as types of communication used to present or explain information, not discourse communication. Skilled communicators present their ideas and demonstrate how they use relevant evidence (Shear et al., 2010 ). An important part of being able to communicate successfully is the ability to connect a product to the needs of a specific audience or client (Warin et al., 2016 ). In doing so, the students need to take into account both the media they are using and the ideas they are communicating so that it is appropriate for the audience (Claro et al., 2012 ; van Laar, van Deursen, van Dijk, & de Haan, 2017 ). Like collaboration, skilled communication is a necessary process to successfully employ knowledge construction and real-world problem solving.

Use of information and communication technology for learning

When students use information and communication technology (ICT) for learning, they are designing, creating, representing, evaluating, or improving a product, not merely demonstrating their knowledge (Koh, Chai, Benjamin, & Hong, 2015 ). In doing so, they need to choose how and when to use the ICT as well as know how to recognize credible online resources (Shear et al., 2010 ). The effective use of ICT requires self-regulation in order to use these tools independently and to keep up with technological advances. Given the continuous advancements in technology, it is essential that students know how to manage and communicate information in order to solve problems (Ainley, Fraillon, Schulz, & Gebhardt, 2016 ).

Conceptual Model of 21 st Century Skills

The six 21 st Century skills presented above are critical for students to develop to prepare for both college (National Science Board, 2016 ) and the future employment (Bybee, 2013 ; Johnson et al., 2016 ). Twenty-first century skills do not exists in isolation. By building one skill, others are reinforced. For example, knowledge construction and real-world problem solving can be enhanced by self-regulation. Likewise, collaboration requires skilled communication to build knowledge and solve problems. These skills coalesce to build the necessary toolkit for students who can learn on their own. Figure 1 shows a working hypothesis of how these six skills, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, interact to foster lifelong learning for student.

figure 1

Working hypothesis of how 21 st Century skills work together to build a 21 st Century student

Knowledge construction and real-world problem solving are the keystones of the model and typically represent the two main goals of student-centered lessons. Knowledge construction is the conceptual formation while real-world problem solving represents the process skills that students are expected to develop. Knowledge construction and real-world problem solving feed each other in a circular fashion. Knowledge construction is built through the inquiry process of real-world problem solving. At the same time, real-world problem solving requires new knowledge to be constructed in order to solve the problem at hand. The connection between knowledge construction and real work problem solving is mediated by collaboration and communication.

While communication and collaboration allow a student to work with others to build their conceptual knowledge and work toward a solution to their real-world problem, self-regulation is an internal process that occurs simultaneously. The student’s self-regulation guides the student’s individual connections, reflections, and revisions between knowledge construction and real-world problem solving.

Information and communication technology provides tools for the students to facilitate communication and collaboration as well as other 21 st Century skills. ICT helps to simplify and assist the communication and collaboration for groups of students. ICT can help streamline the process of analysis and record keeping as well as facilitating the sharing ideas with others. It allows students to more easily document their progress and express their ideas for later reflection. Although ICT is not directly connected with other elements in the model, the use of ICT allows for the learning process to be more efficient.

The six 21 st Century skills addressed in this study, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, are important facets of STEM education. This study documented the extent to which each of the 21 st Century skills were present in both lesson plans and in student work at seven exemplar ISHSs. Given that the schools in the study were highly regarded, understanding the structure and student outcomes of lessons could provide a model for teachers and teacher educators. With that in mind, the study was driven by the following research questions:

To what extent do teacher lesson plans at exemplar ISHSs exhibit 21 st Century learning practices as measured by the 21 st Century Learning Design Learning Activity and Student Work Rubrics?

Do teacher lesson plans and student work samples from exemplar ISHSs show differences in rubric scores by grade level?

During the analysis of these questions, a third research question emerged regarding the duration of lessons. The question and rationale can be found in the data analysis section.

This study is part of a larger multiple instrumental case study of eight exemplar ISHSs. The larger study (Opportunity Structures for Preparation and Inspiration in STEM; OSPrI) examined the common features of successful ISHSs (Lynch et al., 2018 ; Lynch, Peters-Burton, & Ford, 2014 ). OSPrI identified 14 critical components (CC; Table 2 ) that successful ISHSs possess (Behrend et al., 2016 ; Lynch et al., 2015 ; Lynch, Means, Behrend, & Peters-Burton, 2011 ; Peters-Burton, Lynch, Behrend, & Means, 2014 ). Three of the 14 critical components involve the application of 21 st Century skills in the classroom. This study addresses these three critical components: (a) CC1: STEM focused curriculum for all, (b) CC2: reform instructional strategies and project-based learning, and (c) CC3: integrated, innovative technology use.

Cross-case analysis of the eight schools found similarities in how the schools addressed two specific critical components: CC1: college-prep, STEM focused curriculum for all and CC2: reform instructional strategies and project-based learning. From these two critical components, curriculum and instruction, four themes emerged: (a) classroom-related STEM opportunities, (b) cross-cutting school level STEM learning opportunities, (c) school-wide design for STEM opportunities, and (d) responsive design (Peters-Burton, House, Han, & Lynch, 2018 ). The theme of classroom-related STEM opportunities was characterized by the expectation that teachers act as designers of the curriculum and look beyond the typical textbook for resources. While designing the curriculum, teachers took a mastery learning approach and provided students multiple opportunities to master the material. Through the use of collaborative group projects, summative projects, culminating projects, and interdisciplinary studies, the schools demonstrated a cross-cutting school level approach to the STEM learning. School-wide STEM opportunities included a rigorous curriculum, incorporating engineering classes and/or engineering design thinking, emphasizing connections between the curriculum and real-world examples, as well as building strong collaboration between teachers. Finally, these ISHSs had systems such as data-driven decision making and supports for incoming ninth graders built into their schools as a responsive design. In summary, these schools worked to improve students’ 21 st Century skill such as collaboration, problem solving, information and media literacy, and self-directed learning (Lynch et al., 2018 ).

Research design

This study was designed as a conversion mixed methods approach (Tashakkori & Teddlie, 2003 ) in that qualitative data were transformed into quantitative data using established rubrics. Document analysis was used as a tool to identify occasions of evidence within lessons plans and student work products related to the identified 21 st Century skills (Krippendorff, 2012 ). In this conversion approach, the 21 st Century skill demonstrated qualitatively in the documents was scored using the rubrics, ergo integrating qualitative and quantitative methods in the analysis.

Participating schools

The eight exemplar ISHSs for this study came from the same quintain as used by the OSPrI project (Lynch et al., 2018 ). Because this origin project was a cross-case analysis and the IRB did not allow for school to school comparison, the data collected from individual schools was aggregated as one data source. Protocol for inclusion in the OSPrI study was that the school had no academic admission requirements, self-identified as a STEM school, was in operation for grades 9 through 12, and intentionally recruited students typically underrepresented in STEM. For more information on the demographics of the schools and the selection process, see Lynch et al., 2018 . Of the eight schools that were in the original OSPrI project, seven provided teacher lesson plans and/or student work samples during the school visit. All schools have given permission to use their actual names. The sample size from each school was inconsistent, therefore, we treated the data set as one combined group that included all seven schools.

Data sources

Student work samples and teacher lesson plans were collected during OSPrI site visits to the seven schools, which were each visited once between 2012 and 2014. Researchers requested paper copies of typical lesson plans and student work that resulted in an average performance from the lesson plan that was observed at all eight ISHSs during the site visits. Because this was a convenience sample, not all teachers submitted lesson plans, and only a few teachers submitted the student work products related to those lessons. Unfortunately, few parents consented to release student work products. As a result, 67 teacher lesson plans and 29 student work samples were collected from seven of the eight schools. We decided to keep the student work products in the descriptive portion of the analysis, but not the inferential analysis in the study because this is a unique opportunity to gain even a small insight into student work from STEM schools that were considered exemplary and served students who are typically underrepresented in STEM. Table 3 describes the content matter and grade level(s) associated collected teacher lesson plan and corresponding student work product.

Each teacher lesson plan was analyzed using the 21 st Century Learning Design (21CLD) Learning Activity Rubric and each student work product was analyzed using the 21 st Century Learning Design Student Work Rubric (SRI International, n.d.-a; SRI International, n.d.-b). These instruments were found to be valid and reliable for use in high school classrooms, and Shear et al., 2010 reports the details of the development and validation of the rubrics. Although the student work products were related to the teacher lesson plans, they were analyzed independently according to the protocol of the 21CLD rubrics. The 21CLD Activity Rubric and the 21CLD Student Work Rubric were designed by Microsoft Partner’s in Learning with a collaboration between ITL Research and SRI International (SRI International, n.d.-a; SRI International, n.d.-b). These two 21CLD rubrics were the result of a multi-year project synthesizing research-based practices that promote 21 st Century skills (Shear et al., 2010 ). The rubrics, each 44-pages in length, are available online for public use ( https://education.microsoft.com/GetTrained/ITL-Research ). The 21CLD rubrics assess teacher lesson plans or student work products on six metrics aligned with 21 st Century skills: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation (SRI International, n.d.-a; SRI International, n.d.-b). Collaboration, knowledge construction, and use of ICT score ratings range from one to five while real-world problem solving, self-regulation, and skilled communication score ratings range from one to four.

Data analysis

The teacher lessons and student work samples were assessed on (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation using the 21CLD Learning Activity and the 21CLD Student Work Rubrics respectively. Examples of excerpts from teacher lesson plans and student work products for each category can be found in Table 4 . Two raters were used to establish interrater reliability. Both raters have a background as secondary science teachers and were trained on the use of the rubric. One rater has a terminal degree in education and the other rater is a doctoral student in education. The two raters met and discussed the rubric scores until the interrater reliability was 100%. Once consensus scores were established, tests for assumptions, descriptive, and inferential statistics were run.

During the analysis of research questions one and two, unique trends of short-term and long-term lesson plans were noted. From this, a third research question emerged from the analysis:

Are there differences in the 21 CLD Learning Activity scores of short-term lessons and long-term lessons?

The 21CLD Learning Activity and the 21CLD Student Work Rubrics required a lesson to be long-term order to assess self-regulation. The rubric defined long-term as “if students work on it for a substantive period of time” (SRI International, n.d.-a, p. 32). From our reading of the lesson plans, lessons that were scheduled for three or more days met the criterion of a substantive period of time, while lesson that were scheduled for 1 or 2 days did not meet this criterion. For the purposes of this study, we decided to refine the definition of long-term to be a lesson lasting three or more class periods and a short-term lesson lasting less than three class periods. The analyses for all research questions separated lessons into long-term and short-term in order to clarify the category of self-regulation.

The data were checked for normality, skewness, and outliers; only the teacher lesson plans met all assumptions for an ANOVA (comparison of grade levels) and t test (long-term versus short-term). Due to the small number of student work samples collected (see Table 6 ), the data related to student work did not meet the assumptions needed to run a t test therefore was not included in this analysis.

Overall rubric scores

To answer the first research question, a descriptive analysis was run for each of the six categories on the rubric and the total score (found in Tables 5 and 6 ). The average score for all teacher lesson plans was less than 2 for all six categories (out of a total of 4 or 5). Likewise, overall student work sample averages scored below 2 except on the category of Knowledge Construction. Table 6 also shows the median score for long-term student work sample categories to better describe central tendencies of the data. Figure 2 shows the distribution of total rubric scores for all teacher lesson plans. Seventeen of the 67 lessons scored a 6, the lowest possible score. Only 16 of the 67 lessons scored higher than 13 points, half of the total possible points. Out of those 16 scoring over 50%, only three lessons scored 20 points or more out of the possible 27.

figure 2

Distribution of total 21CLD rubric scores for all lessons

Figure 3 illustrates the quantity of 21 st Century skills found in each lesson. Nearly 75% of the teacher lesson plans included at least one 21 st Century skill in the lesson and 67% addressed two or more 21 st Century skills. Although most of the lessons at the ISHSs introduced multiple 21 st Century skills, the overall scores for the quality were low.

figure 3

Distribution of number of 21 st Century skills addressed in a lesson

21 st Century learning by grade

To answer the second research question, an ANOVA was conducted to compare lesson scores by grade level. There were no statistically significant differences between grade level scores for the total rubric score. Data were separated into short-term and long-term lessons by rubric category. There were no significant differences in short-term lessons by grade level (Fig. 4 ). However, there were significant differences across grades for long-term lessons. Total rubric score for grade 12 lessons were significantly higher than grade 9 ( p = 0.023) and grade 11 ( p = 0.032). Difference in total rubric scores for grade 12 lessons were approaching significance with grade 10 ( p = 0.063). As seen in Fig. 5 , category scores for long-term learning activities have small differences in 9th, 10th, and 11th grades but peaks noticeably in 12th grade. The exception to this trend is use of ICT which peaks in 11th grade.

figure 4

The average rubric metric scores for short-term lessons, sorted by grade level for the lesson

figure 5

The average rubric metric scores for long-term lessons, sorted by grade level for the lesson

Long-term versus short-term assignments

To answer the second research question, a t test with Bonferroni correction was performed to compare long-term and short-term lessons for each of the categories. A statistically significant difference was found between short-term ( N = 35) and long-term ( N = 32) lessons on total score, knowledge construction, use of ICT, self-regulation, and skilled communication (Table 7 ). The effect sizes for these categories as calculated by Hedges g (Lakens, 2013 ) were all above 0.8 indicated large effect size (Table 7 ). In all of those categories, long-term lessons scored higher than short-term lessons (Table 5 ). The category of real-world problem solving was approaching statistical significance with the t-score not showing significance [ t = − 2.67, p = .001] but a statistically significant confidence interval [− 1.23, 0.003] and a medium effect size (Table 7 ).

  • 21 st Century skills

Overall, the teacher lesson plans collected at the ISHSs showed evidence of addressing 21 st Century skills. Nearly 75% of the lessons included at least one 21 st Century skill with 67% addressing two or more. Although the majority of lessons addressed multiple 21 st Century skills, the rubric scores for these lessons were low because they addressed these skills at a minimal level. For example, a minimal level of collaboration would be instructions to form a group. A high level of collaboration would include defining roles, explicit instructions on how to share responsibility, and evidence of interdependence. Only five lessons showed evidence of multiple 21 st Century skills implemented at the highest level, as measured by the 21CLD Learning Activity Rubric.

While assessing the lesson plans, we noted that more explicit instructions in the teacher lesson plans would have resulted in higher rubric scores. Placing students in groups, structuring peer feedback, and having students design a final project for a particular audience are three small changes not seen frequently in the lesson plans that are articulated in the Lesson Plan rubrics to encourage multiple 21 st Century skills. When students work in groups, they improve their collaboration and communication skills while constructing knowledge and solving problems (Care et al., 2016 ; Shear et al., 2010 ). When teachers incorporate peer feedback into their lesson, students engage in collaboration. Peer feedback also gives students the opportunity to revise their work based on feedback, increasing self-regulation (Shear et al., 2010 ; Zimmerman, 2000 ). When students design their final project for a specific target audience, rather than simply displaying their knowledge for the teacher, they work on their skilled communication processes (Claro et al., 2012 ; van Laar et al., 2017 ; Warin et al., 2016 ). In summary, placing students in groups, structuring peer feedback, and having students design a final project for a particular audience provides opportunities for students to practice 21 st Century skills.

When lessons addressed more than one 21 st Century skill, they usually demonstrated the use of collaboration or communication in real-world problem solving and knowledge construction (Care et al., 2016 ; Carpenter & Pease, 2013 ). Thirty-three lesson plans in which real-world problem solving or knowledge construction was evident, 31 showed evidence of collaboration or communication. Similarly, 13 of the 18 student work samples showed evidence of collaboration or communication when real-world problem solving or knowledge construction was practiced. The results from the indirect measures of the rubric build support for a conceptual model connecting the components of 21 st Century skills (Fig. 1 ). There was some evidence demonstrating the support that collaboration and communication have for knowledge construction and real-world problem solving.

The findings of this study point to the likelihood of self-regulation being connected to other 21 st Century skills. Each time self-regulation was present in a teacher lesson plan, there was evidence of at least one other 21 st Century skill in that lesson. Seventeen of the 23 lesson plans addressing self-regulation included at least three other 21 st Century skills, showing evidence that self-regulation is a skill that is related to knowledge construction and real-world problem solving. Our findings reflect the findings of other researchers, in that self-regulation guides the students’ individual connections, reflections, and revisions between knowledge construction and real-world problem solving (Brown et al., 1983 ; Carpenter & Pease, 2013 ; Flavell, 1987 ; Shear et al., 2010 ).

Evidence from the lessons showed that there was no consistent connection to the use of ICT and the presence of the other 21 st Century skills. ICT was seen in both low-scoring lessons as the sole 21 st Century skill, as well as in high-scoring lessons in tandem with multiple other 21 st Century skills. As in our model, technology is a tool to help facilitate but is not necessary in the development of the other 21 st Century skills (Koh et al., 2015 ; Shear et al., 2010 ). After examining the data, our model remained unchanged for all 21 st Century skills and their relationship to each other.

Grade level differences

Overall, there were no statistically significant differences in the total 21CLD scores across grade levels. This is consistent with the missions of the ISHSs in this study to shift responsibility for learning to the students by weaving 21 st Century skills throughout high school grade levels (Lynch et al., 2017 ). When looking at trends in long-term projects, there was a jump in total 21CLD score for 12th grade. Again, this aligns with the participating schools’ goals of creating an environment where students have a more independent learning experience during their senior year internships, college classes, and specialized programs CC1 (Lynch et al., 2018 ). This is consistent with the goal of many of the schools to have the students work independently during their senior year either by taking college classes, completing an internship, or taking a career specific set of classes.

Short-term vs. long-term lessons

The data showed that long-term lesson planning had significantly higher scores on the rubric as compared to the short-termed lessons. This difference is consistent with the literature regarding the need for students to have time to develop and practice skills (Lynch et al., 2017 ; NGSS Lead States, 2013 ). The extended time allows students to monitor and reflect on their progress while working toward self-regulation of the skill (Carpenter & Pease, 2013 ; English & Kitsantas, 2013 ). To truly become self-regulated, students need repeated supported attempts to be able to do it on their own (Zimmerman, 2000 ).

Although not significant, collaboration was the only rubric metric where the short-term lessons averaged a higher collaboration score than the long-term lessons. Evidence from the lessons show students worked in pairs or groups, but infrequently shared responsibility, made decisions together, or worked interdependently. This leads to the possibility that incorporating the higher levels of collaborations is difficult, even in long-term projects. In addition, evaluating the higher levels of collaboration is difficult to make based solely on documents. Observations would be required to evaluate how the students within the group were interacting with one another.

Limitations

Because this study used data collected as part of a larger study, there were several limitations. The work collected is a snapshot of the work students were doing at the time of the observation and does not allow for a clear longitudinal look at student growth over time. As stated before, the small student work sample limited what we were able to do with the analysis.

By only analyzing paper copies of the student work, it was not possible to determine a true collaboration score for many of the projects. Higher levels of collaboration such as sharing responsibility, making decisions together, and working interdependently require observation or more detailed notes from the students or teachers. Some lessons may have scored higher in the metric of collaboration had the student interactions been observed or noted.

This study confirmed the presence of all identified 21 st Century skills in the lesson plans at the selected exemplar ISHSs serving underrepresented students in STEM: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation. In light of the patterns that emerged from the rubrics, we posit that in the lesson plans communication and collaboration are the core 21st Century skills that facilitate knowledge construction and real-world problem solving, while student self-regulation creates efficiencies resulting in improved knowledge construction and real-world problem solving. We also saw in the lesson plans that ICT provides tools to support communication and reflection which leads to knowledge construction and real-world problem solving. To further develop knowledge about how 21 st Century skills addressed in lesson plans help to support student work, our model can be a hypothesized starting point to investigate interactions.

While teachers were successful at including 21 st Century skills into lessons, very few lessons practiced higher levels of those skills. This could be an indication that high levels of 21 st Century skills are difficult to teach explicitly at the high school level. Future studies may investigate why teachers are not frequently incorporating higher level 21 st Century skills into their lessons to answer questions as to whether teachers feel that (a) they need more training on incorporating 21 st Century skills, (b) students need more practice and scaffolding to build up to higher levels of 21 st Century skills, or (c) they need more time for long-term projects to work on the higher level skills.

The use of the 21CLD rubric is a tangible way for teachers to self-assess the level of 21 st Century skills in their lessons. Self-evaluation helps encourage reflection, promote professional growth, and recommendations for new aspects of lessons (Akram & Zepeda, 2015 ; Peterson & Comeaux, 1990 ). This can also help teachers make the instructions for the development of 21 st Century skills more explicit in their lesson. In conducting a self-evaluation, teachers may realize that they do not have a deep understanding of the characteristics of 21 st Century skills. If teachers are new to incorporating these skills into their lessons, the teachers may need time to learn the skills themselves before they can incorporate them into their lessons (Yoon et al., 2015 ). Further studies may examine how teachers use the 21CLD rubric to improve their lesson.

Students need time to grapple with and learn new skills (Lynch et al., 2017 ; NGSS Lead States, 2013 ). While we were able to see evidence of higher rubric scores for 21 st Century skills for 12th grade students in the lesson plans, due to the convenience sampling of lesson plans and student work samples, we were not able to look at how students’ 21 st Century skills were built over time. There is a desire to better understand how ISHSs successfully develop these skills. This includes how schools incorporate and build the 21 st Century skills (a) within multiple lessons in one course, (b) across multiple classes over the course of a school year, and (c) throughout the students’ entire high school sequence. Future research may look at a longitudinal study that follows one student’s work over an entire school year to see how the 21CLD scores change. In addition, future studies may also look at how the short-term projects build the skills needed for the students to incorporate higher levels of 21 st Century skills in long-term projects.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

21 st Century Learning Design

Critical component

Information and communication technology

Inclusive STEM high school

National Assessment of Educational Progress

Next-generation science standards

Opportunity Structures for Preparation and Inspiration in STEM

Partnership for 21 st Century Learning

Programme for International Student Assessment

Science, technology, engineering, and mathematics

Technology and engineering literacy

Ainley, J., Fraillon, J., Schulz, W., & Gebhardt, E. (2016). Conceptualizing and measuring computer and information literacy in cross-national contexts. Applied Measurement in Education, 29 , 291–309. https://doi.org/10.1080/08957347.2016.1209205 .

Article   Google Scholar  

Akram, M., & Zepeda, S. J. (2015). Development and validation of a teacher self-assessment instrument. Journal of Research and Reflections in Education, 9 (2), 134–148.

Google Scholar  

Behrend, T. S., Peters-Burton, E. E., Hudson, C., Matray, S., Ford, M., & Lynch, S. J. (2016). STEM High School Inventory. [Measurement instrument]. Retrieved from https://ospri.research.gwu.edu/sites/ospri.research.gwu.edu/files/downloads/CC%20Inventory_FINAL.pdf .

Bell, C. V., & Pape, S. J. (2014). Scaffolding the development of self-regulated learning in mathematics classrooms. Middle School Journal, 45 (4), 23–32.

Boekaerts, M. (2016). Engagement as an inherent aspect of the learning process. Learning and Instruction, 43 , 76–83. https://doi.org/10.1016/j.learninstruc.2016.02.001 .

Brown, A. L., Bransford, J., Ferrara, R., & Campione, J. (1983). Learning, remembering, and understanding. In P. H. Musen (Ed.), Handbook of child psychology (Vol. III, pp. 77–166). New York: Wiley.

Bybee, R. W. (2013). The case for STEM education . Arlington: NSTA press.

Care, E., Scoular, C., & Griffin, P. (2016). Assessment of collaborative problem solving in education environments. Applied Measurement in Education, 29 , 250–264. https://doi.org/10.1080/08957347.2016.1209204 .

Carpenter, J. P., & Pease, J. S. (2013). Preparing students to take responsibility for learning: The role of non-curricular learning strategies. Journal of Curriculum & Instruction, 7 (2), 38–55. https://doi.org/10.3776/joci.2013.v7n2p38-55 .

Carroll, M. (2015). Stretch, dream, and do—A 21 st century design thinking & STEM journey. Journal of Research in STEM Education, 1 (1), 59–70.

Chinn, C. A., & Malhortra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86 , 175–218. https://doi.org/10.1002/sce.10001 .

Claro, M., Preiss, D. D., San Martin, E., Jara, I., Hinostroza, J. E., Valenzuela, S., Cortes, F., & Nussbaun, M. (2012). Assessment of 21 st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59 , 1042–1053. https://doi.org/10.1016/j.compedu.2012.04.004 .

Eisenhart, M., Weis, L., Allen, C. D., Cipollone, K., Stich, A., & Dominguez, R. (2015). High school opportunities for STEM: Comparing inclusive STEM-focused and comprehensive high schools in two US cities. Journal of Research in Science Teaching, 52 (6), 763–789. https://doi.org/10.1002/tea.21213 .

English, M. C., & Kitsantas, A. (2013). Supporting student self-regulated learning in problem- and project based learning. Interdisciplinary Journal of Problem-Based Learning, 7 (2), 127–150. https://doi.org/10.7771/1541-5015.1339 .

Erdogan, N., & Stuessy, C. (2015). Examining the role of inclusive STEM schools in the college and career readiness of students in the United States: A multi-group analysis on the outcome of student achievement. Educational Sciences: Theory & Practice, 15 (6), 1517–1529. https://doi.org/10.12738/estp.2016.1.0072 .

Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners. The role of noncognitive factors in shaping school performance: A critical literature review . Chicago: University of Chicago Consortium on Chicago School Research.

Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. Weinert & U. R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 21–29). Hillsdale: Erlbaum.

Johnson, C. C., Peters-Burton, E. E., & Moore, T. J. (Eds.). (2016). STEM road map: A framework for integrated STEM education . New York: Routledge.

Koh, J. H. L., Chai, C. S., Benjamin, W., & Hong, H. Y. (2015). Technological pedagogical content knowledge (TPACK) and design thinking: A framework to support ICT lesson design for 21st century learning. Asia-Pacific Education Researcher (Springer Science & Business Media B.V.), 24 (3), 535–543. https://doi.org/10.1007/s40299-015-0237-2 .

Krippendorff, K. H. (2012). Content analysis: An introduction to its methodology (3rd ed.). Los Angeles: SAGE Publications, Inc..

LaForce, M., Noble, E., King, H., Century, J., Blackwell, C., Holt, S., Ibrahim, A., & Loo, S. (2016). The eight essential elements of inclusive STEM high schools. International Journal of STEM Education, 3 (21), 1–11. https://doi.org/10.1186/s40594-016-0054-z .

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4 , 863. https://doi.org/10.3389/fpsyg.2013.00863 .

Lynch, S. J., House, A., Peters-Burton, E., Behrend, T., Means, B., Ford, M., Spillane, N., Matray, S., Moore, I., Coyne, C., Williams, C., & Corn, J. (2015). A Logic model that describes and explains eight exemplary STEM-focused high schools with diverse student populations . Washington DC: George Washington University OSPrI Project Retrieved from http://ospri.research.gwu.edu .

Lynch, S. J., Means, B., Behrend, T., & Peters-Burton, E. (2011). Multiple instrumental case studies of inclusive STEM-focused high schools: Opportunity Structures for Preparation and Inspiration (OSPrI). Retrieved from http://ospri.research.gwu.edu

Lynch, S. J., Peters-Burton, E. E., Behrend, T., House, A., Ford, M., Spillane, N., Matray, S., Han, E., & Means, B. (2018). Understanding inclusive STEM high schools as opportunity structures for underrepresented students: Critical components. Journal of Research in Science Teaching., 55 (5), 712–748. https://doi.org/10.1002/tea.21437 .

Lynch, S. J., Peters-Burton, E. E., & Ford, M. (2014). Building STEM opportunities for all. Educational Leadership, 72 (4), 54–60.

Lynch, S. J., Spillane, N., House, A., Peters-Burton, E., Behrend, T., Ross, K. M., & Han, E. M. (2017). A policy-relevant instrument case study of an inclusive STEM-focused high school: Manor New Tech High. International Journal of Education in Mathematics, Science and Technology, 5 (1), 1–20. https://doi.org/10.18404/ijemst.75656 .

McCabe, A., & O’Connor, U. (2014). Student-centered learning: The role and responsibility of the lecturer. Teacher in Higher Education, 19 (4), 350–359. https://doi.org/10.1080/13562517.2013.860111 .

McNutt, M. (2013). Improving scientific communication. Science, 342 , 13. https://doi.org/10.1126/science.1246449 .

Means, B., Wang, H., Viki, Y., Peters, V. L., & Lynch, S. J. (2016). STEM-focused high schools as a strategy for enhancing readiness for postsecondary STEM programs. Journal of Research in Science Teaching, 53 (5), 709–736. https://doi.org/10.1002/tea.21313 .

National Research Council. (2013). Monitoring progress toward successful K-12 STEM education: A nation advancing? Washington, DC: National Academies Press. https://doi.org/10.17226/13509 .

Book   Google Scholar  

National Science Board. (2016). Science and engineering indicators 2016. (NSB-2016-1) . Arlington: National Science Foundation.

NGSS Lead States. (2013). Next generation science standards: For states, by states . Washington, DC: The National Academies Press.

Organisation for Economic Co-operation and Development. (2018). PISA 2015 results in focus. Retrieved from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf

Partnership for 21st Century Learning. (2016). Framework for 21st century learning. Retrieved from www.p21.org/about-us/p21-framework .

Peters, E. E. (2010). Shifting to a student-centered science classroom: An exploration of teacher and student changes in perceptions and practices. Journal of Science Teacher Education, 21 (3), 329–349. https://doi.org/10.1007/s10972-009-9178-z .

Peters, E. E., & Kitsantas, A. (2010). The effect of nature of science metacognitive prompts on science students’ content and nature of science knowledge, metacognition, and self-regulatory efficacy. School Science and Mathematics, 110 , 382–396. https://doi.org/10.1111/j.1949-8594.2010.00050.x .

Peters-Burton, E. E., House, A., Han, E., & Lynch, S. (2018). Curriculum and instruction at inclusive STEM high schools. Journal of Research in STEM Education, 4 (2), 193–212.

Peters-Burton, E. E., Lynch, S. J., Behrend, T. S., & Means, B. B. (2014). Inclusive STEM high school design: 10 critical components. Theory into Practice, 53 (1), 64–71. https://doi.org/10.1080/00405841.2014.862125 .

Peterson, P. L., & Comeaux, M. A. (1990). Evaluating the systems: Teachers’ perspectives on teacher evaluation. Educational Evaluation and Policy Analysis, 12 (1), 3–24. https://doi.org/10.3102/01623737012001003 .

President’s Council of Advisors on Science and Technology. (2010). Prepare and inspire: K-12 education in science, technology, engineering, and math (STEM) for America’s future . Washington, DC: Executive Office of the President.

Prettyman, S. S., Ward, C. L., Jauk, D., & Awad, G. (2012). 21st century learners: Voices of students in a one-to-one STEM environment. Journal of Applied Learning Technology, 2 (4), 6–15.

Shear, L., Novais, G., Means, B., Gallagher, L., & Langworthy, M. (2010). ITL research design . Menlo Park: SRI International Retrieved from https://www.sri.com/sites/default/files/publications/itl_research_design_15_nov_2010.pdf .

SRI International. (n.d.-a). 21CLD learning activity rubrics. Retrieved from https://education.microsoft.com/GetTrained/ITL-Research

SRI International. (n.d.-b). 21CLD student work rubrics. Retrieved from https://education.microsoft.com/GetTrained/ITL-Research

Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social & behavioral research . Thousand Oaks: Sage.

Texas Education Association (n.d.). Texas science, technology, engineering and mathematics initiative (T-STEM). Retrieved from https://tea.texas.gov/T-STEM/

Tofel-Grehl, C., & Callahan, C. M. (2014). STEM high school communities: Common and differing features. Journal of Advanced Academics, 25 (3), 237–271. https://doi.org/10.1177/1932202X14539156 .

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015a). The nation’s report card: 2015 mathematics and reading assessments. (NCES No. 2015136). Retrieved from https://www.nationsreportcard.gov/reading_math_2015/#?grade=4

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015b). The nation’s report card: Technology and engineering literacy. (NCES No. 2016119). Retrieved from https://www.nationsreportcard.gov/tel_2014/

van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21 st -century skills and digital skills: A systematic literature review. Computers in Human Behavior, 77 , 577–588. https://doi.org/10.1016/j.chb.2017.03.010 .

Voss, J. F., & Post, T. A. (1988). On the solving of ill-structured problems. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 261–285). Hillsdale: Lawrence Erlbaum.

Warin, B., Talbi, O., Kolski, C., & Hoogstoel, F. (2016). Multi-role project (MRP): A new project-based learning method for STEM. IEEE Transactions on Education, 59 (2), 137–146. https://doi.org/10.1109/TE.2015.2462809 .

White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16 (1), 3–18.

Williams, P. J., & Mangan, J. (2016). The effectiveness of using young professionals to influence STEM career choices of secondary school students. Journal of Research in STEM Education, 2 (1), 2–18.

Yoon, S. A., Anderson, E., Koehler-Yom, J., Klopfer, E., Sheldon, J., Wendel, D., Schoenfeld, I., Scheintaub, H., Oztok, M., & Evans, C. (2015). Designing curriculum and instruction for computer-supported complex systems teaching and learning in high school science classrooms. Journal of Research in STEM Education, 1 (1), 4–14.

Zimmerman, B. J. (1998). Developing self-fulfilling cycles of academic regulation: An analysis of exemplary instructional models. In D. H. Schunk & B. J. Zimmerman (Eds.), Self-regulated learning: From teaching to self-reflective practice (pp. 1–19). New York: The Guilford Press.

Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). San Diego: Academic Press.

Chapter   Google Scholar  

Download references

Acknowledgments

Publication of this article was funded in part by the George Mason University Libraries Open Access Publishing Fund.

This work was conducted by the OSPrI research project, with Sharon Lynch, Tara Behrend, Erin Peters-Burton, and Barbara Means as principal investigators. Funding for OSPrI was provided by the National Science Foundation (DRL 1118851). Any opinions, findings, conclusions, or recommendations are those of the authors and do not necessarily reflect the position or policy of endorsement of the funding agency.

Author information

Authors and affiliations.

George Mason University, Fairfax, USA

Stephanie M. Stehle & Erin E. Peters-Burton

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors contributed equally to this manuscript. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Stephanie M. Stehle .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Stehle, S.M., Peters-Burton, E.E. Developing student 21 st Century skills in selected exemplary inclusive STEM high schools. IJ STEM Ed 6 , 39 (2019). https://doi.org/10.1186/s40594-019-0192-1

Download citation

Received : 21 December 2018

Accepted : 04 October 2019

Published : 25 November 2019

DOI : https://doi.org/10.1186/s40594-019-0192-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • STEM schools

research papers on 21st century

  • Position paper
  • Open access
  • Published: 28 November 2019

Physics education research for 21 st century learning

  • Lei Bao   ORCID: orcid.org/0000-0003-3348-4198 1 &
  • Kathleen Koenig 2  

Disciplinary and Interdisciplinary Science Education Research volume  1 , Article number:  2 ( 2019 ) Cite this article

36k Accesses

69 Citations

2 Altmetric

Metrics details

Education goals have evolved to emphasize student acquisition of the knowledge and attributes necessary to successfully contribute to the workforce and global economy of the twenty-first Century. The new education standards emphasize higher end skills including reasoning, creativity, and open problem solving. Although there is substantial research evidence and consensus around identifying essential twenty-first Century skills, there is a lack of research that focuses on how the related subskills interact and develop over time. This paper provides a brief review of physics education research as a means for providing a context towards future work in promoting deep learning and fostering abilities in high-end reasoning. Through a synthesis of the literature around twenty-first Century skills and physics education, a set of concretely defined education and research goals are suggested for future research, along with how these may impact the next generation physics courses and how physics should be taught in the future.

Introduction

Education is the primary service offered by society to prepare its future generation workforce. The goals of education should therefore meet the demands of the changing world. The concept of learner-centered, active learning has broad, growing support in the research literature as an empirically validated teaching practice that best promotes learning for modern day students (Freeman et al., 2014 ). It stems out of the constructivist view of learning, which emphasizes that it is the learner who needs to actively construct knowledge and the teacher should assume the role of a facilitator rather than the source of knowledge. As implied by the constructivist view, learner-centered education usually emphasizes active-engagement and inquiry style teaching-learning methods, in which the learners can effectively construct their understanding under the guidance of instruction. The learner-centered education also requires educators and researchers to focus their efforts on the learners’ needs, not only to deliver effective teaching-learning approaches, but also to continuously align instructional practices to the education goals of the times. The goals of introductory college courses in science, technology, engineering, and mathematics (STEM) disciplines have constantly evolved from some notion of weed-out courses that emphasize content drilling, to the current constructivist active-engagement type of learning that promotes interest in STEM careers and fosters high-end cognitive abilities.

Following the conceptually defined framework of twenty-first Century teaching and learning, this paper aims to provide contextualized operational definitions of the goals for twenty-first Century learning in physics (and STEM in general) as well as the rationale for the importance of these outcomes for current students. Aligning to the twenty-first Century learning goals, research in physics education is briefly reviewed to provide a context towards future work in promoting deep learning and fostering abilities in high-end reasoning in parallel. Through a synthesis of the literature around twenty-first Century skills and physics education, a set of concretely defined education and research goals are suggested for future research. These goals include: domain-specific research in physics learning; fostering scientific reasoning abilities that are transferable across the STEM disciplines; and dissemination of research-validated curriculum and approaches to teaching and learning. Although this review has a focus on physics education research (PER), it is beneficial to expand the perspective to view physics education in the broader context of STEM learning. Therefore, much of the discussion will blend PER with STEM education as a continuum body of work on teaching and learning.

Education goals for twenty-first century learning

Education goals have evolved to emphasize student acquisition of essential “21 st Century skills”, which define the knowledge and attributes necessary to successfully contribute to the workforce and global economy of the 21st Century (National Research Council, 2011 , 2012a ). In general, these standards seek to transition from emphasizing content-based drilling and memorization towards fostering higher-end skills including reasoning, creativity, and open problem solving (United States Chamber of Commerce, 2017 ). Initiatives on advancing twenty-first Century education focus on skills that converge on three broad clusters: cognitive, interpersonal, and intrapersonal, all of which include a rich set of sub-dimensions.

Within the cognitive domain, multiple competencies have been proposed, including deep learning, non-routine problem solving, systems thinking, critical thinking, computational and information literacy, reasoning and argumentation, and innovation (National Research Council, 2012b ; National Science and Technology Council, 2018 ). Interpersonal skills are those necessary for relating to others, including the ability to work creatively and collaboratively as well as communicate clearly. Intrapersonal skills, on the other hand, reside within the individual and include metacognitive thinking, adaptability, and self-management. These involve the ability to adjust one’s strategy or approach along with the ability to work towards important goals without significant distraction, both essential for sustained success in long-term problem solving and career development.

Although many descriptions exist for what qualifies as twenty-first Century skills, student abilities in scientific reasoning and critical thinking are the most commonly noted and widely studied. They are highly connected with the other cognitive skills of problem solving, decision making, and creative thinking (Bailin, 1996 ; Facione, 1990 ; Fisher, 2001 ; Lipman, 2003 ; Marzano et al., 1988 ), and have been important educational goals since the 1980s (Binkley et al., 2010 ; NCET, 1987 ). As a result, they play a foundational role in defining, assessing, and developing twenty-first Century skills.

The literature for critical thinking is extensive (Bangert-Drowns & Bankert, 1990 ; Facione, 1990 ; Glaser, 1941 ). Various definitions exist with common underlying principles. Broadly defined, critical thinking is the application of the cognitive skills and strategies that aim for and support evidence-based decision making. It is the thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions (Halpern, 1999 ). It is the “reasonable reflective thinking focused on deciding what to believe or do” (Ennis, 1993 ). Critical thinking is recognized as a way to understand and evaluate subject matter; producing reliable knowledge and improving thinking itself (Paul, 1990 ; Siegel, 1988 ).

The notion of scientific reasoning is often used to label the set of skills that support critical thinking, problem solving, and creativity in STEM. Broadly defined, scientific reasoning includes the thinking and reasoning skills involved in inquiry, experimentation, evidence evaluation, inference and argument that support the formation and modification of concepts and theories about the natural world; such as the ability to systematically explore a problem, formulate and test hypotheses, manipulate and isolate variables, and observe and evaluate consequences (Bao et al., 2009 ; Zimmerman, 2000 ). Critical thinking and scientific reasoning share many features, where both emphasize evidence-based decision making in multivariable causal conditions. Critical thinking can be promoted through the development of scientific reasoning, which includes student ability to reach a reliable conclusion after identifying a question, formulating hypotheses, gathering relevant data, and logically testing and evaluating the hypothesis. In this way, scientific reasoning can be viewed as a scientific domain instantiation of critical thinking in the context of STEM learning.

In STEM learning, cognitive aspects of the twenty-first Century skills aim to develop reasoning skills, critical thinking skills, and deep understanding, all of which allow students to develop well connected expert-like knowledge structures and engage in meaningful scientific inquiry and problem solving. Within physics education, a core component of STEM education, the learning of conceptual understanding and problem solving remains a current emphasis. However, the fast-changing work environment and technology-driven world require a new set of core knowledge, skills, and habits of mind to solve complex interdisciplinary problems, gather and evaluate evidence, and make sense of information from a variety of sources (Tanenbaum, 2016 ). The education goals in physics are transitioning towards ability fostering as well as extension and integration with other STEM disciplines. Although curriculum that supports these goals is limited, there are a number of attempts, particularly in developing active learning classrooms and inquiry-based laboratory activities, which have demonstrated success. Some of these are described later in this paper as they provide a foundation for future work in physics education.

Interpersonal skills, such as communication and collaboration, are also essential for twenty-first Century problem-solving tasks, which are often open-ended, complex, and team-based. As the world becomes more connected in a multitude of dimensions, tackling significant problems involving complex systems often goes beyond the individual and requires working with others who are increasingly from culturally diverse backgrounds. Due to the rise of communication technologies, being able to articulate thoughts and ideas in a variety of formats and contexts is crucial, as well as the ability to effectively listen or observe to decipher meaning. Interpersonal skills can be promoted by integrating group-learning experiences into the classroom setting, while providing students with the opportunity to engage in open-ended tasks with a team of peer learners who may propose more than one plausible solution. These experiences should be designed such that students must work collaboratively and responsibly in teams to develop creative solutions, which are later disseminated through informative presentations and clearly written scientific reports. Although educational settings in general have moved to providing students with more and more opportunities for collaborative learning, a lack of effective assessments for these important skills has been a limiting factor for producing informative research and widespread implementation. See Liu ( 2010 ) for an overview of measurement instruments reported in the research literature.

Intrapersonal skills are based on the individual and include the ability to manage one’s behavior and emotions to achieve goals. These are especially important for adapting in the fast-evolving collaborative modern work environment and for learning new tasks to solve increasingly challenging interdisciplinary problems, both of which require intellectual openness, work ethic, initiative, and metacognition, to name a few. These skills can be promoted using instruction which, for example, includes metacognitive learning strategies, provides opportunities to make choices and set goals for learning, and explicitly connects to everyday life events. However, like interpersonal skills, the availability of relevant assessments challenges advancement in this area. In this review, the vast amount of studies on interpersonal and intrapersonal skills will not be discussed in order to keep the main focus on the cognitive side of skills and reasoning.

The purpose behind discussing twenty-first Century skills is that this set of skills provides important guidance for establishing essential education goals for modern society and learners. However, although there is substantial research evidence and consensus around identifying necessary twenty-first Century skills, there is a lack of research that focuses on how the related subskills interact and develop over time (Reimers & Chung, 2016 ), with much of the existing research residing in academic literature that is focused on psychology rather than education systems (National Research Council, 2012a ). Therefore, a major and challenging task for discipline-based education researchers and educators is to operationally define discipline-specific goals that align with the twenty-first Century skills for each of the STEM fields. In the following sections, this paper will provide a limited vision of the research endeavors in physics education that can translate the past and current success into sustained impact for twenty-first Century teaching and learning.

Proposed education and research goals

Physics education research (PER) is often considered an early pioneer in discipline-based education research (National Research Council, 2012c ), with well-established, broad, and influential outcomes (e.g., Hake, 1998 ; Hsu, Brewe, Foster, & Harper, 2004 ; McDermott & Redish, 1999 ; Meltzer & Thornton, 2012 ). Through the integration of twenty-first Century skills with the PER literature, a set of broadly defined education and research goals is proposed for future PER work:

Discipline-specific deep learning: Cognitive and education research involving physics learning has established a rich literature on student learning behaviors along with a number of frameworks. Some of the popular frameworks include conceptual understanding and concept change, problem solving, knowledge structure, deep learning, and knowledge integration. Aligned with twenty-first Century skills, future research in physics learning should aim to integrate the multiple areas of existing work, such that they help students develop well integrated knowledge structures in order to achieve deep leaning in physics.

Fostering scientific reasoning for transfer across STEM disciplines: The broad literature in physics learning and scientific reasoning can provide a solid foundation to further develop effective physics education approaches, such as active engagement instruction and inquiry labs, specifically targeting scientific inquiry abilities and reasoning skills. Since scientific reasoning is a more domain-general cognitive ability, success in physics can also more readily inform research and education practices in other STEM fields.

Research, development, assessment, and dissemination of effective education approaches: Developing and maintaining a supportive infrastructure of education research and implementation has always been a challenge, not only in physics but in all STEM areas. The twenty-first Century education requires researchers and instructors across STEM to work together as an extended community in order to construct a sustainable integrated STEM education environment. Through this new infrastructure, effective team-based inquiry learning and meaningful assessment can be delivered to help students develop a comprehensive skills set including deep understanding and scientific reasoning, as well as communication and other non-cognitive abilities.

The suggested research will generate understanding and resources to support education practices that meet the requirements of the Next Generation Science Standards (NGSS), which explicitly emphasize three areas of learning including disciplinary core ideas, crosscutting concepts, and practices (National Research Council, 2012b ). The first goal for promoting deep learning of disciplinary knowledge corresponds well to the NGSS emphasis on disciplinary core ideas, which play a central role in helping students develop well integrated knowledge structures to achieve deep understanding. The second goal on fostering transferable scientific reasoning skills supports the NGSS emphasis on crosscutting concepts and practices. Scientific reasoning skills are crosscutting cognitive abilities that are essential to the development of domain-general concepts and modeling strategies. In addition, the development of scientific reasoning requires inquiry-based learning and practices. Therefore, research on scientific reasoning can produce a valuable knowledge base on education means that are effective for developing crosscutting concepts and promoting meaningful practices in STEM. The third research goal addresses the challenge in the assessment of high-end skills and the dissemination of effective educational approaches, which supports all NGSS initiatives to ensure sustainable development and lasting impact. The following sections will discuss the research literature that provides the foundation for these three research goals and identify the specific challenges that will need to be addressed in future work.

Promoting deep learning in physics education

Physics education for the twenty-first Century aims to foster high-end reasoning skills and promote deep conceptual understanding. However, many traditional education systems place strong emphasis on only problem solving with the expectation that students obtain deep conceptual understanding through repetitive problem-solving practices, which often doesn’t occur (Alonso, 1992 ). This focus on problem solving has been shown to have limitations as a number of studies have revealed disconnections between learning conceptual understanding and problem-solving skills (Chiu, 2001 ; Chiu, Guo, & Treagust, 2007 ; Hoellwarth, Moelter, & Knight, 2005 ; Kim & Pak, 2002 ; Nakhleh, 1993 ; Nakhleh & Mitchell, 1993 ; Nurrenbern & Pickering, 1987 ; Stamovlasis, Tsaparlis, Kamilatos, Papaoikonomou, & Zarotiadou, 2005 ). In fact, drilling in problem solving may actually promote memorization of context-specific solutions with minimal generalization rather than transitioning students from novices to experts.

Towards conceptual understanding and learning, many models and definitions have been established to study and describe student conceptual knowledge states and development. For example, students coming into a physics classroom often hold deeply rooted, stable understandings that differ from expert conceptions. These are commonly referred to as misconceptions or alternative conceptions (Clement, 1982 ; Duit & Treagust, 2003 ; Dykstra Jr, Boyle, & Monarch, 1992 ; Halloun & Hestenes, 1985a , 1985b ). Such students’ conceptions are context dependent and exist as disconnected knowledge fragments, which are strongly situated within specific contexts (Bao & Redish, 2001 , 2006 ; Minstrell, 1992 ).

In modeling students’ knowledge structures, DiSessa’s proposed phenomenological primitives (p-prim) describe a learner’s implicit thinking, cued from specific contexts, as an underpinning cognitive construct for a learner’s expressed conception (DiSessa, 1993 ; Smith III, DiSessa, & Roschelle, 1994 ). Facets, on the other hand, map between the implicit p-prim and concrete statements of beliefs and are developed as discrete and independent units of thought, knowledge, or strategies used by individuals to address specific situations (Minstrell, 1992 ). Ontological categories, defined by Chi, describe student reasoning in the most general sense. Chi believed that these are distinct, stable, and constraining, and that a core reason behind novices’ difficulties in physics is that they think of physics within the category of matter instead of processes (Chi, 1992 ; Chi & Slotta, 1993 ; Chi, Slotta, & De Leeuw, 1994 ; Slotta, Chi, & Joram, 1995 ). More details on conceptual learning and problem solving are well summarized in the literature (Hsu et al., 2004 ; McDermott & Redish, 1999 ), from which a common theme emerges from the models and definitions. That is, learning is context dependent and students with poor conceptual understanding typically have locally connected knowledge structures with isolated conceptual constructs that are unable to establish similarities and contrasts between contexts.

Additionally, this idea of fragmentation is demonstrated through many studies on student problem solving in physics and other fields. It has been shown that a student’s knowledge organization is a key aspect for distinguishing experts from novices (Bagno, Eylon, & Ganiel, 2000 ; Chi, Feltovich, & Glaser, 1981 ; De Jong & Ferguson-Hesler, 1986 ; Eylon & Reif, 1984 ; Ferguson-Hesler & De Jong, 1990 ; Heller & Reif, 1984 ; Larkin, McDermott, Simon, & Simon, 1980 ; Smith, 1992 ; Veldhuis, 1990 ; Wexler, 1982 ). Expert’s knowledge is organized around core principles of physics, which are applied to guide problem solving and develop connections between different domains as well as new, unfamiliar situations (Brown, 1989 ; Perkins & Salomon, 1989 ; Salomon & Perkins, 1989 ). Novices, on the other hand, lack a well-organized knowledge structure and often solve problems by relying on surface features that are directly mapped to certain problem-solving outcomes through memorization (Chi, Bassok, Lewis, Reimann, & Glaser, 1989 ; Hardiman, Dufresne, & Mestre, 1989 ; Schoenfeld & Herrmann, 1982 ).

This lack of organization creates many difficulties in the comprehension of basic concepts and in solving complex problems. This leads to the common complaint that students’ knowledge of physics is reduced to formulas and vague labels of the concepts, which are unable to substantively contribute to meaningful reasoning processes. A novice’s fragmented knowledge structure severely limits the learner’s conceptual understanding. In essence, these students are able to memorize how to approach a problem given specific information but lack the understanding of the underlying concept of the approach, limiting their ability to apply this approach to a novel situation. In order to achieve expert-like understanding, a student’s knowledge structure must integrate all of the fragmented ideas around the core principle to form a coherent and fully connected conceptual framework.

Towards a more general theoretical consideration, students’ alternative conceptions and fragmentation in knowledge structures can be viewed through both the “naïve theory” framework (e.g., Posner, Strike, Hewson, & Gertzog, 1982 ; Vosniadou, Vamvakoussi, & Skopeliti, 2008 ) and the “knowledge in pieces” (DiSessa, 1993 ) perspective. The “naïve theory” framework considers students entering the classroom with stable and coherent ideas (naïve theories) about the natural world that differ from those presented by experts. In the “knowledge in pieces” perspective, student knowledge is constructed in real-time and incorporates context features with the p-prims to form the observed conceptual expressions. Although there exists an ongoing debate between these two views (Kalman & Lattery, 2018 ), it is more productive to focus on their instructional implications for promoting meaningful conceptual change in students’ knowledge structures.

In the process of learning, students may enter the classroom with a range of initial states depending on the population and content. For topics with well-established empirical experiences, students often have developed their own ideas and understanding, while on topics without prior exposure, students may create their initial understanding in real-time based on related prior knowledge and given contextual features (Bao & Redish, 2006 ). These initial states of understanding, regardless of their origin, are usually different from those of experts. Therefore, the main function of teaching and learning is to guide students to modify their initial understanding towards the experts’ views. Although students’ initial understanding may exist as a body of coherent ideas within limited contexts, as students start to change their knowledge structures throughout the learning process, they may evolve into a wide range of transitional states with varying levels of knowledge integration and coherence. The discussion in this brief review on students’ knowledge structures regarding fragmentation and integration are primarily focused on the transitional stages emerged through learning.

The corresponding instructional goal is then to help students more effectively develop an integrated knowledge structure so as to achieve a deep conceptual understanding. From an educator’s perspective, Bloom’s taxonomy of education objectives establishes a hierarchy of six levels of cognitive skills based on their specificity and complexity: Remember (lowest and most specific), Understand, Apply, Analyze, Evaluate, and Create (highest and most general and complex) (Anderson et al., 2001 ; Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956 ). This hierarchy of skills exemplifies the transition of a learner’s cognitive development from a fragmented and contextually situated knowledge structure (novice with low level cognitive skills) to a well-integrated and globally networked expert-like structure (with high level cognitive skills).

As a student’s learning progresses from lower to higher cognitive levels, the student’s knowledge structure becomes more integrated and is easier to transfer across contexts (less context specific). For example, beginning stage students may only be able to memorize and perform limited applications of the features of certain contexts and their conditional variations, with which the students were specifically taught. This leads to the establishment of a locally connected knowledge construct. When a student’s learning progresses from the level of Remember to Understand, the student begins to develop connections among some of the fragmented pieces to form a more fully connected network linking a larger set of contexts, thus advancing into a higher level of understanding. These connections and the ability to transfer between different situations form the basis of deep conceptual understanding. This growth of connections leads to a more complete and integrated cognitive structure, which can be mapped to a higher level on Bloom’s taxonomy. This occurs when students are able to relate a larger number of different contextual and conditional aspects of a concept for analyzing and evaluating to a wider variety of problem situations.

Promoting the growth of connections would appear to aid in student learning. Exactly which teaching methods best facilitate this are dependent on the concepts and skills being learned and should be determined through research. However, it has been well recognized that traditional instruction often fails to help students obtain expert-like conceptual understanding, with many misconceptions still existing after instruction, indicating weak integration within a student’s knowledge structure (McKeachie, 1986 ).

Recognizing the failures of traditional teaching, various research-informed teaching methods have been developed to enhance student conceptual learning along with diagnostic tests, which aim to measure the existence of misconceptions. Most advances in teaching methods focus on the inclusion of inquiry-based interactive-engagement elements in lecture, recitations, and labs. In physics education, these methods were popularized after Hake’s landmark study demonstrated the effectiveness of interactive-engagement over traditional lectures (Hake, 1998 ). Some of these methods include the use of peer instruction (Mazur, 1997 ), personal response systems (e.g., Reay, Bao, Li, Warnakulasooriya, & Baugh, 2005 ), studio-style instruction (Beichner et al., 2007 ), and inquiry-based learning (Etkina & Van Heuvelen, 2001 ; Laws, 2004 ; McDermott, 1996 ; Thornton & Sokoloff, 1998 ). The key approach of these methods aims to improve student learning by carefully targeting deficits in student knowledge and actively encouraging students to explore and discuss. Rather than rote memorization, these approaches help promote generalization and deeper conceptual understanding by building connections between knowledge elements.

Based on the literature, including Bloom’s taxonomy and the new education standards that emphasize twenty-first Century skills, a common focus on teaching and learning can be identified. This focus emphasizes helping students develop connections among fragmented segments of their knowledge pieces and is aligned with the knowledge integration perspective, which focuses on helping students develop and refine their knowledge structure toward a more coherently organized and extensively connected network of ideas (Lee, Liu, & Linn, 2011 ; Linn, 2005 ; Nordine, Krajcik, & Fortus, 2011 ; Shen, Liu, & Chang, 2017 ). For meaningful learning to occur, new concepts must be integrated into a learner’s existing knowledge structure by linking the new knowledge to already understood concepts.

Forming an integrated knowledge structure is therefore essential to achieving deep learning, not only in physics but also in all STEM fields. However, defining what connections must occur at different stages of learning, as well as understanding the instructional methods necessary for effectively developing such connections within each STEM disciplinary context, are necessary for current and future research. Together these will provide the much needed foundational knowledge base to guide the development of the next generation of curriculum and classroom environment designed around twenty-first Century learning.

Developing scientific reasoning with inquiry labs

Scientific reasoning is part of the widely emphasized cognitive strand of twenty-first Century skills. Through development of scientific reasoning skills, students’ critical thinking, open-ended problem-solving abilities, and decision-making skills can be improved. In this way, targeting scientific reasoning as a curricular objective is aligned with the goals emphasized in twenty-first Century education. Also, there is a growing body of research on the importance of student development of scientific reasoning, which have been found to positively correlate with course achievement (Cavallo, Rozman, Blickenstaff, & Walker, 2003 ; Johnson & Lawson, 1998 ), improvement on concept tests (Coletta & Phillips, 2005 ; She & Liao, 2010 ), engagement in higher levels of problem solving (Cracolice, Deming, & Ehlert, 2008 ; Fabby & Koenig, 2013 ); and success on transfer (Ates & Cataloglu, 2007 ; Jensen & Lawson, 2011 ).

Unfortunately, research has shown that college students are lacking in scientific reasoning. Lawson ( 1992 ) found that ~ 50% of intro biology students are not capable of applying scientific reasoning in learning, including the ability to develop hypotheses, control variables, and design experiments; all necessary for meaningful scientific inquiry. Research has also found that traditional courses do not significantly develop these abilities, with pre-to-post-test gains of 1%–2%, while inquiry-based courses have gains around 7% (Koenig, Schen, & Bao, 2012 ; Koenig, Schen, Edwards, & Bao, 2012 ). Others found that undergraduates have difficulty developing evidence-based decisions and differentiating between and linking evidence with claims (Kuhn, 1992 ; Shaw, 1996 ; Zeineddin & Abd-El-Khalick, 2010 ). A large scale international study suggested that learning of physics content knowledge with traditional teaching practices does not improve students’ scientific reasoning skills (Bao et al., 2009 ).

Aligned to twenty-first Century learning, it is important to implement curriculum that is specifically designed for developing scientific reasoning abilities within current education settings. Although traditional lectures may continue for decades due to infrastructure constraints, a unique opportunity can be found in the lab curriculum, which may be more readily transformed to include hands-on minds-on group learning activities that are ideal for developing students’ abilities in scientific inquiry and reasoning.

For well over a century, the laboratory has held a distinctive role in student learning (Meltzer & Otero, 2015 ). However, many existing labs, which haven’t changed much since the late 1980s, have received criticism for their outdated cookbook style that lacks effectiveness in developing high-end skills. In addition, labs have been primarily used as a means for verifying the physical principles presented in lecture, and unfortunately, Hofstein and Lunetta ( 1982 ) found in an early review of the literature that research was unable to demonstrate the impact of the lab on student content learning.

About this same time, a shift towards a constructivist view of learning gained popularity and influenced lab curriculum development towards engaging students in the process of constructing knowledge through science inquiry. Curricula, such as Physics by Inquiry (McDermott, 1996 ), Real-Time Physics (Sokoloff, Thornton, & Laws, 2011 ), and Workshop Physics (Laws, 2004 ), were developed with a primary focus on engaging students in cognitive conflict to address misconceptions. Although these approaches have been shown to be highly successful in improving deep learning of physics concepts (McDermott & Redish, 1999 ), the emphasis on conceptual learning does not sufficiently impact the domain general scientific reasoning skills necessitated in the goals of twenty-first Century learning.

Reform in science education, both in terms of targeted content and skills, along with the emergence of knowledge regarding human cognition and learning (Bransford, Brown, & Cocking, 2000 ), have generated renewed interest in the potential of inquiry-based lab settings for skill development. In these types of hands-on minds-on learning, students apply the methods and procedures of science inquiry to investigate phenomena and construct scientific claims, solve problems, and communicate outcomes, which holds promise for developing both conceptual understanding and scientific reasoning skills in parallel (Trowbridge, Bybee, & Powell, 2000 ). In addition, the availability of technology to enhance inquiry-based learning has seen exponential growth, along with the emergence of more appropriate research methodologies to support research on student learning.

Although inquiry-based labs hold promise for developing students’ high-end reasoning, analytic, and scientific inquiry abilities, these educational endeavors have not become widespread, with many existing physics laboratory courses still viewed merely as a place to illustrate the physical principles from the lecture course (Meltzer & Otero, 2015 ). Developing scientific ideas from practical experiences, however, is a complex process. Students need sufficient time and opportunity for interaction and reflection on complex, investigative tasks. Blended learning, which merges lecture and lab (such as studio style courses), addresses this issue to some extent, but has experienced limited adoption, likely due to the demanding infrastructure resources, including dedicated technology-intensive classroom space, equipment and maintenance costs, and fully committed trained staff.

Therefore, there is an immediate need to transform the existing standalone lab courses, within the constraints of the existing education infrastructure, into more inquiry-based designs, with one of its primary goals dedicated to developing scientific reasoning skills. These labs should center on constructing knowledge, along with hands-on minds-on practical skills and scientific reasoning, to support modeling a problem, designing and implementing experiments, analyzing and interpreting data, drawing and evaluating conclusions, and effective communication. In particular, training on scientific reasoning needs to be explicitly addressed in the lab curriculum, which should contain components specifically targeting a set of operationally-defined scientific reasoning skills, such as ability to control variables or engage in multivariate causal reasoning. Although effective inquiry may also implicitly develop some aspects of scientific reasoning skills, such development is far less efficient and varies with context when the primary focus is on conceptual learning.

Several recent efforts to enhance the standalone lab course have shown promise in supporting education goals that better align with twenty-first Century learning. For example, the Investigative Science Learning Environment (ISLE) labs involve a series of tasks designed to help students develop the “habits of mind” of scientists and engineers (Etkina et al., 2006 ). The curriculum targets reasoning as well as the lab learning outcomes published by the American Association of Physics Teachers (Kozminski et al., 2014 ). Operationally, ISLE methods focus on scaffolding students’ developing conceptual understanding using inquiry learning without a heavy emphasis on cognitive conflict, making it more appropriate and effective for entry level students and K-12 teachers.

Likewise, Koenig, Wood, Bortner, and Bao ( 2019 ) have developed a lab curriculum that is intentionally designed around the twenty-first Century learning goals for developing cognitive, interpersonal, and intrapersonal abilities. In terms of the cognitive domain, the lab learning outcomes center on critical thinking and scientific reasoning but do so through operationally defined sub-skills, all of which are transferrable across STEM. These selected sub-skills are found in the research literature, and include the ability to control variables and engage in data analytics and causal reasoning. For each targeted sub-skill, a series of pre-lab and in-class activities provide students with repeated, deliberate practice within multiple hypothetical science-based scenarios followed by real inquiry-based lab contexts. This explicit instructional strategy has been shown to be essential for the development of scientific reasoning (Chen & Klahr, 1999 ). In addition, the Karplus Learning Cycle (Karplus, 1964 ) provides the foundation for the structure of the lab activities and involves cycles of exploration, concept introduction, and concept application. The curricular framework is such that as the course progresses, the students engage in increasingly complex tasks, which allow students the opportunity to learn gradually through a progression from simple to complex skills.

As part of this same curriculum, students’ interpersonal skills are developed, in part, through teamwork, as students work in groups of 3 or 4 to address open-ended research questions, such as, What impacts the period of a pendulum? In addition, due to time constraints, students learn early on about the importance of working together in an efficient manor towards a common goal, with one set of written lab records per team submitted after each lab. Checkpoints built into all in-class activities involve Socratic dialogue between the instructor and students and promote oral communication. This use of directed questioning guides students in articulating their reasoning behind decisions and claims made, while supporting the development of scientific reasoning and conceptual understanding in parallel (Hake, 1992 ). Students’ intrapersonal skills, as well as communication skills, are promoted through the submission of individual lab reports. These reports require students to reflect upon their learning over each of four multi-week experiments and synthesize their ideas into evidence-based arguments, which support a claim. Due to the length of several weeks over which students collect data for each of these reports, the ability to organize the data and manage their time becomes essential.

Despite the growing emphasis on research and development of curriculum that targets twenty-first Century learning, converting a traditionally taught lab course into a meaningful inquiry-based learning environment is challenging in current reform efforts. Typically, the biggest challenge is a lack of resources; including faculty time to create or adapt inquiry-based materials for the local setting, training faculty and graduate student instructors who are likely unfamiliar with this approach, and the potential cost of new equipment. Koenig et al. ( 2019 ) addressed these potential implementation barriers by designing curriculum with these challenges in mind. That is, the curriculum was designed as a flexible set of modules that target specific sub-skills, with each module consisting of pre-lab (hypothetical) and in-lab (real) activities. Each module was designed around a curricular framework such that an adopting institution can use the materials as written, or can incorporate their existing equipment and experiments into the framework with minimal effort. Other non-traditional approaches have also been experimented with, such as the work by Sobhanzadeh, Kalman, and Thompson ( 2017 ), which targets typical misconceptions by using conceptual questions to engage students in making a prediction, designing and conducting a related experiment, and determining whether or not the results support the hypothesis.

Another challenge for inquiry labs is the assessment of skills-based learning outcomes. For assessment of scientific reasoning, a new instrument on inquiry in scientific thinking analytics and reasoning (iSTAR) has been developed, which can be easily implemented across large numbers of students as both a pre- and post-test to assess gains. iSTAR assesses reasoning skills necessary in the systematical conduct of scientific inquiry, which includes the ability to explore a problem, formulate and test hypotheses, manipulate and isolate variables, and observe and evaluate the consequences (see www.istarassessment.org ). The new instrument expands upon the commonly used classroom test of scientific reasoning (Lawson, 1978 , 2000 ), which has been identified with a number of validity weaknesses and a ceiling effect for college students (Bao, Xiao, Koenig, & Han, 2018 ).

Many education innovations need supporting infrastructures that can ensure adoption and lasting impact. However, making large-scale changes to current education settings can be risky, if not impossible. New education approaches, therefore, need to be designed to adapt to current environmental constraints. Since higher-end skills are a primary focus of twenty-first Century learning, which are most effectively developed in inquiry-based group settings, transforming current lecture and lab courses into this new format is critical. Although this transformation presents great challenges, promising solutions have already emerged from various research efforts. Perhaps the biggest challenge is for STEM educators and researchers to form an alliance to work together to re-engineer many details of the current education infrastructure in order to overcome the multitude of implementation obstacles.

This paper attempts to identify a few central ideas to provide a broad picture for future research and development in physics education, or STEM education in general, to promote twenty-first Century learning. Through a synthesis of the existing literature within the authors’ limited scope, a number of views surface.

Education is a service to prepare (not to select) the future workforce and should be designed as learner-centered, with the education goals and teaching-learning methods tailored to the needs and characteristics of the learners themselves. Given space constraints, the reader is referred to the meta-analysis conducted by Freeman et al. ( 2014 ), which provides strong support for learner-centered instruction. The changing world of the twenty-first Century informs the establishment of new education goals, which should be used to guide research and development of teaching and learning for present day students. Aligned to twenty-first Century learning, the new science standards have set the goals for STEM education to transition towards promoting deep learning of disciplinary knowledge, thereby building upon decades of research in PER, while fostering a wide range of general high-end cognitive and non-cognitive abilities that are transferable across all disciplines.

Following these education goals, more research is needed to operationally define and assess the desired high-end reasoning abilities. Building on a clear definition with effective assessments, a large number of empirical studies are needed to investigate how high-end abilities can be developed in parallel with deep learning of concepts, such that what is learned can be generalized to impact the development of curriculum and teaching methods which promote skills-based learning across all STEM fields. Specifically for PER, future research should emphasize knowledge integration to promote deep conceptual understanding in physics along with inquiry learning to foster scientific reasoning. Integration of physics learning in contexts that connect to other STEM disciplines is also an area for more research. Cross-cutting, interdisciplinary connections are becoming important features of the future generation physics curriculum and defines how physics should be taught collaboratively with other STEM courses.

This paper proposed meaningful areas for future research that are aligned with clearly defined education goals for twenty-first Century learning. Based on the existing literature, a number of challenges are noted for future directions of research, including the need for:

clear and operational definitions of goals to guide research and practice

concrete operational definitions of high-end abilities for which students are expected to develop

effective assessment methods and instruments to measure high-end abilities and other components of twenty-first Century learning

a knowledge base of the curriculum and teaching and learning environments that effectively support the development of advanced skills

integration of knowledge and ability development regarding within-discipline and cross-discipline learning in STEM

effective means to disseminate successful education practices

The list is by no means exhaustive, but these themes emerge above others. In addition, the high-end abilities discussed in this paper focus primarily on scientific reasoning, which is highly connected to other skills, such as critical thinking, systems thinking, multivariable modeling, computational thinking, design thinking, etc. These abilities are expected to develop in STEM learning, although some may be emphasized more within certain disciplines than others. Due to the limited scope of this paper, not all of these abilities were discussed in detail but should be considered an integral part of STEM learning.

Finally, a metacognitive position on education research is worth reflection. One important understanding is that the fundamental learning mechanism hasn’t changed, although the context in which learning occurs has evolved rapidly as a manifestation of the fast-forwarding technology world. Since learning is a process at the interface between a learner’s mind and the environment, the main focus of educators should always be on the learner’s interaction with the environment, not just the environment. In recent education developments, many new learning platforms have emerged at an exponential rate, such as the massive open online courses (MOOCs), STEM creative labs, and other online learning resources, to name a few. As attractive as these may be, it is risky to indiscriminately follow trends in education technology and commercially-incentivized initiatives before such interventions are shown to be effective by research. Trends come and go but educators foster students who have only a limited time to experience education. Therefore, delivering effective education is a high-stakes task and needs to be carefully and ethically planned and implemented. When game-changing opportunities emerge, one needs to not only consider the winners (and what they can win), but also the impact on all that is involved.

Based on a century of education research, consensus has settled on a fundamental mechanism of teaching and learning, which suggests that knowledge is developed within a learner through constructive processes and that team-based guided scientific inquiry is an effective method for promoting deep learning of content knowledge as well as developing high-end cognitive abilities, such as scientific reasoning. Emerging technology and methods should serve to facilitate (not to replace) such learning by providing more effective education settings and conveniently accessible resources. This is an important relationship that should survive many generations of technological and societal changes in the future to come. From a physicist’s point of view, a fundamental relation like this can be considered the “mechanics” of teaching and learning. Therefore, educators and researchers should hold on to these few fundamental principles without being distracted by the surfacing ripples of the world’s motion forward.

Availability of data and materials

Not applicable.

Abbreviations

American Association of Physics Teachers

Investigative Science Learning Environment

Inquiry in Scientific Thinking Analytics and Reasoning

Massive open online course

New Generation Science Standards

  • Physics education research

Science Technology Engineering and Math

Alonso, M. (1992). Problem solving vs. conceptual understanding. American Journal of Physics , 60 (9), 777–778. https://doi.org/10.1119/1.17056 .

Article   Google Scholar  

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., … Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives, abridged edition . White Plains: Longman.

Ates, S., & Cataloglu, E. (2007). The effects of students’ reasoning abilities on conceptual understandings and problem-solving abilities in introductory mechanics. European Journal of Physics , 28 , 1161–1171.

Bagno, E., Eylon, B.-S., & Ganiel, U. (2000). From fragmented knowledge to a knowledge structure: Linking the domains of mechanics and electromagnetism. American Journal of Physics , 68 (S1), S16–S26.

Bailin, S. (1996). Critical thinking. In J. J. Chambliss (Ed.), Philosophy of education: An encyclopedia , (vol. 1671, pp. 119–123). Routledge.

Bangert-Drowns, R. L., & Bankert, E. (1990). Meta-analysis of effects of explicit instruction for critical thinking. Research report. ERIC Number: ED328614.

Google Scholar  

Bao, L., Cai, T., Koenig, K., Fang, K., Han, J., Wang, J., … Wu, N. (2009). Learning and scientific reasoning. Science , 323 , 586–587. https://doi.org/10.1126/science.1167740 .

Bao, L., & Redish, E. F. (2001). Concentration analysis: A quantitative assessment of student states. American Journal of Physics , 69 (S1), S45–S53.

Bao, L., & Redish, E. F. (2006). Model analysis: Representing and assessing the dynamics of student learning. Physical Review Special Topics-Physics Education Research , 2 (1), 010103.

Bao, L., Xiao, Y., Koenig, K., & Han, J. (2018). Validity evaluation of the Lawson classroom test of scientific reasoning. Physical Review Physics Education Research , 14 (2), 020106.

Beichner, R. J., Saul, J. M., Abbott, D. S., Morse, J. J., Deardorff, D., Allain, R. J., … Risley, J. S. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. Research-Based Reform of University Physics , 1 (1), 2–39.

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., & Rumble, M. (2010). Draft White paper defining 21st century skills . Melbourne: ACTS.

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Handbook 1: Cognitive domain . New York: Longman.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn , (vol. 11). Washington, DC: National Academy Press.

Brown, A. (1989). Analogical learning and transfer: What develops? In S. Vosniadu, & A. Ortony (Eds.), Similarity and analogical reasoning , (pp. 369–412). New York: Cambridge U.P.

Chapter   Google Scholar  

Cavallo, A. M. L., Rozman, M., Blickenstaff, J., & Walker, N. (2003). Learning, reasoning, motivation, and epistemological beliefs: Differing approaches in college science courses. Journal of College Science Teaching , 33 (3), 18–22.

Chen, Z., & Klahr, D. (1999). All other things being equal: Acquisition and transfer of the control of variables strategy. Child Development , 70 , 1098–1120.

Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science , 13 (2), 145–182.

Chi, M. T., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science , 5 (2), 121–152.

Chi, M. T., & Slotta, J. D. (1993). The ontological coherence of intuitive physics. Cognition and Instruction , 10 (2–3), 249–260.

Chi, M. T., Slotta, J. D., & De Leeuw, N. (1994). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction , 4 (1), 27–43.

Chi, M. T. H. (1992). Conceptual change within and across ontological categories: Examples from learning and discovery in science. In R. N. Giere (Ed.), Cognitive models of science . Minneapolis: University of Minnesota Press.

Chiu, M. H. (2001). Algorithmic problem solving and conceptual understanding of chemistry by students at a local high school in Taiwan. Proceedings-National Science Council Republic of China Part D Mathematics Science and Technology Education , 11 (1), 20–38.

Chiu, M.-H., Guo, C. J., & Treagust, D. F. (2007). Assessing students’ conceptual understanding in science: An introduction about a national project in Taiwan. International Journal of Science Education , 29 (4), 379–390.

Clement, J. (1982). Students’ preconceptions in introductory mechanics. American Journal of Physics , 50 (1), 66–71.

Coletta, V. P., & Phillips, J. A. (2005). Interpreting FCI scores: Normalized gain, preinstruction scores, and scientific reasoning ability. American Journal of Physics , 73 (12), 1172–1182.

Cracolice, M. S., Deming, J. C., & Ehlert, B. (2008). Concept learning versus problem solving: A cognitive difference. Journal of Chemical Education , 85 (6), 873.

De Jong, T., & Ferguson-Hesler, M. G. M. (1986). Cognitive structure of good and poor problem solvers in physics. Journal of Educational Psychology , 78 , 279–288.

DiSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction , 10 (2–3), 105–225.

Duit, R., & Treagust, D. F. (2003). Conceptual change: A powerful framework for improving science teaching and learning. International Journal of Science Education , 25 (6), 671–688.

Dykstra Jr., D. I., Boyle, C. F., & Monarch, I. A. (1992). Studying conceptual change in learning physics. Science Education , 76 (6), 615–652.

Ennis, R. (1993). Critical thinking assessment. Theory Into Practice , 32 (3), 179–186.

Etkina, E., & Van Heuvelen, A. (2001). Investigative science learning environment: Using the processes of science and cognitive strategies to learn physics. In Proceedings of the 2001 physics education research conference , (pp. 17–21). Rochester.

Etkina, E., Van Heuvelen, A., White-Brahmia, S., Brookes, D. T., Gentile, M., Murthy, S., … Warren, A. (2006). Scientific abilities and their assessment. Physical Review Special Topics-Physics Education Research , 2 (2), 020103.

Eylon, B.-S., & Reif, F. (1984). Effects of knowledge organization on task performance. Cognition and Instruction , 1 (1), 5–44.

Fabby, C., & Koenig, K. (2013). Relationship of scientific reasoning to solving different physics problem types. In Proceedings of the 2013 Physics Education Research Conference, Portland, OR .

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction – The Delphi report . Millbrae: California Academic Press.

Ferguson-Hesler, M. G. M., & De Jong, T. (1990). Studying physics texts: Differences in study processes between good and poor solvers. Cognition and Instruction , 7 (1), 41–54.

Fisher, A. (2001). Critical thinking: An introduction . Cambridge: Cambridge University Press.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences , 111 (23), 8410–8415.

Glaser, E. M. (1941). An experiment in the development of critical thinking . New York: Teachers College, Columbia University.

Hake, R. R. (1992). Socratic pedagogy in the introductory physics laboratory. The Physics Teacher , 30 , 546.

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics , 66 (1), 64–74.

Halloun, I. A., & Hestenes, D. (1985a). The initial knowledge state of college physics students. American Journal of Physics , 53 (11), 1043–1055.

Halloun, I. A., & Hestenes, D. (1985b). Common sense concepts about motion. American Journal of Physics , 53 (11), 1056–1065.

Halpern, D. F. (1999). Teaching for critical thinking: Helping college students develop the skills and dispositions of a critical thinker. New Directions for Teaching and Learning , 80 , 69–74. https://doi.org/10.1002/tl.8005 .

Hardiman, P. T., Dufresne, R., & Mestre, J. P. (1989). The relation between problem categorization and problem solving among experts and novices. Memory & Cognition , 17 (5), 627–638.

Heller, J. I., & Reif, F. (1984). Prescribing effective human problem-solving processes: Problem description in physics. Cognition and Instruction , 1 (2), 177–216.

Hoellwarth, C., Moelter, M. J., & Knight, R. D. (2005). A direct comparison of conceptual learning and problem solving ability in traditional and studio style classrooms. American Journal of Physics , 73 (5), 459–462.

Hofstein, A., & Lunetta, V. N. (1982). The role of the laboratory in science teaching: Neglected aspects of research. Review of Educational Research , 52 (2), 201–217.

Hsu, L., Brewe, E., Foster, T. M., & Harper, K. A. (2004). Resource letter RPS-1: Research in problem solving. American Journal of Physics , 72 (9), 1147–1156.

Jensen, J. L., & Lawson, A. (2011). Effects of collaborative group composition and inquiry instruction on reasoning gains and achievement in undergraduate biology. CBE - Life Sciences Education , 10 , 64–73.

Johnson, M. A., & Lawson, A. E. (1998). What are the relative effects of reasoning ability and prior knowledge on biology achievement in expository and inquiry classes? Journal of Research in Science Teaching , 35 (1), 89–103.

Kalman, C., & Lattery, M. (2018). Three active learning strategies to address mixed student epistemologies and promote conceptual change. Frontiers in ICT , 5 (19), 1–9.

Karplus, R. (1964). The science curriculum improvement study. Journal of College Science Teaching , 2 (4), 293–303.

Kim, E., & Pak, S.-J. (2002). Students do not overcome conceptual difficulties after solving 1000 traditional problems. American Journal of Physics , 70 (7), 759–765.

Koenig, K., Schen, M., & Bao, L. (2012). Explicitly targeting pre-service teacher scientific reasoning abilities and understanding of nature of science through an introductory science course. Science Educator , 21 (2), 1–9.

Koenig, K., Schen, M., Edwards, M., & Bao, L. (2012). Addressing STEM retention through a scientific thought and methods course. Journal of College Science Teaching , 41 , 23–29.

Koenig, K., Wood, K., Bortner, L., & Bao, L. (2019). Modifying traditional labs to target scientific reasoning. Journal of College Science Teaching , 48 (5), 28-35.

Kozminski, J., Beverly, N., Deardorff, D., Dietz, R., Eblen-Zayas, M., Hobbs, R., … Zwickl, B. (2014). AAPT recommendations for the undergraduate physics laboratory curriculum , (pp. 1–29). American Association of Physics Teachers Retrieved from https://www.aapt.org/Resources/upload/LabGuidlinesDocument_EBendorsed_nov10.pdf .

Kuhn, D. (1992). Thinking as argument. Harvard Educational Review , 62 (2), 155–178.

Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science , 208 (4450), 1335–1342.

Laws, P. W. (2004). Workshop physics activity guide, module 4: Electricity and magnetism. In Workshop physics activity guide . Wiley-VCH.

Lawson, A. E. (1978), The development and validation of a classroom test of formal reasoning, Journal of Research in Science Teaching , 15 (1), 11–24.

Lawson, A. E. (1992). The development of reasoning among college biology students - a review of research. Journal of College Science Teaching , 21 , 338–344.

Lawson, A. E. (2000). Classroom test of scientific reasoning: Multiple choice version, based on Lawson, A. E. 1978. Development and validation of the classroom test of formal reasoning. Journal of Research in Science Teaching , 15 (1), 11–24.

Lee, H. S., Liu, O. L., & Linn, M. C. (2011). Validating measurement of knowledge integration in science using multiple-choice and explanation items. Applied Measurement in Education , 24 (2), 115–136.

Linn, M. C. (2005). The knowledge integration perspective on learning and instruction. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences , (pp. 243–264). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511816833.016 .

Lipman, M. (2003). Thinking in education , (2nd ed., ). Cambridge: Cambridge University Press.

Liu, X. (2010). Science and engineering education sources. Using and developing measurement instruments in science education: A Rasch modeling approach . Charlotte: IAP Information Age Publishing.

Marzano, R. J., Brandt, R. S., Hughes, C. S., Jones, B. F., Presseisen, B. Z., Rankin, S. C., et al. (1988). Dimensions of thinking, a framework for curriculum and instruction . Alexandria: Association for Supervision and Curriculum Development.

Mazur, E. (1997). Peer instruction: A user’s manual . Upper Saddle River: Prentice Hall.

McDermott, L. C. (1996). Physics by Inquiry: An Introduction to the Physical Sciences . John Wiley & Sons, New York, NY.

McDermott, L. C., & Redish, E. F. (1999). Resource letter: PER-1: Physics education research. American Journal of Physics , 67 (9), 755–767.

McKeachie, W. J. (1986). Teaching and learning in the college classroom: A review of the research literature . Ann Arbor: National Center for Research to Improve Postsecondary Teaching and Learning.

Meltzer, D. E., & Otero, V. K. (2015). A brief history of physics education in the United States. American Journal of Physics , 83 (5), 447–458.

Meltzer, D. E., & Thornton, R. K. (2012). Resource letter ALIP-1: Active-learning instruction in physics. American Journal of Physics , 80 (6), 478–496.

Minstrell, J. (1992). Facets of students’ knowledge and relevant instruction. In R. Duit, F. Goldberg, & H. Niedderer (Eds.), Proceedings of the international workshop: Research in physics learning- theoretical issues and empirical studies , (pp. 110–128). The Institute for Science Education.

Nakhleh, M. B. (1993). Are our students conceptual thinkers or algorithmic problem solvers? Identifying conceptual students in general chemistry. Journal of Chemical Education , 70 (1), 52. https://doi.org/10.1021/ed070p52 .

Nakhleh, M. B., & Mitchell, R. C. (1993). Concept learning versus problem solving: There is a difference. Journal of Chemical Education , 70 (3), 190. https://doi.org/10.1021/ed070p190 .

National Research Council (2011). Assessing 21st century skills: Summary of a workshop . Washington, DC: The National Academies Press. https://doi.org/10.17226/13215 .

Book   Google Scholar  

National Research Council (2012a). Education for life and work: Developing transferable knowledge and skills in the 21st century . Washington, DC: The National Academies Press.

National Research Council (2012b). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas . Washington, DC: National Academies Press.

National Research Council (2012c). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering . Washington, DC: National Academies Press.

National Science & Technology Council (2018). Charting a course for success: America’s strategy for STEM education . Washington, DC: Office of Science and Technology Policy.

NCET. (1987). Critical thinking as defined by the National Council for excellence in critical thinking, statement by Michael Scriven & Richard Paul, presented at the 8th annual conference on critical thinking and education reform. Retrieved December 4, 2018, from http://www.criticalthinking.org/pages/defining-critical-thinking/766 .

Nordine, J., Krajcik, J., & Fortus, D. (2011). Transforming energy instruction in middle school to support integrated understanding and future learning. Science Education , 95 (4), 670–699.

Nurrenbern, S. C., & Pickering, M. (1987). Concept learning versus problem solving: Is there a difference? Journal of Chemical Education , 64 (6), 508.

Paul, R. (1990). Critical thinking: What every person needs to survive in a rapidly changing world . Rohnert Park: Center for Critical Thinking and Moral Critique.

Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context-bound? Educational Researcher , 18 (1), 16–25.

Posner, G., Strike, K., Hewson, P., & Gertzog, W. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education , 66 (2), 211–227.

Reay, N. W., Bao, L., Li, P., Warnakulasooriya, R., & Baugh, G. (2005). Toward the effective use of voting machines in physics lectures. American Journal of Physics , 73 (6), 554–558.

Reimers, F. M., & Chung, C. K. (Eds.) (2016). Teaching and learning for the twenty-first century: Educational goals, policies and curricula from six nations . Cambridge: Harvard Education Press.

Salomon, G., & Perkins, D. N. (1989). Rocky roads to transfer: Rethinking mechanism of a neglected phenomenon. Educational Psychologist , 24 (2), 113–142.

Schoenfeld, A. H., & Herrmann, D. J. (1982). Problem perception and knowledge structure in expert and novice mathematical problem solvers. Journal of Experimental Psychology: Learning, Memory, and Cognition , 8 (5), 484.

Shaw, V. F. (1996). The cognitive processes in informal reasoning. Thinking and Reasoning , 2 (1), 51–80.

She, H., & Liao, Y. (2010). Bridging scientific reasoning and conceptual change through adaptive web-based learning. Journal of Research in Science Teaching , 47 (1), 91–119.

Shen, J., Liu, O. L., & Chang, H.-Y. (2017). Assessing students’ deep conceptual understanding in physical sciences: An example on sinking and floating. International Journal of Science and Mathematics Education , 15 (1), 57–70. https://doi.org/10.1007/s10763-015-9680-z .

Siegel, H. (1988). Educating reason: Rationality, critical thinking and education , (vol. 1). New York: Routledge.

Slotta, J. D., Chi, M. T., & Joram, E. (1995). Assessing students’ misclassifications of physics concepts: An ontological basis for conceptual change. Cognition and Instruction , 13 (3), 373–400.

Smith III, J. P., DiSessa, A. A., & Roschelle, J. (1994). Misconceptions reconceived: A constructivist analysis of knowledge in transition. The Journal of the Learning Sciences , 3 (2), 115–163.

Smith, M. U. (1992). Expertise and organization of knowledge: Unexpected differences among genetic counselors, faculty members and students on problem categorization tasks. Journal of Research in Science Teaching , 29 (2), 179–205.

Sobhanzadeh, M., Kalman, C. S., & Thompson, R. I. (2017). Labatorials in introductory physics courses. European Journal of Physics , 38 , 1–18.

Sokoloff, D. R., Thornton, R. K., & Laws, P. W. (2011). RealTime physics: Active learning laboratories . New York: Wiley.

Stamovlasis, D., Tsaparlis, G., Kamilatos, C., Papaoikonomou, D., & Zarotiadou, E. (2005). Conceptual understanding versus algorithmic problem solving: Further evidence from a national chemistry examination. Chemistry Education Research and Practice , 6 (2), 104–118.

Tanenbaum, C. (2016). STEM 2026: A vision for innovation in STEM education . Washington, DC: US Department of Education.

Thornton, R. K., & Sokoloff, D. R. (1998). Assessing student learning of Newton’s laws: The force and motion conceptual evaluation and the evaluation of active learning laboratory and lecture curricula. American Journal of Physics , 66 (4), 338–352.

Trowbridge, L. W., Bybee, R. W., & Powell, J. C. (2000). Teaching secondary school science: Strategies for developing scientific literacy . Upper Saddle River: Merrill-Prentice Hall.

United States Chamber of Commerce (2017). Bridging the soft skills gap: How the business and education sectors are partnering to prepare students for the 21 st century workforce . Washington DC: Center for Education and Workforce, U.S. Chamber of Commerce Foundation.

Veldhuis, G. H. (1990). The use of cluster analysis in categorization of physics problems. Science Education , 74 (1), 105–118.

Vosniadou, S., Vamvakoussi, X., & Skopeliti, I. (2008). The framework theory approach to the problem of conceptual change. In S. Vosniadou (Ed.), International handbook of research on conceptual change . New York: Routledge.

Wexler, P. (1982). Structure, text, and subject: A critical sociology of school knowledge. In M. W. Apple (Ed.), Cultural and economic reproduction in education: Essays on class, ideology and the state . London: Routledge & Regan Paul.

Zeineddin, A., & Abd-El-Khalick, F. (2010). Scientific reasoning and epistemological commitments: Coordination of theory and evidence among college science students. Journal of Research in Science Teaching , 47 (9), 1064–1093.

Zimmerman, C. (2000). The development of scientific reasoning skills. Developmental Review , 20 (1), 99–149.

Download references

Acknowledgements

The research is supported in part by NSF Awards DUE-1431908 and DUE-1712238. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The research is supported in part by NSF Awards DUE-1431908 and DUE-1712238.

Author information

Authors and affiliations.

The Ohio State University, Columbus, OH, 43210, USA

University of Cincinnati, Cincinnati, OH, 45221, USA

Kathleen Koenig

You can also search for this author in PubMed   Google Scholar

Contributions

LB developed the concept, wrote a significant portion of the review and position, and synthesized the paper. KK wrote and edited a significant portion of the paper. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Lei Bao .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Bao, L., Koenig, K. Physics education research for 21 st century learning. Discip Interdscip Sci Educ Res 1 , 2 (2019). https://doi.org/10.1186/s43031-019-0007-8

Download citation

Received : 17 April 2019

Accepted : 13 June 2019

Published : 28 November 2019

DOI : https://doi.org/10.1186/s43031-019-0007-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Twenty-first century learning
  • STEM education
  • Scientific reasoning
  • Deep learning

research papers on 21st century

Research methodology in the 21st Century

Ieee account.

  • Change Username/Password
  • Update Address

Purchase Details

  • Payment Options
  • Order History
  • View Purchased Documents

Profile Information

  • Communications Preferences
  • Profession and Education
  • Technical Interests
  • US & Canada: +1 800 678 4333
  • Worldwide: +1 732 981 0060
  • Contact & Support
  • About IEEE Xplore
  • Accessibility
  • Terms of Use
  • Nondiscrimination Policy
  • Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Environ Eng Sci

Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition

Over the last 50 years, we argue that incentives for academic scientists have become increasingly perverse in terms of competition for research funding, development of quantitative metrics to measure performance, and a changing business model for higher education itself. Furthermore, decreased discretionary funding at the federal and state level is creating a hypercompetitive environment between government agencies (e.g., EPA, NIH, CDC), for scientists in these agencies, and for academics seeking funding from all sources—the combination of perverse incentives and decreased funding increases pressures that can lead to unethical behavior. If a critical mass of scientists become untrustworthy, a tipping point is possible in which the scientific enterprise itself becomes inherently corrupt and public trust is lost, risking a new dark age with devastating consequences to humanity. Academia and federal agencies should better support science as a public good, and incentivize altruistic and ethical outcomes, while de-emphasizing output.

Introduction

T he incentives and reward structure of academia have undergone a dramatic change in the last half century. Competition has increased for tenure-track positions, and most U.S. PhD graduates are selecting careers in industry, government, or elsewhere partly because the current supply of PhDs far exceeds available academic positions (Cyranoski et al. , 2011 ; Stephan, 2012a ; Aitkenhead, 2013 ; Ladner et al. , 2013 ; Dzeng, 2014 ; Kolata, 2016 ). Universities are also increasingly “balance<ing> their budgets on the backs of adjuncts” given that part-time or adjunct professor jobs make up 76% of the academic labor force, while getting paid on average $2,700 per class, without benefits or job security (Curtis and Thornton, 2013 ; U.S. House Committee on Education and the Workforce, 2014 ). There are other concerns about the culture of modern academia, as reflected by studies showing that the attractiveness of academic research careers decreases over the course of students' PhD program at Tier-1 institutions relative to other careers (Sauermann and Roach, 2012 ; Schneider et al. , 2014 ), reflecting the overemphasis on quantitative metrics, competition for limited funding, and difficulties pursuing science as a public good.

In this article, we will (1) describe how perverse incentives and hypercompetition are altering academic behavior of researchers and universities, reducing scientific progress and increasing unethical actions, (2) propose a conceptual model that describes how emphasis on quantity versus quality can adversely affect true scientific progress, (3) consider ramifications of this environment on the next generation of Science, Technology, Engineering and Mathematics (STEM) researchers, public perception, and the future of science itself, and finally, (4) offer recommendations that could help our scientific institutions increase productivity and maintain public trust. We hope to begin a conversation among all stakeholders who acknowledge perverse incentives throughout academia, consider changes to increase scientific progress, and uphold “high ethical standards” in the profession (NAE, 2004 ).

Perverse Incentives in Research Academia: The New Normal?

When you rely on incentives, you undermine virtues. Then when you discover that you actually need people who want to do the right thing, those people don't exist… —Barry Schwartz, Swarthmore College (Zetter, 2009 )

Academics are human and readily respond to incentives. The need to achieve tenure has influenced faculty decisions, priorities, and activities since the concept first became popular (Wolverton, 1998 ). Recently, however, an emphasis on quantitative performance metrics (Van Noorden, 2010 ), increased competition for static or reduced federal research funding (e.g., NIH, NSF, and EPA), and a steady shift toward operating public universities on a private business model (Plerou, et al. , 1999 ; Brownlee, 2014 ; Kasperkevic, 2014 ) are creating an increasingly perverse academic culture. These changes may be creating problems in academia at both individual and institutional levels ( Table 1 ).

Growing Perverse Incentives in Academia

Modified from Regehr (pers. comm., 2015) with permission.

Quantitative performance metrics: effect on individual researchers and productivity

The goal of measuring scientific productivity has given rise to quantitative performance metrics, including publication count, citations, combined citation-publication counts (e.g., h-index), journal impact factors (JIF), total research dollars, and total patents. These quantitative metrics now dominate decision-making in faculty hiring, promotion and tenure, awards, and funding (Abbott et al. , 2010 ; Carpenter et al. , 2014 ). Because these measures are subject to manipulation, they are doomed to become misleading and even counterproductive, according to Goodhart's Law , which states that “ when a measure becomes a target, it ceases to be a good measure ” (Elton, 2004 ; Fischer et al. , 2012 ; Werner, 2015 ).

Ultimately, the well-intentioned use of quantitative metrics may create inequities and outcomes worse than the systems they replaced. Specifically, if rewards are disproportionally given to individuals manipulating their metrics, problems of the old subjective paradigms (e.g., old-boys' networks) may be tame by comparison. In a 2010 survey, 71% of respondents stated that they feared colleagues can “game” or “cheat” their way into better evaluations at their institutions (Abbott, 2010 ), demonstrating that scientists are acutely attuned to the possibility of abuses in the current system.

Quantitative metrics are scholar centric and reward output, which is not necessarily the same as achieving a goal of socially relevant and impactful research outcomes. Scientific output as measured by cited work has doubled every 9 years since about World War II (Bornmann and Mutz, 2015 ), producing “busier academics, shorter and less comprehensive papers” (Fischer et al. , 2012 ), and a change in climate from “publish or perish” to “funding or famine” (Quake, 2009 ; Tijdink et al. , 2014 ). Questions have been raised about how sustainable this exponential increase in the knowledge industry is (Price, 1963 ; Frodeman, 2011 ) and how much of the growth is illusory and results from manipulation as per Goodhart's Law .

Recent exposés have revealed schemes by journals to manipulate impact factors, use of p-hacking by researchers to mine for statistically significant and publishable results, rigging of the peer-review process itself, and overcitation (Falagas and Alexiou, 2008 ; Labbé, 2010 ; Zhivotovsky and Krutovsky, 2008 ; Bartneck and Kokkelmans, 2011 ; Delgado López-Cózar et al. , 2012 ; McDermott, 2013 ; Van Noorden, 2014 ; Barry, 2015 ). A fictional character was recently created to demonstrate a “spamming war in the heart of science,” by generation of 102 fake articles and a stellar h-index of 94 on Google Scholar (Labbé, 2010 ). Blogs describing how to more discretely raise h-index without committing outright fraud are also commonplace (e.g., Dem, 2011 ).

It is instructive to conceptualize the basic problem from a perspective of emphasizing quality-in-research versus quantity-in-research, as well as effects of perverse incentives ( Fig. 1 ). Assuming that the goal of the scientific enterprise is to maximize true scientific progress, a process that overemphasizes quality might require triple or quadruple blinded studies, mandatory replication of results by independent parties, and peer-review of all data and statistics before publication—such a system would minimize mistakes, but would produce very few results due to overcaution (left Fig. 1 ). At the other extreme, an overemphasis on quantity is also problematic because accepting less scientific rigor in statistics, replication, and quality controls or a less rigorous review process would produce a very high number of articles, but after considering costly setbacks associated with a high error rate, true progress would also be low. A hypothetical optimum productivity lies somewhere in between, and it is possible that our current practices (enforced by peer review) evolved to be near the optimum in an environment with fewer perverse incentives.

An external file that holds a picture, illustration, etc.
Object name is fig-1.jpg

True scientific productivity vis-à-vis emphasis on research quality/quantity.

However, over the long term under a system of perverse incentives, the true productivity curve changes due to increased manipulation and/or unethical behavior ( Fig. 1 ). In a system overemphasizing quality, there is less incentive to cut corners because checks and balances allow problems to be discovered more easily, but in a system emphasizing quantity, productivity can be dramatically reduced by massive numbers of erroneous articles created by carelessness, subtle falsification (i.e., eliminating bad data), and substandard review if not outright fabrication (i.e., dry labbing).

While there is virtually no research exploring the impact of perverse incentives on scientific productivity, most in academia would acknowledge a collective shift in our behavior over the years ( Table 1 ), emphasizing quantity at the expense of quality. This issue may be especially troubling for attracting and retaining altruistically minded students, particularly women and underrepresented minorities (WURM), in STEM research careers. Because modern scientific careers are perceived as focusing on “the individual scientist and individual achievement” rather than altruistic goals (Thoman et al. , 2014 ), and WURM students tend to be attracted toward STEM fields for altruistic motives, including serving society and one's community (Diekman et al. , 2010 , Thoman et al. , 2014 ), many leave STEM to seek careers and work that is more in keeping with their values (e.g., Diekman et al. , 2010 ; Gibbs and Griffin, 2013 ; Campbell, et al. , 2014 ).

Thus, another danger of overemphasizing output versus outcomes and quantity versus quality is creating a system that is a “perversion of natural selection,” which selectively weeds out ethical and altruistic actors, while selecting for academics who are more comfortable and responsive to perverse incentives from the point of entry. Likewise, if normally ethical actors feel a need to engage in unethical behavior to maintain academic careers (Edwards, 2014 ), they may become complicit as per Granovetter's well-established Threshold Model of Collective Behavior (1978). At that point, unethical actions have become “embedded in the structures and processes” of a professional culture, and nearly everyone has been “induced to view corruption as permissible” (Ashforth and Anand, 2003 ).

It is also telling that a new genre of articles termed “quit lit” by the Chronicle of Higher Education has emerged (Chronicle Vitae, 2013–2014 ), in which successful, altruistic, and public-minded professors give perfectly rational reasons for leaving a profession they once loved—such individuals are easily replaced with new hires who are more comfortable with the current climate. Reasons for leaving range from a saturated job market, lack of autonomy, concerns associated with the very structure of academe (CHE, 2013 ), and “a perverse incentive structure that maintains the status quo, rewards mediocrity, and discourages potential high-impact interdisciplinary work” (Dunn, 2013 ).

While quantitative metrics provide an objective means of evaluating research productivity relative to subjective measures, now that they have become a target, they cease to be useful and may even be counterproductive. A continued overemphasis on quantitative metrics will pressure all but the most ethical scientists, to overemphasize quantity at the expense of quality, create pressures to “cut corners” throughout the system, and select for scientists attracted to perverse incentives.

Scientific societies, research institutions, academic journals and individuals have made similar arguments, and some have signed the San Francisco Declaration of Research Assessment (DORA). The DORA recognizes the need for improving “ways in which output of scientific research are evaluated” and calls for challenging research assessment practices, especially the JIF, which are currently in place. Signatories include the American Society for Cell Biology, American Association for the Advancement of Science, Howard Hughes Medical Institute, and Proceedings of The National Academy of Sciences, among 737 organizations and 12,229 individuals as of June 30, 2016. Indeed, publishers of Nature , Science , and other journals have called for downplaying the JIF metric, and the American Society of Microbiology is announcing plans to “purge the conversation of the impact factor” and remove them from all their journals (Callaway, 2016 ). The argument is not to get rid of metrics, but to reduce their importance in decision-making by institutions and funding agencies, and perhaps invest resources toward creating more meaningful metrics (ACSB, 2012 ). DORA would be a step in the right direction of halting the “avalanche” of performance metrics dominating research assessment, which are unreliable and have long been hypothesized to threaten the quality of research (Rice, 1994 ; Macilwain, 2013 ).

Performance metrics: effect on institutions

We had to get into the top 100. That was a life-or-death matter for Northeastern.— Richard Freeland, Former President of Northeastern University (Kutner, 2014 )

The perverse incentives for academic institutions are growing in scope and impact, as best exemplified by U.S. News & World Report annual rankings that purportedly measure “academic excellence” (Morse, 2015 ). The rankings have strongly influenced, positively or negatively, public perceptions regarding the quality of education and opportunities they offer (Casper, 1996 ; Gladwell, 2011 ; Tierney, 2013 ). Although U.S. News & World Report rankings have been dismissed by some, they still undeniably wield extraordinary influence on college administrators and university leadership—the perceptions created by the objective quantitative ranking determines “how students, parents, high schools, and colleges pursue and perceive education” in practice (Kutner, 2014 ; Segal, 2014 ).

The rankings rely on subjective proprietary formula and algorithms, the original validity of which has since been undermined by Goodhart's law —universities have attempted to game the system by redistributing resources or investing in areas that the ranking metrics emphasize. Northeastern University, for instance, unapologetically rose from #162 in 1996 to #42 in 2015 by explicitly changing their class sizes, acceptance rates, and even peer assessment. Others have cheated by reporting incorrect statistics (Bucknell University, Claremont-McKenna College, Clemson University, George Washington University, and Emory University are examples of those who were caught) to rise in the ranks (Slotnik and Perez-Pena, 2012 ; Anderson, 2013 ; Kutner, 2014 ). More than 90% of 576 college admission officers thought other institutions were submitting false data to U.S. News according to a 2013 Gallup and Inside Higher Ed poll (Jaschik, 2013 ), which creates further pressures to cheat throughout the system to maintain a ranking perceived to be fair as discussed in preceding sections.

Hypercompetitive funding environments

If the work you propose to do isn't virtually certain of success, then it won't get funded— Roger Kornberg, Nobel laureate (Lee, 2007 )

The only people who can survive in this environment are people who are absolutely passionate about what they're doing and have the self-confidence and competitiveness to just go back again and again and just persistently apply for funding— Robert Waterland, Baylor College of Medicine (Harris and Benincasa, 2014 )

The federal government's role in financing research and development (R&D), creating new knowledge, or fulfilling public missions like national security, agriculture, infrastructure, and environmental health has become paramount. The cost of high-risk, long-term research, which often has uncertain prospects and/or utility, has been largely borne by the U.S. government in the aftermath of World War II, forming part of an ecosystem with universities and industries contributing to the collective progress of mankind (Bornmann and Mutz, 2015 ; Hourihan, 2015 ).

However, in the current competitive global environment where China is projected to outspend the U.S. on R&D by 2020, some worry that the “edifice of American innovation rests on an increasingly rickety foundation” because of a decline in spending on federal R&D in the past decade (Casassus, 2014 ; OECD, 2014 ; MIT, 2015 ; Porter, 2015 ). U.S. “Research Intensity” (i.e., federal R&D as a share of the country's gross domestic product or GDP) has declined to 0.78% (2014), which is down from about 2% in the 1960 s ( Fig. 2 ). With discretionary spending of federal budgets projected to decrease, research intensity is likely to drop even further, despite increased industry funding (Hourihan, 2015 ).

An external file that holds a picture, illustration, etc.
Object name is fig-2.jpg

Trends in research intensity (i.e., ratio of U.S. R&D to gross domestic product), roles of federal, business, and other nonfederal funding for R&D: 1953–2013. Data source: National Science Foundation, National Center for Science and Engineering Statistics, National Patterns of R&D Resources (annual series). R&D, research and development.

A core mission of American colleges and universities has been “service to the public,” and this goal will be more difficult to reach as universities morph into profit centers churning out patents and new products (Faust, 2009 ; Mirowski, 2011 ; Brownlee, 2014 ; Hinkes-Jones, 2014 ; Seligsohn, 2015 ; American Academy of Arts and Sciences, 2016 ). Until the late 2000s, research institutions and universities went on a building spree fueled by borrowing, with an expectation that increased research funding would allow them to further boost research productivity—a cycle that went bust after the 2007–2008 financial crash (Stephan, 2012a ). Universities are also allowed to offset debt from ill-fated expansion efforts as indirect costs (Stephan, 2012b ), which increases overhead and decreases dollars available to spend on research even if funds raised by grants remain constant.

The static or declining federal investment in research has created the “worst research funding <scenario in 50 years>” and further ratcheted competition for funding (Lee, 2007 ; Quake, 2009 ; Harris and Benincasa, 2014 ; Schneider et al. , 2014 ; Stein, 2015 ), given that the number of researchers competing for grants is rising. The funding rate for NIH grants fell from 30.5% to 18% between 1997 and 2014, and the average age for a first time PI on an R01-equivalent grant has increased to 43 years (NIH, 2008 , 2015 ). NSF funding rates have remained stagnant between 23 and 25% in the past decade (NSF, 2016 ). While these funding rates are still well above the breakeven point of 6%, at which the net cost of proposal writing equals the net value obtained from a grant by the grant winner (Cushman et al. , 2015 ), there is little doubt the grant environment is hypercompetitive, susceptible to reviewer biases, and strongly dependent on prior success as measured by quantitative metrics (Lawrence, 2009 ; Fang and Casadevall, 2016 ). Researchers must tailor their thinking to align with solicited funding, and spend about half of their time addressing administrative and compliance, drawing focus away from scientific discovery and translation (NSB, 2014 ; Schneider et al. , 2014 ; Belluz et al. , 2016 ).

Systemic Risks to Scientific Integrity

Science is a human endeavor, and despite its obvious historical contributions to advancement of civilization, there is growing evidence that today's research publications too frequently suffer from lack of replicability, rely on biased data-sets, apply low or substandard statistical methods, fail to guard against researcher biases, and their findings are overhyped (Fanelli, 2009 ; Aschwanden, 2015 ; Belluz and Hoffman, 2015 ; Nuzzo, 2015 ; Gobry, 2016 ; Wilson, 2016 ). A troubling level of unethical activity, outright faking of peer review and retractions, has been revealed, which likely represents just a small portion of the total, given the high cost of exposing, disclosing, or acknowledging scientific misconduct (Marcus and Oransky, 2015 ; Retraction Watch, 2015a ; BBC, 2016 ; Borman, 2016 ). Warnings of systemic problems go back to at least 1991, when NSF Director Walter E. Massey noted that the size, complexity, and increased interdisciplinary nature of research in the face of growing competition was making science and engineering “more vulnerable to falsehoods” (The New York Times, 1991 ).

Misconduct is not limited to academic researchers. Federal agencies are also subject to perverse incentives and hypercompetition, giving rise to a new phenomenon of institutional scientific research misconduct (Lewis, 2014 ; Edwards, 2016 ). Recent exemplars uncovered by the first author in the Flint and Washington D.C. drinking water crises include “scientifically indefensible” reports by the U.S. Centers for Disease Control and Prevention (U.S. Centers for Disease Control and Prevention, 2004 ; U.S. House Committee on Science and Technology, 2010 ), reports based on nonexistent data published by the U.S. EPA and their consultants in industry journals (Reiber and Dufresne, 2006 ; Boyd et al. , 2012 ; Edwards, 2012 ; Retraction Watch, 2015b ; U.S. Congress House Committee on Oversight and Government Reform, 2016 ), and silencing of whistleblowers in EPA (Coleman-Adebayo, 2011 ; Lewis, 2014 ; U.S. Congress House Committee on Oversight and Government Reform, 2015 ). This problem is likely to increase as agencies increasingly compete with each other for reduced discretionary funding. It also raises legitimate and disturbing questions as to whether accepting research funding from federal agencies is inherently ethical or not—modern agencies clearly have conflicts similar to those that are accepted and well understood for industry research sponsors. Given the mistaken presumption of research neutrality by federal funding agencies (Oreskes and Conway, 2010 ), the dangers of institutional research misconduct to society may outweigh those of industry-sponsored research (Edwards, 2014 ).

A “trampling of the scientific ethos” witnessed in areas as diverse as climate science and galvanic corrosion undermines the “credibility of everyone in science” (Bedeian et al. , 2010 ; Oreskes and Conway, 2010 ; Edwards, 2012 ; Leiserowitz et al. , 2012 ; The Economist, 2013 ; BBC, 2016 ). The Economist recently highlighted the prevalence of shoddy and nonreproducible modern scientific research and its high financial cost to society—posing an open question as to whether modern science was trustworthy, while calling upon science to reform itself (The Economist, 2013 ). And, while there are hopes that some problems could be reduced by practices that include open data, open access, postpublication peer review, metastudies, and efforts to reproduce landmark studies, these can only partly compensate for the high error rates in modern science arising from individual and institutional perverse incentives ( Fig. 1 ).

The high costs of research misconduct

The National Science Foundation defines research misconduct as intentional “fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results” (Steneck, 2007 ; Fischer, 2011 ). Nationally, the percentage of guilty respondents in research misconduct cases investigated by the Department of Health and Human Services (includes NIH) and NSF ranges from 20% to 33% (U.S. Department of Health and Human Services, 2013 ; Kroll, 2015, pers. comm.). Direct costs of handling each research misconduct case are $525,000, and over $110 million are incurred annually for all such cases at the institutional level in the U.S (Michalek, et al. , 2010 ). A total of 291 articles retracted due to misconduct during 1992–2012 accounted for $58 M in direct funding from the NIH, which is less than 1% of the agency's budget during this period, but each retracted article accounted for about $400,000 in direct costs (Stern et al. , 2014 ).

Obviously, incidence of undetected misconduct is some multiple of the cases judged as such each year, and the true incidence is difficult to predict. A comprehensive meta-analysis of research misconduct surveys between 1987 and 2008 indicated that 1 in 50 scientists admitted to committing misconduct (fabrication, falsification, and/or modifying data) at least once and 14% knew of colleagues who had done so (Fanelli, 2009 ). These numbers are likely an underestimate considering the sensitivity of the questions asked, low response rates, and the Muhammad Ali effect (a self-serving bias where people perceive themselves as more honest than their peers) (Allison et al. , 1989 ). Indeed, delving deeper, 34% of researchers self-reported that they have engaged in “questionable research practices,” including “dropping data points on a gut feeling” and “changing the design, methodology, and results of a study in response to pressures from a funding source,” whereas 72% of those surveyed knew of colleagues who had done so (Fanelli, 2009 ). One study included in Fanelli's meta-analysis looked at rates of exposure to misconduct for 2,000 doctoral students and 2,000 faculty from the 99 largest graduate departments of chemistry, civil engineering, microbiology, and sociology, and found between 6 and 8% of both students and faculty had direct knowledge of faculty falsifying data (Swazey et al. , 1993 ).

In life science and biomedical research, the percentage of scientific articles retracted has increased 10-fold since 1975, and 67% were due to misconduct (Fang et al. , 2012 ). Various hypotheses are proposed for this increase, including “lure of the luxury journal,” “pathological publishing,” prevalent misconduct policies, academic culture, career stage, and perverse incentives (Martinson et al. , 2009 ; Harding et al. , 2012 ; Laduke, 2013 ; Schekman, 2013 ; Buela-Casal, 2014 ; Fanelli et al. , 2015 ; Marcus and Oransky, 2015 ; Sarewitz, 2016 ). Nature recently declared that “pretending research misconduct does not happen is no longer an option” (Nature, 2015 ).

Academia and science are expected to be self-policing and self-correcting. However, based on our experiences, we believe there are incentives throughout the system that induce all stakeholders to “pretend misconduct does not happen.” Science has never developed a clear system for reporting, investigating, or dealing with allegations of research misconduct, and those individuals who do attempt to police behavior are likely to be frustrated and suffer severe negative professional repercussions (Macilwain, 1997 ; Kevles, 2000 ; Denworth, 2008 ). Academics largely operate on an unenforceable and unwritten honor system, in relation to what is considered fair in reporting research, grant writing practices, and “selling” research ideas, and there is serious doubt as to whether science as a whole can actually be considered self-correcting (Stroebe et al. , 2012 ). While there are exceptional cases where individuals have provided a reality check on overhyped research press releases in areas deemed potentially transformative (e.g., Eisen, 2010–2015 ; New Scientist, 2016 ), limitations of hot research sectors are more often downplayed or ignored. Because every modern scientific mania also creates a quantitative metric windfall for participants and there are few consequences for those responsible after a science bubble finally pops, the only true check on pathological science and a misallocation of resources is the unwritten honor system (Langmuir et al. , 1953 ).

If nothing is done, we will create a corrupt academic culture

The modern academic research enterprise, dubbed a “Ponzi Scheme” by The Economist , created the existing perverse incentive system, which would have been almost inconceivable to academics of 30–50 years ago (The Economist, 2010 ). We believe that this creation is a threat to the future of science, and unless immediate action is taken, we run the risk of “normalization of corruption” (Ashforth and Anand, 2003 ), creating a corrupt professional culture akin to that recently revealed in professional cycling or in the Atlanta school cheating scandal.

To review, for the 7 years Lance Armstrong won the Tour de France (1999–2005), 20 out of 21 podium finishers (including Armstrong) were directly tied to doping through admissions, sanctions, public investigations, or failing blood tests. Entire teams cheated together because of a “win-at-all cost culture” that was created and sustained over time because there was no alternative in sight (U.S. ADA, 2012 ; Rose and Fisher, 2013 ; Saraceno, 2013 ). Numerous warning signs were ignored, and a retrospective analysis indicates that more than half of all Tour de France winners since 1980 had either been tested positive for or confessed to doping (Mulvey, 2012 ). The resultant “culture of doping” put clean athletes under suspicion (CIRC, 2015 ; Dimeo, 2015 ) and ultimately brought worldwide disrepute to the sport.

Likewise, the Atlanta Public Schools (APS) scandal provides another example of a perverse incentive system run to its logical conclusion, but in an educational setting. Twelve former APS employees were sent to prison and dozens faced ethics sanctions for falsifying students' results on state-standardized tests. The well-intentioned quantitative test results became high stakes to the APS employees, because the law “trigger[s] serious consequences for students (like grade promotion and graduation); schools (extra resources, reorganization, or closure); districts (potential loss of federal funds), and school employees (bonuses, demotion, poor evaluations, or firing)” (Kamenetz, 2015 ). The APS employees betrayed their stated mission of creating a “caring culture of trust and collaboration [where] every student will graduate ready for college and career,” and participated in creating the illusion of a “high-performing school district” (APS, 2016 ). Clearly, perverse incentives can encourage unethical behavior to manipulate quantitative metrics, even in an institution where the sole goal was to educate children.

An uncontrolled perverse incentive system can create a climate in which participants feel they must cheat to compete, whether it is academia (individual or institutional level) or professional sports. While procycling was ultimately discredited and its rewards were not properly distributed to ethical participants, in science, the loss of altruistic actors and trust, and risk of direct harm to the public and the planet raise the dangers immeasurably.

What Kind of Profession Are We Creating for the Next Generation of Academics?

So I have just one wish for you—the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a need to maintain your position in the organization, or financial support, or so on, to lose your integrity. May you have that freedom— Richard Feynman, Nobel laureate (Feynman, 1974 )

The culture of academia has undergone dramatic change in the last few decades—quite a bit of it has been for the better. Problems with diversity, work-life balance, funding, efficient teaching, public outreach, and engagement have been recognized and partly addressed.

As stewards of the profession, we should continually consider whether our collective actions will leave our field in a state that is better or worse than when we entered it. While factors such as state and federal funding levels are largely beyond our control, we are not powerless and passive actors. Problems with perverse incentives and hypercompetition could be addressed by the following:

  • (1) The scope of the problem must be better understood, by systematically mining the experiences and perceptions held by academics in STEM fields, through a comprehensive survey of high-achieving graduate students and researchers.
  • (2) The National Science Foundation should commission a panel of economists and social scientists with expertise in perverse incentives, to collect and review input from all levels of academia, including retired National Academy members and distinguished STEM scholars. The panel could also develop a list of “best practices” to guide evaluation of candidates for hiring and promotion, from a long-term perspective of promoting science in the public interest and for the public good, and maintain academia as a desirable career path for altruistic ethical actors.
  • (3) Rather than pretending that the problem of research misconduct does not exist, science and engineering students should receive instruction on these subjects at both the undergraduate and graduate levels. Instruction should include a review of real world pressures, incentives, and stresses that can increase the likelihood of research misconduct.
  • (4) Beyond conventional goals of achieving quantitative metrics, a PhD program should also be viewed as an exercise in building character, with some emphasis on the ideal of practicing science as service to humanity (Huber, 2014 ).
  • (5) Universities need to reduce perverse incentives and uphold research misconduct policies that discourage unethical behavior.

Acknowledgments

The authors wish to thank PhD Candidate William Rhoads from Virginia Tech and three anonymous reviewers from Environmental Engineering Science for their assistance with the article and valuable suggestions.

Author Disclosure Statement

No competing financial interests exist.

  • Abbott A., Cyranoski D., Jones N., Maher B., Schiermeier Q., and Van Noorden R. (2010). Metrics: Do metrics matter? Nature . 465 , 860. [ PubMed ] [ Google Scholar ]
  • Aitkenhead D. (2013, Dec. 6). Peter Higgs: I wouldn't be productive enough for today's academic system . The Guardian . Available at: www.theguardian.com/science/2013/dec/06/peter-higgs-boson-academic-system (accessed September16, 2016)
  • Allison S.T., Messick D.M., and Goethals G.R. (1989). On being better but not smarter than others: The Muhammad Ali effect . Soc. Cogn. 7 , 275 [ Google Scholar ]
  • American Academy of Arts and Sciences. (2016). Public Research Universities: Serving the Public Good . Cambridge, MA: American Academy of Arts and Sciences [ Google Scholar ]
  • American Society of Cell Biology (ACSB). (2012). San Francisco Declaration on Research Assessment . Available at: www.ascb.org/dora (accessed September16, 2016)
  • Anderson N. (2013, Feb. 6). Five colleges misreported data to U.S. News, raising concerns about rankings, reputation . The Washington Post . Available at: www.washingtonpost.com/local/education/five-colleges-misreported-data-to-us-news-raising-concerns-about-rankings-reputation/2013/02/06/cb437876-6b17-11e2-af53-7b2b2a7510a8_story.html (accessed September16, 2016)
  • Aschwanden C. (2015, Aug. 19). Science isn't broken . FiveThirtyEight . Available at: http://fivethirtyeight.com/features/science-isnt-broken (accessed September16, 2016)
  • Ashforth B.E., and Anand V. (2003). The normalization of corruption in organizations . Res. Organ. Behav. 25 , 1 [ Google Scholar ]
  • Atlanta Public Schools (APS). (2016). About APS | Mission and Vision . Available at: www.atlanta.k12.ga.us/Domain/35 (accessed September16, 2016)
  • Barry F. (2015, Apr. 1). Peer-review cheating ring discredits another 43 papers . In-Pharma Technologist . Available at: www.in-pharmatechnologist.com/Regulatory-Safety/Peer-review-cheating-ring-discredits-another-43-papers (accessed September16, 2016)
  • Bartneck C., and Kokkelmans S. (2011). Detecting h-index manipulation through self-citation analysis . Scientometrics . 87 , 85. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Bedeian A.G., Taylor S.G., and Miller A.N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors . Acad. Manage. Learn. Educ. 9 , 715 [ Google Scholar ]
  • Belluz J., and Hoffman S. (2015, May 13). Science is often flawed. Its time we embraced that . Vox . Available at: www.vox.com/2015/5/13/8591837/how-science-is-broken (accessed September16, 2016)
  • Belluz J., Plumer B., and Resnick B. (2016, Jul. 14). The 7 biggest problems facing science, according to 270 scientists . Vox . Available at: www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process (accessed September16, 2016)
  • Borman S. (2016). Nanoparticle synthesis paper retracted after 12 years . Chemical and Engineering News . Available at: http://cen.acs.org/articles/94/i8/Nanoparticle-Synthesis-Paper-Retracted-After-12-Years.html (accessed September16, 2016)
  • Bornmann L., and Mutz R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references . J. Assoc. Inform. Sci. Technol. 66 , 2215 [ Google Scholar ]
  • Boyd G.R., Reiber S.H., McFadden M., and Korshin G. (2012). Effect of changing water quality on galvanic coupling . J. AWWA. 104 , E136 [ Google Scholar ]
  • British Broadcasting Corporation (BBC). (2016). Saving science from the scientists: Episode 2 . BBC . Available at: www.bbc.co.uk/programmes/b07378cr (accessed September16, 2016)
  • Brownlee J.K. (2014). Irreconcilable Differences: The Corporatization of Canadian Universities (Doctoral dissertation). Carleton University; Available at: https://curve.carleton.ca/b945d1f1-64d4-40eb-92d2-1a29effe0f76 (accessed September16, 2016) [ Google Scholar ]
  • Buela-Casal G. (2014). Pathological publishing: A new psychological disorder with legal consequences? Eur . J. Psychol. Appl. Legal Context. 6 , 91 [ Google Scholar ]
  • Callaway E. (2016). Beat it, impact factor! Publishing elite turns against controversial metric . Nature . Available at: www.nature.com/news/beat-it-impact-factor-publishing-elite-turns-against-controversial-metric-1.20224 (accessed September16, 2016) [ PubMed ]
  • Campbell A.G., Skvirsky R., Wortis H., Thomas S., Kawachi I., and Hohmann C. (2014). Nest 2014: Views from the trainees-talking about what matters in efforts to diversify the stem workforce . CBE Life Sci. Educ. 13 , 587. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Carpenter C.R., Cone D.C., and Sarli C.C. (2014). Using publication metrics to highlight academic productivity and research impact . Acad. Emerg. Med. 21 , 1160. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Casassus B. (2014). China predicted to outspend the US on science by 2020 . Nature . Available at: www.nature.com/news/china-predicted-to-outspend-the-us-on-science-by-2020-1.16329 (accessed September16, 2016)
  • Casper G. (1996). Criticism of College Rankings . Letter from Stanford President Gerhard Casper to James Fallows, Editor of U.S. News & World Report . Available at: http://web.stanford.edu/dept/pres-provost/president/speeches/961206gcfallow.html (accessed September16, 2016)
  • Chronicle of Higher Education (CHE). (2013). What we talk about when we talk about quitting . Chronicle of Higher Education . Available at: https://chroniclevitae.com/news/215-what-we-talk-about-when-we-talk-about-quitting?cid=vem (accessed September16, 2016)
  • Chronicle Vitae (2013–2014). Quit lit . Chronicle Vitae . Available at: https://chroniclevitae.com/news/tags/Quit%20Lit (accessed September16, 2016)
  • Coleman-Adebayo M. (2011). No Fear: A Whistleblower's Triumph Over Corruption and Retaliation at the EPA . Chicago, IL: Chicago Review Press [ Google Scholar ]
  • Curtis J.W., and Thornton S. (2013). Here's the news: The annual report on the economic status of the profession 2012–13 . Academe . 99 , 4 [ Google Scholar ]
  • Cushman P., Hoeksema J.T., Kouveliotou C., Lowenthal J., Peterson B., Stassun K.G., and von Hippel T. (2015). Impact of declining proposal success rates on scientific productivity . AAAC Proposal Pressures Study Group Discussion Draft . Available at: www.nsf.gov/attachments/134636/public/proposal_success_rates_aaac_final.pdf (accessed September16, 2016)
  • Cycling Independent Reform Commission (CIRC). (2015). Report to the President of the Union Cyclist Internationale . Available at: www.uci.ch/mm/Document/News/CleanSport/16/87/99/CIRCReport2015_Neutral.pdf (accessed September16, 2016)
  • Cyranoski D., Gilbert N., Ledford H., Nayar A., and Yahia M. (2011). Education: The PhD factory . Nature . 472 , 276. [ PubMed ] [ Google Scholar ]
  • Delgado López-Cózar N., Robinson-García N., and Torres-Salinas D. (2012). Manipulating Google scholar citations and Google scholar metrics: Simple, easy and tempting . EC3 Working Papers . 6 : 29 May, 2012 [ Google Scholar ]
  • Dem G. (2011). How to Increase Your Papers Citation and H Index . Available at: www.academia.edu/934257/How_to_increase_your_papers_citations_and_ h_index_in_5_simple_steps (accessed September16, 2016)
  • Denworth L. (2008). Toxic Truth: A Scientist, a Doctor, and the Battle Over Lead . Boston, MA: Beacon Press [ Google Scholar ]
  • Diekman A.B., Brown E.R., Johnston A.M., and Clark E.K. (2010). Seeking congruity between goals and roles: A new look at why women opt out of science, technology, engineering, and mathematics careers . Psychol. Sci. 21 , 1051. [ PubMed ] [ Google Scholar ]
  • Dimeo P. (2015, Jul. 10). How cycling's dark history continues to haunt the Tour de France . The Conversation . Available at: http://theconversation.com/how-cyclings-dark-history-continues-to-haunt-the-tour-de-france-44312 (accessed September16, 2016)
  • Dunn S. (2013, Dec. 12). Why so many academics quit and tell . Chronicle of Higher Education . Available at: https://chroniclevitae.com/news/216-why-so-many-academics-quit-and-tell (accessed September16, 2016)
  • Dzeng E. (2014). How academia and publishing are destroying scientific innovation: A conversation with Sydney Brenner . Elizabeth Dzeng's Blog; Available at: https://elizabethdzeng.com/2014/02/26/how-academia-and-publishing-are-destroying-scientific-innovation-a-conversation-with-sydney-brenner/ (accessed September16, 2016) [ Google Scholar ]
  • Edwards M.A. (2012). Discussion: Effect of changing water quality on galvanic coupling . J. Am. Water Works Assoc. 104 , 65 [ Google Scholar ]
  • Edwards M.A. (2014). Foreword . In Lewis D.L., Ed., Science for Sale . New York: Skyhorse Publishing, pp. ix–xiii [ Google Scholar ]
  • Edwards M.A. (2016). Institutional Scientific Misconduct at U.S. Public Health Agencies: How Malevolent Government Betrayed Flint, MI . Testimony to the U.S. Cong. Committee on Oversight and Government Reform on Examining Federal Administration of the Safe Drinking Water Act in Flint , Michigan Hearing, February 3, 2016. 112th Congress. 2nd session Available at: https://oversight.house.gov/wp-content/uploads/2016/02/Edwards-VA-Tech-Statement-2-3-Flint-Water.pdf (accessed September16, 2016) [ Google Scholar ]
  • Eisen J.A. (2010–2015). Microbiomania . Personal Blog . Available at: http://phylogenomics.blogspot.com/p/blog-page.html (accessed September16, 2016)
  • Elton L. (2004). Goodhart's law and performance indicators in higher education . Eval. Res. Educ. 18 , 120 [ Google Scholar ]
  • Falagas M.E., and Alexiou V.G. (2008). The top-ten in journal impact factor manipulation . Arch Immunol Ther Exp (Warsz) . 56 , 223. [ PubMed ] [ Google Scholar ]
  • Fanelli D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data . PLoS One . 4 , e5738. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Fanelli D., Costas R., and Larivière V. (2015). Misconduct policies, academic culture and career stage, not gender or pressures to publish, affect scientific integrity . PLoS One . 10 , e0127556. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Fang F.C., and Casadevall A. (2016). Research funding: The case for a modified lottery . mBio . 7 , e00422. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Fang F.C., Grant Steen R., and Casadevall A. (2012). Misconduct accounts for the majority of retracted scientific publications . Proc. Natl. Aacd. Sci. U. S. A. 109 , 17028 [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Faust D.G. (2009, Sep. 1). The University's Crisis of Purpose . The New York Times . Available at: www.nytimes.com/2009/09/06/books/review/Faust-t.html?_r=0 (accessed September16, 2016)
  • Feynman R.P. (1974). Cargo Cult Science: Caltech's 1974 Commencement Address . Available at: http://calteches.library.caltech.edu/51/2/CargoCult.htm (accessed September16, 2016)
  • Fischer P. (2011). New Research Misconduct Policies . National Science Foundation; Available at: www.nsf.gov/oig/_pdf/presentations/session.pdf (accessed September16, 2016) [ Google Scholar ]
  • Fischer J., Ritchie E.G., and Hanspach J. (2012). An academia beyond quantity: A reply to Loyola et al . and Halme et al. Trends Ecol. Evol . 27 , 587 [ Google Scholar ]
  • Frodeman R. (2011). Interdisciplinary research and academic sustainability: Managing knowledge in an age of accountability . Environ. Conserv. 38 , 105 [ Google Scholar ]
  • Gibbs K.D., and Griffin K.A. (2013). What do I want to be with my PhD? The roles of personal values and structural dynamics in shaping the career interests of recent biomedical science phd graduates . CBE Life Sci. Educ. 12 , 711. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Gladwell M. (2011, Feb. 24). The order of things . The New Yorker . Available at: www.newyorker.com/magazine/2011/02/14/the-order-of-things (accessed September16, 2016)
  • Gobry P.-E. (2016, Apr. 18). Big science is broken . The Week . Available at: http://theweek.com/articles/618141/big-science-broken (accessed September16, 2016)
  • Harding T.S., Carpenter D.D., and Finelli C.J. (2012). An exploratory investigation of the ethical behavior of engineering undergraduates . J. Eng. Educ. 101 , 346 [ Google Scholar ]
  • Harris R., and Benincasa R. (2014, Sep. 9). U.S. science suffering from booms and busts in funding . National Public Radio . Available at: www.npr.org/sections/health-shots/2014/09/09/340716091/u-s-science-suffering-from-booms-and-busts-in-funding (accessed September16, 2016)
  • Hinkes-Jones L. (2014). Bad science . Jacobin . Available at: www.jacobinmag.com/2014/06/bad-science (accessed September16, 2016)
  • Hourihan M. (2015). Federal R&D Budget Trends: A Short Summary . Available at: www.aaas.org/program/rd-budget-and-policy-program (accessed September16, 2016)
  • Huber B.R. (2014, Sep. 22). Scientists seen as competent but not trusted by Americans . Woodrow Wilson Research Briefs . Available at: wws.princeton.edu/news-and-events/news/item/scientists-seen-competent-not-trusted-americans (accessed September16, 2016)
  • Jaschik S. (2013, Sep. 18). Feeling the Heat: The 2013 Survey of College and University Admissions Directors . Inside Higher Ed, www.insidehighered.com/news/survey/feeling-heat-2013-survey-college-and-university-admissions-directors (accessed September16, 2016) [ Google Scholar ]
  • Kamenetz A. (2015, Apr. 1). The Atlanta cheating verdict: Some context . National Public Radio . Available at: www.npr.org/blogs/ed/2015/04/01/396869874/the-atlanta-cheating-verdict-some-context (accessed September16, 2016)
  • Kasperkevic J. (2014, Oct. 7). The harsh truth: US colleges are businesses, and student loans pay the bills . The Guardian . Available at: www.theguardian.com/money/us-money-blog/2014/oct/07/colleges-ceos-cooper-union-ivory-tower-tuition-student-loan-debt (accessed September16, 2016)
  • Kevles D.J. (2000). The Baltimore Case: A Trial of Politics, Science, and Character . New York: W. W. Norton & Company [ Google Scholar ]
  • Kolata G. (2016, Jul. 14). So many research scientists, so few openings as professors . The New York Times . Available at: www.nytimes.com/2016/07/14/upshot/so-many-research-scientists-so-few-openings-as-professors.html (accessed September16, 2016)
  • Kutner M. (2014, Sep.). How Northeastern university gamed the college rankings . Boston Magazine . Available at: www.bostonmagazine.com/news/article/2014/08/26/how-northeastern-gamed-the-college-rankings (accessed September16, 2016)
  • Labbé C. (2010). Ike Antkare One of the Great Stars in the Scientific Firmament . Laboratoire d'Informatique de Grenoble Research Report No. RR-LIG-008. Available at: http://rr.liglab.fr/research_report/RR-LIG-008.pdf (accessed September16, 2016)
  • Ladner D.A., Bolyard S.C., Apul D., and Whelton A.J. (2013). Navigating the academic job search for environmental engineers: Guidance for job seekers and mentors . J. Prof. Issues Eng. Educ. Pract. 139 , 211 [ Google Scholar ]
  • Laduke R.D. (2013). Academic dishonesty today, unethical practices tomorrow? J . Prof. Nurs. 29 , 402 [ PubMed ] [ Google Scholar ]
  • Langmuir I. (1953). Colloquium on Pathological Science. Knolls Research Laboratory , December 18, 1953. Available at: www.cs.princeton.edu/∼ken/Langmuir/langmuir.htm (accessed September16, 2016)
  • Lawrence P.A. (2009). Real lives and white lies in the funding of scientific research . PLoS Biol . 7 , e1000197. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Lee C. (2007, May 28). Slump in NIH funding is taking toll on research . The Washington Post . Available at: www.washingtonpost.com/wp-dyn/content/article/2007/05/27/AR2007052700794.html (accessed September16, 2016)
  • Leiserowitz A.A., Maibach E.W., Roser-Renouf C., Smith N., and Dawson E. (2012). Climategate, public opinion and the loss of trust . Am. Behav. Scientist. 57 , 818 [ Google Scholar ]
  • Lewis D.L. (2014). Science for Sale . New York: Skyhorse Publishing [ Google Scholar ]
  • Macilwain C. (1997). Whistleblowers face blast of hostility . Nature . 385 , 669. [ PubMed ] [ Google Scholar ]
  • Macilwain C. (2013). Halt the avalanche of performance metrics . Nature . 500 , 255. [ PubMed ] [ Google Scholar ]
  • Marcus A., and Oransky I. (2015, May 22). What's behind big science frauds? The New York Times . Available at: www.nytimes.com/2015/05/23/opinion/whats-behind-big-science-frauds.html (accessed September16, 2016)
  • Martinson B.C., Crain A.L., Anderson M.S., and De Vries R. (2009). Institutions' expectations for researchers' self-funding, federal grant holding, and private industry involvement: Manifold drivers of self-interest and researcher behavior . Acad. Med. 84 , 1491. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Massachusetts Institute of Technology (MIT). (2015). The Future Postponed . A Report by the MIT Committee to evaluate the innovation deficit . Available at: https://dc.mit.edu/sites/default/files/Future%20Postponed.pdf (accessed September16, 2016)
  • McDermott J. (2013, Mar. 7). Gaming the system: How to get an astronomical h-index with little scientific impact . Personal Blog . Available at: http://jasonya.com/wp/gaming-the-system-how-to-get-an-astronomical-h-index-with-little-scientific-impact (accessed September16, 2016)
  • Michalek A.M., Hutson A.D., Wicher C.P., and Trump D.L. (2010). The costs and underappreciated consequences of research misconduct: A case study . PLoS Med . 7 , e1000318. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Mirowski P. (2011). Science-Mart . Boston, MA: Harvard University Press [ Google Scholar ]
  • Morse R. (2015, Mar. 27). Debunking myths about The U.S. news best colleges rankings . U.S. News & World Report . Available at: www.usnews.com/education/blogs/college-rankings-blog/2015/03/27/debunking-myths-about-the-us-news-best-colleges-rankings (accessed September16, 2016)
  • Mulvey S. (2012, Oct. 12). Lance Armstrong: Tyler Hamilton on ‘how US Postal cheated’ . BBC News . Available at: www.bbc.com/news/magazine-19912623 (accessed September16, 2016)
  • National Academy of Engineering (NAE). (2004). The Engineer of 2020: Visions of Engineering in the New Century . Washington, DC: The National Academies Press [ Google Scholar ]
  • National Institutes of Health (NIH). (2008). Average Age of Principal Investigators . Available at: https://report.nih.gov/NIH_Investment/PDF_sectionwise/NIH_Extramural_DataBook_PDF/NEDB_SPECIAL_TOPIC-AVERAGE_AGE.pdf (accessed September16, 2016)
  • National Institutes of Health (NIH). (2015). NIH Research Portfolio Online Reporting Tools–Research Project Success Rates of NIH Institute . Available at: https://report.nih.gov/success_rates/Success_ByIC.cfm (accessed September16, 2016)
  • National Science Board (NSB). (2014). Reducing Investigators' Administrative Workload for Federally Funded Research . Report NSB-14-18. Arlington, VA: National Science Foundation [ Google Scholar ]
  • National Science Foundation (NSF). (2016). Budget Internet Information System: Providing Statistical and Funding Information . Available at: http://dellweb.bfa.nsf.gov/awdfr3/default.asp (accessed September16, 2016)
  • Nature. (2015, May 19). Editorial: Publish or perish . Nature . 521 , 159. [ Google Scholar ]
  • New Scientist. (2016, May 17). Microbiomania: The Truth Behind the Hype About Our Bodily Bugs . Available at: www.newscientist.com/article/2088613-microbiomania-the-truth-behind-the-hype-about-our-bodily-bugs (accessed September16, 2016)
  • Nuzzo R. (2015, Oct. 7). How scientists fool themselves–and how they can stop . Nature . 526 , 182. [ PubMed ] [ Google Scholar ]
  • OECD. (2014). OECD Science, Technology and Industry Outlook 2014 . Paris: OECD Publishing [ Google Scholar ]
  • Oreskes N., and Conway E.M. (2010). Merchants of Doubt . New York: Bloomsbury Press [ Google Scholar ]
  • Plerou V., Nunes Amaral L.A., Gopikrishnan P., Meyer M., and Stanley H.G. (1999). Similarities between the growth dynamics of university research and of competitive economic activities . Nature . 400 , 433 [ Google Scholar ]
  • Porter E. (2015, May 19). American Innovation Lies on Weak Foundation . The New York Times . Available at: www.nytimes.com/2015/05/20/business/economy/american-innovation-rests-on-weak-foundation.html (accessed September16, 2016)
  • Price D.J.D.-S. (1963). Little Science, Big Science… and Beyond . New York: Columbia University Press [ Google Scholar ]
  • Quake S. (2009). Letting scientists off the leash . The New York Times Blog , February 10
  • Reiber S., and Dufresne L. (2006). Effects of External Currents and Dissimilar Metal Contact on Corrosion of Lead from Lead Service Lines . Philadelphia: Final Report to USEPA region III [ Google Scholar ]
  • Retraction Watch. (2015a, Nov. 5). Got the Blues? You Can Still See Blue: Popular Paper on Sadness, Color Perception Retracted . Available at: http://retractionwatch.com/2015/11/05/got-the-blues-you-can-still-see-blue-after-all-paper-on-sadness-and-color-perception-retracted (accessed September16, 2016)
  • Retraction Watch. (2015b). Expression of Concern Opens Floodgates of Controversy Over Lead in Water Supply . Available at: http://retractionwatch.com/2015/04/15/expression-of-concern-opens-floodgates-of-controversy-over-lead-in-water-supply (accessed September16, 2016)
  • Rice J.R. (1994). Metrics to Evaluate Academic Departments . Purdue University Computer Science Technical Reports . Paper 1148
  • Rose C., and Fisher N. (2013). Following Lance Armstrong: Excellence corrupted . Harvard Business Review , Case Study # 314015-PDF-ENG, Available at: https://hbr.org/product/following-lance-armstrong-excellence-corrupted/an/314015-PDF-ENG (accessed September16, 2016)
  • Saraceno J. (2013, Jan. 18). Lance Armstrong: ‘I was trying to win at all costs.’ USA Today . Available at: www.usatoday.com/story/sports/cycling/2013/01/18/lance-armstrong-oprah-winfrey-television-interview-tour-de-france-drugs/1843329 (accessed September16, 2016)
  • Sarewitz D. (2016). The pressure to publish pushes down quality . Nature . 533 , 147. [ PubMed ] [ Google Scholar ]
  • Sauermann H., and Roach M. (2012). Science PhD career preferences: Levels, changes, and advisor encouragement . PLoS One . 7 , e36307. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Schekman R. (2013, Dec. 9). How journals like Nature, Cell and Science are damaging science . The Guardian . Available at: www.theguardian.com/commentisfree/2013/dec/09/how-journals-nature-science-cell-damage-science (accessed September16, 2016)
  • Schneider R.L., Ness K.L., Rockwell S., Shaver S., and Brutkiewicz R. (2014). 2012 Faculty Workload Survey Research Report . National Academy of Sciences, Engineering, and Medicine; Available at: http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_087667.pdf (accessed September16, 2016) [ Google Scholar ]
  • Segal M. (2014, Sep. 15). The big college ranking sham: Why you must ignore U.S. News and World's Report list . Salon . Available at: www.salon.com/2014/09/15/the_big_college_ranking_sham_why_you_must_ignore_u_s_news_and_worlds_report_list (accessed September16, 2016)
  • Seligsohn A. (2015, Apr. 29). An observation about the mission of higher education . Campus Compact . Available at: http://compact.org/resource-posts/an-observation-about-the-mission-of-higher-education (accessed September16, 2016)
  • Slotnik D.E., and Perez-Pena R. (2012, Jan. 30). “ College Says It Exaggerated SAT Figures for Ratings .” The New York Times . www.nytimes.com/2012/01/31/education/claremont-mckenna-college-says-it-exaggerated-sat-figures.html?smid=pl-share&_r=0 (accessed September16, 2016)
  • Smaldino P.E., and McElreath R. (2016). The natural selection of bad science . arXiv . Available at: https://arxiv.org/abs/1605.09511 (accessed September16, 2016)
  • Stein S. (2015, Mar. 3). Congrats young scientists, you face the worst research funding in 50 years . The Huffington Post . Available at: www.huffingtonpost.com/2015/03/03/francis-colliins-nih-funding_n_6795900.html (accessed September16, 2016)
  • Steneck N.H. (2007). ORI: Introduction to the Responsible Conduct of Research . U.S. Department of Health and Human Services . Washington, DC: Government Printing Office [ Google Scholar ]
  • Stephan P. (2012a). How Economics Shapes Science . Boston, MA: Harvard University Press [ Google Scholar ]
  • Stephan P. (2012b). Research efficiency: Perverse incentives . Nature . 484 , 29. [ PubMed ] [ Google Scholar ]
  • Stern A.M., Casadevall A., Steen R.G., and Fang F.C. (2014). Financial costs and personal consequences of research misconduct resulting in retracted publications . eLife . 3 , e02956. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Stroebe W., Postmes T., and Spears R. (2012). Scientific misconduct and the myth of self-correction in science . Perspect. Psychol. Sci. 7 , 670. [ PubMed ] [ Google Scholar ]
  • Swazey J.P., Anderson M.S., Lewis K.S., and Louis K.S. (1993). Ethical problems in academic research . Am. Scientist. 81 , 542 [ Google Scholar ]
  • The Economist. (2010, Dec. 16). The Disposable Academic: Why doing a PhD is often a waste of time . Available at: www.economist.com/node/17723223 (accessed September16, 2016)
  • The Economist. (2013, Oct. 19). How Science Goes Wrong . Available at: www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong (accessed September16, 2016)
  • The New York Times. (1991, Jun. 4). Commencements; Science Official Tells M.I.T. of Perils to Ethics. Available at: www.nytimes.com/1991/06/04/us/commencements-science-official-tells-mit-of-peril-to-ethics.html (accessed September16, 2016)
  • Thoman D.B., Brown E.R., Mason A.Z., Harmsen A.G., and Smith J.L. (2014). The role of altruistic values in motivating underrepresented minority students for biomedicine . BioScience . 65 , 183 [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Tierney J. (2013, Sep. 10). Your Annual Reminder to Ignore the U.S. News & World Report College Rankings . The Atlantic . Available at: www.theatlantic.com/education/archive/2013/09/your-annual-reminder-to-ignore-the-em-us-news-world-report-em-college-rankings/279103 (accessed September16, 2016)
  • Tijdink J.K., Verbeke R., and Smulders Y.M. (2014). Publication pressure and scientific misconduct in medical scientists . J. Empir. Res. Hum. Res. Ethics. 9 , 64. [ PubMed ] [ Google Scholar ]
  • U.S. Anti-Doping Agency (U.S. ADA). (2012). Lance Armstrong Receives Lifetime Ban and Disqualification of Competitive Results for Doping Violations Stemming from His Involvement in the United States Postal Service Pro-Cycling Team Doping Conspiracy . Available at: www.usada.org/lance-armstrong-receives-lifetime-ban-and-disqualification-of-competitive-results-for-doping-violations-stemming-from-his-involvement-in-the-united-states-postal-service-pro-cycling-team-doping-conspi (accessed September16, 2016)
  • U.S. Centers for Disease Control and Prevention. (2004). Blood lead levels in residents of homes with elevated lead in tap water–District of Columbia, 2004 . CDC Morbid. Mortal. Weekly Rep. 53 , 268 [ PubMed ] [ Google Scholar ]
  • U.S. Cong. House, Committee on Oversight and Government Reform. (2015). EPA Mismanagement II July 29, 2015. Hearing. 114th Congress. 1st Session Available at: https://oversight.house.gov/hearing/epa-mismanagement-part-ii (accessed September16, 2016) [ Google Scholar ]
  • U.S. Cong. House, Committee on Oversight and Government Reform. (2016). Examining Federal Administration of the Safe Drinking Water Act in Flint , Michigan Hearing, February 3, 2016. Hearing. 114th Congress. 2nd Session Available at: https://oversight.house.gov/hearing/examining-federal-administration-of-the-safe-drinking-water-act-in-flint-michigan (accessed September16, 2016) [ Google Scholar ]
  • U.S. Department of Health and Human Services. (2013). Office of Research Integrity Annual Report 2012
  • U.S. House Committee on Education and the Workforce. (2014). The Just In Time Professor . Available at: http://democrats.edworkforce.house.gov/sites/democrats.edworkforce.house.gov/files/documents/1.24.14-AdjunctEforumReport.pdf (accessed September16, 2016)
  • U.S. House Committee on Science and Technology. (2010). A Public Health Tragedy: How Flawed CDC Data and Faulty Assumptions Endangered Children's Health in the Nation's Capital . Report by the Majority Staff on the Subcommittee on Investigations and Oversight. Available at: http://archives.democrats.science.house.gov/publications/caucus_detail.aspx?NewsID=2843 (accessed September16, 2016)
  • Van Noorden R. (2010). Metrics: A profusion of measures . Nature . 465 , 864. [ PubMed ] [ Google Scholar ]
  • Van Noorden R. (2014). Publishers withdraw more than 120 gibberish papers . Nature . Available at: www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763 (accessed September16, 2016)
  • Werner R. (2015). The focus on bibliometrics makes papers less useful . Nature . 517 , 245. [ PubMed ] [ Google Scholar ]
  • Wilson W.A. (2016, May). Scientific regress . First Things . Available at: www.firstthings.com/article/2016/05/scientific-regress (accessed September16, 2016)
  • Wolverton M. (1998). Treading the tenure-track tightrope: Finding balance between research excellence and quality teaching . Innovative Higher Educ . 23 , 61 [ Google Scholar ]
  • Zetter K. (2009, Feb. 6). TED: Barry Schwartz and the Importance of Practical Wisdom . WIRED . Available at: www.wired.com/2009/02/ted-barry-schwa (accessed September16, 2016)
  • Zhivotovsky L., and Krutovsky K. (2008). Self-citation can inflate h-index . Scientometrics . 77 , 373 [ Google Scholar ]

P21 Resources

Partnership for 21st century learning frameworks & resources.

P21’s Frameworks for 21st Century Learning were developed with input from teachers, education experts, and business leaders to define and illustrate the skills and knowledge students need to succeed in work and life, as well as the support systems necessary for 21st century learning outcomes. They have been used by thousands of educators and hundreds of schools in the U.S. and abroad to put 21st century skills at the center of learning.

Battelle for Kids looks forward to engaging yet again with educators and other experts to update the Frameworks. We encourage personal use of these Frameworks by all who can benefit:

Image

  • Framework for 21st Century Learning – brief and extended versions
  • Early Learning Framework – brief and extended versions

Permission for Use

Permission for commercial use and/or distribution of these resources or other information produced must be  made in writing to Battelle for Kids with a clearly defined request and description of how the material will be used . Permission will be granted provided that:

  • Content remain unchanged.
  • Attribution be given to Battelle for Kids.
  • Attribution will include a link to Battelle for Kids’ website.
  • Content will be used solely for educational purposes.
  • The party using or referring to Battelle for Kids’ intellectual property will not represent themselves in a manner that can be interpreted as implying that they are officially certified or endorsed by Battelle for Kids.

In addition, please note that:

  • The Frameworks for 21st Century Learning or its representational images may not be used to endorse specific products or services
  • Battelle for Kids reserves the right to restrict usage of any intellectual property that Battelle for Kids finds is being used in an inappropriate manner
  • Battelle for Kids network, member, or partner names; logos; and other content may not be used without permission for any purposes other than to demonstrate the organizations and corporations that are partners or members of Battelle for Kids or its networks. Partners and members may not be listed selectively. Any other requests for use of Battelle for Kids partners’ or members’ trademarks and logos must be approved by the member organization prior to use

Battelle for Kids will continue publishing frequently requested P21 materials and refreshing them for the benefit of the 21st century education movement and community. Please check back often for updates.

For additional information about the Partnership for 21st Century Learning and the available resources available, contact the Battelle for Kids team.

© 2024, Battelle for Kids. All Rights Reserved. | Privacy Policy

Economics of Transportation in the 21st Century

To promote research in transportation economics and to strengthen the economic foundations for transportation policies in the 21st century, the National Bureau of Economic Research (NBER), with the support of the US Department of Transportation (DOT), is carrying out a multi-year research initiative on “Transportation Economics in the 21st Century.”  This initiative, led by NBER researchers Edward Glaeser (Harvard University), James Poterba (MIT), and Stephen Redding (Princeton University), brings together researchers in various subfields of economics -- energy economics, industrial organization, macroeconomics, environmental economics, regional and urban economics, regulatory economics, and public finance, as well as transportation economics -- to study issues of current importance in the transportation sector and to develop an agenda for future research. This research project will host a virtual conference on Friday, May 3, 2024, to showcase new findings in transportation economics.  It will address a range of possible topics, including but not limited to:

  • The demand for electric vehicles and how it is affected by public and private infrastructure provision.
  • The effects tax and regulatory policies on transportation-related pollution, and the cost of reducing the carbon emissions from the transportation sector. 
  • The impact of differential access to transportation services and historical infrastructure investments on disparities in economic outcomes across racial, ethnic, geographic, and socioeconomic groups. 
  • The measurement of the benefits and costs of, and willingness to pay for, transport infrastructure improvements, along with market-based mechanisms for pricing such improvements.
  • The economic factors that affect the development of new transportation modalities, such as ridesharing, and new options for transportation services, such as autonomous vehicles.
  • The impact of new technologies for managing surface freight transportation, including data-intensive analysis using tools such as machine learning.
  • The impact of the transportation sector on aggregate economic activity  and the distribution of economic activity within urban areas and between rural and urban areas. 
  • The implications of the COVID-19 pandemic and subsequent developments such as increased work-from-home on public transit systems and transportation infrastructure more generally.
  • The returns to, and financing of, investments in and maintenance of transportation infrastructure, including roads, rail, air, pipelines, ports, and liquid natural gas terminals.
  • The economic forces affecting transportation safety and the impact of new technologies, such as in-vehicle tests for sobriety and inter-vehicle communication and monitoring tools. 

Submission of papers by researchers with and without NBER affiliations, from early career scholars and from researchers from under-represented groups, are welcome.  Please do not submit papers that will be published by April 2024. To be considered for inclusion on the program,  upload papers by 11:59pm ET on Tuesday, March 5, 2024. All papers should include a comprehensive conflict of interest statement that describes any financial or other interests that the researchers might have with regard to the research.  Decisions about which papers will be included on the program will be announced by late March 2024.  Please feel free to forward this call to others who might be conducting research on related topics.  Questions about this meeting may be directed to [email protected] .

IMAGES

  1. (PDF) Entrepreneurship and Education in the 21st Century: Analysis and

    research papers on 21st century

  2. (PDF) Importance of Human Resource Management in 21st Century: A

    research papers on 21st century

  3. (PDF) Implementation of 21st Century Learning Through Lesson Study

    research papers on 21st century

  4. 🌷 21st century education essay. Essay On Education In 21st Century

    research papers on 21st century

  5. (PDF) Promoting 21st century skills in Higher Education through

    research papers on 21st century

  6. (PDF) 21st Century Learning Skills: A Challenge in Every Classroom

    research papers on 21st century

VIDEO

  1. Eng_303 Novel (18th & 19th Century) 2018 Past Papers Solution || 5th Semester || BS English ❤️ 🌼

  2. Eng_303 Novel (18th & 19th Century) 2017 Past Papers Solution || 5th Semester || BS English ❤️ 🌼

  3. Eng_303 Novel (18th & 19th Century) 2023 Past Papers Solution || 5th Semester || BS English ❤️ 🌼

  4. Eng_303 Novel (18th & 19th Century) 2020 Past Papers Solution || 5th Semester || BS English ❤️ 🌼

  5. Eng_303 Novel (18th & 19th Century) 2019 Past Papers Solution || 5th Semester || BS English ❤️ 🌼

COMMENTS

  1. Determinants of 21st-Century Skills and 21st-Century Digital Skills for

    To study differences in digital skills and to develop interventions for skill improvements, in the past years several skill frameworks and definitions have been introduced (e.g., 21st-century skills, digital skills, digital competence, digital literacy, e-skills, internet skills).

  2. Improving 21st-century teaching skills: The key to effective 21st

    The 21st-century skillset is generally understood to encompass a range of competencies, including critical thinking, problem solving, creativity, meta-cognition, communication, digital and technological literacy, civic responsibility, and global awareness (for a review of frameworks, see Dede, 2010 ).

  3. Research on Adolescence in the Twenty-First Century

    Research on adolescence has also changed dramatically. This review discusses recent developments in this literature, being cognizant of their historical underpinnings while focusing on the future. ... New Directions for the 21st Century. New York: Oxford; 1993. pp. 13-37. [Google Scholar] Crosnoe R. Low-income students and the socioeconomic ...

  4. Developing 21st century teaching skills: A case study of teaching and

    18,294 Views 4 CrossRef citations to date 0 Altmetric Listen TEACHER EDUCATION & DEVELOPMENT Developing 21st century teaching skills: A case study of teaching and learning through project-based curriculum Corinne Martinez Article: 2024936 | Received 23 Sep 2021, Accepted 21 Nov 2021, Published online: 17 Jan 2022 Cite this article

  5. (PDF) Toward an understanding of 21st-century skills ...

    Toward an understanding of 21st-century skills: From a systematic review DOI: Authors: Dandan Chen Pearson Abstract and Figures The first two decades of the twenty-first century witnessed the...

  6. 21st CENTURY LEARNING SKILLS IN EDUCATION AND EMPLOYABILITY

    CENTURY LEARNING SKILLS IN EDUCATION AND EMPLOYABILITY William Xavier Toro The purpose of this mixed-methods study was to research a diverse school in Westchester County, New York to analyze whether it is aligned to 21. st. century practices. This study used both qualitative and quantitative data from focus-group interviews, surveys and non-

  7. Taking 21st century skills from vision to classroom: What teachers

    https://doi.org/10.1016/j.tate.2021.103286 Modelling effective strategies are useful in order to teach according to reform visions. The purpose of this paper is to accentuate teachers' voice and examine what teachers consider as useful professional development for promoting 21st century skills as presented in educational reforms.

  8. The Futures of learning 2: what kind of learning for the 21st century?

    education research and foresight 14 november 2015 united nations educational, scientific and cultural organization working papers introduction overall vision of twenty-first century learning • learning to know • learning to do • learning to be • learning to live together conclusions, next steps and future issues the futures of learning 2: what kind of learning for the 21st century?

  9. Developing student 21st Century skills in selected exemplary inclusive

    Research Open access Published: 25 November 2019 Developing student 21 st Century skills in selected exemplary inclusive STEM high schools Stephanie M. Stehle & Erin E. Peters-Burton International Journal of STEM Education 6, Article number: 39 ( 2019 ) Cite this article 116k Accesses 85 Citations Metrics Abstract Background

  10. The Futures of Learning 3: what kind of pedagogies for the 21st century?

    Twenty-first century pedagogy must employ innovative and research- supported teaching strategies, learning technologies and real- world applications (Saavedra and Opfer, 2012). Opportunities for learners to apply twenty-first century skills across content areas are also essential for deeper understanding.

  11. Physics education research for 21st century learning

    Following the conceptually defined framework of twenty-first Century teaching and learning, this paper aims to provide contextualized operational definitions of the goals for twenty-first Century learning in physics (and STEM in general) as well as the rationale for the importance of these outcomes for current students.

  12. Consumer behavior research in the 21st century: Clusters, themes, and

    Search for more papers by this author. Guangrui Guo, Guangrui Guo. Odette School of Business, University of Windsor, Windsor, Ontario, Canada. ... This study provides a quantitative overview of contemporary consumer behavior research in the 21st century (2001-2020) to inform future research directions in consumer behavior research. Using co ...

  13. PDF 21st Century Learning: Research, Innovation and Policy

    21ST CENTURY LEARNING: RESEARCH, INNOVATION AND POLICY DIRECTIONS FROM RECENT OECD ANALYSES Learning is central in knowledge-based societies and economies. In many countries there is a push to reflect this by ensuring that reforms of the education system focus more strongly on learning itself rather than simply changing

  14. Global glacier change in the 21st century: Every increase in ...

    We produce a set of global glacier projections for every glacier on Earth for SSPs from 2015 to 2100 by leveraging global glacier mass balance data and near-global frontal ablation data (10-13).To provide policy-relevant scenarios, our projections are grouped based on mean global temperature increases by the end of the 21st century compared with preindustrial levels to explicitly link ...

  15. Research methodology in the 21st Century

    Research methodology in the 21st Century Abstract: Within this paper, we consider recent developments in methods of research and consider the context of how students are introduced to this compared with late 20th Century methodologies.

  16. (PDF) Artificial Intelligence in the 21st Century

    In this paper, we study the evolution of AI at the beginning of the 21st Century using publication metadata extracted from 9 top-tier journals and 12 top-tier conferences of this discipline....

  17. Why parenting matters for children in the 21st century

    The paper outlines how changes in the 21st century have altered family life and summarises evidence from 29 meta-studies and 81 quantitative studies for the developmental impact of different parenting styles and dimensions.

  18. Digital Transformation: An Overview of the Current State of the Art of

    This paper qualitatively classifies the literature on digital business transformation into three different clusters based on technological, business, and societal impacts. ... the number of publications started to noticeably rise only in the first decade of the 21st century, especially in the last 5 years, with the majority of the articles ...

  19. Research for development in the 21st century

    Research for development in the 21st century C. Rumpel, a,⁎ V. Ann, b H. Bahri, c M. Calabi Floody, d S. Cheik, e T.T. Doan, f A. Harit, g J.L. Janeau, a P. Jouquet, a M.L. Mora, d P. Podwojewski, a T.M. Tran, f Q.A. Ngo, h P.L. Rossi, a and M. Sanaullah i Author information Copyright and License information PMC Disclaimer

  20. Cancer Biology, Epidemiology, and Treatment in the 21st Century

    Further Challenges in the 21st Century. Beyond the challenges regarding oncologic biomedical research, the 21 st century is facing important issues that must be solved as soon as possible if we truly wish to gain significant ground in our fight against cancer. Three of the most important have to do with prevention, early diagnosis, and access ...

  21. (PDF) Nursing research in the 21 st century

    Nursing research in the 21 st century Journal of Health Specialties License CC BY-NC-SA 4.0 Authors: Roger Watson University of Hull Abstract Aim: To explore the development of nursing...

  22. Academic Research in the 21st Century: Maintaining Scientific Integrity

    In this article, we will (1) describe how perverse incentives and hypercompetition are altering academic behavior of researchers and universities, reducing scientific progress and increasing unethical actions, (2) propose a conceptual model that describes how emphasis on quantity versus quality can adversely affect true scientific progress, (3) ...

  23. Transformation of Labor: Robotic Education in Elementary Schools ...

    Despite the short time and limited conditions for the application, the increase in averages on posttests implies advances in students' 21st century skills. Consequently, the results of this research indicate that the use of digital technologies such as robotic design and coding helps elementary school students to think more deeply about given ...

  24. P21 Resources

    P21's Frameworks for 21st Century Learning were developed with input from teachers, education experts, and business leaders to define and illustrate the skills and knowledge students need to succeed in work and life, as well as the support systems necessary for 21st century learning outcomes. They have been used by thousands of educators and ...

  25. (PDF) Life Skills for 21st century

    Life Skills for 21st century Authors: Bhagyashree Dudhade SWAMI VIVEKANAND COLLEGE OF EDUCATION, WAKAD Discover the world's research Public Full-text Content uploaded by Bhagyashree Dudhade...

  26. Economics of Transportation in the 21st Century

    To promote research in transportation economics and to strengthen the economic foundations for transportation policies in the 21st century, the National ... interest statement that describes any financial or other interests that the researchers might have with regard to the research. Decisions about which papers will be included on the program ...