PI: Jamie Jensen

This report will outline how effectively we met our academic and mentoring objectives. I have listed below both the academic and mentoring objectives put forth in the original proposal and have outlined how they were met or modified over the course of the past two years. Let me just start out by saying how proud I am of my undergraduate team of researchers. They have been absolutely AMAZING and have taken this project on with incredible autonomy. Although we are a bit behind our anticipated schedule (due to the autonomy that I have tried to foster with the students), the careful and meticulous planning we have done and slow and deliberate pace will result in more effective outcomes and a greater growing experience for students.

Academic Objectives

Academic Objective #1: To empirically test the plasticity of scientific reasoning ability and investigate a way to improve these skills through carefully designed, adaptive online modules that scaffold the developmental process.

The process of accomplishing this first objective will be described in more detail through the mentoring objectives as this was almost entirely done by the students. In summary, students have created an online application using the Canvas platform that teaches scientific reasoning skills. Distractors are based on carefully investigated misconceptions solicited from in-­‐depth student interviews. In addition, each distractor is accompanied by video-­‐tutorials specifically designed to target misconceptions. The entire program has been conceived and designed by students under careful mentorship.

This first phase of development has resulted in some very interesting qualitative data concerning student misconceptions and missteps in the scientific reasoning process and some vital insight into the development of these skills in highly-­‐intelligent college students (i.e., BYU students). The students will be writing up these results for publication starting in January (estimated submission in April). This will hopefully result in publication with 8 student co-­‐authors.

Academic Objective #2: To investigate the relationship between scientific reasoning skills and STEM performance and retention.

We are preparing to implement this phase in the Winter 2017 semester. We have finalized the Build Expertise in STEM (BEST) Module and will be implementing it in introductory STEM courses and tracking student progress, performance, and retention, as outlined in the original proposal.

Mentoring Objectives

The research team consists of nine mentored undergraduates, hereafter referred to as ‘researchers’. Although the original proposal was for five undergraduates, the popularity of this program has spread and four additional students have asked to be a part of the team (in addition, we recruited the technology experts out of Engineering Education, instead of IP&T):

• Austen Deal (Biology)
• Sam Pollack (Biology)
• Shelby Kurtz (Biology Science Education)
• Faith Hathenbruck (Biology)
• Jenica Wooley (Engineering Education)
• Trent Park (Biology)
• Juliette Green (Biology)
• Bryant Transtrum (Exercise Science)
• Aubria Hunt (Engineering Education) – Aubria has since left our team due to personal reasons but she participated for two semesters before leaving.

Phase 1 Student Evidences of Success:

• Students will become proficient in our current understanding of the development and teaching of scientific reasoning skills by participating in group discussions.
• Students will write a draft ORCA proposal based on this project
• Students will write an IRB proposal for this project.

We have successfully accomplished all objectives under this phase. Through weekly meetings, students tackled the literature and built expertise on scientific reasoning skills. Faith Hathenbruck submitted an ORCA proposal related to this project in October of this year. We are awaiting results. Student wrote and submitted the IRB proposal and received permission to use human subjects in the research. This was a valuable experience for all of the students as they all had to complete CITI training and become familiar with human subjects protocol.

Phase 2 Student Evidences of Success:

• Students will complete workable modules.

This has been an incredible process of learning, trial, and error and the researchers have developed the most through this process. They began by creating sample module questions which they then administered to students both at BYU and UVU. As a team, we analyzed student responses and, using grounded theory, developed common missteps that students make in each of the different reasoning patterns. To confirm these missteps, the researchers called in individual students who had missed problems and conducted interviews to further understand student missteps and to characterize tactics that successfully helped students overcome their missteps. These interviews were video recorded after which, we again analyzed them as a group to identify common successful tutorial applications.

Researchers then mass produced reasoning questions based on the results we had gathered. These questions were again administered and refined. The researchers worked together as a team to accomplish their goals and to refine the modules. Researchers met with me weekly for guidance and support. Researchers have now created enough questions for the modules.

In addition, researchers then used the common tutorial applications gleaned from interviews to build video tutorials using “Explain Everything”. These video tutorials correspond to each of the common missteps used as distractors in the modules. In other words, if a student participant chooses answer choice “D” that represents one of the common missteps, a tutorial pops up that addresses that specific misstep and walks students through an explanation that has been proven successful in prior interviews with students. The researchers got together regularly to view and critique each other’s tutorials and make refinements. They have now successfully completed all tutorials required by the BEST Application.

Students will fully automate (i.e., put on line) these modules

Using the expertise of our researchers from Engineering Education (Jenica and Aubria), we have fully automated the modules using the Canvas platform. We will be testing it out for the first time in its automated form starting in January.

• Students will have a research plan in place with collaborating instructors Researchers have been in contact with instructors of introductory STEM courses and have coordinated with them to implement the modules in the Winter 2017 semester.

Phase 3 Student Evidences of Success:

• Students will obtain, sort, and organize pre-­test, post-­test, and longitudinal data
• Students will produce readily analyzable spreadsheets of the data

Due to other intense research projects running in the lab, this project was delayed in its start. In addition, once started, I decided to encourage more autonomy and independence in the research team, having them rely more upon each other than on me, than I have done in the past. I feel like this was an incredibly successful approach and resulted in greater confidence and learning in the researchers, however, it further delayed the project. Thus, we will be implementing our first round of the modules in Winter 2017 (as opposed to Fall 2015, as originally planned). We will be running it Winter and Fall of 2017 and will have publishable data by possibly April, but definitely December 2017. So, they will be accomplishing these objectives then.

Phase 4 Student Evidences of Success

• Students will complete a full statistical analysis of the data
• Students will prepare posters for presentation at the annual SABER meeting

As stated above, the statistical analyses will take place during the summer of 2017 and again in December 2017, once the modules have been run in two successive semesters.

This summer (July 2017), we will be presenting this research at the annual SABER meeting. I plan to take any members of the research team who have not graduated and who are available to go to present their research. We will apply for an oral presentation, with a poster presentation as a back-­‐up.

Phase 5 Student Evidences of Success

• Students will prepare a manuscript

As stated before, students will be preparing their qualitative work on student missteps in reasoning to be submitted by April. We will be running a ‘publishing’ course as our lab meeting this coming semester in order to facilitate the writing process. The results of the implementation phase will be written up in April, if data is sufficient, or in December if additional data is needed. I plan to include all 8 researchers (minus Aubria, who is no longer a part of the team) on both manuscripts.

Budget Use

Several of the students participating in this project did so either with 494R credit (Austen, Bryant, and Juliette). A good portion of the budget went (and is still going) to the employment of the rest of the research team (Jenica, Shelby, Trent, Sam, Aubria, and Faith). In addition, we purchased three iPads in order to build video tutorials.

The remainder of the money will be used for travel to take students to the SABER meeting in 2017.