Carol Ward, Sociology

Overview of project

The MEG evaluation project began with development of an approach appropriate for the assessment of short-­‐term and long-­‐term impacts of the program. Thus, the project included planning and development of process assessment activities and assessment of program outcomes and impact. Evaluation planning also included development of a sampling plan for construction of the treatment and control groups needed for this evaluation design.

The first step for developing the sampling plan involved working with the school administration to identify students with characteristics—low income, potential first generation college student, and low academic achievement (GPAs of 2.0 and below). Collaboration to identify eligible students began before the end of the 2013-­‐14 school year. Provo High School administrators were asked to help select a minimum of 50 students eligible to participate in Teens Act. The students selected and their parents were contacted about participation in the study.

Next, a process was developed for random assignment of students to treatment and control groups. Because the Teens Act class accepts 25 students each year, 25 students were assigned to the program participant (treatment) group and 25 additional students were assigned to the comparison (control) group. Replacement of students, in the case of attrition, was done by school personnel who were asked to use the same selection criteria.

The next step involved development of appropriate data collection plans to obtain baseline data on relevant skill levels and attitudes of students as well as changes in skills, behaviors and attitudes at the end of the program year. This included refining instruments to obtain the types and quality of data needed. Data collection tasks included the following:

1. The 50 students selected and their parents were asked to complete the Teens Act pre-­‐ program survey, which included questions about student and parent attitudes towards academics and their interest in, resources for and plans for post-­‐secondary education for the student. These surveys were administered in person to minimize measurement error as well as to provide clarification of questions if necessary.
2. At the end of the school year students were asked to take a post-­‐program survey that provided data for the assessment of changes in attitudes and plans that can be compared to the pre-­‐program survey.
3. Data were obtained from the school for each student on school performance (GPAs), which were used to analyze short and long-­‐term performance changes.
4. In addition to the quantitative measurement of academic performance and achievement, qualitative data were collected through small group interviews that obtained more detailed data about students’ personal growth, attitudes and educational outlooks as well as their experiences with the program and feedback on the program’s effectiveness for meeting their individual goals.

BYU student researchers participated in all phases of the review and analyses of these data collection strategies and instruments and worked closely with faculty mentors and Teens Act staff to develop and implement an evaluation plan that addressed the outcomes and impact assessment needs of the program.

The evaluation project began in Winter 2014 following selection of 2 BYU undergraduate students who had completed social research methods courses. Three additional undergraduates continued the project when the first two graduated. Two graduate students were involved in the project as well, providing leadership, mentoring and assistance on key research tasks and reports. Methods appropriate for qualitative and quantitative data were used for the analysis of preliminary evaluation data available from Teens Act. Statistical procedures were used to analyze pre-­‐program survey data for both program participants and the control group and to analyze post-­‐program survey data for participants. Qualitative data analysis identified patterns and themes in pre-­‐program survey data and focus group interviews. Key outcomes analyzed include school performance as well as attitudes toward school and college aspirations. Analyses of the assessment data will be used for both the refinement of effective evaluation plans for the Teens Act program and for assessing current program progress. Analyses of preliminary data will be used for both the refinement of effective evaluation plans for the Teens Act program and for assessing current program progress.

How well academic objectives of project were met

This mentoring project was assessed by considering the principles of mentored research: quality of students’ experiences with project tasks such as reviews of evaluation plans and procedures, data analyses, reports and papers produced by students, the distribution of the research (presentations, publications in academic outlets, and reports to the program), and feedback from Teens Act staff. For this project, students practiced sound thinking as they used research methods, increased their skills in analytical reasoning as they analyzed data, and learned about effective communication as they presented results to in applied and academic contexts. Participating in this project allowed students to delve more deeply into the things they read about in class by actually doing the types of research professionals do in their field. This project provided extensive opportunities for students to actually use and refine the knowledge and methods they learned in their sociology methods classes in the context of an applied project.

Finally, this project encouraged service by showing how sociologists can use the tools of their discipline to serve the organizations in the communities in which they live

Specific student activities and deliverables included

• Students were involved in assessment data collection activities related to focus groups, surveys and program records.
• They learned about the appropriate use and analysis of qualitative, survey and program data for formative and summative evaluations of a program.
• The project enhanced student skills in using data management programs such as Excel, doing qualitative data analyses using programs (e.g., Dedoose), using statistical programs (e.g., SPSS or Stata), conducting appropriate literature reviews and constructing logical arguments based on the research, and writing research findings for both applied purposes and academic audiences.
• Students involved in this project produced interim reports for the Teens Act program staff that provided details of the reviews of the current program evaluation and analyses of the preliminary program data.
• Additionally, students prepared a final report of the Teens Act evaluation that provided process evaluation findings and demonstrated program outcomes and impact.
• Students used preliminary data produced during the project for the development of presentations on the program at professional meetings.
• The students who participated used their new skills as they competed for jobs and graduate programs.

Evaluation of the mentoring environment

Mentoring occurred for this project as planned. For this project, graduate student research assistants worked closely with me and Tim Heaton on each aspect of the research project and activities and helped coordinate the undergraduate researchers’ work, both on and off campus. We used project funds to hire 2 undergraduate students as researchers at the beginning of this project. When they graduated at the end of 2014, I replaced them with 2 more students in Winter 2015. When these students graduated (or went abroad for an internship) in spring 2015, I added 1 more student to help finish the evaluation activities. Three of the five undergraduates spoke Spanish, which was helpful with data collection.

The entire project team—faculty, graduate student and undergraduates—met weekly to discuss project tasks and progress and to involve the students in planning the research activities. The research team also attended meetings with the Teens Act staff to discuss the evaluation design, research tasks, preliminary analyses of program data, and project progress. The graduate student RAs coordinated with me in the development of research products–data, analyses, reports, papers, presentations, etc.–and then worked closely with the undergraduate students on research assignments related to these products.

Students involved

Undergraduates:

• Adriana Lovo (2014)
• Brooke Murphy (2014)
• Barry Jacobsen (2015)
• Megan Hilmo (2015)
• Tess Huntington (2015)

Graduate students:

• Can Cheng (2014-­15)
• Janie Demetropolous (2015)

Results/findings of the MEG project

The evaluation data analyzed indicate that the effects for the students participating in Teens Act during 2014-15 were positive overall. The most important assessment finding was that the Teens Act students achieved positive changes in their GPAs from the baseline year (2013-14) to the end of the program (2014-15). These positive changes occurred in the lower-starting GPA treatment group (i.e., students that started the school year with GPAs at 2.0 or below, the target group for Teens Act): their GPAs changed from an average of 1.674 to 1.982 (an 18.4% improvement) compared to an average GPA change of 1.442 to 1.669 (an improvement of 15.7%) for the control group. A comparison of GPAs for the lower-starting GPA treatment and control groups showed that the average GPA difference in 2014-15 was statistically significant.

Pre- and post-program student survey results show that Teens Act participants benefited in terms of education and work aspirations and more explicit plans to graduate from high school and attend college after high school graduation. Students reported that mentoring and homework help were the most effective aspects of the Teens Act program. Important changes the students reported included that they worked harder on schoolwork, completed their homework, and earned better grades. The majority of Teens Act participants reported that they expected college to have important positive effects on their work or career plans and would help to improve the quality of their lives.

Focus group interview data identified details of the Teens Act participant experiences that show how the program worked to benefit the students. Overall, the Teens Act program benefitted students by providing them with time to do homework as well as positive peer pressure to be diligent students. The relaxed environment and relatable mentors helped motivate them to keep coming to class. They also appreciated the lessons and activities about the real world and possibilities for their future. These data indicate that students typically experienced more success in school because of the structure of the Teens Act class, the activities offered and the mentoring support provided by the Teens Act class. This suggests that the unique combination of factors offered by Teens Act made a positive difference in the participants’ experiences with and attitudes toward school.

How the budget was spent

The majority of project funds (more than 90%) were spent on student wages. A small portion was used for research expenses, such as access to Dedoose software for coding qualitative data, copy costs, etc.