"Research Brief: SMART/THRIVE in the Little Rock School District"

University of Memphis. Center for Research in Educational Policy
CREP ' Center for Research itt Educational Policy Center for Research in Educational Policy Research Brief The University of Memphis 325 Browning Hall Memphis, Tennessee 38152 Toll Free
1-866-670-6147 SMART/THRIVE in the Little Rock School DistrictII II II CREP ' Caiierftr Research in Educational Policy II II Center for Research in Educational Policy The Unwersity of Memphis 325 Browning Hall Memphis, Tennessee 38152 Toll Free: 1-866-670-6147 Research Brief II SMART/THRIVE in the Little Rock School District II II II II II II January 2006 n II IB Lyle Hull Davis, Ph.D. Ying Huang Center for Research in Educational Policy n John Nunnery, Ph.D. Old Dominion University Gail Weems, Ph.D. University of Arkansas at Little RockII II RESEARCH BRIEF: SMART/THRIVE IN THE LITTLE ROCK SCHOOL DISTRICT II II II M M Background SMART and THRIVE are programs designed by two veteran Little Rock School District (LRSD) teachers. In 1999, the programs were implemented in LRSD, designed to serve at-risk students. These programs have been funded in part by the Little Rock Comprehensive Partnerships for Mathematics and Science Achievement Program and the National Science Foundation. The overall purposes of the SMART and THRIVE programs are to provide supplemental pre-algebra support to students entering Algebra I and to prepare students to meet state standards in Algebra I, respectively. The programs intentions/goals are to: a) provide a solid foundation for Algebra, b) encourage mathematical exploration
c) make mathematics fun
d) enable students to achieve academic success in mathematics
and e) create the confidence necessary to pursue higher level mathematics courses. The SMART program meets for two consecutive weeks during the summer. The THRIVE program meets every other Saturday during the Spring semester. SMART uses a co-teaching model with one teacher and one high school student mentor while THRIVE uses a co-teaching model with two certified teachers. Purpose, Plan and Participants M HI ni Purpose This report summarizes the 2004-2005 evaluation study of the SMART and THRIVE programs. The global purpose of the evaluation was to: a) examine the extent to which the programs have been effective in improving and remediating academic achievement of African-American students
b) provide cumulative evidence of SMART/THRIVE implementation practices and c) document perceptions of participants, parents and teachers as well as the level of student participation in SMART and THRIVE. Research Questions IH IH Seven research questions provided the focus of the evaluation. The primary research question focused on the extent to which SMART/THRIVE programs improved and/or remediated math achievement among African American students. Evaluation questions were
1. Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? IH 2. What is the level of participation in SMART and THRIVE by African American students? 3. What instructional strategies are used during the tutoring sessions? IH 4. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? IH 5. 6. IH What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? 7. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? 1n n Plan n The evaluation was conducted during the spring of 2005. To determine how effectively SMART and THRIVE were meeting the needs of African American students, a cadre of qualitative and quantitative measures were used. II Evaluation Measures II Five measurement strategies were used to collect the evaluation data: standardized achievement test scores, classroom observations, surveys, focus groups, and program-developed achievement benchmarks. Administration procedures and descriptions of instrumentation are provided below. IB Direct Classroom Observation Measures School Observation Measure (SOtvf'): Examines frequency of usage of 24 instructional strategies. IB Rubric for Student-Centered Activities (RSCA)-. Provides measurement of the degree of learner engagement in cooperative learning, project-based learning, higher-level questioning, IB experiential/hands-on learning, student independent inquiry/research, student discussion and students as producers of knowledge using technology. Math Addendum: Rates teachers actions in emphasizing conceptual understanding, connecting the content to daily life, and promoting students confidence in mathematics. IB Surveys Student Survey. Collects participant impressions and perceptions regarding the satisfaction, shortcomings, strengths and influence of SMART and THRIVE programs. IB Parent Survey. Collects parent perceptions and impressions regarding availability (access), influence, transportation and opinions of SMART and THRIVE. IB Algebra I Teacher Survey. Collects Algebra I teacher perceptions and impressions of influence of program on student performance as well as global awareness and opinions of SMART and THRIVE. IB SMART/THRIVE Teacher Survey. Collects SMARTfTHRIVE teacher perceptions and impressions regarding professional development, influence of technology, influence of program on student performance and overall opinions of the program. Focus Groups II Student Focus Groups: Collects program participant impressions and perceptions regarding the influence of each program on Algebra I achievement, test scores, and self-confidence about math. Also collects information about SMART and THRIVE program weaknesses, strengths and needs. IB Mentor Focus Group: Collects high school SMART mentor impressions and perceptions of SMART program strengths, weaknesses, needs, and influence on students. n student Records Attendance records from the 2004 SMART program and 2005 THRIVE program were examined in light of achievement data. Participation rates are reported as well. 2n V student Academic Performance n Math Benchmark Test (2003 - 2004): The Benchmark Test is a state-mandated criterion-referenced test aligned with state math standards and developed by Arkansas teachers and the Arkansas Department of Education. Benchmark scores are reported as Advanced, Proficient, Basic, and Below Basic. The mathematics portion of the exam is comprised of multiple-choice and open response items. fl fl End of Course Exams (EOC) 2004-2005: The Algebra I End of Course Examination is a criterion- referenced test with questions that align with the goals and subject-specific competencies described by the Arkansas Mathematics Curriculum Framework and Algebra I Course Goals. The purpose of the test is to assess student mastery of Algebra I knowledge and skills. Students take the exam at the completion of Algebra I. Scores are reported as Advanced, Proficient, Basic, and Below Basic. fl ITBS Math Subtests: The Iowa Test of Basic Skills (Math) assesses students ability to compute, deal with math concepts and solve problems dealing with numeration, graph/table interpretation, and computation. This test is norm-referenced. fl SMART and THRIVE Pre/Post Tests: The curriculum-based tests are measures of student ability to utilize algebraic concepts to solve equations, judge inequalities, evaluate and simplify basic problems, graph equations, plot data, and solve applied problems. The THRIVE program tests were comprised of content more advanced than that of SMART program tests. Pre-test problems were identical to post-test problems in both programs. fl Procedure fl fl To examine the quality of teaching and extent to which students were engaged in learning, CREP researchers conducted a series of random observations during the THRIVE program. Targeted observations were conducted in February and March of 2005 using the SOM, Math-addendum, and RSCA instruments. Observations were scheduled for random classrooms participating in the THRIVE program on three different days. fl To examine teacher, student, and parent perceptions, surveys and focus groups were designed and lead by CREP researchers. fl To examine effects of SMART and THRIVE on achievement, data from a variety of norm-referenced, criterion-referenced and informal tests were collected. District and state student performance measures were administered in late spring. Historical records of prior student Benchmark performance were collected through the district database. fl Table 1 summarizes the data collection procedure. fl fl 3d fl Table 1. Data Collection Summary fl Type of Measure Instrument Number Completed/Data Source Total Description fl fl fl fl fl fl fl fl fl fl fl fl fl Targeted Classroom Observations Surveys Focus Groups Achievement T ests Attendance Data SOM RSCA Math AdderxJum SMART/THRIVE Teacher Parent Algebra I Teacher Student student Focus Group Mentor Focus Group Math Benchmark Test 2003 - 2004 and 2004 -2005 End of Course Exams (Spring, 2005) ITBS Informal curriculumbased measures (specific to SMART/THRIVE) SMART 2004 Atterxiance THRIVE 2005 Attendance 5 19 5 10 SMART/18 THRIVE 21 SMART/35 THRIVE 25 SMART/33 THRIVE 70 SMART/142 THRIVE 21 Participants 2 Participants See Table 2 210 students (88% African American) 143 students (84% African American) *A series of observation sessions in which random THRIVE classes were observed. During a one hour period, one* RSCA form was completed every 15 minutes. Completed during Spring, 2005 THRIVE sessions. Items specific to SMART and THRIVE were included in the surveys and administered to all groups in Spring, 2005. THRIVE student surveys were distributed in class and collected by researchers. THRIVE teacher surveys were distributed during a faculty meeting and administered by researchers. THRIVE parent sun/eys were distributed by teachers via students, returned to the school and sent to researchers. Algebra I teacher surveys were distributed through the district, returned to the school and sent to researchers or sent directly to researchers. During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental pennission slips were included in the focus groups. Each focus group inten/iew lasted approximately 45 minutes. Mentors from the SMART 2004 program were contacted by researchers. Researchers used the Student Focus Group protocol modified to capture mentor perceptions to conduct an onsite, 45-minute focus group with 2 of the 7 mentors. Researchers used data collected from three major math measures designed by indeperxJent entities and distributed throughout the LRSD for achievement data analysis. Researchers used the attendance and demographic data from SMART, 2004 and THRIVE, 2005 to examine level of participation among African American students One observation session was comprised of three rather than four observation segments due to scheduling constraints. 4IB n Participants KM HR The primary context of observations, focus groups and surveys was the 2005 THRIVE program. Additional attendance data were collected from the 2004 SMART program. Survey and focus group responses concerning both programs also were collected. Both programs served students from the following middle schools: Cloverdale, Dunbar, Forest Heights, Henderson, Mablevale, Mann, Pulaski Heights, and Southwest. Students from the following high schools were also served: Central, Fair, Hall, McClellan, and Parkview. Table 1 provides a breakdown of students, mentors, teachers and parents who participated in surveys and focus groups. Table 2 provides a breakdown of the student achievement samples. n The sample sizes used for achievement comparisons varied depending upon the availability of achievement data. The following table illustrates the student populations as they relate to comparison or SMART/THRIVE programs. Due to the relative small total number of 8* grade, non-African American students participating in SMART/THRIVE (n=11), eighth grade comparisons were only made for African American students. I Table 2. Description of Subject Pool H s* grade ECX) Comparison 82 Thrive SMART SMARTS THRIVE 15 48 African- American 32 M 9^ grade EOC aggrade ITBS 581 790 NonAfrican- American 150 356 African- American 70 72 NonAfrican- American 14 12 African- American 13 17 NonAfrican- American 0 0 African- American 23 23 NonAfrican- American 3 2 n Findings M Direct Classroom Observation Results Targeted Observations II n 11 Analysis of data collected using SOM, the Math Addendum and RSCA during the five observation sessions revealed that THRIVE teachers are using a variety of instructional strategies aligned with Arkansas state-designed Student Learner Expectations (SLE). Team teaching and collaborative/cooperative learning dominated the curriculum. Additionally, notable use of higher level questioning strategies and teachers acting as coaches rather than lecturers was observed during the sessions. The Math Addendum items revealed that teachers were frequently emphasizing conceptual understanding over rote learning during the observation sessions. Teachers were also noted promoting students confidence and flexibility in doing mathematics. Overall, students in the THRIVE program had a high level of interest and were observed being engaged in instruction that emphasized practical and conceptual learning in a comfortable, student-centered setting. The RSCA results suggest that the THRIVE program integrated technology into the curriculum. Average or above average implementation (e.g., level of intensity, meaningfulness, quality) was noted for activities associated with higher level thinking strategies, teamwork and independent technology use. Survey Results I Student Survey The overall trend in response types was positive for survey items pertaining to THRIVE and SMART. Regarding THRIVE, students demonstrated high levels of agreement for items probing self-confidence, comfort level, growth related to problem solving skills, and overall satisfaction with the program. Two areas in 5 h H u n which some students showed some disagreement were related to generalization and application. Students indicated an overall satisfaction with the program on the open ended portion of the student survey, but did not feel that the program changed their study habits or helped them apply information to real life situations. Responses to items pertaining to SMART indicated agreement among students for items pertaining to preparedness, motivation, and application. Open-ended items also suggest overall satisfaction with SMART
however, a number of student suggestions were made about changing the time during which the summer program met. Parent Survey H U H Responses from the 35 parents who responded to the THRIVE parent survey suggest that they have positive feelings towards the program. Agreement or strong agreement was consistently noted across all survey items. No negative responses to the items were reported. The 21 SMART parent respondents provided similar information, particularly for those items reflecting their satisfaction with the teachers, their childrens preparedness for Algebra I. and their comfort level with the program meeting time. None of the respondents indicated disagreement or strong disagreement with any of the items. Neutral responses were noted for one item pertaining to whether SMART had improved attitudes about math. Parent perceptions measured with analysis of open-ended responses reflected an overall satisfaction with both programs. When asked specifically about transportation, the majority of parents did not cite it as an issue
however, some parents reported access to transportation being inconsistent. Algebra I Teacher Survey 11 Algebra I teachers responses to SMART items showed slightly more varied distribution across the survey scale (Strongly Agree/Agree-Neutral-Strongly Disagree/Disagree-Dont Know). Responses from the 33 Algebra I teachers responding to THRIVE items and the 25 Algebra I teachers responding to SMART items were globally positive. These teachers agreed that both programs had improved students self-confidence and positively impacted student achievement
however, they responded more neutrally to items probing the programs influence on childrens problem solving ability and influence on general Algebra I instruction. When asked specifically about whether they felt SMART and THRIVE had had an impact on achievement differences among races, a small percentage disagreed, approximately one third responded neutrally or did not know, and the remaining 48% agreed. Ml SMART/THRIVE Teacher Survey Ml Ml SMART/THRIVE Teachers responses overwhelmingly favored agreement or strong agreement with items pertaining to positive attributes among professional development, resources/materials, methodology and instructional delivery within the THRIVE and SMART programs. Two items for which some teachers responded neutrally were related to the extent to which they felt the program had improved their own ability or their students ability to use calculators. No negative responses were noted among the items
however, some teachers indicated that the limited time available for teaching was considered a shortcoming for both SMART and THRIVE. Focus Groups Ml Student Focus Group w Students perceptions of both programs suggest that there is a generally positive feeling towards SMART and THRIVE. Specifically, the consensus among the twenty-one students who participated in the focus groups was that SMART and THRIVE bolstered math skills and boosted confidence. Despite some mild distaste for required meeting times (Saturdays and during the summer), most student participants indicated that both SMART and THRIVE had reinforced Algebra I skills and provided a fun. relaxing context for learning. Many students noted that the programs had provided new strategies for problem solving and made Algebra I less daunting. There were no outwardly negative comments made. 6 I Il II Mentor Focus Group II II Two high school student mentors who had participated in SMART, 2004 provided input about the SMART program primarily. The mentors viewed SMART as a preparatory program focused on reviewing prealgebra principles. Both mentors, however, also noted that SMART provided a unique environment in which competition helped motivate students and make math more enjoyable. The mentors global perception of the program was that SMART allowed students to get a head start on Algebra 1 principles with a simultaneous boost in confidence. II Attendance II II Collectively, the 2004 SMART program served 210 students for which attendance data were available and the 2005 THRIVE program served 143 students for which data were available. Compared to the general population served by the district (69.0% African American), SMART and Thrive programs served a higher percentage of African American students (88.0% SMART
84.8% Thrive
89.6% Both). The attendance rates for SMART yielded an average rate of 97% (range: 94%-100%) and the attendance rates for Thrive yielded an average rate of 90% (range
81%-100%). Achievement I II II The results provide evidence that, in 9** grade, the SMART program, the Thrive program, or the combination of the two programs were associated with substantial improvements in the achievement on the Algebra I EOC exam. The strongest evidence was noted among students who participated in both programs. Eighth grade students participating in any combination of the programs were less likely than their non-parficipant peers to perform at basic or below basic proficiency levels on the Algebra I EOC exam. More specifically, African American students in both eighth and ninth grade who were enrolled in any of the three program configurations were more likely to perform at proficient or advanced levels of proficiency than the nonparticipant comparison group. Gains were also noted on the program-specific assessments for the SMART and Thrive programs. No significant differences between comparison and SMART/THRIVE students nor racial differences were noted among scores from the ITBS. n Conclusions n Each of the major research questions will provide a framework around which the conclusions for the present study will be structured. II Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? n Student-level achievement analyses show that participation in SMART. Thrive and a combination of programs is associated with improvements in achievement when compared to non-participating students. Specifically, our interpretation of these outcomes focuses on the extent to which these differences in achievement relate to growth among African American students. II n Findings suggest that participation in any of the program configurations was associated with a major reduction in the achievement gap between African American students and students of other races, from a one standard deviation deficit on 2003-04 Benchmark scores to virtually no difference on 2004-5 Algebra I EOC scores. For 2004-05 S' graders, the results suggest a positive effect of the programAfrican American students attending any program configuration were more likely to achieve at Proficient or Advanced levels than peers who did not attend. Although results were favorable for African American students enrolled in both programs, no statistically significant effects were noted from ITBS scores for race or program, demonstrating that although SMART and Thrive have a direct and significant impact on curriculum-based performance measures, they have a lesser influence on standardized, norm-referenced math achievement performance. In light of the current goal in increasing achievement among African American students, particularly in the domain of mathematics, the results from EOC and Benchmark exams are highly important. These results 7 are encouraging as implementation of the SMART and THRIVE programs seems to have had a positive impact on African American student achievement. What is the level of participation in SMART and THRIVE by African American students? Il a Overall, the level of participation for both programs was high. The majority of students enrolled in SMART, THRIVE or both programs missed no sessions, yielding average attendance rates of 90% or above. Additionally, data from classroom observations suggested a high degree of student engagement and participation across classrooms within the THRIVE program. Given the demographic characteristics of the participants, the significant quantitative relationship between participation and student achievement gains, and the high participation rates among participants, the results provide evidence that SMART and THRIVE have a positive impact on participation among African American students and that participation has had significant influence on achievement. II What instructional strategies are used during the tutoring sessions? If During classroom visits, the majority of observed instructional strategies were couched in a collaborative teaching model focused on higher level thinking skills and student grouping. Students were observed utilizing technology throughout their lessons and were highly engaged during observation sessions. Teachers frequently posed hypothetical questions to groups to accommodate use of higher-level thinking skills and generate more varied student responses. Even when students were engaged in independent seatwork, which was rare, teachers were observed circulating among various student groups, requesting more in-depth questioning strategies in their problem solving approaches. Additionally, students were frequently observed encouraging fellow students in preparation for competition while working in collaborative groups. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? M U U SMARTfTHRIVE teachers responded positively to survey items for both programs. Teachers generally believed that the core components of both programs supported student learning, particularly among students who were struggling to perform well in Algebra I classes. Additionally, they reported feeling comfortable with expectations and resources, noting that having opportunities to collaborate with other teachers and utilize competition to motivate students were essential to success. Teachers cited time constraints as a recognized weakness across both programs. The general consensus was that they wanted more time to develop deeper skills among students and to reinforce concepts taught in Algebra I classrooms. None of the respondents expressed feelings of dissatisfaction with either program and demonstrated the highest percentage of positive responses when compared to other groups (students, parents, Algebra I teachers). What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? ai Algebra I teachers were generally positive in their reaction to both SMART and THRIVE programs, representative of an overall feeling that SMART and THRIVE facilitated student learning and generalized to the K Algebra 1 classroom. Algebra I teachers noted observed improvements among students participating in SMART/THRIVE and globally agreed that the programs effectively enabled students to use technology in the Algebra I classrooms. A number of Algebra I teachers indicated that both programs needed to reach a larger and broader group of students. There were no indications that Algebra I teachers disliked SMART or THRIVE
however, there was some evidence that some respondents felt that the programs did more to supplement what was being taught in Algebra I classrooms than cultivate more meaningful understanding of algebraic concepts. I 8 h sa am What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? n Students were somewhat mixed in their reactions to SMART and THRIVE, although they were generally positive. While most of the students believed that SMART and/or THRIVE helped them with Algebra I class, others indicated mild dislike for time spent doing academics during summer and/or Saturdays. Students perceived the programs as motivating, fun, and globally helpful. It was rewarding to note that despite some dislike of the programs scheduled meeting times, most students indicated a desire to continue with the programs modifying very little about the actual instructional delivery. These comments were corroborated by comments students made during focus group sessions. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? n Parents' reactions to both programs were globally positive. Specifically, parents commented that the motivational, fun nature of the programs was fostering an interest in math among students. H M M The change that parents requested most was that the program be expanded both in breath/scope and duration. None of the parents cited negative features
however, some parents requested more rigor within the curriculum. When asked specifically about transportation, a concern of SMART/THRIVE program heads, the majority (84%) of parents indicated that it was not a problem. Of the parents who did indicate that transportation was a problem, inconsistency of service was cited as the primary issue. The majority of parents felt comfortable sending their children to the programs during the summer and/or on Saturdays, particularly because transportation on those days/during that time did not conflict with work or other activities that typically dominated school days. II Compliance Remedy Questions Recommendations M n A series of recommendations have been designed to provide guidance for future implementation of SMART and THRIVE programs. The recommendations primarily focus upon two broad areas in which the Little Rock School District could take additional action to ensure proper implementation of SMART/THRIVE as they relate to academic improvement among African American children: 1. Expansion 2. Accountability n Program Expansion II M A recurring theme among surveys collected during this evaluation related to a general desire to expand both programs. Specifically, teachers and parents generally called to extend the duration and frequency of the programs as well as expand the programs to meet the needs of larger and broader groups of students. Before expanding the frequency and duration of each program, the district would be well-served to explore effective ways to recruit students and teachers to expand the programs. An expanded program would have to include more SMART/THRIVE teachers directly proportionate to the number of students added to the program. The success of the program is in part dependent upon the amount of flexibility afforded to each teaching team as well as the ratio of teachers to students. If more students are added without adding more teachers, the design of the SMART/THRIVE model will be compromised. u Given the unique cooperative teaching model used in the program, pre-program professional development and staff training of new SMART/THRIVE teachers would be paramount to successful expansion. Specific focus on technology use and development of higher level 9 b a If If thinking skills would help further develop the current model. Additionally, the level of problem solving and critical thinking used in SMART/THRIVE classrooms would require new SMART/THRIVE teachers to receive training aimed at fostering these skills in the context of Algebra. Current and new SMART/THRIVE teachers would benefit from professional development aimed at applying algebraic concepts in authentic contexts (i.e.. real world application) and/or simulations. If If If Currently, THRIVE meets every other week during the Spring semester. The district should consider expanding the number of participating students and teachers by using alternating Saturdays to accommodate the same numbers of students per THRIVE session rather than simply making each session bigger. For example, if the district were to add 100 additional THRIVE students and proportional numbers of teachers, they would be less likely to disrupt the current model by creating two groups of THRIVE students meeting on different Saturdays during each month. The numbers, therefore would remain more manageable and the staff more effective in delivering instruction and fostering learning rather than discipline during competition and class time. SMART could potentially integrate two, two-week summer sessions to serve more students. al If If In light of budgetary constraints, the prospects of expanding SMART/THRIVE programs could be limited potentially
however, LRSD should examine alternative funding opportunities to bolster prospects for expansion. The following list, although not exhaustive, provides examples of alternative funding sources that potentially match the goals and purposes of SMART/THRIVE: American Society of Engineering Education, The National Science Foundation, Oak Ridge Institute for Science and Education, the Spencer Foundation and the Society for Industrial and Applied Mathematics. Some grant opportunities could foster potential secondary opportunities for qualified African American students to participate in further research and competition. The purpose in seeking these grants is bifurcate in nature: to obtain funding for expansion
to explore further opportunities aimed at enriching mathematics and fostering motivation among African American children If If Transportation costs and availability must be considered if the programs are expanded. Although evidence of negative experiences with transportation was minimal among respondents within the 2004-2005 subject pool, the district will need to determine the extent to which access to transportation limits participation. Appropriate measures should be taken during recruitment to disclose availability of transportation and ensure that availability is viable. II Accountability II Attendance rates among SMART and THRIVE participants were extraordinarily high
however, for the purposes of longitudinal studies, it would be instrumental to follow participants through graduation to examine longitudinal effects of the programs on math achievement. Future studies might examine indepth analyses of participants to determine what factors impacted their achievement significantly. n Additional data such as graduation rates, enrollment in advanced math courses and school attendance should be examined longitudinally once students complete SMART and THRIVE programs. A Efforts to recruit former SMART/THRIVE students as high school mentors should be made to foster continuity within the program and make students gainfully aware of future effects. n Future comparisons of SMART/THRIVE students should continue to disaggregate assessment results by race and ethnicity to effectively examine the progress of African American students in SMART/THRIVE relative to their peers. A 10 I 0 a II Given the limited research on proven programs, intensification on research of this program through long-term studies is recommended. If positive. SMART/THRIVE may have potential to serve as a leader in developing effective Algebra programs for African American children. Expectations of Program Modifications II II II Findings from the recent evaluation were globally positive. In light of these findings, SMART/THRIVE has value in raising student achievement among African American students in LRSD. With regard to change in the instructional design of the programs, little guidance is provided and change is not recommended. However, future modifications would include programmatic expansion to serve a larger, broader group of students. Offerings to African American students should remain paramount. Additionally, given the limited amount of research on successful programs with at-risk populations across the country, further, more intensive research of SMART/THRIVE programs would provide the district not only with a model that can be easily replicated, but also information about factors that set SMART/THRIVE apart from classroom Algebra and pre- Algebra classes. Finally, the district will need to examine the sustained achievement, retention and Il coursework decisions of SMART/THRIVE participants to determine whether the growth noted among participants during the program is sustainable. I II n II II I II II n I 11 a 11 M CREP ' CtHter for Reteareh in Educational foUcy n Center for Research in Educational Policy Evaluation of SMART/THRIVE a The University of Memphis 325 Browning Hall Memphis, Tennessee 38152 Toll Free: 1-86&67&6147 In the Little Rock School District a TECHNICAL REPORT a a a CREP Ce/iU/ for Research in E^ueaiional Policy Center for Research in Educational Policy The University of Memphis 325 Browning Hall Memphis, Tennessee 38152 Toll Free
1-866-670-6147 Evaluation of SMART/THRIVE In the Little Rock School District TECHNICAL REPORT January 2006 Lyle Hull Davis, Ph.D. Ying Huang Center for Research in Educational Policy John Nunnery, Ph.D. Old Dominion University Gail Weems, Ph.D. University of Arkansas at Little Rockp EXECUTIVE SUMMARY Background SMART and THRIVE are programs designed by two, veteran LRSD teachers. In 1999, the programs were implemented in LRSD, designed to serve at-risk students. These programs have been funded in part by the Little Rock Comprehensive Partnerships for Mathematics and Science Achievement Program and the National Science Foundation. The overall purposes of the SMART and THRIVE programs are to provide supplemental pre-algebra support to students entering Algebra I and to prepare students to meet state standards in Algebra I, respectively. The programs intentions/goals are to: a) provide a solid foundation for Algebra, b) encourage exploration
c) make mathematics fun
d) enable students to achieve academic success in mathematics
and e) create the confidence necessary to pursue higher level courses. The SMART program meets for two consecutive weeks during the summer. The THRIVE program meets every other Saturday during the Spring semester. SMART uses a co-teaching model with one teacher and one high school student mentor while THRIVE uses a co-teaching model with two certified teachers. Purpose, Plan and Participants Purpose This report summarizes the 2004-2005 evaluation study of the SMART and THRIVE programs. The global purpose of the evaluation was to: a) provide cumulative evidence of SMART/THRIVE implementation practices
b) examine the extent to which the programs have been effective in improving and remediating academic achievement of African-American students
and c) explore perceptions of participants, parents and teachers as well as level of student participation in SMART and THRIVE. Research Questions Seven research questions provided the focus of the evaluation. The primary research question focused on the extent to w4iich SMART/THRIVE programs improved and/or remediated math achievement among African American students. Evaluation questions were: 1. 2. 3. 4. 5. 6. 7. Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? What is the level of participation in SMART and THRIVE by African American students? What instructional strategies are used during the tutoring sessions? What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? 1Plan The evaluation was conducted during the spring of 2005. To determine how effectively SMART and THRIVE were effectively meeting the needs of African American students, a cadre of qualitative and quantitative measures were used. Evaluation Measures Six measurement strategies were used to collect the evaluation data: standardized achievement test scores, classroom observations, surveys, focus groups, and program-developed achievement benchmarks. Administration procedures and descriptions of instrumentation are provided below. Direct Classroom Observation Measures School Observation Measure (SOM^)'. Examines frequency of usage of 26 instructional strategies. Rubric for Student-Centered Activities (RSCA)'. Provides measurement of the degree of learner engagement in cooperative learning, project-based learning, higher-level questioning. experiential/hands-on learning, student independent inquiry/research, student discussion and students as producers of knowledge. Math Addendum'. Rates teachers actions in emphasizing conceptual understanding, connecting the content to daily life, and promoting students confidence in mathematics. Surveys Student Survey. Collects participant impressions and perceptions regarding the satisfaction, shortcomings, strengths and influence of SMART and THRIVE programs. Parent Survey. Collects parent perceptions and impressions regarding availability (access), influence, transportation and opinions of SMART and THRIVE. Algebra I Teacher Survey. Collects Algebra I teacher perceptions and impressions of influence of program on student performance as well as global awareness and opinions of SMART and THRIVE. SMART/THRIVE Teacher Survey. Collects THRIVE teacher perceptions and impressions regarding professional development, influence of technology, influence of program on student performance and overall opinions of the program. Focus Groups Student Focus Groups: Collects program participant impressions and perceptions regarding the influence of each program on Algebra I achievement, test scores, and self-confidence about math. Also collects information about SMART and THRIVE program weaknesses, strengths and needs. Mentor Focus Group: Collects high school SMART mentor impressions and perceptions of SMART program strengths, weaknesses, needs, and influence on students. Student Records Attendance records from the 2004 SMART program and 2005 THRIVE program were examined in light of achievement data. Participation rates are reported as well. Student Academic Performance Math Benchmark Test (2003 - 2004) scores were analyzed. The Benchmark Test is a state- mandated criterion-referenced test aligned with state math standards and developed by Arkansas teachers and the Arkansas Department of Education. Benchmark scores are reported as Advanced, Proficient, Basic, and Below Basic. The mathematics portion of the exam is comprised of multiplechoice and open response items. 2if End of Course Exams (EOC) 2004-2005 scores were analyzed. The Algebra I End of Course Examination is a criterion-referenced test with questions that align with the goals and subject-specific competencies described by the Arkansas Mathematics Curriculum Framework and Algebra I Course Goals. The purpose of the test is to assess student mastery of Algebra I knowledge and skills. Students take the exam at the completion of Algebra I. Scores are reported as Advanced, Proficient, Basic, and Below Basic. ITBS Math Subtests scores were analyzed. The Iowa Test of Basic Skills (Math) assesses students ability to compute, deal with math concepts and solve problems dealing with numeration, graph/table interpretation, and computation. This test is norm-referenced. SMART and THRIVE Pre/Post Tests: The curriculum-based tests are measures of student ability to utilize algebraic concepts to solve equations, judge inequalities, evaluate and simplify basic problems, graph equations, plot data, and solve applied problems. The THRIVE program tests were comprised of content more advanced than that of SMART program tests. Pre-test problems were identical to post-test problems in both programs. Procedure To examine the quality of teaching and extent to which students were engaged in learning, CREP researchers conducted a series of random observations during the THRIVE program. Targeted observations were conducted in February and March of 2005 using the SOM, Math-addendum, and RSCA instruments. Observations were scheduled for random classrooms participating in the THRIVE program on three different days. To examine teacher, student, and parent perceptions, surveys and focus groups were designed and lead by CREP researchers. To examine effects of SMART and THRIVE on achievement, data from a variety of normed, criterion-referenced and informal tests were collected. District and state student performance measures were administered in late spring. Historical records of prior student Benchmark performance were collected through the district database. Table 1 summarizes the data collection procedure. i 3 I 0 a Table 1. Data Collection Summary I Type of Measure Targeted Classroom Observations SOM Instrument TC---------------------- Number Completed/Data Source Total fl RSCA Math Addendum 5 19 5 Description A series of observation sessions in which random THRIVE classes were observed. During a one hour period, four* note forms were completed every 15 minutes. Completed during Spring, 2005 THRIVE sessions. Surveys SMART/THRIVE Teacher Parent Algebra I Teacher Student 10 SMART/18 THRIVE 21 SMART/35 THRIVE 25 SMART/33 THRIVE 70 SMART/142 THRIVE Items specific to SMART and THRIVE were included in the surveys and administered to all groups in Spring, 2005. THRIVE student surveys were distributed in class and collected by researchers. THRIVE teacher surveys were distributed during a faculty meeting and administered by researchers THRIVE parent surveys were distributed by teachers via students, returned to the school and sent to researchers Algebra I teacher surveys were distributed through the district, returned to the school and sent to researchers or sent directly to researchers. Focus Groups Student Focus Group Mentor Focus Group 21 Participants 2 Participants During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups. Each focus group interview lasted approximately 45 minutes. Mentors from the SMART 2004 program were contacted by researchers. Researchers used the Student Focus Group protocol to conduct an on-site, 45-minute focus group with 2 of the 7 mentors. Achievem ent T ests Math Benchmark Test 2003-2004 and 2004 -2005 End of Course Exams ITBS See Table 2 Researchers used data collected from three major math measures designed by independent entities and distribute throughout the LRSD for achievement data analysis. Exams were taken during the Spring of 2005. Attendance Data SMART 2004 Attendance THRIVE 2005 Attendance 210 students (88% African American) 143 students (84% African American) Researchers used the attendance and demographic data from SMART, 2004 and THRIVE, 2005 to examine level of participation among African American students i H *One observation session was comprised of three rather than four observation segments due to scheduiing constraints. Participants The primary context of observations, focus groups and surveys was the 2005 THRIVE program. Additional attendance data were collected from the 2004 SMART program. Survey and focus group responses concerning both programs also were collected. Both programs served students from the following middle schools: Cloverdale, Dunbar, Forest Heights, Henderson, Mablevale, Mann, Pulaski Heights, and Southwest. Students from the following high schools were also served: Central, Fair, Hall, McClellan, and Parkview. Table 1 provides a breakdown of students, mentors, teachers and parents who participated surveys and focus groups. Table 2 provides a breakdown of the student achievement samples. in The sample sizes used for achievement comparisons varied depending upon the availability achievement data. The following table illustrates the student populations as they relate to comparison of or 4- I SMART/THRIVE programs. Due to the relative small total number of 8* grade, non-African American students participating in SMART/THRIVE (n=11), eighth grade comparisons were only made for African American students. Table 2. Student Populations 8'*' grade EOC 9" grade EOC S'" grade ITBS Findings Comparison 82 African- American 581 790 NonAfrican- American 150 356 Thrive SMART SMART & THRIVE 15 48 32 African- American 70 72 NonAfrican- American 14 12 African- American 13 17 NonAfrican- American 0 0 African- American 23 23 NonAfrican- American 3 2 Direct Classroom Observation Results Targeted Observations Analysis of data collected using SOM, the Math Addendum and RSCA during the five observation sessions revealed that THRIVE teachers are using a variety of instructional strategies aligned with Arkansas state-designed Student Learner Expectations (SLE). The most striking result was the extent to which team teaching and collaborative/cooperative learning dominated the curriculum. Additionally, notable use of higher level questioning strategies and teachers acting as coaches rather than lecturers was noted among observed sessions. The Math Addendum items revealed that teachers were frequently emphasizing conceptual understanding over rote learning during the observation sessions. Teachers were also noted promoting students confidence and flexibility in doing mathematics. Overall, students in the THRIVE program had a high level of interest and were observed being engaged in instruction that emphasized practical and conceptual learning in a comfortable, student-centered setting. The RSCA results suggest that the THRIVE program integrated technology into the curriculum. Average or above average implementation (e.g., level of intensity, meaningfulness, quality) was noted for activities associated with higher level thinking strategies, teamwork and independent technology use. Survey Results Student Survey The overall trend in response types was positive for survey items pertaining to THRIVE and SMART. Regarding THRIVE, students demonstrated high levels of agreement for items probing setf-confidence, comfort level, growth among problem solving skills, and overall satisfaction with the program. Two areas in which some students showed some disagreement were related to generalization and application. Students indicated an overall satisfaction with the program on the open ended portion of the student survey, but did not feel that the program changed their study habits or helped them apply information to real life situations. Responses to items pertaining to SMART indicated agreement among students for items pertaining to preparedness, motivation, and application. Open-ended items also suggest overall satisfaction with SMART
however, a number of student suggestions were made about changing the time during which the summer program met. Parent Survey Responses from the 35 parents who responded to the THRIVE parent survey suggest that they have positive feelings towards the program. Agreement or strong agreement was consistently noted across ail survey items. No negative responses to the items were reported. The 21 parent respondents provided 5 I at similar information, particularly for those items reflecting their satisfaction with the teachers, their childrens preparedness for Algebra I, and their comfort level with the program meeting time. None of the respondents indicated disagreement or strong disagreement with any of the items. Neutral responses were noted for one item pertaining to whether SMART had improved attitudes about math. Parent perceptions measured with analysis of open-ended responses reflected an overall satisfaction with both programs. When asked specifically about transportation, the majority of parents did not cite it as an issue
however, some parents reported access to transportation being inconsistent. Algebra I Teacher Survey I n I Algebra I teachers responses to SMART items showed slightly more varied distribution across the survey scale (Strongly Agree/Agree-Neutral-Strongly Disagree/Disagree-Dont Know). Responses from the 33 Algebra I teachers responding to THRIVE items and the 25 Algebra I teachers responding to SMART items were positive. These teachers agreed that both programs had improved students self-confidence and positively impacted student achievement
however, they responded more neutrally to items probing the programs influence on childrens problem solving ability and influence on general Algebra I instruction. When asked specifically about whether they felt SMART and THRIVE had had an impact on achievement differences among races, a small percentage disagreed, approximately one third responded neutrally or did not know, and the remaining 48% agreed. SMART/THRIVE Teacher Survey SMART/THRIVE Teachers responses overwhelmingly favored agreement or strong agreement with items pertaining to positive attributes among professional development, resources/materials, methodology and instructional delivery within the THRIVE and SMART programs. Two items for which some teachers responded neutrally were related to the extent to which they felt the program had improved their own ability or their students ability to use calculators. No negative responses were noted among the items
however, some teachers indicated that the limited time available for teaching was considered a shortcoming for both SMART and THRIVE. Focus Groups student Focus Group Students perceptions of both programs suggest that there is a generally positive feeling towards SMART and THRIVE. Specifically, the consensus among the twenty-one students who participated in the focus groups was that SMART and THRIVE bolstered math skills and boosted confidence. Despite some mild distaste for required meeting times (Saturdays and during the summer), most student participants indicated that both SMART and THRIVE had reinforced Algebra I skills and provided a fun, relaxing context for learning. Many students noted that the programs had provided new strategies for problem solving and made Algebra I less daunting. There were no outwardly negative comments made. Mentor Focus Group Two high school student mentors who had participated in SMART, 2004 provided input about the SMART program primarily. The mentors viewed SMART as a preparatory program focused on reviewing pre-algebra principles. Both mentors, however, also noted that SMART provided a unique environment in which competition helped motivate students and make math more enjoyable. The mentors global perception of the program was that SMART allowed students to get a head start on Algebra I principals with a simultaneous boost in confidence. Attendance Collectively, the 2004 SMART program served 210 students for which attendance data was available and the 2005 THRIVE program served 143 students for which data was available. Compared to the general population served by the district (69.0% African American), SMART and Thrive programs served a higher 6percentage of African American students (88.0% SMART
84.8% Thrive
89.6% Both). The attendance rates for SMART yielded an average rate of 97% (range: 94%-100%) and the attendance rates for Thrive yielded an average rate of 90% (range: 81%-100%). Achievement The results provide evidence that, in 9* grade, the SMART program, the Thrive program, or the combination of the two programs were associated with substantial improvements in the achievement on the Algebra I EOC exam. The strongest evidence was noted among students who participated in both programs. Eighth grade students participating in any combination of the programs were less likely than their nonparticipant peers to perform at basic or below basic proficiency levels on the Algebra I EOC exam. More specifically. African American students in both eighth and ninth grade who were enrolled in any of the three program configurations were more likely to perform at proficient or advanced levels of proficiency than the non-participant comparison group. Gains were also noted on the program-specific assessments for the SMART and Thrive programs. No significant differences between comparison and SMART/THRIVE students nor racial differences were noted among scores from the ITBS. I Conclusions Each of the major research questions will provide a framework around which the conclusions for the present study will be structured. Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? Student-level achievement analyses show that participation in SMART. Thrive and a combination of programs is associated with comparative improvements in achievement when compared to non-participating students. Specifically, our interpretation of these outcomes focuses on the extent to which these differences in achievement relate to growth among African American students. II Findings suggest that participation in any of the program configurations was associated with a major reduction in the achievement gap between African American students and students of other races, from a one standard deviation deficit on 2003-04 Benchmark scores to virtually no difference on 2004-5 Algebra I EOC scores. For 2004-05 8** graders, the results suggest a positive effect of the program^African American students attending any program configuration were more likely to achieve at Proficient or Advanced levels than peers who did not attend. Since 8* graders do not comprise the majority of SMART/THRIVE participants, further analysis of these students sustained achievement would be interesting. Although results were favorable for African American students enrolled in both programs, no statistically significant effects were noted from ITBS scores for race or program, demonstrating that although SMART and Thrive have a direct and significant impact on curriculum-based performance measures, they have a lesser influence on standardized, norm-referenced math achievement performance. fl In light of the current goal in increasing achievement among African American students, particularly in the domain of mathematics, the results from EOC and Benchmark exams are highly significant. These results are encouraging as implementation of the SMART and Thrive programs seems to have had a positive impact on African American student achievement. What is the level of participation in SMART and THRIVE by African American students? Overall, the level of participation for both programs was high. The majority of students enrolled in SMART. Thrive or both programs missed no sessions, yielding average attendance rates of 90% or above. Additionally, data from classroom observations suggested a high degree of student engagement and participation across classrooms within the Thrive program. Given the demographic characteristics of the participants, the significant quantitative relationship between participation and student achievement gains, and the high participation rates among participants, the results provide evidence that SMART and Thrive 79 have a positive impact on participation among African American students and that participation has had significant influence on achievement. What instructional strategies are used during the tutoring sessions? n During classroom visits, the majority of observed instructional strategies were couched in a collaborative teaching model focused on higher level thinking skills and student grouping. Students were observed utilizing technology throughout their lessons and were highly engaged during observation sessions. Teachers frequently posed hypothetical questions to groups to accommodate use of higher-level thinking skills and generate more varied student responses. Even when students were engaged in independent seatwork, which was rare, teachers were observed circulating among various student groups, requesting more in-depth questioning strategies in their problem solving approaches. Additionally, students were frequently cited encouraging fellow students in preparation for competition while working in collaborative groups. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? SMART/THRIVE teachers responded positively to survey items for both programs. Teachers generally believed that the core components of both programs supported student learning, particularly among students who were struggling to perform well in Algebra I classes. Additionally, they reported feeling comfortable with expectations and resources, noting that having opportunities to collaborate with other teachers and utilize competition to motivate students were essential to success. Teachers cited time constraints as a recognized weakness across both programs. The general consensus was that they wanted more time to develop deeper skills among students and to reinforce concepts taught in Algebra I classrooms. None of the respondents expressed feelings of dissatisfaction with either program and demonstrated the highest percentage of positive responses when compared to other groups (students, parents. Algebra I teachers). What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? Algebra I teachers were generally positive in their reaction to both SMART and THRIVE programs, representative of an overall feeling that SMART and THRIVE facilitated student learning and generalized to the Algebra I classroom. Algebra I teachers noted observed improvements among students participating in SMART/THRIVE and globally agreed that the programs effectively enabled students to use technology in the Algebra 1 classrooms. A number of Algebra I teachers indicated that both programs needed to reach a larger and broader group of students. There were no indications that Algebra I teachers disliked SMART or THRIVE
however, there was some evidence that some respondents felt that the programs did more to supplement what was being taught in Algebra I classrooms than cultivate more meaningful understanding of algebraic concepts. What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? Students were somewhat mixed in their reactions to SMART and THRIVE, although they were generally positive. While most of the students believed that SMART and/or THRIVE helped them with Algebra I class, others indicated mild dislike for time spent doing academics during summer and/or Saturdays. Students perceived the programs as motivating, fun, and globally helpful. It was rewarding to note that despite some dislike of the programs scheduled meeting times, most students indicated a desire to continue with the programs modifying very little about the actual instructional delivery. These comments were corroborated by comments students made during focus group sessions. 8What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? Parents reactions to both programs were globally positive. Specifically, parents commented that the motivational, fun nature of the programs was fostering an interest in math among students. The change that parents requested most was that the program be expanded both in breath/scope and duration. None of the parents cited negative features
however, some parents requested more rigor within the curriculum. When asked specifically about transportation, a concern of SMART/THRIVE program heads, the majority (84%) of parents indicated that it was not a problem. Of the parents who did indicate that transportation was a problem, inconsistency of service was cited as the primary issue. The majority of parents felt comfortable sending their children to the programs during the summer and/or on Saturdays, particularly because transportation on those days/during that time did not conflict with work or other activities that typically dominated school days. Compliance Remedy Questions Recommendations A series of recommendations have been designed to provide guidance for future implementation of SMART and THRIVE programs. The recommendations primarily focus upon two broad areas in which the Little Rock School District could take additional action to ensure proper implementation of SMART/THRIVE as they relate to academic improvement among African American children: 1. Expansion 2. Accountability Program Expansion A recurring theme among surveys collected during this evaluation related to a general desire to expand both programs. Specifically, teachers and parents generally called to extend the duration and frequency of the programs as well as expand the programs to meet the needs of larger and broader groups of students. Before expanding the frequency and duration of each program, the district would be well- served to explore effective ways to recruit students and teachers to expand the programs. An expanded program would have to include more SMART/THRIVE teachers directly proportionate to the number of students added to the program. The success of the program is in part dependent upon the amount of flexibility afforded to each teaching team as well as the ratio of teachers to students. If more students are added without adding more teachers, the design of the SMART/THRIVE model will be compromised. Given the unique cooperative teaching model used in the program, pre-program professional development and staff training of new SMART/THRIVE teachers would be paramount to successful expansion. Specific focus on technology use and development of higher level thinking skills would help further develop the current model. Additionally, the level of problem solving and critical thinking used in SMART/THRIVE classrooms would require new SMART/THRIVE teachers to receive training aimed at fostering these skills in the context of Algebra. Current and new SMART/THRIVE teachers would benefit from professional development aimed at applying algebraic concepts in authentic contexts (i.e., real world" application) and/or simulations. Currently, THRIVE meets every other week during Spring semester. The district would be well- advised to expand the number of participating students and teachers by using alternating Saturdays to accommodate the same numbers of students per THRIVE session rather than simply making each session bigger. For example, if the district were to add 100 additional 9THRIVE students and proportional numbers of teachers, they would be less likely to disrupt the current model by creating two groups of THRIVE students meeting on different Saturdays during each month. The numbers, therefore would remain more manageable and the staff more effective in delivering instruction and fostering learning rather than discipline during competition and class time. SMART could potentially integrate two, two-week summer sessions to serve more students. In light of budgetary constraints, the prospects of expanding SMART/THRIVE programs could be limited potentially
however, LRSD should examine alternative funding opportunities to bolster prospects for expansion. The following list, although not exhaustive, provides examples of alternative funding sources that potentially match the goals and purposes of SMART/THRIVE: American Society of Engineering Education. The National Science Foundation. Oak Ridge Institute for Science and Education, the Spencer Foundation and the Society for Industrial and Applied Mathematics. Some grant opportunities could foster potential secondary opportunities for qualified African American students to participant in further research and competition. The purpose in seeking these grants is bifurcate in nature: to obtain funding for expansion
to explore further opportunities aimed at enriching mathematics and fostering motivation among African American children Transportation costs and availability must be considered if the programs are expanded. Although evidence of negative experiences with transportation was minimal among respondents within the 2004-2005 subject pool, the district will need to determine the extent to which access to transportation limits participation. Appropriate measures should be taken during recruitment to disclose availability of transportation and ensure that availability is viable. Accountability Attendance rates among SMART and THRIVE participants were extraordinarily high
however, for the purposes of tracking, it would be instrumental to follow past participants to examine longitudinal effects of the programs on math achievement. Future studies might examine in-depth analyses of participants to determine what factors impacted their achievement significantly. Additional data such as graduation rates, enrollment in advanced math courses and school attendance should be examined longitudinally once students complete SMART and THRIVE programs. Efforts to recruit former SMARTfTHRIVE students as high school mentors should be made to foster continuity within the program and make students gainfully aware of future effects. Future comparisons of SMART/THRIVE students should continue to disaggregate assessment results by race and ethnicity to effectively examine the progress of African American students in SMART/THRIVE relative to their peers. Given the limited research on proven programs, instensification on research of this program through long-term studies is recommended. If positive, SMART/THRIVE may have potential to serve as a leader in developing effective Algebra programs for African American children in failing schools. Expectations of Program Modifications Findings from the recent evaluation were globally positive. In light of these findings. SMART/THRIVE has value in raising student achievement among African American students in LRSD. With regard to change in the instructional design of the programs, little guidance is provided and change is not recommended. However, future modifications would include programmatic expansion to serve a larger, broader group of students. Offerings to African American students should remain paramount. Additionally, given the limited amount of research on successful programs in failing schools across the country, further, more intensive research of SMART/THRIVE programs would provide the district not only with a model that can be easily 10replicated, but also information about factors that set SMART/THRIVE apart from classroom Algebra and preAlgebra classes. Finally, the district will need to examine the sustained achievement, retention and coursework decisions of SMART/THRIVE participants to determine whether the growth noted among participants during the program is sustainable. 11Evaluation of SMART/THRIVE In the Little Rock School District Research Report: 2004-2005 INTRODUCTION The Little Rock School District contracted with the Center for Research in Educational Policy (CREP) 'J at The University of Memphis to collect formative evaluation data in schools implementing SMART/THRIVE. This report summarizes the 2004-2005 evaluation study of the SMART and THRIVE programs. The global purpose of the evaluation was to examine the extent to which the SMART/THRIVE programs have been effective in improving and remediating academic achievement of African American students. Additionally, the evaluation explores perceptions of participants, parents and teachers as well as level of student participation in SMART and THRIVE. The overall purposes of the SMART and THRIVE programs are to provide supplemental pre-algebra support to students entering Algebra I and to prepare students to meet state standards in Algebra I. It was designed as an intervention for rising and current S- and 9*'-grade students who are entering or will be enrolled in Algebra I. Applications for SMART are disseminated through the Sth grade teachers for any rising 9th grade student who has not taken Algebra I. Eligible rising 9th grade students who are registered to attend high school in the LRSD and will take algebra are selected on a first come-first served basis. Seventh-grade math teachers are asked to recommend students, particularly African American or Hispanic students, who have the potential to be successful in Algebra I in the Sth grade. Students who participated in SMART are invited to participate in THRIVE. After a designated enrollment period, remaining seats are opened to any student taking Algebra I. Applications for these remaining seats are disseminated by the Algebra 1 teachers. The programs intentions/goals are to: Provide a solid foundation for algebra, encourage exploration Make mathematics fun Enable students to achieve academic success in mathematics Create the confidence necessary to pursue higher-level math courses The goals encompass two components: pre-algebra instruction for two weeks during the summer (SMART Program) and 10 Saturdays across the school year (THRIVE Program). Instruction in both 12programs utilizes small class sizes (fewer than 18) with two teachers per class. Each program currently (2004-2005) engages approximately 10 percent of the total African American student population enrolled in Algebra I classes. Various local grants have funded both programs since 1999. Summer Mathematics Advanced Readiness Training (SMART) provides opportunities for students to gain the knowledge, skills, and confidence needed for success in Algebra 1. SMART is a two-week half-day summer institute for eligible rising S' and 9* grade students who will be enrolled in Algebra 1. SMART serves approximately 200 students each year. Each SMART classroom consists of a qualified mathematics teacher. one high school mentor who demonstrates outstanding mathematical skills and a positive attitude, and no more than seventeen SMART students. Each teacher uses a variety of strategies and tools to enhance their instructional delivery. Activities include real-life situations requiring the use of math skills. SMART students are exposed to and become familiar with a range of technological tools. The most popular tool used is the Tl- 84 Plus graphing calculator. SMART students learn how to use the calculator to further and deepen their knowledge of mathematics. In addition to the TI-84 Plus calculator, students use the Calculator-Based Laboratory (CBL) and the Calculator-Based Ranger (CBR) to solve problems and perform experiments. Students who successfully complete the SMART program receive a free TI-84 Plus graphing calculator, a SMART T-shirt, and a certificate. Instructional delivery and structure within the THRIVE program is virtually identical to that noted within SMART sessions
however, students do not have a high school mentor acting as a facilitator. THRIVE serves approximately 150 students, many of whom generally have been enrolled in SMART during the previous summer. Students currently enrolled in Algebra I meet bi-monthly on Saturdays during the spring semester from 8:45 a.m. - 12:25 p.m. The sessions are divided into 45-minute instructional blocks with learning activities prescribed for each block. The last block consists of two classes competing in a mathematics game/competition featuring the material covered that day. Teaching methods mirror those used in SMART, but integrate more advanced content and focus on material covered in Algebra I classrooms across the district. RESEARCH QUESTIONS Seven research questions provided the structure around which the evaluation was developed. The primary research question focused on the extent to which SMARTfTHRIVE programs improved and/or 13remediated math achievement among African American students. A substantive supplemental question addressed the level of participation in SMART and THRIVE by African American students. Additional attention was focused upon academically focused time and student engagement as well as parent, student, teacher and Algebra I teacher perceptions of the programs. The questions are listed below and are followed by a brief explanation of the areas addressed in the present evaluation. PRIMARY EVALUATION QUESTION Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African American students? A treatment-control student pre-test/post-test designed was employed for this evaluation. The analysis controlled for pretest, ethnicity and population free lunch status. Four types of Algebra I students were compared: a) no program/comparison, b) SMART program only, c) THRIVE program only and d) both SMART and THRIVE programs. The pretest data was collected from the 2003-2004 Math Benchmark Test and the posttest data was collected from the 2004-2005 Math Benchmark Test. Additional test information gathered for treatment-control comparison includes the 2004-2005 ITBS Math subtests and Algebra I End of Course Exams (EOC). Descriptive data from SMART and THRIVE participants were derived from pre- and post-test program exams. All standardized tests were administered by and throughout the Little Rock School District. Pre- and post- test program exams were administered through SMART and/or THRIVE and should be considered supplemental. SUPPLEMENTAL EVALUATION QUESTIONS 1. What is the level of participation in SMART and THRIVE by African American students? Attendance data from the SMART program of 2004 and the THRIVE program of 2005 were examined. In addition to descriptive information, the levels of participation were analyzed in light of student achievement. 2. What instructional strategies are used during the tutoring sessions? This question was addressed through classroom observations using an instrument focused on 26 research-based teaching strategies associated with increased academically focused instructional time and technology use within the classroom. Three additional math-specific items were designed to gather 14observation data about math-specific instructional practice. Five random observation visits were conducted during the THRIVE program sessions. 3. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? 4. What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? 5. What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? 6. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? Questions 3, 4, 5 and 6 were addressed through use of self-report surveys, including objective and open-ended items. Demographic data was also collected and analyzed. In addition, three student focus groups were created during which students perceptions of program strengths
weaknesses and overall satisfaction were collected, and then analyzed. PARTICIPANTS Currently, SMART/THRIVE serves students from the following eight Little Rock School District middle schools: Cloverdale: 82% African American student population
Dunbar: 61% African American student population
Forest Heights: 77% African American student population
Henderson: 82% African American student population
Mablevale: 81% African American student population
Mann: 52% African American student population
Pulaski Heights: 57% African American student population
Southwest: 94% African American student population. Students from the following high schools also were sen/ed: Central: 51 %, Fair: 81%, Hall: 75%, McClellan: 91%, and Parkview: 51%. Of 274 eighth- and ninth-grade students who participated in SMART, THRIVE, or both programs, matching district demographic data were available for 258 (94.2%). Tables 1 and 2 provide summaries of demographic and grade-level data for the various subsamples of SMART, THRIVE, and Comparison students in the study. These programs served a higher percentage of African American students than existed in the student population at large (69.0%): 88.0% of SMART students. 84.8% of THRIVE students, and 89.6% of students attending both programs were African American. Of the 258 students with matching data, 125 attended SMART only, 66 attended THRIVE only, and 67 attended both 15 \programs. Most students attending SMART only were ninth graders (83.2%), while the majority attending THRIVE only were eighth graders (66.7%). About equal percentages of eighth graders (53.7%) and ninth graders (46.3%) attended both programs. Whereas the district percentage of female students in eighth and ninth grades was 50.1%, the programs served a much larger proportion of females: 63.2%, 66.7%, and 59.7% for SMART, THRIVE, and both programs, respectively. Table 1 Percentage of Students Served by Race: SMART/THRIVE and Comparison PROGRAM Total Am Indian Comparison 0.2 SMART 0.0 THRIVE 0.0 BOTH 1.5 0.3 Asian 1.7 0.0 6.1 1.5 1.7 African American Hispanic Other 69.0 88.0 84.8 89.6 70.2 4.8 0.8 3.0 4.5 4.6 1.4 4.0 0.0 0.0 1.4 White 22.9 7.2 6.1 3.0 21.8 Total n 3726 125 66 67 3984 Note. Comparison group is all eighth and ninth grade students not served by either the SMART or THRIVE programs. Table 2 Program Attended by Grade Level PROGRAM Total GRADE 08 09 Total Count % within PROGRAM Count % within PROGRAM Count % within PROGRAM Comparison 1727 46.3% 1999 53.7% 3726 100.0% SMART 21 16.8% 104 83.2% 125 100.0% THRIVE 44 66.7% 22 33.3% 66 100.0% BOTH 36 53.7% 31 46.3% 67 100.0% 1828 45.9% 2156 54.1% 3984 100.0% EVALUATION DESIGN The evaluation was conducted during the spring of 2005. A mixed-methods design was employed to address the research questions. Results are primarily descriptive in nature
inferential statistics from 16achievement analyses are discussed. Both quantitative and qualitative data were collected by trained observers. The observers administered and collected surveys to THRIVE teachers and students and conducted focus groups with students and SMART mentors. The majority of the time was spent observing classrooms using the School Observation Measure (SOM) described below. Observers also conducted focus groups with students and SMART mentors and administered or distributed surveys to students. teachers, Algebra I teachers and parents. Additionally, achievement data from students participating in SMART, 2004 and THRIVE, 2005 was collected and analyzed by CREP. Details about all of the instruments and evaluation procedures are provided in subsequent sections INSTRUMENTATION To address the questions proposed in this mixed-methods evaluation, data from achievement tests. student records, classroom observations, surveys, and focus groups were collected. Six measurement strategies were used to collect the evaluation data: standardized achievement test scores, classroom observations, surveys, focus groups, and program-developed achievement benchmarks. ACHIEVEMENT TEST SCORES Math Benchmark Test 2003 - 2004 and 2004 - 2005: The Benchmark Test is a state-mandated criterion-referenced test aligned with state math standards and developed by Arkansas teachers and the Arkansas Department of Education. Benchmark scores are reported as advanced, proficient. basic, and below basic. The mathematics portion of the exam is comprised of multiple-choice and open response items. Internal consistency reliability estimates, from the 2002-2004 administrations ranged from Cronbachs Alpha of .84 - .87. End of Course Exams (EOC) 2005: The Algebra I End of Course Examination is a criterion- referenced test with questions that align with the goals and subject-specific competencies described by the Arkansas Mathematics Curriculum Framework and Algebra I Course Goals. The purpose of the test is to assess student mastery of Algebra I knowledge and skills. Students take the exam at the completion of Algebra I. Scores are reported as advanced, proficient, basic, and below basic. Internal consistency reliability estimates, from the 2001-2004 administrations ranged from Cronbachs Alpha of .81 - .88. 17 ITBS Math Subtests
The Iowa Test of Basic Skills (Math) assesses students ability to compute, deal with math concepts and solve problems dealing with numeration, graph/table interpretation, and computation. The ITBS test is a norm-referenced. The ITBS information is proprietary and, therefore. estimates of reliability and validity could not be obtained. SMART and THRIVE Pre/Post Tests: These curriculum-based tests measured students ability to utilize algebraic concepts to solve equations, judge inequalities, evaluate and simplify basic problems. graph equations, plot data, and solve applied problems. The THRIVE program tests were comprised of content more advanced than that of SMART program tests. Pre-test problems were identical to post-test problems in both programs. STUDENT RECORDS Attendance records from the 2004 SMART program and 2005 THRIVE program were examined in comparison to achievement measures. CLASSROOM OBSERVATION MEASURES The data collection instruments for classroom observations were the School Observation Measure (SOM), including three math-specific items designed to better tailor the instrument, and the Rubric for Student-Centered Activities (RSCA). The SOM and addendum items were designed to gather information about instructional practices and classroom activities. The RSCA was used to gather more detailed information about the level of student engagement during target activities throughout target observations. The instruments used for classroom observation are described below. School Observation Measure (SOM): School Observation Measure (SOM"). The SOM was developed to determine the extent to which different common and alternative teaching practices (e.g., direct instruction, cooperative learning, student inquiry, experiential learning) are used by teachers (Ross, Smith & Alberg, 1999) in typical (non-evaluative) classroom contexts. During each recording session, notes were completed every 15 minutes in different classrooms for a total of four at the end of one, 60-minute observation period. Researchers objectively recorded the relative use, non-use and frequency of 26 observation items covering a variety of classroom practices. Additionally, researchers recorded the extent to which high academically focused instructional time and high student attention/interest were observed. At the conclusion 18of a one-hour targeted visit, a trained observer summarizes the frequency with which each of the strategies was observed, yielding one SOM Data Summary Form. The frequency is recorded via a 5-point rubric that ranges from (0) Not Observed to (4) Extensively Observed. Two global items rate, respectively, the academically-focused instructional time and degree of student attention and interest. A reliability study by Lewis, Ross, and Alberg (1999) found that pairs of trained observers selected identical SOM categories 67% of the time and rated within one category 95% of the time. Math Addendum: Because the program under evaluation was specifically a mathematics program. three additional observations items unique to mathematics were developed. The items were written by a member of the research team with an Arkansas teaching certificate in mathematics based on the standards of the National Council of Teachers of Mathematics (NCTM). The three items addressed the teachers actions in emphasizing conceptual understanding, connecting the content to daily life, and promoting students confidence in mathematics. Each item was rated as 1 indicating not observed to 5 reflecting strong application. Rubric for Student-Centered Activities (RSCA): This Rubric (Ross & Lowther, 2002) is applied in conjunction with SOM visits to determine the quality and depth of teacher applications of selected strategies (e.g., cooperative learning, higher-order questioning, project-based learning, and technology as a learning tool). These strategies reflect emphasis on higher-order learning and attainment of deep understanding of content. Such learning outcomes seem consistent with those likely to be engendered by well designed, real-world linked exercises, projects, or problems utilizing technology as a learning tool. Each item includes a two-part rating scale. The first is a four-point scale, with 1 indicating a very low level of application, and 4 representing a high level of application. The second is a Yes/No option to the question: Was technology used? with space provided to write a brief description of the technology use. The RSCA was completed as part of SOM observation periods. The RSCA reliability results indicate that observer ratings were within one category for 97% of the whole-school observations and for 90% of the targeted observations (Sterbinsky & Burke, 2004). A RSCA was completed at the end of each targeted classroom observation. 19 SURVEYS H Four surveys were developed by the evaluation team to collect data pertaining to the effectiveness of the SMART/THRIVE Program. The specific audiences around which the current evaluation focused were parents, Algebra I teachers currently teaching in Little Rock schools, SMART/THRIVE teachers, and students currently enrolled in THRIVE. Items were designed to address both the primary and secondary research questions of the study and to glean open-ended perceptions from the four data collection groups. Items specific to SMART and THRIVE were included as were open-ended questions about both programs. Program-specific items were presented to gain general and specific knowledge about both programs as separate entities. This also eliminated the need for respondents to complete two separate surveys. Each survey was divided into three sections. The first section was used to collect demographic information specific to each group. The second section was comprised of objective items specific to the research questions and each program (SMART and THRIVE). In the final section, the respondents were presented with four, open-ended questions which probed strategies, strengths, weaknesses and general additional comments for each program. Responses across all surveys were scored through the use of Likert-type ratings ranging from strong disagreement (1) to strong agreement (5). Demographic information was also collected. Drafts of all surveys were presented at the evaluation team meetings in Little Rock and revisions were made from team participants suggestions. Surveys are found in Appendix A. FOCUS GROUPS m Two sets of focus group protocols were developed by the evaluation team. Students were asked their perceptions of the effectiveness of the SMART/THRIVE program. SMART mentors also were asked their impressions of program effectiveness. Additionally, participants were also asked to report their perceptions of program impact. Student and Mentor protocols were similar, but tailored to their specific audience. Question drafts were presented at an evaluation team meeting in Little Rock and revisions were made from team participants suggestions. PROCEDURES Data for this evaluation were collected during the spring of 2005. Targeted observations were conducted during late spring, 2005 using the SOM (including 3 math-specific addendum items) and RSCA 20 - instruments. Surveys, focus groups and program-designed achievement tests were also administered during this time period. The classroom observations were conducted during February and March, 2005. Observations were scheduled for random classrooms participating in the THRIVE program on three different ri days. Student performance measures were administered in late spring. Table 3 summarizes the data collection procedure. Table 3 Data Collection Summary Type of Measure Instrument Targeted Classroom SOM Observations RSCA Number Completed/Data Source Total 5 19 Surveys SMART/THRIVE Teacher Parent Algebra I Teacher Student 10 SMART/18 THRIVE 21 SMART/35 THRIVE 25 SMART/33 THRIVE 70 SMART/142 THRIVE Focus Groups Student Focus Group Mentor Focus Group 21 Participants 2 Participants Achievement fests MaSi Benchmark Test Attendance Data 2003 - 2004 and 2004 -2005 End of Course Exams ITBS ......SMART 2004 Attendance THRIVE 2005 Attendance 210 students 143 students Description Observation sessions in which random THRIVE classes were observed. During the observation period, note forms were completed every 15 minutes. Four^ sets of observation notes were completed for each SOM. Completed during Spring, 2005 THRIVE sessions. Four RSCAs were completed for each SOM. ItOTS specific to SMART and THRIVE were included in the surveys and administered to all groups in Spring, 2005. THRIVE student surveys were distributed in class and collected by researchers. THRIVE teacher sun/eys were distributed during a faculty meeting and administered by researchers. THRIVE parent surveys were distributed by teachers via students, returned to the school and sent to researchers. Algebra 1 teacher surveys were distributed through the district, returned to the school and sent to researchers or sent directly to researchers. During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups. Each focus group interview lasted approximately 45 minutes. Researchers used the Student Focus Group protocol to conduct an on-site, 45-minute focus group with 2 of the 7 mentors^_____ ___________ Researchers used data collected from three major math measures designed by independent entities and distributed throughout the LRSD for achievement data analysis. Exams were taken during the Spring of 2005. Researchers used the attendance and demographic data from SMART, 2004 and THRIVE, 2005 to examine level of participation among African American students. Data were collected from program heads and sent to researchers. Note: One SOM session was cut short due to scheduling, yielding 3 observations sheets and 3 RSCAs. ACHIEVEMENT TESTS End-of-course (EOC) Algebra I test scores were used as 2004-05 outcomes for 9* grade students. with 2003-04 Benchmark Mathematics scale scores used as a covariate. No covariate was available for 8* grade students, because no comparable district or state tests were administered to 7* graders in 2003-04. 21 The End of Course Exams were administered at the end of the academic year to all students enrolled in Algebra I during the 2004-2005 school year. The ITBS exam was administered only to 9* grade students, thus outcomes for this exam were available only for 9*^ graders. With no 8* grade scores available, 2003-2004 Benchmark scores were used for covariate comparisons. Additionally, curriculum-based pre- and postprogram exams were administered on the first and last day of both programs. Both pre- and post-program-designed exams were comprised of 20 items, asking students to simplify, evaluate, graph, and solve equations. The SMART exams were geared to test pre-algebra and early algebra content and the THRIVE exams were designed to examine content expected of students completing Algebra I. The pre-program exams and post-program exams were identical. These exams are provided in Appendix B. CLASSROOM OBSERVATIONS Trained observers completed a series of classroom visits to collect frequency data regarding observed instructional practices, use of technology and level of student engagement. Classroom teachers were advised that the observers would be present throughout the semester, but were instructed to deliver lessons as usual. A targeted procedure was used for the present study given that the observations were of the THRIVE sessions rather than of a school-wide, cross-categorical program. The majority of the observations were conducted by one researcher, with the second researcher observing for one SOM to examine inter-rater reliability. The inter-rater reliability for the joint observation was high
observers selected identical SOM categories 88% of the time and rated within one category 100% of the time. The standard SOM procedure involves evaluation of an entire school by visiting 10-12 randomly selected classrooms for 15 minutes each. However, the goal of this evaluation was to examine the practices within a single program, where all classrooms were participating in the same activities at the same times, not assess an entire school. Therefore, procedures were modeled from those used by Lowther, Ross, and Morrison (2003) in their examination a school laptop computer program. A total of five SOM observations were conducted. Both the RSCA and the Math Addendum items were completed at the same time. A total of 19 RSCAs were completed because one observation session was interrupted by a scheduled class dismissal. 22 I d M SURVEYS d E The program teacher surveys were administered and collected by researchers on March 5, 2005 during a faculty meeting. Student surveys were administered and collected by researchers during a March 5 class. Algebra I teacher surveys were distributed via district mail. Parent surveys were distributed through students and collected by teachers before being sent to CREP for analysis. Additional parent surveys were administered and collected by a member of the research team at the student Math Fair. FOCUS GROUPS During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups, each of which lasted 45 minutes. The mentors were contacted individually by researchers via phone, mail and email. Two of the seven mentors responded and attended. The mentor focus group interview lasted approximately 45 minutes. RESULTS The results of this evaluation are presented below and synthesized to address each research question in the Discussion section. These results are presented by measurement strategy
however, the findings are synthesized across instruments to reflect each research question in the Discussion section of this report. SCHOOL OBSERVATION MEASURE (SOM) As noted previously, observers focused on the 26 instructional strategies provided in the SOM using a standard five-point rubric (0 =not observed, 1 = rarely, 2 = occasionally, 3 = frequently, and 4 = extensively). The results are presented with a full, categorical breakdown in Table 4. Four strategies were observed occasionally to extensively at a frequency level of at least 75% or higher. These strategies include team teaching (100%), cooperative/collaborative learning (100%), use of higher-level questioning strategies (80%), and teacher acting as a coach/facilitator (100%). The data indicates that students were engaged in collaborative learning situations with high teacher support where they were asked to use problem solving and critical thinking skills to solve problems. Strategies that were observed occasionally to extensively during 40-60% of the sessions included direct instruction (lecture), higher level instructional feedback (40%), experiential hands-on learning (40%), independent inquiry or research (40%) and student 23M discussion (60%). Student discussion heavily dominated the instructional context (60% observed frequency) M with teams of students working together to solve problems and propose new ideas. Some individual seat work and independent research was observed (40% each respectively), however, these frequently were noted to lead into a group-based activity in which individuals proposed ideas to a team that was used to create a corporate answer to problems proposed by the teachers. Sustained reading was observed in one classroom. fl Technology was heavily used (80%) as teachers relied upon graphing calculators and problem solving with H technology as an integral part of the curriculum. Teachers demonstrated different ways to solve problems and/or work towards solutions via calculator and multi-media use. High academically focused time was fl observed extensively (80% of the time) as was high student engagement (in 60% of the observation sessions). Since the evaluation context was an Algebra I Saturday supplementary program with specific model parameters, some instructional strategies within the SOM were not observed. These strategies included individual tutoring, ability and multi-age groupings, work centers, integration of subject areas, project-based learning, independent seatwork, sustained wzriting/composition and systematic individual instruction. Generally, the aggregated data reflect efficient use of class time with teachers using a team teaching approach to deliver instruction to collaborative teams of students. The aforementioned math addendum items served to capture specific strategies used frequently among math teachers. Results from observations showed that THRIVE teachers frequently emphasized conceptual understanding over rote learning 60% of the time observed. Teachers also demonstrated connections between math and daily life frequently in at least 60% of the observations. One additional strategy of high interest was teachers cultivation of students confidence, flexibility and inventiveness in doing mathematics. This was observed frequently among 80% of the observations. None of the math-specific strategies were observed extensively. 24 n Table 4 II School Observation Measure (SOM) Data Summary for Little Rock-SMART/THRIVE Project N = 5 H II II II n n The extent to which each of the following was used or present in the school...________ Instructional Orientation Direct instruction (lecture) Team teaching Cooperative/collaborative learning Individual tutoring (teacher, peer, aide, adult volunteer) Classroom Organization Ability groups Multi-age grouping Work centers (for individuals or groups) Instructional Strategies Higher-level instructional feedback (written or verbal) to enhance student learning Integration of subject areas (interdisciplinary/thematic units) Project-based learning Use of higher-level questioning strategies Teacher acting as a coach/tacilitator Parent/community involvement in learning activities Student Activities Independent seatwork (self-paced worksheets, individual assignments) Experiential, hands-on learning Systematic individual instruction (differential assignments geared to individual needs) Sustained writing/composition (self-selected or teacher-generated topics) Sustained reading Independent inquiry/research on the part of students Student discussion Technology Use Computer for instructional delivery (e.g. CAI, drill & practice) Technology as a learning tool or resource (e.g., Internet research, spreadsheet or database creation, multi-media, CD Rom, Laser disk) Assessment Performance assessment strategies Student self-assessment (portfolios, individual record books) Summary Items High academically focused class time High level of student Percent None Percent Rarely Percent Occasionally Percent Percent Frequently Extensively 20.0 0.0 0.0 100.0 100.0 100.0 100.0 60.0 100.0 100.0 20.0 0.0 80.0 100.0 20.0 100.0 100.0 80.0 40.0 40.0 80.0 0.0 100.0 100.0 0.0 0.0 attention/interest/engagement________________________ Note: One SOM is comprised of approximately 8 classroom visits. 20.0 0.0 0.0 20.0 20.0 0.0 40.0 20.0 20.0 0.0 60.0 80.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 0.0 40.0 0.0 0.0 0.0 20.0 0.0 0.0 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 40.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 0.0 0.0 0.0 40.0 0.0 60.0 60.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 0.0 20.0 20.0 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 20.0 0.0 20.0 0.0 0.0 20.0 40.0 40.0 0.0 0.0 0.0 20.0 20.0 0.0 40.0 0.0 0.0 80.0 60.0 25 n RUBRIC FOR STUDENT-CENTERED ACTIVITIES (RSCA) IB The RSCA results reflect the percentage of observed sessions in which selected strategies are II observed at least once. The quality of the application and the percentage of sessions in which technology was used with the observed strategy are also recorded. A descriptive summary of the RSCA results is presented in Table 5. IB The RSCA results suggest that the level of application with which THRIVE program teachers applied certain instructional practices, particularly with regard to cooperative learning and higher-level questioning strategies was high. Additionally, students evidenced appropriate use of technology (calculators, multimedia. and spreadsheets) in self-directed activities to create new knowledge. The most notable area of technology use was among cooperative learning groups, where students collaboratively utilized multimedia, calculators and graphing applications to solve problems
teachers often helped direct collaborative groups. Project-based learning was not observed and although student discussion, experiential hands on learning, and independent research were applied, the level of application was limited or somewhat limited without technology use. Additionally, aggregate results suggest that technology was used in more than a third of the instances during which 5/7 remaining student-centered activities were observed and in over 25% of the instances during which 6/7 remaining activities or strategies were applied. 26 II Table 5 Rubric for Student-Centered Activities (RSCA) N = 19 Student-Centered Activities Not Limited Observed Application Somewhat Limited Somewhat Strong Application Application Strong Application Technology In Use * Cooperative Learning 8.7 13.0 8.7 21.7 47.8 60.9 II Project-Based Learning 100.0 0.0 0.0 0.0 0.0 0.0 Higher-Level Questioning Strategies Experiential Hands-on Learning Independent Inquiry/Research Student Discussion 26.1 56.5 56.5 56.5 0.0 4.3 8.7 17.4 30.4 39.1 4.3 26.1 4.3 0.0 8.7 21.7 30.4 17.4 13.0 39.1 4.3 0.0 39.1 30.4 Students as Producers of Knowledge Using 30.4 8.7 21.7 39.1 0.0 Technology 'Percentages Indicating Observed Levels of Application 'Rating Scale: 1= limited application: 4=strong application * See addendum description of technology use SCHOOL OBSERVATION SUMMARY Analysis of data collected using SOM, the Math Addendum and RSCA during the observations revealed that THRIVE teachers are using a variety of instructional strategies aligned with Arkansas state-designed SLEs. The most striking result was the extent to which team teaching and collaborative/cooperative learning dominated the curriculum. Additionally, notable use of higher level questioning strategies and teachers acting as coaches rather than lecturers was noted among observed sessions. During the teaching sessions that were observed, the Math Addendum items revealed that teachers were frequently emphasizing conceptual understanding over rote learning. Furthermore, this practice facilitated frequent instances of promoting students confidence and flexibility in doing mathematics. Overall, students in the THRIVE program had a high level of interest and were observed being engaged in instruction that emphasized practical and conceptual learning in a comfortable, student-centered setting. 27 SURVEY RESULTS Four surveys (Algebra I Teacher, Student, Parent, SMART/THRIVE Teacher) were administered in late March, 2005. Results for each of the four instruments were examined independently. Demographic characteristics of the respondents are representative: however, not all respondents completed the demographic section of the surveys. Additionally, the total number of respondents to THRIVE items varied from the total number of SMART respondents across questionnaires. Demographic differences are described n for each group. Finally, item percentages across surveys did not always total 100% because of missing input from some of the respondents (i.e., there were questions to which some people did not respond). PROGRAM PARTICIPANT SURVEY THRIVE total of 142 students responded to items pertaining to THRIVE and 70 students responded to items pertaining to SMART\ Of the 138 THRIVE respondents who reported their ethnicity, 77% specified African American, 7% specified Multi-Ethnic, and 4.2% specified Asian. The remaining categories of Caucasian (3.5%), Hispanic (2.8%), and Other (2.1%) were specified less frequently. A total of 63.4% of the respondents were female
35.9% of the respondents were male. Of the 70 students responding to SMART questionnaire items and specifying demographic information, 38.6% were female and 61.4% were male. Regarding ethnicity, 82.9% of the SMART respondents specified African American, 4.3% specified Caucasian or Hispanic respectively, 2.9% specified Multi-Ethnic and 1.4% specified Other or Asian. Table 6 contains the results for THRIVE and SMART objective survey items. These results are summarized in text below. Summaries of the open ended questions are shown in subsequent Tables. The overall trend in response types was positive
fifty percent or more students indicated agreement or strong agreement across all items. Well over eighty percent of students responding to the survey reported that team competitions made THRIVE classes more fun. Over seventy-five percent of respondents agreed or strongly agreed that THRIVE made algebra more enjoyable and instilled confidence in doing well on the algebra Benchmark Exam. Over sixty percent of student respondents agreed or strongly agreed that THRIVE improved their self-confidence about math, helped them feel comfortable asking questions in THRIVE class. Because some students participating in the THRIVE program did not participate in SMART during the previous summer, differences between respondent numbers were expected. 28 n II and facilitated problem solving through calculator use. A total of 66.6% of respondents indicated that THRIVE IB teachers helped them with problems they were having in their regular algebra class. At least one half (50.0%) of the students responding to the survey indicated that THRIVE helped improve their algebra grades and II helped them understand how to apply algebra concepts in real life. In contrast to the positive trend in responses across items, two areas in which some students showed disagreement were related to n generalization and application. This is somewhat consistent with responses noted within the open ended II portion of the student survey where students indicated an overall satisfaction with the program but did not feel that the program changed their study habits or helped them apply information to real life situations. II Additionally, more students responded neutrally to items pertaining to content, grades and application when compared to responses for items related to motivation, fun and self-confidence. II Table 6 II Student Survey - THRIVE N = 142 THRIVE items II 1. Because of THRIVE, I have learned how to use a calculator to help solve algebra problems. Percent Strongly Agree/ Agree 67.6 Percent Neutral 23.9 Percent Strongly Disagree/ Disagree 8.5 II 2. THRIVE makes algebra more enjoyable. 77.5 20.4 2.1 3. II 4. II 5. 6. II 7. 8. THRIVE has helped me get good grades in algebra. I think I will do well on the algebra Benchmark Exam because of THRIVE. In THRIVE, I have learned how algebra can be used in real life. THRIVE has made me more confident about math. My THRIVE teacher helps me with problems I am having in my algebra class. I feel comfortable asking questions in THRIVE class. 50.0 78.9 55.6 66.2 66.2 74.7 40.9 16.2 33.8 24.7 25.4 21.1 8.5 4.2 10.6 7.8 8.5 4.2 9. Team competitions make THRIVE classes fun. 87.3 6.3 2.8 29 a n II SMART II Responses indicating agreement or strong agreement for items pertaining to the SMART portion of survey were consistently above 75%. Of the 70 respondents, over 85% agreed or strongly agreed that II SMART helped prepare them for Algebra I in the fall, and 80% agreed or strongly agreed that SMART made algebra more fun. Seventy-eight percent agreed or strongly agreed that the SMART program helped them II remember math skills from the previous year. Students agreed or strongly agreed that they were motivated to II go to their SMART classes and that the program helped them apply math to solve real-life problems across 75% of the surveys. Fewer students indicated neutral or negative responses for the SMART survey items II than noted among THRIVE survey responses. Of those that indicated disagreement or strong disagreement, the item noted most frequently (4.3%) was related to application (We learned how to use math to solve real- II life problems.). Overall, the trend towards agreement with items probing the attributes of the programs was positive. Table 7 presents detailed results for the SMART student survey. II Table 7 II Student Survey - SMART N = 70 II SMART Items 1. II 2. II 3. 4. II 5. SMART helped me remember the math skills I learned last school year. SMART made algebra more fun. In SMART, we learned how to use math to solve real-life problems. I was motivated to go to my SMART class during the summer. Because of SMART, I was prepared to begin Algebra I in the fall. Percent Strongly Agree/ Agree 78.6 Percent Neutral 18.6 Percent Strongly Disagree/ Disagree 2.9 81.4 75.7 75.7 88.6 14.3 15.7 18.6 5.7 1.4 4.3 1.4 1.4 il OPEN-ENDED RESPONSES THRIVE students had an opportunity to respond to four, open-ended questions as part of the survey. Student impressions of the SMART program were polled across all students even though SMART was not in session and not all students participated in it during the previous summer. SMART and THRIVE were evaluated separately using the same questions to preserve the integrity of their differences. Student responses are summarized below. 30I H 11 Question 1: Students were asked to comment on what they liked about the THRIVE program. The II majority of responses reflected an appreciation of the programs use of competition and a sense that the program made math more fun/enjoyable. Additional comments suggested that students enjoyed having a II program that allowed for review and expansion of concepts taught in the regular Algebra I classroom. Students also noted that they enjoyed the teachers and felt comfortable asking for clarification when II necessary. Additionally, snacks and rewards were notably cited as positive influences of THRIVE. Table 8 II summarizes what students liked about THRIVE. TABLE 8 II What do you like about THRIVE? II Description II II II II II Makes math fun Uses competition/games in Provides expansion/review of math concepts Clarifies/Remediates confusing concepts Teachers are good/likeable Snacks People/friends in THRIVE classes Class size/Format Money/Rewards NA - responses did not pertain to question Enjoy using a calculator Provides a head start for following week in class Improves self-confidence/provides comfortable place to learn Class Time/Format TOTAL Frequency 42 37 18 17 17 14 11 8 7 7 4 3 3 2 190 Percentage 22% 19% 9% 9% 9% 7% 6% 4% 4% 4% 2% 2% 2% 1% 100% II Question 2: When asked what they would change about the THRIVE program, the ovenwhelming majority of students responded that they would make no changes. Students who did comment on needed II changes mostly cited a later start time or a more compressed time period (e.g., two hours rather than four hours) as potential prospects. Non-specific time changes (e.g., change time, time, different time) were also H noted among open-ended comments but coded separately due to the large number of students commenting on previously cited specifics about THRIVE hours. Table 9 provides an illustration of student comments. The Hi miscellaneous category captured unique comments made by individual students (e.g., field trips, guest speakers. Physical Education). 31 I TABLE 9 What would you change about THRIVE? Description No change Less time Later start time Miscellaneous Class format/more varied activities Longer breaks Non-specific time change Snacks/lunch Enhance competition More time More peers/neighborhood school Rewards Earlier time Teacher changes TOTAL Frequency 57 17 14 11 10 8 8 8 5 4 3 3 2 2 152 Percentage 38% 11% 9% 7% 7% 5% 5% 5% 3% 3% 2% 2% 1% 1% 100% Question 3: Students were asked to comment on how THRIVE has helped them over the course of the semester. Most comments suggested that THRIVE helped students enhance comprehension of general algebraic concepts. Many students also indicated that THRIVE helped improve their grades and improved their understanding of how to use a calculator to solve and graph equations. Additionally, students' comments suggested that THRIVE reinforced specific concepts (e.g., graphing equations with polynomials) and made the students feel that they were staying ahead in weekly algebra classes. Of 150 responses, three indicated that THRIVE was not helpful
none indicated that it was detrimental. Table 10 summarizes the results. TABLE 10 If THRIVE has helped you, how has it helped you? Description Enhanced comprehension of algebra Improved grades Use of calculator Reinforced specific concepts in algebra Helps stay ahead in math Builds confidence/makes math more positive Miscellaneous (individual answers) Negative-Did not help TOTAL Frequency 63 20 18 16 14 8 8 3 150 Percentage 42% 13% 12% 11% 9% 5% 5% 2% 100% 32 II l< Questions about SMART were analyzed. Response rates across SMART questions, although Il somewhat different, do not vary significantly. Question 4: When asked to identify the most appealing aspects of SMART, most comments were If positive, yet nonspecific in nature. Examples included responses that alluded to liking everything about the program or how the program helped me (in general)". A number of students (13%) noted that SMART II prepared them for math in the fall and reduced anxiety when entering first year algebra. Calculator use and II team competition garnered notably high response rates (13% and 12% respectively). Some students made specific comments about the positive influence of the SMART teachers and others made comments about rewards delivered through the program. Response categories are provided in Table 11. TABLE 11 What do you like about SMART? Description Nonspecific positive responses (e.g.. Everything
How it helped me) More prepared for math/ reduced anxiety Using the calculator Team competitions The teachers Reinforcement and rewards Social aspects/visiting with peers from other schools The snacks Games and content delivery NA (e.g., I did not go to SMART
I do not know) Format/time/class size TOTAL Frequency 31 11 11 10 5 5 5 3 2 2 1 86 Percentage 36% 13% 13% 12% 6% 6% 6% 3% 2% 2% 1% 100% Question 5: Students also provided information about what aspects of the SMART program they would change. Over sixty percent of the responses favored no change. Of the remaining responses, 23% indicated a time change as appropriate. Of those, the large majority of comments suggested a later starting time rather than a reduced time period. One student noted a desire to reduce the amount of time spent at SMART. A few students who did not participate in SMART simply indicated such
no additional comments were recorded among these respondents. The remaining responses indicated a desire to enhance the program by adding team competitions, extending the time during which SMART was available, and providing more applied skills for use in everyday situations. Data are summarized in Table 12. 33 TABLE 12 IVhaf would you change about the SMART program? Response No change Time Snacks NA (I did not go to SMART
1 like Math) More team competitions More applied skills Extend time/day and/or hours________ TOTAL Frequency Percentage 41 15 3 3 2 1 1 66 62% 23% 5% 5% 3% 1% 1% 100% Question 6: When asked about how SMART helped students, the overwhelming majority of comments identified the programs purpose in preparing students for Algebra I as most important. Many responses also indicated the use of a calculator as instrumental in helping students solve algebra problems. Additionally, students noted that SMART helped improve their grades and built confidence in math. TABLE 13 If SMART helped you, describe how. Response Made me feel prepared Use of calculator Built confidence in math Improved grades NA(I did not go to SMART
1 liked THRIVE) Reinforced specific concepts Miscellaneous (I would like field trips) Increased engagement/interest TOTAL PARENT SURVEY THRIVE Frequency Percentage 41 11 6 6 3 1 1 1 70 59% 16% 9% 9% 4% 1% 1% 1% 100% A total of 35 parents responded to survey items pertaining to THRIVE. Of those, 57% were female and 40% were male. A total of 80% of parent respondents indicated that their child was African American, 8.6% specified Asian and 2.9% specified Caucasian, Hispanic and Other respectively. No parental respondents specified Multi-Ethnic as the ethnicity of their child. 34 n II Responses indicating agreement or strong agreement were consistently above 80% across all survey il items. Overall, parents agreed or strongly agreed that THRIVE was a good program (94%), that they felt comfortable having their child attend classes on Saturdays (91.4%) and that they felt teachers in the program II helped their children feel successful in math (91.4%). The neutral responses that were noted were made in response to items pertaining to childrens attitudes about math and improvement in grades. No negative II responses to the items were reported. Table 14 provides a summary of the responses. H Table 14 11 Parent Survey - THRIVE N = 35 THRIVE items 11 Percent Strongly Agree/Agree Percent Percent Strongly Neutral Disagree/ 1. II 2. 3. II 4. 5. II 6. II II II II n The teachers in this program make my child feel that he/she can succeed. Because of this program, my child is more motivated to complete algebra homework. This program has helped improve my childs attitude about math. I am comfortable having my child attend classes on Saturdays. Because of this program, I have seen an improvement in my childs Algebra I grades. Overall, I think this is a good program. SMART 91.4 88.6 82.9 91.4 80.0 94.3 5.7 8.6 17.1 5.7 14.3 2.9 Disagree 0.0 Percent Don't Know 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 A smaller number of parents (21) responded to the five survey items pertaining to SMART. Of those, 61.9% were female and 33.3% were male. Eighty-five percent of parental respondents cited their childs ethnicity as African American. The remaining respondents indicated Asian (4.8%) or Other (4.8%) as their childs ethnicity. Ninety percent of parent respondents indicated agreement or strong agreement when asked whether SMART did a good job of preparing children for Algebra I in the fall and improving childrens perception that they could succeed in math. Results are noted in tables 15 and 16. Slightly over 85% of respondents indicated agreement or strong agreement with items related to whether SMART improved their childs attitude about math and whether they felt comfortable sending their children to classes during the summer. A total of 35 h II n 76% of parents agreed or strongly agreed that the program classes were easy for their children to attend. n None of the respondents indicated disagreement or strong disagreement with any of the items. Of those that responded neutrally, the highest number (14.3%) were noted for the item pertaining to whether SMART had II improved attitudes about math. No parents indicated negative responses for any of the items. II Table 15 II Parent Survey- SMART N=21 SMART Items II 1. II 2. 3. II 4. 5. II Because of this program, my child felt like he/she could succeed in Math. The summer classes were easy for my child to attend. This program helped improve my childs attitude about Math. I am comfortable having my child attend classes during the summer. This program did a good job of preparing my child for beginning Algebra I in the fall. Percent Strongly Agree/ Agree 90.5 Percent Neutral 4.8 Percent Strongly Disagree/ Disagree 0.0 Percent Dont Know 0.0 76.2 85.7 85.7 90.5 4.8 0.0 4.8 14.3 0.0 0.0 0.0 0.0 4.8 4.8 0.0 0.0 OPEN-ENDED RESPONSES II Question 1: When asked to provide their impressions of THRIVE, many parents responses described II it as a motivating math program (19%) targeting childrens math skills. Numerous comments suggested that parents characterize THRIVE as a program that improves general math skills (19%) and/or focuses on II problem solving and/or critical thinking (15%). A number of parents comments also reflected an awareness of the programs use of motivating games and competition to foster interest in math (15%). The balance of the II comments were positive and reflected an awareness of THRIVE as a supplemental alternative to the weekly Algebra I curriculum. II II 36 H bII II TABLE 16 II Please tell us your impression of the THRIVE program. What do you see as taking place during the sessions? II II II II Description Helps child feel fulfilled/motivated about math Helps child improve math skills Focuses critical thinking and problem solving Integrates games and fun into math Non-specific positive commendation Practical application Provides individualized tutoring provides review of school curriculum Class format: small groups and team competition TOTAL Frequency 5 5 4 4 2 2 2 21 27 Percentage 19% 19% 15% 15% 7% 7% 7% 7% 4% 100% II Question 2 and Question 3: Questions 2 and 3 addressed parents opinions of THRIVE. One third of the comments made about parents positive opinions indicated that it was motivating. Parents also noted that II they liked how THRIVE provided additional help (17%) and made efforts to improve specific math skills (12%). n Some parents reported global positive experiences and satisfaction with THRIVE (14%). Table 17 provides additional data regarding what parents liked about the program. II Regarding parents opinions about whether aspects of THRIVE should be changed, 29% recommended no change and/or responded positively about the program. Of the comments requesting n change, one third recommended expanding the breadth and length of the program. Specifically, these parents made note that more time (23%) should be allocated for THRIVE and that more students (10%) should be II involved. Table 18 provides additional information about isolated comments for changing THRIVE. II II n 37 II II TABLE 17 II What did you like about the THRIVE program? Description Frequency Percentage II II II II Motivating Provides additional help Non-specific positive commendation Improve math skills Class size, time, format Learning new strategies Games Parent gatherings Prepares for benchmark exam Teaches calculator skills TOTAL 14 7 6 5 3 3 1 1 1 1 42 33% 17% 14% 12% 7% 7% 2% 2% 2% 2% 100% II II TABLE 18 If anything, what would you change about the THRIVE program? II Description Frequency Percentage II II II II II No change Extend time/meet more frequently Expand program to more students More challenge Expand into summer Expand program to after school Get credit Later start Measurement of progress More communication between parents and teachers More funding More hands on learning More structure Transportation
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.