Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 2 of 69 CREP Center for Research in Educational Policy Evaluation of SMART/THRIVE In the Little Rock School District Center for Research n Educational Policy The Universiy of Memphis 325 Browning H<d t4emphis. Tennessee 38152 ToB Free. 1-866^70^147 Research Report 2004-2005 DRAFT REPORT FOR REVIEW BY Little Rock School District December 2005 Lyle Hull Davis, Ph.D. Gail Weems, Ph.D. John Nunnery, Ph.D. Center for Research in Educational PolicyCase 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 3 of 69 EXECUTIVE SUMMARY INTRODUCTION DHlUOilNO W N0lW5)3H33S3a Nvr aaAiaoaa This report summarizes the 2004-2005 evaluation study of the SMART and THRIVE programs. The global purpose of the evaluation was to: a) provide cumulative evidence of SMART/THRIVE implementation practices
b) examine the extent to which the programs have been effective in improving and remediating academic achievement of African-American students
and c) explore perceptions of participants, parents and teachers as well as level of student participation in SMART and THRIVE. The overall purposes of the SMART and THRIVE programs are to provide supplemental prealgebra support to students entering Algebra I and to prepare students to meet state standards in Algebra I, respectively. The programs intentions/goals are to: a) provide a solid foundation for Algebra, encourage exploration
b) make mathematics fun
c) enable students to achieve academic success in mathematics
and c) create the confidence necessary to pursue higher level courses. The SMART program meets for two consecutive weeks during the summer and the THRIVE program meets every other Saturday during the Spring semester. SMART uses a co-teaching model with one teacher and one high school student mentor while THRIVE uses a co-teaching model with two certified teachers. The primary context of observations, focus groups and surveys was the 2005 THRIVE program. Additional attendance data were collected from the 2004 SMART program. Survey and focus group responses concerning both programs also were collected. Both programs served students from the following schools: Cloverdale, Dunbar, Forest Heights, Henderson, Mablevale, Mann, Pulaski Heights, and Southwest. Research Questions Seven research questions provided the focus of the evaluation. The primary research question focused on the extent to which SMART/THRIVE programs improved and/or remediated math achievement among African American students. Evaluation questions were: 1. Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? 2. What is the level of participation in SMART and THRIVE by African American students? 3. 4. What instructional strategies are used during the tutoring sessions? What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? 5. What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? 6. What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? 7. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? RECEIVED JAN 3 2006 1 OFFICE OF DESEGREGATION MONITORINGCase 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 4 of 69 Design The evaluation was conducted during the spring of 2005. A mixed-methods design was employed to address the research questions. Results within this summary are primarily descriptive in nature
however, achievement data will be used for comparative analyses among SMART participants. THRIVE participants^ students participating in both programs, and students participating in neither program. These data will be examined in light of attendance data provided by program heads. Evaluation Measures Six measurement strategies were used to collect the evaluation data: standardized achievement test scores, classroom observations, surveys, focus groups, and program-developed achievement benchmarks. Administration procedures and descriptions of instrumentation are provided below. Direct Classroom Observation Measures School Observation Measure (SOM*): Examines frequency of usage of 26 instructional strategies. Rubric for Student-Centered Activities (RSCA)
Provides measurement of the degree of learner engagement in cooperative learning, project-based learning, higher-level questioning, experientiai/hands-on learning, student independent inquiry/research, student discussion and students as producers of knowledge. Math Addendum: Rates teachers' actions in emphasizing conceptual understanding, connecting the content to daily life, and promoting students confidence in mathematics. Surveys Student Survey: Collects participant impressions and perceptions regarding the satisfaction, short-comings, strengths and influence of SMART and THRIVE programs. Parent Survey: Collects parent perceptions and impressions regarding availability (access), influence, transportation and opinions of SMART and THRIVE. Algebra I Teacher Survey: Collects Algebra I teacher perceptions and impressions of influence of program on student performance as well as global awareness and opinions of SMART and THRIVE. SMART/THRIVE Teacher Survey: Collects THRIVE teacher perceptions and impressions regarding professional development, influence of technology, influence of program on student performance and overall opinions of the program. Focus Groups Student Focus Groups: Collects program participant impressions and perceptions regarding the influence of each program on Algebra I achievement, test scores, and self-confidence about math. Also collects information about SMART and THRIVE program weaknesses, strengths and needs. Mentor Focus Group: Collects high school SMART mentor impressions and perceptions of SMART program strengths, weaknesses, needs, and influence on students. Student Records Attendance records from the 2004 SMART program and 2005 THRIVE program were examined in light of achievement data. Participation rates are reported as well. 2Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 5 of 69 student Academic Performance Math Benchmark Test (2003 - 2004) scores were analyzed. The Benchmark Test is a state- mandated criterion-referenced test aligned with state math standards and developed by Arkansas teachers and the Arkansas Department of Education. Benchmark scores are reported as Advanced, Proficient, Basic, and Below Basic. The mathematics portion of the exam is comprised of multiple-choice and open response items End of Course Exams (EOC) 2004-2005 scores were analyzed. The Algebra I End of Course Examination is a criterion-referenced test with questions that align with the goals and subjectspecific competencies described by the Arkansas Mathematics Curriculum Framework and Algebra I Course Goals. The purpose of the test is to assess student mastery of Algebra I knowledge and skills. Students take the exam at the completion of Algebra I. Scores are reported as Advanced, Proficient, Basic, and Below Basic. ITBS Math Subtests scores were analyzed. The Iowa Test of Basic Skills (Math) assesses students ability to compute, deal with math concepts and solve problems dealing with numeration, graph/table interpretation, and computation. SMART and THRIVE Pre/Post Tests: The curriculum-based tests are measures of student ability to utilize algebraic concepts to solve equations, judge inequalities, evaluate and simplify basic problems, graph equations, plot data, and solve applied problems. The THRIVE program tests were comprised of content more advanced than that of SMART program tests. Pre-test problems were identical to post-test problems in both programs. Procedure Data for this evaluation were collected during the ^ring of 2005. Targeted observations were conducted in February and March of 2005 using the SOM, Math-addendum, and RSCA instruments. J , IVIUK I UUUOl lUUl I I, dl lU ITOV/ZA II lO U UI I I Cl I lb. Surveys, focus groups and program-designed achievement tests were also administered during this time period. Observations were scheduled for random classrooms participating in the THRIVE program on three different days. Student perfonnance measures were administered in late spring. The following table summarizes the data collection procedure. 3Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 6 of 69 Data Collection Summary Type of Measure Targeted Classroom Observations Instrument SOM RSCA Math Addendum Number Completed/Data _____ Source Total 5 19 5 Description A series of observation sessions in which random THRIVE classes were observed. During a one hour period, four^ note forms were completed every 15 minutes. Completed during Spring, 2005 THRIVE sessions. Surveys SMART/THRIVE Teacher Parent Algebra I Teacher Student 10SMART/18 THRIVE 21 SMART/35 THRIVE 25 SMART/33 THRIVE 70 SMART/142 THRIVE Items specific to SMART and THRIVE were included in the surveys and administered to all groups in Spring, 2005. THRIVE student surveys were distributed in class and collected by researchers. THRIVE teacher surveys were distributed during a faculty meeting and administered by researchers THRIVE parent surveys were distributed by teachers via students, returned to the school and sent to researchers Algebra I teacher surveys were distributed through the district, returned to the school and sent to researchers or sent directly to researchers. Focus Groups Student Focus Group Mentor Focus Group 21 Participants 2 Participants During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups. Each focus group interview lasted approximately 45 minutes. Mentors from the SMART 2004 program were contacted by researchers. Researchers used the Student Focus Group protocol to conduct an on-site, 45-minute locus group with 2 of the 7 mentors. Achievement Tests Math Benchmark Test 2003-2004 and 2004 -2005 End of Course Exams ITBS Researchers used data collected from three major math measures designed by independent entities and distribute throughout the LRSD for achievement data analysis. Exams were taken during the Spring of 2005. Attendance Data SMART 2004 Attendance THRIVE 2005 Attendance 210 students (88% African American) 143 students (84% African American) Researchers used the attendance and demographic data from SMART, 2004 and THRIVE, 2005 to examine level of participation among African American students *One observation session was comprised of three rather than four observation segments due to scheduling constraints. 4Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 7 of 69 Results Direct Classroom Observation Results Targeted Observations Analysis of data collected using SOM, the Math Addendum and RSCA during the five observation sessions revealed that THRIVE teachers are using a variety of instructional strategies aligned with Arkansas state-designed Student Learner Expectations (SLE). The most striking result was the extent to which team teaching and coilaborative/cooperative learning dominated the curriculum. Additionally, notable use of higher level questioning strategies and teachers acting as coaches rather than lecturers was noted among observed sessions. During the teaching sessions that were observed, the Math Addendum items revealed that teachers were frequently emphasizing conceptual understanding over rote learning. Furthermore, this practice facilitated frequent instances of promoting students confidence and flexibility in doing mathematics. Overall, students in the THRIVE program had a high level of interest and were observed being engaged in instruction that emphasized practical and conceptual learning in a comfortable, student-centered setting. The RSCA results suggest that the THRIVE program integrated technology into the curriculum. Average or above average implementation (e.g., level of intensity, meaningfulness, quality) was noted for activities associated with higher level thinking strategies, teamwork and independent technology use. Survey Results Student Survey The overall trend in response types was positive for survey items pertaining to THRIVE and SMART. Regarding THRIVE, students demonstrated high levels of agreement for items probing self-confidence, comfort level, growth among problem solving skills, and overall satisfaction with the program. Two areas in which some students showed some disagreement were related to generalization and application. Students indicated an overall satisfaction with the program on the open ended portion of the student survey, but did not feel that the program changed their study habits or helped them apply information to real life situations. Responses to items pertaining to SMART indicated agreement among students for items pertaining to preparedness, motivation, and application. Open-ended items also suggest overall satisfaction with SMART
however, a number of student suggestions were made about changing the time during which the summer program met. Parent Survey Responses from the 35 parents who responded to the THRIVE parent survey suggest that they have positive feelings towards the program. Agreement or strong agreement was consistently noted across all survey items. No negative responses to the items were reported. The 21 parent respondents provided similar information, particularly for those items reflecting their satisfaction with the teachers, their childrens preparedness for Algebra I, and their comfort level with the program meeting time. None of the respondents indicated disagreement or strong disagreement with any of the items. Neutral responses were noted for one item pertaining to whether SMART had improved attitudes about math. Parent perceptions measured with analysis of open-ended responses reflected an overall satisfaction with both programs. When asked specifically about transportation, the majority of parents did not site it as an issue
however, some parents reported access to transportation being inconsistent. Algebra I Teacher Survey In contrast to response trends from the parents, students and THRIVE teachers surveyed, Algebra I teachers responses to SMART items showed slightly more varied distribution across the survey scale (Strongly Agree/Agree-Neutral-Strongly Disagree/Disagree-Dont Know). Responses from the 33 Algebra I teachers responding to THRIVE items and the 25 Algebra I teachers responding to SMART items showed a 5Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 8 of 69 positive response trend. These teachers agreed that both programs had bolstered students self-confidence and positively impacted student achievement
however, they responded more neutrally to items probing the programs' influence on childrens problem solving ability and influence on general Algebra I instruction. When asked specifically about whether they felt SMART and THRIVE had had an impact on achievement differences among races, a small percentage disagreed, approximately one third responded neutrally or did not know and the remaining 48% agreed. SMART/THRIVE Teacher Survey SMARTTTHRIVE Teachers responses overwhelmingly favored agreement or strong agreement with items pertaining to positive attributes among professional development, resources/materials, methodology and instructional delivery within the THRIVE and SMART programs. Two items for which some teachers indicated neutrality across both programs were related to the extent to which they felt the program had improved their own ability or their students ability to use calculators. No negative responses were noted among the items
however, some teachers indicated that the limited time available for teaching was considered a shortcoming for both SMART and THRIVE. Focus Groups Student Focus Group Students perceptions of both programs suggest that there is a generally positive feeling towards SMART and THRIVE. Specifically, the consensus among the twenty-one students who participated in the focus groups was that SMART and THRIVE bolstered math skills and boosted confidence. Despite some mild distaste for required meeting times (Saturdays and during the summer), most student participants indicated that both SMART and THRIVE had reinforced Algebra I skills and provided a fun, relaxing context for learning. Many students noted that the programs had provided new strategies for problem solving and made Algebra I less daunting. There were no outwardly negative comments made. Mentor Focus Group Two high school student mentors who had participated in SMART, 2004 provided input about the SMART program primarily. The mentors viewed SMART as a preparatory program focused on reviewing prealgebra principals. Both mentors, however, also noted that SMART provided a unique environment in which competition helped motivate students and make math more enjoyable. The mentors global perception of the program was that SMART allowed students to get a head start on Algebra I principals with a simultaneous boost in confidence. Attendance Collectively, the 2004 SMART program served 210 students for which attendance data was available and the 2005 THRIVE program served 143 students for which data was available. Compared to the general population served by the district (69.0% African American), SMART and Thrive programs served a higher percentage of African American students (88.0% SMART
84.8% Thrive
89.6% Both). The attendance rates for SMART yielded an average rate of 97% (range: 94%- 100%) and the attendance rates for Thrive yielded an average rate of 90% (range: 81 %-100%). Achievement The results provide evidence that, in 9'*' grade, the SMART program, the Thrive program, or the combination of the two programs were associated with substantial and statistically significant improvements in the achievement of participating students relative to non-participating students on the Algebra I EOC exam. The strongest effect size was associated with a combination of the programs. Additionally, eighth grade students participating in any combination of the programs were less likely than their non-participant peers to perform at basic or below basic proficiency levels on the Algebra I EOC exam. More specifically, African American students in both eight and ninth grade who were 6Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 9 of 69 enrolled in any of the three program configurations were more likely to perform at proficient or advanced levels of proficiency than the non-participant comparison group. Statistically significant gains were also noted on the program-specific assessments for the SMART and Thrive programs. No statistically significant program effects or racial differences noted among scores from the ITBS. CONCLUSIONS Each of the major research questions will provide a framework around which the conclusions for the present study will be structured. Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? StudenMevel achievement analyses yielded results suggesting that participation in SMART, Thrive and a combination of programs is associated with comparative improvements in achievement relative to nonparticipating students. Specifically, our interpretation of these outcomes focuses on the extent to which these differences in achievement relate to growth among African American students. Findings suggest that although there was not a statistically significant interaction term specific to race, participation in any of the program configurations was associated with a major reduction in the achievement gap between African American students and students of other races from 8* grade to 9* grade, from a one standard deviation deficit on 2003-04 Benchmark scores to virtually no difference on 2004-5 Algebra I EOC scores. For 2004-05 8 graders, the results suggest a positive effect of the programAfrican American students attending any program configuration were more likely to achieve at Proficient or Advanced levels. However, the lack of 2003-04 achievement data tor use as a covariate limits the strength of this evidence. Although results were favorable for African American students enrolled in both programs, no statistically significant effects were noted from ITBS scores for race or program, demonstrating that although SMART and Thrive have a direct and significant impact on curriculum-based performance measures, they have a lesser influence on standardized math achievement performance. In light of the current goal in increasing achievement among African American students, particularly in the domain of mathematics, the results from EOC and Benchmark exams are highly significant. These results are encouraging as implementation of the SMART and Thrive programs seems to have had a positive impact on African American student achievement. What is the level of participation in SMART and THRIVE by African American students? Overall, the level of participation for both programs was high. The vast majority of students enrolled in SMART, Thrive or both programs missed no sessions, yielding average attendance rates of 90% or above. Additionally, data from classroom observations suggested a high degree of student engagement and participation across classrooms within the Thrive program. Given the demographic characteristics of the participants, the significant quantitative relationship between participation and student achievement gains, and the high participation rates among participants, the results provide evidence that SMART and Thrive have a positive impact on participation among African American students and that participation has had significant influence on achievement. What instructional strategies are used during the tutoring sessions? During classroom visits, the majority of observed instructional strategies were couched in a collaborative teaching model focused on higher level thinking skills and student grouping. Students were observed utilizing technology throughout their lessons and were highly engaged during observation sessions. Teachers frequently posed hypothetical questions to groups to accommodate use of higher-level thinking skills and generate more varied student responses. Even when students were engaged in independent seatwork, which was rare, teachers were observed circulating among various student groups, requesting more in-depth questioning strategies in their problem solving approaches. Additionally, although the competition component 7Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 10 of 69 of the program was not observed directly, students were frequently cited encouraging fellow students in preparation for competition while working in collaborative groups. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? There was a positive response trend among SMART/THRIVE teachers across questionnaire items for both programs. Teachers generally believed that the core components of both programs bolstered student learning, particularly among students who were struggling to perform well in Algebra I classes. Additionally, they reported feeling comfortable with expectations and resources, noting that having opportunities to collaborate with other teachers and utilize competition to motivate students were essential to success. Teachers cited time constraints as a recognized weakness across both programs. The general consensus was that they wanted more time to develop deeper skills among students and to reinforce concepts taught in Algebra I classrooms. None of the respondents expressed feelings of dissatisfaction with either program and demonstrated the highest percentage of positive responses when compared to other groups (students, parents. Algebra I teachers). What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? Algebra I teachers were generally positive in their reaction to both SMART and THRIVE programs, representative of an overall feeling that SMART and THRIVE facilitated student learning and generalized to the Algebra I classroom. Algebra I teachers noted observed improvements among students participating in SMART/THRIVE and globally agreed that the programs effectively enabled students to use technology in the Algebra I classrooms. A number of Algebra I teachers indicated that both programs needed to reach a larger and broader group of students. There were no indications that Algebra I teachers disliked SMART or THRIVE
however, there was some evidence that some respondents felt that the programs did more to supplement what was being taught in Algebra I classrooms rather than cultivate more meaningful understanriinn nf algebraic concepts, ~ What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? Students were somewhat mixed in their reactions to SMART and THRIVE, although the response trend among respondents across items was generally positive. While most of the students believed that SMART and/or THRIVE helped them with Algebra I class, others indicated mild dislike for time spent doing academics during summer and/or Saturdays. Students did perceive the programs as motivating, fun, and globally helpful. It was rewarding to note that despite some dislike of the programs' scheduled meeting times, most students indicated a desire to continue with the programs modifying very little about the actual instructional delivery. These comments were corroborated by comments students made during focus group sessions. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? Parents' reactions to both programs were globally positive. Specifically, parents commented that the motivational, fun nature of the programs was fostering an interest in math among students. Comments for change were largely dominated by parents requesting that the program be expanded both in breath/scope and duration. None of the parents cited negative features
however, some parents requested more rigor within the curriculum. When asked specifically about transportation, a concern of SMART/THRIVE program heads, the majority (84%) of parents indicated that it was not a problem. Of the parents who did indicate that transportation was a problem, inconsistency of service was cited as the primary issue. The majority of parents felt comfortable sending their children to the programs during the summer and/or on Saturdays, particularly because transportation on those days/during that time did not conflict with work or other activities that typically dominated school days. 8Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 11 of 69 Evaluation of SMART/THRIVE In the Little Rock School District Research Report: 2004-2005 INTRODUCTION The Arkansas Department of Education contracted with the Center for Research in Educational Policy (CREP) at The University of Memphis to collect formative evaluation data in schools implementing SMART/THRIVE. This report summarizes the 2004-2005 evaluation study of the SMART and THRIVE programs. The global purpose of the evaluation was to examine the extent to which the SMART/THRIVE programs have been effective in improving and remediating academic achievement of African-American students. Additionally, the evaluation explores perceptions of participants, parents and teachers as well as level of student participation in SMART and THRIVE. The overall purposes of the SMART and THRIVE programs are to provide supplemental pre-algebra support to students entering Algebra I and to prepare students to meet state standards in Algebra I. It was designed as an intervention for rising and current 8'"- and 9-grade students who are entering or will be enrolled in Algebra I. Applications for SMART are disseminated through the Sth grade teachers for any rising 9th grade student who has not taken Algebra I. Eligible rising 9th grade students who are registered to attend high school in the LRSD and will take Algebra are selected on a first come-first served basis. Seventh grade math teachers are asked to recommend students, particularly African-American or Hispanic students, who have the potential to be successful in Algebra I in the Sth grade. Students who participated in SMART are invited to participate in THRIVE. After a designated enrollment period, remaining seats are opened to any student taking Algebra I. Applications for these remaining seats are disseminated by the Algebra 1 teachers. The programs intentions/goals are to: Provide a solid foundation for Algebra, encourage exploration Make mathematics fun Enable students to achieve academic success in mathematics Create the confidence necessary to pursue higher level courses The goals encompass two components: pre-algebra instruction for two weeks during the summer (SMART Program) and 10 Saturdays across the school year (THRIVE Program). Instruction in both 9Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 12 of 69 programs utilizes small class sizes (fewer than 18) with two teachers per class. Each program currently (2004-2005) engages approximately 10 percent of the total African-American student population enrolled in Algebra I classes. Various local grants have funded both programs since 1999. Summer Mathematics Advanced Readiness Training (SMART) provides opportunities for students to gain the knowledge, skills, and confidence needed for success in Algebra 1. SMART is a two-week half-day summer institute for eligible rising 8'" and 9' grade students who will be enrolled in Algebra 1. SMART serves approximately 200 students each year. Each SMART classroom consists of a qualified mathematics teacher, one high school mentor who demonstrates outstanding mathematical skills and a positive attitude, and no more than seventeen SMART students. Each teacher uses a variety of strategies and tools to enhance their instructional delivery. Activities include real-life situations requiring the use of math skills. SMART students are exposed to and become familiar with a range of technological tools. The most popular tool used is the Tl- 84 Plus graphing calculator. SMART students learn how to use the calculator to further and deepen their knowledge of mathematics. In addition to the TI-84 Plus calculator, students use the Calculator-Based Laboratory (CBL) and the Calculator-Based Ranger (CBR) to solve problems and perform experiments. Students who successfully complete the SMART program receive a free TI-84 Plus graphing calculator, a SMART T-shirt, and a certificate. Instructional delivery and structure within the THRIVE program is virtually identical to that noted within SMART sessions
however, students do not have a high school mentor acting as a facilitator. THRIVE serves approximately 150 students, many of whom generally have been enrolled in SMART during the previous summer. Students currently enrolled in Algebra I meet bi-monthly on Saturdays during the Spring semester from 8:45 a.m. 12:25 p.m. The sessions are divided into 45-minute instructional blocks with learning activities prescribed for each block. The last block consists of two classes competing in a mathematics game/competition featuring the material covered that day. Teaching methods mirror those used in SMART, but integrate more advanced content and focus on material covered in Algebra I classrooms across the district. RESEARCH QUESTIONS Seven research questions provided the structure around which the evaluation was developed. The primary research question focused on the extent to which SMART/THRIVE programs improved and/or remediated math achievement among African American students. A substantive supplemental question 10Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 13 of 69 addressed the level of participation in SMART and THRIVE by African-American students. Additional attention was focused upon academically focused time and student engagement as well as parent, student, teacher and Algebra I teacher perceptions of the programs. The questions are listed below and are followed by a brief explanation of the areas addressed in the present evaluation. PRIMARY EVALUATION QUESTION Have the SMART/THRIVE programs been effective in improving and remediating the academic achievement of African-American students? A treatment (2-levels)-control student pre-tesVpost-test designed was employed for this evaluation. The analysis controlled for pretest, gender, ethnicity and SES. Four types of Algebra I students were compared: a) no program/comparison, b) SMART program only, c) THRIVE program only and d) both SMART and THRIVE programs. The pretest data was collected from the 2003-2004 Math Benchmark Test and the posttest data was collected from the 2004-2005 Math Benchmark Test. Additional test information gathered for treatment-control comparison includes the 2004-2005 ITBS Math subtests and Algebra I End of Course Exams (EOC). Descriptive data from SMART and THRIVE participants were derived from pre- and post-test program exams. All standardized tests were administered by and throughout the Little Rock School District. Pre- and post- test program exams were administered through SMART and/or THRIVE. SUPPLEMENTAL EVALUATION QUESTIONS 1. What is the level of participation in SMART and THRIVE by African American students? Attendance data from the SMART program of 2004 and the THRIVE program of 2005 were examined. In addition to descriptive information, the levels of participation were analyzed in light of student achievement. 2. What instructional strategies are used during the tutoring sessions? This question was addressed through classroom observations using an instrument focused on 26 research-based teaching strategies associated with increased academically focused instructional time and technology use within the classroom. Three additional math-specific items were designed to gather observation data about math-specific instructional practice. Five, random observation visits were conducted during the THRIVE program sessions. 11Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 14 of 69 3. What are the perceptions of SMART/THRIVE Teachers regarding program impacts, strengths, and weaknesses? 4. What are the perceptions of Algebra I teachers regarding program impacts, strengths, and weaknesses? 5. What are the perceptions of participating students regarding program impacts, strengths, and weaknesses? 6. What are the perceptions of parents/guardians of SMART/THRIVE students regarding program impacts, strengths, and weaknesses? Questions 3, 4, 5 and 6 were addressed through use of self-report surveys, including objective and open-ended items. Demographic data was also collected and analyzed. In addition, three student focus groups were created during which students perceptions of program strengths
weaknesses and overall satisfaction were collected, and then analyzed. PARTICIPANTS Currently, SMART/THRIVE serves students from the following eight Little Rock School District middle schools: Cloverdale: 82% African-American student population
Dunbar: 61% African-American student population
Forest Heights: 77% African-American student population
Henderson: 82% African-American student population
Mabievale: 81% African-American student population
Mann: 52% African-American student population
Pulaski Heights: 57% African-American student population
Southwest: 94% African- American student population. Of 274 eighth- and ninth-grade students who participated in SMART, Thrive, or both programs, matching district demographic data were available for 258 (94.2%). Tables 1 and 2 provide summaries of demographic and grade-level data for the various subsamples of SMART, THRIVE, and Comparison students in the study. These programs served a higher percentage of African American students than existed in the student population at large (69.0%): 88.0% of SMART students, 84.8% of Thrive students, and 89.6% of students attending both programs were African American. Of the 258 students with matching data, 125 attended SMART only, 66 attended Thrive only, and 67 attended both programs. Most students attending SMART only were ninth graders (83.2%), while the majority attending Thrive only were eighth graders (66.7%). About equal percentages of eighth graders (53.7%) and ninth graders (46.3%) attended both programs. Whereas the district percentage of female students in eighth and ninth grades was 50.1%, the 12Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 15 of 69 programs served a much larger proportion of females: 63.2%, 66.7%, and 59.7% for SMART, Thrive, and both programs, respectively. Table 1 Percentage of Students Served by Race: SMART^hrive and Comparison PROGRAM Am Indian Asian Black Hispanic Other White Total n Comparison 0.2 1.7 69.0 4.8 1.4 22.9 3726 SMART 0.0 0.0 88.0 0.8 4.0 7.2 125 THRIVE 0.0 6.1 84.8 3.0 0.0 6.1 66 BOTH 1.5 1.5 89.6 4.5 0.0 3.0 67 Total 0.3 1.7 70.2 4.6 1.4 21.8 3984 Note. Comparison group is all eighth and ninth grade students not served by either the SMART or Thrive programs. Table 2 Program Attended by Grade Level PROGRAM Total GRADE 08 09 Total Count % within PROGRAM Count % within PROGRAM Count % within PROGRAM Comparison 1727 SMART 21 THRIVE 44 BOTH 36 1828 46.3% 16.8% 66.7% 53.7% 45.9% 1999 104 22 31 2156 53.7% 83.2% 33.3% 46.3% 54.1% 3726 125 66 67 3984 100.0% 100.0% 100.0% 100.0% 100.0% EVALUATION DESIGN The evaluation was conducted during the spring of 2005. A mixed-methods design was employed to address the research questions. Results are primarily descriptive in nature
inferential statistics from achievement analyses are discussed. Both quantitative and qualitative data were collected by trained observers. The observers administered and collected surveys to THRIVE teachers and students and conducted focus groups with students and SMART mentors. The majority of the time was spent observing 13Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 16 of 69 classrooms using the School Observation Measure (SOM) described below. Observers also conducted focus groups with students and SMART mentors and administered or distributed surveys to students, teachers. Algebra I teachers and parents. Additionally, achievement data from students participating in SMART, 2004 and THRIVE, 2005 was collected and analyzed by CREP. Details about all of the instruments and evaluation procedures are provided in subsequent sections INSTRUMENTATION To address the questions proposed in this mixed-methods evaluation, data from achievement tests, student records, classroom observations, surveys, and focus groups were collected. Six measurement strategies were used to collect the evaluation data
standardized achievement test scores, classroom observations, surveys, focus groups, and program-developed achievement benchmarks. ACHIEVEMENT TEST SCORES Math Benchmark Test 2003 - 2004 and 2004 - 2005: The Benchmark Test is a state-mandated criterion-referenced test aligned with state math standards and developed by Arkansas teachers and the Arkansas Department of Education. Benchmark scores are reported as advanced, proficient, basic, and below basic. The mathematics portion of the exam is comprised of multiple-choice and open response items. Internal consistency reliability estimates, from the 2002-2004 administrations ranged from Cronbachs Alpha of .84 - .87. End of Course Exams (EOC) 2005: The Algebra I End of Course Examination is a criterionreferenced test with questions that align with the goals and subject-specific competencies described by the Arkansas Mathematics Curriculum Framework and Algebra I Course Goals. The purpose of the test is to assess student mastery of Algebra I knowledge and skills. Students take the exam at the completion of Algebra I. Scores are reported as advanced, proficient, basic, and below basic. Internal consistency reliability estimates, from the 2001-2004 administrations ranged from Cronbachs Alpha of .81 - .88. ITBS Math Subtests
The Iowa Test of Basic Skills (Math) assesses students ability to compute, deal with math concepts and solve problems dealing with numeration, graph/table interpretation, and 14Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 17 of 69 computation. The ITBS test is a norm-referenced. The ITBS information is proprietary and, therefore, estimates of reliability and validity could not be obtained. SMART and THRIVE Pre/Post Tests
These curriculum-based tests measured students' ability to utilize algebraic concepts to solve equations, judge inequalities, evaluate and simplify basic problems, graph equations, plot data, and solve applied problems. The THRIVE program tests were comprised of content more advanced than that of SMART program tests. Pre-test problems were identical to post-test problems in both programs. STUDENT RECORDS Attendance records from the 2004 SMART program and 2005 THRIVE program were examined in comparison to achievement measures. CLASSROOM OBSERVATION MEASURES The data collection instruments for classroom observations were the School Observation Measure (SOM), including three math-specific items designed to better-tailor the instrument, and the Rubric for Student-Centered Activities (RSCA). The SOM and addendum items were designed to gather information about instructional practices and classroom activities. The RSCA was used to gather more detailed information about the level of student engagement during target activities throughout target observations. The instruments used for classroom observation are described below. School Observation Measure (SOM): School Observation Measure (SObP). The SOM was developed to determine the extent to which different common and alternative teaching practices (e.g., direct instruction, cooperative learning, student inquiry, experiential learning) are used by teachers (Ross, Smith & Alberg, 1999) in typical (non-evaluative) classroom contexts. During each recording session, notes were completed every 15 minutes in different classrooms for a total of four at the end of one, 60-minute observation period. Researchers objectively recorded the relative use, non-use and frequency of 26 observation items covering a variety of classroom practices. Additionally, researchers recorded the extent to which high academically focused instructional time and high student attention/interest were observed. At the conclusion of a one-hour targeted visit, a trained observer summarizes the frequency with which each of the strategies was observed, yielding one SOM Data Summary Form. The frequency is recorded via a 5-point rubric that 15Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 18 of 69 ranges from (0) Not Observed to (4) Extensively Observed. Two global items rate, respectively, the academically-focused instructional time and degree of student attention and interest. A reliability study by Lewis, Ross, and Alberg (1999) found that pairs of trained observers selected identical SOM categories 67% of the time and rated within one category 95% of the time. Math Addendum: Because the program under evaluation was specifically a mathematics program, three additional observations items unique to mathematics were developed. The items were written by a member of the research team with an Arkansas teaching certificate in mathematics based on the standards of the National Council of Teachers of Mathematics (NCTM). The three items addressed the teachers actions in emphasizing conceptual understanding, connecting the content to daily life, and promoting students confidence in mathematics. Each item was rated as 1 indicating not observed to 5 reflecting strong application. Rubric for Student-Centered Activities (RSCA): This Rubric (Ross & Lowther, 2002) is applied in conjunction with SOM visits to determine the quality and depth of teacher applications of selected strategies (e.g., cooperative learning, higher-order questioning, project-based learning, and technology as a learning tool). These strategies reflect emphasis on higher-order learning and attainment of deep understanding of content. Such learning outcomes seem consistent with those likely to be engendered by well-designed, real- world linked exercises, projects, or problems utilizing technology as a learning tool. Each item includes a two- part rating scale. The first is a four-point scale, with 1 indicating a very low level of application, and 4 representing a high level of application. The second is a Yes/No option to the question: Was technology used? with space provided to write a brief description of the technology use. The RSCA was completed as part of SOM observation periods. The RSCA reliability results indicate that observer ratings were within one category for 97% of the whole-school observations and for 90% of the targeted observations (Sterbinsky & Burke, 2004). A RSCA was completed at the end of each targeted classroom observation. SURVEYS Four surveys were developed by the evaluation team to collect data pertaining to the effectiveness of the SMART/THRIVE Program. The specific audiences around which the current evaluation focused were parents. Algebra I teachers currently teaching in Little Rock schools, SMART/THRIVE teachers, and students 16Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 19 of 69 currently enrolled in THRIVE. Items were designed to address both the primary and secondary research questions of the study and to glean open-ended perceptions from the four data collection groups. Items specific to SMART and THRIVE were included as were open-ended questions about both programs. Program-specific items were presented to gain general and specific knowledge about both programs as separate entities. This also eliminated the need for respondents to complete two separate surveys. Each survey was divided into three sections. The first section was used to collect demographic information specific to each group. The second section was comprised of objective items specific to the research questions and each program (SMART and THRIVE), in the final section, the respondents were presented with four, open-ended questions which probed strategies, strengths, weaknesses and general additional comments for each program. Responses across all surveys were scored through the use of Likert- type ratings ranging from strong disagreement (1) to strong agreement (5). Demographic information was also collected. Drafts of all surveys were presented at the evaluation team meetings in Little Rock and revisions were made from team participants suggestions. Sun/eys are found in Appendix A. FOCUS GROUPS Two sets of focus group protocols were developed by the evaluation team. Students were asked their perceptions of the effectiveness of the SMART/THRIVE program. SMART mentors also were asked their impressions of program effectiveness. Additionally, participants were also asked to report their perceptions of program impact. Student and Mentor protocols were similar, but tailored to their specific audience. Question drafts were presented at an evaluation team meeting in Little Rock and revisions were made from team participants' suggestions. PROCEDURES Data for this evaluation were collected during the spring of 2005. Targeted observations were conducted during late spring, 2005 using the SOM (including 3 math-specific addendum items) and RSCA instruments. Surveys, focus groups and program-designed achievement tests were also administered during this time period. The classroom observations were conducted during February and March, 2005. Observations were scheduled for random classrooms participating in the THRIVE program on three different days. Student performance measures were administered in late spring, collection procedure. Table 3 summarizes the data 17Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 20 of 69 Table 3 Data Collection Summary Type of Measure Instrument Targeted Classroom Observations SOM RSCA Number Completed/Oata Source Total 5 19 Description Surveys SMART/THRIVE Teacher Parent 10 SMART/18 THRIVE 21 SMART/35 THRIVE 25 SMART/33 THRIVE Algebra I Teacher 70 SMART/142 Student THRIVE Focus Groups Student Focus Group Mentor Focus Group 21 Participants 2 Participants Achievement Tests Attendance Data Math Benchmark Test 2003 - 2004 and 2004 - 2005 End of Course Exams ITBS SMART 2004 Attendance THRIVE 2005 Attendance 210 students 143 students Note: One SOM session was cut short due to scheduling, yielding 3 observations sheets and 3 FiSCAs. Observation sessions in which random THRIVE classes were observed. During the observation period, note forms were completed every 15 minutes. Four* sets of observation notes were completed for each SOM. Completed during Spring, 2005 THRIVE sessions. Four RSCAs were completed for each SOM._____________ Items specific to SMART and THRIVE were included in the surveys and administered to all groups in Spring, 2005. THRIVE student surveys were distributed in class and collected by researchers. THRIVE teacher surveys were distributed during a faculty meeting and administered by researchers. THRIVE parent surveys were distributed by teachers via students, returned to the school and sent to researchers. Algebra I teacher surveys were distributed through the district, returned to the school and sent to researchers or sent directly to researchers._________________________ During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups. Each focus group interview lasted approximately 45 minutes. Researchers used the Student Focus Group protocol to conduct an on-site, 45- minute focus group with 2 of the 7 mentors. Researchers used data collected from three major math measures designed by independent entities and distributed throughout the LRSD for achievement data analysis. Exams were taken during the Spring of 2005._______________________ Researchers used the attendance and demographic data from SMART, 2004 and THRIVE, 2005 to examine level of participation among African American students. Data were collected from program heads and sent to researchers. 18Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 21 of 69 ACHIEVEMENT TESTS End-of-course (EOC) Algebra I test scores were used as 2004-05 outcomes for g** grade students, with 2003-04 Benchmark Mathematics scale scores used as a covariate. No covariate was available for 8' grade students, because no district or state tests were administered to graders in 2003-04. The End of Course Exams were administered at the end of the academic year to all students enrolled in Algebra I during the 2004-2005 school year. The ITBS exam was administered only to O' grade students, thus outcomes for this exam were available only for 9* graders. With no S grade scores available, 2003-2004 Benchmark scores were used for covariate comparisons. Additionally, curriculum-based pre- and post- program were administered on the first and last day of both programs. Both pre- and post-program-designed exams exams were comprised of 20 items, asking students to simplify, evaluate, graph, and solve equations. The SMART exams were geared to test pre-algebra and early algebra content and the THRIVE exams were designed to examine content expected of students completing Algebra I. The pre-program exams and post-program exams were identical. These exams are provided in Appendix B. CLASSROOM OBSERVATIONS Trained observers completed a series of classroom visits to collect frequency data regarding observed instructional practices, use of technology and level of student engagement. Classroom teachers were advised that the observers would be present throughout the semester, but were instructed to deliver lessons as usual. A targeted procedure was used for the present study given that the observations were of the THRIVE sessions rather than of a school-wide, cross-categorical program. The majority of the observations were conducted by one researcher, with the second researcher observing for one SOM to examine inter-rater reliability. The inter-rater reliability for the joint observation was high
observers selected identical SOM categories 88% of the time and rated within one category 100% of the time. The standard SOM procedure involves evaluation of an entire school by visiting 10-12 randomly selected classrooms for 15 minutes each. However, the goal of this evaluation was to examine the practices within a single program, where all classrooms were participating in the same activities at the same times, not assess an entire school. Therefore, procedures were modeled from those used by Lowther, Ross, and Morrison (2003) in their examination a school laptop computer program. A total of five SOM observations were conducted. Both the RSCA and the Math Addendum items were completed at the same time. A total of 19Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 22 of 69 19 RSCAs were completed because one observation session was interrupted by a scheduled class dismissal. SURVEYS The program teacher surveys were administered and collected by researchers on March 5, 2005 during a faculty meeting. Student surveys were administered and collected by researchers during a March 5 class. Algebra I teacher surveys were distributed via district mail. Parent surveys were distributed through students and collected by teachers before being sent to CREP for analysis. Additional parent surveys were administered and collected by a member of the research team at the student Math Fair. FOCUS GROUPS During spring 2005, student and mentor focus groups were conducted at the THRIVE site by researchers. All students returning parental permission slips were included in the focus groups, each of which lasted 45 minutes. The mentors were contacted individually by researchers via phone, mail and email. Two of the seven mentors responded and attended. The mentor focus group interview lasted approximately 45 minutes. RESULTS The results of this evaluation are presented below and synthesized to address each research question in the Discussion section. These results are presented by measurement strategy
however, the findings are synthesized across instruments to reflect each research question in the Discussion section of this report. SCHOOL OBSERVATION MEASURE (SOM) As noted previously, observers focused on the 26 instructional strategies provided in the SOM using a standard five-point rubric (0 =not observed, 1 = rarely, 2 = occasionally, 3 = frequently, and 4 = extensively). The results are presented with a full, categorical breakdown in Table 4. Four strategies were observed occasionally to extensively at a frequency level of at least 75% or higher. These strategies include team teaching (100%), cooperative/collaborative learning (100%), use of higher-level questioning strategies (80%), and teacher acting as a coach/facilitator (100%). The data indicates that students were engaged in collaborative learning situations with high teacher support where they were asked to use problem solving and critical thinking skills to solve problems. Strategies that were observed occasionally 20Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 23 of 69 to extensively during 40-60% of the sessions included direct instruction (lecture), higher level instructional feedback (40%), experiential hands-on learning (40%), independent inquiry or research (40%) and student discussion (60%). Student discussion heavily dominated the instructional context (60% observed frequency) with teams of students working together to solve problems and propose new ideas. Some individual seat work and independent research was observed (40% each respectively), however, these frequently were noted to lead into a group-based activity in which individuals proposed ideas to a team that was used to create a corporate answer to problems proposed by the teachers. Sustained reading was observed in one classroom. Technology was heavily used (80%) as teachers relied upon graphing calculators and problem solving with technology as an integral part of the curriculum. Teachers demonstrated different ways to solve problems and/or work towards solutions via calculator and multi-media use. High academically focused time was observed extensively (80% of the time) as was high student engagement (in 60% of the observation sessions). Since the evaluation context was an Algebra I Saturday supplementary program with specific model parameters, some instructional strategies within the SOM were not observed. These strategies included individual tutoring, ability and multi-age groupings, work centers, integration of subject areas, project-based learning, independent seatwork, sustained writing/composition and systematic individual instruction. Generally, the aggregated data reflect efficient use of class time with teachers using a team teaching approach to deliver instruction to collaborative teams of students. The aforementioned math addendum items served to capture specific strategies used frequently among math teachers. Results from observations showed that THRIVE teachers frequently emphasized conceptual understanding over rote learning 60% of the time observed. Teachers also demonstrated connections between math and daily life frequently in at least 60% of the observations. One additional strategy of high interest was teachers cultivation of students confidence. flexibility and inventiveness in doing mathematics. This was observed frequently among 80% of the observations. None of the math-specific strategies were observed extensively. 21Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 24 of 69 Table 4 School Observation Measure (SOM) Data Summary for Little Rock-SMART/THRIVE Project N = 5 The extent to which each of the following was used or present in the school..._______ Instructional Orientation Direct instruction (lecture) Team teaching Cooperative/collaborative learning Individual tutoring (teacher, peer, aide, adult volunteer) Classroom Organization Ability groups Multi-age grouping Work centers (for individuals or groups) Instructional Strategies Higher-level instructional feedback (written or verbal) to enhance student learning Integration of subject areas (interdisciplinary/thematic units) Project-based learning Use of higher-level questioning strategies Teacher acting as a coach/facilitator Parent/community involvement in learning activities Student Activities Independent seatwork (self-paced worksheets, individual assignments) Experiential, hands-on learning Systematic individual instruction (differential assignments geared to individual needs) Sustained writing/composition (self-selected or teacher-generated topics) Sustained reading Independent inquiry/research on the part of students Student discussion Technology Use Computer for instructional delivery (e.g. CAI, drill & practice) Technology as a learning tool or resource (e.g., Internet research, spreadsheet or database creation, multi-media, CD Rom, Laser disk) Assessment Performance assessment strategies Student self-assessment (portfolios, individual record books) Summary Items High academically focused class time High level of student attention/interest/engagement Percent None Percent Rarely Percent Occasionally Percent Percent Frequently Extensively 20.0 0.0 0.0 100.0 100.0 100.0 100.0 60.0 100.0 100.0 20.0 0.0 80.0 100.0 20.0 100.0 100.0 8Q.0 40.0 40.0 80.0 0.0 100.0 100.0 0.0 0.0 Wole
One SOM is comprised of approximately 8 classroom visits. 20.0 0.0 0.0 0.0 20.0 20.0 0.0 0.0 40.0 20.0 20.0 0.0 0.0 60.0 80.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 0.0 40.0 0.0 0.0 0.0 20.0 0.0 0.0 20.0 0.0 0.0 0.0 0.0 40.0 0.0 0.0 20.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 0.0 20.0 20.0 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 40.0 0.0 0.0 0.0 60.0 60.0 0.0 0.0 0.0 0.0 0.0 0.0 20.0 20.0 0.0 20.0 0.0 0.0 20.0 40.0 0.0 40.0 0.0 0.0 0.0 20.0 20.0 0.0 40.0 0.0 0.0 80.0 60.0 22Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 25 of 69 RUBRIC FOR STUDENT-CENTERED ACTIVITIES (RSCA^ The RSCA results reflect the percentage of observed sessions in which selected strategies are observed at least once. The quality of the application and the percentage of sessions in which technology was used with the observed strategy are also recorded. A descriptive summary of the RSCA results is presented in Table 5. The RSCA results suggest that the level of application with which THRIVE program teachers applied certain instructional practices, particularly with regard to cooperative learning and higher-level questioning strategies was high. Additionally, students evidenced appropriate use of technology (calculators, multimedia, and spreadsheets) in self-directed activities to create new knowledge. The most notable area of technology use was among cooperative learning groups, where students collaboratively utilized multimedia, calculators and graphing applications to solve problems
teachers often helped direct collaborative groups. Project-based learning was not observed and although student discussion, experiential hands on learning, and independent research were applied, the level of application was limited or somewhat limited without technology use. Additionally, aggregate results suggest that technology was used in more than a third of the instances during which 5/7 remaining student-centered activities were observed and in over 25% of the instances during which 6/7 remaining activities or strategies were applied. 23Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 26 of 69 Table 5 Rubric for Student-Centered Activities (RSCA) N = 19 Student-Centered Activities Not Limited Somewhat Somewhat Cooperative Learning Project-Based Learning Higher-Level Questioning Strategies Experiential Hands-on Learning Independent Inqulry/Ftesearch Student Discussion Students as Producers of Knowledge Using Observed Application Limited Strong Application Application Strong Application Technology In Use * 8.7 13.0 8.7 21.7 47.8 60.9 100.0 26.1 56.5 56.5 56.5 30.4 0.0 0.0 0.0 0.0 0.0 0.0 4.3 8.7 17.4 8.7 Technoiogy Percentages Indicating Observed Levels of Application Rating Scale: 1- limited application
4=strong application 30.4 39.1 4.3 26.1 4.3 0.0 8.7 21.7 21.7 30.4 17.4 39.1 13.0 39.1 4.3 0.0 0.0 39.1 30.4 * See addendum description of technology use SCHOOL OBSERVATION SUMMARY Analysis of data collected using SOM, the Math Addendum and RSCA during the observations revealed that THRIVE teachers are using a variety of instructional strategies aligned with Arkansas state- designed SLEs. The most striking result was the extent to which team teaching and collaborative/cooperative learning dominated the curriculum. Additionally, notable use of higher level questioning strategies and teachers acting as coaches rather than lecturers was noted among observed sessions. During the teaching sessions that were observed, the Math Addendum items revealed that teachers were frequently emphasizing conceptual understanding over rote learning. Furthermore, this practice facilitated frequent instances of promoting students confidence and flexibility in doing mathematics. Overall, students in the THRIVE program had a high level of interest and were observed being engaged in instnjction that emphasized practical and conceptual learning in a comfortable, student-centered setting. 24Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 27 of 69 SURVEY RESULTS Four surveys (Algebra I Teacher, Student, Parent, SMART/THRIVE Teacher) were administered in late March, 2005. Results for each characteristics of the respondents demographic section of the surveys. of the four instruments were examined independently. Demographic are representative
however, not all respondents completed the Additionally, the total number of respondents to THRIVE items varied from the total number of SMART respondents across questionnaires. Demographic differences are described for each group. Finally, item percentages across surveys did not always total 100% because of missing input from some of the respondents (i.e., there were questions to which some people did not respond). PROGRAM PARTICIPANT SURVEY THRIVE A total of 142 students responded to items pertaining to THRIVE and 70 students responded to items pertaining to SMART'. Of the 138 THRIVE respondents who reported their ethnicity, 77% specified African- American, 7% specified Multi-Ethnic, and 4.2% specified Asian. The remaining categories of Caucasian (3.5%), Hispanic (2.8%), and Other (2.1%) were specified less frequently. A total of 63.4% of the respondents were female
35.9% of the respondents were male. Of the 70 students responding to SMART questionnaire items and specifying demographic information, 38.6% were female and 61.4% were male. Regarding ethnicity, 82.9% of the SMART respondents specified African American, 4.3% specified Caucasian or Hispanic respectively, 2.9% specified Multi-Ethnic and 1.4% specified Other or Asian. Table 6 contains the results for THRIVE and SMART objective survey items. These results are summarized in text below. Summaries of the open ended questions are shown in subsequent Tables. The overall trend in response types was positive
fifty percent or more students indicated agreement or strong agreement across all items. Well over eighty percent of students responding to the survey reported that team competitions made THRIVE classes more fun. Over seventy-five percent of respondents agreed or strongly agreed that THRIVE made Algebra more enjoyable and instilled confidence in doing well on the Algebra Benchmark Exam. Over sixty percent of student respondents agreed or strongly agreed that THRIVE improved their self-confidence about math, helped them feel comfortable asking questions in THRIVE class. Because some students participating in the THRIVE program did not participate in SMART during the previous summer, differences between respondent numbers were expected. 25Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 28 of 69 and facilitated problem solving through calculator use. A total of 66.6% of respondents indicated that THRIVE teachers helped them with problems they were having in their regular algebra class. At least one half (50.0%) of the students responding to the survey indicated that THRIVE helped improve their algebra grades and helped them understand how to apply algebra concepts in real life. In contrast to the positive trend in responses across items, two areas in which some students showed disagreement were related to generalization and application. This is somewhat consistent with responses noted within the open ended portion of the student survey where students indicated an overall satisfaction with the program but did not feel that the program changed their study habits or helped them apply information to real life situations. Additionally, more students responded neutrally to items pertaining to content, grades and application when compared to responses for items related to motivation, fun and self-confidence. Table 6 Student Survey - THRIVE N = 142 1. 2. 3. 4. 5. 6. 7. 8. 9. THRIVE items Because of THRIVE, I have learned how to use a calculator to help solve algebra problems. THRIVE makes Algebra more enjoyable. THRIVE has helped me get good grades in Algebra. I think I will do well on the Algebra Benchmark Exam because of THRIVE. In THRIVE, I have learned how algebra can be used in real life. THRIVE has made me more confident about math. My THRIVE teacher helps me with problems I am having in my algebra class. I feel comfortable asking questions in THRIVE class. Team competitions make THRIVE classes fun. Percent Strongly Agree/ Agree 67.6 77.5 50.0 78.9 55.6 66.2 66.2 74.7 87.3 Percent Neutral 23.9 20.4 40.9 16.2 33.8 24.7 25.4 21.1 6.3 Percent Strongly Disagree/ Disagree 8.5 2.1 8.5 4.2 10.6 7.8 8.5 4.2 2.8 26Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 29 of 69 SMART Responses indicating agreement or strong agreement for items pertaining to the SMART portion of survey were consistently above 75%. Of the 70 respondents, over 85% agreed or strongly agreed that SMART helped prepare them for Algebra I in the fall, and 80% agreed or strongly agreed that SMART made algebra more fun. Seventy-eight percent agreed or strongly agreed that the SMART program helped them remember math skills from the previous year. Students agreed or strongly agreed that they were motivated to go to their SMART classes and that the program helped them apply math to solve real-life problems across 75% of the surveys. Fewer students indicated neutral or negative responses for the SMART survey items than noted among THRIVE survey responses. Of those that indicated disagreement or strong disagreement. the item noted most frequently (4.3%) was related to application (We learned how to use math to solve reallife problems.). Overall, the trend towards agreement with items probing the attributes of the programs was positive. Table 7 presents detailed results for the SMART student survey. Table 7 Student Survey -SMART N = 70 1. 2. 3. 4. 5. SMART Items SMART helped me remember the math skills I learned last school year. SMART made algebra more fun. In SMART, we learned how to use math to solve real-life problems. I was motivated to go to my SMART class during the summer. Because of SMART, I was prepared to begin Algebra I in the fall. Percent Strongly Agree/ Agree 78.6 Percent Neutral 18.6 Percent Strongly Disagree/ Disagree 2.9 81.4 75.7 75.7 88.6 14.3 15.7 18.6 5.7 1.4 4.3 1.4 1.4 OPEN-ENDED RESPONSES THRIVE students had an opportunity to respond to four, open-ended questions as part of the survey. Student impressions of the SMART program were polled across all students even though SMART was not in session and not all students participated in it during the previous summer. SMART and THRIVE were evaluated separately using the same questions to preserve the integrity of their differences. Student responses are summarized below. 27Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 30 of 69 Question 1: Students were asked to comment on what they liked about the THRIVE program. The majority of responses reflected an appreciation of the programs use of competition and a sense that the program made math more fun/enjoyable. Additional comments suggested that students enjoyed having a program that allowed for review and expansion of concepts taught in the regular Algebra I classroom. Students also noted that they enjoyed the teachers and felt comfortable asking for clarification when necessary. Additionally, snacks and rewards were notably cited as positive influences of THRIVE. Table 8 summarizes what students liked about THRIVE. TABLE 8 What do you like about THRIVE? Description Makes math fun Uses competition/games in Provides expansion/review of math concepts Clarifies/Remediates confusing concepts Teachers are good/likeable Snacks People/friends in THRIVE classes Class size/Format Money/Rewards NA - responses did not pertain to question Enjoy using a calculator Provides a head start for following week in class Improves self-confidence/provides comfortable place to learn Class Time/Format TOTAL Frequency 42 37 18 17 17 14 11 8 7 1 4 3 3 2 190 Percentage 22% 19% 9% 9% 9% 7% 6% 4% 4% 4% 2% 2% 2% 1% 100% Question 2: When asked what they would change about the THRIVE program, the overwhelming majority of students responded that they would make no changes. Students who did comment on needed changes mostly cited a later start time or a more compressed time period (e.g., two hours rather than four hours) as potential prospects. Non-specific time changes (e.g., change time, time, different time) were also noted among open-ended comments but coded separately due to the large number of students commenting on previously cited specifics about THRIVE hours. Table 9 provides an illustration of student comments. The miscellaneous category captured unique comments made by individual students (e.g., field trips, guest speakers, PE). 28Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 31 of 69 TABLE 9 Whaf would you change about THRIVE? Description No change Less time Later start time Miscellaneous Class format/more varied activities Longer breaks Non-specific time change Snacks/lunch Enhance competition More time More peers/neighborhood school Rewards Earlier time Teacher changes TOTAL Frequency 57 17 14 11 10 8 8 8 5 4 3 3 2 2 152 Percentage 38% 11% 9% 7% 7% 5% 5% 5% 3% 3% 2% 2% 1% 1% 100% Question 3'. Students were asked to comment on how THRIVE has helped them over the course of the semester. Most comments suggested that THRIVE helped students enhance comprehension of general Algebraic concepts. Many students also indicated that THRIVE helped improve their grades and improved their understanding of how to use a calculator to solve and graph equations. Additionally, students' comments suggested that THRIVE reinforced specific concepts (e.g., graphing equations with polynomials) and made the students feel that they were staying ahead in weekly Algebra classes. Of 150 responses, three indicated that THRIVE was not helpful
none indicated that it was detrimental. Table 10 summarizes the results. TABLE 10 If THRIVE has helped you, how has it helped you? Description Enhanced comprehension of Algebra Improved grades Use of calculator Reinforced specific concepts in Algebra Helps stay ahead in math Builds confidence/makes math more positive Miscellaneous (individual answers) Negative-Did not help TOTAL Frequency 63 20 18 16 14 8 8 3 150 Percentage 42% 13% 12% 11% 9% 5% 5% 2% 100% 29Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 32 of 69 Questions about SMART were analyzed. Response rates across SMART questions, although somewhat different, do not vary significantly. Question 4: When asked to identify the most appealing aspects of SMART, most comments were positive, yet nonspecific in nature. Examples included responses that alluded to liking everything about the program or how the program helped me (in general). A number of students (13%) noted that SMART prepared them for math in the fall and reduced anxiety when entering first year Algebra. Calculator use and team competition garnered notably high response rates (13% and 12% respectively). Some students made specific comments about the positive influence of the SMART teachers and others made comments about rewards delivered through the program. Response categories are provided in Table 11. TABLE 11 What do you like about SMART? Description Nonspecific positive responses (e.g.. Everything: How it helped me) More prepared for math/ reduced anxiety Using the calculator Team competitions The teachers Reinforcement and rewards Social aspects/visiting with peers from other schools The snacks Games and content delivery NA (e.g., I did not go to SMART
I do not know) Format/time/class size TOTAL ~ Frequency 31 11 11 10 5 5 5 3 2 2 1 86 Percentage 36% 13% 13% 12% 6% 6% 6% 3% 2% 2% 1% 100% Question 5: Students also provided information about what aspects of the SMART program they would change. Over sixty percent of the responses favored no change. Of the remaining responses, 23% indicated a time change as appropriate. Of those, the large majority of comments suggested a later starting time rather than a reduced time period. One student noted a desire to reduce the amount of time spent at SMART. A few students who did not participate in SMART simply indicated such
no additional comments were recorded among these respondents. The remaining responses indicated a desire to enhance the program by adding team competitions, extending the time during which SMART was available, and providing more applied skills for use in everyday situations. Data are summarized in Table 12. 30Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 33 of 69 TABLE 12 What would you change about the SMART program? Response No change Time Snacks NA (I did not go to SMART
I like Math) More team competitions More applied skills Extend time/day and/or hours TOTAL Frequency 41 15 3 3 2 1 1 66 Percentage 62% 23% 5% 5% 3% 2% 2% 100% Question 6: When asked about how SMART helped students, the ovenwhelming majority of comments identified the programs purpose in preparing students for Algebra I as most important. Many responses also indicated the use of a calculator as instrumental in helping students solve Algebra problems. Additionally, students noted that SMART helped improve their grades and built confidence in math. TABLE 13 If SMART helped you, describe how. Response Made me feel prepared Use of calculator Built confidence in math Improved grades NA(I did not go to SMART
I liked THRIVE) Reinforced specific concepts Miscellaneous (I would like field trips) Increased engagement/interest TOTAL Frequency 41 11 6 6 3 1 1 1 70 Percentage 59% 16% 9% 9% 4% 1% 1% 1% 100% PARENT SURVEY THRIVE A total of 35 parents responded to survey items pertaining to THRIVE. Of those, 57% were female and 40% were male. A total of 80% of parent respondents indicated that their child was African American, 8.6% specified Asian and 2.9% specified Caucasian, Hispanic and Other respectively. No parental respondents specified Multi-Ethnic as the ethnicity of their child. 31Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 34 of 69 Responses indicating agreement or strong agreement were consistently above 80% across all survey items. Overall, parents agreed or strongly agreed that THRIVE was a good program (94%), that they felt comfortable having their child attend classes on Saturdays (91.4%) and that they felt teachers in the program helped their children feel successful in math (91.4%). The neutral responses that were noted were made in response to items pertaining to childrens attitudes about math and improvement in grades. No negative responses to the items were reported. Table 14 provides a summary of the responses. Table 14 Parent Survey - THRIVE N = 35 1. 2. 3. 4. 5. 6. THRIVE items The teachers in this program make my child feel that he/she can succeed. Because of this program, my child is more motivated to complete algebra homework. This program has helped improve my child's attitude about math. I am comfortable having my child attend classes on Saturdays. Because of this program, I have seen an improvement in my child's Algebra I grades. Overall, I think this is a good program. Percent Strongly Agree/Agree 91.4 88.6 82.9 91.4 80.0 94.3 Percent Percent Strongly Neutral Disagree/ 5.7 8.6 17.1 5.7 14.3 2.9 Disagree 0.0 0.0 0.0 0.0 0.0 0.0 Percent Don't Know 0.0 0.0 0.0 0.0 0.0 0.0 SMART A slightly smaller number of parents (21) responded to the five survey items pertaining to SMART. Of those, 61.9% were female and 33.3% were male. Eighty-five percent of parental respondents cited their childs ethnicity as African-American. The remaining respondents indicated Asian (4.8%) or Other (4.8%) as their childs ethnicity. Ninety percent of parent respondents indicated agreement or strong agreement when asked whether SMART did a good job of preparing children for Algebra I in the fall and improving childrens perception that they could succeed in math. Slightly over 85% of respondents indicated agreement or strong agreement with items related to whether SMART improved their childs attitude about math and whether they felt comfortable sending their children to classes during the summer. A total of 76% of parents agreed or strongly agreed that 32Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 35 of 69 the program classes were easy for their children to attend. None of the respondents indicated disagreement or strong disagreement with any of the items. Of those that responded neutrally, the highest number (14.3%) were noted for the item pertaining to whether SMART had improved attitudes about math. No parents indicated negative responses for any of the items. Table 15 Parent Survey - SMART N = 21 1. 2. 3. 4. 5. SMART Items Because of this program, my child felt like he/she could succeed in Math. The summer classes were easy for my child to attend. This program helped improve my child's attitude about Math. I am comfortable having my child attend classes during the summer. This program did a good job of preparing my child for beginning Algebra I in the fall. Percent Strongly Agree/ Agree 90.5 76.2 85.7 85.7 90.5 Percent Neutral 4.8 4.8 14.3 0.0 4.8 Percent Strongly Disagree/ Disagree 0.0 0.0 0.0 0.0 0.0 Percent Don't Know 0.0 4.8 0.0 4.8 0.0 OPEN-ENDED RESPONSES Question 7
When asked to provide their impressions of THRIVE, many parents responses described it as a motivating math program (19%) targeting childrens math skills. Numerous comments suggested that parents characterize THRIVE as a program that improves general math skills (19%) and/or focuses on problem solving and/or critical thinking (15%). A number of parents comments also reflected an awareness of the programs use of motivating games and competition to foster interest in math (15%). The balance of the comments were positive and reflected an awareness of THRIVE as a supplemental alternative to the weekly Algebra I curriculum. 33Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 36 of 69 TABLE 16 Please tell us your impression of the THRIVE program. What do you see as taking place during the sessions? Description Helps child feel fulfilled/motivated about math Helps child improve math skills Focuses critical thinking and problem solving Integrates games and fun into math Non-specific positive commendation Practical application Provides individualized tutoring provides review of school curriculum Class format: small groups and team competition TOTAL Frequency 5 5 4 4 2 2 2 2 1 27 Percentage 19% 19% 15% 15% 7% 7% 7% 7% 4% 100% Question 2 and Question 3: Questions 2 and 3 addressed parents opinions of THRIVE. One third of the comments made about parents positive opinions indicated that it was motivating. Parents also noted that they liked how THRIVE provided additional help (17%) and made efforts to improve specific math skills (12%). Some parents reported global positive experiences and satisfaction with THRIVE (14%). Table 17 provides additional data regarding what parents liked about the program. Regarding parents' opinions about whether aspects of THRIVE should be changed, 29% 'O recommended no change and/or responded positively about the program. Of the comments requesting change, one third recommended expanding the breadth and length of the program. Specifically, these parents made note that more time (23%) should be allocated for THRIVE and that more students (10%) should be involved. Table 18 provides additional information about isolated comments for changing THRIVE. 34Case 4:82-cv-00866-\/VRW-JTR Document 3996-2 Filed 12/27/2005 Page 37 of 69 TABLE 17 What did you like about the THRIVE program? Description Motivating Provides additional help Non-specific positive commendation Improve math skills Class size, time, format Learning new strategies Games Parent gatherings Prepares for benchmark exam Teaches calculator skills_________ TOTAL ' Frequency 14 7 6 5 3 3 1 1 1 1 42 Percentage 33% 17% 14% 12% 7% 7% 2% 2% 2% 2% 100% TABLE 18 If anything, what would you change about the THRIVE program? Description No change Extend time/meet more frequently Expand program to more students More challenge Expand into summer Expand program to after school Get credit Later start Measurement of progress More communication between parents and teachers More funding More hands on learning More structure Transportation__________ TOTAL Frequency 9 7 3 2 1 1 1 1 1 1 1 1 1 1 31 Percentage 29% 23% 10% 6% 3% 3% 3% 3% 3% 3% 3% 3% 3% 3% 100% Two questions probed the operational aspects of the THRIVE program. Regarding notification and enrollment, parents comments indicated that their childrens schools (34%) and teachers (26%) were the primary sources of information about THRIVE. Children reportedly were additional sources of information (11%). A substantial percentage of parents also indicated that their childs participation in SMART the previous summer served as a source of information about THRIVE. 35Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 38 of 69 The majority of respondents suggested that transportation was not a problem. Of the four of respondents who viewed transportation as problematic, one indicated that bus services were not provided in her area. Two respondents noted that bus services were provided, but were inconsistent. Of the 28 respondents who did not identify transportation as a problem, none elaborated as to whether or not their children made use of the buses that were provided. Tables 19 and 20 provide frequency and percentage data for these questions. TABLE 19 How did you find out about THRIVE? Description Child's school Child's teacher Child SMART Child's friend District Math programs Mailings, flyers PTA Respondent is a THRIVE Teacher School Counselor TOTAL Frequency 12 9 4 3 2 1 1 1 1 1 35 Percentage 34% 26% 11% 9% 6% 3% 3% 3% 3% 3% 100% TABLE 20 IVas transportation a problem? Please explain. Description No Yes. Bus provided, but inconsistent No. Weekend no problem Yes. Yes. No bus provided TOTAL Frequency 27 2 1 1 1 32 Percentage 84% 6% 3% 3% 3% 100% Parents also had an opportunity to respond to questions regarding the SMART summer program. These questions were exactly the same as those asked about the THRIVE program. Bearing in mind that fewer students participated in SMART than in THRIVE, it was expected that fewer parents would respond to 36Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 39 of 69 questions regarding SMART. Three respondents indicated that their child did not participate in SMART
these three respondents did not answer the remaining questions. Parents impressions of the SMART program were positive. The majority of parents responding indicated that the program exposed their children to a variety of methods for solving Algebra problems (24%) and improved their motivation (19%) tor math. Five parents made very general positive comments (14%) about the program and/or indicated that the teachers were integral in motivating students (10%) to learn about Algebra. TABLE 21 Please tell us your impression of the SMART program. Description Use of variety of teaching methods Improving motivation for math NA (response indicated that child did not participate in SMART) Non-specific positive commendation Teachers making math enjoyable Description of time spent in program Improving grades in math Motivating Teachers making math enjoyable TOTAL Frequency 5 4 Percentage 24% 19% 14% 3 2 1 1 1 1 21 14% 10% 5% 5% 5% 5% 100% 3 Parents were asked to comment on whether or not they would recommend changes being made to the program. Results suggest that many parents desire to extend the duration of the program (43%) and would like to expand the program to more students (14%). The remaining 43% of respondents indicated that they would make no changes. Table 22 provides an illustration of the comments regarding change. TABLE 22 What, if anything, would you change about SMART? Description Extend duration (weeks) No change Expand to more students TOTAL Frequency 6 6 2 14 Percentage 43% 43% 14% 100% 37Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 40 of 69 Parents were asked how they found out about SMART and whether or not transportation posed a problem for them to examine operational needs of the program. Results from these questions are summarized in Tables 23 and 24. The majority of parents indicated that they were notified about SMART through their childs school (59%). Of these respondents, a specific person (e.g., general math specialist) or department was noted as a source. Adult peers (12%) also were among sources of information notably contributing to disseminating information about SMART. Five respondents (30%) indicated that specific school personnel notified them about SMART
of these, one was a SMART teacher. Transportation was not viewed as problematic. Of the respondents, three provided more specific information about how they traveled to and/or from SMART. None of the respondents indicated negativity toward bus or car. TABLE 23 How did you find out about the SMART program? Description Frequency Childs school Friends of parent Math Specialist at Childs School Childs teacher LRSD Math Dept Respondent is a SMART Teacher TOTAL 10 2 2 1 1 1 17 Percentage 59% 12% 12% 6% 6% 6% 100% TABLE 24 IVas transportation a problem? Description Frequency No No. Bus Provided No. Parents Drove TOTAL 13 2 1 16 Percentage 81% 13% 6% 100% 38Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 41 of 69 SMART/THRIVE TEACHER SURVEY THRIVE A total of 23 THRIVE teachers were asked to provide opinions and perceptions of SMART and THRIVE. Of those, 18 completed objective survey items for the THRIVE program. Seventeen of the respondents reported their ethnicity. Over sixty percent of these respondents specified their ethnicity as African-American
the remaining respondents identified their ethnicity as Caucasian (27%) or Other (5%). Regarding experience, the majority of the teachers responding to the survey had between 6-10 years of teaching experience and had worked with the THRIVE program for 3 or more years. Over 30% of respondents indicated having 1-2 years experience with THRIVE. Over 25% indicated having 5 or more years of experience with THRIVE. Teachers' responses overwhelmingly favored agreement or strong agreement with items pertaining to positive attributes among professional development, resources/materials, methodology and instructional delivery within the THRIVE program. Two items for which some teachers indicated neutrality were related to the extent to which they felt the program had improved their own ability or their students ability to use calculators. Overall, however, teachers responses suggest an overall positive perception of the program. This is consistent with the opinions shared from open-ended questions designed to glean similar information. Table 25 presents summary information from the objective items. 39Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 42 of 69 Table 25 SMART/THRIVE Teacher Survey - THRIVE N = 18 Please indicate the extent to which you agree with the following items: 1. I have received adequate professional development 2. 3. to implement this program. I have a thorough understanding of the objectives of this program. The students enjoy the planned activities. Percent Strongly Agree/ Agree 100.0 Percent Neutral 0.0 Percent Strongly Disagree/ Disagree 0.0 Percent Dont Know 0.0 4. 5. 6. 7. 8. 9. 10. The facility at which the classes are held is adequate. Materials (books, copies, equipment) needed for this program are readily available. The daily briefing and debriefing sessions are valuable. The program has improved my ability to use graphing calculators. The program has improved my students ability to use graphing calculators. The program is aligned with state and district math standards. The co-teacher (THRIVE) and Mentor (SMART) approaches are effective instructional methods for teaching and reinforcing Algebra concepts. 100.0 100.0 100.0 100.0 94.4 94.4 94.4 100.0 100.0 0.0 0.0 0.0 0.0 0.0 5.6 5.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 SMART Of the 18 surveys collected from THRIVE teachers, ten contained completed items pertaining to SMART. Demographic data from all ten suggests that this cohort consisted of teachers with more than 5 years of teaching experience. The majority of respondents reported between 6-10 years experience (60%). Teachers with 11-15 years experience comprised 20% of the respondent group and teachers with 16 or more years comprised the remaining 20% of the group. The majority of respondents indicated having four or more years experience with THRIVE (70%) and all respondents indicated having 4 or more years with SMART (100%). With regard to ethnicity, the majority of respondents specified African American (70%). The remaining 30% specified Caucasian. Respondents indicated 100% agreement or strong agreement across eight of the ten items. For two items probing teachers perceptions of how SMART improved their ability or their students ability to use graphing calculators, total agreement was noted among 90% of the surveys. The remaining 10% of the survey 40Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 43 of 69 respondents indicated neutral responses for these two items. None of the respondents indicated disagreement or strong disagreement for any of the items. Table 26 summarizes this data. Table 26 SMART/THRIVE Teacher Survey - SMART N = 10 1. 2. 3. 4. 5. 6. 7. 8. 9. Please indicate the extent to which you agree with the following items: I have received adequate professional development to implement this program. I have a thorough understanding of the objectives of this program. The students enjoy the planned activities. Percent Strongly Agree/ Agree 100.0 Percent Neutral 0.0 Percent Strongly Disagree/ Disagree 0.0 Percent Don't Know 0.0 10. The facility at which the classes are held is adequate. Materials (books, copies, equipment) needed for this program are readily available. The daily briefing and debriefing sessions are valuable. The program has improved my ability to use graphing calculators. The program has improved my students' ability to use graphing calculators. The program is aligned with state and district math standards. The co-teacher (THRIVE) and Mentor (SMART) approaches are effective instructional methods for teaching and reinforcing Algebra concepts. 100.0 100.0 100.0 100.0 100.0 90.0 90.0 100.0 100.0 0.0 0.0 0.0 0.0 0.0 10.0 10.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 OPEN-ENDED RESPONSES In addition to closed-ended items, THRIVE Teachers also had an opportunity to respond to four open- ended questions as part of the survey. The open ended items also gave the teachers who were involved with the SMART program an opportunity to provide comments regarding the program from the prior summer. A summary of these questions and respective responses are listed below. Question 1. In response to the question, What SMART/THRIVE strategies do you use in your classroom? teachers listed a total of 41 distinct responses. The most frequently made comments related to strategies pertaining to games (29%) and calculator applications (22%). Strategies relating to cooperative learning, computer applications and general strategies were also noted. The results are found in Table 27. 41Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 44 of 69 TABLE 27 What SMART/THRIVE strategies do you use in your classroom? Description Games Calculator applications Cooperative learning Specific Strategies (warm-up/review) Computer applications (beyond calculators) General strategies TOTAL Frequency 12 9 9 7 2 2 41 Percentage 29% 22% 22% 17% 5% 5% 100% Question 2: In response to the question, What are the most/least beneficial aspects to the professional development component of SMART and/or THRIVE?, teachers listed a total of 24 responses. Beneficial aspects of the professional development portion of the SMART and THRIVE programs most frequently noted were: new strategies, opportunities to collaborate with other teachers, hands-on application, co-teaching component, and application in regular algebra class. Three comments were made about the least beneficial aspects of the program
all three noted time constraints as a major limitation. See Table 28 for a summary. TABLE 28 What are the most/least beneficial aspects to the professional development component of SMART and/or THRIVE? Description Hands-on application Opportunities to collaborate with other teachers Application in regular algebra class New strategies Time is an issue TOTAL Frequency 7 5 5 4 3 24 Percentage 29 21 21 17 13 100% Question 3: A total of 14 comments were made in response to the question, 'If you were a teacher tor SMART during the past summer, are there any additional comments that you would like to make about the SMART Program? The overwhelming majority of comments cited SMART as a strong pre-algebra preparation program. Additionally, teachers noted the following comments about SMART: reinforcing concepts, fostering technology awareness, improving teacher relationships, and preparing students for 42Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 45 of 69 algebra. Two teachers noted a desire to expand and continue the SMART program. See Table 29 for a summary. TABLE 29 If you were a teacher for SMART during the past summer, are there any additional comments that you would like to make about the SMART Program? Description Reinforcing concepts Preparing for Algebra Expansion Fostering teacher relationships Fostering technology awareness TOTAL Frequency 6 4 2 1 1 14 Percentage 43% 29% 14% 7% 7% 100% ALGEBRA I TEACHER SURVEY SMART Demographic information from the Algebra I Teacher survey show that of the twenty-five Algebra I teachers who responded to six survey items pertaining to SMART 68% were female and 28% were male. A total of 44% of respondents specified their ethnicity as Caucasian. Thirty-six percent specified African American and 12% specified Multi-Ethnic. The remaining categories of Asian and Hispanic were not specified by any of the respondents. The number of respondents indicating that they had worked for SMART or THRIVE in the past comprised 44% percent of the total group
fifty-two percent indicated not having worked with either program. Few respondents (8%) reported having fewer than 5 years teaching experience and the majority of respondents (total 56%) reported having 11 or more years of experience. In contrast to response trends from the parents, students and THRIVE teachers surveyed, Algebra I teachers' responses to SMART items showed slightly more varied distribution across the survey scale (Strongly Agree/Agree-Neutral-Strongly Disagree/Disagree-Dont Know). Over 80% of the teachers surveyed agreed or strongly agreed that SMART positively impacted student achievement in math and effectively enabled students to use technology to solve algebra problems. Eighty percent of respondents indicated agreement or strong agreement when asked to determine the extent to which they agreed that SMART helped students become more confident of their abilities in algebra. A little over 72% of responses indicated agreement or strong agreement with the statement that SMART offered strategies that could be used in the 43Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 46 of 69 Algebra I classroom and 65% agreed or strongly agreed that SMART offered strategies that could be used in the Algebra I classroom. When asked to judge whether SMART had helped close the achievement gap between African American and Caucasian students, just over 48% of the respondents agreed or strongly agreed. Eight percent disagreed or strongly disagreed with this statement, while 24% remained neutral and 20% indicated that they did not know. Table 30 Algebra I Teacher Survey - SMART N = 25 1. 2. 3. 4. 5. 6. The program has... Positively impacted student achievement in math Helped close the achievement gap between African American and Caucasian students Helped students become more confident of their abilities in Algebra Effectively enabled students to use technology to solve Algebra problems Facilitates students' meaningful understanding of algebraic concepts Offered me strategies that I use in my own classroom Percent Strongly Agree/ Agree 88.0 Percent Neutral 4.0 Percent Strongly Disagree/ Disagree 0.0 Percent Don't Know 8.0 48.0 80.0 88.0 72.0 68.0 24.0 8.0 20.0 8.0 0.0 16.0 8.0 4.0 0.0 4.0 8.0 8.0 12.0 8.0 16.0 THRIVE Among responses from the 33 Algebra I teachers who responded to the six THRIVE items, 66.7% were female and 24.2% were male. Reports of teaching experience and experience with THRIVE and SMART were comparable to those noted among the group responding to SMART. The demographic distribution of this group was also similar to that noted among SMART respondents (39.4% Caucasian, 36.4% African American, and 12.1% Multi-Ethnic). Over 84% of the respondents agreed or strongly agreed that THRIVE effectively enabled students to solve problems using technology. Eighty-one percent agreed that THRIVE positively impacted student achievement in math. Over 75% agreed or strongly agreed that THRIVE facilitated conceptual understanding of algebra and helped students become more confident in Algebra. Seventy-two percent of the respondents agreed or strongly agreed that THRIVE offered strategies that could be used in the Algebra I classroom. The same percentage of respondents (48%) as those responding to the same item about SMART agreed or 44Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 47 of 69 strongly agreed that THRIVE had closed the achievement gap between African American and Caucasian students. A total of 18% remained neutral about this item, 6.1% disagreed or strongly disagreed and 24.2% indicated that they did not know. Table 31 Algebra I Teacher Survey - THRIVE N = 33 Percent Percent 1. 2. 3. 4. 5. 6. The program has... Positively impacted student achievement in math Helped close the achievement gap between African American and Caucasian students Helped students become more confident of their abilities in Algebra Effectively enabled students to use technology to solve Algebra problems Facilitates students meaningful understanding of algebraic concepts Offered me strategies that I use in my own classroom Strongly Percent Strongly Agree/ Agree 81.8 48.5 75.8 84.9 75.8 72.7 Neutral Disagree/ Disagree Percent Dont Know 3.0 0.0 12.1 18.2 6.1 24.2 9.1 0.0 9.1 6.1 3.0 0.0 3.0 6.1 9.1 12.1 9.1 12.1 OPEN-ENDED RESPONSES To further evaluate teachers perceptions of SMART and THRIVE, four open-ended questions were administered to all district Algebra I teachers, regardless of whether or not they were directly involved in the programs. Tables 32 and 33 illustrate how SMART and THRIVE programs are perceived among ail Algebra I teachers polled in Little Rock. Since not all of the Algebra I teachers polled were SMART/THRIVE teachers, it was important to gather information about teachers awareness of both programs. Question 1: A total of 50 comments were made when teachers were asked to briefly describe their understanding of the SMART program. SMART was cited as a strong pre-algebra preparation program by the large majority of respondents. Algebra I teachers comments reflected a general understanding of the nature and target population, and several noted specific elements about its focus on promoting understanding Algebra I concepts through use of technology, particularly among at-risk populations. Results are summarized in Table 32. 45Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 48 of 69 Table 32 Please briefly describe your understanding of the SMART Program. Description Program designed to provide a preview of Algebra I concepts and build math confidence Program designed to integrate calculators Frequency 20 Percentage 40% 10 20% Identified time (day) and location of program Program designed to motivate students and make Algebra I fun Identified program for grades 8 and 9 only 14% 8% 8% Program designed to target at-risk children Program designed to address ARK and ACTAAP goals___________________________ TOTAL 6% 4% 50 100% 7 4 4 3 2 Question 2: When Algebra I teachers were asked to briefly describe their understanding of the THRIVE program, a total of 55 comments were made. THRIVE was most frequently characterized as a supplement to the Algebra I curriculum designed to bolster concepts taught in school. Teachers also generally described THRIVE program's time (Saturdays), location and relationship to SMART. Additionally, Algebra I teachers made comments about the motivational nature of the program as well as its purpose in reinforcing principles for various standardized tests (e.g.. Benchmark and ACCTAP). Results are summarized in Table 33. 46Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 49 of 69 TABLE 33 Please briefly describe your understanding of the THRIVE program. Description Program designed to supplement Algebra I concepts and build math confidence Identified time (day) and location of program Frequency 23 Percentage 42% 10 18% Related to or a follow-up to SMART Program designed to motivate students and make Algebra I fun Program designed to address standardized Benchmark tests, ARK and ACTAAP goals Program designed to integrate calculators 10 18% 9% 7% 4% Program designed to target at-risk children TOTAL 2% 55 100% 5 4 2 1 Questions 3 (Table 34) and 4 (Table 35). Questions three and four were designed to gather information about teachers perceptions of effective and ineffective components of SMART and THRIVE. These questions did not differentiate between the two programs. Responses suggested that teachers evaluative open-ended comments for both programs were similar. Responses to question 3 yielded complimentary commentary describing the efficacious elements of the programs. Question 4 yielded comments describing the programs shortcomings. TABLE 34 What do you see as the most effective aspects of the SMART and/or THRIVE programs? Description Enriches Algebra concepts in authentic contexts Integrates calculatorAechnology use Reinforces Algebra concepts Fosters motivation and integrates fun into teaching Fosters confidence and success Enhances peer relationships/collaborative learning Enhances professional development Provides tutoring for children having trouble with Algebra I Helps prepare for standardized tests Emphasizes co-teaching TQTAL Frequency 10 10 8 5 4 2 1 1 1 1 43 Percentage 23% 23% 19% 12% 9% 5% 2% 2% 2% 2% 100 47Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 50 of 69 TABLE 35 What do you see as the feast effective aspects or areas in need of improvement? Description Need to serve broader group of students Over-reliance on calculator More funding needed No direct evaluation of the program (grades) Enrollment/recruitment Too much emphasis on games Space TOTAL Frequency 9 4 3 2 2 2 1 23 Percentage 39% 17% 13% 9% 9% 9% 4% 100% FOCUS GROUP RESULTS PROGRAM PARTICIPANT FOCUS GROUP The questions asked of the selected students were designed to glean more information about their perceptions of THRIVE and the perceived effects of participation in the program. Each focus group began by asking students to describe THRIVE. A little over one third of the comments (36%) indicated that the students viewed THRIVE as a program that focused on math tutoring. An additional 21% of participants indicated that THRIVE made math more fun. Other comments referenced THRIVE as a (standardized) test-preparation program, a program designed to build specific math skills and improve grades, and/or a program that used competition and snacks to help motivate students interest in math. Some students made note of the heavy use of calculators in instruction. In a similar vein, students noted that they thought the purpose of THRIVE was to help struggling students, improve math scores, provide differentiated instruction and improve district math programs. When asked to describe more discrete details of the Saturday program, student participants frequently noted use of game play and rewards to facilitate teaching and test taking skills in Algebra 1. To get a better sense of students perceptions of math in general, respondents were asked to comment on their level of comfort or dislike in math. Almost half (46%) of the student participants indicated that they have not traditionally liked math, but have enjoyed it more through THRIVE. Twenty-three percent of respondents noted always having enjoyed math. Two students (15%) noted not liking math. One student noted having mild disdain for math and another reportedly enjoyed math but felt that Algebra I was difficult. Students were asked to provide opinions of their overall student satisfaction of the program. When asked whether or not they liked THRIVE, all respondents replied positively. When probed about why they liked the 48Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 51 of 69 program, fewer individuals provided further commentary. Of those who did, hands-on activities and the personal, comfortable environment were noted as elements that made them feel more confident. Games, rewards and use of competition in math application were also noted as positive attributes. None of the respondents indicated negative feelings toward the program in response to this question, although one student recommended that the program start later in the day. When asked whether they would recommend THRIVE to other students, half of the participants (10) responded. No negative responses to this question were noted. When asked to elaborate, the majority of respondents who did not respond with a simple yes noted that the program would be particularly appropriate for students struggling with Algebra and/or those students who desire to get ahead in Algebra. Students also were asked to comment on specific changes that they would foresee as improvements to THRIVE. Among those who responded, the most frequently noted request for change related to the time or duration of the program (4 respondents). Three students indicated they would make no changes, and two students indicated that they would like to get credit for THRIVE. The remaining responses were noted among individuals and are listed in Appendix C. Because students come to THRIVE with a variety of instructional backgrounds, the next two questions addressed whether the students felt that THRIVE changed how they approached math problems and whether new ways of studying math had emerged as a result of participating in THRIVE. Over a third of the respondents indicated that THRIVE had given them different ways to solve problems. Individual comments suggested that THRIVE helped students improve grades, get ahead in class and use calculators in problem solving. One student indicated that THRIVE made math less intimidating
another suggested that the practice it provided made classroom concepts easier to grasp. None of the respondents indicated that THRIVE had significantly changed how they studied math. When further probed, it was noted that students perceived this question as one pertaining to test-taking and test preparation. They indicated that they continued to prepare for traditional tests as they did before participating in THRIVE. The core purpose of the question was to determine whether students applied study strategies learned in THRIVE to Algebra I class. Although it is apparent that students felt THRIVE helped them in Algebra I, their responses did not indicate that it significantly changed their study habits. When asked how THRIVE is specifically helping in Algebra, the three most frequently made comments related to the programs role in reviewing material presented in Algebra I classes, improving grades and 49Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 52 of 69 helping apply concepts. Other comments were proposed by individuals
however, all but one related to THRIVE supporting math-specific needs through basic instruction that complimented but could be differentiated from classroom practices. Three probe questions were asked as follow-up. The first probe helped elicit information about how calculators serve as facilitative devices in THRIVE. One student stated that calculators could be used as a crutch and should be relied upon less heavily
however, another student noted the use of calculators as helpful in solving problems more efficiently. Additional comments included references to calculators as instruments that could be used to do anything (not merely calculations) and tools that reduce the amount of thinking" required to solve problems. Regarding the use of games and competitions, students responses suggest that they help reinforce concepts, make concepts easier to understand, make time spent learning math more fun. and help retention. When asked how students apply strategies learned in THRIVE to Algebra I classrooms. two students noted that THRIVE gives them more practice. Other students reported learning different ways to solve problems and having information from the Algebra class reinforced as helpful. Additional comments are noted in Appendix D. Since Benchmark exams play a significant role in district-level evaluations of math programs, students were asked to comment about their performance expectations on the exams as well as whether or not they felt their THRIVE experience would enhance their performance. Over 50% of respondents said they felt ready/prepared to take the Algebra subject area exams. Three students indicated that they were unsure about whether their efforts would transfer or not and one student reported feeling that he had more tricks (strategies) for solving some of the Benchmark problems than he did before attending THRIVE. Students were asked to comment on what Algebra I would be like without THRIVE or SMART. Well over one third of respondents indicated that math/algebra would be difficult and/or that they would have to study harder to keep up in class. Other comments indicated that it would be difficult to remember concepts and problems would be more complicated. One student noted that he would not have taken Algebra this year had he not been enrolled in THRIVE. 50Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 53 of 69 MENTOR FOCUS GROUP The questions posed to the SMART mentors were designed to glean information about the mentors perceptions of both SMART and THRIVE. Specifically, the purpose was to gather data about how the programs affected students. Two of the eight THRIVE mentors attended the focus group. As mentors, their jobs were reported as conducting icebreakers with the students such as scrambled math problems and providing individual help/instruction during the class sessions. While the teacher is providing instruction, the mentors walk the room and offer assistance to any student in need. The mentors viewed SMART as beneficial citing benefits that extend through pre-calculus. In describing SMART, the mentors stated that the program is exactly what the name entails Summer Math Algebra Readiness Training." They noted that the SMART program trains students on the graphing calculator for their first year in algebra. It makes them successful, otherwise the kids that dont go they just buy the calculator and dont know how to use it. According to the two focus group participants, participation in SMART boosts students confidence, they begin enter algebra knowing they have a head start on the major topics like order of operations, factoring, inequalities, and stuff like that. The mentors indicated that they would recommend SMART to other students because it will help them in the long run. One mentor noted that, "Id recommend this to students who hate math, cause you get in there, give up 2 weeks of your summer, and then come out saying 'Wow, I learned and had so much fun. Especially in the competitions.' When asked how working as mentors for SMART benefited them, the mentors attributed ail of their algebra knowledge to their work with SMART. Additionally, it was noted that the program helped the mentors learn to work with students of different ages. learn teaching techniques, and new learning techniques for themselves. When asked about THRIVE, the mentors described it as the extension of SMART. The SMART program was described as a program that, gets students going, and THRIVE keeps students going by retaining their skills, tricks, and calculator skills throughout the year. However, neither tutor had participated in THRIVE so all knowledge of THRIVE is from their peers. The mentors concluded that THRIVE must be good to get you up on Saturdays because students hate to miss it. They love it. 51Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 54 of 69 ATTENDANCE Attendance data were available for 143 Thrive students. A total of eight Thrive sessions were held once every two weeks during the school year. Attendance rates ranged from 81 % to 100%, with an average attendance rate of 90% (see Figure 1). A plurality of students {n = 64,44.8%) attended all sessions, while 30.0% (n=43) missed one session, 21.7% (n=31) missed two sessions, and 3.5% (n=5) missed three sessions (see Figure 2). SMART students (n=210) attended up to eight sessions held during the summer. SMART attendance rates ranged from 94% to 100%, with an average attendance rate of 97% (see Figure 3). The vast majority of students (n=183,87.1%) missed no sessions, while 10.9% {n = 23) missed one session, and 2% (n=4) missed two or more sessions (see Figure 4). 52Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 55 of 69 Figure 1. Thrive Attendance Rate by Session. Figure 2. Number of Students by Number of Sessions Missed: Thrive. 70 SOSO 40 10 53 3 0 2 0 0 6 4 T 0 4 3 T 1 3 1 T 2 0= T 3 Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 56 of 69 Figure 3. SMART Attendance Rates by Session. Figure 4. Number of Students by Number of Sessions Missed
SMART. 20 0 150 1 00 if' 1 83 50 HD 54 0 T 0 T 1 2 4 Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 57 of 69 STUDENT PERFORMANCE-BASED ASSESSMENT 2003-2004 AND 2004-2005 MATH BENCHMARK AND EOC SCORES For ninth graders, a 4 (Treatment Status) X 2 (African American vs. Other) analysis of covariance was performed on Algebra I EOC scores, using 2003-04 Benchmark scores and free lunch status as covariates. Effect size estimates were computed for the ninth grade programs by subtracting the adjusted mean for the comparison group from the adjusted mean for each program group, then dividing by the pooled-within groups standard deviation. Standardized (z) scores (with overall M = 0.00 and SO = 1.00) were computed for Benchmark Math scale scores and EOC Algebra I scale scores by subtracting the district mean from each score and dividing by the district standard deviation. Mean z-scores were then computed for comparison students and students participating in any program configuration by race to describe relative shifts in the achievement gap across years. For eighth graders, a descriptive analysis was conducted by computing the percentages of students at various levels of proficiency for each treatment condition. Because very few nonAfrican American students participated in any of these programs in eighth grade (n = 11), eighth grade comparisons were conducted only for African American students. All results for Benchmark and EOC data were derived from a pool of students whose scores on both exams were available. Algebra I EOC test: Ninth Grade. The ANCOVA model, which explained 55.9% of the variance in 2004-05 Algebra I EOC scores, revealed two significant main effects: 2003-04 Benchmark Mathematics scale scores (F|,a45 = 855,4, p <.001), used as a covariate, and treatment status (F3345 = 6.10, p <.001). Follow-up tests indicated that the adjusted mean scores for SMART students (M=175.58, ES = +0.28), Thrive students (AT = 187.88, ES = +0.56), and students attending both programs (M= 189.77, ES = +0.60) were all significantly higher than the adjusted comparison group mean (Af=163.42
see Figure 5). There were no main or interaction effects of student race. 55Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 58 of 69 Figure 5. Effect Size Estimates by Program: 2004-05 Ninth Grade Algebra I End-of-course Test. 0.70 - 0.60 - 0.56 0.60 0.50 0.40------ 0.30 0.20-------- 0.10 0.00 0.28 SMART Thrive Both Table 36 provides a breakdown of mean Algebra I test scores by race and treatment status. As can be seen in the table, all program means were significantly higher (p < .05) than the comparisons means within each racial category. Table 36. 2004-05 Algebra I Observed and Covariate-adjusted EOC Test Means by Race and Treatment Condition: Ninth Grade Race Program Mean Non-AA Comparison SMART Thrive Both 184.47 201.07 Adjusted Mean 163.65 178.54 SD ES African American Comparison SMART Thrive Both 186.33 187.30 155.84 163.18 45.22 45.16 52.81 40.62 180.49 192.23 202.00 172.61 187.87 192.15 39.29 53.64 35.72 150 14 0 3 581 70 13 23 -t-0.34 -1-0.54 -^0.21 -hO.56 -hO.66 N Note. All Smart, Thrive, and Both means are significantly higher than comparison group mean within racial category at p <.05. Note. Total SD = 43.93. Note. Scores were derived from 9" grade students with available EOC scores from 2004-2005 and Benchmark scores from 2003-2004. 56Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 59 of 69 Supplementary analyses indicated that students in any of the three program groupings were more likely to achieve at Basic, Proficient, or Advanced proficiency levels than the comparison group, while the comparison group had a large percentage of students (42.1%) achieving at a Below Basic level (see Table 37, Figure 6). Further, as shown in Figure 7, African American students participating in any of the three program configurations (z- -0.16) scored more than one standard deviation below other students in the comparison condition (z= +0.90) on 2003-04 Benchmark Mathematics, but had a nearly equal score on the 2004-05 Algebra I test (z= +0.42) relative to comparison students (z= +0.50). Table 37 Percentage of Students by Proficiency Level and Program: 2004-05 Grade Algebra I End-of-Course Test Algebra I Proficiency Level Program Comparison SMART Thrive Both n % within program n % within program n % within program n % within 1-Below Basic 409 42.1% 15 17.0% 4 25.0% 2 6.7% 2-Basic 347 35.7% 45 51.1% 6 37.5% 13 43.3% 3-Proficient 174 17.9% 22 25.0% 3 18.8% 11 36.7% 4-Advanced 41 4.2% 6 6.8% 3 18.8% 4 _____________________________program Note. Scores were derived from 3^ grade students with available EOC scores from 2004-2005 and Benchmark scores from 2003-2004. 13.3% 57Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 60 of 69 IVE ISON 58 Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 61 of 69 Figure 7. Mean Z-score by Program Status and Race
2003-04 Benchmark Mathematics and 2004-05 Algebra I. Algebra I EOC test: Eighth Grade. As shown in Figure 8, African American students who attended SMART (73.3%), Thrive (69.7%), or both programs (71.9%) were more likely to achieve at Proficient or Advanced Levels than the comparison group (58.6%
see Table 38. Figure 8). Comparison group students were more likely than either of the three program groups to achieve at Below Basic (12.2%) or Basic (29.3%) levels. 59Case 4
82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 62 of 69 Table 38 Percentage of Students by Proficiency Level and Program: 2004-05 Grade Algebra I End-of-Course Test (African American Students) Algebra I Proficiency Level Program Comparison SMART Thrive Both n % within program n % within program n % within program n % within program 1-Below Basic 10 12.2% 6.7% 3.0% .0% 2-Basic 24 29.3% 20.0% 27.3% 28.1% 3-Proficient 35 42.7% 60.0% 17 51.5% 16 50.0% 4-Advanced 13 15.9% 13.3% 18.2% 21.9% Note. Scores were derived from Sf" grade students with available EOC scores from 2004-2005 and Benchmark scores from 2003-2004. 60 1 1 0 3 9 2 9 9 6 7Case 4:82-cv-00866-WRW-JTR Document 3996-2 Filed 12/27/2005 Page 63 of 69 Figure 8. Percentage of Students by Proficiency Level and Program: 2004-05 8* Grade Algebra I End-of-Course Test (African American Students). 60 50 40 c 03 O O Q. 30 20 10 PROGRAM ----------- COMPARISON -------- smart ---------thrive ----------- BOTH 1 -Below Basic 2-Basic 3-Proficient 4-Advanced 0 T T T T Proficiency Level 2004-2005 MATH ITBS SCORES (9 GRADE ONLY) ITBS Mathematics NCE scores: Grade. The ANCOVA model, which explained 70.5% of the variance in ITBS Mathematics NCE scores revealed two significant main effects: 2003-04 Benchmark Mathematics scale scores (Fuses =
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.