"Little Rock School District Compass Learning Evaluation," Center for Research in Education Policy, University of Memphis, Memphis, Tennessee (final report for review)

This transcript was created using Optical Character Recognition (OCR) and may contain some errors.
CREP Cl!ntu for Ral!orcl, in duc11/lonol Polic)' RECEIVED JAN 5 2006 OFACEOF DESEGREGATION MONITORING slearning DRAFT REPORT FOR REVIEW BY Little Rock School District CREP Cenlu for Research in Educutional Policy t oc s tri t Co Le .. s n ng 10n Fina eport DRAFT REPORT FOR REVIEW BY Little Rock School District De ember 2005 Dubor
_ l owther St~vc1 1 Ro rrey Martind le l t I tt Rock EXECUTIVE SUMMARY INTRODUCTION This report summarizes the evaluation study results of the Little Rock School District's {LSRD) 2004-2005 CompassLearning (CL) program. The overall purpose of the evaluation was threefold: (1) to assess the effects of CL in raising the academic achievement of African American and other students, (2) to examine CL implementation processes and practices, and (3) to document the perceptions of students, teachers, principals, and district and school personnel involved with CL regarding the strengths, weaknesses, and needed improvements of the CL program. RESEARCH QUESTIONS Primary Evaluation Question 1. What are the effects of participation in CL on the achievement of African American and other students? Supplemental (Step 2) Evaluation Questions 1. What is the quality, nature, and level of implementation of CL at the 19 elementary schools identified as implementing the program in 2004-05? 2. What is the level of participation in CL by African American students relative to other ethnic groups at the implementing schools? 3. What are the perceptions of teachers, lab attendants, and technology specialists regarding CL program implementation, impacts, strengths, and weaknesses? 4. What are the perceptions of parents/guardians of CL students regarding program impacts, strengths, and weaknesses? 5. What are the perceptions of school principals, whose schools no longer use CL, with regard to past use of the CL program and possible adoption of a different program? DESIGN Participants. The LSRD identified 20 elementary schools, two middle schools, and one high school to participate in the spring 2005 evaluation. Collectively the schools enrolled 9,318 students and employed 606 classroom teachers. Over 68% of the overall student enrollment of the selected schools was African American. The CL program was provided to kindergarten through fifth grade students in 19 elementary schools. The students spent 30- to 60-minutes per week working on CL activities that occurred either in computer labs that typically had a lab attendant and classroom teacher present or in the students' regular classrooms. The lab attendants, technology specialists, and/or teachers planned student CL activities that were primarily focused on identified areas of student deficiencies in language arts/reading and/or math. Design. The evaluation utilized a mixed-method design. Both quantitative and qualitative data were collected from the participating schools by trained external researchers. The researchers observed CL computer labs, administered surveys to teachers and parents, conducted student focus groups, and interviewed school principals, lab attendants, technology specialists, district personnel and CL personnel. INSTRUMENTATION Five measurement strategies were used to collect the evaluation data: direct classroom observations, surveys, interviews, focus groups, and assessment of student academic achievement. Following are descriptions of the instruments. Classroom Observation Measure Direct classroom observation. The Survey of Computer Use for Compasslearning (SC UCL) was used to record teacher and student activities during the use CL in school computer labs during two-hour observation periods. Teacher and Parent Survey Teacher survey. The Teacher Compasslearning Questionnaire (TCLQ) is a two-part instrument that was used to collect teacher knowledge and perceptions of CL. Parent survey. The LSRD Compasslearning Parent Survey was designed to capture parent perceptions regarding their child's use of the CL program. Student Focus Groups The Student Focus Group Protocol was used to document student impressions of the nature of CL activities
the usefulness of CL
and student enjoyment when using CL. Interviews Five interview protocols were developed for the evaluation. The lab attendant, technology specialist, and district (CL) personnel interviews focused on the same general areas: CL responsibilities, level, nature, and quality of CL implementation, support for the CL program, the strongest and weakest aspects of CL, and if the program should be continued. The primary purpose of the national CL representative interview was to determine Cl's guidelines for effective implementation of the CL Program. The principal interview was for principals that had previously used the CL program in their schools. The interview focused on past use of CL, why it was discontinued, and if the school was using a different integrated learning system, and if yes, why that program was chosen. Student Achievement Analyses of student achievement on the Development Readiness Assessment (ORA), Iowa Tests of Basic Skills, (ITBS), and the Arkansas Benchmark tests were conducted to address the following research questions: What are the effects of the Compass Learning program on student achievement in Reading and Mathematics? Is the program effective in meeting the academic needs of African American students? How does the achievement of African American students compare to that of other students in the same or similar schools? 2 PROCEDURE All data were collected in the spring of 2005 by external researchers. Direct observations of CL classrooms were conducted during prearranged two hour sessions at 7 randomly selected elementary schools. The SCU-CL note forms were completed every 15 minutes during the two-hour period. External researchers conducted 20-minute student focus groups at four schools that were randomly selected from the seven schools being observed. The teacher surveys (TCLQ) were administered at a faculty meeting at each of 19 elementary schools, while the parent surveys were distributed at 5 randomly selected elementary schools by having one class from each grade at the school send surveys to the parents. Surveys were returned to the school and sent to evaluators. Researchers individually interviewed lab attendants at each of the 7 randomly selected schools and conducted 45- to 60-minute phone interviews with all CL technology specialists. The researchers also interviewed two district level personnel with responsibility for the CL program
a national CL representative
and 4 school principals from schools that no longer used CL. Each interview lasted approximately sixty minutes. Student Achievement Analysis Procedure The elementary school sample used in the student achievement analysis included a total of 7,853 LRPS students in first grade (n = 2,016), second grade (n = 1984 ), third grade (n = 1904) or fifth grade (n = 1,949) during the 2004-'05 school year. Kindergarten and fourth grade were excluded because these grades lacked corresponding 2003-04 data to be used as a "pretest" (statistical covariate). All 34 LRSD elementary schools were included in the analysis. The Developmental Reading Assessment (DRA) was administered in grades K-2 in 2003-04 and 2004-05, and the Iowa Test of Basic Skills (ITBS) was administered in grades K-5 in 2004-05. The ITBS provides normal curve equivalent (NCE) scores in Reading and Mathematics. The Arkansas Benchmark test was administered in grade 4 in 2003-04, and in grades 3-5 in 2004-05. However, scale scores from the 2004-05 Arkansas Benchmark test were not available as of December 2005. RESULTS The results of the study are presented below, grouped by measurement strategy. In the conclusion section, findings are synthesized across instruments in order to address each research question. Classroom Observation Results Data from the Survey of Computer Use for Compasslearning (SCU-CL) reveled that the 19 observed CL classes were implemented in computer labs in which each student used a modern, upto- date computer. All African American and non-African American students in the classes participated equally and exhibited a high level of attention and interest in the CL activities. All observed classrooms had a high level of academically focused class time in which students were typically engaged in CL activities for language arts, mathematics, or reading. Survey Results Teacher Compasslearning Questionnaire (TCLQ) A total of 356 teachers (Caucasian= 63.2%
African American= 32.6%) working at schools implementing CL completed the TCLQ. Nearly all of the teachers reported familiarity with CL and over 60% indicated that they had received enough training to address student needs through the use of CL resources. More than one-half of the teachers indicated that they frequently used Cl's class by 3 ethnicity and class by gender reports. There was general agreement that teachers, administrators, and parents supported the continued use of CL. The primary reasons given for the continuation of CL were that the program reinforces the curriculum
enhances student learning and skills
and increases student engagement and excitement in learning. LRSD CompassLearning Parent Survey (Parent Survey) A total of 451 parents/guardians of primarily African American students completed the Parent Survey. Parental awareness of CL was high and most agreed that time in the computer lab was an important aspect of their child's education, had improved their child's interest in school, and had improved their child's academic achievement. Approximately one-third of the parents wrote a response when asked to describe the "best thing" about their child or children using CUthe computer lab. The most common responses were learning how to use computers and acquiring new learning skills. Less than one in five parents/guardians indicated that the best aspect was improvement of math and reading skills. Of the parents who provided recommended changes for CL, most indicated that their child or children needed more computer time. Student Focus Group A total of 23 fifth-grade students from four randomly selected schools participated in the focus groups, which lasted approximately 20-minutes. Student experience with CL ranged from less than one year to six years, with 43% indicating they had used CL since pre-kindergarten or kindergarten. The students reported that they primarily completed CL activities in language arts, mathematics, and reading in both computer labs and their classrooms. Of the 23 students, 10 wanted to spend more time using CL, 6 indicated the CL time was appropriate, and 2 students responded that they spent too much time in the CL lab. When asked how they would change the CL activities, the most common responses were to make the activities more fun or game-like and to add Internet access. Nearly all students thought they did better in school because CL helped them remember what they learned in class. Students indicated that the disadvantages to using CL were that sometimes the activity problems were too hard or that some were too easy or boring. All of the students responded that they would like to continue using CL. Interview Results Interview responses from the lab attendants, technology specialist, LRSD personnel, CL national representative, and principals who used CL in the past are synthesized in the summary below. Lab Attendant, Technology Specialist, District Personnel The lab attendants, technology specialists, and district (CL) personnel revealed common agreement that there was a high level of teacher support for the CL program, that the CL program was well matched to LRSD curriculum standards, and that the program has been implemented well. Interview responses indicated that the primary users of CL were students in grades K through 5 who used CL from one to five days per week. Equitable student access and appropriate implementation of CL was ensured from close monitoring by the lab attendants, technology specialists, and principals. Respondents also indicated that the teachers were regularly provided CL reports on student progress and that the teachers actively used the reports to plan student CL activities. School administrators most frequently used the CL data for year-end reports. There was unanimous agreement that the CL program should be continued due to its positive effects on student academic achievement. However, the respondents suggested that CL implementation would be improved if district guidelines were created, supervision of CL was increased, and other remediation options explored. 4 National CL Representative To obtain background regarding the CL program, an interview was conducted with a CL national representative. The responses are divided into guidelines for students, teachers, and administrators. Students. With regard to student use of CL, or time-on-task, the recommended guidelines are that students in grades 3 through 12 should spend from 60- to 90-minutes per week completing 4 to 5 new activities per content area. For students in grades K through 2, the recommended time for computer activities is 60-minutes per week per content area. Student activities should be focused on the subject that needs the most improvement. Teachers and administrators. The representative recommended that teachers should be actively involved in using the CL reports to evaluate the progress of individual students. The teachers should review student reports every two weeks in order to align classroom instruction to identified needs, plan off-line (non-CL) remediation, and adjust CL activities to address areas of non-mastery. A monthly review of CL class-by-class reports by school principals to identify any areas of need is also recommended . The CL representative indicated that teachers and administrators, who implement these "best practices" strategies, would be provided with formative evaluation data to help map student progress and plan needed interventions. However, the CL representative indicated that most schools do not achieve this level of implementation. School Principals The principals from four LRSD schools (1 elementary, 2 middle, 1 high) who no longer used CompassLearning were interviewed. The principals reported the primary reasons for not continuing with CL were related to technical (server and/or system) problems. Other reasons for the discontinuation of CL were insufficient Title 1 funds to fully support CL, teacher complaints that aligning lessons with CL took too much time, and the program had too many technical problems. Three principals indicated that their school had adopted different integrated learning systems (ILS). Principal impressions of the new programs were positive indicating that fewer problems were occurring and teachers were more accepting of the new systems. The fourth principal reported that a technology integration approach was being implemented rather than using an ILS. Student Achievement Results Across all grades, subjects, and achievement measures, a total of 14 separate analyses (8 Reading
6 Math) examining treatment x School Set effects were conducted. Of these, 4 (50%) in Reading and 3 (50%) in Math showed no significant treatment main effects or interactions. Where significant effects did occur in Reading, two were main effects, respectively, showing comparison group advantages over Compass in Grade 1-ITBS (ES= -.25) and Grade 7-ITBS (ES= -0.21). The other two effects were interactions, one showing a Compass advantage within the low-poverty School Set (ES = +0.41) in Grade 1-DRA, and the other showing a comparison group advantage within the medium-poverty School Set (ES = -0.26) in Grade 2-ITBS. Significant effects in Math were associated with three interactions, showing Compass advantages in low-poverty schools in Grade 2-ITBS (ES= +0.26) and Grade 3-ITBS (ES= +0.26), and in high-poverty schools in Grade 5-ITBS (ES = +0.10). Comparison schools showed an advantage in medium-poverty schools in Grade 5-ITBS (ES = -0.15). Overall, Compass learning effects were inconsistent but positive in direction in Mathematics. Benefits were more frequent in lowpoverty than high-poverty schools. 5 CONCLUSIONS The conclusions of the present study are presented in association with each of the major research questions in the respective sections below. Primary Evaluation Question What are the effects of participation in CL on the achievement of African American and other students? Taken as a whole, the evidence provided by this study suggests that usage of Compass Learning was not associated with noticeable achievement gains in Reading or Math for the overall sample of students or for African American students in particular. Median effect size, across all analysis, indicate a weak positive effect pf Compass in the low poverty setting but essentially zero effects in medium and high-poverty settings. Supplemental (Step 2) Evaluation Questions What is the quality, nature, and level of implementation of CL at the 19 elementary schools identified as implementing the program in 2004-05? The observation data indicated that CL was implemented in well-equipped, well-supervised computer labs that provided one computer for each student. All students in each class participated equally and were highly engaged and attentive to the CL activities. The interview results indicated teachers more frequently based instructional decisions on class-level performance rather than Cl's recommended strategy of using student-level performance to achieve individualized instruction. With regard to the level of implementation, most students' scheduled time for CL activities fell below program recommendations of 60-minutes per content area per week. Thus, the quality, nature, and level of implementation was generally effective with regard to computer lab environments and student engagement during CL use, but was inconsistent with regard to the degree to which student CL activities were individualized and the amount of scheduled CL time. What is the level of participation in CL by African American students relative to other ethnic groups at the implementing schools? The Compasslearning program was implemented in 19 LRSD Title 1 schools comprised of student populations that were 67.9% African American. Triangulation of data from multiple sources indicate that African American and non-African American students fully and equally participated in CL program activities at their schools. Specifically, data from direct classroom observations of 327 students in CL computer labs revealed that all of the 277 African American and 50 non-African American students were highly engaged in the use of CL in a well-supported environment. When lab attendants and technology specialists were asked which students used CL, the responses indicated a// students within the specified grade ranges regularly participated in CL classroom and lab activities. The district personnel involved with CL reported that CL is implemented in schools with the highest level of student need, or Title 1 Schools. What are the perceptions of teachers, lab attendants, and technology specialists regarding CL program implementation, impacts, strengths, and weaknesses? There was common agreement across the teachers, lab attendants, and technology specialists that the CL program was generally implemented well and that the teachers and administrators supported the use of CL in their schools. The interviewees from each group cited program flexibility to meet individual student needs as one of the strongest aspects of CL. When asked about the weaknesses of the CL program, respondents from the three groups expressed concern that aspects of the CL software were difficult for students to use
students needed more computer time to benefit from the remediation, and the CL program lacked sufficient mapping to the curriculum. There was 6 unanimous agreement that the CL program should be continued due to its positive impact on student learning. What are the perceptions of parents/guardians of CL students regarding program impacts, strengths, and weaknesses? Most parents responded that they were aware of CL and believed that the program was an important part of their child or children's education. However, there was less agreement that CL increased student interest in school or that CL improved achievement in math and reading. Most parents seemed to think that their children were going to the computer lab to learn about computers, rather than to improve math and reading skills. When parents were asked about changes they would like to see in CUcomputer lab program, half wanted more computer time for their children to improve students' computer skills. The parents who suggested more variety in CL lessons seemed to have a better grasp of Cl's purpose. Overall, parents were supportive of their child/children using computers, but they were generally unaware of the overall instructional purpose of students using the CL program. What are the perceptions of school principals, whose schools no longer use CL, with regard to past use of the CL program and possible adoption of a different program? Of the four principals who discontinued use of CL, three were generally positive regarding its past use. One of the principals, however, was less positive due to teacher dislike of the program. This principal indicated that teachers thought it took too much time to align CL activities with their classroom lesson plans, that there were too many technical problems, and the teachers did not enjoy monitoring the CL computer lab. Three of the four principals adopted different integrated learning systems (ILSs), while one chose to implement a non-lLS computer integration model. The principals that adopted new programs indicated that fewer technical problems had occurred and teachers were more accepting of the new systems. These results suggest that although these principals had discontinued use of CL, they had a generally positive regard for the use of technology to improve student learning in schools with a predominately African American population. 7 LITTLE ROCK SCHOOL DISTRICT COMPASSLEARNING EVALUATION FINAL REPORT This report summarizes the evaluation study results of the Little Rock School District's {LSRD) 2004-2005 CompassLearning (CL) program. The overall purpose of the evaluation was threefold (1) to assess the effects of CL in raising the academic achievement of African American and other students, (2) to examine CL implementation processes and practices, and (3) to document the perceptions of students, teachers, principals, and district and school personnel involved with CL regarding the strengths, weaknesses, and needed improvements of the CL program. CL is a computer-based program designed to develop students' skills in reading, writing, and spelling. The program also assists and supports teachers with the management of student performance and allows instruction to be personalized to meet student needs. The lessons are designed around themes that incorporate cross-curricular student activities that are based on "realworld" contexts. In the 2004-05 school year, 19 LRSD elementary schools utilized CL programs, while 1 elementary school, 2 middle schools and 1 high school discontinued the program after using it in previous years. The LRSD schools that implemented the CL program were provided technology specialists and lab attendants to assist classroom teachers with integrating the CL lessons into their curriculum and to assist with technology-related concerns. EVALUATION QUESTIONS The CL evaluation was structured around one over-arching, primary question concerning the impact of CL on student achievement, and five supplemental (Step 2) questions that addressed contextual factors related to implementation of the CL program. 8 Primary Evaluation Question 1. What are the effects of participation in CL on the achievement of African American and other students? Supplemental (Step 2) Evaluation Questions 1. What are the quality, nature, and level of implementation of CL at the 19 elementary schools identified as implementing the program in 2004-05? 2. What is the level of participation in CL by African American students relative to other ethnic groups at the implementing schools? 3. What are the perceptions of teachers, lab attendants, and technology specialists regarding CL program implementation, impacts, strengths, and weaknesses? 4. What are the perceptions of parents/guardians of CL students regarding program impacts, strengths, and weaknesses? 5. What are the perceptions of school principals, whose schools no longer use CL, with regard to past use of the CL program and possible adoption of a different program? EVALUATION DESIGN AND MEASURES Participants The Little Rock School District identified 20 elementary schools, two middle schools, and one high school to participate in the spring 2005 evaluation. Collectively the schools had an enrollment of 9,318 students and employed 606 classroom teachers. The percentage of enrollment of African American students ranged from a low of 20% to a high of 100%. The overall percentage of the schools' African American population was 68.4% (see Appendix A). The majority of participating schools were either pre-kindergarten through fifth grade or kindergarten through fifth grade. Specific distribution of schools by grade level is shown in Table 1. Table 1. Schools Participating in the Compasslearning Evaluation by Grade Level Grade Levels PreK-5 K-5 6,7,8 9 thru 12 Number of Schools 15 5 2 9 The 19 schools that actively implemented the CL program during 2004-2005 all served prekindergarten or kindergarten through fifth grade students. Of those, 13 schools (72.2%) implemented CL to all students in grades K-5, two schools (11.1 %) provided CL to all students in grades 1 through 5, while the grade levels of students using CL varied at the 3 three remaining schools (see Appendix 8). Although, the amount of weekly CL computer lab time for students ranged from 30- to 60-minutes across the schools, the schedule for majority of the students (grades K-2 = 78.9%
grades 3-5 = 73.6%) consisted of less than 60 minutes per week (see Appendix 8). Student CL activities occurred either in computer labs that typically had a lab attendant and classroom teacher present or in the students' regular classrooms. The lab attendants, technology specialists, and/or teachers planned student CL activities that were primarily focused on identified areas of student deficiencies in language arts/reading and/or math. Design The evaluation, which utilized a mixed-method design, was conducted during the spring of 2005. Both quantitative and qualitative data were collected from the participating schools by trained external researchers (e.g., university faculty and staff). The researchers observed computer labs, administered surveys to teachers and parents, conducted student focus groups, and interviewed school principals, lab attendants, technology specialists, district personnel and CompassLearning personnel. A description of the evaluation instruments and a summary table of the participants and data sources, presented with associated research questions, are in Table 2. 10 Table 2. Summary of Instruments, Participants, and Data Sources by Evaluation Question Evaluation Question Pnmary Quesllon 1. What are the effects of participation in CompassLeaming (CL) on the achievement of African American and other students? Supplemental (Step 2) Questions 2. What are the quality, nature, and level of implementation of CL at the 19 elementary schools identified as implementing the program In 2004-05? 3. What is the level of participation in CL by African American students relative to other ethnic groups at the schools concerned? 4. What are the perceptions of teachers, lab attendants, and technology specialists regarding CL program implementation, impacts, strengths, and weaknesses? 5. What are the perceptions of parents/guardians of CL students regarding program impacts, strengths, and weaknesses? 6. What are the perceptions of school principals, whose schools no longer use CL, with regard to past use of the CL program and possible adoption of a different program? Participants The elementary school sample included a total of 7,853 LRPS students in first grade (n = 2,016), second grade (n = 1984), third grade (n = 1904) or fifth grade (n = 1,949) during the 2004-'05 school year. Kindergarten and fourth grade were excluded because these grades lacked corresponding 2003-04 data to be used as a "pretest" (statistical covanate). All 34 LRSD elementary schools were included in the analysis. All CL school teachers All technology specialists at schools implementing CL CL lab attendants at the 7 schools randomly selected for observations Two district personnel involved with CL Student Focus Groups at 4 randomly selected elementary schools National CL representative All Compass schools All CL school teachers All technology specialists at schools implementing CL CL lab attendants at the 7 schools randomly selected for observations Parents of CL students Principals at one elementary school, two middle schools, and one high school Data Sources ITBS as pretest for Grades K-5 Mansas Benchmarks as posttest for 3- 5) 2004-05 ITBS Reading and Math subtests (as posttest in grades 1-5) Teacher CompassLeaming Questionnaire administered dunng school faculty meetings Technology specialist Phone Interview District Personnel (involved with CL) Phone Interview Lab attendant Phone Interview Two-hour CL Laboratory Observations (7 randomly selected elementary schools) 20-min. Student Focus Groups (n = 5-7 students), one each at 4 schools randomly selected from the 7 observation schools National CL representative phone interview School records/archival data Two-hour CL Laboratory Observations (7 randomly selected elementary schools) Teacher Survey (faculty meeting) Technology specialist Phone Interview Lab attendant Phone Interview CL Parent Survey Distributed to one class at each grade level at 5 elementary schools Principal Phone Interview 11 Instrumentation Five measurement strategies were used to collect the evaluation data: direct classroom observations, surveys, interviews, focus groups, and assessment of student achievement. Details of the instrumentation and administration procedures are provided in the following sections. Copies of the instruments are in Appendix C. Classroom Observation Measure The Survey of Computer Use for CompassLearning (SCU-CL) was designed to document the processes and practices used to implement CompassLearning in school computer labs during twohour observation periods. The instrument was used to record contextual data (e.g., number of students using CL by ethnicity, the number of teachers, lab attendants, or technology specialists present, and the number of classes observed during a two-hour observation visit) and computer configuration data (e.g. , number, type, and working condition of the computers). Data were also recorded regarding student use of CL including: percent of students using CL
subject area of the activities
teacher/student interactions during CL use
level of African American and non African American student engagement/interest
and level of academically-focused class time. The data were recorded every 15 minutes then summarized on a data summary form at the end of the 2-hour observation period. Nineteen individual classes were observed for a total of 14 hours in 7 randomly selected elementary school computer labs. Teacher and Parent Survey Teacher Survey. The Teacher CompassLearning Questionnaire (TCLQ) is a two-part instrument that was used to collect teacher knowledge and perceptions of CL. The survey began by asking teachers their gender, ethnicity, and if they knew about the CompassLearning (computer lab) program at the teacher's respective school. Teachers who were aware of the CL program were then asked to rate their level of agreement with 16 statements regarding the impact of CL on students, teacher readiness to integrate CL resources into their instructional practices, and overall support for the CL program. Items were rated with a five-point Likert-type scale that ranged from "(1) -- Strongly Disagree" to "(5) -- Strongly Agree". The next TCLQ section asked teachers to provide information regarding their experience with CL and how they used CL resources (e.g., selection of activities, use 12 of specific reports). The final items asked teachers to list the strongest and weakest aspects of CL and whether or not CL use should be continued at their school. Parent Survey. The LSRD CompassLearning Parent Survey was designed to ascertain parent awareness and perceptions regarding their child's use of the CL program. They were asked to complete one survey per household, but indicate the number and ethnicity of their school-aged child/children. If the parent or guardian were aware of the CL program, they were asked five general questions regarding student attitudes about CL and the value of CL. The final section of the survey consisted of three open-ended items to record parents' perceptions of the best and worst aspects of their child's/children's use of CL and what changes they recommended . Student Focus Groups The Student Focus Group Protocol focused on three aspects regarding student use of CL. The first aspect was the nature of CL activities (e.g., how long students had been using CL, if CL lab time was long enough). The second aspect was student perceptions of the usefulness of CL. This was documented by asking students if they thought that CL helped them to do better in school, if there were any disadvantages to using CL, and what parents and friends thought of CL. The third aspect focused on student enjoyment of using CL. At the conclusion of the focus groups, students were asked to contribute any additional comments. Interviews Interviews. The following five interview protocols were developed for the evaluation: Lab Attendant, Technology Specialist, District (CL) Personnel, Principal, and CL National Representative. The lab attendant, technology specialist, and district (CL) personnel interviews focused on the same general areas: CL responsibilities
level, nature, and quality of CL implementation
support for the CL program
and , overall questions regarding the strongest and weakest aspects of CL. Participants were also asked if the program should be continued. The interview protocol for the National CL representative used a more open format, as the primary purpose was to determine the CL guidelines for effective implementation of the CL program. The principal interview was designed for principals that had previously used the CL program in their schools. The interview questions examined past use of CL, e.g., years used, overall impression 13 - of CL, how it was implemented, and why they discontinued its use. The final section included questions about current use of an ILS. PROCEDURES The nine data collection measures are summarized in Table 3 by type of measure, instrument, number completed and the data collection procedure. Table 3. Data Collection Summary Type of Measure Classroom Observations Surveys Student Focus Groups Interviews Instrument SCU-CL TCLQ LRSDCL Parent Survey Student Focus Group Protocol Lab attendant Interview Protocol Technology Specialist Interview Protocol District (CL) Personnel Interview Protocol Compass Leaming Representative Interview Protocol Pnncipal Interview Protocol Number Completed 7 356 451 4 groups
23 5"' grade students 6 3 2 4 Data Collection Procedure Prearranged two-hour sessions in which random classes were observed using CL at 7 randomly selected elementary schools. Note forms were completed every 15 minutes during the two-hour period. TCLQs were administered at a faculty meeting at each of 19 elementary schools. Compass Leaming Parent Surveys were distributed at 5 randomly selected elementary schools. One class from each grade at the school was selected and all parents of children in that class were asked to complete a survey. Surveys were returned to the school and sent to evaluators. Student Focus Groups were conducted by researchers at 4 randomly selected elementary schools. Each focus group interview was approximately 20-minutes in duration. Researchers individually interviewed lab attendants at 6 of the 7 randomly selected schools. Interviews lasted approximately 45 minutes to one hour. Only six interviews were conducted as one person was out on sick leave. Researchers conducted 45- to 60-minutes phone interviews with all CL technology specialists Researchers interviewed two district level personnel with responsibility for the Compass Leaming program. Interviews lasted approximately 45- to 60-m1nutes. A researcher conducted a 45-minute phone interview with a national Compass Leaming representative that was familiar with the Little Rock program. Researchers individually interviewed principals from 1 elementary school, 2 middle schools, and 1 high school that no longer used CL. Each interview lasted approximately 60 minutes. 14 RESULTS The results of the study are presented below by measurement strategy. In the Conclusion section, the findings are synthesized across instruments to address each research question. Classroom Observation Results The Survey of Computer Use for CompassLearning (SCU-CL) results reflect data collected during 14 hours of observations conducted at seven randomly selected elementary schools that were implementing CompassLearning (see Table 4). During these observations, 19 kindergarten through 51h grade classes, comprised of 277 African American students and 50 non-African American students engaged in CL activities in labs equipped with one up-to-date computer for each student. Student time-on-task for the CL activities ranged from 30- to 60-minutes. At least one lab attendant was present and in almost every observation a classroom teacher was also present during each session. All students in each class, both African American and non-African American, equally participated in CL activities during the observations. The majority (71.4%) of the CL activities were either for language arts or mathematics, while some (42.9%) were focused on reading. Activities for other subject areas were rarely observed. Over one-half (57.2%, frequently to extensively) of the observed CL lessons started with no instructions being given to the students. During the lessons, students asked questions related to the CL content in 43.1 % (frequently to extensively) of the visits. Student questions regarding the CL software, the computers, or other issues occurred less frequently during CL lab time. The lab attendants were observed actively moving among the students while they worked on CL activities during slightly over half of the visits (57.1 %, occasionally to extensively). Both African American and non-African American students exhibited a high level of attention and interest in the CL activities, and all observed classrooms had a high level of academically focused class time. 15 Table 4. Survey of Computer Use for Compasslearning (SCU-CL) Results n = 7 two-hour observations ( 19 classrooms) Number of students by ethnicity Elementary School All grades observed African Non-African Brady Chicot Geyer Springs McDemiott Rightsell Stephens Wakefield Total Kand 5 Kand 3 4 and 5 Kand 1 2and 3 K and2 3,4and5 American 29 49 42 44 33 29 51 277 (84.7%) Computer Configuration and Use How many computers were available for Compassleaming? Only one 2-4 5--10 11 or more During the observation, what percentage of the following students used Compassleaming? African American Caucasian CompassLearning Activities Subject Areas of Observed CL Activities Language Arts Mathematics Reading Science Social Studies Writing Cross-Cultural Types of student questions while using CL Content Area (e.g. how to solve a problem-the meaning of a word) Software use (e.g. how to log in or change sections) Computer use (e.g. how to use keyboard or mouse) Non-CL question (e .g. Do I have to sit next to .. . ? May I go to the restroom?) Types of Instruction given to students at the start of the lesson Content area review (e.g. reading, math) Software use (e.g. how to log in, find correct lesson) Computers use (e.g. locate software, use mouse) Classroom behavior rules No instruction given Teacher movement among students High level of teacher movement among students during CompassLeaming labs Student attention and academic focus High level of Afncan American student attention, interest, and engagement while using CL High Level of Non-Afncan American student attention, interest, and engagement while using CL High level of academically focused class time American 5 3 14 0 0 27 50 (15.3%) % Observed 0.0 0.0 0.0 100.0 Only a few (Less than 10%} 0.0 0.0 Not Observed 0.0 0.0 14.3 57.1 85.7 85.7 71.4 Not Observed 0.0 28.6 57.1 14.3 Not Observed 85.7 71.4 85.7 57.1 42.9 Not Observed 14.3 Not Observed 0.0 28.6 0.0 Number of teachers present Number of classes observed 2 3 2 3 2 3 4 19 Lab attendant 2 8 Most of the computers used for Compassleaming were Up to date Aging but adequate Outdated/limited capacity Some {About 10%} Most (51-90%) 0.0 0.0 0.0 0.0 Percentage of classes Rarely Occasionally F reg uently 0.0 28.6 0.0 0.0 0.0 28.6 42.9 0.0 0.0 28.6 0.0 0.0 14.3 0.0 0.0 14.3 0.0 0.0 28.6 0.0 0.0 Teachers 2 3 2 3 2 4 17 % Observed 100.0 0.0 0.0 Near1yaIVall (About91- 100%} 100.0 100.0 Extensively 71.4 71.4 42.9 0.0 0.0 0.0 0.0 Rarely Occasionally Frequently Extensively 14.2 14.3 42.9 71.4 Rarely 0.0 14.3 0.0 28.6 0.0 Rarely 28.6 Rarely 0.0 0.0 0.0 42.7 57.1 0.0 14.3 Occasionally 0.0 14.3 14.3 14.3 0.0 28.8 0.0 0.0 0.0 Frequently 0.0 0.0 0.0 0.0 28.6 14.3 0.0 0.0 0.0 Extensively 14.3 0.0 0.0 0.0 28.6 Occasionally F reguently Extensively 28.6 14.3 14.3 Occasionally Frequently Extensively 0.0 0.0 0.0 42.9 0.0 0.0 57.1 71.4 100.0 16 Survey Results This section contains the results of the Teacher CompassLearning Questionnaire {TCLQ) and the CompassLearning Parent Survey (Parent Survey). Teacher CompassLearning Questionnaire (TCLQ) The Teacher CompassLearning Questionnaire {TCLQ) was completed by a total of 356 teachers working at schools implementing the CompassLearning program. The respondents were primarily Caucasian (63.2%), and secondarily African American (32.6%), with most respondents being female (88.2%). As shown in Table 5, The TCLQ begins with two questions about the visibility and perceived value of the CL program. A very high percentage of teachers (96.1 %) reported familiarity with CL and most (93.3%) supported the continued use of CL within their respective schools. The next 16 TCLQ items assessing teacher perceptions of CL were indicative of a relatively high level of teacher approval for the program . Many items were rated highly on the survey (as measured by a combination of the categories "Strongly Agree" and "Agree". At least 85% of the teachers agreed or strongly agreed that the school computers for CL were in good condition (90.5%), the administration supported teacher use of CL (88.8%), and the teachers were supportive of CL (88.5%). Additional positive responses included agreement that CL increased student learning (78.4%), increased the level of student attention, interest, and engagement (77.3%), and was valuable for improving the achievement of African American students (73.0%). Approximately 70% of the teachers agreed that they routinely aligned CL with their lessons (72.5%), reviewed CL content in class (71.0%), and customized CL activities to meet individual needs of students. The concluding section of survey items addressed the use of the CL reporting functions. Teachers reported frequent use of "Class by Ethnicity'' (59.8%), "Class by Gender" (64.6%), and "Other" (78.6%) CL reports, while over one-half (53.7%) reported never using "Whole Class" and 45.2% reported never using "Individual Student" reports. 17 Table 5. Teacher Compass Learning Questionnaire (TCLQ) Results n = 356 Items Do you know about the CompassLeaming (computer lab} program at your school? Do you think your school should continue using CompassLeaming? %No 03.93 06.74 % Yes 96.07 93.26 Percent of Teacher Response Items Most of our school computers that are used for CompassLeaming are kept in good working condition. The administration fully supports teacher use of CompassLeaming resources. Teachers in this school are generally supportive of the CompassLeaming program. My computer skills are adequate to access CompassLeaming resources. I can readily obtain answers about CompassLeaming. The use of CompassLeaming has increased student learning and achievement. The use of Compass Leaming has increased the level of student attention, interest, and engagement in learning. Our school has a well-developed plan that guides the CompassLeaming program. Overall, this program seems valuable for improving the achievement of African American students. I routinely align CompassLeaming with my lessons and the district's standards based curriculum. I routinely provide academic review of content covered during the use of Compass Leaming. The use of CompassLeaming has improved the quality of student work. I routinely customize CompassLeaming activities to meet the individual needs of students. I have received enough training to address student learning needs through the use of CompassLeaming resources. Parents and community members support our school's use of CompassLeaming. I routinely modify my instructional practices on the basis of student performance in CompassLeaming. How frequently do you use the following Compassleaming reports? Whole Class Individual Student Class by Ethnicity Class by Gender Other Disagree & Strorgly Disagree 4.21 1.12 1.40 4.78. 4.21 2.25 4.49 4.78 1.97 4.78 7.30 3.93 7.87 14.33 2.81 12.92 Never 53.65 45.22 5.62 6.18 3.09 Neutral Strongly Agree & Agree 5.34 90.45 7.58 88.76 9.27 88.48 10.67 83.71 12.08 83.43 16.57 78.37 17.42 77.25 19.10 75.28 22.75 73.03 21 .35 72.47 20.22 71 .00 23.88 70.22 23.03 67.13 19.66 65.45 31 .74 64.61 30.90 54.78 Seldom Frequentiy 23.60 14.04 25.56 20.22 23.31 59.83 21 .35 64.64 5.06 78.65 18 The teachers also responded to open-ended questions about the use and effectiveness of the CL system. Following are highlights of the responses. Detailed results for the TCLQ open-ended items are found in Appendix D. Familiarity with CL. As seen in Table 5, almost all of the teachers completing the TCLQ were famil iar with CL. However, the open-ended comments revealed that only about one-third (37.2%) had used CL for more than 3 years. Perhaps the relative lack of experience with CL contributed to the low response rate to the item asking teachers to describe their understanding of the CompassLearning program. Less than one-fourth (84 of 356) of the teachers responded to this item. Within those responses, all the teachers revealed a general understanding of the overall purpose of the CL program. The most common responses were that CL reinforces skills addressed by the district framework (58.3%) and that CL was a program designed to enhance student learning (21 .4%). Teacher responses were somewhat evenly divided when asked how subject areas and performance levels of activities were selected. Approximately one-fourth of the respondents indicated that decisions were made on the basis of student abilities, skills, and/or grade level (25.1 %), while a similar number responded that testing results were used (23.2%). CL reports. Respondents clearly identified assessing student growth and academic progress (63.3%) as the primary reason for using CL reports. Only 56 out of 356 (15.7%) teachers indicated that they did not use CL reports. The most common reasons were: use of "other" assessment data/tools (33.9%)
lack of time (26.7%), and lack of skill in using CL (23.2%). Representative teacher comments included: I don't use them frequently because I don't think it's always a good assessment for all learners. Arkansas Reading First data is my primary diagnostic tool. I seldom use the reports. I monitor while they are working. I consider CompassLearning to be a good way to practice skills. I have very little knowledge of Compass. Strongest and weakest aspects of CL. The strongest aspect of CL identified was reinforcement of skills and the curriculum (41.2%), followed by the ability to adjust the program according to the needs of the teachers and of the students (32.9%). Student engagement or interest in learning was mentioned by 15.8% of the respondents. CL weaknesses reported by the teachers were primarily related to software issues or difficulty that students had in using some of the program features (58.6%) 19 and that the schedule did not allow enough time for students to benefit from the program (21.7%). Seventy-five percent of the teachers wrote a response to the final question, which asked teachers why their school should continue using CL. The responses were fairly evenly distributed across three reasons: CL reinforces the curriculum (33.3%)
CL enhances student learning and skills (26.2%)
and use of CL increases student engagement and excitement in learning (20.9%). The following are sample teacher responses highlighting why their schools should keep CL: The children love it. It is great practice for skills. Compass is a great tool for addressing specific student needs. Students enjoy working at their own pace and ability and it provides good reinforcement for what you are teaching in class. It allows students to work toward mastery of certain academic skills presented in class on an individual basis. It is a wonderful tool for records of student growth. It has been a great plus to student learning. It allows teachers the freedom to set it to their teaching and student needs. This program has been a bright innovative way of reinforcing methods of skills and the students work and learn on their own pace which shows results of what they are learning. I think it is a very valuable tool to help strengthen our students' skills in a fun way they enjoy. Going to computer is a treat! LRSD Compasslearning Parent Survey (Parent Survey) Of approximately 625 Parent Surveys distributed to children in five randomly selected schools, 451 parents responded (72.2% response rate). Over 84% (84.3%) of the children represented by parents completing surveys were African American, followed by 8.0% who were Hispanic and 2.0% who were Caucasian. Parental awareness of the CL program was high (82.5%), and most parents indicated they learned about the CL program via their children (55.7%). When asked how excited their child or children get about using CL, 60.1 % responded "a lot", 21 .7% reported "some", while less than 3% (2.9%) indicated that their child or children did not get exited about using CL. Parent responses indicate general approval of the CL program by agreeing that time in the computer lab: was important in their child's education (84.5%)
had improved their child's interest in school (63.2%)
and had improved their child's achievement (57.7%). The parents' responses are summarized in Table 6. 20 Table 6. LRSD Compasslearning Parent Survey Results n = 451 Items Do you know about the Computer Lab (Compassleaming) program at your child's/children's school? Indicate the ethnicity of your child/children Caucasian African American Hispanic Asian Multi-Ethnic How did you learn about the computer lab activities? School Teacher My Kids Other parents/friends How excited do your child/children get about using the computer lab? Aloi Some Not at all Not sure % Yes 82.48 % 02.00 84.26 07.98 00.00 01.33 % 11 .53 09.53 55.65 01 .55 % 60.09 21 .73 02.88 04.66 Percentage: Yes Somewhat No Not Sure Do you think time in the computer lab is an important part of your child's/children's education? 84.48 05.54 00.44 00.67 Do you think time in the computer lab has improved your child's/children's interest in school? 63.19 17.74 03.55 04.21 Do you think time in the computer lab has improved your child's/children's achievement in reading and math? 57.65 18.40 01.77 06.43 In order to gain a more complete understanding of parental perception of the CL program, parents were asked three additional questions about CL. These questions were designed to elicit open-ended responses and comments from the parents. Highlights of the parent responses are presented below and a full summary is located in Appendix E. Approximately two-thirds (273 of 451
60.5%) of the parents wrote a response when asked to describe the best thing about their child or children using the "computer lab program". The most common responses were that students were learning how to use computers (43.9%) and were acquiring new learning skills or learning in a better way (38.0%). Less than 20% (17.9%) indicated that the best aspect was improvement of math and reading skills. When asked about negative or "worst" aspects of CL, there was a 34.1 % response rate (154 of 451) with 59% of the respondents indicating that there were no negative aspects. Thirty-five parents or guardians (22.7%) were concerned that their child or children might have access to inappropriate Internet-based materials 21 when using CL. A small percentage (7.7%) of parents indicated the worst aspect of CL was that it replaced traditional classroom/textbook learning with computer-based learning. Sample parent comments reflecting this concern included "The act of just using the computer lab program may take away from the child's own thinking and reasoning capacity." And, "It may get them too adapted until they wouldn't want to sit in a classroom." The last open-ended item on the Parent Survey asked parents what changes they would like to see in the computer lab program or CL. Of the 158 (35.0% of 451) parents who responded to the last question, 50.6% indicated that their child or children needed more computer time and 25.3% wanted more variety in the lessons, content areas, and types of activities. Nearly the same percentage (24.5%) of parents responded that no changes were needed. Student Focus Group Results A total of 23 5 th grade students from 4 randomly selected elementary schools participated in the 20-minute focus groups. The students' experience with CL ranged from less than one year up to six years, with 42.8% indicating they had used CL since pre-kindergarten or kindergarten. Full results of the Student Focus Group are in Appendix F. Nature of the CL activities. The students reported that they primarily completed CL activities in language arts (33.3%), mathematics (28.6%), and reading (19.0%), in both computer labs and their classrooms. Of the 23 students, 10 wanted to spend more time using CL, 6 replied that CL time was appropriate, and 2 students indicated they spent too much time in the CL lab. Technical concerns reported by the students were focused on hardware (computer freezing, slow startup) and software malfunctions. When asked how they would change the CL activities, the most common response was to make the activities more fun or game-like (42.8%), while 21.4% requested Internet access, and more math, science, and spelling activities. Usefulness of CL activities. When asked if CL helped them to "do better in school", 17 of the 23 students responded, "Yes". The most common reason cited for this response (83.3%) was that CL helped the students remember what they learned in class and provided practice in and out of the classroom. Students indicated that the disadvantages to using CL were that sometimes the activity problems were too hard (52.8%) or that some were too easy or boring (32.0%). The majority (71.4%) of the students responded that their parents were supportive of the CL program. The students were 22 less positive when asked what their friends said about using CL, with 50.0% saying their friends thought the program was dumb or boring. Responses were mixed when asking which types of students would benefit the most and least from using CL. Students indicated that CL would be most beneficial for students who were immature (27.2%), tried their best or liked CL (27.2%), were smart (18.2%), or disadvantaged (18.2%). They thought that CL would be least beneficial to students who didn't try (66 .7%) or were smart (22.2%). Enjoyment of CL. The majority (89.5%) of the students indicated that they really liked or loved CL, while only 2 students indicated that they liked it a "little bit" (10.5%). Students' favorite things about using CL were that it helped them to learn/understand class work (31 .2%), was exciting and fun , helped learn new and harder things, and helped with math and reading (18.7% each). The least favorite things were that students could not use calculators when completing CL math lessons (27.2%), the spelling, social studies, and math lessons (27.7%), and repeating lessons (22.2%). All (23) students responded that they would like to continue using CL. Interviews The interview results are presented in three sections. First are the interviews with those directly responsible for overseeing implementation of the CompassLearning program: lab attendants (n = 6), technology specialist (n = 3), and LRSD personnel (n = 2). This is followed by the CL national representative interview summary. The section ends with the interview results from four high school principals whose schools did not use CL in 2004-2005, but had previously used CL at their respective schools. Lab Attendant, Technology Specialist, and District (CL) Personnel Highlights from the Lab Attendant, Technology Specialist, and District (CL) Personnel interview results will be discussed by these major interview categories: CL responsibilities, implementation, overall strengths and weaknesses, and if the program should be continued. Detailed results of these three interviews are in Appendix G. CL Responsibilities. The interview responses revealed that CL responsibilities were distributed as would be expected with the lab attendants having the greatest level of interaction with the students during CL lab time. Both the lab attendants and technology specialists worked with teachers to align 23 CL activities with lessons and to produce CL reports. The technology specialists provided teacher training, while the district personnel were responsible for general oversight of the program and served as liaisons between all CL district personnel. CL implementation. When asked to evaluate the CL software and support from the vendor, responses from the three groups ranged from "fine" to "great" with regard to the software and from "OK" to "excellent" concerning vendor support. The groups did, however, display consensus that there was a high level of teacher support for the CL program and that the program had been implemented very well. The best aspects of CL implementation included integration of CL activities with LRSD curriculum, custom sequencing of the units, and the use of reports for tracking student progress. Suggestions for improving the implementation process were to create district guidelines, increase supervision, and examine other remediation options. When examining how the program was implemented, interview responses indicated that the primary users of the CL program were students in grades pre-kindergarten through 5 who used CL from 1 to 5 days per week. Respondents from the three groups indicated that the teachers were regularly provided CL reports on student progress and actively used the reports to plan student CL activities. School administrators' most frequent use of the CL data was for year-end reports. Equitable student access and appropriate implementation of CL were ensured by means of close monitoring by the lab attendants, technology specialists, and principals. Overall strengths, weaknesses, continuation. There was once again consensus among the respondents with regard to the strongest aspects of the CL program. The interviewees from each group cited program flexibility to meet individual student needs, alignment of CL with LSRD curriculum standards (except math), and student enjoyment/engagement as the strongest aspects of CL. Differences were noted, however, in responses about the weakness of the CL program. The lab attendants indicated that students needed more CL time and that some of the lessons required special equipment or software. The technology specialists responded that the CL math curriculum was not well aligned with LSRD curriculum, the program lacked curriculum mapping, and sometimes the oral commands were difficult for students to understand. From the district perspective, one respondent indicated the CL program lacked oversight, while the second one did not state a 24 weakness. There was unanimous agreement that the CL program should be continued due to its positive contribution to student learning. National CL Representative One of Compasslearning's national representatives whose responsibilities included managing Cl's research and evaluation program was interviewed. This person was also knowledgeable about the LRSD program. The purpose of the interview was to obtain the representative's perspective on how schools should implement CL to improve student academic achievement. Her responses are divided into guidelines for students, teachers, and administrators. Students. With regard to student use of CL, or time-on-task, the recommended guidelines are that students in grades 3 through 12 should spend from 60 to 90 minutes per week completing 4 to 5 new activities per content area. Student activities should be focused on the subject that needs the most improvement. For students in grades K through 2, the recommended time for computer activities is 60 minutes per week per content area. Teachers. The CL representative indicated that teachers should be actively involved in using the CL reports to evaluate the progress of individual students. She recommended that teachers review student reports eve,y two weeks in order to align classroom instruction to identified needs, plan off-line (non-CL) remediation and adjust CL activities to address areas of non-mastery. Administrators. The CL representative recommended that school principals review CL class-byclass reports on a monthly basis to identify any areas of need. When teachers and administrators implement these "best practices" strategies, they will be provided with formative evaluation data to help determine if: student learning is "on-track
if the curriculum needs to be altered
and, whether or not students need more off-line activities". However, the CL representative indicated that most schools do not achieve this level of implementation. School Principals The principals from four LRSD schools (1 elementary school, 2 middle schools, and 1 high school) who no longer use Compasslearning were interviewed to examine how CL was used in the past, why it was discontinued, and if the school had implemented a different integrated learning system (ILS). Collectively, the student population of the four schools was 90% African American. Key findings from the interviews are presented below and the full interview summary is in Appendix H. 25 Past use of CL. Three of the four principals indicated a positive overall impression of CL, while one reported that the teachers disliked it. All four principals reported that their school had implemented the CL program in computer labs. However, the amount of time students spent using the software and the student populations that used CL were different for each school. One principal indicated that all students used CL, one responded that only students in English classes used it, another reported that CL was used by magnet students, and the fourth indicated that CL was used by students who needed extra academic assistance. When asked how teachers were involved with CL, 3 principals indicated their teachers aligned CL lessons with classroom instruction, 2 said teachers helped monitor the labs, and 1 reported that the teachers did very little with CL. The principals reported the primary reasons for not continuing with CL were related to server or system problems. Other reasons were lack of enough Title 1 funds, teacher complaints that the program took too much time to align with lessons, and teacher complaints of too many technical problems. Current use of an /LS. Three principals indicated that their schools were currently using different integrated learning systems {ILSs). One middle school principal reported that his school was using two systems (Accelerated Reader, and Destination Math), while the other middle school principal had adopted River Deep. The ILS adopted by the high school was Plato. The principals reported positive impressions of the new programs, indicating that fewer problems were occurring and that teachers were more accepting of the new systems. The fourth principal reported that a technology integration approach was being implemented rather than using an ILS. The principal further explained that with this approach students use basic software applications like spreadsheets, presentation and word processing software to enhance classroom environments. Student Achievement Results Research Questions Analyses of student achievement on the Development Readiness Assessment (ORA), Iowa Tests of Basic Skills, (ITBS), and the Arkansas Benchmark tests were conducted to address the following research questions: 26 1. What are the effects of the Compass Learning program on student achievement in Reading and Mathematics? 2. Is the program effective in meeting the academic needs of African American students? 3. How does the achievement of African American students compare to that of other students in the same or similar schools? Sample The elementary school sample included a total of 7,853 LRPS students in first grade (n = 2,016), second grade (n = 1984 ), third grade (n = 1904) or fifth grade (n = 1,949) during the 2004-'05 school year. Kindergarten and fourth grade were excluded because these grades lacked corresponding 2003-04 data to be used as a "pretest" (statistical covariate) .. All 34 LRSD elementary schools were included in the analysis. Overall, 67% of the student sample was eligible for free or reduced-price lunch and 68% was African American. A majority (60%
n = 4,728) attended one of the 21 schools that offered the Compass Learning program. School Matching Process Elementary schools. To facilitate analysis of program effects, elementary schools were matched via hierarchical cluster analysis into homogeneous sets on the basis of percentage eligible for free or reduced price lunch and percentage African American enrollment. Ward's minimum variance method was used as a clustering algorithm, and squared Euclidean distances were used as similarity measures. Three homogeneous School Sets were identified: Set 1 (Low Poverty, Majority White) contained 4r schools with a mean percentage of students eligible for free lunch of 24.5% and mean African-American enrollment of 35.0%
Set 2 (Medium Poverty, Integrated) contained 9 schools with a mean percentage of students eligible for free lunch of 55.5% and mean African-American enrollment of 56.4%
and Set 3 (High Poverty, Majority African American) contained 21 schools with a mean percentage of students eligible for free lunch of 84.9% and mean African-American enrollment of 83.3%. Middle schools. Two middle schools, Cloverdale and Henderson, implemented the Compass Learning program. Two comparison middle schools, Southwest and Mabelvale, were selected on the 27 basis of similarities in percentage of African American students and students eligible for free lunch. Percentages of African American students were 81.1 % at Cloverdale, 81.3% at Henderson, 81.4% at Mabelvale, and 93.1 % at Southwest. Percentages of students eligible for free or reduced-price lunch were 82.9% at Cloverdale, 69.6% at Henderson, 74.4% at Mabelvale, and 83.5% at Southwest. A total of 562 sixth graders and 531 seventh graders were included in the analyses. Measures The Developmental Reading Assessment (ORA) was administered in grades K-2 in 2003-04 and 2004-05, and the Iowa Test of Basic Skills (ITBS) was administered in grades K-5 in 2004-05. The ITBS provides normal curve equivalent (NCE) scores in Reading and Mathematics. The Arkansas Benchmark test was administered in grade 4 in 2003-04, and in grades 3-5 in 2004-05. However, scale scores from the 2004-05 Arkansas Benchmark test were not available as of December 2005. Construction of ORA NCE scores. ORA scores in 2004-05 were used as outcome variables for grades 1 and 2. The kindergarten and first-grade forms of the ORA administered provides six separate scale scores, in letter recognition, word recognition, capitalization, writing, dictation, and a composite reading score. The second-grade forms provide scores for word recognition, writing, dictation, and composite reading. The subscale scores were factor-analyzed using principal components analysis to generate regression-based factor scores for use in subsequent analyses. For 2004-05 first graders, a single factor accounted for 45.7% of the variance across 2004-05 ORA subtests. For 2004-05 second graders, a single factor accounted for 63.0% of the variance across the four 2004-05 ORA subtests. The regression-based factor scores were then converted to nonmal curve equivalent (NCE) scores with mean of 50 and standard deviation of 21.06 to aid in the interpretation of results. The norm group for these NCE scores was comprised of all Little Rock students for whom data were available in a given year. Analyses A 2 (Compass versus comparison) X 2 (African American versus non-African American) X 3 (Homogeneous School Set) analysis of covariance was performed at each grade level (1 and 2), with 2004-05 ORA and ITBS test scores as outcome variables and free or reduced price lunch status and 2003-04 ORA test scores as covariates. Follow-up analyses of covariance were performed within levels of the independent variables when significant interaction effects were observed. Effect size 28 estimates were determined by computing mean differences between groups for which significant differences were observed, then dividing by the total standard deviation. In cases where the outcome variable was an NCE score, a standard deviation of 21.06 was used in the effect size estimate computation, because the population standard deviation is known. Table 7 provides an overview of pretests, posttests, and pretest-posttest correlations for the analyses of covariance. As shown, the covariates used were 2003-04 ORA scores in grades 1-3, ITBS scores in grade 6, and Benchmark scores in grades 5 and 7. In addition, a descriptive profile of 2004-05 mean ITBS NCE scores in Reading and Mathematics across grade levels by race was generated to provide on overall basis for comparing African American achievement to that of other students. Table 7. Covariate-Outcome Correlation Coefficients by Grade Level: LRPS Elementary Schools Grade 2003-04 Pretests (Covariates) 2004-05 Posttests (Outcomes) Covariate-Outcome r 1 ORA Dictation ITBS Reading NCE 0.51 ORA Capitalization ITBS Mathematics NCE o.5i ORA Word Recognition DRANCE1 0.582 2nd ORA Composite Score ITBS Reading NCE 0.61 2 ORA Composite Score ITBS Mathematics NCE 0.502 ORA Word Recognition DRANCE1 0.712 3' ORA Total Score ITBS Reading NCE 0.592 ORA Total Score ITBS Mathematics NCE o.5i 5lh Benchmark Literacy SS ITBS Reading NCE 0.772 Benchmark Math SS ITBS Mathematics NCE 0.842 6th ITBS Reading NCE ITBS Reading NCE 0.792 ITBS Mathematics Concepts NCE ITBS Total Mathematics NCE 0.802 ITBS Reading NCE - 7th Benchmark Literacy 0.71 ' Benchmark Mathematics ITBS Mathematics NCE 0.782 'Score derived from factor analysis and normed on Little Rock School District students. 2Significant at p < .01. - 29 RESULTS First Grade !TBS Reading NCE. The model accounted for 32.4 % of the variance in 2004-05 ITBS Reading NCE scores. ANCOVA indicated significant main effects for 2003-04 ORA dictation (F, .1335 = 338.4, p < .001 ), free lunch status (F ,.1335 = 25.4, p < .001 ), race (F1.1335 = 12.92, p <.001 }, treatment (F 1_1335 = 5.827, p = 0.02), and School Set (F2_1335 = 3.00, p = .05). Variables that predicted a significant amount of variance in the outcome accounted for 19.5% of the variance in 2004-05 ITBS Reading scores. As shown in Figure 1, most of the explained variance (17%) was attributable to 2003-04 ORA dictation scores, with modest percentages attributable to free lunch status (1 .3%), race (0.6%), treatment (0.3%), or School Set (0 .3%). Post hoc comparisons showed that non-African American students had a significantly higher adjusted mean score (M' = 57.5) than African American students (M' = 52.4
ES = -0.25)
comparison students had a significantly higher adjusted mean score (M' = 56.6) than Compass students (M' = 53.4
ES = -0 .16)
and students in majority White, low-poverty schools had a significantly higher adjusted mean score (M' = 57.4) than students in majority African American, highpoverty schools (M' = 52.8
ES = -0.23). 30 0.3% 0.6% 1 0.3% 1.3% 17.0% Pretest Free Lunch Race School Set Treatment Figure 1. Percentage of Variance Explained in 2004-05 ITBS Outcomes by Significant Predictor: First Grade Table 8 shows mean ITBS Reading NCE scores by race, treatment, and School Set. Observed mean scores generally fell near or above the national average of 50.0, with the lowest average observed score of 46.5 observed for comparison group African American students in high-poverty, majority African American schools. This score is about one-sixth of a standard deviation below the national average. 31 Table 8. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race, Treatment, and Homogeneous School Set: First Grade Race Treatment School Set Mean Adjusted SD ES N Mean Non-AA Comparison Low Poverty, 68.52 60.45 15.8 21 Majority White Medium Poverty, 65.34 59.14 19.9 79 Integrated High Poverty, 52.98 56.85 18.9 56 Majority African American Compass Low Poverty, 66.59 57.75 18.2 -0.13 108 Learning Majority White Medium Poverty, 63.18 58.08 20.0 -0.05 114 Integrated High Poverty, 53.58 52.90 20.6 -0 .19 60 Majority African American African- Comparison Low Poverty, 61 .83 59.96 17.5 12 American Majority White - Medium Poverty, 53 .39 52.34 17.4 129 Integrated High Poverty, 46.53 50.96 19.8 273 Majority African American Compass Low Poverty, 54.37 51 .50 21 .5 -0.40 43 Learning Majority White Medium Poverty, 49.48 49.45 17.2 -0.14 156 Integrated High Poverty, 46.74 50.40 17.7 -0.03 298 Majority African American 32 ORA NCE. The model accounted for 24 .6% of the variance in 2005 ORA NCE scores. ANCOVA indicated a significant main effects for 2003-04 ORA Word Recognition {F1.2002 = 534.0, p < .001
20.0% of variance accounted for) and treatment status (F1.2002 = 9.15, p <.01
0.3% of variance accounted for), and a significant interaction effect between treatment status and School Set (F22002 = 3.33, p <.05
0.2% of variance accounted for). Follow-up analyses revealed that Compass students in majority White, low-poverty scores had a significantly higher adjusted mean score (M' = 54.5
ES = +0.41) than comparison students in similar schools (M' = 45.9
see Figure 2). 30.00- 55.0D - 50.00 - 45.00- 40 .00 - J 5UI! 45.80 I Lov,.
Pove,t y, t ,laio1 ity V'vhite .. .. 50 .74 I t.ledium Pove,ty. t11tegI:ited School Set I High Pove11y, t ,1ajorit y .A. t, ican .American T1 eatrnent --Comparison - - - Compass Figure 2. Adjusted Mean DRA NCE by School Set and Treatment Status: 2004-05 First Grade 33 - Table 9 shows mean ORA NCE scores by race, treatment, and School Set. Mean observed DRA NCE scores ranged from a low of M = 43.64 for non-African American comparison students in high poverty, majority African American schools to a high of M = 57.69 for non-African American Compass students in low-poverty, majority White schools. Table 9. Observed and Covariate-adjusted 2004-05 ORA NCE Mean Scores by Race, Treatment, and Homogeneous School Set: First Grade Adjusted Race Treatment School Set Mean Mean SD ES N Comparison Low Poverty, 51.64 48.58 30.1 40 Non-AA Majority White Medium Poverty, 55.32 50.44 14.7 112 Integrated High Poverty, 43.64 46.57 28.9 103 Majority African American Compass Low Poverty, 59.33 53.20 11.6 0.22 154 Learning Majority White Medium Poverty, 54.92 50.70 18.9 0.01 163 Integrated High Poverty, 48.76 48.84 21.4 0.11 97 Majority African American African- Comparison Low Poverty, 44.22 43.13 38.1 21 American Majority White Medium Poverty, 49.28 48.35 22.3 171 Integrated High Poverty, 47.38 50.38 21.6 388 Majority African American Compass Low Poverty, 57.69 55.82 10.2 0.60 60 Learning Majority White Medium Poverty, 51 .74 50.78 16.7 0.12 196 Integrated High Poverty, 46.72 49.29 21.2 -0.05 511 Majority African American 34 /TBS Mathematics NCE. The model accounted for 38.8% of the variance in ITBS Mathematics NCE scores. Significant main effects were observed for 2003-04 ORA Capitalization (F, .1323 = 303.1, p <.001
13.9% of variance accounted for), African American status (F, .1323=44.5, p < .001
2% of variance accounted for), eligibility for free lunch (F, .1323=26.5, p < .001
1.2% of variance accounted for), and School Set (F2_1323=18.9, p =.001
0.6% of variance accounted for). Follow-up analyses showed that: (a) African American students had a significantly lower adjusted mean score (M' = 42.44
ES= -0.44) than White students (M' = 51.70), and (b) students attending high-poverty, majority African American schools had a significantly lower adjusted mean (M' =43.76) than students attending either medium poverty, integrated schools (M' = 47.12
ES= +0.16) or low-poverty, majority White schools (M'= 50.34
ES= +0.38). As shown in Table 10, African American students in mediumpoverty, integrated schools and in high-poverty, majority African American schools scored substantially below the national average of 50.0. 35 Table 10. Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race, Treatment, and Homogeneous School Set: First Grade Race Treatment School Set Mean Adjusted SD ES N Mean Non-AA Comparison Low Poverty, 62.10 55.23 18.3 20 Majority White Medium Poverty, 58.75 52.26 21.0 76 Integrated High Poverty, 44.19 47.67 20.9 Majority African American 58 Compass Low Poverty, 65.66 56.62 16.5 0.07 108 Learning Majority White Medium Poverty, 57.78 51 .81 19.8 -0.02 114 Integrated High Poverty, 46.78 46.60 16.1 -0.05 58 Majority African American African- Comparison Low Poverty, 46.25 45.10 20.7 12 American Majority White Medium Poverty, 42.95 41 .30 15.5 122 - Integrated High Poverty, 37.18 40.02 19.7 268 Majority African American Compass Low Poverty, 47.37 44.40 16.7 -0 .03 43 Learning Majority White Medium Poverty, 41.68 43.10 18.5 0.09 161 Integrated High Poverty, 35.86 40.73 16.5 0.03 297 Majority African American 36 Second Grade /TBS Reading NCE. The model accounted for 47.2% of the variance in ITBS Reading NCE scores . Significant main effects were observed for 2003-04 ORA Composite scores (F1.1347 = 723.1, p <.001
28.1 % of variance accounted for), African American status (F 1_1347 = 57.8, p <.001
2.2% of variance accounted for), and free lunch status (F 1.1347 = 29.0, p <.001
1.1 % of variance accounted for). A significant interaction was observed between treatment and School Set (F2,1347 = 9.95, p <.001 ). Follow-up tests showed that comparison students had significantly higher adjusted mean scores (M' =51 .88) than Compass students (M' =57.44
ES = -0.26) in medium-poverty, integrated schools, while Compass students had significantly higher adjusted mean scores (M'= 53.35
ES = +0.15) than comparison students (M' =50.17) in high-poverty, majority African American schools (see Figure 3). As shown in Table 11 all groups scored near or above the national average of 50.0 except African American comparison students in high poverty, majority African American schools. 37 - Table 11. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Second Grade Race Treatment School Set Mean Adjusted SD N Mean Non-AA Comparison Low Poverty, 66.07 59.95 22.4 29 Majority White Medium 68.52 64.40 20.8 86 Poverty, Integrated High Poverty, Majority African 48.00 53.35 20.4 54 American Compass Low Poverty, 70.45 61.48 20.0 O.Q7 Learning Majority White 88 Medium Poverty, 62.01 55.25 20.9 -0.43 106 Integrated High Poverty, Majority African 55.67 57.02 23.0 0.17 66 American African- Comparison Low Poverty, 52.35 48.05 16.6 23 American Majority White Medium Poverty, 51 .07 50.48 19.1 110 Integrated High Poverty, Majority African 43.05 46.99 18.8 264 American Compass Low Poverty, 57.47 54.31 19.8 0.30 60 Learning Majority White Medium Poverty, 48.82 48.50 20.5 -0 .09 169 Integrated High Poverty, Majority African 46.34 49.68 18.4 0.13 306 American 38 58 56 54 - 50 ~ ' Low Poverty. tAajority'1',hite \ ' ' \ Medium Poverty , Integrated School Set High Poverty . Majority t .trican .A.111erica11 Treatment - Comparison - - Compass Figure 3. Mean Adjusted ITBS Reading NCE Scores by Treatment and School Set: 2004-05 Second Grade ORA NCE. The model accounted for 52.9% of the variance in ORA NCE scores. Significant main effects were observed for 2003-04 ORA Word Recognition scores (F1.1482 = 1424.6, p <.001
44.9% of variance explained), free lunch status (F 1_1482 = 19.76, p <.001
0.6% of variance explained), and School Set (F2_1482 = 6.1, p <.01
0.4% of variance explained). Follow-up tests indicated that in low-poverty, majority White schools had significantly higher adjusted mean scores (M' = 54.1 O ) than students in either medium poverty, integrated schools (M' = 49.57
ES = -0 .22) or students in highpoverty, majority African American schools (M' = 50.81
ES= -0.16). Students in low-poverty, majority White schools scored substantially above the district average of 50.0, regardless of race or treatment (see Table 12). No treatment effects (Compass vs. comparison) were indicated. 39 - Table 12. Observed and Covariate-adjusted 2004-05 DRA NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Second Grade Race Treatment School Set Mean Adjusted Mean SD ES N Non-AA Comparison Low Poverty, Majority White 57.64 52.05 20.4 30 Medium Poverty, Integrated 52.64 50.55 21.7 102 High Poverty, Majority African 47.45 51 .29 26.7 57 American Compass Low Poverty, Learning Majority White 64.65 57.69 8.9 0.27 96 Medium Poverty, Integrated 55.65 51.45 15.3 0.04 118 High Poverty, Majority African 49.25 50.98 24.5 -0.01 72 American African- Comparison Low Poverty, American Majority White 57.54 53.00 13.7 24 - Medium Poverty, Integrated 48.70 46.61 18.5 118 High Poverty, Majority African 45.73 50.80 23.9 292 American Compass Low Poverty, Learning Majority White 55.83 53.66 18.1 0.03 62 Medium Poverty, Integrated 51 .19 49.65 16.8 0.14 189 High Poverty, Majority African 48 .66 50.15 20.3 -0.03 336 American 40 /TBS Mathematics NCE. The model accounted for 42.3% of the variance in ITBS Mathematics NCE scores. Significant main effects were observed for 2003-04 ORA Composite score (F1,1354 = 358.54, p <.001 ), African American status (F1 ,1354 = 99.36, p <.001 ), and free lunch status (F1,1354 = 26.7, p <.001 ). Post hoc comparisons indicated that non-African American students had a significantly higher adjusted mean (M' = 53.60) than African American students (M' = 41. 78
ES = - 0.56). A significant interaction effect was observed between treatment and School Set (F2_1354 = 4.09, p =0.02). Follow-up tests, as graphically depicted in Figure 4, indicated that, in low-poverty majority White schools, Compass students had a significantly higher adjusted mean (M' = 54.80) than comparison students (M' = 49.23
ES = +0.26). The mean NCE scores for African American students in either treatment at medium-poverty, integrated or high poverty, majority African American schools were substantially below the national average (see Table 13). 41 - Table 13. Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Second Grade Race Treatment School Set Mean Adjusted SD ES N Mean Non-AA Comparison Low Poverty, 60.97 55.82 22.0 29 Majority White Medium Poverty, 59.65 55.88 20.1 85 Integrated High Poverty, Majority African 42.44 46.60 20.1 54 American Compass Low Poverty, 69.31 61 .98 19.2 0.29 89 Learning Majority White Medium Poverty, 56.28 50.98 20.9 -0.23 106 Integrated High Poverty, Majority African 49.26 50.30 21 .6 0.18 66 American - African- Comparison Low Poverty, 45.83 42.64 17.9 American Majority White 23 Medium Poverty, 41 .86 41 .66 17.5 114 Integrated High Poverty, Majority African 34 .29 37.42 16.2 266 American Compass Low Poverty, 49.97 47.61 17.9 0.24 60 Learning Majority White Medium Poverty, 42.59 42.47 20.3 0.04 169 Integrated High Poverty, Majority African 36.25 38.90 14.7 0.07 307 American 42 56 - 50- 48- 44- J ~-1 sj . . . . . I Low Pove1ty. I ,lcijo1 rt 1 \,\ hrte . . . . . . r:wT:l .. ... I l,ledium Pove1t y, lntegrnted School Set -1~.0 I i High Pove1ty, t,lcijo1 rty t-.11 ican .~meric:in Treatment - Comprnison - Compass Figure 4. Mean Adjusted ITBS Mathematics NCE Scores by Treatment and School Set: 2004- 05 Second Grade Third Grade /TBS Reading NCE. The ANCOVA model explained 48.0% of the variance in 2004-05 ITBS Reading NCE scores. Significant main effects were observed for 2003-04 ORA Composite scores (F1,1310 =600.8, p <.01
15.1% of variance explained), African American status (F1,1310 = 71 .6, p < .01
4.2% of variance explained), free lunch status (F 1,1310 = 28.11 , p < .01
1.1 % of variance explained) and School Set (F2. 1310 = 21 .57
p <.001
1.3% of variance explained). A significant interaction effect was observed between African American status and School Set (F2.1310 = 6.19, p < .01 ). As shown in Figure 5, the interaction was ordinal over levels of race. Mean adjusted scores for African American students were sign ificantly lower than 43 those for students of other races in low-poverty majority White schools (M' = 52.79 versus M' =63.86
ES= -0.52), in medium-poverty, integrated schools (M' = 49.99 versus M' = 60.39
ES= -0.49), or high-poverty, majority African American schools (M' = 46.93 versus M' = 53.48
ES= -0 .31
see Figure 5). The interaction was attributable to a successively widening achievement gap between African American and other students as the percentage of poor and African American students decreased. Most groups performed near or above the national average, except for African American students attending high-poverty majority African American schools, who scored substantially below the national average (see Table 14). 44 Table 14. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Third Grade Race Treatment School Set Mean Adjusted SD N Mean Non-AA Comparison Low Poverty, 72.47 65.84 18.4 34 Majority White Medium Poverty, 63.07 59.21 20.5 76 Integrated High Poverty, Majority African 50.78 50.52 18.8 45 American Compass Low Poverty, 72.84 64.18 17.6 -0.08 91 Learning Majority White Medium Poverty, 61.93 57.62 23.3 -0.08 123 Integrated High Poverty, Majority African 50.45 50.70 18.9 0.01 53 American African- Comparison Low Poverty, 49.52 52.66 17.0 23 - American Majority White Medium Poverty, 46.46 49.28 17.1 110 Integrated High Poverty, Majority African 43.46 46.18 17.5 264 American Compass Low Poverty, 51 .26 48.40 13.3 -0.20 47 Learning Majority White Medium Poverty, 46.86 47.72 17.8 -0.07 188 Integrated High Poverty, Majority African 43.05 45.67 16.3 -0 .02 270 American 45 - 55 &~...7 ~.). .. ... 50 45 Low Pc,verty. l,lctiorij y',',hite ... ., , JU.. P..J .. I ,ledium Poverty, lntegrmed School Set .. .. .. .. High Povert 1, l,k1iority Africm1 ,t!merican Pao:8 -Other .t-.fricrn1 ti.meric:m Figure 5. Mean Adjusted ITBS Reading NCE Scores by Race and School Set: 2004-05 Third Grade /TBS Mathematics NCE. The model explained 42.0% of the variance in 2004-05 ITBS Mathematics NCE scores. Significant main effects were observed for 2004 ORA Composite scores {F11304 =426.4, p <.01
18.5% of variance explained), African American status (F1,1304 = 111 .3, p < .01
4.8% of variance explained), free lunch status (F 1,1304 = 17 .3, p < .01
0.7% of variance explained) and School Set (F2 1304 = 6.12
p <.01
0.5% of variance explained). Significant interaction effects were observed between African American status and School Set (F2., 304 = 3.36, p < .05) and treatment and School Set (F2,1304 = 3.10, p < .05). Followup analyses of the treatment X School Set interaction showed that Compass students in low-poverty majority White schools had a significantly higher adjusted posttest mean (M' = 55.93
ES = +0.21) 46 than their counterparts attending similar schools (M'=51.59
see Figure 6). Treatment differences in other School Sets were nonsignificant. Follow-up analyses of the race X School Set interaction indicated that non-African American students in all school types performed significantly higher than African American students in all school types, but the gap was especially large in low-poverty majority White schools (see Figure 7). As Table 15 shows, non-African American students achieved at or above the national average score regardless of treatment or School Set, whereas African American students achieved substantially below the national average in every condition except Compass students in low-poverty majority White schools. 47 - Table 15. Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Third Grade Race Treatment School Set Mean Adjusted Mean SD ES N Non-AA Comparison Low Poverty, Majority White 65.21 59.51 18.04 34 Medium Poverty, Integrated 62.11 58.77 19.18 75 High Poverty, Majority African 55.41 55.09 19.56 44 American Compass Low Poverty, Learning Majority White 72.96 65.43 17.32 0.28 91 Medium Poverty, Integrated 60.71 56.99 20.37 -0.08 123 High Poverty, Majority African 51 .57 51 .78 18.42 -0.16 53 American African- Comparison Low Poverty, American Majority White 40.87 43.67 16.81 23 - Medium Poverty, Integrated 45.18 47.69 18.06 110 High Poverty, Majority African 42.89 45.21 17.70 260 American Compass Low Poverty, Learning Majority White 48.91 46.42 15.75 0.13 47 Medium Poverty, Integrated 44.14 44.91 17.63 -0.13 188 High Poverty, Majority African 40.50 42.78 18.29 -0.12 270 American 48 \ Tr,-:itrn..,nt -- l ump::iri::
011 , - - ('" 1..lllj)CICS ' \ ::
4- \ 48 - I L1J \t</ Puv~1tv 1 M::ii,
,rity 'v'\hit,.. '\ \ ' ' \ J5U.OG I ' ... I 1,1..
,Jium Puvee11y, lnteqr::it.
d School Set I HiJfl P,Jv.,
1ty. 1,h:iic,rity A fri,:::111 .ll.meri,,011 Figure 6. Mean Adjusted ITBS Mathematics NCE Scores by Treatment and School Set: 2004-05 Third Grade 49 60- 55- 50- 45- 62.47 57 .88 -~ Ji5.o5 l - - ........ I Low Poverty, Majority White I Medium Poverty, Integrated School Set 53.43 ........ ffe I High Poverty, Majority African American Race -Non-AA __ African American Figure 7. Mean Adjusted ITBS Mathematics NCE Scores by Race and School Set: 2004-05 Third Grade Fifth Grade /TBS Reading NCE. The AN COVA accounted for 62.1 % of the variance in ITBS Reading NCE scores. Significant main effects were observed for 2003-04 Benchmark Literacy scale scores (F, .1507 = 1320.1, p <.001
46.7% of variance explained), African American status (F, .1507 = 62.2, p < .001
4.0% of variance explained), free lunch status (F, .1507 = 17.2, p = .01
1.1 % of variance explained) and School Set (F2.1507 = 3.3
p <.05
0.4% of variance explained). A significant interaction effect was observed between African American status and School Set (F2.1507 = 4.08, p < .05). Follow-up analyses of the race X School Set interaction indicated that non-African American students in all school types performed significantly higher than African American students in all school types, but the gap was especially large in low-poverty majority White schools (see Figure 8). Students in most 50 treatment X School Set categories performed near or above the national average, with the exception of African American students attending high-poverty majority African American schools-these students scored nearly one-half of a standard deviation below the national average regardless of treatment condition (see Table 16). Table 16. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Fifth Grade Race Treatment School Set Mean Adjusted SD ES N Mean Non-AA Comparison Low Poverty, 76.70 56.72 23.29 33 Majority White Medium Poverty, 66.57 54.90 19.77 94 Integrated High Poverty, Majority African 47.33 48.55 20.00 60 American Compass Low Poverty, 68.89 56.82 18.33 0.00 91 Learning Majority White Medium Poverty, 64.60 53.71 20.29 -0.06 Integrated 124 High Poverty, Majority African 53.07 52.80 17.44 0.20 55 American African- Comparison Low Poverty, 46.38 43.79 19.70 26 American Majority White Medium Poverty, 45.48 46.11 18.49 109 Integrated High Poverty, Majority African 39.56 46.57 17.18 304 American Compass Low Poverty, 51 .27 49.81 19.54 0.29 77 Learning Majority White Medium Poverty, 46.22 45.80 16.19 -0.01 182 Integrated High Poverty, Majority African 40.45 46.48 16.70 0.00 366 American 51 47 ,,'.
L1~1w P1Jv..-1ty. l:l
:i jc,1ity V\-hit.- J,.1.-.Jiurn h ,v.-11y . Hi,il1 P,.,v.-11y, Int.-, II ~ ... ,J l:laj,_,, ity P..tri,-011 ,1,_ m.-1i, :::111 School Set Afli'Oll /'.',.n,..,ri,
:in Figure 8. Mean Adjusted ITBS Reading NCE Scores by Race and School Set: 2004-05 Fifth Grade /TBS Mathematics NCE. The ANCOVA accounted for 71 .9% of the variance in ITBS Reading NCE scores. Significant main effects were observed for 2003-04 Benchmark Mathematics scale scores (F 1 1504 = 2368.1, p <.001
61.2% of variance explained), African American status (F1.1504 = 37 .9, p < .001
2.5% of variance explained), free lunch status (F1,1504 = 4.4, p <.05
0.3% of variance explained) and School Set (F2 1504 = 5.5
p <.01
0. 7% of variance explained). African American students had a significantly lower adjusted mean score (M' = 46.30) than other students (M' = 51 .63). A significant interaction effect was observed between treatment status and School Set (F2.1504 = 5.48, p < .01 ). Post hoc comparisons revealed that the comparison group had a significantly higher adjusted mean (M'=51.90) than the Compass group (M'=48.81
ES =-0.15) in medium-poverty, integrated schools, while the Compass students had a significantly higher mean (48.67
ES= +0.10) 52 than comparison students (M' = 46.60) in high-poverty African American schools (see Figure 9). As shown in Table 17, African American students scored below the national average of 50.0 regardless of school type or treatment, while other students scored above the national average regardless of school type or treatment. Table 17. Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race, Treatment, and Homogeneous School Set: Fifth Grade Race Treatment School Set Mean Adjusted SD ES N Mean Non-AA Comparison Low Poverty, 73.73 52.58 20.73 33 Majority White Medium Poverty, 66.69 55.45 21.38 94 Integrated High Poverty, Majority African 51 .03 48.47 20.32 60 American Compass Low Poverty, 69.75 51 .64 17.63 -0.04 91 Learning Majority White Medium Poverty, 63.48 51 .04 21.28 -0.21 124 Integrated High Poverty, Majority African 53.96 50.60 21.41 0.10 55 American African- Comparison Low Poverty, 47.35 44.87 21 .56 26 American Majority White Medium Poverty, 44.30 48.36 19.66 109 Integrated High Poverty, Majority African 38.24 44.74 17.81 301 American Compass Low Poverty, 47.22 46.54 19.46 0.08 77 Learning Majority White Medium Poverty, 43.92 46.58 16.60 -0 .08 182 Integrated High Poverty, Majority African 39.85 46.74 18.20 0.09 366 American 53 48 Lc, ,.iv p,_, ,,,..1ty, I 1l8i.>I tty 'v"il1tt.- ----~148.811 ~ - - - - ,6---' 1,1.-,Jilllll Puv .. 1t y, lrrt.-,Jr:rt.-.J School Set Hi, 1h P,Jv.-rty. li18i.rtty 6 f1i, 811 .<l 111.-1 i1:~,n - ( ,_,1n1 ,::u i::::,.,n - t vmpoc3 Figure 9. Mean Adjusted ITBS Mathematics NCE Scores by Treatment and School Set: 2004-05 Fifth Grade Sixth Grade /TBS Reading NCE. The model accounted for 63.5% of the variance in !TBS Reading NCE scores. Only two significant main effects were observed: 2003-04 !TBS Reading scores (F,.556 = 834.7, p <.001) and race (F,.556 = 4.58, p <.05). African American students had a statistically significantly lower adjusted mean score {M'= 38. 77) than other students (M'= 41.91 ), although the magnitude of the difference was small. All groups except non-African American comparison students scored substantially below the national average (see Table 18). 54 - Table 18. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race and Treatment: Sixth Grade Race Treatment Mean Adjusted Mean SD ES N Non-AA Comparison 54.06 42.32 24.04 33 Compass 40 .56 41 .50 19.78 -0.04 39 Learning African- Comparison 37.29 38.28 17.53 209 American Compass 38.76 39.27 18.15 0.05 281 Learning /TBS Mathematics NCE. The model accounted for 64.6% of the variance in ITBS Reading NCE scores. The only significant main effect was for 2003-04 ITBS Reading scores (F1,576 = 920.0, p <.001
61.5% of variance accounted for). As shown in Table 19, African American students in both the comparison and the Compass groups scored nearly a standard deviation below the national average (national M = 50, SD= 21 .06). Table 19. Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race and Treatment: Sixth Grade Race Treatment Mean Adjusted Mean SD ES Non-AA Comparison 51 .03 39.24 25.05 Compass 42.80 38.60 21 .21 -0.03 Learning N 35 40 African- Comparison 33.63 34.66 19.35 223 American Compass 36.33 37.57 16.56 0.14 284 Learning 55 Seventh Grade /TBS Reading NCE. The model explained 51.6% of the variance in ITBS Reading NCE scores. Significant main effects were observed for 2003-04 Benchmark Literacy scale scores (F, .528 = 458.1, p <.001
46.5% of variance), race (F1.528 = 8.15, p <.01
1.5% of variance), treatment (F1.528 = 7.01 , p <.01
1.3% of variance), and free lunch status (F1.528 = 5.73, p <.05
1.1 % of variance). Follow-up tests showed that African American students had significantly lower adjusted mean scores (M' = 38.65) than other students (M' = 43.51), and that Compass learning students had significantly lower adjusted mean scores (M' = 38.85
ES= -0.21) than comparison students (M' = 43.31 ). As shown in Table 20, African American students performed substantially below the national average (about onehalf standard deviation), while other students performed at or near the national average. Table 20. Observed and Covariate-adjusted 2004-05 ITBS Reading NCE Mean Scores by Race and Treatment: Seventh Grade Race Treatment Mean Adjusted Mean SD ES N Non-AA Comparison 50.85 46.63 19.31 26 Compass 46.70 40.40 18.79 -0.30 33 Learning African- Comparison 38.56 40.00 15.85 230 American Compass 37.36 37 .30 17.79 -0.13 245 Learning /TBS Mathematics NCE. The model explained 60. 7% of the variance in ITBS Mathematics scores. Only two significant main effects were observed: 2004 Benchmark Mathematics scale scores (F1,525 = 686.2, p <.001
56.7% of variance) and race (F,.525 = 5.15, p <.05). African American students had a statistically significantly lower adjusted mean score (M'= 38.52) than other students (M' = 42.19), although the magnitude of the difference was small. As with Reading, African American students performed substantially below the national average (about one-half standard deviation), while other students performed at or near the national average (see Table 21 ). 56 - Table 21 . Observed and Covariate-adjusted 2004-05 ITBS Mathematics NCE Mean Scores by Race and Treatment: Seventh Grade Race Treatment Mean Adjusted Mean SD ES N Non-AA Comparison 53.12 44.22 17.21 25 Compass 49.45 40.15 16.62 -0.19 33 Learning African- Comparison 37.25 38.16 16.88 American 229 Compass 37 .57 38.88 17.94 0.03 244 Learning Descriptive Profile of ITBS Mean NCE scores Reading. Figure 10 shows mean ITBS Reading NCE scores for African American and other students across grades 1 to 9. As seen in the figure, African American students in first grade score near the national average, but their performance declines across successive grade levels until grade 8, where the mean NCE was nearly one-half of a standard deviation below the national average. For other students, mean NCE scores were between 60 and 65 at all grades levels, or approximately onehalf standard deviation above the national average. Thus, the achievement gap is substantially wider at higher grade levels. 57 - t35- -- 45 - .... 40- 35- I I I I 1 ,-., 3 4 -... I I 5 13 GRADE - - - 1 I ,' 8 I ~ Ra,:~ - J-Jon- /Jo.A __ .t .fricon ,t .mericon Figure 10. Mean 2004-05 ITBS Reading Score by Grade Level and Race. Note: The norm group average score is 50.0 Mathematics. As seen in Figure 11, the patterns of mean ITBS Mathematics scores for African American and other students are strikingly similar, with a distinct peak occurring in fourth grade for both groups. The achievement gap is relatively constant across grade levels, with some widening of the gap at upper grade levels. As with Reading, African American students generally performed about one-half standard deviation below the national average, while other students generally performed about one-half standard deviation above the national average. 58 - 70 GO 50 ,.. / I' 40 / --I 2 3 4 ' ' ' ,,,. .,, 5 ,3 GRADE ..... ..... .. 8 -- 9 R:.ice -Non-/> .. A. .t,.fricrn1 .A.merican Figure 11. Mean 2004-05 ITBS Mathematics Score Grade Level and Race. Note: The norm group average score is 50.0. 59 - CONCLUSIONS The conclusions of this present study are presented in relation to each of the major research questions in the following sections. Primary Evaluation Question What are the effects of participation in CL on the achievement of African American and other students? What are the effects of the Compass Learning program on student achievement in Reading and Mathematics? Is the program effective in meeting the academic needs of African American students? The only simple main effects of Compass Learning on students were statistically significant negative effects on first grade ITBS Reading (ES= -0.16) Where program effects were observed, they varied by school context: On first-grade ORA, Compass students had significantly higher adjusted mean scores than comparison students only in low-poverty majority White schools (ES = +0.26). On second-grade ITBS Reading, comparison students had higher mean scores than Compass students in medium-poverty integrated schools (ES = -0.26), while Compass students had higher mean scores in high-poverty majority African American schools (ES = +0.15). On second-grade ITBS Mathematics, Compass students had significantly higher adjusted mean scores than comparison students only in low-poverty majority White schools (ES = +0.38). On third-grade ITBS Mathematics, Compass students had significantly higher adjusted mean scores than comparison students only in low-poverty majority White schools (ES = +0.26). On fifth-grade ITBS Mathematics, comparison students had higher mean scores than Compass students in medium-poverty integrated schools (ES = -0.15), while Compass students had higher mean scores in high-poverty majority African American schools (ES = +0.10). Across all grades, subjects, and achievement measures, a total of 14 separate analyses (8 Reading
6 Math) examining treatment x School Set effects were conducted. Of these, 4 (50%) in Reading and 3 (50%) in Math showed no significant treatment main effects or interactions. Where significant effects did occur in Reading, two were main effects, respectively, showing comparison group advantages over Compass in Grade 1-ITBS (ES= -.25) and Grade 7-ITBS (ES= -0.21). The 60 other two effects were interactions, one showing a Compass advantage within the low-poverty School Set (ES = +0.41) in Grade 1-DRA, and the other showing a comparison group advantage within the medium-poverty School Set (ES = -0 .26) in Grade 2-ITBS. Thus, taken as a whole, the evidence provided by this study suggests that usage of Compass Learning was not associated with noticeable achievement gains in Reading for the overall sample of students or for African American students in particular. Significant effects in Math were associated with three interactions, showing Compass advantages in low-poverty schools in Grade 2-ITBS (ES = +0.26) and Grade 3-ITBS (ES = +0.26), and in high-poverty schools in Grade 5-ITBS (ES= +0.10). Comparison schools showed an advantage in medium-poverty schools in Grade 5-ITBS (ES = -0.15). Overall, Compass learning effects were inconsistent but positive in direction in Mathematics. Benefits were more frequent in lowpoverty than high-poverty schools. A summary of aggregate effect sizes across grades, subjects, and tests is provided in Figures 11 and 12. Specifically, the box plots in Figure 11 show the interquartile ranges (middle 50% distribution) of the computed effect sizes for each School Setting. As can be seen, the ranges extend above ES = +0.20 only for the low-poverty setting. The median effect size, represented by the dark horizontal lines in Figure 11 and the bar graphs in Figure 12, indicate a weak positive effect of Compass in the low-poverty setting but essentially zero effects in the medium- and high-poverty settings. 61 0.60 - 0.40 - 0.20 - en w 0.00 - -0.20 - -0.40 - -- I Low Poverty, MaJority White T _L I Medium Poverty, Integrated School Setting T I I J_ I High Poverty, Majority African American Figure 12. Box Plot of Compass Learning Effect Size Estimates Across All Outcomes and Grade Levels by Setting: African American Students 62 0.15 0.11 0.10 -+--- 0. 05 +-----1 0.00 --1-------'-l ' ___ __,_ ____ ~---------~,---~, ----i -0.0.1 -0.05 -+---------------u_,,unt-4------------ -0.10 +----------------------- -0.15 Low Poverty, Majority White Medium Poverty, Integrated High Poverty, Majority AA Figure 13. Median Compass Learning Effect Size Estimates Across All Outcomes and Grade Levels by Setting: African American Students How does the achievement of African American students compare to that of other students in the same or similar schools? In both Reading and Mathematics across grades 1-9, ~frican American students generally performed about one-half standard deviation below the national average, while other students generally performed about one-half standard deviation above the national average. The pattern of Mathematics achievement across grade levels is virtually identical between African American and other students (see Figure 11) African American achievement in Reading declines precipitously as grade levels increase, while the Reading achievement of other students remains relatively steady across grades (see Figure 10). 63 Supplemental (Step 2) Evaluation Questions What is the quality, nature, and level of implementation of CL at the 19 elementary schools identified as implementing the program in 2004-05? The observation data indicated that CL was implemented in well-equipped computer labs that provided one computer for each student. The CL labs were almost always supervised by both a lab attendant and a classroom teacher that were responsive to student questions and actively monitored classroom activities. The data also revealed that all students in each class, both African American and non-African American, equally participated and were highly engaged and attentive to the CL activities. The interview results indicated that most teachers routinely customized their classroom instruction based on CL performance. However, nearly one-half (45.2%) of the teachers responded that they never used CL reports on individual student performance, but rather frequently used class reports by ethnicity (59.8%) and by gender (64.6%). Overall, these results indicate that the quality and nature of CL implementation was generally effective with regard to how students use CL. However, the data also revealed that teachers more frequently based instructional decisions on class-level performance rather than Cl's recommended strategy of using student-level performance to achieve individualized instruction. With regard to the level of implementation, most students' scheduled time for CL activities fell below program recommendations of 60-minutes per content area per week. Thus, the quality, nature, and level of implementation was generally effective with regard to computer lab environments and student engagement during CL use, but was inconsistent with regard to the degree to which student CL activities were individualized and the amount of scheduled CL time. What is the level of participation in CL by African American students relative to other ethnic groups at the implementing schools? The Compasslearning program was implemented in 19 LRSD Title 1 schools comprised of student populations that were 67.9% African American during 2004-2005. Triangulation of data from multiple sources indicated that African American and non-African American students fully and equally participated in CL program activities at their schools. Specifically, the direct classroom observation data revealed that of 327 students in CL computer labs a// of the 277 African American and 50 nonAfrican American students were highly engaged in the use of CL in well-supported environments. 64 When lab attendants and technology specialists were asked which students used CL, the responses indicated a// students within the specified grade ranges regularly participated in CL classroom and lab activities. The district personnel involved with CL reported that CL is implemented in schools with the highest level of student need, or Title 1 Schools. What are the perceptions of teachers, lab attendants, and technology specialists regarding CL program implementation, impacts, strengths, and weaknesses? There was common agreement across the teachers, lab attendants, and technology specialists that the CL program was generally implemented well and that the teachers and administrators supported the use of CL in their schools. Teacher responses reflected their support for CL through their agreement that CL not only increased student learning but also increased the level of student attention, interest, and engagement and was valuable for improving the academic achievement of African American students. Additionally, nearly three-fourths of the teachers agreed that they routinely aligned CL with their lessons and reviewed CL content in class. There was general consensus among the respondents that the strongest aspect of CL was program flexibility to meet individual student needs. However, teachers also identified reinforcement of skills and the curriculum as the strongest aspect of CL, whereas, the lab attendants and technology specialists noted alignment of CL with LSRD curriculum standards and student enjoyment or engagement as additional areas of strength. When asked about the weakness of the CL program, respondents from the three groups were concerned about software issues due to difficulty students had in using some aspects of the CL program. The teachers and lab attendants thought students needed more computer time to benefit from the program. The technology specialists remarked that the CL program lacked sufficient curriculum mapping. There was unanimous agreement that the CL program should be continued due to its positive effect on student learning. What are the perceptions of parents/guardians of CL students regarding program impacts, strengths, and weaknesses? The Parent Survey results reflect the target population for this evaluation in that over 84% of the surveys represented African American children who use CL at school. Most parents responded that they were aware of CL and believed that the program was an important part of their child or children's education. However, there was less agreement that CL increased student interest in school or improved achievement in math and reading. Reasoning behind these responses was revealed in 65 remarks to open-ended questions, which suggested that parents' understanding of CL was somewhat misguided. Most parents seemed to believe that their children attended the computer lab to learn about computers, rather than to improve math and reading skills. For example, the most common reply regarding the "best" part of CL was improving computer skills and the "worst" aspect was possible access to inappropriate Internet-based materials - both of which were unrelated to use of CL. Responses from less than 20% of the parents reflected a greater understanding of the overall purpose of CL in that their reported "best" aspect of the program was improvement of reading and math skills and the worst aspect was replacement of traditional learning with computer-based learning. When parents were asked about what changes they would like to see in CUcomputer lab program, half wanted more computer time for their children. However, the data suggested that this extra time was requested to improve students' computer skills. The parents that suggested more variety in CL lessons seemed to have a better grasp of Cl's purpose. Overall, parents were supportive of their child or children using computers, but many were unaware of the overall instructional purpose the CL program. What are the perceptions of school principals, whose schools no longer use CL, with regard to past use of the CL program and possible adoption of a different program? Of the four LRSD principals who no longer used CL at their schools, three indicated that CL was generally effective and that teachers liked it, however, the program was discontinued due to technical problems. One of the principals, however, was less positive about CL as a result of teacher complaints about the program. Specifically, teachers indicated it took too much time to align CL activities with their classroom lesson plans, there were too many technical problems, and they did not like monitoring the CL computer lab. Three of the four principals adopted different integrated learning systems (ILSs}, while one chose to implement a non-lLS computer integration model. The principals that adopted the new programs indicated that fewer technical problems had occurred and teacher acceptance was greater. These results suggest that although CL use had been discontinued at the four schools, the principals had a generally positive regard for the use of technology to improve student learning in schools with a predominately African American population. 66 Appendix A Compasslearning Schools by Percentage of African American Students 67 Little Rock School District Compasslearning Schools by Percentage of African American Students School Name CL Schools in 2004-2005: Elementary Schools Bale Booker Brady Carver Chicot Fair Park Forest Park Franklin Fulbright Geyer Springs Gibbs Mabelvale McDermott Otter Creek Rightsell Rockefeller Stephens Wakefield Williams Past CL Participating Schools: Elementary School Mitchell Middle Schools Cloverdale Henderson High School Accelerated Learning Center (ALC) Percentage of African American Students 82% 53% 78% 52% 73% 75% 20% 96% 26% 88% 53% 80% 62% 60% 100% 67% 95% 78% 52% 96% 82% 82% 92% 68 Appendix B 2004-2005 Time Scheduled for Compasslearning by School and Grade 69 Little Rock School District 2004-2005 Time* Scheduled for Compasslearning by School and Grade Grade Level School K 2nd 3rd McDermott 60 60 60 60 Williams 60 60 60 60 Carver 60 60 60 60 Fair Park 60 60 60 60 Forest Park n/a 30 60 60 Chicot n/a n/a 45 45 Rightsell 45 45 45 45 Stephens 30 30 30 30 Booker 30 30 30 30 Brady 30 30 30 30 Bale n/a n/a n/a 30 Franklin 30 30 30 30 Fulbright 30 30 30 30 Geyer Springs n/a 30 30 30 Gibbs 30 30 30 30 Mabelvale 30 30 30 30 Otter Creek 30 30 30 30 Rockefeller n/a n/a n/a n/a Wakefield 30 30 30 30 Overall averages 39.6 38.4 40.6 40.0 Time in minutes .. "nla" indicates grade does not use CompassLeaming 4th 5th 60 60 60 60 60 60 60 60 60 60 45 45 45 45 60 60 60 60 45 45 30 n/a 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 43.4 44.1 School average 60 60 60 60 54 45 45 40 40 35 30 30 30 30 30 30 30 30 30 40.5 70 Appendix C Evaluation Instruments 71 Survey of Computer Use for Compasslearning School ____________ Observer Name __________ Observation Date Grade(s) Observed ______ Setting: __ Classroom __ Lab Time In __ Time Out Number of Students Using CompassLeaming: __ African American __ Non-African American Number of Teachers present: _ Classroom Teacher(s) _ lab attendant(s) _ technology specialist(s) Number of Classes observed during the two hours: ___ _ Computer Configuration and Use How many computers were available for Compasslearning? One 2-4 5-10 11 or more How frequently did malfunctions occur on computers used for Compasslearning? Never _Rarely _ Occasionally _ Frequently _ Extensively Compasslearning Activities In which subject areas did students complete CompassLearning work ( check all that were observed)? _ Language Arts Social Studies Mathematics _Writing _ Reading Cross-Curricular Science TALLY the types of questions students asked while using CompassLearning. _ Content area (e.g. how to solve a problem
the meaning of a word) _ Software use (e.g., how to log in
how to move to next section, how to take a test) _ Computer use (e.g., how to get the mouse or keyboard to work properly) _ Non-CompassLeaming questions. (e.g., Do I have to sit next to John? Can I go to the restroom?) What was the overall level of African American student attention, interest, and engagement while using CompassLearning? Low Moderate _High What was the level of academically focused class time? Low Moderate _High Observer Notes Most of the computers used for Compasslearning were _ Up-to-date _ Aging but adequate _ Outdated/limited capacity During the observation, computers were used for CompassLeaming by _ Only a few students (less than 10%) _ Some students (about 10-50%) _ Most (students about 51-90%) _ Nearly all or all students (91-100%) What types of instructions were given to students at the start of the lesson (check all that were observed)? _ Content area review (e.g. reading, math) _ Software use (e.g., how to log in, find correct lesson) _ Computer use (e.g., locate software, use mouse) Classroom behavior rules _ No instructions were given What was the predominate role of the teacher(s) during student use of CompassLeaming? _ Continuously moved among students to actively monitor student work and answer questions _ Occasionally moved among students to monitor student work and answer questions _ Rarely moved among students to monitor student work and answer questions _ Remained at one location (e.g., at desk) rather than moving among the students. What was the overall level of NON-African American student attention, interest, and engagement while using CompassLeaming? Low Moderate _High 72 Teacher Compasslearning Questionnaire (TCLQ) School ______ _ Your Gender__ Your Ethnicity _________ _ Do you know about the Compasslearning (computer lab) program at your school? Yes No If Yes, please answer the remaining questions. If No, please stop hear and return the survey when they are collected. TCLQ Items 2, ~ ~ Ii i 2, ~ I! l ij I~ z 1. Most of our school computers that are used for Compasslearning are kept in good working condition. 2. I can readily obtain answers to questions about Compasslearning. 3. The use of Compasslearning has increased the level of student attention, interest, and engagement in learning. 4. Parents and community members support our school's use of Compasslearning. 5. The use of Compasslearning has increased student learning and achievement. 6. I am able to align Compasslearning with my lessons and the district's standards-based curriculum. 7. Overall, this program seems valuable for improving the achievement of African American students. 8. I have received enough training to address student learning needs through the use Compasslearning resources. 9. My computer skills are adequate to access Compasslearning resources. 10. The administration fully supports teacher use of Compasslearning resources. 11. I routinely customize Compasslearning activities to meet the individual needs of students. 12. Our school has a well-developed plan that guides the Compasslearning program. 13. I routinely provide academic review of content covered during student use of Compasslearning. 14. Teachers in this school are generally supportive of the Compasslearning program. 15. I routinely modify my instructional practices on the basis of student performance in Compasslearning. 16. The use of Compasslearning has improved the quality of student work. 73 Directions: Please use the space provided to write a brief response to the following questions. 1) How many years of experience do you have with Compasslearning? _____ Years 2) Circle the grade level(s) you currently teach that use Compasslearning K 1 2 3 4 5 6 7 8 3) How are the subject areas and performance levels of Compasslearning activities selected for each of your students? 4) How frequently do you use the following Compasslearning Reports? Whole class - Never - Seldom _Frequently Individual Student - Never - Seldom _Frequently Class by Ethnicity - Never - Seldom _Frequently Class by Gender - Never - Seldom _Frequently Other - Never - Seldom _Frequently 5.a) If you use Compasslearning reports, what is your primary reason? 5.b) If you do NOT use Compasslearning reports, what is your primary reason? ______ _ 6) What are the strongest aspects of the Compasslearning Program? _________ _ 7) What are the weakest aspects of the Compasslearning Program? __________ _ 8) Do you think your school should continue using Compasslearning? _Yes No Why?--------------------------------- Page 2 TCLQ 74 Little Rock School District (LSRD) Compasslearning Parent Survey Dear Parent/Guardian We would like to know what you think about your child or children using the CompassLearning program (the program your child or children work on in the computer Jab). Please take a few minutes to complete the following survey. Only complete 1 (one) survey if you have more than one child in the school district Directions Circle your response or fill- in the requested information in the space provided. Do not put your name on the survey. Share with us your honest opinions to help improve this program. Information about your Child/Children Circle the grade levels of your child/children K 1 2 3 4 5 6 7 8 9 10 11 12 Circle the ethnicity of your child/children Caucasian African American Hispanic Asian Multi-Ethnic Computer Lab Program (Compasslearning) Do you know about the computer lab program at your child 's/children 's school? Yes No If Yes, please answer the remaining questions. If No, please have your child return the survey to his/her teacher. How did you learn about the computer lab activities? School Teacher My kids Other parents/friends Do you think your child/children get excited about using the computer lab? Yes Somewhat No Not sure Do you think computer lab time is an important part of your child 's/children 's education? Yes Somewhat No Not sure Do you think use of the Computer Lab has improved your child 's/children 's interest in school? .. ... .. ...... ..... ....... ..... .... ... Yes Somewhat achievement in reading and math? ... ...... Yes 1 Somewhat No No Not Sure Not Sure Your Comments Please respond in the space provided and use back of sheet if more space is needed. What is the Best thing about your child/children using the Computer Lab program? What is the Worst thing about your child/children using the Computer Lab program? What Changes would you like to see in the Computer Lab program? 75 Little Rock School District (LSRD) Compasslearning Student Focus Group Protocol Nature of the Activities 1. How long have you been using Compasslearning? a. What kinds of activities do you do when using Compasslearning? b. Have you been using it in your classroom or in a lab? 2. What do you think about the amount of time that you spend using Compasslearning? a. Is it enough - or too much? b. Why? 3. What kinds of computer problems have you experienced when using it? 4. What types of changes would you like to see in the Compasslearning program? Usefulness of CompassLearning 5. Do you think Compasslearning helps you to do better in sc
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.