RECEIVED JAN 1 3 2004 CREP OFFICE OF DESEGREGATION MONITORING Center for Research in Educational Policy > Little Rock School District Literacy Program Evaluation Steven M. Ross John Nunnery Lana Smith Aaron McDonald Allan Sterbinsky Center for Research in Educational Policy University of Memphis 325 Browning Hall Memphis, TN 38152 Toll Free: 1-866-670-6147 November 2003Little Rock School District Literacy Program Evaluation Executive Summary The present report provides the results from a study of the different literacy programs used in the Little Rock School District (LRSD). After expending substantial effort and resources to improve the reading ability of students in the district, administrators at LRSD wanted to examine the effectiveness of the different programs used within the district for literacy instruction. To facilitate this exanunation, the Center for Research in Educational Policy (CREP) at The University of Memphis was employed to provide an independent, third party evaluation. The evaluation methodology and data analysis were oriented around the following research questions: 1. 2. 3. What are teacher perceptions of and reactions to the different literacy programs? After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement, did African American students exhibit similar levels of academic achievement as other students? What proportion of the variance in 2003 literacy achievement was uniquely attributable H to whether students were African American? 4. What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? 5. Were differences in achievement gains between African American students and other students similar at different grade levels and for different test instruments? 6. Was there a relationship between the literacy program implemented at the elementary schools, school composition variables (i.e., school poverty and percentage of African American students enrolled), and the achievement of African American students? Method H M The evaluation design was based on both quantitative student achievement data as well as qualitative data from K-12 faculty members who are responsible for literacy instruction. The primary data sources were (a) a questionnaire completed by teachers, (b) focus groups conducted _ - ^.1__ with faculty members representing different literacy programs in the district, and (c) student achievement data including the Literacy Benchmark scale score, SAT-9 Reading subscale score. II H M and the SAT-9 Language subscale score. During the 2002-2003 school year, two CREP researchers conducted seven focus groups at the Neighborhood Resource Center using a structured interview guide. Each focus group was approximately one-hour in duration. Teachers signed a permission form to be interviewed and were given assurance that their comments would be confidential and anonymous. The sessions were audiotape-recorded and supplemented with the researchers hand-written notes. III The Literacy Program Teacher Questionnaires were printed and shipped to LRSD personnel. The district staff members disseminated the questionnaires to the individual schools along with instructions for completing and returning the forms to the district. After the district III staff received the completed forms, they were sent to CREP for analysis. Similarly, district Page 1 of 471 H Id Id personnel assembled the student achievement data into an electronic format. The data files were then sent to CREP researchers for analysis. Id The focus group and surveys were analyzed thematically and descriptively, respectively. A synthesis was then developed to highlight findings by literacy program and grade level (e.g., elementary and secondary) as provided by the district. The achievement methodology and analysis is discussed in the achievement section below. Id Results Id Teacher Focus Groups and Literacy Program Teacher Questionnaire Id A synthesis of themes and related findings from the seven focus groups and teacher questionnaire is provided in bulleted format for a concise overview. The reader is encouraged to examine the full report for detailed findings. Id Id Id Most Effective Program Elements (Elementary) Professional development (PD) when received New materials Emergent literacy/readiness skills for Kindergarten Positive impact on student writing Literacy Coaches (in RR program) Providing instruction at students level d d Most Effective Program Elements (Secondary) Positive Impact on student writing Paired reading (instructional strategy) Portfolios and interactive journals d M Least Effective Program Elements (Elementary) Inconsistent implementation across and within schools Texts not aligned with SAT-9 Some leveled texts and vocabulary are not appropriate Transient student population is problematic Parent/community involvement is not at desired levels d d Least Effective Program Elements (Secondary) Gaps in training that is offered to teachers Lack of consistency in literacy instruction and programs (across and within schools) Lack of teacher support from non-literacy subject areas Parent/community involvement is not at desired levels d Teacher support for the Programs (Elementary) Level of support varies across schools (and within schools) Page 2 of 47V Support for ELLA and Effective Literacy is high because they fit well with what teachers were already doing or moying toward anyway SEA is most polarized (loye or hate the program) Teacher support (Secondary) Generally positiye attitudes, but not much support from non-literacy teachers District support (Elementary) Lack of consistency for teachers to attend training (substitutes
ayailability of training) District proyides materials, but could also use teacher aides in the classroom District support (Secondary) Literacy coaches would be beneficial Need to use district time set aside for inseryices to plan a more comprehensiye literacy approach * M Professional Deyelopment (Elementary) Quality of professional deyelopment receiyed has been mixed, but teachers are generally positiye Literacy Coach (proyiding leadership and training) has been beneficial General consensus that there is a lack of ongoing training (or opportunity to attend recommended/mandated number of days) II II Professional Deyelopment (Secondary) Quality of professional deyelopment receiyed has been mixed Would like more training Teachers do not see training sessions as tying together. Not sure what big picture literacy plan is II II Classroom Changes (Elementary) Positiye impact on Kindergarten students More emphasis on student writing Students are learning reading strategies There has been some return to traditional instructional practices II Classroom Changes (Secondary) More cooperatiye learning (with mixed results) Special Education teachers assist students in classes (instead of pullout program) * * M Impact on Students (Elementary) Learned better cooperation skills Increased confidence in reading Students are learning reading strategies Page 3 of 47I hl u H Impact on Students (Secondary) There is some increased motivation to read Assessments and leveled readers are creating more success and confidence for students Students are writing more often II M Impact on Teachers (Elementary) Negative impact on time and stamina Additional instructional strategies and materials have been beneficial Increased sharing of ideas H H Impact on Teachers (Secondary) Increased sharing of strategies Focus on bringing lower performing students to proficiency level High school teachers had lower levels of agreement on all survey items (comparatively) II Student Achievement Data M The primary purpose of research focus was to examine the achievement of African American students in reading and language arts in the Little Rock School District. The five achievement oriented research questions (#2 #5 above) were used to guide the methodology and analyses. Methodology IM IM IM Subjects of the study included all students enrolled in grades 3 to 11 in the Little Rock School District during the 2002-2003 school year for whom 2003 Literacy Benchmark or 2003 SAT-9 scores were available. This included a total of 11,934 students, of whom 23.4% were Caucasian, 68.2% were African-American, and 48.5% were certified as eligible for free or reduced-price lunch. Three measures were used to assess literacy, the Literacy Benchmark scale score (Grades 4, 6, and 8), and the SAT-9 Reading subscale score, and SAT-9 Language subscale score (both Grades 5, 7, and 10). I I I i I IM Analyses IM IM District-wide achievement effects. The basic analytic model used to gauge district-wide achievement effects was a 2 (free lunch status) X 2 (gender) X 2 (African-American, nonAfrican American) analysis of variance (ANOVA). This basic model was adapted to each grade level to reflect: (a) the availability of achievement data from the prior year, in which case a 2 X 2 X 2 analysis of covariance (ANCOVA) was used
(b) the specific 2003 outcome data that were available at each grade level (either Literacy Benchmark scores or SAT9 scores)
and (c) the number of outcome variables. School composition and program effects. For elementary schools, information was available regarding specific literacy programs being implemented in the schools. For 5 grade, a two level hierarchical linear model (HLM) was performed to examine relationships between Page 4 of 47Ml school composition factors (aggregate poverty, mean achievement at pretest, and percentage African American enrollment), school literacy programs, and student achievement. Longitudinal cohort performance on Benchmark Examinations. For fourth and eighth grades, three consecutive years of Literacy Benchmark Performance Level data were available. The percentage of African American students scoring in Below Basic, Basic, Proficient, and Advanced categories was computed for each year from 2001 to 2003 to provide a basis for examining overall trends in performance across time. Elementary Level Results Below, conclusions and results based on analyses performed on fourth and fifth grade data are presented by research question. After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement. did African American students exhibit similar levels of academic achievement as other students? I African American students had substantially lower absolute performance than did other students. The academic gains on literacy tests were lower for African American students than for other students. What proportion of the variance in 2003 literacy achievement was uniquely attributable to whether students were African American? Although there was a significant relationship between African American status and student achievement, the proportion of variance in academic performance attributable to African American status was very low4.6% for fourth grade, and 5.7% for fifth grade. What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? The performance of African American fourth grade students on the Benchmark Literacy examination improved dramatically between 2001 and 2003, with nearly half perfoiming at a Below Basic level in 2001, compared to only one-fifth in 2003. Was there a relationship between the literacy program implemented at the school, school composition variables, and the achievement of African American students ? No significant relationship was observed between the type of literacy program implemented and the achievement of African American students. The percentage of African American students enrolled in a school did not predict overall achievement or the achievement of African American students. School poverty, as measured by the percentage of students eligible for free or reduced-price lunch, had a negative effect on the achievement gains of students. Page 5 of 47Id Secondary Level Results ri The secondary conclusions and results by research question are as follows: ri After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement, did African American students exhibit similar levels of academic achievement as other students? ri The absolute level of achievement of African American students was substantially lower than that of other students of similar gender and free lunch eligibility status. Generally, the gains in academic achievement of African American students were similar to those of other students in grades 6 to 11. ri What proportion of the variance in 2003 literacy achievement was uniquely attributable to whether students were African American? ri When data on prior achievement were available, the proportion of variance in 2003 achievement attributable to African American status was quite low at the secondary level, ranging from 0% to 5%. ri What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? ri ri The performance of African American 8* graders on the Literacy Benchmark exam improved substantially and consistently between 2001 and 2003, with the percentage scoring Below Basic dropping from 48.5% to 34.5%, and the percentage scoring Proficient increasing from 14.9% to 23.9%. ri Were dijferences in achievement gains between African American students and other students similar at different grade levels and for different test instruments? ri ri ni From the 5* to the 8* grade cohort, the achievement gains of African American students became more similar to those of other students, and for some subgroups surpassed those of other students in 8* grade. The gap in achievement gains was greater on the SAT9 than on the Literacy Benchmark examination. Presumably, the Literacy Benchmark examination is more closely aligned to the mandated curriculum than is the SAT9, which is intentionally designed to be insensitive to curricular differences. Summary and Conclusions ri The Little Rock School District is commended for the emphasis given to increasing literacy in all schools in the district and not just those in the lowest performing strata. The state and local initiatives in literacy for early learners, including ELLA and Effective Literacy, are well grounded in current research of best practice. In addition, the Reading Recovery and Success for All models are among the best researched and proven programs in the nation for lower-performing students. Impressions from interviews and survey data, however, are that Page 6 of 47ri ri ri ri these programs are often perceived as separate and discrete entities instead of integral to a district comprehensive literacy program. Teachers describe themselves or their schools as doing ELLA or Success for All and only the certified tutors doing Reading Recovery. Middle and high school teachers comments seemed to indicate that they also did not perceive themselves as being involved in a literacy plan beyond the traditional roles they have had as English teachers. Thus, it is recommended that the districts plan, or big picture of literacy, be developed and presented to teachers in a format that communicates how each program, school, grade level, and teacher contributes to and accomplishes literacy goals. ri ri ri The professional development in basic literacy has been well received and represents an enormous accomplishment for the district. Management and delivery of the professional development, however, needs to be made more consistent and available to teachers. The primary concerns voiced by teachers were scheduling problems, inadequate space and availability of training for the numbers of teachers needing to be trained, retraining for teachers who change grade levels, training for new teachers, and obtaining qualified substitutes for teachers while they attend training. The impressions of professional development communicated by upper grade teachers and those who were not implementing special programs were that professional development has been minimal, targeted to cunent hot topics (e.g., portfolios), and inconsistent in quality. ri ri ri ri Teachers perceptions of the impact of literacy programs were extremely mixed. Writing and composition were literacy areas that all teachers agreed had been emphasized and improved in their schools and classrooms as a consequence of literacy initiatives in the district. Some of the positive comments relative to ELLA had more to do with the materials teachers had received as a part of the training than the new ideas they had been provided. During the focus groups, the most impressive level of agreement that teachers voiced was by teachers who had Reading Recovery teachers as Literacy Coaches in their schools. The concept of highly trained Literacy Coaches being placed in and available to all elementary schools is a national issue presently and it is recommended that the district find ways to support this concept not only in elementary schools with large at-risk populations, but in all schools in the district. ri ri ri Regarding student achievement, substantial differences exist in the overall achievement of African American students and other students in LRSD. However, the differences in academic gains tend to be smaller at higher grade levels. The three-year trend in Literacy Benchmark scores shows substantial, sustained improvement in the academic achievement of African American students between 2001 and 2003. African American status of the student, as well as the percentage of African American students enrolled in a school, explain only small amounts of variance in student outcomes compared to prior achievement. None of the curricular programs examined in this study had a significant impact on the achievement of 5 grade students however, these programs are most likely to have an impact at the primary grade levels. Page 7 of 47Little Rock School District Literacy Program Evaluation Report II ri The present report provides the results from a study of the different literacy programs used in the Little Rock School District (LRSD). After expending substantial effort and resources to improve the reading ability of students in the district, administrators at LRSD wanted to examine the effectiveness of the different programs used within the district for literacy instruction. To facilitate this examination, the Center for Research in Educational Policy (CREP) at The University of Memphis was employed to provide an independent, third party evaluation. The evaluation methodology and data analysis were oriented around the following research questions: ri 1. 2. ri 3. ri What are teacher perceptions of and reactions to the different literacy programs? After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement, did African American students exhibit similar levels of academic achievement as other students? What proportion of the variance in 2003 literacy achievement was uniquely attributable to whether students were African American? ri ri 4. What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? 5. Were differences in achievement gains between African American students and other students similar at different grade levels and for different test instruments? 6. Was there a relationship between the literacy program implemented at the elementary schools, school composition variables (i.e., school poverty and percentage of African American students enrolled), and the achievement of African American students? ri Method ri The evaluation design was based on both quantitative student achievement data as well as qualitative data from K-12 faculty members who are responsible for literacy instruction. The data were collected during the 2002-2003 school year., ri Instrumentation ri ri ri Literacy Program Teacher Questionnaire. A questionnaire was developed by CREP researchers to gather faculty members perceptions of their schools literacy program. The specific areas addressed included understanding of the program, professional development, resources, pedagogical change, support, and outcomes associated with program implementation. The questionnaire was comprised of two sections. The first section included 21 items to which teachers responded using a likert-type scale ranging from strong disagreement (1) to strong agreement (5). The second section contained six items that gathered demographic information about the respondents. Teacher Focus Group. Structured interview protocols were used to conduct group interviews with randomly selected teachers. Interview guides were developed by CREP researchers to ensure consistency of questions during the different interviews. The areas addressed in the guide were as follows: general information, professional development. Page 8 of 47 i Hri classroom level changes, results/outcomes, and parent and community support. Appendix A contains a copy of the interview guide. Student Achievement Data. A variety of achievement data sources were analyzed because of the different types of literacy assessments used at each grade level. The achievement assessments included in the analysis were from all students enrolled in grades 3 to 11 in the district during the 2002-2003 school year. The three primary data sources included the Literacy Benchmark scale score, SAT-9 Reading subscale score, and the SAT-9 Language subscale score. Procedure To establish the focus groups, teachers were randomly selected from all schools in the district and then grouped for the interviews to represent the districts various grade divisions and established comprising 38 teachers (see table 1). literacy programs. A total of seven groups were Table 1 ri Number of Focus Group Participants ri ri ri ri ri ri ri Literacy Program Number of Participants ELLA____________________________ Effective Literacy___________________ Harcourt Brace_____________________ Reading Recovery__________________ Success For All____________________ Middle School Language Arts (English) High School English Total _______________ 4 6 4 6 6 7 5 38 Two CREP researchers conducted the seven focus groups at the Neighborhood Resource Center using the structured interview guides. Each focus group was approximately one-hour in duration. Teachers signed a permission form to be interviewed and were given assurance that their comments would be confidential and anonymous. The sessions were audiotape-recorded and supplemented with the researchers hand-written notes. The Literacy Program Teacher Questionnaires were printed and shipped to LRSD personnel. The district staff members disseminated the questionnaires to the individual schools along with instructions for completing and returning the forms to the district. After the district staff received the completed forms, they were sent to CREP for analysis. Similarly, district personnel assembled the student achievement data into an electronic format. The data files were then sent to CREP researchers for analysis. M Page 9 of 47r I Results Teacher Focus Group To analyze the focus groups, the tape-recorded sessions and researcher notes were summarized and recoded into bullet statements. The individual focus group summaries were then analyzed thematically and synthesized by grade level and program. The following is a synthesis of findings from the focus groups (see Table 2 for an overview). The literacy initiative in LRSD is a combination of programs supported by national, state, and district funds to provide a balanced, comprehensive approach to reading and writing instruction. Professional development is organized, supported, and provided by five district personnel. The following programs are currently used: Early Literacy Learning in Arkansas (ELLA), (Grades K-2). ELLA is a three-year staff development process designed for grades K-2. The training began in the LRSD in 1999 and consists of 12 days (6 hours per day) of staff development covering a range of topics and instructional techniques that support emergent learners. Comments from the four K-2 teachers who participated in this group were integrated with comments regarding ELLA training from three additional K-2 teachers (total n = 7) who had been trained in ELLA but were interviewed as part of the Effective Literacy group. Effective Literacy (Grades 3-5). Effective Literacy is a two-year staff development process designed for grades 3-5. It began in the district as a state initiative in 1999. The components of the program are organized into 8 full day sessions (6 hours) and include a range of topics and instructional techniques to support reading and writing development. A total of six teachers participated in the focus group although only three were teachers were currently in grades 3-5. Signatures Reading Series (1999), Harcourt Brace Publishers. This is the basal reading textbook series adopted by the district in 2000. Professional training related to the use of the reading series was provided during the first year of adoption. Since that time, no other staff development has occuned relative to the reading series. Four teachers participated in this focus group from grades 1, 2,4, and 5. Reading Recovery (Grade 1). Reading Recovery is an extensively researched and widely used early intervention/prevention program that provides at-risk first graders with one-to-one tutoring from specially trained certified teachers. The yearlong professional training of teachers is considered to be one of the greatest assets of the program. In the six schools that were represented in the focus group, the Reading Recovery teacher served as their schools Literacy Coach. Success for All (SFA), (Grades K-5). SFA is a comprehensive school reform model that uses a reading curriculum founded on research-based instructional practices, cooperative learning, and one-to-one tutoring for at-risk learners. The program was introduced in the district in 1997-98 and presently exists in six schools in the district. According to the six teachers who Page 10 of 47 participated in the interview, three of the current SFA programs will continue next year while three others are being discontinued or reconfigured to exist only in the upper grades. Middle Schools (Grades 6-9), (English/Language Arts teachers). Middle schools were reformed in the district in 1999 and encompass grades 6-9. Most of the schools now have a double-block English class (2 hours and 45 minutes) for all students. All grades participate in Reading and Writing Workshop concepts. Professional development has been largely shaped by current trends and research in literacy. Seven English teachers participated in the interview. High Schools (Grades 10-12), (English teachers). According to the five teachers participating in this focus group, all of the high schools are on block schedules (learning periods that exceed the traditional 45 to 55 minutes) with various configurations of time for English classes to accommodate different levels of learners. As in the middle schools, teacher professional development has been implemented based on national research findings, trends, and issues in literacy development. Most ejfective elements of the literacy programs Although teachers have had varying amounts of training in ELLA (from one day to as many as eight days), they were generally positive about the professional development and the materials they had received as part of the training. They indicated that one of the most effective components of ELLA training was the Developmental Reading Assessment that enabled teachers to effectively measure student progress. When implementation of the assessment first began, teachers said they had to share the testing materials and that this was very inconvenient and stressful. However, every teacher was reported to have his/her own testing materials this year. Other positive comments related to the emergent literacy emphasis for kindergarten students and the resulting improvements, especially in childrens writing. Teachers also commented positively about currently having books to use to begin teaching students to read in kindergarten (a relatively new philosophical change in reading instruction in kindergarten). I According to the teachers, the most positive aspects of Effective Literacy training were the emphases on delivery of instruction in variable groupings (whole class and small groups) and the changes in school schedules that provide two and one-half hours of uninterrupted time for reading first thing in the morning. The increased time for literacy has also given teachers more time for writing. Teachers reported the most effective aspects of the Harcourt Brace Signatures reading series to be the organization of the book around thematic units, consistent lesson structures for each day of the week, and the vocabulary emphasis in the series. In addition, the series provides English as a Second Language (ESL) materials, reinforcement of skills with workbooks, and suggested trade books to support the basal materials. Teachers from schools implementing the Reading Recovery program indicated the most effective elements included the one-to-one tutoring of students, extra time (30 minutes) for instruction of at-risk learners, and the structure of the lessons. One teacher claimed that the program works even for the hyperactive students. The most common point of agreement Page 11 of 47IM H II regarding effectiveness of this program, however, was that the training Reading Recovery teachers received to become certified was excellent and that schools were greatly benefiting from the leadership these teachers were providing to the schools in their roles as Literacy Coaches. Elements of the Success for All program that teachers perceived as most effective included the regrouping of students for reading instruction into more narrow ranges of ability levels and the routines of the instruction. One teacher referred to having an aide to help with instruction as one of the most effective elements of SEA
When I have an aide to help teach, the kids really do much better and discipline problems go down significantly. Middle school Language Arts/English teachers reported that students are writing more as a consequence of teacher professional development and emphasis on reading/writing workshop approaches to instruction. Other effective elements of the literacy curriculum included its H reorganization into quarters that were more manageable for instruction, and implementation of M writing portfolios that move through the grades with students. Teachers indicated that portfolios were helping to encourage high expectations for all students and the writing prompts and rubrics were especially helpful to teachers in structuring and evaluating student writing. High school English teachers were more reserved in attributing effectiveness to any H particular elements of the literacy programs that are operational in their schools. They explained, We dont know (whats been most effective) until we see the test scores. Two strategies. paired reading and interactive journals, are relatively recent approaches teachers referred to as II working well. II II II II III II Least effective elements of the literacy programs One primary concern about ELLA was the inconsistency of implementation across schools. Teachers explained that judging the effectiveness of ELLA is complicated by high rates of student mobility, especially when students move from one school with strong ELLA implementation to another school where there is less emphasis on ELLA concepts and strategies. Teachers also mentioned that although they liked the new approaches to student assessment of progress, it was difficult to individually test a class of 25 students, and there had been a change in the cap or cutoff for first grade to level 8 even though some students are at level 25 or 30. Teachers were not clear about why the level was changed and what purpose it served in helping teachers and students move to appropriate levels of instruction. Effective Literacy teachers indicated that they did not perceive any ineffective or undesirable elements of the literacy initiatives in the district. In the teachers words: We like it and We want all the training we can get. Teachers concerns about the Harcourt Brace reading series centered around the amount and depth of comprehension work required by the books and workbooks, the number and simplicity levels of the stories, and vocabulary that was too difficult for some grades and too easy for others. Another concern was that the texts were not good matches for the SAT-9 achievement test. Page 12 of 47d d d d d d d d d d d II II II Reading Recovery programs have specified guidelines regarding acceptance of students into the program and discontinuance of students after 20 weeks of instruction. Teachers indicated concerns that these policies interrupted student success and were not always in the best interests of students and their continuing progress. One teacher said, Students dont continue to make the same kind of progress in literacy groups that they have made in individual tutoring sessions. Others commented that in some cases, students have been discontinued because of behavior and not reading ability and that some schools are not accepting the lowest students or students who speak English as a Second Language (ESL). Success for All teachers most common concerns related to the lack of flexibility in the program. One teacher said that the timing of the individual activities presented in a days lesson did not include enough time to monitor for student understanding. Other related concerns were that there was no room in the schedules for review and that the program did not allow teachers to move students down in ability level even if they needed to be moved to more appropriate reading levels. Thus, when students at lower levels got behind a grade level or more, it was difficult to find an appropriate group for them. Because of a highly transient student population, teachers find that they constantly have to train the students to the routines and in cooperative learning. Teachers reported that the least effective elements in middle school literacy initiatives were the gaps in the training. Because the training started several years ago, new teachers have not received much. In addition, teachers indicated that there needed to be reading classes in middle school for all students and not just for the lowest performers. Additional concerns about literacy were that grammar was not being emphasized any longer and that increased classroom emphasis on writing did not match assessment formats. One teacher explained. When there is so much emphasis on writing, test scores go down because students are tested in multiple choice formats. One concern that teachers emphasized was the lack of consistency in literacy instruction and programs in middle schools. One teacher said, All schools are doing something different. Another teacher suggested that there was need to rewrite literacy programs and have guidelines that make programs consistent for all teachers and students. High school English teachers indicated that getting everybody (teachers) to participate and support the literacy initiatives across subject areas was difficult. They heard complaints from teachers about the increase in the amount of work for them when they gave even tiny writing assignments to students. In addition, it was hard for teachers to check to ensure that students read 25 books across multiple classes. II d Teacher Support for Literacy Initiatives Overall, teachers reported that support for ELLA, and Effective Literacy initiatives is high primarily because the concepts and strategies fit well with what they were already doing or moving toward anyway. One teacher said, It comes naturally and is working well at our school. Another indicated that much of ELLA was what she was already doing and by combining and adding the ELLA principles to it, she had created a true balanced literacy environment. Page 13 of 47 h d d d d Teachers were more equivocal about their support of the Harcourt Brace reading textbook series. According to one teacher, Its there. We use it. Another explained, If we find the books lacking in some way, we search out other materials, such as novels, and supplement the reading. Reading Recovery teachers indicated that teacher support for the program varies from school to school. Some reported that their schools were more positive than others
however, there were no reports of any strong dislike of the program. The widest range of variability in teacher support for a district literacy program was reported by Success for All teachers who indicated that in some schools there was no support, in others the support was low and in some most of the teachers support the program. One teachers comment reflected how she as an individual had also ranged in her opinion of the program as she experienced it over time. She said, I hated it at first, but I like it a lot more now that I am working with the higher level kids. Middle school teachers had no comments about general support for the literacy programs at their schools. High school teachers, however, indicated that their colleagues had generally positive attitudes about literacy efforts in their schools. They indicated that they, as literacy experts, could possibly contribute more to other teachers confidence and support of literacy if the English teachers had more time to work with their colleagues in other subject areas and train them to use rubrics for grading writing assignments. d d II II II d d District Support for Literacy Initiatives The most common concern about ELLA relative to the question of district support was the lack of consistency for supporting teachers to attend training. The teachers explained that the district expects teachers to commit to eight days of training and assures them that substitutes will be supplied to relieve them of teaching duties so that they can attend the training. However, in the words of one teacher, They wont let you call and arrange for your own subs even when you know someone who is available. Then when the day comes, they don t send subs. So you can t go, or the kids get supervised by aides. Another issue of concern to teachers was not receiving supplies (e.g., phonics charts, magnetic letters), in the most recent ELLA trainings as teachers had received in earlier sessions. Teachers also expressed the need for more physical support for implementing ELLA. Specifically, they wanted to have teaching assistants for early grades who were trained in EJ.IA concepts and strategies to provide daily support during reading instruction. Effective literacy teachers also expressed concerns about having qualified substitutes in their classrooms when they are away from their classes for training. They indicated that this problem may be centered more at the individual school administrative level, however, than at the district level. Teachers responses relative to the Harcourt Brace reading texts indicated that all the materials needed to support literacy instruction were made available to teachers through the district. One teacher reported that when she was new in a school, however, she had to hunt for the materials for herself and that she believed that other new teachers often had this same Page 14 of 47ri ri experience. She expressed the opinion that more should be done to help new teachers become aware of the resources and textbooks that were available and where they could be found. ri ri ri ri ri ri ri Reading Recovery is largely supported through Title I, a federal source of dollars that teachers did not attribute as support from the district. Teachers comments were that when schools dont qualify for federal funds, such as Title I, then the school does not receive the same level of support that other schools do. Success for All teachers indicated that the district has not supported the number of refresher trainings recommended by the model developers due to economic conditions. Teachers also reported that three schools were forced to drop the SFA program due to Reading First requirements. Middle school teachers made no comments about district support, whereas, high school teachers made suggestions for ways in which the district could provide greater support. Specifically, teachers expressed the need for literacy coaches to not only help the English teachers do a better job, but also to help train teachers in other subject areas to use rubrics, writing prompts, and consistent scoring techniques to improve students writing. Teachers also indicated the need to use district time set aside for inservice training to plan a more comprehensive literacy approach. ri ri ri ri ri ri Professional Development Teachers reports of the amount of ELLA professional development they had received relative to the 12 required days of training was highly variable. Some teachers reported receiving as little as 3 days of training while others who had taught the same number of years (or more) had completed 12. Teachers who were new to the district indicated they are expected to go one day a month for training. According to teachers, there were also differences among schools regarding monies that paid for training (Title I in some schools). Some teachers were also offered stipends to attend training, whereas others were not. One teacher indicated that some training sessions had not been accessible because the number of participants was limited. Although teachers were generally positive about the quality of the training, some were not. One teacher said, I got more from my college class than I did from ELLA training, and another suggested that If training were better, people would buy into it more. Teachers offered several specific suggestions for ways the professional development could be improved and made more effective. Some of these ideas included selecting successful inservice teachers to do the training
using videos that were more realistic in terms of numbers of students in a classroom
making the training more hands-on and less video and lecture
and offering the training at better times, even during the summer. ri Although professional development in Effective Literacy began in the district at the time that ELLA began (1999), the most sessions any one teacher in the focus group had completed was four out of the eight. According to teachers, the district presently expects same teachers to complete all eight days by the end of their first year of teaching, but as with ELLA training, sessions have been limited by numbers of participants, and thus teachers have not always been able to attend them. The teachers were generally positive about the quality of the Page 15 of 47 *ri ri ri ri ri training. One teacher commented that, If we hear just one thing to help change up the day and make things better for the students, thats good. Teachers also liked scheduling the training during school hours and immediately after school. Teachers reported that the professional development provided for using the Harcourt Brace reading series was offered by company representatives at the time of adoption and that there had not been any ongoing training since that time even for new teachers or for teachers who change grades. One teacher commented that the training that was done started halfway into the year and that put me behind in using it [reading textbook]. Another concern for teachers was that the trade books that are referenced in the reading series, and modeled and suggested in the training, were not available in the school libraries, or there weren t enough of them for the students to use. ri ri The most positive reports regarding professional development were made by teachers in Reading Recovery schools. The specially trained and certified Reading Recovery teacher at each school also serves as the schools Literacy Coach. Each has received additional training to be Literacy Coaches for whole schools beyond their specific training in the Reading Recovery program. In the role of coaches, the teachers have brought their training back to the school and trained others. Teachers were positive in their response to this model of professional development indicating that they [the coaches] help, model, and are non-threatening for teachers. ri M M Success for All teachers have had access to two sets of professional development activities. The training specific to the implementation of the SFA model was originally offered to schools when the model was first implemented in the district. Since that time follow-up trainings have been limited by funding, and as a result, have been combined and shared with other schools in the area that were implementing SFA. All teachers in SFA schools have also had access to ELLA training. Teacher descriptions and judgments of the quality of SFA training varied. Two teachers descriptions of the SFA training were highly positive. In their words: They were awesome presentations. Other responses were more critical with the primary concern being that they were redundant and provided too much information. ni II II II W Middle school English teachers response to professional development was also mixed. Some of the teachers indicated they had benefited from the trainings, while others described them as a waste of the districts money. Another concern was that some teachers need specific types of training (e.g., portfolios) while others dont. In general, all of the teachers indicated that they needed more training than they have received, especially in such topics as conferencing,,, hands-on application of teaching strategies, and how to do writers workshop like in ELLA. High school teachers comments regarding professional development indicated that they have no sense of a district coordinated plan for teachers. One teacher said, We have no overall picture of what we are doingjust pieces. They said also that they have lots of inservices that have been adequate, but that their time might be better spent if they worked with their faculties in their individual schools to brainstorm and plan their own comprehensive literacy programs. Page 16 of 47 kri ri Classroom Level Changes ri ri ri ELLA teachers described two major changes in kindergarten. Teachers are returning to doing centers as they had in the past, and they are emphasizing and involving students in writing. They also noted that there were increases in pressure put on five year olds to perform academically. Teachers at other grades had noted less change in classrooms as a consequence of ELLA. Comments such as, Teachers are falling back to the regular ways
It isnt anything different than what I have always done
We arent big on ELLA at our school
and Sometimes we use pieces of ELLA indicated that changes in classrooms that could be attributed to ELLA would be difficult to observe. ri t Changes in Effective Literacy classrooms primarily involved more emphasis on writing activities. Teachers also indicated they had greater variety of reading materials available, including more technology-based programs such as Accelerated Reader. ri ri Teachers using Harcourt Brace basal textbooks as their primary vehicle for delivery of instruction also indicated that teachers are falling back into traditional methods even after we have trained on ELLA. One change that was noted, however, was that ELLA emphasizes placing students at their instructional levels and doing extra reading programs to meet their needs. The reading materials provide teachers with additional materials and suggestions for trade books that enable them to differentiate instruction. One teacher noted, however, that the text materials do not do enough to address the needs of lower level kids. ri ri Teachers in Reading Recovery schools provided the most positive and specific descriptions of how classrooms had changed as a consequence of the literacy program. Teachers reported that students were achieving higher, students were learning strategies to help themselves become more strategic readers, and writing was being emphasized more and had subsequently improved. One teacher commented that these changes had resulted from a combination of things
Results have been convincing
materials are good
the literacy coach has been helpful
Reading Recovery fills in gaps from the classroom. ri ri Success for All teachers found it difficult to describe changes as a consequence of the program because as two of them said: SEA is all I know. One teacher indicated that she does more cooperative learning than in the past. Others comments such as, I do my own thing when the door closes and I follow the program but I modify it seemed to imply that in some classrooms instruction is drifting from SEA practices to more traditional. ri ri Middle school teachers made no comments about notable changes to classroom instruction beyond the presence of special education teachers who now come into classrooms with students who have individualized instructional plans to assist them in the classrooms rather than pulling students out for separate classes in literacy. ri ri High school teachers said there had been lots of changes in classroom instruction recently. However, the only major change they discussed in-depth was cooperative learning. They indicated that although teachers were trying to use cooperative learning more, they are not completely satisfied with how it is working. I Page 17 of 47n n Impact on Students RUA teachers indicated that achievement has gone up but qualified their judgments by I saying
.. .but we cant tease out if the changes are due to ELLA or not. They also reported that the use of leveled books had given students more confidence to try things, kindergarten students were being provided with stronger foundations for transitioning into first grade reading . _J '1^ V 1 1 f -I i"! T1 rr I and writing expectations, and that students were really involved in writing. was too early to tell how the program had impacted I V Effective Literacy teachers said that it students. One aspect of the emphasis on literacy that they noted had impacted students was the wide range of resources and materials that they now had available to them. Another observation teachers reported was their judgment that students that were new to the district were starting at lower achievement levels than students who had consistently been enrolled in the district. at According to teachers using the Harcourt Brace reading series, students have enjoyed the stories in the texts and have learned better cooperation skills through partner reading strategies. Teachers also reported that students have learned what good readers are and that this is helping them to be more successful. q I I I q q( Teachers in the Reading Recovery schools indicated that the impact of the program on students is largely determined by the follow through and support of the regular classroom teacher. In one teachers words: Without strong communication between the Reading Recovery teacher and the classroom teacher, it wont happen for students. Although teachers indicated that many students have been helped with Reading Recovery, they also reflected again on their concern that many students regress once they are dropped from the program and do not continue to improve. Two Success for All teachers said that achievement scores of students had gone up, while another reported this was not the case at her school. Teachers reported that students were getting bored with the structure and routines of the program at this time of the year (late spring). Middle Schools teachers indicated that the Accelerated Reader program has been useful in increasing the motivation of students to read. They also reported that the assessments and leveled readers were creating more success and confidence for students and that students were writing more. According to the teachers, behavior and classroom control of students (management) are still very big issues for achievement of middle school students. q was too early to tell what impact their teaching High School teachers indicated that it had on students this year as they are judged by whether students can read and write (as indicated q on their achievement tests). Impact on Teachers 3ie One T.f.I A teacher commented that because there is stuff going on all the time, teachers constantly busy, but not necessarily in a good way. Others agreed that the pressures and Page 18 of 47 ri time investments have had a negative impact on teacher stamina. Another teacher explained that when schools work together and all are doing ELLA, that it is not because of ELLA necessarily, but because they are working together as a team around a central focus. A final comment was that ELLA had provided teachers with strategies and ideas that helped all teachers in a school provide more consistency across a variety of student learning levels and across schools as student mobility continues to increase. fl Ejfective Literacy teachers were reported to be excited about it. They like having the additional materials and seem to be sharing ideas among themselves more than in the past. fl Harcourt Brace Reading Series teachers indicated that they have observed teachers in their schools discussing the stories in the texts with each other. They indicated that teachers were sharing ideas and expectations they have for students more than they have in the past. fl fl Reading Recovery classroom teachers reported borrowing and successfully using many of the strategies the tutors use that are particular to the Reading Recovery program. The modeling that the Literacy Coaches have done in the Reading Recovery schools was reported to be well received and useful to teachers. fl fl Success for All teachers responses were ambivalent about the impact of the program on teachers. Three teachers said that teachers at her school hated the program and that this commonality brought them together. Another teacher claimed that teachers at her school loved the program and that it brought them together also. Another teacher reported middle ground response to the program at her school by saying, They dont hate the program, but they arent particularly enthusiastic about it. n fl Middle School teachers gave no response to the question of impact on teachers while High School teachers reported that as a consequence of the emphasis on literacy in the district, they are sharing a lot more strategies. Although some of the teachers expected their schools to be on warning status, they indicated that teachers were staying focused on trying to identify students who are in most need and work at ways to bring them to proficiency level. H fl fl fl Page 19 of 47Table 2 Themes from Teacher Focus Group Theme ELLA Most Effective Element Professional Development Materials Effective Literacy______ Instruction for different student groupings (whole class/small group) Harcourt Brace______ Thematic units Reading Recovery 1 to 1 tutoring Success For AU Regrouping of students Middle School High School Developmental Reading Assessment 2.5 hours of Reading instruction Emergent literacy emphasis More time for writing Consistent lesson structures Vocabulary ESL materials Skill reinforcement Extra instruction time for at-risk students Reading/writing workshop approach Paired Reading Teacher Aides Student writing portfolios/rubrics Interactive journals Training to become RR certified Reorganization of curriculum into quarters Least Effective Element Teacher Support for Program District Support for Program Inconsistency of implementation across schools Individually testing all students (time) First grade cutoff/ level Support is high due to fit with existing practices Lack of support to attend training Lack of materials and teacher aides (None noted) Support is high due to fit with existing practices Lack of support to attend training Amount of comprehension work Story levels No link with SAT-9 Not high or low
it is one resource Materials are available
new teachers need assistance Guidelines (timeframe) for acceptance to and discontinuance from program (20 week limit) Varies across schools (no strong dislike) Program primarily supported by Federal funds Lack of flexibility Gaps in training Inability to move students to lower reading level Frequent retraining on routines Varies across schools (strong dislike to strong support) Lack of support to attend training Lack of support to continue SFA Reading classes for low perf students only Lack of consistency between schools (No comments) (No comments) Lack of support from non-literacy teachers Ensuring students read 25 books across multiple classes Generally positive
need more time to work with peers Need Literacy Coaches Need inservice time for planning Page 20 of 47Theme ELLA Professional Development Classroom Level Changes Impact on Students Impact on Teachers Number of days received varies across schools Stipend is inconsistent Delivery and time could be improved More work centers and writing in Kindergarten Some regression to traditional methods Increased confidence Better prepared to transition to P grade Strain on time New strategies/ideas (consistency within and across schools) Effective Literacy Number of days received varies across schools Good quality Emphasis on writing More materials available More resources and materials available More materials Sharing of ideas Harcourt Brace No ongoing training Trade books not widely available Use of materials and trade books to differentiate instruction Some regression to traditional methods Learned cooperation skills and what agood reader is Sharing expectations and ideas Reading Recovery Training to become certified/ Literacy Coach is beneficial Teacher-trainer model is positive Increased student achievement and writing ability Students becoming strategic readers Literacy Coach Gains made during program, but not always continued after program New strategies available Literacy Coach well received Success For All Ongoing training is limited Varying quality of training More cooperative learning Some regression to traditional methods Mixed achievement results Boredom with routines Unity through like or dislike of program Middle School Ongoing training is limited Varying quality and applicability of training Special Ed teachers work with students in classrooms Student behavior still problematic AR program increases motivation Success and confidence from assessments/ leveled readers (No response) High School No overall training plan
just series of unrelated inservices Need training relevant to own school More cooperative learning (varying quality) Too early to tell Sharing of ideas Focus on bringing students to proficiency Page 21 of 47 M M M Literacy Program Teacher Questionnaire (LPTQ) The questionnaires were analyzed descriptively using the program groupings provided by the district. The groups were as follows
(a) Balanced Literacy, (b) Direct Instruction, (c) Success For All, (d) Middle Schools, and (e) High Schools. See table 3 for a comparative overview. a Professional Development. Most teachers in the different programs agreed that they had _ thorough understanding of their schools literacy program and that they have received adequate professional development for program implementation. High school teachers tended to be less positive than other groups with regard to receiving effective professional development and support provided by external partners, although more than one-half of the high school teachers agreed with these items (60.9% and 56.5%, respectively). 1 II II n M M n Resources. Most respondents agreed that they had the materials needed to implement their literacy program, with the exception of high school teachers (39.1% agreement). A similar pattern was seen with regard to having sufficient faculty and staff for program implementation, where high school teachers agreed less often than teachers in other literacy program groups. In comparison with other teachers, faculty members from schools implementing Success For All and teachers at the High School level had lower levels of agreement that technological resources had become more available as a result of their literacy program. The majority of teachers (80.9%) implementing Direct Instruction agreed that they were given sufficient planning time to implement their literacy program. Other respondents were less positive with regard to this item, with two-thirds of the middle school respondents agreeing that they were given sufficient planning time, and less than one-half of Balanced Literacy (47.2%) and High School English (32.6%) teachers agreeing. Impact/Outcomes. Most respondents agreed that their literacy program had changed classroom learning activities and had a positive impact on students. High school teachers, however, were less likely to agree with these items. Teachers were also positive regarding the impact that their literacy program had on student enthusiasm for learning and encouraging students to have higher standards for their own work. Middle and High school teachers, however, had lower levels of agreement with these items in comparison with elementary teachers. Lower levels of agreement (less than 50%) were seen from all respondent groups regarding increased parental and community involvement as a result of the literacy programs. Teachers were also less likely to agree that they are now more involved in decision making at their schools since implementing their literacy program. Encouragingly, most teachers agreed that they are supportive of the literacy program at their school (ranging from 65.2% agreement by High School teachers to 89.4% agreement from Direct Instruction teachers). Page 22 of 47H M Table 3 Percentage of Teacher Agreement (Agree or Strongly Agree) with LPTQ Items by Program LPTQ Item H d I I II II II II II d I have a thorough understanding of this school's literacy program.________________________________________ I have received adequate initial and ongoing professional development/ training for implementation of my school's literacy program._________________ Professional development provided by external trainers, model developers, and/or designers has been valuable. Guidance and support provided by our school's external facilitator, support team, or other resource personnel have helped our school implement its literacy program. Teachers are given sufficient planning time to implement our literacy program.____________________ Materials (books and other resources) needed to implement our literacy program are readily available. Our school has sufficient faculty and staff to fully implement its literacy program.____________________ Because of our program, technological resources have become more available.___________________________ Our literacy program has changed classroom-learning activities a great deal.__________________________ Overall student achievement has been positively impacted by our literacy program.__________________ Children in this school are more enthusiastic about learning because of our literacy program.____________ Because of our literacy program, parents are more involved in the educational program of this school. Community support for this school has increased since our literacy program has been implemented._________ Students have higher standards for their own work because of our school's literacy program.____________ Teachers are more involved in decision making at this school than they were before we implemented our literacy program._______________________________ Our literacy program adequately addresses the requirements of children with special needs.__________ Because of our literacy program, teachers in this school spend more time working together to develop curriculum and plan instruction.___________________ Teachers in this school are generally supportive of our literacy program._______________________________ The elements of our literacy program are effectively integrated to help us meet school improvement goals. This school has a plan for evaluating all components of our literacy program.____________________________ Achievement for African American students has been BL N = 551 84.9 DI N = 47 91.5 SFA N= 115 85.2 MS N = 88 84.1 HS N = 46 80.4 81.3 87.2 79.1 79.5 71.7 78.0 74.0 47.2 75.0 68.4 55.4 72.1 71.9 60.8 33.6 27.0 55.2 47.5 61.2 60.8 80.9 80.6 70.2 89.4 89.4 80.9 91.5 80.9 68.1 85.1 78.7 63.8 46.8 42.6 66.0 53.2 72.3 63.8 89.4 85.1 89.4 80.0 85.2 60.9 85.2 59.1 82.6 65.2 41.7 77.4 72.2 59.1 40.0 35.7 57.4 41.7 56.5 55.7 72.2 84.3 81.7 72.7 67.0 60.2 67.0 62.5 73.9 67.0 45.5 20.5 25.0 47.7 42.0 51.1 51.1 73.9 70.5 60.2 56.5 32.6 39.1 32.6 19.6 28.3 45.7 13.0 6.5 8.7 26.1 17.4 26.1 26.1 65.2 41.3 28.3 positively impacted by our literacy program. i 60.3 76.6 71.3 60.2 32.6 BL = Balanced Literacy
DI = Direct Instruction: SFA = Success For All
MS = Middle School
HS - High School Page 23 of 47 II ri Student Achievement Data ri ri The purpose of this portion of the study was to examine the achievement of African American students in reading and language arts in the Little Rock School District. Specifically, the examination sought to determine whether (a) African American students academic gains were similar to those of other students
(b) there were any evident longitudinal trends in the overall achievement or in achievement gains for African American students
and (c) at the elementary school level, there was any relationship between the literacy program implemented at the school and achievement gains of African American students. Subjects I The subjects included all students enrolled in grades 3 to 11 in the Little Rock School District during the 2002-2003 school year for whom 2003 Literacy Benchmark or 2003 SAT-9 scores were available. This included a total of 11,934 students, of whom 23.4% were Caucasian, 68.2% were African-American, and 48.5% were certified as eligible for free or reduced-price lunch. Three measures were used to assess literacy: the Literacy Benchmark scale score, SAT-9 Reading subscale score, and the SAT-9 Language subscale score. Preliminary Data Screening Data Screening: Grades 5, 7, and 10. For grades 5,7, and 10, 2003 SAT-9 scores served as outcome variables, while Literacy Benchmark scores from 2002 served as a covariate for grades 5 and 7. A total of 5,320 student records were comprised of 1,842 fifth grade
1,932 seventh grade
and 1,445 tenth grade records101 records (1.9%) did not include a 2003 grade level indicator. These 101 records were eliminated from subsequent analyses, leaving a total of 5,219 cases. Less than 1% {n = 41) of cases did not include a valid school location code, and were thus eliminated from subsequent analyses, leaving 5,178 for further processing. A score of zero on the SAT-9 subtests indicated that a valid score was not obtained. For fifth grade, 307 students were missing 2002 Literacy Benchmark Scores, 82 scored 0 on the SAT-9 Reading subtest, and 16 scored 0 on the SAT-9 Language subtest. For seventh grade, 325 students were missing benchmark scores, 57 had scores of 0 on the Reading subtest, and 47 had scores of 0 on the Language subtest. Tenth grade students did not have benchmark scores from the preceding year
43 had scores of 0 in Reading, and 54 had scores of 0 in Language. Data screening: Grades 4, 6, 8, and 11. For grades 4, 6, 8, and 11, 2003 Literacy Benchmark scores served as the outcome variable, while 2002 SAT-9 scores served as a covariate for all except grade ILA total of 6,770 student records were comprised of 1,841 fourth grade, 1,863 sixth grade, 1,759 eight grade, and 1,307 eleventh grade records. All records contained a valid grade level indicator. Only four records were deleted due to missing school location indicatorsall of these were 11* grade. All 4* grade records contained a Literacy Benchmark score. For 6* grade, 1,487 records had matching pretest (SAT-9) and posttest (Benchmark) scores
respective numbers for 8* and 11* grades were 1,447 and 985. After eliminating cases with a SAT-9 score of zero, the number of cases retained for analysis was 1,395 for 6* grade, 1,359 for 8* grade, and 946 for 11* grade. Finally, to be included in Page 24 of 47ri ri ri multivariate analyses using Benchmark, SAT-9 Reading, and SAT-9 Language scores, cases were required to have valid scores on all three instruments (or both SAT-9 scores for 10 grade). ri ri The results of data screening for all grades are depicted in Table 4, along with the percentage of children eligible for free or reduced-price lunch, the percentage of African American students, and the mean Literacy Benchmark score for unscreened and screened samples. Across grades, match rates ranged from 72.4% for 11* grade, to 100% for 4' grade. Comparisons on key variables indicate that the screening procedures did not substantially alter the composition of the samplesthe largest observed discrepancy for any demographic variable was a 1.8% difference in the number of African Americans in the 11* grade sample (see Table ri 4). Likewise, differences on the Benchmark test means were small, the largest being a difference of 3.7 for 11* grade, which represents an increase of about one-tenth of a standard deviation unit from prescreen to postscreen. It is worth noting that the percentage of children eligible for free 1 1_______J__________11,. j i ith rimhohlv rliip tn nhildrpn flt ri or reduced-price lunch dramatically declines in lO'** and 11* grades, probably due to children at these grade levels not applying for the program. Table 4 Pretest-posttest Match Rates and Screened Sample Characteristics ri Grade n Match Rate % Free Lunch % African American Mean Benchmark' ri ri ri ri ri Fourth Prescreen Postscreen Fifth Prescreen Postscreen Sixth Prescreen Postscreen Seventh Prescreen Postscreen Eighth Prescreen Postscreen Tenth Prescreen Postscreen Eleventh Prescreen Postscreen 1,841 1,841 1,842 1,464 1,863 1,395 1,932 1,509 1,759 1,359 1,445 1,342 1,307 946 100.0% 79.5% 74.9% 78.2% 77.3% 92.9% 72.4% 63.0 55.8 54.2 60.4 60.6 47.4 45.8 53.0 51.6 26.0 26.0 20.6 20.6 '2002 scores for grades 5,7, and 10
2003 scores for other grades. 68.5 69.2 69.9 67.4 68.2 72.3 71.7 68.1 68.3 64.9 63.8 64.3 62.5 204.8 196.7 197.9 175.6 176.9 172.0 173.4 184.8 187.5 n/a n/a 187.0 190.7 Page 25 of 47Analyses H ri ri ri District-wide achievement effects. The basic analytic model used to gauge district-wide achievement effects was a 2 (free lunch status) X 2 (gender) X 2 (African-American, nonAfrican American) analysis of variance (ANOVA). This basic model was adapted to each grade level to reflect: (a) the availability of achievement data from the prior year, in which case a 2 X 2 X 2 analysis of covariance (ANCOVA) was used
(b) the specific 2003 outcome data that were available at each grade level (either Literacy Benchmark scores or SAT9 scores)
and (c) the number of outcome variableswhere both Reading and Language SAT9 scores were available, multivariate analysis of variance or multivariate analysis of covariance was used. Within each grade level, outcome variables were converted to Z-scores to aid in the interpretation of results and to permit comparisons across grade levels for which different outcome variables were available. Z-scores express the original scale scores in standard deviation units. Partial eta- squared (e^) values were computed for prior achievement, gender, race, and free lunch status. Partial indicates the proportion of variance in the outcome variable that is uniquely attributable to each predictor variable (see Tables 5 and 6). ri Table 5 Average Unadjusted Z-scores by Grade Level, Free Lunch Eligibility, Gender, and Race ri Grade Level Not eligible Female Male Female Free lunch eligible Male ri ri Fourth Non African-American African-American Fifth Non African-American African-American Sixth Non African-American African-American Seventh Non African-American African-American Eighth Non African-American African-American Tenth Non African-American African-American Eleventh Non African-American African-American 0.85 0.32 0.95 0.10 0.78 0.25 1.06 0.17 0.93 0.37 0.90 -0.10 0.83 -0.10 0.53 -0.13 0.84 -0.22 0.49 -0.17 0.95 -0.05 0.58 -0.19 0.70 -0.34 0.50 -0.22 0.23 -0.14 0.16 -0.30 0.21 0.01 0.08 -0.24 -0.13 -0.05 0.13 -0.51 0.36 -0.11 -0.09 -0.54 0.01 -0.56 -0.13 -0.50 0.09 -0.58 -0.37 -0.57 -0.30 -0.60 -0.07 -0.60 * 'Literacy Benchmark scores for grades 4, 6, 8, and 11
SAT9 Reading subtest scores for grades 5, 7, 9, and 10. Page 26 of 47If If Table 6 If Proportion of Variance Explained in Achievement Outcomes: Partial Eta-squared (^) Values for Prior Achievement, Gender, Race, and Free Lunch Status by Grade Level Grade Level Prior Achievement Gender Race Free Lunch Status Fourth n.a. 0.028 0.046 0.058 If Fifth 0.480 0.002 0.057 0.028 If Sixth 0.477 0.024 0.000 0.000 Seventh 0.541 0.029 0.050 0.026 If Eighth 0.450 0.047 0.001 0.011 If Tenth n.a. 0.020 0.090 0.068 Eleventh 0.502 0.013 0.009 0.000 If If If Performance gap analysis. A performance gap analysis was performed for grades that had data available for two consecutive years (i.e., matching pretest and posttest scores). The performance gap was defined as the standardized difference in pretest-adjusted means between African American students and other students within gender and free lunch status categories
thus, four performance gap estimates were computed for each grade level. These performance gaps were then regressed on grade level to ascertain whether there was a trend across age cohorts. If If If If School composition and program effects. For elementary schools, information was available regarding specific literacy programs being implemented in the schools. For 5 grade, a two level hierarchical linear model (HLM) was performed to examine relationships between school composition factors (aggregate poverty, mean achievement at pretest, and percentage African American enrollment), school literacy programs, and student achievement. In HLM, a student-level (Level 1) model of reading achievement is constructed for each school. The Level 1 model provides a mean (or predicted mean) for each school, as well as a slope coefficient for each student-level predictor. The means and slopes computed for each school in the Level 1 modeling process become outcome variables in the Level 2 (school) model. At level two, HLM uses school composition and program variables as predictors of the means and slopes computed in the Level 1 model. A major advantage of HLM over traditional analyses, in addition to producing more reliable and accurate statistical tests, is that it allows one to assume that the relationships between Page 27 of 47 H student achievement and other student variables are different from school to school. These differences can then be modeled as a function of school characteristics. For example, in addition to ascertaining whether particular features of a school are associated with the overall average effectiveness of a school, one can determine whether school characteristics are associated a more equitable distribution of achievement gains across levels of prior achievement, or are differentially effective for students with different characteristics (e.g., African American students). Because development of HLM models is exploratory, full specification of the analyses is presented in the results section. Longitudinal cohort performance on Benchmark Examinations. For fourth and eighth grades, three consecutive years of Literacy Benchmark Performance Level data were available. The percentage of African American students scoring in Below Basic, Basic, Proficient, and Advanced categories was computed for each year from 2001 to 2003 to provide a basis for examining overall trends in performance across time. Elementary Level Results District-wide Achievement Effects Fourth grade. ANOVA revealed significant main effects for gender (Fi,i83i=52.15, p < .001), free lunch status (Fi,i83i=l 13.30, p < .001), and race (Fi,i83i=87.75,p < .001). None of the interaction effects were significant, which indicates that the main effects for each variable were constant across levels of the other variables. Mean Literacy Benchmark scores were significantly higher for females (M = 214.6) than for males (M = 203.0), for students not eligible for free lunch (M = 217.4 versus M = 200.2), and for non-African American students (Af = 216.4 versus M = 201.2). As shown in Table 7 and in Figures 1 and 2, mean scores for African American students were approximately four-tenths of a standard deviation lower than those of non- African American students who were of similar gender and free lunch eligibility status. An examination of longitudinal cohort performance on the Benchmark Literacy examination showed a dramatic reduction in the percentage of African American students scoring in the Below Basic range from 2001 (49.1%) to 2003 (20.4%
see Figure 3). Simultaneously, the percentage of African American fourth graders who scored at a Proficient level increased from 19.1% in 2001 to 45.2% in 2003 (see Figure 3). Page 28 of 47Id Id Table 7 Id Average Z-scores by Grade Level, Free Lunch Eligibility, Gender, and Race^, Adjusted for Prior Achievement Id Grade Level Not eligible Female Free lunch eligible Male Female Male Id Id ri ri ri Fifth Non African-American African-American Sixth Non African-American African-American Seventh Non African-American African-American Eighth Non African-American African-American Eleventh Non African-American African-American 0.38 0.03 0.09 0.16 0.38 0.04 0.32 0.33 0.28 0.15 ri ri ri ri ri ri 0.60 -0.02 -0.04 -0.07 0.57 -0.01 -0.07 -0.10 0.06 0.04 0.15 -0.18 0.11 -0.16 0.21 0.19 -0.06 -0.14 0.01 -0.14 0.16 -0.13 0.00 0.17 0.46 0.14 -0.19 -0.17 0.15 -0.20 'Literacy Benchmark scores for grades 4, 6, 8, and 11
SAT9 Reading subtest scores for grades 5,7, 9, and 10. Page 29 of 47M ri ri 1.00 ri .80- .85 ri .60 .53 N .40 < .20- ri 0.00- I -.09 IZZlNon Afr-American H -.20 HAfrican American Female Male I Figure 1. Fourth Grade Average Z-scores by Gender and Race
Students Not Eligible for Free or Reduced-price Lunch. M .40 ri .20 ri -.00 EEL ri -.20 ri -.40 ri -.60 Female II Male CeSnoh Afr-American ^HAfrican American Figure 2. Fourth Grade Average Z-scores by Gender and Race
Students Eligible for Free or Reduced-price Lunch. Page 30 of 47 ri ri ri ri ri 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 49.1% --------------^2v4%r 32.4% 31.6% 31.8% Below Basic Basic 4&.2% 3474 19.1 Proficient 0.2%0.8% Advanced M N M 1 % 2001 112002 112003 Figure 3. Percentage of African American Fourth Graders Scoring at Each Performance Level on the Literacy Benchmark Examination by Year, 2001-2003. ri ri Fifth grade. MANCOVA showed significant two-way interaction effects for between gender and African-American status (F2,i463=3 194, p <.05) and African-American and free lunch status (F2,i463=3.195, p <.05). Follow-up univariate tests indicated that the gender X African- American interaction effect was significant only for SAT9 Language scores (Fi,i464=6.37, p =.012), whereas the free lunch status X African-American interaction was significant only for SAT9 Reading scores (Fi,1464=4.598, p < .05). Accordingly, univariate post hoc tests were performed on the interaction effects. Four gender X race groups were formed to test the interaction effect on language scores: African-American males, African-American females, other males, and other females. These groups differed significantly on pretest-adjusted SAT9 Language scores (F 3,1538=37.55, p<.001). Post hoc tests indicated that each adjusted group mean was significantly different from the others, with other males having the highest adjusted Z-score (M=0.38), followed by other females (Af=0.25), African-American females (M= -0.04), and African-American males (M=-0.15). The interaction was attributable to a larger performance gap between males than females across levels of race (see Figure 4). ri Page 31 of 47d d d d d d d .50 .40 .30 .25 .20 .10 0.00' -.10 -.20 E^Non Afr-American ^lAfrican American d Female Male d Figure 4. Adjusted Mean Z-scores by Gender and Race
Fifth Grade SAT9 Language. d d Four groups were also formed to follow-up the/ree lunch X race interaction effect: African-American students eligible for free lunch, African-American students not eligible for free lunch, Other students eligible for free lunch, and Other students not eligible for free lunch. Post hoc tests showed that: (a) the adjusted mean SAT9 Reading score for African-American students eligible for free lunch {M= -0.18) was significantly lower than the adjusted mean for all other groups
(b) the adjusted means for non-eligible African American students (M=0.007) and Other students eligible for free lunch (M = 0.134) were not significantly different
and (c) the adjusted mean for non-eligible, non-African American students (A/=0.501) was significantly higher than that of all other groups. The interaction was attributable to a larger performance gap between African American and Other students who were not eligible for free or reduced-price lunch compared to the gap for students who were eligible (see FigureS). d Page 32 of Vlri ri .60 ri .50 ri .40 ri .20 I -13 I ri 0.00' -.20 -.40 [Z^Non Afr-American HAtrican American Not Eligible Free Lunch Eligible Figure 5. Adjusted Mean Z-scores by Free Lunch Status and Race: Fifth Grade SAT9 Language. School-level predictors of achievement gains and the relationships between student characteristics and student achievement: A hierarchical linear model of 5'* grade achievement. To further explore the relationships among school-level variables and student achievement, a 2- level hierarchical linear model was constructed for fifth grade achievement. First, student-level regression equations were estimated for each school, using 2003 SAT9 reading scale scores as the outcome variable, and 2002 Benchmark Literacy, African American status, free lunch eligibility status, gender, and special education status as predictor variables. Benchmark Literacy scores were centered on the grand mean for this variable, meaning that the intercept of the regression equation for each school was equal to the pretest-adjusted mean score. Relationships were then estimated between the regression coefficients for each school and school composition variables (i.e., reading program, presence of a literacy coach, percentage of African American students, percentage of students eligible for free or reduced price lunch). School composition variables associated with a t-value greater than 2.0 were then incorporated into the final model. As Table 5 indicates, the percentage of students eligible for free lunch was a potential predictor of mean adjusted achievement (Bq, or the school-level intercept), the slope associated with gender, the slope associated with special education status, and the slope associated with African American status. The percentage of African American students was a potential predictor of the slope associated with special education status, as was implementation of the Success for All reading program (see Table 8). Page 33 of 47 d ri Table 8 Results of Exploratory HLM Analyses: School-level Variables as Predictors of Student-level Regression Coefficients II Level-1 Coefficient Potential Level-2 Predictors (School-level) SFA DI RR COACH ri INTRCPTl,BO Coefficient Standard Error t value AFRAMER LUNCH ri GENDER,Bl Coefficient Standard Error t value LUNCH,B2 Coefficient Standard Error t value SPED,B3 Coefficient Standard Error t value PRETEST,B4 Coefficient Standard Error t value AFRAMER,B5 Coefficient Standard Error t value Note. -3.142 2.720 -1.155 0.411 6.250 0.066 -3.137 2.094 -1.498 -1.212 2.321 -0.522 -0.100 0.053 -1.903 -0.109 0.044 -2.484 SFA DI RR COACH AFRAMER LUNCH -2.351 1.675 -1.404 1.586 3.876 0.409 -0.069 1.348 -0.051 0.046 1.449 0.032 -0.069 0.032 -2.132 -0.071 0.027 -2.629 SFA DI RR COACH AFRAMER LUNCH 0.527 1.625 0.324 0.796 3.659 0.217 -0.978 1.258 -0.778 0.273 1.365 0.200 0.011 0.033 0.329 0.010 0.028 0.339 SFA DI RR COACH AFRAMER LUNCH 4.992 2.181 2.289 -5.857 5.200 -1.126 -1.258 1.826 -0.689 -2.003 1.946 -1.029 0.094 0.044 2.131 0.086 0.038 2.271 SFA DI RR COACH AFRAMER LUNCH 0.024 0.044 0.534 -0.001 0.100 -0.013 -0.005 0.035 -0.147 -0.002 0.037 -0.060 -0.000 0.001 -0.049 -0.000 0.001 -0.206 SFA DI RR COACH AFRAMER LUNCH 2.655 2.322 1.143 -1.055 5.330 -0.198 2.394 1.799 1.331 0.566 1.987 0.285 0.075 0.046 1.636 0.080 0.038 2.073 SFA = Success for All program. Recovery program. American enrollment, reduced-price lunch. Coach = Literacy Coach. DI= Direct Instruction program. RR = Reading AFRAMER (level 2) = percentage African Lunch (level 2) = percentage of children eligible for free or t-values over 2.0 in bold. I I Page 34 of 47n H II II II II As shown in Table 9, the percentage of students eligible for free or reduced price lunch was significantly, negatively related to the mean adjusted pretest score for schools f = -4.31, df= 31, p< .001), indicating that schools with higher percentages of children eligible for free lunch tended to have lower student achievement after adjusting for prior achievement. The percentage eligible for free lunch also was negatively related to the slope for gender f = -2.385, df =31, p =0.023), meaning that the achievement differences between boys and girls grew smaller as school poverty grew higher. The percentage of students eligible for free or reduced price lunch was positively related to the slope indicating African American status (r=3.305, df= 31, p = 0.003), which shows that the achievement gap between African American and other students became lower as school poverty rates increased. The type of reading program implemented in schools was not significantly related to any outcome. Table 9 n Hierarchical Linear Model of Fifth Grade Achievement: Final Estimation of Fixed Effects fl Fixed Effect Coefficient Standard Error T-ratio Approx. d.f. P-value fl For For INTRCPTl, INTRCPT2, GOO LUNCH, GOl GENDER slope, INTRCPT2, GIO LUNCH, Gll LUNCH slope. BO Bl 674.570463 -0.245310 3.896852 0.056874 173.107 -4.313 31 31 0.000 0.000 fl For 8.742665 -0.101186 2.638103 0.042419 3.314 -2.385 31 31 0.003 0.023 B2 INTRCPT2, G20 For SPED slope, -6.642145 1.682618 -3.948 32 0.001 B3 fl For For INTRCPT2, G30 SFA, G31 AFRAMER, G32 LUNCH, G33 PRETEST slope, INTRCPT2, G40 AFRAMER slope, INTRCPT2, G50 LUNCH, G51 -30.431707 8.625169 0.296650 -0.028009 8.907047 9.294506 0.289108 0.228295 -3.417 0.928 1.026 -0.123 29 29 29 29 0.002 0.361 0.314 0.904 B4 B5 0.772208 0.037128 20.798 32 0.000 -26.731249 0.181304 3.617033 0.054857 -7.390 3.305 31 31 0.000 0.003 H I LUNCH = Free/reduced price lunch eligibility status. AFRAMER = African American. SPED = Special education Note. status. Secondary Level Results I Sixth grade. ANCOVA resulted in a significant main effect for gender (Fi.i465= 34.18, p < .001) after controlling for 2002 SAT9 Reading scores. As with fourth grade, females (M = 0.252) had significantly higher Literacy Benchmark Z-scores than males -0.110). A significant interaction effect was observed between race and free lunch status (Fi.i465= 6.397, p =0.012). Post hoc tests showed that
(a) the adjusted mean SAT9 Reading score for African- American students eligible for free lunch {M= -0.23) was significantly lower than the adjusted mean for all other groups
(b) the adjusted means for non-eligible African American students (M=0.03) and Other students eligible for free lunch (M = 0.01) were not significantly different
I Page 35 of 47Il II II and (c) the adjusted mean for non-eligible, non-African American students (A/=0.51) was significantly higher than that of all other groups (see Figure 6). .60 II I M M I I I fl fl fl fl fl .40' .20 0.00 -.20 -.40 h .51 Not Eligible B Free Lunch Eligible Q^Non Afr-American HAfrican American Figure 6. Sixth Grade Average Adjusted Z-scores by Free Lunch Eligibility and Race. Seventh grade. MANCOVA showed significant interaction effects for race Xfree lunch status (F2,i507=4.426, p <.O5) and race X gender (F2,i507=3.592, p <.05). Follow-up univariate tests indicated that the race X free lunch status interaction was significant for both SAT9 Reading (Fi.i507=3.592, p <.05) and SAT9 Language (Fi,i507 = 3.592, p <.05), whereas the race X gender interaction was significant only for SAT9 Reading (Fi,1507 = 3.592, p <.05). Follow-up tests were performed using grouping variables as was done for fifth grade. Post hoc comparisons showed that (a) adjusted mean Z-scores for African American students eligible for free or reduced price lunch (Mseading = -0.13, MLanguage= -0.10) were significantly lower than all other groups
(b) there was no significant difference between non-eligible African American students (MReading = 0.01, MLanguage= -0.03) and free lunch eligible other students (MReading= 0.09, MLanguage= 0.14), and non-cligible, non-African American students had higher adjusted mean scores than all other groups (MReading= 0.47, MLanguage= 0.44). For both Reading and Language scores, the interaction was attributable to a more pronounced difference between African American students and Other students who were not eligible for free lunch (see Figures 7 and 8). Follow-up tests of the gender X race interaction effect indicated that the adjusted means for African American females (MReading = -0.08, MLanguage= -0.11) did not differ significantly than those of African American males (MReading = -0.08, MLanguage= -0.14), while both groups had significantly lower adjusted means than Other males (MReading = 0.47, MLanguage= 0.31) and Other females (MReading = 0.29, MLanguage= 0.43). Page 36 of 47ri ri .60 ri .50 ri .40 ri .30 .20 1 A ri .10 ''27'' t. j T J 9|'- ri 0.00 -.10 ri -.20 Not Eligible Free Lunch Eligible EZjNon Afr-American ^BAfrican American Figure 7. Adjusted Mean Z-scores by Free Lunch Status and Race: Seventh Grade SAT9 Reading. .50 ri .40 ri .30 .20 .10 0.00 Oil ri -.10 EIZlNon Afr-American -.20 ^lAfrican American Not Eligible Free Lunch Eligible Figure 8. Adjusted Mean Z-scores by Free Lunch Status and Race: Seventh Grade SAT9 Language. ri I Page 37 of 47 ri ri ri ri Eighth grade. ANCOVA showed significant main effects for gender (Fi.1421=66.41, p < .001) and free lunch status (Fi.uzi = 6.59, p < .001), but not for race (F1.1421 = 0-19, p = .890). There were no significant interaction effects. Females (M =0.179) had higher Literacy Benchmark scores than males (M =-0.165), and students not eligible for free lunch (M=0.010) had significantly higher scores than those who were eligible (M =-0.08). African American students had slightly higher pretest-adjusted mean Z scores (M = 0.009) than other students (M=0.003), mostly attributable to the fact that African American students who were eligible for free or reduced price lunch having a higher adjusted mean Z score than other eligible students (see Figure 9). ri .20 ri .13 .10 ftu)*-: ri ri 0.00 ri C.Jae^'.gig -.10 -12 , 'i ri ri -.20 Not Eligible Free Lunch Eligible ri ri ri ri H IZZlNon AfrAmerican ^BAfrican American Figure 9. Eighth Grade Average Adjusted Z-scores by Free Lunch Eligibility and Race As shown in Figure 10, a substantial decline in the percentage of African American eighth grade students scoring Below Basic on the literacy exam occurred between 2001 (48.5%) and 2003 (34.5%). During the same time period, the percentage scoring at a Proficient level rose from 14.9% to 23.9%, and the percentage scoring at a Basic level rose from 35.7% to 40.5%. Page 38 of 47 h d ri ri 60.0% 50.0% 48.5% ri 40.0% 34.7% 33.0% 40.5% ri 30.0% ri 20.0% 10.0% ri ri ri ri ri ri ri ri ri ri 0.0% 34.5% 357r%qj ll JUI J rz Below Basic Basic '% Proficient 02001 112002 112003 0.9% 0.7%) 1.1% Advanced Figure 10. Percentage of African American Eighth Graders Scoring at Each Performance Level on the Literacy Benchmark Examination by Year, 2001-2003. Tenth grade. No prior achievement data were available for 10* grade students. MANOVA indicated a significant interaction effect between race and free lunch status (F2,i34o = 9.727, p < .001). Follow-up post hoc tests showed that (a) mean Z-scores for African American students eligible for free or reduced price lunch (MReading = -0.54, MLanguage= -0.51) were significantly lower than all other groups
(b) there was no significant difference between non- eligible African American students (MReading = -0.21 , MLanguage= -0.16) and free lunch eligible Other students (MReading= -0.10, MLangiiage= "0.12), and non-eligible, non-African American students had higher mean scores than all other groups (MReading= 0.83, MLanguage= 0.75). For both Reading and Language scores, the interaction was attributable to a more pronounced difference between African American students and Other students who were not eligible for free lunch (see Figures 11 and 12). Page 39 of 47d ri 1.00 ri .80 'itv. .83 .60 .40 ri .20' 'X I 1 ri 0.00 I l-^l -.20' ri -.40' I ri -.60 [^]Non Afr-American -.80 HlAfrican American ri Not Eligible Free Lunch Eligible Figure 11. Mean Z-scores by Free Lunch Status and Race: Tenth Grade SAT9 Reading. M 1.00 II .80 II .60 .40 II .20 ri -.00 -.12 id ri -.20 -.40 EniNon Afr-American ri -.60 ^lAfrican American Not Eligible Free Lunch Eligible Figure 12. Mean Z-scores by Free Lunch Status and Race: Tenth Grade SAT9 Language. Page 40 of 47 M ri ri Eleventh grade. ANCOVA revealed significant main effects for gender (Fi,966=4.885, p < .001
Mfemales=0.235, Mmales=-0.020) and race (Fl,966= 3.699, P =.002
MAfr-Amer=-008, Mnoh Afr-Amer=0.221). No Significant interaction effects were observed. As shown in Figure 13, the average adjusted Z-score for African American students was about 0.20 lower than that of other students within levels of free lunch eligibility. ri .40 ri .30 .20 .10 0.00 -.10 :r 'rm II ^1 [iZlNon Atr-American -.20 ^African American M M M H llr< 'a II Female Male Figure 13. Eleventh Grade Average Adjusted Z-scores by Gender and Race. Il II II II ri Page 41 of 47ri ri Trends in Student Gains Across Age Cohorts ri ri ri A performance gap estimate was computed within levels of gender and free lunch eligibility status by subtracting the average adjusted Z-score for other students from that of African American students. Negative gap scores indicate that African American students are gaining in achievement at a slower rate, whereas positive scores indicate that African American students are gaining at a greater rate. A cubic regression of these scores on grade level was performed to determine whether there were any trends in the differential gain rates. As Figure 14 illustrates, the performance gap grew smaller as grade level increased, and was near or above zero for 6* and 8 grades, in which the Literacy Benchmark score was the outcome. Performance gap indices were negative and relatively large for 5* and 7* grades, particularly among male students not eligible for free lunch. The SAT9 was the outcome variable in these two grades. ri .2 ri 0.0' -.2' ri ri -.4. ri -.6' ri O. < 0 -.8 ri 4.0 Male, Not Eligible Female, Not Eligible Male, Free Lunch Female, Free Lunch Rsq = 0.7435 5.0 6.0 7.0 8.0 9.0 GRADE A X Figure 14. Cubic Regression of Performance Gap on Grade Level. ri Note. Performance gap is defined as the difference in mean adjusted Z-scores between African American and other students of the same gender and free lunch eligibility status. An upward trend indicates a reduction in the gap. Negative values indicate that non-African American students within the category made greater gains in achievement, whereas positive values indicate that African American students made greater gains. Page 42 of 47 ri Summary and Conclusions ri The Little Rock School District is commended for the emphasis given to increasing literacy in all schools in the district and not just those in the lowest performing strata. The state and local initiatives in literacy for early learners, including ELLA and Effective Literacy, are well grounded in current research of best practice. In addition, the Reading Recovery and Success for All models are among the best researched and proven programs in the nation for lower-performing students. Impressions from interviews and survey data, however, are that these programs are often perceived as separate and discrete entities instead of integral to a district comprehensive literacy program. Teachers describe themselves or their schools as doing ELLA or Success for All and only the certified tutors doing Reading Recovery. Unfortunately, teachers using the basal reading series dont perceive their schools as doing a literacy program at all. Teachers commented that their schools weren t big on ELLA, or that they close the door and do their own thing. Middle and high school teachers comments seemed to indicate that they also did not perceive themselves as being involved in a literacy plan beyond the traditional roles they have had as English teachers. Thus, it is recommended that the districts plan, or big picture of literacy, be developed and presented to teachers in a format that communicates how each program, school, grade level, and teacher contributes to and accomplishes literacy goals. The professional development in basic literacy has been well received and represents an enormous accomplishment for the district. It is recommended that efforts be intensified in this regard as teachers indicated that they were appreciative of and eager for training. Management and delivery of the professional development, however, needs to be made more consistent and available to teachers. The primary concerns voiced by teachers were scheduling problems, inadequate space and availability of training for the numbers of teachers needing to be trained, retraining for teachers who change grade levels, training for new teachers, and obtaining qualified substitutes for teachers while they attend training. In addition, the same level of professional development needs to also be made available to upper grade, middle, and high school teachers. The impressions of professional development communicated by upper grade teachers and those who were not implementing special programs were that professional development has been minimal, targeted to cunent hot topics (e.g., portfolios), and inconsistent in quality. Teachers perceptions of the impact of literacy programs were extremely mixed. Writing and composition were literacy areas that all teachers agreed had been emphasized and improved in their schools and classrooms as a consequence of literacy initiatives in the district. Although ELLA has generally been well received by teachers, the level of change it has engendered in teachers instruction is difficult to determine. Some of the positive comments relative to ELLA had more to do with the materials teachers had received as a part of the training than the new ideas they had been provided. During the focus groups, the most impressive level of agreement that teachers voiced was by teachers who had Reading Recovery teachers as Literacy Coaches in their schools. Teachers recognized the level of expertise that the Reading Recovery teachers had and voiced strong appreciation for and dependence on these teachers leadership in implementing literacy instruction in their schools. The concept of highly trained Literacy Coaches being placed in and available to all elementary schools is a national issue presently and it is Page 43 of 47I I fl fl fl recommended that the district find ways to support this concept not only in elementary schools with large at-risk populations, but in all schools in the district. In terms of achievement, the conclusions based on the results of analyses performed on are presented below by research question. It should be noted that these conclusions might not apply if data were available for grades K - 3, or if pretest data were available for fourth grade. Special caution is in order related to interpreting the effects of school programs on student achievement, because those analyses were based only on 5* grade data, whereas the programs tend to focus more strongly on the lower grades. Elementary Achievement 1. After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement, did African American students exhibit similar levels of academic achievement as other students? African American students had substantially lower absolute performance than did other students. The academic gains on literacy tests were lower for African American students than for other students. 2. What proportion of the variance in 2003 literacy achievement was uniquely attributable to whether students were African American? Although there was a significant relationship between African American status and student achievement, the proportion of variance in academic performance attributable to African American status was very low4.6% for fourth grade, and 5.7% for fifth grade. 3. What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? The performance of African American fourth grade students on the Benchmark Literacy examination improved dramatically between 2001 and 2003, with nearly half performing at a Below Basic level in 2001, compared to only one-fifth in 2003. 4. Was there a relationship between the literacy program implemented at the school, school composition variables, and the achievement of African American students? No significant relationship was observed between the type of literacy program implemented and the achievement of African American students. The percentage of African American students enrolled in a school did not predict overall achievement or the achievement of African American students. School poverty, as measured by the percentage of students eligible for free or reduced- price lunch, had a negative effect on the achievement gains of students. Page 44 of 47I Secondary Achievement 5, After controlling for gender, eligibility for free or reduced-price lunch, and prior achievement, did African American students exhibit similar levels of academic achievement as other students? The absolute level of achievement of African American students was substantially lower than that of other students of similar gender and free lunch eligibility status. Generally, the gains in academic achievement of African American students were similar to those of other students in grades 6 to 11. a 6. What proportion of the variance in 2003 literacy achievement was uniquely attributable to whether students were African American? !l When data on prior achievement were available, the proportion of variance in 2003 achievement attributable to African American status was quite low at the secondary level, ranging from 0% to 5%. a 7. What was the trend in the achievement of African American students on the Literacy Benchmark examination from 2001 to 2003? a The performance of African American 8* graders on the Literacy Benchmark exam improved substantially and consistently between 2001 and 2003, with the percentage scoring Below Basic dropping from 48.5% to 34,5%, and the percentage scoring Proficient increasing from 14.9% to 23.9%. 8. Were differences in achievement gains between African American students and other students similar at different grade levels and for different test instruments? a From the 5 to the 8* grade cohort, the achievement gains of African American students became more similar to those of other students, and for some subgroups surpassed those of other students in 8* grade. The gap in achievement gains was greater on the SAT9 than on the Literacy Benchmark examination. Presumably, the Literacy Benchmark examination is more closely aligned to the mandated curriculum than is the SAT9, which is intentionally designed to be insensitive to curricular differences. Overall Achievement a Substantial differences exist in the overall achievement of African American students and other students in the Little Rock School District. African American students tend to gain in literacy achievement at a lower rate than other students, especially at the elementary level. The differences in academic gains tend to be smaller at higher grade levels. The three-year trend in Literacy Benchmark scores shows substantial, sustained improvement in the academic achievement of African American students between 2001 and 2003. African American status of the student, as well as the percentage of African American students enrolled in a school, explain only small amounts of variance in student outcomes compared to prior achievement. The Page 45 of 47Il II II aggregate poverty level of the school, however, is significantly related to achievement gains made by students attending the school. None of the curricular programs examined in this study had a significant impact on the achievement of 5* grade studentshowever, these programs are most likely to have an impact at lower grade levels. II II II II II II II II II II II d d d L Page 46 of 47Il II Appendix A II Teacher Focus Group Interview Guide 1. General information A. B. C. D. E. F. How long has (program name) been implemented in the district? Did teachers have input in the decision to implement (program name)? What elements of (program name) are the most effective? What elements of (program name) are the least effective or least desirable? How would you describe teacher support for (program name)? How would you describe ongoing support from the district for (program name)? n. II Professional Development A. What professional development related to (program name) has been provided for teachers? B. How would you rate the quality of that professional development? II III. Classroom Level Changes A. B. C. What changes have been made at the classroom level as a result of (program name) {e.g. teaching to standards, technology, interdisciplinary and project-based learning, cooperative and team-based approaches, authentic, alternative assessments)! How does (program name) address special needs children? Within the framework of (program name) are there specific strategies or programs for addressing the academic needs of disadvantaged students? Il IV. I II Results A. How has (program name) impacted students? B. Are there differences in student achievement because of (program name)? C. What differences do you see in student motivation, attendance, or conduct because of (program name)? D. How has (program name) impacted teachers? V. Il II Parent/Community support A. How has parental support for the school changed as a result of (program name)? B. How has community support for the school changed as a result of (program name)? Closure - is there anything else you would like to say about (program name)? il I I Page 47 of 47
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.

<dcterms_creator>Center for Research in Educational Policy, University of Memphis, Memphis, Tennessee</dcterms_creator>