'Research Brief: SMART/THRIVE in the Little Rock School District''

CREP \cmitrfar iaE^htcatuaudPaiiey Center for Research in Educational Policy Research Brief The University of Memphis 325 Browning Hall Memphis, Tennessee 38t 52 Toll Free: 1-866-670-6147 Reading Recovery in the Little Rock School District\M^^ T0 ssS*^*^ WffWWIIIffltfiTglTrTWgl 1^ R CREP crater Jaf Ratarck ia EJueatiaaai Miey Center for Research in Educational Policy The University of Memphis 325 Browning Hall Memphis, Tennessee 38152 Toll Free: 1-866-670-6147 Research Brief Reading Recovery in the Little Rock School District fl fl January 2006 Anna W. Grehan Steven M. Ross Lynn Harrison Center for Research in Educational Policy fl John Nunnery Old Dominion University fl IRESEARCH BRIEF: READING RECOVERY IN THE LITTLE ROCK SCHOOL DISTRICT Background Reading Recovery is an internationally used intervention program, designed by noted literacy expert Dr. Marie M. Clay. It is a short-term intervention program for first grade students who are struggling in learning to read. The goal of Reading Recovery is to provide early, effective interventions, through one-on-one tutoring, which not only empowers struggling readers and increases their chances of continued success, but also reduces the cost of these learners to educational systems. In 1995. the Little Rock School District (LRSD) began implementing Reading Recovery in two elementary schools. Over the years the LRSD has expanded the Reading Recovery program and developed a partnership with the University of Arkansas at Little Rock (UALR) to conduct training and provide professional assistance for Reading Recovery specialists. At the end of the 2004-2005 school year the LRSD had 28 trained Reading Recovery teachers who were serving 18 of the 34 elementary schools in the district. In February 2005. researchers from the Center for the Research in Educational Policy (CREP) at the University of Memphis presented an evaluation plan for the Reading Recovery program to principals, representatives of the LRSD and other stakeholders. The evaluation plan for Reading Recovery was presented as part of a larger district study in which Reading Recovery and three other programs were evaluated. The evaluation plan for Reading Recovery included: (1) analyses of Reading Recovery student achievement and program data. (2) principal, teacher, and parent surveys and interviews, and (3) observations of Reading Recovery tutoring sessions. The evaluation plan was approved and researchers from CREP immediately began implementation. Purpose, Plan and Participants Purpose V The major goal of the study was to understand the impact Reading Recovery has had on African American students in the Little Rock School District. Specifically, through Reading Recovery, are African American students showing improvement in academic achievement? With that goal in mind, the primary research question for this evaluation was: Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? In order to gain an even clearer picture of how well the Reading Recovery program has been used in the schools and the impact it has had on African-American students, six other questions were addressed. The information gathered to answer these questions provided a more complete understanding of how well the program has been carried out in the district. These additional questions were: I What are the quality and level of implementation of Reading Recovery at the 18 schools implementing in 2004-2005? . What is the level of participation in Reading Recovery by African American students relative to other ethnic groups at the school? What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? [NOTE: A 1 LI Reading Recovery student is discontinued when he/she demonstrates, through the appropriate assessments, ability to read on grade level with the regular class.] What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of principals, regular first grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? Plan To answer these questions a research plan that collected a variety of information from numerous sources was required. Researchers used scores from the following standardized achievement tests or in- class assessments: H Iowa Test of Basic Skills (ITBS) Developmental Reading Assessment (DRA) Dynamic Indicators of Basic Early Literacy Skills (DIBELS) An Observation of Early Literacy Achievement
Reading Recovery Subtests The CREP researchers also needed to determine how well the Reading Recovery teachers understood their tasks as teachers and how well they were able to follow the Reading Recovery program design. To answer these questions, data was collected data by using a Reading Recovery Teacher Questionnaire and a Reading Recovery observation tool. The questionnaire contained items for teachers to rate on an agreement-disagreement scale and open-ended items soliciting comments about the program. With the observation tool, trained Reading Recovery specialists monitored Reading Recovery teachers as they worked with students and recorded how well they followed Reading Recovery guidelines. CREP researchers interviewed, via telephone, four Reading Recovery teachers who were currently being trained to understand how effectively their training was being conducted. To determine how the program was thought of by principals, other teachers, and parents, questionnaires were specifically designed for each group. Classroom teachers from kindergarten through third grade in schools that had a Reading Recovery program were surveyed and asked both ratings items and open-ended questions. Parents with children currently in the Reading Recovery program were given a questionnaire to complete, also containing both types of items. Ten principals from Reading Recovery schools were randomly selected to participate in a phone interview in which their comments were recorded and transcribed. All of the data from observations were collected in March through May of 2005. Copies of all the instruments used and a complete breakdown of all results can be found the technical report. Participants Students: The overall sample of students that was used in this research included 1,094 first graders in the schools with Reading Recovery programs (18 schools) for the school year 2004-2005. Of this larger sample, 230 students were referred to the Reading Recovery program and became the Reading Recovery Group and 864 students who were not in the Reading Recovery program became the Comparison Group. There were similar numbers of African-Americans in both groups, but students in the Reading Recovery group were more likely to be male, in a lower income bracket, and to receive special education services. 2 ' -Principals: Ten principals from Reading Recovery schools were randomly selected to participate in a phone interview. The phone interview lasted approximately 45 minutes and responses to questions regarding the Reading Recovery program and its effectiveness were recorded and transcribed. Reading Recovery Teachers in-Training: Four randomly selected teachers currently being trained to become Reading Recovery specialists were interviewed through a 45-minute phone interview. These teachers provided insight into the current training efforts and professional development needs for new n teachers. Reading Recovery Teachers: There were 22 Reading Recovery Teachers who were surveyed through a specially designed questionnaire. In addition, 14 Reading Recovery Teachers were observed as they worked with their students. Classroom Teachers: Kindergarten through third grade teachers, working in schools with Reading Recovery programs, were also surveyed and 156 responded. Parents: Parents of students currently participating in a Reading Recovery program were sent a questionnaire and 95 were completed and returned. A survey in Spanish was sent when necessary, and 9 of those returned were completed in Spanish. n n Findings Researchers spent the summer and early fall compiling the information learned from the phone interviews, surveys, and observations. In late fall, achievement data and assessment records were obtained. Through careful analysis and with the use of statistical techniques, a thorough analysis of the Reading Recovery program was completed. The findings are best understood in context of the questions that were asked in the evaluation plan. Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? The impact of the Reading Recovery program on African-American students was comparable to students of other ethnicities. This impact, as it regards reading success, was more significant when students completed the entire program and were successfully discontinued. In addition, the benefits were more noticeable in areas of beginning reading skills that included basics such as hearing and recording sounds (Observation Survey) and phoneme segmentation (DIBELS), than in more advanced reading skills, such as text reading (DIBELS & DRA). However, there were no noticeable gains on statewide standardized tests (ITBS) for Reading Recovery students. What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? Reading Recovery appeared to be well implemented and there was a high degree of consistency in the way the Reading Recovery teachers carried out the program. Classroom observations, conducted by outside experts in Reading Recovery, agreed that in the majority of areas, teachers were well prepared and were following the guidelines outlined by the Reading Recovery program. In addition, when looking at students achievement information, there was no difference between those students who had a more experienced teacher and those having a teacher with less experience. This outcome suggested that teachers were well trained and were using the Reading Recovery model appropriately. What is the level of participation in Reading Recovery by African-American students relative to other ethnic groups at the school? Most of the students participating in Reading Recovery were African-American, which was expected given that African-American students were the majority in the 18 schools being studied. 3 g| What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued ? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? There are three categories in which a Reading Recovery student may be placed at the end of the program year: (1) Discontinued: successful completion of the program
(2) Incomptete^stilHn the program without having enough time to complete the program
and (3) Recommended for Further Action: Reading Recovery teachers believe that the student is in need of some other instructional support to be successful. African American students in Reading Recovery were as likely as other ethnic groups to be placed in the Discontinued or Incomplete categories
and were slightly more likely to be Recommended * . .. . . . ... I'l__I. . iL .. AO rvlor'orJ fhn for Further Action. African-American students were more likely than other ethnicities to be placed in the Low/Lower Middle reading group by their classroom teachers, after completing Reading Recovery. Reading Recovery is a program designed to be a daily intervention. However, about one-fourth 4 of sessions were missed due to either teacher or student absence or teacher or student being unavailable. So. on average, there were 3.5 sessions held per week, instead of 5. B V What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? Reading Recovery teachers were extremely positive about the Reading Recovery program and were eager to see it continued. Teachers appeared to be very knowledgeable about the program- well- trained, prepared, and supported by their school and district. Reading Recovery teachers thought the program was having a positive impact on their students, including their African-American students. Reading Recovery teachers expressed a need for greater planning time and for more opportunities to continue to monitor their students after they completed the program. Teachers in-training were equally positive and committed to the Reading Recovery program. These teachers reported being well prepared by their training. What are the perceptions of principals, regular first grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? Principals were very supportive of Reading Recovery and reported being actively involved in ensuring that it was carried out correctly. Most principals agreed that, through Reading Recovery, the achievement gap was being bridged for their African American students. Classroom teachers at Reading Recovery schools had a clear understanding of the Reading They praised the program and felt it had a very positive impact on student Recovery program, achievement. What are the perceptions of parents/ guardians or Reading Recovery students regarding program impacts, strengths, and weaknesses? Parents were generally very pleased with the Reading Recovery program and believed that the program was helping their child become a successful reader. Nearly all of the parents indicated a clear understanding of the program and appreciated the one-on-one attention their child was receiving. 4Compliance Remedy Questions Recommended Program Modifications Reading Recovery was extremely well received by parents, teachers and administrators and all reported a positive impact on those students who were in the program. It would be expected that higher achievement scores would also be detected
yet this was not the case. However, given the focus of the program in improving reading performance, it might also be unrealistic to expect highly noticeable inrrM-sPs on norm-referenced multiole-choice standardized tests. Possible program modifications increases on norm-referenced multiple-choice standardized tests. needed to produce greater achievement gains are: An expanded program that would include more Reading Recovery teachers. With more teachers, there could be greater monitoring of students after they discontinue the program to ensure that their momentum is maintained. There are many students in need of Reading Recovery in the Little Rock School District and Reading Recovery teachers may be pressed to discontinue one qroup of students in order to begin assisting another group. The necessary follow-up contact can not occur, and therefore the slippage in achievement that has been noted in the program in other states and districts may be found among students in the Little Rock School District. With an expanded program, these students could be more closely monitored and their gains could be maintained. In addition to an expanded program, a transitional plan for students who have discontinued in aaoiiion lo an expanueu progiam, ci uciieiuuiiai ------------ --------------- should be explored. Such a plan could involve daily monitored reading that would provide n another buffer against slippage. Increased professional development of classroom teachers would enable them to understand how to integrate their Reading Recovery students back into the classroom once they have been discontinued, and how to provide the appropriate instruction and feedback so that students would continue to improve. Increased partnership with University of Arkansas at Little Rock (UALR) to help with the development of transitional plan and professional development for classroom teachers. n Many of the principals, as well as Reading Recovery teachers, cited examples of profound student achievement and continued noteworthy success of former Reading Recovery African- American students. Although evidence of these success stories were not detected by analyzing standardized test scores as a whole, some African-American students are reported to be experiencing continued, sustained high levels of achievement. Future studies might provide a more in-depth analysis of a small number of students whose gains were maintained to determine what factors contributed to their successes and how these factors can be generalized to the Hi I population as a whole. Expectations of Program Modifications Reading Recovery has valuable components that, with adaptation and modification, can be even more effective. With program modifications, the Little Rock School District could expect: Progressive gains on standardized test scores over time. An increased number of students involved in the Reading Recovery program. A greater adherence to Reading Recovery guidelines, especially those relating to the number of sessions required for optimum benefits. 5 More teachers throughout the district better able to serve students at-risk in the areas of literacy and reading. Sustained achievement of students upon completion of the Reading Recovery program. A stronger relationship with UALR professionals that would continue to provide the Little Rock School District with the most up-to-date research findings and best practices for reading and literacy instruction. a I 6 e I i-
*CREP ' CenUrf^r Httevek ui Educational ^oEcy Center for Research in Educational Policy The Univetsity of Memphis 325 Browning Hal Memphis. Tennessee 38152 Tol Free 1-866-670-6147 Evaluation of Reading Recovery In the Little Rock School District TECHNICAL REPORT 9 A n nn H CREP xca/er ieutrci itt Educational policy Center for Research in Educational Policy The University of Memphis 325 Browning Hall Memphis, T ennessee 38152 Toll Free 1-866^70-6147 Evaluation of Reading Recovery In the Little Rock School District ai TECHNICAL REPORT 91 January 2006 -r ,1.
Anna W. Grehan Steven M. Ross Lynn Harrison Center for Research in Educational Policy John Nunnery Old Dominion UniversityEXECUTIVE SUMMARY INTRODUCTION ^1 The Little Rock School District began implementing Reading Recovery during the 1995-1996 school year in two schools. During the first four years of implementation, the Little Rock School District was part of the Pulaski County Reading Recovery Site. By 1998 there were eight trained Reading Recovery teachers in seven schools in the Little Rock School District. In July 1999 the district became a Reading Recovery Site hiring a full-time teacher leader. In 2000-2001 the University of Arkansas at Little Rock (UALR) began conducting the Reading Recovery teacher training for the district. Between the 2000-2001 and 2004-2005 school years UALR trained 17 Reading Recovery teachers for the district. At the end of the 2004-2005 school year the Little Rock School District had 28 trained Reading Recovery teachers serving 18 of the 34 elementary schools in the district. Reading Recovery is one of eight literacy programs, interventions, and/or models used by Little Rock schools. Currently, Little Rock School District funds are used to support the program. The goal of Reading Recovery is to dramatically reduce the number of first grade students who have difficulty learning to read and write and to reduce the cost of these learners to educational systems. Reading Recovery is a short-term intervention program of one-to-one tutoring for the lowest-achieving first graders. The intervention is most effective when it is available to all students who need it and is used as a supplement to good classroom teaching. Individual students receive a half-hour lesson each school day for 12 to 20 weeks with a specially trained Reading Recovery teacher. As soon as students can read within the average range of their class and demonstrate that they can continue to achieve, their lessons are discontinued and new students begin individual instruction. The evaluation plan for the Reading Recovery program in Little Rock School District included: (1) analyses of Reading Recovery student achievement and program data, (2) principal, teacher, and parent surveys and interviews, and (3) observations of Reading Recovery tutoring sessions. This report is part of a larger district study of four programs evaluating the effectiveness in improving and remediating the academic achievement of African-American students. This report includes results from 18 elementary schools participating in the Reading Recovery program. RESEARCH QUESTIONS The major goals of this research study were to evaluate African-American student achievement outcomes, program implementation fidelity, and principal, teacher, and parent perceptions concerning the Reading Recovery tutoring program for first grade students. Student achievement results on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement were analyzed to compare the progress of first graders enrolled in the Reading Recovery intervention program and comparison students in 2004-2005. Program implementation ratings were obtained from observations of 14 tutoring sessions in nine schools. The survey and interview results are based on the perceptions of 156 classroom teachers in grades K-3, 22 experienced Reading Recovery teachers, four teachers in-training, 10 principals, and 95 parents. The Reading Recovery evaluation was structured around the following seven primary and supplemental research questions. Primary evaluation question: Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? 1n Supplemental (Qualitative/Step 2) evaluation questions: E What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? What is the level of participation in Reading Recovery by African-American students relative to other ethnic groups at the school? What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? DESIGN 4 The evaluation period extended from February 2005 through May 2005. The evaluation design was based on both quantitative and qualitative data collected from Reading Recovery intervention observations, principal and teacher in-training interviews, classroom teachers, Reading Recovery teachers, parent surveys, and Reading Recovery program data. Reading Recovery student-level achievement data on the ITBS, DRA, DIBELS, and An Observation of Early Literacy Achievement was received in fall 2005 and incorporated in this report. The primary data collectors in this study were Center for Research in Educational Policy (CREP) trained site researchers. Site researchers: (1) conducted Reading Recovery intervention observations, (2) administered the Reading Recovery teacher survey, (3) conducted principal and teacher in-training interviews, and (4) collected program data. Principals at Reading Recovery schools were responsible for administering the classroom teacher and Reading Recovery parent surveys. Participants Little Rock School District is located in Central Arkansas and serves approximately 26,500 students, with African-Americans representing approximately 67% of the district student population, in 49 schools in an urban area with a population of 184,000. In the 2004-2005 school year 18 w elementary schools and 230 first grade students, of which 173 are African-American, 27 are Caucasian, 22 are Hispanic, and the remaining 8 are other ethnicities, participated in the Reading Recovery program in the district. Three schools were in their first year of program implementation (Bale, Stephens, and Terry) and survey and observation data were not collected from these schools. However, randomly selected teachers in-training and principals from these three schools were interviewed for this study. INSTRUMENTATION Six instruments were developed by CREP to collect the evaluation data: a classroom observation tool, a principal interview, a teacher in-training interview, a Reading Recovery teacher questionnaire, a classroom teacher questionnaire, and a parent survey. 2 Program Data Reading Recovery program information was obtained from data submitted to the National Data Evaluation Center at The Ohio State University in 2004-2005. Each year the schools and district submit program data and receive a site report. The report represents an examination of Reading Recovery student outcomes for Little Rock and accounts for all children served by Reading Recovery within the site during the 2004-2005 school year. In addition, attention is given to implementation factors that may be supporting or hindering the success of the intervention at the site. Student Achievement Results In addition to the program data, interviews, survey, and observation tools cited above, reading achievement data are derived from scores on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement. PROCEDURE Data for the evaluation were collected March-May for the 2004-2005 school year. On February 16, 2005 principals were given an overview of the evaluation and timelines for collecting data. On March 3-4 and April 14-15, 14 tutoring observations were conducted in nine randomly 91 selected experienced Reading Recovery schools by two Reading Recovery content experts from Georgia State and The Ohio State Universities. Only experienced Reading Recovery teachers were observed. On March 23 a CREP trained site researcher administered the Reading Recovery teacher survey to experienced teachers at a regularly scheduled monthly meeting at UALR. In April and May principals in experienced Reading Recovery schools administered the classroom teacher questionnaire to K-3 grade teachers. Principals also administered the parent survey to parents whose children were currently receiving first grade intervention services in the 15 experienced Reading Recovery schools. Four teachers in-training and 10 principals were randomly selected from the 18 Reading Recovery schools to participate in a phone interview in April and May conducted by CREP researchers. Reading Recovery program data were received in summer 2005 from the National Data Evaluation Center at The Ohio State University. Student achievement data were received from the district in fall 2005. METHODS - STUDENT ACHIEVEMENT Sample. The sample included 1,094 first grade students who attended one of 18 schools that implemented the Reading Recovery program during the 2004-2005 school year. Of these, 230 were referred to the Reading Recovery program, and 864 were in the comparison group. The percentages of students who were African-American or of Limited English Proficiency were similar between the comparison and Reading Recovery groups (73.6% versus 75.6% and 7.3% versus 8.7%, respectively). However, Reading Recovery students were more likely to be eligible for free or reduced-price lunch (84.3% versus 73.1%), to receive special education services (15.7% versus 8.6%), and to be male (58.1 % versus 48.0%). Reading Recovery Treatment Level. In the Reading Recovery program, students who are deemed to have attained a reading level equivalent to their peers are assigned an end-of-program status of discontinued. Children who received 20 or more weeks of Reading Recovery services, but who do not attain a reading level equivalent to their peers are assigned a status of recommended action. Children who have received fewer than 20 weeks were assigned incomplete program status. Children designated as unknown status were removed from the program in fewer than 20 weeks due to reasons other than the school year ending. Other children were designated as having moved during the school year. The median number of sessions of recommended action (Md = 68.50
n = 68) children was actually higher than the median for discontinued students (Md = 57.00
n 3 = 90). The medians for incximplete students {n = 46) and unknown" students (n = 12) were very similar (Md = 44.63 versus 46.08, respectively). For the purposes of this study, recommended action and discontinued students were categorized as complete program
incomplete and 158 with a unknown students were categorized as incomplete program
and students who moved were eliminated from the analyses. This left a total of 216 Reading Recovery students
133 ...3. 4 4 complete program, and 58 with an incomplete program. Measures. Pretest (covariate) measures included: (a) Spring, 2004 Developmental Reading Assessment (DRA) scores, (b) fall 2005 Observation Survey: Reading Recovery program subtests that included Letter Identification, Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds in Words, and Text Reading, and (c) the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in Letter Naming Fluency, Phoneme Segmentation Fluency, Nonsense Word Fluency, and Word Use Fluency. Outcome measures included Iowa Test of Basic Skills (ITBS) Reading Normal Curve Equivalent Scores, DRA subtests. Observation Survey: Reading Recovery subtests, and DIBELS subtests administered in Spring 2005. To achieve some parsimony in the analyses, pretest measures were subjected to a principal components analysis, and regressionbased factors scores were constructed for pretest DRA, Observation Survey
Reading Recovery subtests, and DIBELS subtests. A single factor accounted for 60.1% and 60.5% of the variance in the DRA and Observation Survey: Reading Recovery subtests, and DIBELS subtests, respectively. Outcome measures included the DRA and Observation Survey: Reading Recovery, DIBELS subtests, and ITBS Reading. Student achievement analyses. A 3 (Program Status) by 2 (African-American versus other) multivariate analysis of covariance (MANCOVA) was performed on two test batteries, using student gender, the pertinent pretest factor score, free lunch status, special education status, and LEP status as covariates. One test battery included the DRA and subscales of the Observation Survey: Reading Recovery. The second test battery included subtests from the DIBELS. A similar 3 by 2 analysis of covariance was performed on ITBS Reading NCE scores using the 2003-2004 DRA score as a pretest covariate. For the multivariate analyses, Wilks lambda was used as the criterion of multivariate significance. Where Wilks lambda indicated a significant multivariate effect, follow-up univariate analyses were performed on each outcome variable using the Bonferroni procedure to control for experimentwise alpha. When significant univariate results were found, post hoc analyses were performed using Scheffes procedure. Effect size estimates were computed for all posttests by subtracting the adjusted mean for the comparison group from the adjusted mean from the Reading Recovery group within levels of race, then dividing by the total standard deviation of the posttest (Cohen, 1988
Hedges & Olkin, 1985). For ITBS Reading NCE effect size estimates, the population standard deviation of 21.06 was used. The effect size estimate (ES) represent the standardized difference between treatment and comparison group means, which allows for comparison across measures that have different metrics. Exploratory and supplementary analyses. Impact of teacher experience and number of sessions. The number of years experience with Reading Recovery was recorded for each teacher. Years experience ranged from 1 to 10 years. ITBS Reading NCE scores and DRA test scores for 2004-2005 were regressed on 2003-2004 DRA/RR Observation Survey factor scores and dummy- coded variables representing student ethnicity, free lunch status, gender, special education status, and LEP status. Standardized residuals were saved from each of these analyses, and plotted against teacher years of experience to graphically assess the nature of the relationship between teacher experience and teacher effectiveness in Reading Recovery for students who received a Complete program. Standardized residuals for each of these 2004-2005 tests were also plotted against number of Reading Recovery sessions received for all students (i.e., those who received either a Complete or Incomplete program). 4 r.iRESULTS Classroom Observation Results Reading Recovery Implementation Assessment Instrument (RRIAI) As indicated in the description of the RRIAI, the observation procedure primarily focuses on Reading Recovery Program Components and Program Strategies. The site observers used a four-point rubric (1 = poor or unacceptable, 2 = below average in comparison to other programs observed, 3 = meets nearly all standards of program quality, and 4 = above average in comparison to other programs) to rate the frequency and application of components and strategies of Reading Recovery instruction. m Reading Recovery Program Components The overall mean rating for the Reading Recovery Program Components was 3.46 which suggests a high level of program implementation. Of the six subcategories the highest observed ratings were: assembling a cut-up story and introducing and reading a new book observed to be above average in 85.7% of tutoring sessions. The program component subcategory with the lowest observed rating was working with letters and/or words using magnetic letters observed to be above average in only 35.7% of tutoring sessions. The six program components were observed in at least 92.9% of 14 tutoring sessions. m Reading Recovery Program Strategies The overall mean rating for the Reading Recovery Program Strategies was 3.61 which also indicates a high level of instructional effectiveness. Of the eight During Tutoring Lesson subcategories, the highest observed rating was for appropriate text selected throughout the lesson observed to be above average in 85.7% of tutoring sessions. The During Tutoring Lesson subcategory with the lowest observed rating was echo of focus throughout the lesson observed to be above average in just 42.9% of tutoring sessions. For the After Tutoring Lesson, has high expectations for the child and articulates child's strengths and needs were observed to be above average in 78.6% of tutoring sessions and accurate and up-to-date records were observed to be above average in 71.4% of tutoring sessions. The Reading Recovery Program Strategies were observed in at least 85.7% of tutoring sessions. Observer Perceptions of Reading Recovery Program Implementation Site observers reported being impressed with the dedication and commitment of the Reading Recovery teachers to the fidelity of the teaching procedures and the integrity of the implementation of the program. Almost all teachers observed were meeting all the standards, guidelines, and expectations of the Reading Recovery Council of North America and the North American Trainers Group. Since observations occurred in March and April, most of the students observed were second-round students, since teachers had discontinued their first-round students. Teachers reported that students who did not discontinue from Reading Recovery during the first-round were being considered for further intervention services. Site observers also reported that the Reading Recovery program in the Little Rock district receives an adequate allocation of time, materials, and other resources. However, several teachers and principals expressed the need for additional Reading Recovery teachers in their schools. Reading Recovery teachers also reported teaching literacy small groups the rest of the day, which enables them to give their Reading Recovery students more attention during the instructional day. By teaching these literacy groups, site observers suggested that the Reading Recovery 5 teachers expertise and knowledge gained from their training and practice benefits children across several grade levels. During visits site observers suggested areas in which some Reading Recovery teachers needed improvement. These instructional areas included: (1) hearing and recording sounds in words
(2) making and breaking
(3) doing away with the helping hand
and (4) maintaining up-to-date records on each child as a basis for instruction. INTERVIEW RESULTS Reading Recovery Principal Interview 4 Principals at 10 of the Reading Recovery schools were randomly selected to participate in a 45-minute phone interview. Principals were asked a series of questions regarding general program implementation, classroom-level changes, program results, professional development opportunities and parental and community involvement. 81 Overall, principals were positive about the Reading Recovery program and the impact it has had on their schools. Most of the schools have been utilizing Reading Recovery for several years and faculty and staff are very comfortable with the program. Nearly all of the principals interviewed were instrumental in bringing Reading Recovery to their schools, and the decision to utilize Reading Recovery was made after considerable research and thoughtful consideration. Principals reported that Reading Recovery is a wonderful complement to the schools balanced literacy programs. The one-on-one attention the reading Recovery students receive is overwhelmingly the most effective of the strategies that the program employs. Other effective strategies mentioned included push-ins, literacy groups, running records, and the writing component. Principals reported being active advocates in their Reading Recovery programs. They described their roles as one of support and involvement, ranging from oversight of the program to more direct involvement including student selection for the program, review of student progress, and ongoing meetings and collaboration with the Reading Recovery specialists. 4 All principals noted that teachers were very supportive of the Reading Recovery program and appreciated the impact it has had on overall student achievement. Most of the resources needed for effective program implementation are available
however, principals reported an ongoing need for books, additional teachers and tutors, and more planning time. Principals described the African- American population as being well-served by Reading Recovery, and most agreed that through Reading Recovery the achievement gap is being bridged for their African-American students. In most of the schools, African-American students are a large percentage of the Reading Recovery program, and the early intervention provided by Reading Recovery allows the student to be encouraged by being successful at a younger age. Teacher In-Training Interview q In the spring 2005, four teachers in their first year in the Reading Recovery program were contacted by phone for a 30 to 45-minute phone interview. The teachers in-training were attending classes concerning Reading Recovery instruction, as well as working in their schools implementing the Reading Recovery strategies. Feedback from the teachers was solicited regarding general program information, classroom level changes, results, professional development, and parental involvement and support. All teachers described the process of integrating Reading Recovery into the schools literacy program as well planned and organized. Reading Recovery teachers work individually with the lowest performing students and then follow up individual instruction with literacy groups. Reading Recovery 6^1 teachers reported a thorough selection process that involved collaboration with the classroom teacher and comprehensive testing and assessment. In general, the Reading Recovery teachers reported strong support from the classroom teachers, that classroom teachers appreciated the effectiveness of the one-on-one approach, and the fact that Reading Recovery is able to provide this type of support, freeing the classroom teacher to work more effectively with the other students. The Reading Recovery teachers in-training strongly felt that Reading Recovery helped to equalize learning and achievement opportunities for African-American students at their schools. Most of the students in Reading Recovery are African-American, and these teachers were able to see positive gains. Reading Recovery helps all struggling readers by actively engaging them in reading and allowing them to feel successful at reading. The Reading Recovery teachers in-training reported a significant increase in self-confidence and improvement in overall attitude among the Reading Recovery students. As the students begin to experience success, they are less frustrated and angry. This improvement in attitude improves their classroom behavior and relationships with other teachers and students. Some of these teachers have seen dramatic increases in test scores and other achievement data, while others have not yet been able to see these gains. Reading Recovery teachers in-training reported professional training that ranged from adequate to thorough. Changes in Reading Recovery course instructors led to some confusion for some of the teachers in training, and some reported the need for increased classroom hours. However, overall, these teachers reported feeling well-prepared to work with the students. The strategies they have learned have been very helpful. Although the teachers in-training reported having significant classroom experience, they are learning unique techniques that have been very beneficial. Survey Results 4 Reading Recovery Teacher Questionnaire (RRTQ) Descriptive results. Reading Recovery teachers had extremely favorable attitudes toward the program at their schools. All of the teachers strongly agreed or agreed that they have a thorough understanding of the program, teachers in their school are generally supportive of the program, ongoing communication exists between Reading Recovery tutors and classroom reading teachers, Reading Recovery monthly meetings are effective and useful, instructional materials needed to implement the Reading Recovery program are readily available, and the Reading Recovery program is aligned with state and district reading and language arts standards. There was also strong agreement that Reading Recovery teachers received support, with 86.4% of the teachers reporting that the school administration and Reading Coach supported their efforts as a Reading Recovery teacher. Almost 75% (72.7%) of teachers reported receiving extensive district support. The items with the highest level of disagreement (disagree and strongly disagree) concerned Reading Recovery teachers having sufficient planning time (36.4%) and enough tutors to fully implement the Reading Recovery program (18.2%). Demographic data. All Reading Recovery teachers (100%) reported having at least six years of teaching experience and 31.8% reported at least six years experience as a Reading Recovery teacher. Approximately 85% (86.4%) of Reading Recovery teachers reported having a Masters Degree and beyond. The majority of Reading Recovery teachers were Caucasian (72.7%) and 13.6% reported their ethnicity as African-American. H Reading Recovery Classroom Teacher Questionnaire (RRCTQ) Descriptive results. For the classroom teachers, the three items on which 80% or higher agreed included: teacher support of the program (93.6%), positive impact on student achievement 1(87.8%), and improving achievement of African-American students (82.1%). The two items on which classroom teachers expressed the strongest disagreement or disagreement included: sufficient faculty and staff to fully implement the program (23.1%) and because of Reading Recovery, parents are more involved in the literacy program (14.7%). Demographic data. K-3 classroom teachers reported less teaching experience and education attainment than Reading Recovery teachers. Close to 30% (28.9%) reported less than one year to five years teaching experience in any school and 57.7% reported having a Bachelors degree. However, more classroom teachers reported their ethnicity as African-American (28.9%) than Reading Recovery teachers with the majority (68.6%) reporting ethnicity as Caucasian. Reading Recovery Parent Survey (RRPS) 4 Descriptive results. 4 Descriptive results. Generally, parents had favorable attitudes toward the Reading Recovery program. A majority of the parents (90.5%) reported that, because of Reading Recovery tutoring, they believed that their child would be successful in school and 86.3% reported that Reading Recovery had improved their child's reading skills. However, less than 75% (66.3%) of parents strongly agreed or agreed that they have many opportunities to talk with the Reading Recovery teacher about their childs progress. Demographic data. Almost 70% (68.4%) of parents reported the ethnicity of their child as African-American and 13.7% reported their childs ethnicity as Hispanic and 7.4% reported the ethnicity of their child as Caucasian. m Reading Recovery Level of Participation and Program Measures African-American students were in the majority in all of the 18 schools in the study. Not surprisingly, at 72.5% of the comparison student population and 75.0% of the Reading Recovery student population, African-American students were also a majority of the Reading Recovery students in this study. In 10 of the 18 schools, the percentage of African-American students in Reading Recovery exceeded their percentage of the comparison population. How meaningful this difference is may be debatable given instances where 100% of the Reading Recovery students are African-American in a school in which 98.3% of the comparison students are African-American (Watson Elementary) or where there are just 8 Reading Recovery students in a school with more than 100 first grade students (Terry Elementary School). Reading Recovery End of Program Status by Race A comparison of the total African-American Reading Recovery student population to the total other students involved in Reading Recovery indicates that the students were nearly equally represented in two of the three program specific measures. The percentage of African-American students Discontinued (43.3%) was not considerably different from the percent of other students Discontinued (46.3%). In addition, in the Incomplete status, the percentage of African-American students (21.3%) was not very much different than that of the other students (25.9%). Only in the Recommended status did the percentage of African-American students considerably exceed the percentage of other students (34.5% vs. 27.8%). fl Reading Recovery Year End Reading Group by Race With-in school comparisons are again difficult to make due to the unequal number of African- American students compared to other students participating in the program. On an overall basis however, the percentage of African-American students in the high/upper-middle group at 25.7% was 8 I fl fl fl fl lower than the percent of other students in this group (38.2%). In addition, almost 75% of the African-American students were in the low/lower-middle group compared to less than 66% of other students. STUDENT ACHIEVEMENT RESULTS DRA and Observation Survey: Reading Recovery Subtests. A total of 142 Reading Recovery students (66% pretest-posttest match rate) and 562 comparison students (65% rate) had matching 2005 DRA subtest scores, demographic information, and 2004 DRA/Observation Survey factor scores. Wilks lambda indicated significant multivariate effects for Reading Recovery status (Fa 1380 = 6.83, p < .001), special education status ( Ft.ego = 3.93, p <.01), and 2003-2004 DRA/Observation Survey factor scores (F4 ego = 111.32, p <.001). Follow-up univariate tests indicated significant Reading Recovery status effects on Hearing and Recording Sounds (^2,093 = 6.34. p <.01) and DRA scores (F2,693 = 9.99, p < 001). Post hoc analyses revealed that: (a) Reading Recovery students in both the Incomplete Program = 36.32
S =+0.43) and the Complete Program (M = 35.96
ES =+0.37) had a significantly higher adjusted mean Hearing and Recording Sounds score than students in the comparison condition (Ivf = 33.82)
and (b) students in the comparison condition //Vf = 17.64) and the Complete program 16.42
ES = -0.18) had a higher mean DRA score than students receiving the Incomplete Program (M = 13.02
ES = -0.68). No program by race interaction effect occurred, indicating that African-American and other students were equally affected by participation in Reading Recovery. DIBELS Subtests. A total of 67 Reading Recovery students (31%) and 53 comparison students (28%) had matching 2005 DIBELS subtest scores, demographic information, and 2003-2004 DIBELS factor scores. Wilks lambda indicated significant multivariate effects for Reading Recovery status (3212 = 4.12. p < .001). special education status ( E4.106 = 3.50. p <.01). and 2003-2004 DRA factor scores (4,106 = 3.69. p <.01). Follow-up univariate tests indicated significant Reading Recovery status effects on Phoneme Segmentation Fluency (2,109 = 3.39. p <.O5) and Oral Reading Fluency (2109 = 6.59, p < .01). Post hoc analyses revealed that: (a) Reading Recovery students in both the Incomplete Program (M = 54.81
ES = +0.65) and the Complete Program {M = 54.02
ES = +0.58) had a significantly higher adjusted mean Phoneme Segmentation Fluency score than students in the comparison condition (M = 47.23 )
(b) students in the comparison condition (M = 40.35) and the Complete program (M- 33.45) had a higher mean Oral Reading Fluency score than students receiving the Incomplete Program {M = 21.62
ES = -0.91)
and (c) students in the comparison condition had a higher adjusted mean Oral Reading Fluency score (A/f = 40.35) than students receiving the Complete program (Af = 33.45
ES =-0.33). No program by race interaction effect occurred, indicating that African-American and other students were equally affected by participation in Reading Recovery. ITBS Reading NCE. A total of 140 Reading Recovery students (65%) and 562 comparison students (65%) had matching 2004-2005 Reading Recovery subtest scores, demographic information, and 2003-2004 DRA factor scores. The ANCOVA indicated statistically significant effects for Reading Recovery status (2,591 = 6.62. p <.001). free lunch status (1,591 = 7.83. p <.01). and 2003-2004 DRA/Observation Survey factor scores (1,591 = 195.81. p <.001). No program by race interaction effect occurred, indicating that African-American and other students were equally affected by participation in Reading Recovery. Post hoc analyses showed that comparison students (M = 53.82) had a significantly higher adjusted mean ITBS Reading NCE score than students receiving the Complete program (AT = 46.65
ES = -0.34). The effect size for African-American students receiving a complete program was -0.46. versus 0.09 for those receiving an incomplete program. Exploratory and supplemental results. Exploratory analyses of second and third grade results showed no effects on 2004-2005 DRA and effect sizes ranged from -0.16 to -1.34 on ITBS for Reading Recovery students. These results need to be viewed with caution, however, due to low matching rates in second grade and the lack of a true pretest measure. There was no relationship between number of teacher years of experience with Reading Recovery and ITBS standardized 9n residuals or DRA standardized residuals for students receiving a complete program, after controlling for 2003-2004 DRA factor scores and student ethnicity, gender, free lunch status, special education status, and LEP status. Likewise, there was no relationship between number of sessions attended and ITBS residuals. A statistically significant, small positive relationship was observed between total number of sessions attended and DRA residuals (r = 0.21, p <.O5). The median effect size estimate across all posttests for African-American students with a complete program was Md = +0.17, with a range from -0.25 to +0.52. For African-American students receiving an incomplete program, effect size estimates ranged from -0.78 to +0.50, with a median of -0.23. Thus, receiving a complete program yielded a directional advantage for African-American students, whereas the reverse occurred for receiving an incomplete program. FINDINGS Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? 4 m The Reading Recovery program had equal effects on African-American and other students. Students receiving the complete program had significantly higher adjusted means than comparison students on Phoneme Segmentation (ES = +0.58) and Hearing and Recording Sounds in Words (ES = +0.36) Students receiving an incomplete program had significantly higher adjusted means than comparison students on Phoneme Segmentation (ES = +0.65). Students in the comparison condition had significantly higher adjusted means than students receiving a complete program on Oral Reading Fluency (ES = -0.33) and on ITBS Reading NCE scores (ES = -0.34). Students in the comparison condition had significantly higher adjusted means than students receiving an incomplete program on DRA test scores (ES = -0.68) and Oral Reading Fluency (ES = -0.91). No relationship was observed between teacher experience with Reading Recovery and 2004- 2005 student achievement outcomes after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. No relationship was observed between number of Reading Recovery sessions and 2004- 2005 ITBS Reading NCE scores, after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. A small but statistically significant positive relationship was observed between number of Reading Recovery sessions and 2004-2005 DRA test scores, after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. The median effect size estimate across all posttests for African-American students receiving a complete program was +0.16
for students receiving an incomplete program, the median effect size estimate was -0.09. Positive effects of Reading Recovery tended to be associated with lower-order or beginning reading skills like phoneme segmentation and hearing and recording sounds, while less positive effects tended to be associated with more complex, higher-order skills like Oral Reading Fluency and DRA scores. n What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? Classroom observations indicate that Reading Recovery teachers instructional practices conform to the recommendations and requirements of the program throughout the district. Given that there are no national comparisons or benchmarks for the RRIAl, a mean of approximately 3.50 on a 4.00 scale suggests a high level of Reading Recovery implementation in the district. Site researchers noted only three areas in which some teachers were observed below average to some degree, reading familiar stories, appropriate pacing of the lesson components, and working 10 with letters and or/words. However, the observed lack of quality implementation in some classrooms in reading familiar stories and appropriate pacing of the lesson components might begin to explain the lack of oral reading fluency, text reading, and ITBS effects for Reading Recovery students. There appears to be a high level of consistency of program delivery across the district. This suggests that generally teachers have a high degree of fidelity to the model. In addition, the student achievement analysis found that there was no relationship between teacher experience with Reading Recovery and 2005 achievement scores after controlling for 2004 achievement and other variables. What is the level of participation in Reading Recovery by African-American students relative to other ethnic groups at the school? The data indicate that African-American students made up a majority of the students participating in Reading Recovery in the 18 schools included in this study. This finding shouldnt be surprising since African-Americans are the majority of the students in each of the participating schools. Information compiled from the student achievement analyses also indicates the percentage of African-American students receiving Reading Recovery services (75.2%) is very similar to the ethnic makeup of the students used for comparison purposes (73.6%). However, Reading Recovery students were more likely to receive free or reduced-price lunch (84.3% vs. 73.1%) and special education services (15.7% vs. 8.6%) than comparison students, and were more likely to be male (58.1% vs. 48.0%). What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? Table 11 indicates that African-American students, when compared with Reading Recovery students of other ethnicities, were nearly equally represented in two of the three program specific measures examined. The percentage of African-American students Discontinued at 43.3%, was not considerably different than the percentage of students Discontinued of other ethnic backgrounds, 46.3%. In addition, in the Incomplete status, the percentage of African-Americans (21.3%) was, again, not much different than students from other ethnicities (25.9%). African- American students, at 34.5%, were more likely to be Recommended for further actions than other students (27.8%). 4 African-American students were, however, more likely to be in the Low/Lower Middle reading group at the end of the school year than other students (74.3% vs. 61.8%). This finding may present a dilemma for the program and the district. While African-American students are generally progressing similarly to other students on program-specific measures, at the end of the school year, the majority of African-American students are still struggling to maintain or falling below grade level in reading. About 22% of scheduled sessions were missed due to the teacher being unavailable (7%), student absence (6%), teacher absence (5%), or the student being unavailable (4%). These missed sessions could contribute to the mean number of sessions per week being 3.5. What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? Reading Recovery teachers had extremely favorable attitudes toward the Reading Recovery program. Teachers reported a thorough understanding of the program, that they received adequate professional development which was valuable for improving the achievement of African- American students, and that they had the support from teachers in the school. The majority of Reading Recovery teachers also reported receiving extensive administrative, Reading Coach, 114 - V and district support. The items and areas of most concern were sufficient planning time, enough tutors to fully implement the program, and time to routinely monitor first grade students progress after they were discontinued from Reading Recovery tutoring. Additionally, Reading Recovery teachers indicated that only 63.6% of faculty, staff, and administration believe that all children can read at grade level or above by the end of third grade and that parents are more involved in the literacy program of this school as a result of the program. Reading Recovery teachers, on average, appear to be more experienced and better educated. Eighty percent had a Masters degree or beyond in educational attainment and 100% reported at least six year or more years of teaching experience. In addition, the majority (68.18%) reported one to five years of experience as a Reading Recovery teacher. The four teachers in-training were equally committed and positive about the program and overall, felt they were well prepared to work with students. The teachers in-training also felt strongly that Reading Recovery helped to equalize learning and achievement opportunities for African-American students. Teachers in-training emphasized the importance of using data to monitor the progress of the students to develop effective teaching strategies based on the individual needs of each student. Finally, teachers in-training also reported the need for more time to plan and implement as well as for continuing support to understand Reading Recovery components more thoroughly. What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? A review of Reading Recovery principal interview responses indicates that principals are very supportive and actively involved in the program. All of the principals interviewed reported that they understood the program and were advocates of their program having a positive impact on overall student achievement. Principals indicated that the one-on-one tutoring program supplements and enhances the schools balanced literacy program. Most principals agreed that, through Reading Recovery, the achievement gap is being bridged for their African-American students. Principals also noted that teachers were very supportive of the program. K-3 classroom teachers shared the principals enthusiasm for the Reading Recovery program, as evidenced by responses on the RRCTQ. A majority of the classroom teachers reported that they had an understanding of the program, were generally supportive, and that student achievement had been positively impacted. Principals and teachers also agreed that most of the resources and support needed for effective program implementation was available
however, they also reported an ongoing need for additional teachers and tutors to support more students and time to plan, review student progress, and collaborate together. All (102) of the teachers responding agreed that their school should continue the Reading Recovery program. What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? Parents were generally very pleased with the results of the Reading Recovery program. Approximately 90% of the parents responding to the parent survey agreed that: Reading Recovery tutoring had improved their childs reading skills and because of Reading Recovery their child will be successful in school. Less than three percent of those who responded reported that they did not know or understand the program. In the three open-ended responses parents indicated a very good understanding of the program, appreciation of one-on-one tutoring sessions, and the improvement in their childs reading skills. However, a few parents did express the need for longer and more frequent tutoring sessions and more opportunities to talk with the Reading Recovery teacher about their childs progress. 12 RECOMMENDED PROGRAM MODIFICATIONS In summary, the Little Rock School District has a strong Reading Recovery program implementation. Teachers, principals, and parents appear to be actively engaged in the program and the district tries to provide adequate levels of resources and support. However, the lack of clear program effects may be the result of factors that have been identified in prior studies of Reading Recovery. Possible factors and recommended program modifications to produce greater achievement gains include: It would be expected that Reading Recovery students would perform better on assessments more closely aligned with the instructional program (DRA, DIBELS, Observation Survey) than the norm-referenced, group-administered ITBS. In m particular, Reading Recovery enhanced learning for complete program students tests involving Phoneme Segmentation and Hearing and Recording Sounds in Words, However, it had less positive effects on tests assessing Oral Reading Fluency and ITBS Reading NCE scores. Reading Recovery is also most likely serving as a first line of defense for students who many later be referred for special education services. While classroom teachers are receiving help for their neediest students, this may put an extra burden on the program that was designed to help students who could benefit quickly from quality instruction in 20 weeks. w The district should examine the feasibility of expanding the program to provide tutoring support to all incoming first grade students who need services. In the 2004- 2005 school year, the 18 Reading Recovery schools indicated that 365 students needed tutoring services and approximately half of this number received a complete round of lessons and were discontinued. The Reading Recovery program guidelines state that if a school has more children who need services than one teacher can provide, then it will never realize the full benefit of Reading Recovery for later school achievement. It is especially difficult for classroom teachers to continue to scaffold discontinued students learning while supporting a large number of other at-risk students reading below grade level. q q q In addition to an expanded program, a transitional plan for students who have discontinued should be explored. The data suggest that after Reading Recovery students are discontinued and return to the classroom at the same reading level as their peers, they may not maintain the same growth rate and achievement does not keep pace with their peers. Although research indicates that former Reading Recovery students perform well in their classes, some slippage in achievement can occur (Clay, 1993). Although Clay (1993) provides guidelines for transition back to the classroom after the student is discontinued
it is possible that students were returned to the classroom without benefit of a transition plan. As noted by Reading Recovery teachers, few teachers have the opportunity to routinely monitor discontinued students progress. Such a plan could involve daily monitored reading RI that would provide another buffer against slippage. Also in tutoring sessions, children have opportunities to read texts at their instruction level on a daily basis, but they may not have adequate time for daily reading in the regular classroom. q q The quality of instruction that Reading Recovery students receive once they return to the classroom is an important factor that was not examined in this study. Increased professional development of classroom teachers would enable them to understand how to integrate their Reading Recovery students back into the classroom once they have been discontinued, and how to provide the appropriate instruction and feed back so that students would continue to improve. 13m9 4 Bl Increased partnership with UALR to help with the development of a transition plan and professional development for classroom teachers would seem to be warranted given these study results. Additional research could provide critical insight into the optimum classroom environment for discontinued, recommended, and incomplete Reading Recovery students. Further studies might also provide a more in-depth analysis of a small number of students whose gains were maintained to determine what factors contributed to their success and how these factors can be generalized to all Reading Recovery schools. EXPECTATIONS OF PROGRAM MODIFICATIONS Reading Recovery has valuable components that, with adaptation and modification, can be even more effective. With the recommended program modifications, the Little Rock School District could expect: Progressive gains on standardized test scores for African-American students over time. An increased number of students involved in the Reading Recovery program. A greater adherence to Reading Recovery guidelines, especially those relating to transition services and the number of sessions required for optimum benefits. More teachers throughout the district better able to serve students at-risk in the areas of literacy and reading. Sustained achievement of students upon completion of the Reading Recovery program. A stronger relationship with UALR professionals that would continue to provide the Little Rock School District with the most up-to-date research findings and best instructional practices for reading and literacy instruction. 14 4 4 EVALUATION OF READING RECOVERY IN THE LITTLE ROCK SCHOOL DISTRICT: 2005 AGGREGATE REPORT INTRODUCTION The Little Rock School District began implementing Reading Recovery during the 1995-1996 school year in two schools. During the first four years of implementation, the Little Rock School District was part of the Pulaski County Reading Recovery Site. By 1998 there were eight trained Reading Recovery teachers in seven schools in the Little Rock School District. In July 1999 the district became a Reading Recovery Site hiring a full-time teacher leader. In 2000-2001 the University of Arkansas at Little Rock (UALR) began conducting the Reading Recovery teacher training for the district. Between the 2000-2001 and 2004-2005 school years UALR trained 17 Reading Recovery teachers for the district. At the end of the 2004-2005 school year the Little Rock School District had 28 trained Reading Recovery teachers serving 18 of the 34 elementary schools in the district. Reading Recovery is one of eight literacy programs, interventions, and/or models used by Little Rock schools. Currently, Little Rock School District funds are used to support the program. The goal of Reading Recovery is to dramatically reduce the number of first grade students who have difficulty learning to read and write and to reduce the cost of these learners to educational systems. Reading Recovery is a short-term intervention program of one-to-one tutoring for the lowest-achieving first graders. The intervention is most effective when it is available to all students who need it and is used as a supplement to good classroom teaching. Individual students receive a half-hour lesson each school day for 12 to 20 weeks with a specially trained Reading Recovery teacher. As soon as students can read within the average range of their class and demonstrate that they can continue to achieve, their lessons are discontinued and new students begin individual instruction. Reading Recovery was developed by New Zealand educator and researcher Dr. Marie M. Clay over 20 years ago. More than one million first graders have been served in 49 states since Reading Recovery was introduced in the United States in 1984. Professional development is an 15 essential component of the Reading Recovery program. Training utilizes a three-tiered approach that includes teachers, teacher leaders, and university trainers. In schools, special trained teachers work with children. At the site level, teacher leaders work with children, train teachers, and assist and monitor implementation with the help of a site coordinator. In university training centers, trainers work with children, train teacher leaders, engage in research, and support program implementation at affiliated sites. Professional development for teachers and teacher leaders begins with year-long graduate level study and is followed by ongoing training in succeeding years. Since 1984, the program reports that 80% of students who completed the full 12 to 20 week series of lessons, and 59% of all students who have any lessons in Reading Recovery, can read and write with the average range of performance of their class. Program follow-up studies also indicate 4 that most Reading Recovery students also do well on standardized tests and maintain their gains in later years. The evaluation plan for the Reading Recovery program in Little Rock School District included: (1) analyses of Reading Recovery student achievement and program data, (2) principal, teacher, and 4 parent surveys and interviews, and (3) observations of Reading Recovery tutoring sessions. This 4 report is part of a larger district study of four programs evaluating the effectiveness in improving and remediating the academic achievement of African-American students. This report includes results from 18 elementary schools participating in the Reading Recovery program. RESEACH QUESTIONS The major goals of this research study were to evaluate African-American student achievement outcomes, program implementation fidelity, and principal, teacher, and parent perceptions concerning the Reading Recovery tutoring program for first grade students. Student achievement results on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment a (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement were analyzed to compare the progress of first graders enrolled in the Reading a Recovery intervention program and comparison students in 2004-2005. Program implementation ratings were obtained from observations of 14 tutoring sessions in nine schools. The survey and a 16interview results are based on the perceptions of 156 classroom teachers in grades K-3, 22 experienced Reading Recovery teachers, four teachers in-training, 10 principals, and 95 parents. The Reading Recovery evaluation was structured around the following seven primary and supplemental research questions
Primary evaluation question: Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? Supplemental (Qualitative/Step 2) evaluation questions: What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? q What is the level of participation in Reading Recovery by African-American students relative to other ethnic groups at the school? What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? 17EVALUATION DESIGN AND MEASURES The evaluation period extended from February 2005 through May 2005. The evaluation design was based on both quantitative and qualitative data collected from Reading Recovery intervention observations, principal and teacher in-training interviews, classroom teachers, Reading Recovery teachers, parent surveys, and Reading Recovery program data. Reading Recovery student-level achievement data on the ITBS, DRA, DIBELS, and An Observation of Early Literacy Achievement was received in fall 2005 and incorporated in this report. The primary data collectors in this study were Center for Research in Educational Policy (CREP) trained site researchers. Site researchers (1) conducted Reading Recovery intervention observations, (2) administered the Reading Recovery teacher survey, (3) conducted principal and teacher in-training interviews, and (4) collected program data. Principals at Reading Recovery schools were responsible for administering the classroom teacher and Reading Recovery parent surveys. Participants Little Rock School District is located in Central Arkansas and serves approximately 26,500 students, with African-Americans representing approximately 67% of the district student population, in 49 schools in an urban area with a population of 184,000. In the 2004-2005 school year 18 elementary schools and 230 first grade students, of which 173 are African-American, 27 are Caucasian, 22 are Hispanic, and the remaining 8 are other ethnicities, participated in the Reading Recovery program in the district. However, the schools indicated that 365 first grade students needed Reading Recovery services. Three schools were in their first year of program implementation (Bale, 1 Stephens, and Terry) and survey and observation data were not collected from these schools. However, randomly selected teachers in-training and principals from these three schools were interviewed for this study. A profile of the Reading Recovery schools and participants included in this study is shown in Table 1. The profile data were obtained from either the 2003-2004 Common Core of Data from the National Center for Education Statistics, the 2004-2005 Reading Recovery Site Report for Little Rock from the National Data Evaluation Center at The Ohio State University, or provided by the district. As indicated in Table 1, the number of years schools have implemented the 18 Reading Recovery program ranged from one to 10. The Reading Recovery schools were predominately African-American, ranging from 50% of the student population to 99%. The district reported that four Reading Recovery schools did not receive Title I funding and the percentage of students eligible for free and reduced price lunch ranged from a low of 33% to a high of 94% at all Reading Recovery schools. Table 1 Reading Recovery Participating Schools: 2004-2005 School Name Students Teachets Aftican- Ametican Bale Elementary School 319 27 87% School Wide Populalion Asian 0% Hispanic Caucasian 7% 5% %Freeand Reduced Lurch 88.4% ReaiiTgRecowiyParticiMnlWaTOlion % Below Proficient* Number of RR Teachets Number of K-3 Teachers Nutter of RR Yeats n RR Students Piogram 45% 10 Bi Booker Arts Magnet ES 605 55 55% 1% 4% 40% 63.3% 22% 20 32 Bl Carver Magnet Elem School Chicot Elem School David ODodd El School Franklin Incentive Elem School Geyer Springs El School 496 536 261 387 299 43 44 27 35 23 54% 75% 58% 97% 89% 3% 3% 40% 53.0% 18% 16 16 0% 0% 0% 0% 15% 10% 86.6% 38% 17 26 10% 32% 68.9% 30% 10 0% 2% 943% 51% 12 18 4% 7% 806% 48% Bl Gibbs Magnet El School Meadowdiff Elem School Mitchell Incentive Elem School Otter Creek El School Rightsell Incentive Elem School Stephens Elem School 310 349 156 511 262 499 30 24 22 31 26 39 52% 77% 98% 56% 99% 95% 4% 3% 41% 43.9% 11% 14 1% 9% 13% 85.1% 44% 0% 0% 2% 91.7% 59% 1% 6% 36% 55.7% 19% 14 10 0% 0% 1% 876% 49% 1% 3% 2% 90.6% 59% 18 B! Terry Elem School 577 36 51% 7% 5% 36% 475% 12% 18 Wakefield Elem School 451 29 83% 0% 15% 2% 920% 33% 13 18 Watson Elem School 456 34 95% 0% 3% 2% 932% 64% 15 10 Williams Magnet Elem School 461 36 52% 9% 1% 38% 33.6% 10% 13 13 Wilson Elem School 285 27 89% 1% 4% 6% 91.9% 36% H 4 * Proficiency levels are based on 2003-2004 school year ACTAAP Grade 4 Reading, Language, and Writing data. 1 8 1 4 7 2 3 3 5 1 8 9 2 7 1 8 8 7 2 8 6 1 1 1 1 1 1 2 1 2 1 9 5 8 8 8 8 8 9 9 8 8 3 2 3 1 1 3 3 9 5 4 19Instrumentation Six instruments were developed by CREP to collect the evaluation data: a classroom observation tool, a principal interview, a teacher in-training interview, a Reading Recovery teacher questionnaire, a classroom teacher questionnaire, and a parent survey. A detailed description of each instrument follows. Classroom Observation Measure Reading Recovery Implementation Assessment Instrument (RRIAl) 4 The RRIAl was designed for observation in Reading Recovery classrooms and was developed by researchers at CREP and Reading Recovery faculty at UALR. Ratings are organized Ql around two categories: Reading Recovery Program Components and Reading Recovery Program Strategies. The RRIAl observation tool provides an overall rating for each category based on a rubric that ranges from (1) poor or unacceptable
(2) below average in comparison to other programs observed
(3) meets nearly all standards of program quality
and (4) above average in comparison to other programs. The RRIAl has been aligned to the essential components of the Reading Recovery program. Sub-categories of the program components include: reading familiar stories, reading a story that was read for the first time the day before, working with letters, writing a story, assembling a cut-up story, and introducing and reading a new book. The tool was designed to be utilized by experienced Reading Recovery trainers during the 30-minute tutoring session with additional time allocated for observer questions and examination of student records after the session. Site observers received observation protocol training in February 2005 and a copy of the observation guidelines are included in Appendix A. Site observers were also asked to provide overall perceptions of Reading Recovery program implementation in the Little Rock schools. Interviews Reading Recovery Principal Interview Randomly selected principals from 10 Reading Recovery schools participated in approximately 45-minute phone interviews with CREP researchers. Interview questions addressed 20 the principals general experiences and reactions to the Reading Recovery implementation and the associated outcomes for the school, students, teachers, and parents. A copy of the principal interview protocol with summary responses is included in Appendix B. Reading Recovery Teacher in-Training Interview In the 2004-2005 school year, six teachers were beginning their first year as a Reading Recovery teacher and considered as teachers in-training. Three were located in experienced schools and three were located in the schools beginning their first year of implementation: Bale, Stephens, and Terry. Four randomly selected teachers in-training participated in an approximately 45-minute phone interview with CREP researchers. Interview questions addressed the teachers in-training experiences with and perceptions of the Reading Recovery program implementation with regard to such areas as q resources, professional development, parent involvement, support, and student outcomes. A copy of the teacher in-training interview protocol with summary responses is included in Appendix C. Surveys Reading Recovery Classroom Teacher Questionnaire (RRCTQ) 4 Reading Recovery teachers in the 15 experienced Reading Recovery schools were asked to complete the RRTQ, which includes four sections. Section I contains 20 items to which teachers respond using a five-point Likert-type scale ranging from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the general impressions of the Reading Recovery program, professional I development, support for the program, impacts on student achievement, and alignment of state and district reading and language arts standards. In a second section, teachers were asked to indicate to what degree the listed items occurred. The focus of the four items were (1) administration support of Reading Recovery teacher efforts
(2) Reading Coach support of Reading Recovery teacher efforts
(3) district support of Reading Recovery teacher efforts
and (4) the time to routinely monitor first n grade students progress after they were discontinued from tutoring. Open-ended questions asked Reading Recovery teachers to respond to the statements: Whaf are the strongest aspects of the Reading Recovery program? What are the weakest aspects of the Reading Recovery program? Do 21 you think your school should continue the Reading Recovery program? A final section of the RRTQ contained general questions regarding years of teaching experience, race, and level of education. Reading Recovery Classroom Teacher Questionnaire (RRCTQ) All K-3 teachers who taught in experienced Reading Recovery schools were asked to complete the RRCTQ, which contains 13 items to which teachers respond using a five-point Likert- type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the specific program elements of Reading Recovery such as: understanding of the program, impacts on student achievement, teacher support for the program, and parent involvement. Open-ended questions asked classroom teachers to respond to the statements: Please describe your understanding of the Reading Recovery program in your school. What are the strongest aspects of the Reading Recovery Bl program? Do you think your school should continue the Reading Recovery program? final section of the RRCTQ contained general questions regarding years of teaching experience, race, and level of 4 education. Reading Recovery Parent Survey (RRPS) Parents whose first grade children were currently receiving Reading Recovery tutoring services were asked to complete the RRPS, that contains six items to which parents respond using a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed general impressions of the program such as: improvement in childs reading skills, childs enjoyment of tutoring sessions, and opportunities to communicate with the Reading Recovery teacher. Both English and Spanish versions were made available to schools. Open-ended questions asked parents to respond to the statements: Please describe your understanding of the Reading Recovery tutoring program at your childs school. What is the BEST thing about your childs involvement with the Reading Recovery tutoring program? What CHANGES would you like to see in the Reading Recovery tutoring program? A final section of the RRPS asked parents to indicate the race, grade, and age of their child. 22Program Data Reading Recovery program information was obtained from data submitted to the National Data Evaluation Center at The Ohio State University in 2004-2005. Each year the schools and district submit teacher, school, and student program data and receive a site report. The report represents an examination of Reading Recovery student outcomes for Little Rock and accounts for all children served by Reading Recovery within the site during the 2004-2005 school year. In addition, attention is given to implementation factors that may be supporting or hindering the success of the intervention at the site. Student Achievement Results ^1 In addition to the program data, interviews, survey, and observation tools cited above, reading achievement data are derived from scores on the Iowa Tests of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement. A description of each assessment follows. Iowa Tests of Basic Skills (ITBS). The ITBS Form A is a norm-referenced group administered test that measures the skills and achievement of students in grades K-8 and was developed at The University of Iowa. The ITBS provides an in-depth assessment of students achievement of important educational objectives. Tests in Reading, Language Arts, Mathematics, Social Studies, Science, and Source of Information yield reliable and comprehensive information both about the development of students skills and about their ability to think critically. The emphasis of the K-1 assessment is on academic skills found in the early childhood curriculum. These tests are neither measures of readiness for school nor readiness to read. Rather, they assess the extent to which a child is cognitively prepared to begin work in the academic aspects of the curriculum. Test materials have been extensively field tested for psychometric soundness and evaluated for fairness to gender. racial, ethnic, and cultural groups. Developmental Reading Assessment (DRA). The Developmental Reading Assessment provides teachers with a method for assessing and documenting primary students development as readers over time with a literature-based instructional reading program. The DRA is designed to be 23used in K-3 classrooms with rich literacy environments. The assessments are conducted during one- on-one reading conferences as children read specially selected assessment texts. A set of 20 stories, which increase in difficulty, are used for the assessment. The DRA evaluates two major aspects of reading
accuracy of oral reading and comprehension through reading and retelling of narrative stories. Questions pertaining to concepts about print are also included in the assessment with lower leveled texts. Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS is a tool for early identification of children with potential literacy problems and an assessment of response to instruction. The assessment is individually administered to K-3 children at least three times per year. The DIBELS assessment is designed to enable educators to modify their approach if a student is not on course to achieve reading goals. The Early Childhood Research Institute on Measuring Growth and Development at the University of Oregon constructed DIBELS (2000). The Institute reports it has validated the instruments ability to predict outcomes and has tested its reliability with young children across the country. Reading Recovery schools that have received Reading Excellence Act or Reading First grant funding use the DIBELS assessment in Little Rock. An Observation of Early Literacy Achievement: Reading Recovery Subtests. Six tasks in Marie Clays (2002) An Observation of Early Literacy Achievement (Observation Survey) were used as pretest and posttest measures. The Survey tasks have the qualities of sound assessment instruments with established reliabilities and validities. The purpose of the assessment is to determine 4 r an appropriate level of text difficulty and to record, using a running record, what the child does when reading continuous text. All six tasks of the Observation Survey were administered to Reading Recovery students in the fall (start of the school year) and/or at entry to the intervention. These scores served as pretest measures in the evaluation design. The six tasks were also administered to Reading Recovery students upon discontinuing or exiting from the program. In the spring (end of school year), the six tasks were again administered to all students who received Reading Recovery services during the year. Year-end scores served as the posttest measures in comparing the progress made by Reading Recovery children in the various end-of-program status groups. 24PROCEDURE Data for the evaluation were collected March-May 2005 for the 2004-2005 school year. On February 16, 2005 principals were given an overview of the evaluation and timelines for collecting data. On March 3-4 and April 14-15, 14 tutoring observations were conducted in nine randomly selected experienced Reading Recovery schools by two Reading Recovery content experts from Georgia State and The Ohio State Universities. Only experienced Reading Recovery teachers were observed. On March 23 a CREP trained site researcher administered the Reading Recovery teacher survey to experienced teachers at a regularly scheduled monthly meeting at UALR. In April and May principals in experienced Reading Recovery schools administered the classroom teacher q questionnaire to K-3 grade teachers. Principals also administered the parent survey to parents whose children were currently receiving first grade intervention services in the 15 experienced Reading Recovery schools. Four teachers in-training and 10 principals were randomly selected from the 18 Reading Recovery schools to participate in a phone interview in April and May conducted by CREP researchers. Reading Recovery program data were received in summer 2005 from the National Data Evaluation Center at The Ohio State University. Student achievement data were received from the district in fall 2005. METHODS - STUDENT ACHIEVEMENT Sample. The sample included 1,094 first grade students who attended one of 18 schools that implemented the Reading Recovery program during the 2004-2005 school year. Of these, 230 q were referred to the Reading Recovery program, and 864 were in the comparison group. The percentages of students who were African-American or of Limited English Proficiency were similar q between the comparison and Reading Recovery groups (73.6% versus 75.6% and 7.3% versus 8.7%, respectively
see Figure 1). However, Reading Recovery students were more likely to be eligible for q free or reduced-price lunch (84.3% versus 73.1%), to receive special education services (15.7% versus 8.6%), and to be male (58.1% versus 48.0%
see Figure 1). A random sample of about 11 comparison students was selected from each school (total n = 189) for administration of the DIBELS and Observation Survey: Reading Recovery test batteries. 25I Figure 1. Percentage of Students with Selected Demographic Characteristics by Reading Recovery Status 100.0 90.0------- 84.3 80.0 70.0 - 60.0 - 50.0 - 40.0 - 30.0 20.0 10.0 0.0 73.6 75.6 ft l?>( if African American 73.1 15.7 p. 7.3 8-7 8.6p^ 48.0 58^1 S Free lunch LEP Special Education Male o 7- ?? B Comparison Reading Recovery a Reading Recovery Treatment Level. In the Reading Recovery program, students who are deemed to have attained a reading level equivalent to their peers are assigned an end-of-program status of discontinued. Children who received 20 or more weeks of Reading Recovery services, but who do not attain a reading level equivalent to their peers are assigned a status of recommended action. Children who have received fewer than 20 weeks were assigned incomplete program status. Children designated as unknown status were removed from the program in fewer than 20 weeks due to reasons other than the school year ending. Other children were designated as having moved during the school year. As shown in Figure 2, the median number of sessions of recommended action {Md = 68.50
n = 68) children was actually higher than the median for discontinued students {Md = 57.00
n = 90). The medians for incomplete students {n = 46) and unknown students (n = 12) were very similar {Md = 44.63 versus 46.08, respectively
see Figure 2). a For the purposes of this study, recommended action and discontinued students were categorized as complete program
incomplete and unknown students were categorized as incomplete 26program
" and students who moved were eliminated from the analyses. This left a total of 216 Reading Recovery students: 158 with a complete program, and 58 with an incomplete program. Figure 2. Boxplot of Number of Reading Recovery Sessions by Student Status Upon Exiting the Program 80- (A C o w (A Q s Ct 40- M o o I- 20- Discontinued T T T Incomplete Moved Unknown Recommended action o o o T * T Status Note. Heavy dark lines indicate median. Gray boxes indicate interquartile range. 'Whiskers indicate range, excepting extreme values. Extreme values denoted by circles or asterisks. n 21 Measures. Pretest (covariate) measures included: (a) Spring. 2004 Developmental Reading Assessment (DRA) scores, (b) Fall 2005 Observation Survey: Reading Recovery program subtests that included Letter Identification, Word Test, Concepts About Print, Writing Vocabulary, Hearing and 4 Recording Sounds in Words, and Text Reading
and (c) the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in Letter Naming Fluency. Phoneme Segmentation Fluency, Nonsense Word Fluency, and Word Use Fluency. Outcome measures included Iowa Test of Basic Skills (ITBS) Reading Normal Curve Equivalent Scores. DRA scores. Observation Survey: Reading Recovery subtests, and DIBELS subtests administered in Spring 2005. To achieve some parsimony in the analyses, pretest measures were subjected to a principal components analysis, and regressionbased factors scores were constructed for pretest DRA and Observation Survey: Reading Recovery 4 4 subtests, and the DIBELS subtests. A single factor accounted for 60.1% and 60.5% of the variance in the DRA and Observation Survey: Reading Recovery subtests, and the DIBELS subtests, respectively. Outcome measures included the DRA and Observation Survey: Reading Recovery. DIBELS subtests, and ITBS Reading. Student achievement analyses. A 3 (Program Status) by 2 (African-American versus other) multivariate analysis of covariance (MANCOVA) was performed on two test batteries, using student gender, the pertinent pretest factor score, free lunch status, special education status, and LEP status as covariates. One test battery included the DRA and subscales of the Observation Survey: Reading Recovery. The second test battery included subtests for the DIBELS. A similar 3 by 2 analysis of covariance was performed on ITBS Reading NCE scores using the 2003-2004 DRA score as a pretest covariate. For the multivariate analyses, Wilks lambda was used as the criterion of multivariate significance. Where Wilks lambda indicated a significant multivariate effect, follow-up univariate analyses were performed on each outcome variable using the Bonferroni procedure to control for experimentwise alpha. When significant univariate results were found, post hoc analyses were performed using Scheffes procedure. Effect size estimates were computed for all posttests by subtracting the adjusted mean for the comparison group from the adjusted mean from the Reading Recovery group within levels of race, then dividing by the total standard deviation of the posttest (Cohen. 1988: Hedges & Olkin. 1985). For ITBS Reading NCE effect size estimates, the population 28 standard deviation of 21.06 was used. The effect size estimates (ES) represent the standardized difference between treatment and comparison group means, which allows for comparison across measures that have different metrics. Exploratory and supplementary analyses: Impact of teacher experience and number of sess/ons. The number of years experience with Reading Recovery was recorded for each teacher. Years experience ranged from 1 to 10 years. ITBS Reading NCE scores and DRA test scores for 2004-2005 were regressed on 2003-2004 DRA/RR Observation Survey factor scores and dummy-coded variables representing student ethnicity, free lunch status, gender, special education status. and LEP status. Standardized residuals were saved from each of these analyses and plotted against teacher years of experience to graphically assess the nature of the relationship between teacher experience and teacher effectiveness in Reading Recovery for students who received a Complete program. Standardized residuals for each of these 2004-2005 tests were also plotted against number of Reading Recovery sessions received for all students (i.e., those who received either a Complete or Incomplete program). DATA COLLECTION Table 2 provides the type of measures, instrument names, administration timeline, and a brief data collection description for each of the instruments. 29 4 Table 2 Data Collection Summary t-fl Type of Measure Site Visits Surveys Interviews Data Analysis and Reporting Instrument RR Implementation Assessment Instrument RR Teacher Questionnaire RR Classroom Teacher Questionnaire RR Parent Survey Principal Interviews Teacher In-Training Interviews ITBS, DRA. DIBELS, & Observation Survey
RR Little Rock School District Reading Recovery Aggregate Report Timeline Spring 2005 Spring 2005 Spring 2005 Spring 2005 Spring 2005 Spring 2005 Fall 2005 Description/ _____ Response rate 14 conducted 22 respondents/100% 156 respondents/ approximately 90% 95 respondents/9 Spanish approximately 86%_____ 10 conducted 4 conducted 1,094 first grade students . 230 RR 864 comparison group 1 Final Report RESULTS The results of the Reading Recovery evaluation are presented below by instrument. A copy of each instrument is included in Appendix I. In the Conclusion section, findings are synthesized across instruments to address each research question. J -i Reading Recovery Implementation Assessment Instrument (RRIAl) As indicated in the description of the RRIAl, the observation procedure primarily focuses on Reading Recovery Program Components and Program Strategies. The site observers used a four-point rubric (1 = poor or unacceptable, 2 = below average in comparison to other programs observed. 3 = meets nearly all standards of program quality, and 4 = above average in comparison to other programs) to rate the frequency and application of components and strategies of Reading Recovery instruction. More precise data regarding observed Reading Recovery instructional practices measured by the four-point rubric for the 14 observations are presented in Table 3. 30 Reading Recovery Program Components The overall mean rating for the Reading Recovery Program Components was 3.46 which suggests a high level of program implementation. Of the six subcategories the highest observed ratings were
assembiing a cut-up story and introducing and reading a new book observed to be above average in 85.7% of tutoring sessions. The program component subcategory with the lowest observed rating was working with tetters and/or words using magnetic tetters observed to be above average in only 35.7% of tutoring sessions. Reading a famitiar story and reading a story that was read for the first time the day before was observed to above average in 64.3% of visits. The six program components were observed in at least 92.9% of 14 tutoring sessions. Reading Recovery Program Strategies Cl The overall mean rating for the Reading Recovery Program Strategies was 3.61 which also indicates a high level of instructional effectiveness. Of the eight During Tutoring Lesson subcategories the highest observed rating was for appropriate text setected throughout the tesson 4 observed to above average in 85.7% of tutoring sessions. The During Tutoring Lesson subcategory with the lowest observed rating was echo of focus throughout the tesson observed to be above average in just 42.9% of tutoring sessions. For the After Tutoring Lesson, has high expectations for the chiid and articutates chitds strengths and needs were observed to be above average in 78.6% of tutoring sessions and accurate and up-to-date records were observed to be above average in 71.4% of tutoring sessions. The Reading Recovery Program Strategies were observed in at least 85.7 of tutoring sessions. Observer Perceptions of Reading Recovery Program Implementation Site observers reported being impressed with the dedication and commitment of the Reading Recovery teachers to the fidelity of the teaching procedures and the integrity of the implementation of the program. Almost all teachers observed were meeting all the standards, guidelines, and expectations of the Reading Recovery Council of North America and the North American Trainers m Group. Since observations occurred in March and April, most of the students observed were second- round students, since teachers had discontinued their first-round students. Teachers reported that students who did not discontinue from Reading Recovery during the first-round were being considered 31for further intervention services. Site observers also reported that the Reading Recovery program in the Little Rock district receives an adequate allocation of time, materials, and other resources. However, several teachers and principals expressed the need for additional Reading Recovery n w teachers in their schools. Reading Recovery teachers also reported teaching literacy small groups the rest of the day which enables them to give their Reading Recovery students more attention during the instructional day. By teaching these literacy groups, site observers suggested that the Reading Recovery teachers expertise and knowledge gained from their training and practice benefits children across 4 q q several grade levels. During visits site observers suggested areas in which some Reading Recovery teachers needed improvement. These instructional areas included: (1) hearing and recording sounds in words
(2) making and breaking
(3) doing away with the helping hand
and (4) maintaining up-to-date records on each child as a basis for instruction. A summary of observers general findings for each classroom observation is included in Appendix D. q q q q q 32 ATable 3 Reading Recovery Implementation Assessment Instrument: Spring 2005 N=14 1. II. Please check: Observed: 0 Not Observed: N Please rate each of the following items in terms of the quality of implementation by using the appropriate number according to the following scales: Quality 1 = Poor or unacceptable 2 = Below average in comparison to other programs obsen/ed 3 = Meets nearly all standards of program quality 4 = Above average in comparison to other programs Obseived Not Observed Reading Recoveiy Progtam Components % Poor 4 100.0 0.0 100.0 0.0 92.86 7.14 Reading familiar stories Reading a story that was read for the first time the day before - incorporates nmning record Working with letters and /or words using magnetic tetters 0.0 0.0 0.0 92.86 7.14 Writing a story 0.0 92.86 7.14 Assembling a cut-up story 0.0 92.86 7.14 Introducing and reading a new book 0.0 4 (Overall rating: Follows the Reading Recovery lesson frameworks Observed Not Obsen/ed Reading Recovery Program Strategies % Poor 92.86 7.14 Appropriate pacing of lesson components 0.0 4 92.86 7.14 100.0 0.0 Appropriate text selected throughout the lesson Appropriate prompts are used for scaffolding the child to problem solve 0.0 0.0 100.0 0.0 Child is engaged in constaictive problem solving 7.14 92.86 7.14 Echo of focus throughout the lesson 0.0 92.86 7.14 Procedures are adjusted according to child's needs 0.0 92.86 7.14 Balance of fluency phrasing practice and problem solving 0.0 85.71 7.14 Opportunities to develop phonological awareness within the lesson 0.0 85.71 85.71 7.14 7.14 Accurate up-to-date records Articulates childs strengths and needs 0.0 0.0 85.71 7.14 Has high expectations for the child 0.0 % Below Average 1429 7.14 4286 0.0 0.0 0.0 % Below Average 21.43 7.14 7.14 0.0 7.14 0.0 7.14 7.14 7.14 0.0 0.0 Quality % Meets 21.43 28.57 1429 21.43 7.14 7.14 Quality % Meets 7.14 0.0 1429 21.43 42.86 1429 7.14 21.43 14.29 14.29 14.29 Overall Rating: "NOTE: Item percentages may not total 100% because of missing input from observers. % Above Average 6429 6429 35.71 71.43 85.71 85.71 Mean:3.46 SD
0.54 % Above Average 64.29 85.71 78.57 71.43 42.86 78.57 78.57 64.29 71.43 78.57 78.57 Mean: 3.61 SO: 0.66 33 INTERVIEW RESULTS Reading Recovery Principal Interview Principals at 10 of the Reading Recovery schools were randomly selected to participate in a 45-minute phone interview. Principals were asked a series of questions regarding general program implementation, classroom level changes, program results, professional development opportunities and parental and community involvement. A summary of the principal responses can be found in Appendix B. Overall, principals were positive about the Reading Recovery program and the impact it has had on their schools. Most of the schools have been utilizing Reading Recovery for several years and faculty and staff are very comfortable with the program. Nearly all of the principals interviewed were instrumental in bringing Reading Recovery to their schools, and the decision to utilize Reading Recovery was made after considerable research and thoughtful consideration. Principals reported that Reading Recovery is a wonderful complement to the schools balanced literacy programs. The one-on-one attention the reading Recovery students receive is overwhelmingly the most effective of the strategies that the program employs. Other effective strategies mentioned included push-ins, literacy groups, running records, and the writing component. Principals reported being active advocates in their Reading Recovery programs. They described their roles as one of support and involvement, .ranging from oversight of the program to more direct involvement including student selection for the program, review of student progress, and i - ongoing meetings and collaboration with the Reading Recovery specialists. All principals noted that teachers were very supportive of the Reading Recovery program and appreciated the impact it has had on overall student achievement. Most of the resources needed for effective program implementation are available
however, principals reported an ongoing need for books, additional teachers and tutors, and more planning time. Principals described the African- American population as being well-served by Reading Recovery, and most agreed that through Reading Recovery the achievement gap is being bridged for their African-American students. In most of the schools, African-American students are a large percentage of the Reading Recovery program. 34and the early intervention provided by Reading Recovery allows the students to be encouraged by being successful at a younger age. Teacher in-Training Interview In the spring 2005, four teachers in their first year in the Reading Recovery program were contacted by phone for a 30 to 45-minute phone interview. The teachers in-training were attending classes concerning Reading Recovery instruction, as well as working in their schools implementing the Reading Recovery strategies. Feedback from the teachers was solicited regarding general program information, classroom level changes, results, professional development, and parental involvement and support. A summary of the teacher in-training responses can be found in Appendix C. All teachers described the process of integrating Reading Recovery into the schools literacy program as well planned and organized. Reading Recovery teachers work individually with the lowest performing students and then follow up individual instruction with literacy groups. Reading Recovery teachers reported a thorough selection process that involved collaboration with the classroom teacher and comprehensive testing and assessment. In general, the Reading Recovery teachers reported strong support from the classroom teachers, that classroom teachers appreciate the effectiveness of the one-on-one approach, and the fact that Reading Recovery is able to provide this type of support. freeing the classroom teacher to work more effectively with the other students. 4 The Reading Recovery teachers in-training strongly felt that Reading Recovery helped to equalize learning and achievement opportunities for African-American students at their schools. Most 4 of the students in Reading Recovery are African-American, and these teachers were able to see positive gains. Reading Recovery helps all struggling readers by actively engaging them in reading and allowing them to feel successful at reading. Through detailed daily records and periodic testing all inherent in the Reading Recovery programstudent progress is effectively monitored and lessons can be planned that are tailor made for each individual child. Additionally, the Reading Recovery program often serves as a first line of defense in determining special education needs. Often, Reading Recovery will be used prior to special education referral. 4 Teachers in-training emphasized the importance of the daily data reports they collect and 4 view them as essential to the success of the program. Reading Recovery teachers collect and use 35 data to monitor the progress of the students and to help develop appropriate, effective teaching strategies based on the individual needs of the student. Data are often shared with the classroom teacher and the principal so that a collaborative team develops to help plan and implement instruction for the student. The Reading Recovery teachers in-training report a significant increase in self-confidence and improvement in overall attitude among the Reading Recovery students. As the students begin to experience success, they are less frustrated and angry. This improvement in attitude improves their classroom behavior and relationships with other teachers and students. Some of these teachers have seen dramatic increases in test scores and other achievement data, while others have not yet been able to see these gains. Again, from the perspective of the teachers in-training, the Reading Recovery program is helping to close the achievement gap of African-American students. Reading Recovery teachers in-training reported professional training that ranged from adequate to thorough. Changes in Reading Recovery course instructors led to some confusion for some of the teachers in-training, and some reported the need for increased classroom hours. However, overall, these teachers reported feeling well-prepared to work with the students. The strategies they have learned have been very helpful. Although the teachers in training reported having significant classroom experience, they are learning unique techniques that have been very beneficial. 4 There is considerable support from the district teacher, and all teachers expressed a desire and need for continuing support in the upcoming school year. Parental support has been mixed. although all the teachers reported active attempts at engaging the parents in the Reading Recovery program. Reading Recovery teachers in-training concurred that the exposure to print and the one-on-one attention were the most critical elements of the Reading Recovery program. Teachers reported the need to understand several components better, including the make and break lesson and the writing component
and the need for time to plan and implement the program is always a factor. Overall, the components of Reading Recovery were well understood and effective. r,1 36 SURVEY RESULTS Reading Recovery Teacher Questionnaire (RRTQ) Descriptive results. The results of the RRTQ are summarized in Table 4. Reading Recovery teachers had extremely favorable attitudes toward the program at their schools. As 4 illustrated in Table 4,100% of teachers strongly agreed or agreed that they have a thorough understanding of the program, teachers in their school are generally supportive of the program, ongoing communication exists between Reading Recovery tutors and classroom reading teachers, Reading Recovery monthly meetings are effective and useful, instructional materials needed to implement the Reading Recovery program are readily available, and the Reading Recovery program is aligned with state and district reading and language arts standards. There was also strong agreement that Reading Recovery teachers received support, with 86.4% of the teachers reporting that the school administration and Reading Coach supported their efforts as a Reading Recovery teacher. Almost 75% (72.7%) of teachers reported receiving extensive district support. The items with the highest level of disagreement (disagree and strongly disagree) concerned Reading Recovery teachers having sufficient planning time (36.4%), enough tutors to fully implement the Reading Recovery program (18.2%), and time to routinely monitor first grade students progress after they were discontinued from Reading Recovery (9.1 % not at all, and 72.7% somewhat). Demographic data. All Reading Recovery teachers (100%) reported having at least six years of teaching experience and 31.8% reported at least six years experience as a Reading Recovery teacher. Approximately 85% (86.4%) of Reading Recovery teachers reported having Masters Degree or beyond. The majority of Reading Recovery teachers were Caucasian (72.7%) R R and 13.6% reported their ethnicity as African-American. Open-ended responses. Reading Recovery teachers were asked to respond to three open-ended questions in addition to the 24 items on the questionnaire. Respondents were instructed to list the strongest aspects of the Reading Recovery program, the weakest aspects of the Reading Recovery program, and reasons for continuing or discontinuing Reading Recovery at their school. There were 22 Reading Recovery teachers who filled out the questionnaire, and 21 of those also answered the open-ended questions. Many of the Reading Recovery teachers answered in detail. 37 listing multiple responses for each of the questions. The open-ended responses are summarized in Table 5, and a verbatim listing of teacher responses can be found in Appendix E. Of the 21 who listed strong aspects of the Reading Recovery program, the two most popular answers were individualized, one-on-one instruction, 57.1%, and the ability to reach the lowest performing students and bridge the achievement gap for these students, 47.6%. Reading Recovery teaching strategies and components were listed in 19% of the responses, and both the early intervention element and the support from the other teachers were listed in 14.3% of the responses. Respondents were also asked to list what they considered the weakest elements of the Reading Recovery program. Time was a factor in many of the responses with lack of planning time most frequently mentioned at 33.3%. Lack of time to complete the lesson was mentioned in 19.0% of 4 the responses. Teachers' inability to help all the students who need help was also mentioned in 19.0% of the responses. All of the 21 teachers responded positively to the question Do you think your school should continue the Reading Recovery program? Why?"
and 20 teachers elaborated on their positive response listing reasons for continuing Reading Recovery. The ability of Reading Recovery to help J struggling readers was listed in half of the responses (50.0%), and strong progress and results were mentioned in 40.0% of the responses. The fact that Reading Recovery decreases the number of special education referrals was listed in 15.0% of the responses. There was one respondent who pointed out that although he/she felt Reading Recovery should be continued at the school, there needed to be better implementation to make it effective. Overall, as reflected in the other items on the questionnaire, the open-ended responses were very positive. Reading Recovery teachers appear knowledgeable and committed to their roles as Reading Recovery teachers and believe strongly in the positive impact the program is having at their schools. Based on survey response and comments provided through the open-ended questions. Reading Recovery teachers are able to see a strong impact from the individualized instruction they provide. 38 Table 4 Reading Recovery Teacher Questionnaire (RRTQ): 2004-2005 N = 22 ______________________________________________ RRTQ Items 1. a 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. I have a thorough understanding of the schools Reading Recovery program. I have received adequate initial and ongoing professional development/training for implementation of the Reading Recovery program. Our Reading Recovery program has positively impacted student achievemenl Because ot Reading Recovery, Literacy Group interventions occur for students in grades K-3. Overall, this program seems valuable for improving the achievement of African-American students. Reading Recovery teachers are given sufficient planning time to implement the program. Our school has enough tutors to fully implement its Reading Recovery program. The administration protects the time for daily uninterrupted Reading Recovery tutoring and Literacy Small Group interventions. Because of our Reading Recovery program, parents are more involved in the literacy program of this school. This school has a plan for evaluating aU elements of our Reading Recovery program. Teachers in this school are generally supportive of the Reading Recovery program. Ongoing communication exists between Reading Recovery tutors and classroom reading teachers. Reading Recovery teachers are encouraged to communicate concerns, questions, and constnictive ideas regarding the program. Our Reading Recovery program adequately addresses the requirements of children with special needs. Reading Recovery teachers participate in the special education referral process to provide early literacy intervention. Because of Reading Recovery, teachers in this school spend more time working togethrer to plan instruction and review student progress. Reading Recovery monthly meetings (continuing contact) are effective and useful. Instructional materials (books, assessments, and other resources) needed to implement our Reading Recovery program are readily available. The faculty, staff, and administration believe that all children can read at grade level or above by the end of third grade. The Reading Recovery program is aligned with state and district reading and language arts staiidards. Percent Strongly Agree And Agree 100.00 95.45 95.45 95.45 86.36 54.55 59.09 7727 63.64 86.36 100.00 100.00 81.82 7727 86.36 81.82 100.00 100.00 63.64 100.00 1. 3. Percent Neutral 0.00 4.55 0.00 0.00 4.55 9.09 18.18 22.73 18.18 13.64 0.00 0.00 18.18 13.64 9.09 18.18 0.00 0.00 31.82 0.00 4. RRTQ Items To what degree did your school administration support your efforts as a Reading Recovery teacher? To what degree did your school Reading Coach support your efforts as a Reading Recovery teacher? To what degree does the district support your efforts as a Reading Recovery teacher? To what degree did your schedule allow the time to routinely monitor first grade students' progress after they were discontinued from Reading Percent Extensively 86.36 86.36 72.73 18.18 Percent Somewtiat 13.64 4.55 2727 72.73 Percent Disagree and Strongly Disagree 0.00 0.00 0.00 4.55 0.00 36.36 18.18 0.00 13.64 0.00 0.00 0.00 0.00 9.09 4.55 0.00 0.00 0.00 4.55 0.00 Percent Not at all 0.00 0.00 0.00 9.09 ________Recovery tutoring?________________________________________________________________________________________________________ NOTE: Item percentages may not total 100% because of missing input from some respondents. 39Table 4, continued Reading Recovery Teacher Questionnaire (RRTQ): 2004-2005 N = 22________________________________________ Total Years of Experience in this School Less than 1 year 1-5 years 6-10 years 11-15 years More than 15 years Number of Respondents Total Years of Experience in any School Less than 1 year 1-5 years 6-10 years 11-15 years More than 15 years ______________Number of Respondents How many years experiences have you had as a Reading Recovery teacher? Less than one year 1-5 years 6-10 years 11-15 years More than 15 years ______________Number of Respondents Educational Attainment Bachelor's degree Master's degree Master's plus 20 hrs Education Specialist's Doctoral ______________ Number of Respondents ______________________ Ethnicity/Race Asian or Pacific Islander American Indian or Alaskan Native Black, not of Hispanic origin Hispanic, regardless of race White, not of Hispanic origin Multi-racial / Other Number of Respondents Percent 0.00 36.36 31.82 18.18 9.09 22 Percent 0.00 0.00 18.18 13.64 68.18 22 Percent 0.00 68.18 31.82 0.00 0.00 22 Percent 13.64 36.36 40.91 9.09 0.00 22 Percent 0.00 0.00 13.64 0.00 72.73 9.09 22 NOTE: Item percentages may not total 100% because of missing input from some respondents. 40 Table 5 Open-Ended Responses on the RRTQ Question Positive Comments 1. What are the strongest aspects of the Reading Recovery Program? 2. 3. 4. 5. 6. 1. 8. 9. Teacher Selection_________________ __________Responses_________________ One-on-one intervention/ Individualized lessons Bridging achievement gap by reaching lowest performing students Reading Recovery teaching strategies and components of instnjction Early intervention Support from other RR teachers Continuing contact with students Professional training and development Addresses both reading and writing Increases students' confidence Frequency Percent 12 10 57.1 47.6 10. Collaboration with classroom teacher 11. Close contact with parents_________ Total Responses N = 21 43 32 221 11 19.0 14.3 14.3 9.5 9.5 9.5 4.8 4.8 4.8 Sample responses Working one-on-one with an at-risk child.
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.