Evaluation of Reading Recovery In the Little Rock School District, 2005 Aggregate Report, Draft

RECEIVED JAN 5 2006 OFRCEOF DESEGREGATION MONITORING Research in EducalionaJ Pcticy Evaluation of Reading Recovery In the Little Rock School District Ceruer for Researdi in Educational Policy The UniversSy of Memphis 325 Browning Ha# f.temohcs Tennessee 38152 To(Free1-866^7(M147 2005 Aggregate Report DRAFT REPORT FOR REVIEW BY Little Rock School District CREP Center for Research in Educational Policy Center for Rcsea'di in Edixatkinal Potcy The University' of Fvlempliis 325 Browning Mali Memohis Tennessee 38152 Tea Free 1-8I56-67M147 Evaluation of Reading Recovery In the Little Rock School District 2005 Aggregate Report DRAFT REPORT FOR REVIEW BY Little Rock School District December 2005 Anna W. Grehan Steven M. Ross Lynn Harrison Center for Research in Educational Policy John Nunnery Old Dominion UniversityEXECUTIVE SUMMARY INTRODUCTION The Little Rock School District began implementing Reading Recovery during the 1995-1996 school year in two schools. During the first four years of implementation, the Little Rock School District was part of the Pulaski County Reading Recovery Site. By 1998 there were eight trained Reading Recovery teachers in seven schools in the Little Rock School District. In July 1999 the district became a Reading Recovery Site hiring a full-time teacher leader. In 2000-2001 the University of Arkansas at Little Rock (UALR) began conducting the Reading Recovery teacher training for the district. Between the 2000-2001 and 2004-2005 school years UALR trained seventeen Reading Recovery teachers for the district. At the end of the 2004-2005 school year the Little Rock School District had 28 trained Reading Recovery teachers serving 18 of the 34 elementary schools in the district. Reading Recovery is one of eight literacy programs, interventions, and/or models used by Little Rock schools. Currently, Little Rock School District funds are used to support the program. The goal of Reading Recovery is to dramatically reduce the number of first grade students who have difficulty learning to read and write and to reduce the cost of these learners to educational systems. Reading Recovery is a short-term intervention program of one-to-one tutoring for the lowest-achieving first graders. The intervention is most effective when it is available to all students who need it and is used as a supplement to good classroom teaching. Individual students receive a half-hour lesson each school day for 12 to 20 weeks with a specially trained Reading Recovery teacher. As soon as students can read within the average range of their class and demonstrate that they can continue to achieve, their lessons are discontinued, and new students begin individual instruction. The evaluation plan for the Reading Recovery program in Little Rock School District included: (1) analyses of Reading Recovery student achievement and program data, (2) principal, teacher, and parent surveys and interviews, and (3) observations of Reading Recovery tutoring sessions. This report is part of a larger district study of four programs evaluating the effectiveness in improving and 1remediating the academic achievement of African-American students. This report includes results from 18 elementary schools participating in the Reading Recovery program. RESEARCH QUESTIONS The major goals of this research study were to evaluate African-American student achievement outcomes, program implementation fidelity, and principal, teacher, and parent perceptions concerning the Reading Recovery tutoring program for first grade students. Student achievement results on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement were analyzed to compare the progress of first graders enrolled in the Reading Recovery intervention program and comparison students in 2004-2005. Program implementation ratings were obtained from observations of 14 tutoring sessions in nine schools. The survey and interview results are based on the perceptions of 156 classroom teachers in grades K-3,22 experienced Reading Recovery teachers, four teachers in-training, 10 principals, and 95 parents. The Reading Recovery evaluation was structured around the following seven primary and supplemental research questions. Primary evaluation question: Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? Supplemental (Qualitative/Step 2) evaluation questions: What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? What is the level of participation in Reading Recovery by African American students relative to other ethnic groups at the school? What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? 2 What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? DESIGN The evaluation period extended from February 2004 through May 2005. The evaluation design was based on both quantitative and qualitative data collected from Reading Recovery intervention observations, principal and teacher in-training interviews, classroom teachers, Reading Recovery teachers, and parent surveys, and Reading Recovery program data. Reading Recovery student-level achievement data on the ITBS, DRA, DIBELS, and An Observation of Early Literacy Achievement was received in Fall 2005 and incorporated in this report. The primary data collectors in this study were Center for Research in Educational Policy (CREP) trained site researchers. Site researchers: (1) conducted Reading Recovery intervention observations, (2) administered the Reading Recovery teacher survey, (3) conducted principal and teacher in-training interviews, and (4) collected program data. Principals at Reading Recovery schools were responsible for administering the classroom teacher and Reading Recovery parent surveys. Participants Little Rock School District is located in Central Arkansas and serves approximately 26,500 students, with African-Americans representing approximately 67% of the district student population, in 49 schools in an urban area with a population of 184,000. In the 2004-2005 school year 18 elementary schools and 230 first grade students, of which 173 are African American, 27 are white, 22 are Hispanic, and the remaining 8 are other ethnicities, participated in the Reading Recovery program in the district. However, the schools indicated that 365 first grade students needed Reading Recovery services. Three schools were in their first year of program implementation (Bale, Stephens, and 3Terry) and survey and observation data were not collected from these schools. However, randomly selected teachers in-training and principals from these three schools were interviewed for this study. INSTRUMENTATION Six instruments were developed by CREP to collect the evaluation data: a classroom observation tool, a principal interview, a teacher in-training interview, a Reading Recovery teacher questionnaire, a classroom teacher questionnaire, and a parent survey. A detailed description of each instrument follows. Classroom Observation Measures Reading Recovery Implementation Assessment Instrument (RRIAI) The RRIAI was designed for observation in Reading Recovery classrooms and was developed by researchers at CREP and Reading Recovery faculty at ULAR. Ratings are organized around two categories: Reading Recovery Program Components and Reading Recovery Program Strategies. The RRIAI observation tool provides an overall rating for each category based on a rubric that ranges from (1) poor or unacceptable
(2) below average in comparison to other programs observed
(3) meets nearly all standards of program quality
and (4) above average in comparison to other programs. The RRIAI has been aligned to the essential components of the Reading Recovery program. Sub-categories of the program components include: reading familiar stories, reading a story that was read for the first time the day before, working with letters, writing a story, assembling a cut-up story, and introducing and reading a new book. The tool was designed to be utilized by experienced Reading Recovery trainers during the 30-minute tutoring session with additional time allocated for observer questions and examination of student records after the session. Interviews Reading Recovery Principal Interview Randomly selected principals from 10 Reading Recovery schools participated in approximately 45-minute phone interviews with CREP researchers. Interview questions addressed 4the principals general experiences and reactions to the Reading Recovery implementation and the associated outcomes for the school, students, teachers, and parents. Reading Recovery Teacher in Training Interview In the 2004-2005 school year, six teachers were beginning their first year as a Reading Recovery teacher and considered as teachers in-training. Three were located in experienced schools and three were located in the schools beginning their first year of implementation: Bale, Stephens, and Terry. Four randomly selected teachers in-training participated in an approximately 45-minute phone interview with CREP researchers. Interview questions addressed the teacher's in-training experiences with and perceptions of the Reading Recovery program implementation with regard to such areas as resources, professional development, parent involvement, support, and student outcomes. Surveys Reading Recovery Teacher Questionnaire (RRTQ) Reading Recovery teachers in the 15 experienced Reading Recovery schools were asked to complete the RRTQ, which includes four sections. Section I contains 20 items to which teachers respond using a five-point Likert-type scale ranging from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the general impressions of the Reading Recovery program, professional development, support for the program, impacts on student achievement, and alignment of state and district reading and language arts standards. In a second section, teachers were asked to indicate to what degree the listed items occurred. The focus of the 4 items were (1) administration support of Reading Recovery teacher efforts
(2) Reading Coach support of Reading Recovery teacher efforts
(3) district support of Reading Recovery teacher efforts
and (4) the time to routinely monitor first grade students progress after they were discontinued from tutoring. Open-ended questions asked Reading Recovery teachers to respond to the statements: What are the strongest aspects of the Reading Recovery program? What are the weakest aspects of the Reading Recovery program? Do you think your school should continue the Reading Recovery program? A final section of the RRTQ contained general questions regarding years of teaching experience, race, and level of education. 5Reading Recovery Classroom Teacher Questionnaire (RRCTQ) All K-3 teacher who taught in experienced Reading Recovery schools were asked to complete the RRCTQ, which contains 13 items to which teachers respond using a five-point Likert- type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the specific program elements of Reading Recovery such as: understanding of the program, impacts on student achievement, teacher support for the program, and parent involvement. Open-ended questions asked classroom teachers to respond to the statements: Please describe your understanding of the Reading Recovery program In your school. What are the strongest aspects of the Reading Recovery program? Do you think your school should continue the Reading Recovery program? A final section of the RRCTQ contained general questions regarding years of teaching experience, race, and level of education. Reading Recovery Parent Survey (RRPS) Parents whose first grade children were currently receiving Reading Recovery tutoring services were asked to complete the RRPS, that contains 6 items to which parents respond using a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed general impressions of the program such as: improvement in childs reading skills, child's enjoyment of tutoring sessions, and opportunities to communicate with the Reading Recovery teacher. Open-ended questions asked parents to respond to the statements: Please describe your understanding of the Reading Recovery tutoring program at your child's school. What is the BEST thing about your childs involvement with the Reading Recovery tutoring program? What CHANGES would you like to see in the Reading Recovery tutoring program? A final section of the RRPS asked parents to indicate the race, grade, and age of their child. Program Data Reading Recovery program information was obtained from data submitted to the National Data Evaluation Center at The Ohio State University in 2004-2005. Each year the schools and district submit data and receive a site report. The report represents an examination of Reading Recovery student outcomes for Little Rock and accounts for all children served by Reading Recovery within the 6site during the 2004-2005 school year. In addition, attention is given to implementation factors that may be supporting or hindering the success of the intervention with the site. Student Achievement Results In addition to the program data, interviews, survey, and observation tools cited above, reading achievement data is derived from scores on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and An Observation of Early Literacy Achievement. A description of each assessment follows. Iowa Tests of Basic Skills (ITBS). The ITBS Form A is a norm-referenced group administered test that measures the skills and achievement of students in grades K-8 and was developed at The University of Iowa. The ITBS provides an in-depth assessment of students achievement of important educational objectives. Tests in Reading, Language Arts, Mathematics, Social Studies, Science, and Source of Information yield reliable and comprehensive information both about the development of students skills and about their ability to think critically. The emphasis of the K-1 assessment is on academic skills found in the early childhood curriculum. These tests are neither measures of readiness for school nor readiness to read. Rather, they assess the extent to which a child is cognitively prepared to begin work in the academic aspects of the curriculum. Test materials have been extensively field tested for psychometric soundness and evaluated for fairness to gender. racial, ethnic, and cultural groups. Developmental Reading Assessment (DRA). The Developmental Reading Assessment provides teachers with a method for assessing and documenting primary students development as readers over time with a literature-based instructional reading program. The DRA is designed to be used in K-3 classrooms with rich literate environments. The assessments are conducted during one- on-one reading conferences as children read specially selected assessment texts. A set of 20 stories. which increase in difficulty, are used for the assessment. The DRA evaluates two major aspects of reading: accuracy of oral reading and comprehension through reading and retelling of narrative stories. Questions pertaining to concepts about print are also included in the assessment with lower leveled texts. 7Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS is a tool for early identification of children with potential literacy problems and an assessment of response to instruction. The assessment is individually administered to K-3 children at least three times per year. The DIBELS assessment is designed to enable educators to modify their approach if a student is not on course to achieve reading goals. The Early Childhood Research Institute on Measuring Growth and Development at the University of Oregon constructed DIBELS (2000). The Institute reports it has validated the instruments ability to predict outcomes and has tested its reliability with young children across the country. Reading Recovery schools that have received Reading Excellence Act or Reading First grant funding use the DIBELS assessment in Little Rock. An Observation of Early Literacy Achievement: Reading Recovery Subtests. Six tasks in Marie Clays (2002) An Observation of Early Literacy Achievement (Observation Survey) were used as pretest and posttest measures. The Survey tasks have the qualities of sound assessment instruments with reliabilities and validities. The purpose of the assessment is to determine an appropriate level of text difficulty and to record, using a running record, what the child does when reading continuous text. All six tasks of the Observation Survey were administered to Reading Recovery students in the fall (start of the school year) and/or at entry to the intervention. These scores serve as pretest measure in the evaluation design. The six tasks were also administered to Reading Recovery students upon discontinuing or exiting from the program. In the spring (end of school year), the six tasks were again administered to all students who received Reading Recovery services during the year. Year-end scores served as the posttest measures in comparing the progress made by Reading Recovery children in the various end-of-program status groups. PROCEDURE Data for the evaluation were collected March-May for the 2004-2005 school year. On February 16, 2005 principals were given an overview of the evaluation and timelines for collecting data. On March 3-4 and April 14-15,14 tutoring observations were conducted in nine randomly selected experienced Reading Recovery schools by two Reading Recovery content experts from Georgia State and The Ohio State Universities. Only experienced Reading Recovery teachers were observed. On March 23 a CREP trained site researcher administered the Reading Recovery teacher 8survey to experienced teachers at a regularly scheduled monthly meeting at ULAR. In April and May principals in experienced Reading Recovery schools administered the classroom teacher questionnaire to K-3 grade teachers. Principals also administered the parent survey to parents whose children were currently receiving first grade intervention services in the 15 experienced Reading Recovery schools. Four teachers in-training and ten principals were randomly selected from the 18 Reading Recovery schools to participate in a phone interview in April and May conducted by CREP researchers. Reading Recovery program data was received in summer 2005 from the National Data Evaluation Center at The Ohio State University. Student achievement data was received from the district in fall 2005. METHODS - STUDENT ACHIEVEMENT Sample. The sample included 1,094 first grade students who attended one of 18 schools that implemented the Reading Recovery program during the 2004-2005 school year. Of these, 230 were referred to the Reading Recovery program, and 864 were in the comparison group. The percentages of students who were African American or of Limited English Proficiency were similar between the comparison and Reading Recovery groups (73.6% versus 75.6% and 7.3% versus 8.7%, respectively). However, Reading Recovery students were more likely to be eligible for free or reduced-price lunch (84.3% versus 73.1%), to receive special education services (15.7% versus 8.6%), and to be male (58.1 % versus 48.0%). A random sample of about 11 comparison students was selected from each school (total n = 189) for administration of the DIBELS and Observation Survey: Reading Recovery test batteries. Reading Recovery Treatment Level. In the Reading Recovery program, students who are deemed to have attained a reading level equivalent to their peers are assigned an end-of-program status of discontinued. Children who received 20 or more weeks of Reading Recovery services, but who do not attain a reading level equivalent to their peers are assigned a status of recommended action. Children who have received fewer than 20 weeks were assigned incomplete program status. Children designated as unknown status were removed from the program in fewer than 20 weeks due to reasons other than the school year ending. Other children were designated as having moved during the school year. The median number of sessions of recommended action {Md = 968.50
n = 68) children was actually higher than the median for discontinued students (Md = 57.00
n = 90). The medians for incomplete students (n = 46) and unknown students (n = 12) were very similar (Md = 44.63 versus 46.08, respectively). For the purposes of this study, recommended action and discontinued students were categorized as complete program
incomplete and unknown students were categorized as incomplete program
and students who moved were eliminated from the analyses. This left a total of 216 Reading Recovery students: 158 with a complete program, and 58 with an incomplete program. Measures. Pretest (covariate) measures included: (a) Spring. 2004 Developmental Reading Assessment (DRA) subtests in Letter Recognition, Word Recognition, Capitatization, Writing, Dictation, and a DRA Test score, (b) Fall 2005 Observation Survey: Reading Recovery program subtests that included Letter Identification, Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds in Words, and Text Reading, and (c) the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in Letter Naming Fluency, Phoneme Segmentation Fluency, Nonsense Word Fluency, and Word Use Fluency. Outcome measures included Iowa Test of Basic Skills (ITBS) Reading Normal Curve Equivalent Scores. DRA subtests. Observation Survey: Reading Recovery subtests, and DIBELS subtests administered in Spring 2005. To achieve some parsimony in the analyses, pretest measures were subjected to a principal components analysis, and regressionbased factors scores were constructed for pretest DRA. Observation Survey: Reading Recovery, and DIBELS subtests. A single factor accounted for 60.1 %. 60.5%. and 43.8% of the variance in the DRA. Observation Survey: Reading Recovery, and DIBELS subtests, respectively. Outcome measures included the DRA. Observation Survey: Reading Recovery, and DIBELS subtests, as well as ITBS. The DRA Letter Recognition, DRA Capitalization, and DIBELS Letter Naming Fluency subtests were not administered in Spring 2005. DIBELS and Observation Survey: Reading Recovery tests were administered to a small random sample of comparison students within each school. Student achievement analyses. A 3 (Program Status) X 2 (African American versus other) multivariate analysis of covariance (MANCOVA) using student gender, the pertinent pretest factor score, free lunch status, special education status, and LEP status was performed with the various subtests as outcomes for the DRA. Observation Survey: Reading Recovery, and DIBELS test 10batteries. A similar 3X2 analysis of covariance was performed on ITBS Reading NCE scores using the 2003-2004 DRA factor score as a pretest covariate. For the multivariate analyses, Wilks lambda was used as the criterion of multivariate significance. Where Wilks lambda indicated a significant multivariate effect, follow-up univariate analyses were performed on each outcome variable using the Bonferroni procedure to control for experimentwise alpha. When significant univariate results were found, post hoc analyses were performed using Scheffes procedure. Effect size estimates were computed for all posttests by subtracting the adjusted mean for the comparison group from the adjusted mean from the Reading Recovery group within levels of race, then dividing by the total standard deviation of the posttest. For ITBS Reading NCE effect size estimates, the population standard deviation of 21.06 was used. Exploratory and supplementary analyses. Impact of teacher experience and number of sessions. The number of years experience with Reading Recovery was recorded for each teacher. Years experience ranged from 1 to 10 years. ITBS Reading NCE scores and DRA test scores for 2004-2005 were regressed on 2003-2004 DRA factor scores and dummy-coded variables representing student ethnicity, free lunch status, gender, special education status, and LEP status. Standardized residuals were saved from each of these analyses, and plotted against teacher years of experience to graphically assess the nature of the relationship between teacher experience and teacher effectiveness in Reading Recovery for students who received a Complete program. Standardized residuals for each of these 2004-2005 tests were also plotted against number of Reading Recovery sessions received for all students (i.e., those who received either a Complete or Incomplete program). Finally, effect size boxplots were constructed for African American students who had Complete and Incomplete Reading Recovery programs. 11RESULTS Classroom Observation Results Reading Recovery Implementation Assessment Instrument (RRIAI) As indicated in the description of the RRIAI, the observation procedure primarily focuses on Reading Recovery Program Components and Program Strategies. The site observers used a four- point rubric (1 = poor or unacceptable, 2 = below average in comparison to other programs observed, 3 = meets nearly all standards of program quality, and 4 = above average in comparison to other programs) to rate the frequency and application of components and strategies of Reading Recovery instruction. More precise data regarding observed Reading Recovery instructional practices measured by the four-point rubric for the 14 observations. Reading Recovery Program Components The overall mean rating for the Reading Recovery Program Components was 3.46 which suggest a high level of program implementation. Of the six subcategories the highest observed ratings were
assembling a cut-up story and introducing and reading a new book observed to be above average in 85.71 % of tutoring sessions. The program component subcategory with the lowest observed rating was working with letters and/or words using magnetic letters observed to be above average in only 35.71% of tutoring sessions. The six program components were observed in at least 92.86% of 14 tutoring sessions. Reading Recovery Program Strategies The overall mean rating for the Reading Recovery Program Strategies was 3.61 which also indicate a high level of instructional effectiveness. Of the eight During Tutoring Lesson subcategories the highest observed rating was for appropriate text selected throughout the lesson observed to above average in 85.71% of tutoring sessions. The During Tutoring Lesson subcategory with the lowest observed rating was echo of focus throughout the lesson observed to be above average in just 42.86% of tutoring sessions. For the After Tutoring Lesson, has high expectations for the child and articulates childs strengths and needs were observed to be above average in 78.57% of tutoring sessions and accurate and up-to-date records were observed to be above average in 71.43% of 12tutoring sessions. The Reading Recovery Program Strategies were observed in at least 85.71 of tutoring sessions. Observer Perceptions of Reading Recovery Program Impiementation Site observers reported being impressed with the dedication and commitment of the Reading Recovery teachers to the fidelity of the teaching procedures and the integrity of the implementation of the program. Almost all teachers observed were meeting all the standards, guidelines, and expectations of the Reading Recovery Council of North America and the North American Trainers Group. Since observations occurred in March and April, most of the students observed were second- round students, since teachers had discontinued their first-round students. Teachers reported that students who did not discontinue from Reading Recovery during the first-round were being considered for further intervention services. Site observers also reported that the Reading Recovery program in the Little Rock district receives an adequate allocation of time, materials, and other resources. However, several teachers and principals expressed the need for additional Reading Recovery teachers in their schools. Reading Recovery teachers also reported teaching literacy small groups the rest of the day which enables them to give their Reading Recovery students more attention during the instructional day. By teaching these literacy groups, site observers suggested that the Reading Recovery teachers expertise and knowledge gained from their training and practice benefits children across several grade levels. During visits site observers did suggest areas in which some Reading Recovery teachers need improvement. These instructional areas include: (1) hearing and recording sounds in words
(2) making and breaking
(3) do away with the helping hand
and (4) maintaining up-to-date records on each child as a basis for instruction. 13INTERVIEW RESULTS Reading Recovery Principal Interview Principals at ten of the Reading Recovery schools were randomly selected to participate in a 45 minute phone interview. Principals were asked a series of questions regarding general program implementation, classroom level changes, program results, professional development opportunities and parental and community involvement. Overall, principals were positive about the Reading Recovery program and the impact it has had on their schools. Most of the schools have been utilizing Reading Recovery for several years and faculty and staff are very comfortable with the program. Nearly all of the principals interviewed were instrumental in bringing Reading Recovery to their schools, and the decision to utilize Reading Recovery was made after considerable research and thoughtful consideration. Principals reported that Reading Recovery is a wonderful compliment to the school's balanced literacy programs. The one-on-one attention the reading Recovery students receive is overwhelmingly the most effective of the strategies that the program employs. Other effective strategies mentioned included push-ins, literacy groups, running records, and the writing component. Principals reported being active advocates in their Reading Recovery programs. They described their roles as one of support and involvement, ranging from oversight of the program to more direct involvement including student selection for the program, review of student progress, and ongoing meetings and collaboration with the Reading Recovery specialists. All principals noted that teachers were very supportive of the Reading Recovery program and appreciated the impact it has had on overall student achievement. Most of the resources needed for effective program implementation are available
however principals reported an ongoing need for books, additional teachers and tutors, and more planning time. Principals described the African American population as being well-served by Reading Recovery, and most agree that through Reading Recovery the achievement gap is being bridged for their African American students. In most of the schools, African American students are a large percentage of the Reading Recovery program, 14and the early intervention provided by Reading Recovery allov/s the student to be encouraged by being successful at a younger age. Teacher in-Training Interview In the spring 2005, four teachers in their first year in the Reading Recovery program were contacted by phone for a 30 to 45 minute phone interview. The teachers in-raining were attending classes concerning Reading Recovery instruction, as well as working in their schools implementing the Reading Recovery strategies. Feedback from the teachers was solicited regarding general program information, classroom level changes, results, professional development, and parental involvement and support. All teachers described the process of integrating Reading Recovery into the school's literacy program as well planned and organized. Reading Recovery teachers work individually with the lowest performing students and then follow up individual instruction with literacy groups. Reading Recovery teachers reported a thorough selection process that involved collaboration with the classroom teacher and comprehensive testing and assessment. In general, the Reading Recovery teachers reported strong support from the classroom teachers, that classroom teachers appreciate the effectiveness of the one-on-one approach, and the fact that the Reading Recovery is able to provide this type of support, freeing the classroom teacher to work more effectively with the other students. The Reading Recovery teachers in training strongly felt that Reading Recovery helped to equalize learning and achievement opportunities for African American students at their schools. Most of the students in Reading Recovery are African American, and these teachers were able to see positive gains. Reading Recovery helps all struggling readers by actively engaging them in reading and allowing them to feel successful at reading. The Reading Recovery teachers in training report a significant increase in self-confidence and improvement in overall attitude among the Reading Recovery students. As the students begin to experience success, they are less frustrated and angry. This improvement in attitude improves their classroom behavior and relationships with other teachers and students. Some of these teachers have seen dramatic increases in test scores and other achievement data, while others have not yet been able to see these gains. 15Reading Recovery teachers in- training reported professional training that ranged from adequate to thorough. Changes in Reading Recovery course instructors led to some confusion for some of the teachers in training, and some reported the need for increased classroom hours. However, overall, these teachers reported feeling well-prepared to work with the students. The strategies they have learned have been very helpful. Although the teachers in training reported having significant classroom experience, they are learning unique techniques that have been very beneficial. Survey Results Reading Recovery Teacher Questionnaire (RRTQ) Descriptive results. Reading Recovery teachers had extremely favorable attitudes toward the program at their schools. All of the teachers strongly agreed or agreed that they have a thorough understanding of the program, teachers in their school are generally supportive of the program. ongoing communication exists between Reading Recovery tutors and classroom reading teachers. Reading Recovery monthly meetings are effective and useful, instructional materials needed to implement our Reading Recovery program are readily available, and the Reading Recovery program is aligned with state and district reading and language arts standards. There was also strong agreement that Reading Recovery teachers received support, with 86.36% of the teachers reporting that the school administration and Reading Coach supported their efforts as a Reading Recovery teacher. Almost 75% (72.73%) of teachers reported receiving extensive district support. The items with the highest level of disagreement (disagree and strongly disagree) concerned Reading Recovery teachers having sufficient planning time (36.36%) and enough tutors to fully implement the Reading Recovery program (18.18%). Demographic data. All Reading Recovery teachers (100%) reported having at least six years of teaching experience and 31.82% reported at least six years experience as a Reading Recovery teacher. Approximately 85% (86.36%) of Reading Recovery teachers reported having a Masters Degree and beyond. The majority of Reading Recovery teachers were white (72.73%) and 13.64% reported their ethnicity as African-American. 16Open-ended responses. Reading Recovery teachers were asked to respond to three open- ended questions in addition to the 24 items on the questionnaire. Respondents were instructed to list the strongest aspects of the Reading Recovery program, the weakest aspects of the Reading Recovery program, and reasons for continuing or discontinuing Reading Recovery their school. There were twenty-two Reading Recovery teachers who filled out the questionnaire, and twenty-one of those also answered the open-ended questions. Of the 21 who listed strong aspects of the Reading Recovery program, the two most popular answers were individualized, one-on-one instruction, 57.1%, and the ability to reach the lowest performing students and bridge the achievement gap for these students, 47.6%. Reading Recovery teaching strategies and components were listed in 19% of the responses, and both the early intervention element and the support from the other teachers were listed in 14.3% of the responses. Respondents were also asked to list what they considered the weakest elements of the Reading Recovery program. Time was a factor in many of the responses with lack of planning time most frequently mentioned at 33.3%. Lack of time to complete the lesson was mentioned in 19.0% of the responses. Teachers inability to help all the students who need help was also mentioned in 19.0% of the responses. All of the 21 teachers responded positively to the question Do you think your school should continue the Reading Recovery program? Why?-, and 20 teachers elaborated on their positive response listing reasons for continuing Reading Recovery. The ability of Reading Recovery to help struggling readers was listed in half of the responses (50.0%), and strong progress and results were mentioned in 40.0% of the responses. The fact that Reading Recovery decreases the number of special education referrals was listed in 15.0% of the responses. There was one respondent who pointed out that although he/she felt Reading Recovery should be continued at the school, there needed to be better implementation to make it effective. Reading Recovery Classroom Teacher Questionnaire (RRCTQ) Descriptive results. For the classroom teachers, the three items on which there was 80% or higher agreement included: teacher support of the program (93.59%), positive impact on student achievement (87.82%), and improving achievement of African-American students (82.05%). The two 17items on which classroom teachers expressed the strongest disagreement or disagreement included
sufficient faculty and staff to fully implement the program (23.08%) and because of Reading Recovery, parents are more involved in the literacy program (14.74%). Demographic data. K-3 classroom teachers reported less teaching experience and education attainment than Reading Recovery teachers. Close to 30% (28.85%) reported less than one year to five years teaching experience in any school and 57.69% reported having a Bachelors degree. However, more classroom teachers reported their ethnicity as African-American (28.85%) than Reading Recovery teachers with the majority (68.59%) reporting ethnicity as white. Open-ended responses. In addition to the 13 items the teachers were asked to rate on the questionnaire, several open-ended questions were also asked of the classroom teachers. Teachers were asked to describe their understanding of the Reading Recovery program, relate the strongest aspects of the program, and discuss whether or not Reading Recovery should be continued at their school. There were 151 teachers who described their understanding of the Reading Recovery program. Classroom teachers at Reading Recovery schools have a very good understanding of the program, which corresponds with the almost 80% (78.2%) who agreed with the item on the questionnaire concerning their understanding of the program. Nearly all of the respondents were able to articulate the main elements of Reading Recovery. A total of 136 teachers responded to the question regarding the strongest aspect of the Reading Recovery program. Many of the teachers listed more than one component as being a strong aspect of Reading Recovery. One-on-one or individualized instruction was listed in 56.6% of the responses as being the strongest aspect of the program. Working with students in small groups was mentioned in 24.2% of the responses and the ability of Reading Recovery to help those students who are most in need was listed in 14.7% of the responses. For the open-ended responses, over half (53.4%) listed the fact that Reading Recovery helps students achieve as a significant reason for keeping the program, and 22.4% stated Reading Recovery should be continued because it helps those in greatest need. Nearly twenty percent (19.8%) listed individualized instruction. Although all of the respondents to the question replied that 18Reading Recovery should continue at their schools, six percent (6.0%) stated the need for more Reading Recovery teachers at the school, and 6.0% felt greater implementation was needed. Reading Recovery Parent Survey (RRPS) Descriptive results. Generally, parents had favorable attitudes toward the Reading Recovery program. A majority of the parents (90.53%) reported that, because of Reading Recovery tutoring, they believed that their child would be successful in school and 86.32% reported that Reading Recovery had improved their child's reading skills. However, less than 75% (66.32%) of parents strongly agreed or agreed that they have many opportunities to talk with the Reading Recovery teacher about their childs progress. Demographic data. Almost 70% (68.42%) of parents reported the ethnicity of their child as African-American and 13.68% reported their childs ethnicity as Hispanic and 7.37% reported the ethnicity of their child as Caucasian. Open-ended responses. In addition to the six scaled items parents of Reading Recovery students were asked to answer, parents were also asked three open-ended questions regarding the program. There were 80 parents who responded to the first open-ended question, which asked them to describe their understanding of the Reading Recovery tutoring program at their childs school. Overall, parents seemed to have a very good understanding of the Reading Recovery program, and many listed several components that are instrumental in Reading Recovery. Over 80% (83.8%) responded that Reading Recovery helps children read better and improve their reading skills, and 21.3% listed the one-on-one help and individualized attention that the program provides. Children bringing books home and reading at home was mentioned in 13.8% of the responses and improvement in writing skills was listed in 8.8% of the responses. Only 2.5% of those who responded indicated that they did not know or understand Reading Recovery. Reading Recovery Level of Participation and Program Measures African American students were in the majority in all of the 18 schools in the study. Not surprisingly, at 75.2% of the total student population, African American students were also a majority of the Reading Recovery students in the schools. In 11 of the 18 schools, the percentage of African American students in Reading Recovery exceeded their percentage of the school wide population. 19How meaningful this difference is may be debatable given instances where 100% of the Reading Recovery students are African American in a school which is 99% African American (Rightsell Incentive Elementary) or where there are just 8 Reading Recovery students in a school with more than 100 first grade students (Terry Elementary School). Reading Recovery End of Program Status by Race A comparison of the total African American Reading Recovery student population to the total other students involved in Reading Recovery indicates that the students were nearly equally represented in two of the three program specific measures. The percentage of African American students Discontinued (43.3%) was not significantly different from the percent of other students Discontinued (46.3%). In addition, in the Incomplete status, the percentage of African American students (21.3%) was not very much different than that of the other students (25.9%). Only in the Recommended status did the percentage of African American students considerably exceed the percentage of other students (34.5% vs. 27.8%). Reading Recovery Year End Reading Group by Race With-in school comparisons are again difficult due to the unequal number of African American students compared to others student participating in the program. On an overall basis however, the percentage of African American students placed in the high/upper-middle group at 25.7% was significantly lower than the percent of other students placed in this group (38.2%). In addition. almost three quarters of the African American students were placed in the low/lower-middle group compared to less than two thirds of other students. STUDENT ACHIEVEMENT RESULTS DRA Subtests. A. total of 142 Reading Recovery students (66% pretest-posttest match rate) and 562 comparison students (65% rate) had matching 2005 DRA subtest scores, demographic information, and 2004 DRA factor scores. Wilks lambda indicated significant multivariate effects for Reading Recovery status (Fejaso = 6.83, p < .001), special education status (F^^so = 3.93, p <.01), and 2003-2004 DRA factor scores (= 111 -32, p <.001). Follow-up univariate tests indicated 20significant Reading Recovery status effects on Dictation (F2,693 = 6.34, p <.01) and Test scores (^2,693 = 9.99, p < .001). Post hoc analyses revealed that: (a) Reading Recovery students in both the Incomplete Program {M = 36.32
ES =+0.43) and the Complete Program (W = 35.96
S =+0.37) had a significantly higher adjusted mean Dictation score than students in the comparison condition (M = 33.82)
and (b) students in the comparison condition (M = 17.64) and the Complete program {M= 16.42
S = -0.18) had a higher mean Test score than students receiving the Incomplete Program (W = 13.02
ES = -0.68). No program X race interaction effect occurred, indicating that African American and other students were equally affected by participation in Reading Recovery. DIBELS Subtests. total of 67 Reading Recovery students (31 %) and 53 comparison students (28%) had matching 2005 DIBELS subtest scores, demographic information, and 2003-2004 DIBELS factor scores. Wilks lambda indicated significant multivariate effects for Reading Recovery status (Eg,212 = 4.12, p < .001), special education status ( E4JO6 = 3.50, p <.01), and 2003-2004 DRA factor scores ( F4JO6 = 3.69, p <.01). Follow-up univariate tests indicated significant Reading Recovery status effects on Phoneme Segmentation Fluency (F2,io9 = 3.39, p <.O5) and Oral Reading Fluency (F2.109 = 6.59, p < .01). Post hoc analyses revealed that: (a) Reading Recovery students in both the Incomplete Program {M = 54.81
ES = +0.65) and the Complete Program (M = 54.02
ES = +0.58) had a significantly higher adjusted mean Phoneme Segmentation Fluency score than students in the comparison condition {M = 47.23)
(b) students in the comparison condition {Ivf = 40.35) and the Complete program (1^= 33.45) had a higher mean Oral Reading Fluency score than students receiving the Incomplete Program (M = 21.62
ES = -0.91)
and (c) students in the comparison condition had a higher adjusted mean Oral Reading Fluency score (M = 40.35) than students receiving the Complete program (M = 33.45
ES =-0.33). No program X race interaction effect occurred, indicating that African American and other students were equally affected by participation in Reading Recovery. Observation Survey: Reading Recovery Subtests. A total of 161 Reading Recovery students (75%%) and 90 comparison students (47%) had matching 2004-2005 Reading Recovery subtest scores, demographic information, and 2003-2004 Reading Recovery factor scores. Wilks lambda indicated significant multivariate effects for Reading Recovery status (Fio,474 = 4.85, p < .001), 21special education status (Fs ^s? = 3.09, p <.01), 2003-2004 Reading Recovery factor scores (Fs^s? = 22.02, p <.001), and gender {F5,237 = 4.92, p <.01). No program X race interaction effect occurred. indicating that African American and other students were equally affected by participation in Reading Recovery. Follow-up univariate tests indicated significant Reading Recovery status effects on Concepts About Print (F2,241 = 11.87, p <.001), Hearing and Recording Sounds in Words {F2,24-1 = 3.19, p < .05), and Text Reading Levei (F2,24i = 5.38, p < .01). Post hoc analyses revealed that: (a) Reading Recovery students in the Complete Program {M = 20.00
ES = +0.83 ) had a significantly higher adjusted mean Concepts About Print score than student in the comparison condition {M = 17.08) or the Incomplete Program {M = 18.52
ES = +0.41)
(b) students in the Complete program (M = 34.79
ES = +0.36) had a higher mean Hearing and Recording Sounds in Words score than students in the comparison condition (M = 32.77)
and (c) students in the comparison condition {M = 13.67) and the Complete program (M= 13.73) had a higher mean Text Reading score than students receiving the Incomplete Program (M = 10.28
ES = -0.53). ITBS Reading NCE. A total of 140 Reading Recovery students (65%) and 562 comparison students (65%) had matching 2004-2005 Reading Recovery subtest scores, demographic information, and 2003-2004 DRA factor scores. The ANCOVA indicated statistically significant effects for Reading Recovery status (F2,69i = 6.62, p <.001), free lunch status (Fi gg, = 7.83, p <.01), and 2003-2004 DRA factor scores (Fi.esi = 195.81, p <.001). No program X race interaction effect occurred, indicating that African American and other students were equally affected by participation in Reading Recovery. Post hoc analyses showed that comparison students (M = 53.82) had a significantly higher adjusted mean ITBS Reading NCE score than students receiving the Complete program (W = 46.65
ES = -0.34). The effect size for African American students receiving a complete program was -0.46, versus -0.09 for those receiving an incomplete program. Exploratory and supplemental results. Exploratory analyses of second and third grade results showed no effects on 2004-2005 DRA and negative effects on ITBS. These results need to be viewed with caution, however, due to low matching rates in second grade and the lack of a true pretest measure. There was no relationship between number of teacher years of experience with Reading Recovery and ITBS standardized residuals or DRA standardized residuals for students 22receiving a complete program, after controlling for 2003-2004 DRA factor scores and student ethnicity. gender, free lunch status, special education status, and LEP status. Likewise, there was no relationship between number of sessions attended and ITBS residuals. A statistically significant, small positive relationship was observed between total number of sessions attended and DRA residuals (r = 0.21, p <.O5). The median effect size estimate across all posttests for African American students with a complete program was Md = +0.17, with a range from -0.25 to +0.52. For African American students receiving an incomplete program, effect size estimates ranged from -0.78 to +0.50, with a median of -0.23. Thus, receiving a complete program yielded a directional advantage for African American students, whereas the reverse occurred for receiving an incomplete program. CONCLUSIONS Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? The Reading Recovery program had equal effects on African American and other students. Students receiving the complete program had significantly higher adjusted means than comparison students on Dictation {ES = +0.37), Phoneme Segmentation {ES = +0.58), Concepts About Print (ES = +0.83), and Hearing and Recording Sounds in Words (ES = +0.36) Students receiving an incomplete program had significantly higher adjusted means than comparison students on Dictation (ES = +0.43) and Phoneme Segmentation (ES = +0.65). Students in the comparison condition had significantly higher adjusted means than students receiving a complete program on Oral Reading Fluency (ES = -0.33) and on ITBS Reading NCE scores (ES = -0.34). Students in the comparison condition had significantly higher adjusted means than students receiving an incomplete program on DRA test scores (ES = -0.68), Oral Reading Fluency (ES = -0.91), and Text Reading (ES = -0.53). 23No relationship was observed between teacher experience with Reading Recovery and 2004- 2005 student achievement outcomes after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. No relationship was observed between number of Reading Recovery sessions and 2004- 2005 ITBS Reading NCE scores, after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. A small but statistically significant positive relationship was observed between number of Reading Recovery sessions and 2004-2005 DRA test scores, after controlling for 2003-2004 achievement, student ethnicity, gender, free lunch status, special education status, and LEP status. The median effect size estimate across all posttests for African American students receiving a complete program was +0.17
for students receiving an incomplete program, the median effect size estimate was -0.22. Positive effects of Reading Recovery tended to be associated with lower-order or beginning reading skills like dictation, phoneme segmentation, and concepts about print while negative effects tended to be associated with more complex, higher-order skills like Oral Reading Fluency and Text Reading. What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005. Classroom observations indicate that Reading Recovery teachers instructional practices conform to the recommendations and requirements of the program throughout the district. Given that there are no national comparisons or benchmarks for the RRIAI, a mean of approximately 3.50 on a 4.00 scale suggests a high level of Reading Recovery implementation in the district. Site researchers noted only three areas in which some teachers were observed below average to some degree, reading familiar stories, appropriate pacing of the lesson components, and working with letters and or/words. However, the observed lack of quality implementation in some classrooms in reading familiar stories and appropriate pacing of the lesson components might 24begin to explain the lack of oral reading fluency, text reading, and ITBS effects for Reading Recovery students. There appears to be a high level of consistency of program delivery across the district. The analysis of student achievement data supports this observation that there is consistency in program delivery. The student achievement analysis found that there was no relationship between teacher experience with Reading Recovery and 2005 achievement scores after controlling for 2004 achievement and other variables. This suggests that generally teachers have a high degree of fidelity to the model. What is the level of participation in Reading Recovery by African American students relative to other ethnic groups at the school? The data indicate that African American students made up a majority of the students participating in Reading Recovery in the 18 schools included in this study. This finding shouldnt be surprising since African Americans are the majority of the students in each of the participating schools. Information compiled from the student achievement analyses also indicates the percentage of African American students receiving Reading Recovery services (75.2%) is very similar to the ethnic makeup of the students used for comparison purposes (73.6%). However, Reading Recovery students were more likely to receive free or reduced-price lunch (84.3% vs. 73.1%) and special education services (15.7% vs. 8.6%) than comparison students, and were more likely to be male (58.1% vs. 48.0%). What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? Table 11 indicates that African American students, when compared with Reading Recovery students of other ethnicities, were nearly equally represented in two of the three program specific measures examined. The percentage of African American students Discontinued", at 43.3%, 25was not considerably different than the percentage of students Discontinued of other ethnic backgrounds, 46.3%. In addition, in the Incomplete status, the percentage of African Americans (21.3%) was, again, not much different than students from other ethnicities (25.9%). African American students, at 34.5%, were more likely to be Recommended" for further actions than other students (27.8%). African American students were, however, more likely to be placed in the Low/Lower Middle reading group by their classroom teachers at the end of the school year than other students (74.3% vs. 61.8%). This finding may present a dilemma for the program and the district. While African American students are generally progressing similarly to other students on program specific measures, their classroom teachers appear to consider that, at the end of the school year, the majority of African American students are still struggling to maintain or falling below grade level in reading. About 22% of scheduled sessions were missed due to the teacher being unavailable (7%), student absence (6%), teacher absence (5%), or the student being unavailable. These missed sessions could contribute to the mean number of sessions per week being just 3.5. What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? Reading Recovery teachers had extremely favorable attitudes toward the Reading Recovery program. Teachers reported a thorough understanding of the program, that they received adequate professional development which was valuable for improving the achievement of African- American students, and that they had the support from teachers in the school. The majority of Reading Recovery teachers also reported receiving extensive administrative, Reading Coach, and district support. The items and areas of the most concern was sufficient planning time. enough tutors to fully implement the program, and time to routinely monitor first grade students progress after they were discontinued from Reading Recovery tutoring. Additionally, Reading Recovery teachers were concerned that only 63.64% of faculty, staff, and administration believe that all children can read at grade level or above by the end of third grade thanks to the Reading Recovery, and that parents are more involved in the literacy program of this school. 26Reading Recovery teachers, on average, appear to be more experienced and better educated. Eighty percent had a Masters degree or beyond in educational attainment and 100% reported at least six year or more years of teaching experience. In addition, the majority (68.18%) reported one to five years of experience as a Reading Recovery teacher. The four teachers in-training were equally committed and positive about the program and overall, felt they were well prepared to work with students. The teachers in-training also felt strongly that Reading Recovery helped to equalize learning and achievement opportunities for African American students. Teachers in-training emphasized the importance of using data to monitor the progress of the students to develop effective teaching strategies based on the individual needs of each student. Finally, teachers in training also reported the need for more time to plan and implement as well as for continuing support to understand Reading Recovery components more thoroughly. What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? A review of Reading Recovery principal interview responses indicates that principals are very supportive and actively involved in the program. All of the principals interviewed reported that they understood the program and were advocates of their program having a positive impact on overall student achievement. Principals indicated that the one-on-one tutoring program supplements and enhances the schools balanced literacy program. Most principals agreed that. through Reading Recovery, the achievement gap is being bridged for their African American students. Principals also noted that teachers were very supportive of the program. K-3 classroom teachers shared the principals enthusiasm for the Reading Recovery program, as evidenced by responses on the RRCTQ. A majority of the classroom teachers reported that they had an understanding of the program, were generally supportive, and that student achievement had been positively impacted. Principals and teachers also agreed that most of the resources and support needed for effective program implementation was available
however they also reported an ongoing need for additional teachers and tutors to support more 27students and time to plan, review student progress, and collaborate together. All (102) of the teachers responding agreed that their school should continue the Reading Recovery program. What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? Parents were generally very pleased with the results of the Reading Recovery program. Approximately 90% of the parents responding to the parent survey agreed that
Reading Recovery tutoring has improved my childs reading skills and because of Reading Recovery their child will be successful in school. Less than three percent of those who responded reported that they did not know or understand the program. In the three open-ended responses parents indicated a very good understanding of the program, appreciation of one-on-one tutoring sessions, and the improvement in their childs reading skills. However, a few parents did express the need for longer and more frequent tutoring sessions and more opportunities to talk with the Reading Recovery teacher about their childs progress. In summary, the Little Rock School District has a strong Reading Recovery program. Teachers, principals, and parents appear to be actively engaged in the program and the district tries to provide adequate levels of resources and support. However, the lack of clear program effects may be the result of factors that have been identified in prior studies of Reading Recovery. These factors include: It would be expected that Reading Recovery students would perform better on assessments more closely aligned with the instructional program (DRA, DIBELS, Observation Survey) than the norm-referenced group administered ITBS. In particular, Reading Recovery enhanced learning for complete program students tests involving Dictation, Phoneme Segmentation, Concepts About Print, and Hearing and Recording Sounds in Words. However, it had no effects on tests assessing Oral Reading Fluency and ITBS Reading NCE scores. The district should examine the feasibility of providing tutoring support to all incoming first grade students who need services. In the 2004-2005 school year, the 18 Reading Recovery schools indicated that 365 students needed tutoring services and 28approximately half of this number were discontinued and received a complete round of lessons. The Reading Recovery program guidelines state that if a school has more children who need services than one teacher can provide then it will never realize the full benefit of Reading Recovery for later school achievement. It is especially difficult for teachers to continue to scaffold discontinued students learning while supporting a large number of other at-risk students reading below grade level. This may explain why so many Reading Recovery students are placed in the low/lower middle reading groups by their classroom teachers. The data suggest that after Reading Recovery students are discontinued and return to the classroom at the same reading level as their peers, it appears that they do not maintain the same growth rate and achievement does not keep pace with their peers. Although research indicates that former Reading Recovery students perfomi well in their classes, some slippage in achievement can occur (Clay, 1993). Although Clay (1993) provides guidelines for transition back to the classroom after the student is discontinued
it is possible that students were returned to the classroom without benefit of a transition plan. As noted by Reading Recovery teachers, few teachers have the opportunity to routinely monitor discontinued students progress. Also in tutoring sessions, children have opportunities to read texts at their instruction level on a daily basis, but they may not have adequate time for daily reading in the regular classroom. The quality of instruction that Reading Recovery students receive once they return to the classroom is an important factor that was not examined in this study. Additional research could provide critical insight into the optimum classroom environment for discontinued, recommended, and incomplete Reading Recovery students. 29EVALUATION OF READING RECOVERY IN THE LITTLE ROCK SCHOOL DISTRICT: 2005 AGGREGATE REPORT INTRODUCTION The Little Rock School District began implementing Reading Recovery during the 1995-1996 school year in two schools. During the first four years of implementation, the Little Rock School District was part of the Pulaski County Reading Recovery Site. By 1998 there were eight trained Reading Recovery teachers in seven schools in the Little Rock School District. In July 1999 the district became a Reading Recovery Site hiring a full-time teacher leader. In 2000-2001 the University of Arkansas at Little Rock (UALR) began conducting the Reading Recovery teacher training for the district. Between the 2000-2001 and 2004-2005 school years UALR trained seventeen Reading Recovery teachers for the district. At the end of the 2004-2005 school year the Little Rock School District had 28 trained Reading Recovery teachers serving 18 of the 34 elementary schools in the district. Reading Recovery is one of eight literacy programs, interventions, and/or models used by Little Rock schools. Currently, Little Rock School District funds are used to support the program. The goal of Reading Recovery is to dramatically reduce the number of first grade students who have difficulty learning to read and write and to reduce the cost of these learners to educational systems. Reading Recovery is a short-tenn intervention program of one-to-one tutoring for the lowest-achieving first graders. The intervention is most effective when it is available to all students who need it and is used as a supplement to good classroom teaching. Individual students receive a half-hour lesson each school day for 12 to 20 weeks with a specially trained Reading Recovery teacher. As soon as students can read within the average range of their class and demonstrate that they can continue to achieve, their lessons are discontinued, and new students begin individual instruction. Reading Recovery was developed by New Zealand educator and researcher. Dr. Marie M. Clay over 20 years ago. More than one million first graders have been served in 49 states since Reading Recovery was introduced in the United States in 1984. Professional development is an 30essential component of the Reading Recovery program. Training utilizes a three-tiered approach that includes teachers, teacher leaders, and university trainers. In schools, special trained teachers work with children. At the site level, teacher leaders work with children, train teachers, and assist and monitor implementation with the help of a site coordinator. In university training centers, trainers work with children, train teacher leaders, engage in research, and support program implementation at affiliated sites. Professional development for teachers and teacher leaders begins with year-long graduate level study and is followed by ongoing training in succeeding years. Since 1984, the program reports that 80% of students who completed the full 12 to 20 week series of lessons, and 59% all students who have any lessons in Reading Recovery, can read and write with the average range of performance of their class. Program follow-up studies also indicate that most Reading Recovery students also do well on standardized tests and maintain their gains in later years. The evaluation plan for the Reading Recovery program in Little Rock School District included: (1) analyses of Reading Recovery student achievement and program data, (2) principal, teacher, and parent surveys and interviews, and (3) observations of Reading Recovery tutoring sessions. This report is part of a larger district study of four programs evaluating the effectiveness in improving and remediating the academic achievement of African-American students. This report includes results from 18 elementary schools participating in the Reading Recovery program. RESEACH QUESTIONS The major goals of this research study were to evaluate African-American student achievement outcomes, program implementation fidelity, and principal, teacher, and parent perceptions concerning the Reading Recovery tutoring program for first grade students. Student achievement results on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and An Observation of Early Literacy Achievement were analyzed to compare the progress of first graders enrolled in the Reading Recovery intervention program and comparison students in 2004-2005. Program implementation ratings were obtained from observations of 14 tutoring sessions in nine schools. The survey and 31interview results are based on the perceptions of 156 classroom teachers in grades K-3,22 experienced Reading Recovery teachers, four teachers in-training, 10 principals, and 95 parents. The Reading Recovery evaluation was structured around the following seven primary and supplemental research questions: Primary evaluation question: Has the Reading Recovery program been effective in improving and remediating the academic achievement of African-American students? Supplemental (Qualitative/Step 2) evaluation questions: What are the quality and level of implementation of Reading Recovery at the 18 schools implementing it in 2004-2005? What is the level of participation in Reading Recovery by African American students relative to other ethnic groups at the school? What is the progress demonstrated by African-American and other student participants in Reading Recovery in improving achievement, as demonstrated on program-specific measures? What percentage of students are discontinued or not discontinued? What proportion of scheduled sessions are actually held, and what are the reasons for missed sessions? What are the perceptions of Reading Recovery teachers regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of principals, regular first-grade teachers, and other teachers in the school regarding Reading Recovery program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of Reading Recovery students regarding program impacts, strengths, and weaknesses? 32EVALUATION DESIGN AND MEASURES The evaluation period extended from February 2004 through May 2005. The evaluation design was based on both quantitative and qualitative data collected from Reading Recovery intervention observations, principal and teacher in-training interviews, classroom teachers, Reading Recovery teachers, and parent surveys, and Reading Recovery program data. Reading Recovery student-level achievement data on the ITBS, DRA, DIBELS, and An Observation of Early Literacy Achievement was received in Fall 2005 and incorporated in this report. The primary data collectors in this study were Center for Research in Educational Policy (CREP) trained site researchers. Site researchers: (1) conducted Reading Recovery intervention observations, (2) administered the Reading Recovery teacher survey, (3) conducted principal and teacher in-training interviews, and (4) collected program data. Principals at Reading Recovery schools were responsible for administering the classroom teacher and Reading Recovery parent surveys. Participants Little Rock School District is located in Central Arkansas and serves approximately 26,500 students, with African-Americans representing approximately 67% of the district student population, in 49 schools in an urban area with a population of 184,000. In the 2004-2005 school year 18 elementary schools and 230 first grade students, of which 173 are African American, 27 are white, 22 are Hispanic, and the remaining 8 are other ethnicities, participated in the Reading Recovery program in the district. However, the schools indicated that 365 first grade students needed Reading Recovery services. Three schools were in their first year of program implementation (Bale, Stephens, and Terry) and survey and observation data were not collected from these schools. However, randomly selected teachers in-training and principals from these three schools were interviewed for this study. A profile of the Reading Recovery schools and participants included in this study is shown in Table 1. The profile data were obtained from either the 2003-2004 Common Core of Data from the National Center for Education Statistics, the 2004-2005 Reading Recovery Site Report for Little Rock from the National Data Evaluation Center at The Ohio State University, or provided by the district. As indicated in Table 1 the number of years schools have implemented the Reading Recovery program ranged 33from one to 10. The Reading Recovery schools were predominately African-American, ranging from 50% of the student population to 99%. The district reported that four Reading Recovery schools did not receive Title I funding and the percentage of students eligible for free and reduced price lunch ranged from a low of 33% to a high of 94% at all Reading Recovery schools. Table 1 Reading Recovery Participating Schools: 2004-2005 School Name Bale Elementary School Booker Aris Magn^ES Carver Magnet Elem School Chicot Eiem^ool David O'Dodd Elem School Franklin Incentive Elem School C^er Springs ElemSctwol Gibbs Magnet Elem School tvie^cwcliff Elem School Mitohell Incentive Elem School Otter Creek Elem School Rights^ Incentive Bern School Stephens Elem School terry Elem School Wakefield Elem School Watson Elem School VWIiams Magnet Elem School Wilson Elem School School Wide Population Students Teacheis Asian 319 605 496 536 261 387 299 310 349 156 511 262 499 577 451 456 461 285 27 55 43 44 27 35 23 30 24 22 31 25 39 36 29 34 36 27 0% 1% 3% 0% 0% 0% 0% 4% 1% 0% 1% 0% 1% 7% 0% 0% 9% 1% School Wide Student Population African American 87% 55% 54% 75% 58% 97% 89% 52% 77% 96% 56% 99% 95% 51% 83% 95% 52% 89% Hispanic White 7% 5% 4% 3% 15% 10% 0% 4% 3% 9% 0% 6% 0% 3% 5% 15% 3% 1% 4% 40% 40% 10% 32% 2% 7% 41% 13% 2% 36% 1% 2% 36% 2% 2% 38% 6% % Free and Reduced Lunch 88.4% 63.3% 53.0% 85.6% 68.9% 943% 80.6% 43.9% 85.1% 91.7% 55.7% 87.8% 90.6% 47.5% 920% 932% 33.6% 91.9% Reading Recowiy Participant tifonnation % Below Pralicienr Number of RR Teachers Number of K-3 Number of RR Years in RR Teachets Students Program 45% 22% 18% 38% 30% 51% 48% 11% 44% 59% 19% 49% 59% 12% 33% 64% 10% 35% 10 20 32 16 17 12 14 18 18 13 15 13 16 26 10 18 14 18 10 13 10 1 8 1 4 7 2 3 3 5 1 2 1 2 1 1 1 1 1 1 2 1 2 1 8 9 7 8 8 7 8 6 9 8 3 5 8 2 8 8 8 9 9 8 8 3 1 1 3 3 9 5 Proficiency levels are based on 2003-2004 school year ACTAAP Grade 4 Reading, Language, and Writing data. 34Instrumentation Six instruments were developed by CREP to collect the evaluation data: a classroom observation tool, a principal interview, a teacher in-training interview, a Reading Recovery teacher questionnaire, a classroom teacher questionnaire, and a parent survey. A detailed description of each instrument follows. Classroom Observation Measure Reading Recovery Implementation Assessment Instrument (RRIAI) The RRIAI was designed for observation in Reading Recovery classrooms and was developed by researchers at CREP and Reading Recovery faculty at ULAR. Ratings are organized around two categories: Reading Recovery Program Components and Reading Recovery Program Strategies. The RRIAI observation tool provides an overall rating for each category based on a rubric that ranges from (1) poor or unacceptable
(2) below average in comparison to other programs observed
(3) meets nearly all standards of program quality
and (4) above average in comparison to other programs. The RRIAI has been aligned to the essential components of the Reading Recovery program. Sub-categories of the program components include: reading familiar stories, reading a story that was read for the first time the day before, working with letters, writing a story, assembling a cut-up story, and introducing and reading a new book. The tool was designed to be utilized by experienced Reading Recovery trainers during the 30-minute tutoring session with additional time allocated for observer questions and examination of student records after the session. Site observers received observation protocol training in February 2005 and a copy of the observation guidelines are included in Appendix A. Site observers were also asked to provide overall perceptions of Reading Recovery program implementation in the Little Rock schools. Interviews Reading Recovery Principal Interview Randomly selected principals from 10 Reading Recovery schools participated in approximately 45-minute phone interviews with CREP researchers. Interview questions addressed 35the principals general experiences and reactions to the Reading Recovery implementation and the associated outcomes for the school, students, teachers, and parents. A copy of the principal interview protocol with summary responses is included in Appendix B. Reading Recovery Teacher in Training Interview In the 2004-2005 school year, six teachers were beginning their first year as a Reading Recovery teacher and considered as teachers in-training. Three were located in experienced schools and three were located in the schools beginning their first year of implementation: Bale, Stephens, and Terry. Four randomly selected teachers in-training participated in an approximately 45-minute phone interview with CREP researchers. Interview questions addressed the teachers in-training experiences with and perceptions of the Reading Recovery program implementation with regard to such areas as resources, professional development, parent involvement, support, and student outcomes. A copy of the teacher in-training interview protocol with summary responses is included in Appendix C. Surveys Reading Recovery Teacher Questionnaire (RRTQ) Reading Recovery teachers in the 15 experienced Reading Recovery schools were asked to complete the RRTQ, which includes four sections. Section I contains 20 items to which teachers respond using a five-point Likert-type scale ranging from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the general impressions of the Reading Recovery program, professional development, support for the program, impacts on student achievement, and alignment of state and district reading and language arts standards. In a second section, teachers were asked to indicate to what degree the listed items occurred. The focus of the 4 items were (1) administration support of Reading Recovery teacher efforts
(2) Reading Coach support of Reading Recovery teacher efforts
(3) district support of Reading Recovery teacher efforts
and (4) the time to routinely monitor first grade students progress after they were discontinued from tutoring. Open-ended questions asked Reading Recovery teachers to respond to the statements: What are the strongest aspects of the Reading Recovery program? What are the weakest aspects of the Reading Recovery program? Do 36you think your school should continue the Reading Recovery program ? A final section of the RRTQ contained general questions regarding years of teaching experience, race, and level of education. Reading Recovery Classroom Teacher Questionnaire (RRCTQ) All K-3 teacher who taught in experienced Reading Recovery schools were asked to complete the RRCTQ, which contains 13 items to which teachers respond using a five-point Likert- type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed the specific program elements of Reading Recovery such as: understanding of the program, impacts on student achievement, teacher support for the program, and parent involvement. Open-ended questions asked classroom teachers to respond to the statements: Please describe your understanding of the Reading Recovery program in your school. What are the strongest aspects of the Reading Recovery program? Do you think your school should continue the Reading Recovery program? /\ final section of the RRCTQ contained general questions regarding years of teaching experience, race, and level of education. Reading Recovery Parent Survey (RRPS) Parents whose first grade children were currently receiving Reading Recovery tutoring services were asked to complete the RRPS, that contains 6 items to which parents respond using a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Items assessed general impressions of the program such as: improvement in childs reading skills, childs enjoyment of tutoring sessions, and opportunities to communicate with the Reading Recovery teacher. Both English and Spanish versions were made available to schools. Open-ended questions asked parents to respond to the statements: Please describe your understanding of the Reading Recovery tutoring program at your childs school. What is the BEST thing about your child's involvement with the Reading Recovery tutoring program? What CHANGES would you like to see in the Reading Recovery tutoring program? A final section of the RRPS asked parents to indicate the race, grade, and age of their child. 37Program Data Reading Recovery program information was obtained from data submitted to the National Data Evaluation Center at The Ohio State University in 2004-2005. Each year the schools and district submit data and receive a site report. The report represents an examination of Reading Recovery student outcomes for Little Rock and accounts for all children served by Reading Recovery within the site during the 2004-2005 school year. In addition, attention is given to implementation factors that may be supporting or hindering the success of the intervention with the site. Student Achievement Results In addition to the program data, interviews, survey, and observation tools cited above, reading achievement data is derived from scores on the Iowa Test of Basic Skills (ITBS), Developmental Reading Assessment (DRA), Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and An Observation of Early Literacy Achievement. A description of each assessment follows. Iowa Tests of Basic Skills (ITBS). The ITBS Form A is a norm-referenced group administered test that measures the skills and achievement of students in grades K-8 and was developed at The University of Iowa. The ITBS provides an in-depth assessment of students achievement of important educational objectives. Tests in Reading, Language Arts, Mathematics, Social Studies, Science, and Source of Information yield reliable and comprehensive information both about the development of students skills and about their ability to think critically. The emphasis of the K-1 assessment is on academic skills found in the early childhood curriculum. These tests are neither measures of readiness for school nor readiness to read. Rather, they assess the extent to which a child is cognitively prepared to begin work in the academic aspects of the curriculum. Test materials have been extensively field tested for psychometric soundness and evaluated for fairness to gender. racial, ethnic, and cultural groups. Developmental Reading Assessment (DRA). The Developmental Reading Assessment provides teachers with a method for assessing and documenting primary students development as readers over time with a literature-based instructional reading program. The DRA is designed to be used in K-3 classrooms with rich literate environments. The assessments are conducted during one- 38on-one reading conferences as children read specially selected assessment texts. A set of 20 stories. which increase in difficulty, are used for the assessment. The DRA evaluates two major aspects of reading: accuracy of oral reading and comprehension through reading and retelling of narrative stories. Questions pertaining to concepts about print are also included in the assessment with lower leveled texts. Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS is a tool for early identification of children with potential literacy problems and an assessment of response to instruction. The assessment is individually administered to K-3 children at least three times per year. The DIBELS assessment is designed to enable educators to modify their approach if a student is not on course to achieve reading goals. The Early Childhood Research Institute on Measuring Growth and Development at the University of Oregon constructed DIBELS (2000). The Institute reports it has validated the instruments ability to predict outcomes and has tested its reliability with young children across the country. Reading Recovery schools that have received Reading Excellence Act or Reading First grant funding use the DIBELS assessment in Little Rock. An Observation of Early Literacy Achievement: Reading Recovery Subtests. Six tasks in Marie Clays (2002) An Observation of Early Literacy Achievement (Observation Survey) were used as pretest and posttest measures. The Survey tasks have the qualities of sound assessment instruments with reliabilities and validities. The purpose of the assessment is to determine an appropriate level of text difficulty and to record, using a running record, what the child does when reading continuous text. All six tasks of the Observation Survey were administered to Reading Recovery students in the fall (start of the school year) and/or at entry to the intervention. These scores serve as pretest measure in the evaluation design. The six tasks were also administered to Reading Recovery students upon discontinuing or exiting from the program. In the spring (end of school year), the six tasks were again administered to all students who received Reading Recovery services during the year. Year-end scores served as the posttest measures in comparing the progress made by Reading Recovery children in the various end-of-program status groups. 39PROCEDURE Data for the evaluation were collected March-May for the 2004-2005 school year. On February 16, 2005 principals were given an overview of the evaluation and timelines for collecting data. On March 3-4 and April 14-15,14 tutoring observations were conducted in nine randomly selected experienced Reading Recovery schools by two Reading Recovery content experts from Georgia State and The Ohio State Universities. Only experienced Reading Recovery teachers were observed. On March 23 a CREP trained site researcher administered the Reading Recovery teacher survey to experienced teachers at a regularly scheduled monthly meeting at ULAR. In April and May principals in experienced Reading Recovery schools administered the classroom teacher questionnaire to K-3 grade teachers. Principals also administered the parent survey to parents whose children were currently receiving first grade intervention services in the 15 experienced Reading Recovery schools. Four teachers in-training and ten principals were randomly selected from the 18 Reading Recovery schools to participate in a phone interview in April and May conducted by CREP researchers. Reading Recovery program data was received in summer 2005 from the National Data Evaluation Center at The Ohio State University. Student achievement data was received from the district in fall 2005. METHODS - STUDENT ACHIEVEMENT Sample. The sample included 1,094 first grade students who attended one of 18 schools that implemented the Reading Recovery program during the 2004-2005 school year. Of these, 230 were referred to the Reading Recovery program, and 864 were in the comparison group. The percentages of students who were African American or of Limited English Proficiency were similar between the comparison and Reading Recovery groups (73.6% versus 75.6% and 7.3% versus 8.7%, respectively
see Figure 1). However, Reading Recovery students were more likely to be eligible for free or reduced-price lunch (84.3% versus 73.1%), to receive special education services (15.7% versus 8.6%), and to be male (58.1 % versus 48.0%
see Figure 1). A random sample of about 11 comparison students was selected from each school (total n = 189) for administration of the DIBELS and Observation Survey: Reading Recovery test batteries. 40Figure 1. Percentage of Students with Selected Demographic Characteristics by Reading Recovery Status 100.0 -I 90.0 - 84.3 -i 80.0 - 73.6 75.6 " 73.1 70.0 - 60.0 - 40.0 - 48.0 58.1 50.0 - 30.0 - 20.0 10.0 - 7.3 15.7 8.6 0.0 African American Free lunch LEP Special Education Male J j Comparison Reading Recovery Reading Recovery Treatment Level. In the Reading Recovery program, students who are deemed to have attained a reading level equivalent to their peers are assigned an end-of-program status of discontinued." Children who received 20 or more weeks of Reading Recovery services, but who do not attain a reading level equivalent to their peers are assigned a status of recommended action. Children who have received fewer than 20 weeks were assigned incomplete program status. Children designated as unknown status were removed from the program in fewer than 20 weeks due to reasons other than the school year ending. Other children were designated as having moved during the school year. As shown in Figure 2, the median number of sessions of recommended action {Md = 68.50
n = 68) children was actually higher than the median for discontinued students {Md = 57.00
n = 90). The medians for incomplete students {n = 46) and unknown students {n = 12) were very similar {Md = 44.63 versus 46.08, respectively
see Figure 2). For the purposes of this study, recommended action and discontinued students were categorized as complete program
incomplete and unknown students were categorized as incomplete 41program
and students who moved were eliminated from the analyses. This left a total of 216 Reading Recovery students: 158 with a complete program, and 58 with an incomplete program. Figure 2. Boxplot of Number of Reading Recovery Sessions by Student Status Upon Exiting the Program 80 c o w 00- V) V) a 2 cr O 40- o o 20- O O o * DisiJuntinueiJ HovhU I I )nl nown Pricominrjndrid a<
tion T T T T lni
uin|:'He Status Note. Heavy dark lines indicate median. Gray boxes indicate interquartile range. Whiskers indicate range, excepting extreme values. Extreme values denoted by circles or asterisks. 42Measures. Pretest (covariate) measures included: (a) Spring, 2004 Developmental Reading Assessment (DRA) subtests in Letter Recognition, Word Recognition, Capitalization, Writing, Dictation, and a DRA Test score, (b) Fall 2005 Observation Survey: Reading Recovery program subtests that included Letter Identification, Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds in Words, and Text Reading
and (c) the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in Letter Naming Fluency, Phoneme Segmentation Fluency, Nonsense Word Fluency, and Word Use Fluency. Outcome measures included Iowa Test of Basic Skills (ITBS) Reading Normal Curve Equivalent Scores, DRA subtests. Observation Survey: Reading Recovery subtests, and DIBELS subtests administered in Spring 2005. To achieve some parsimony in the analyses, pretest measures were subjected to a principal components analysis, and regressionbased factors scores were constructed for pretest DRA, Observation Survey: Reading Recovery, DIBELS subtests. A single factor accounted for 60.1%, 60.5%, and 43.8% of the variance in the DRA, Observation Survey: Reading Recovery, and DIBELS subtests, respectively. Outcome measures included the DRA, Observation Survey: Reading Recovery, and DIBELS subtests, as well as ITBS. The DRA Letter Recognition, DRA Capitalization, and DIBELS Letter Naming Fluency subtests were not administered in Spring 2005. DIBELS and Observation Survey: Reading Recovery tests were administered to a small random sample of comparison students within each school. Student achievement analyses. A 3 (Program Status) X 2 (African American versus other) multivariate analysis of covariance (MANCOVA) using student gender, the pertinent pretest factor score, free lunch status, special education status, and LEP status was performed with the various subtests as outcomes for the DRA, Observation Survey: Reading Recovey, and DIBELS test batteries. A similar 3X2 analysis of covariance was performed on ITBS Reading NCE scores using the 2003-2004 DRA factor score as a pretest covariate. For the multivariate analyses, Wilk's lambda was used as the criterion of multivariate significance. Where Wilks lambda indicated a significant multivariate effect, follow-up univariate analyses were performed on each outcome variable using the Bonferroni procedure to control for experimentwise alpha. When significant univariate results were found, post hoc analyses were performed using Scheffes procedure. Effect size estimates were computed for all posttests by subtracting the adjusted mean for the comparison group from the 43adjusted mean from the Reading Recovery group within levels of race, then dividing by the total standard deviation of the posttest. For ITBS Reading NCE effect size estimates, the population standard deviation of 21.06 was used. Exploratory and supplementary analyses: Impact of teacher experience and number of sessions. The number of years experience with Reading Recovery was recorded for each teacher. Years experience ranged from 1 to 10 years. ITBS Reading NCE scores and DRA test scores for 2004-2005 were regressed on 2003-2004 DRA factor scores and dummy-coded variables representing student ethnicity, free lunch status, gender, special education status, and LEP status. Standardized residuals were saved from each of these analyses, and plotted against teacher years of experience to graphically assess the nature of the relationship between teacher experience and teacher effectiveness in Reading Recovery for students who received a Complete program. Standardized residuals for each of these 2004-2005 tests were also plotted against number of Reading Recovery sessions received for all students (i.e., those who received either a Complete or Incomplete program). Finally, effect size boxplots were constructed for African American students who had Complete and Incomplete Reading Recovery programs. DATA COLLECTION Table 2 provides the type of measures, instrument names, administration timeline, and a brief data collection description for each of the instruments. 44Table 2 Data Collection Summary Type of Measure Site Visits Instrument Timeline RR Implementation Assessment Instrument Spring 2005 Description/ Response rate 14 conducted Surveys RR Teacher Questionnaire RR Classroom Teacher Questionnaire RR Parent Survey Spring 2005 Spring 2005 Spring 2005 22 respondents/100% 156 respondents/ approximately 90% Interviews Principal Interviews Teacher In-Training Interviews Spring 2005 Spring 2005 95 respondents/9 Spanish approximately 86% 10 conducted 4 conducted Data Analysis and Reporting ITBS, DRA, DIBELS, & Observation Survey: RR Fall 2005 Little Rock School District Reading Recovery Aggregate Report 1,094 first grade students . 230 RR 864 comparison group 1 Final Report RESULTS The results of the Pleading Recovery evaluation are presented below by instrument. In the Conclusion section, findings are synthesized across instruments to address each research question. Reading Recovery Implementation Assessment Instrument (RRIAI) As indicated in the description of the RRIAI, the observation procedure primarily focuses on Reading Recovery Program Components and Program Strategies. The site observers used a four- point rubric (1 = poor or unacceptable, 2 = below average in comparison to other programs observed. 3 = meets nearly all standards of program quality, and 4 = above average in comparison to other programs) to rate the frequency and application of components and strategies of Reading Recovery instruction. More precise data regarding observed Reading Recovery instructional practices measured by the four-point rubric for the 14 observations are presented in Table 3. 45Reading Recovery Program Components The overall mean rating for the Reading Recovery Program Components was 3.46 which suggest a high level of program implementation. Of the six subcategories the highest observed ratings were
assembling a cut-up story and introducing and reading a new book observed to be above average in 85.71 % of tutoring sessions. The program component subcategory with the lowest observed rating was working with letters and/or words using magnetic letters observed to be above average in only 35.71% of tutoring sessions. Reading a familiar story and reading a story that was read for the first time the day before was observed to above average in 64.29% of visits. The six program components were observed in at least 92.86% of 14 tutoring sessions. Reading Recovery Program Strategies The overall mean rating for the Reading Recovery Program Strategies was 3.61 which also indicate a high level of instructional effectiveness. Of the eight During Tutoring Lesson subcategories the highest observed rating was for appropriate text selected throughout the lesson observed to above average in 85.71% of tutoring sessions. The During Tutoring Lesson subcategory with the lowest observed rating was echo of focus throughout the lesson observed to be above average in just 42.86% of tutoring sessions. For the After Tutoring Lesson, has high expectations for the child and articulates child's strengths and needs were observed to be above average in 78.57% of tutoring sessions and accurate and up-to-date records were observed to be above average in 71.43% of tutoring sessions. The Reading Recovery Program Strategies were observed in at least 85.71 of tutoring sessions. Observer Perceptions of Reading Recovery Program Implementation Site observers reported being impressed with the dedication and commitment of the Reading Recovery teachers to the fidelity of the teaching procedures and the integrity of the implementation of the program. Almost all teachers observed were meeting all the standards, guidelines, and expectations of the Reading Recovery Council of North America and the North American Trainers Group. Since observations occurred in March and April, most of the students observed were second- round students, since teachers had discontinued their first-round students. Teachers reported that students who did not discontinue from Reading Recovery during the first-round were being considered 46for further intervention services. Site observers also reported that the Reading Recovery program in the Little Rock district receives an adequate allocation of time, materials, and other resources. However, several teachers and principals expressed the need for additional Reading Recovery teachers in their schools. Reading Recovery teachers also reported teaching literacy small groups the rest of the day which enables them to give their Reading Recovery students more attention during the instructional day. By teaching these literacy groups, site observers suggested that the Reading Recovery teachers' expertise and knowledge gained from their training and practice benefits children across several grade levels. During visits site observers did suggest areas in which some Reading Recovery teachers need improvement. These instructional areas include: (1) hearing and recording sounds in words
(2) making and breaking
(3) do away with the helping hand
and (4) maintaining up-to-date records on each child as a basis for instruction. A summary of observers general findings for each classroom observation is included in Appendix D. 47Table 3 Reading Recovery Implementation Assessment Instrument: Spring 2005 N=14 Please check: Observed: 0 Not Observed: N Please rate each of the following items in terms of the quality of implementation by using the appropriate number according to the following scales: I. II. Quality 1 = Poor or unacceptable 2 = Below average in comparison to other programs observed 3 = Meets nearly all standards of program quality 4 = Above average in comparison to other programs Observed Not Obsented Reading Recovery Program Components Poor 100.0 0.0 100.0 0.0 92.86 7.14 Reading familiar stories Reading a story that was read for the first time the day before - incorporates ronning record Working with letters and /or words using magnetic letters 0.0 0.0 0.0 92.86 7.14 Writing a story 0.0 92.86 7.14 Assembling a cut-up story 0.0 92.86 7.14 Introducing and reading a new book 0.0 Below Average 14.29 7.14 42.86 0.0 0.0 0.0 Quality Meets 21.43 28.57 14.29 21.43 7.14 7.14 Above Average 64.29 64.29 35.71 71.43 85.71 85.71 Overall rating: Folkws the Reading Recovery lesson frameworks Mean:3.46 SD:0.54 Observed Not Observed Reading Recovery Program Strategies Poor 92.86 7.14 Appropriate pacing of lesson components 0.0 Below Average 21.43 Quaflty Meets 7.14 Above Average 64.29 92.86 7.14 100.0 0.0 Appropriate text selected throughout the lesson Appropriate prompts are used for scaffolding the child to problem s<3lve 0.0 0.0 7.14 7.14 0.0 85.71 14.29 78.57 100.0 0.0 Child is engaged in constructive problem solving 7.14 0.0 21.43 71.43 92.86 7.14 Echo of focus throughout the lesson 0.0 7.14 42.86 42.86 92.86 7.14 Procedures ate adjusted according to child's needs 0.0 0.0 14.29 78.57 92.86 7.14 Balance of fluency phrasing practice and problem solving 0.0 7.14 7.14 78.57 85.71 7.14 Opportunities to develop phonological awareness within the lesson 0.0 7.14 21.43 64.29 85.71 7.14 Accurate up-tobate records 0.0 7.14 14.29 71.43 85.71 7.14 Articulates child's strengths and needs 0.0 0.0 14.29 78.57 85.71 7.14 Has high expectations for the child 0.0 0.0 14.29 78.57 Overall Rating: 'NOTE: Item percentages may not total 100% because of missing input from observers. Mean: 3.61 SD:0.66 48INTERVIEW RESULTS Reading Recovery Principal Interview Principals at ten of the Reading Recovery schools were randomly selected to participate in a 45 minute phone interview. Principals were asked a series of questions regarding general program implementation, classroom level changes, program results, professional development opportunities and parental and community involvement. A summary of the principal responses can be found in Appendix B. Overall, principals were positive about the Reading Recovery program and the impact it has had on their schools. Most of the schools have been utilizing Reading Recovery for several years and faculty and staff are very comfortable with the program. Nearly all of the principals interviewed were instrumental in bringing Reading Recovery to their schools, and the decision to utilize Reading Recovery was made after considerable research and thoughtful consideration. Principals reported that Reading Recovery is a wonderful compliment to the schools balanced literacy programs. The one-on-one attention the reading Recovery students receive is overwhelmingly the most effective of the strategies that the program employs. Other effective strategies mentioned included push-ins", literacy groups, running records, and the writing component. Principals reported being active advocates in their Reading Recovery programs. They described their roles as one of support and involvement, ranging from oversight of the program to more direct involvement including student selection for the program, review of student progress, and ongoing meetings and collaboration with the Reading Recovery specialists. All principals noted that teachers were very supportive of the Reading Recovery program and appreciated the impact it has had on overall student achievement. Most of the resources needed for effective program implementation are available
however principals reported an ongoing need for books, additional teachers and tutors, and more planning time. Principals described the African American population as being well-served by Reading Recovery, and most agree that through Reading Recovery the achievement gap is being bridged for their African American students. In most of the schools, African American students are a large percentage of the Reading Recovery program. 49and the early intervention provided by Reading Recovery allows the student to be encouraged by being successful at a younger age. Teacher in-Training Interview In the spring 2005, four teachers in their first year in the Reading Recovery program were contacted by phone for a 30 to 45 minute phone interview. The teachers in-raining were attending classes concerning Reading Recovery instruction, as well as working in their schools implementing the Reading Recovery strategies. Feedback from the teachers was solicited regarding general program information, classroom level changes, results, professional development, and parental involvement and support. A summary of the teacher in training responses can be found in Appendix C. All teachers described the process of integrating Reading Recovery into the schools literacy program as well planned and organized. Reading Recovery teachers work individually with the lowest performing students and then follow up individual instruction with literacy groups. Reading Recovery teachers reported a thorough selection process that involved collaboration with the classroom teacher and comprehensive testing and assessment. In general, the Reading Recovery teachers reported strong support from the classroom teachers, that classroom teachers appreciate the effectiveness of the one-on-one approach, and the fact that the Reading Recovery is able to provide this type of support, freeing the classroom teacher to work more effectively with the other students. The Reading Recovery teachers in training strongly felt that Reading Recovery helped to equalize learning and achievement opportunities for African American students at their schools. Most of the students in Reading Recovery are African American, and these teachers were able to see positive gains. Reading Recovery helps all struggling readers by actively engaging them in reading and allowing them to feel successful at reading. Through detailed daily records and periodic testing all inherent in the Reading Recovery programstudent progress is effectively monitored and lessons can be planned that are tailor made for each individual child. Additionally, the Reading Recovery program often serves as a first line of defense in determining special education needs. Often Reading Recovery will be used prior to special education referral. Teachers in training emphasized the importance of the daily data reports they collect and view them as essential to the success of the program. Reading Recovery teachers collect and use 50data to monitor the progress of the students and to help develop appropriate, effective teaching strategies based on the individual needs of the student. Data are often shared with the classroom teacher and the principal so a collaborative team develops to help plan and implement instruction for the student. The Reading Recovery teachers in training report a significant increase in self-confidence and improvement in overall attitude among the Reading Recovery students. As the students begin to experience success, they are less frustrated and angry. This improvement in attitude improves their classroom behavior and relationships with other teachers and students. Some of these teachers have seen dramatic increases in test scores and other achievement data, while others have not yet been able to see these gains. Again, from the perspective of the teachers in training, the Reading Recovery program is helping to closing the achievement gap of African American students. Reading Recovery teachers in- training reported professional training that ranged from adequate to thorough. Changes in Reading Recovery course instructors led to some confusion for some of the teachers in training. and some reported the need for increased classroom hours. However, overall, these teachers reported feeling well-prepared to work with the students. The strategies they have learned have been very helpful. Although the teachers in training reported having significant classroom experience, they are learning unique techniques that have been very beneficial. There is considerable support from the district teacher, and all teachers expressed a desire and need for continuing support in the upcoming school year. Parental support has been mixed. although all the teachers reported active attempts at engaging the parents in the Reading Recovery program. Reading Recovery teachers in-training concurred that the exposure to print and the one-on- one attention were the most critical elements of the Reading Recovery program. Teachers reported the need to understand several components better, including the make and break lesson and the writing component
and the need for time to plan and implement the program is always a factor. Overall, the components of Reading Recovery were well understood and effective. 51SURVEY RESULTS Reading Recovery Teacher Questionnaire (RRTQ) Descriptive results. The results of the RRTQ are summarized in Table 4. Reading Recovery teachers had extremely favorable attitudes toward the program at their schools. As illustrated in Table 4,100% of teachers strongly agreed or agreed that they have a thorough understanding of the program, teachers in their school are generally supportive of the program. ongoing communication exists between Reading Recovery tutors and classroom reading teachers. Reading Recovery monthly meetings are effective and useful, instructional materials needed to implement our Reading Recovery program are readily available, and the Reading Recovery program is aligned with state and district reading and language arts standards. There was also strong agreement that Reading Recovery teachers received support, with 86.36% of the teachers reporting that the school administration and Reading Coach supported their efforts as a Reading Recovery teacher. Almost 75% (72.73%) of teachers reported receiving extensive district support. The items with the highest level of disagreement (disagree and strongly disagree) concerned Reading Recovery teachers having sufficient planning time (36.36%), enough tutors to fully implement the Reading Recovery program (18.18%), and time to routinely monitor first grade students' progress after they were discontinued from Reading Recovery (9.09% not at all and 72.73% somewhat). Demographic data. All Reading Recovery teachers (100%) reported having at least six years of teaching experience and 31.82% reported at least six years experience as a Reading Recovery teacher. Approximately 85% (86.36%) of Reading Recovery teachers reported having a Masters Degree and beyond. The majority of Reading Recovery teachers were white (72.73%) and 13.64% reported their ethnicity as African-American. Open-ended responses. Reading Recovery teachers were asked to respond to three open- ended questions in addition to the 24 items on the questionnaire. Respondents were instructed to list the strongest aspects of the Reading Recovery program, the weakest aspects of the Reading Recovery program, and reasons for continuing or discontinuing Reading Recovery their school. There were 22 Reading Recovery teachers who filled out the questionnaire, and 21 of those also answered the open-ended questions. Many of the Reading Recovery teachers answered in detail, listing multiple 52responses for each of the questions. The open-ended responses are summarized in Table 5, and a verbatim listing of teacher responses can be found in Appendix E. Of the 21 who listed strong aspects of the Reading Recovery program, the two most popular answers were individualized, one-on-one instruction, 57.1%, and the ability to reach the lowest performing students and bridge the achievement gap for these students, 47.6%. Reading Recovery teaching strategies and components were listed in 19% of the responses, and both the early intervention element and the support from the other teachers were listed in 14.3% of the responses. Respondents were also asked to list what they considered the weakest elements of the Reading Recovery program. Time was a factor in many of the responses with lack of planning time most frequently mentioned at 33.3%. Lack of time to complete the lesson was mentioned in 19.0% of the responses. Teachers inability to help all the students who need help was also mentioned in 19.0% of the responses. All of the 21 teachers responded positively to the question Do you think your school should continue the Reading Recovery program? Why?"-, and 20 teachers elaborated on their positive response listing reasons for continuing Reading Recovery. The ability of Reading Recovery to help struggling readers was listed in half of the responses (50.0%), and strong progress and results were mentioned in 40.0% of the responses. The fact that Reading Recovery decreases the number of special education referrals was listed in 15.0% of the responses. There was one respondent who pointed out that although he/she felt Reading Recovery should be continued at the school, there needed to be better implementation to make it effective. Overall, as reflected in the other items on the questionnaire, the open-ended responses were very positive. Reading Recovery teachers appear knowledgeable and committed to their roles as Reading Recovery teachers and believe strongly in the positive impact the program is having at their schools. Based on survey response and comments provided through the open-ended questions. Reading Recovery teachers are able to see a strong impact from the individualized instruction they provide. 53Table 4 Reading Recovery Teacher Questionnaire (RRTQ): 2004-2005 N = 22 RRTQ Items 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. I have a thorough understanding of the schoofs Reading Recovery program. I have received adequate initial and ongoing professional development/lraining for implementation of the Reading Recovery program. Our Reading Recovery program has positively impacted student achievement Because of Reading Recovery, Literacy Group interventions occur for students in grades K-3. Overall, this program seems valuable for improving the achievement of African-American students. Reading Recovery teachers are given sufficient planning lime to implement the program. Our school has enough tutors to fully implement its Reading Recovery program. The administration protects the time for daily unintenupled Reading Recovery tutoring and Literacy Small Group interventions. Because of our Reading Recovery program, parents are more involved In the literacy program of this school. This school has a plan for evaluating all elements of our Reading Recovery program. Teachers in this school are generally supportive of the Reading Recovery program. Ongoing communication exists between Reading Recovery tutors and dasstoom reading teachers. Reading Recovery teachers are encouraged to communicate concerns, questions, and constnrctive Ideas regarding the program. Our Reading Recovery program adequately addresses the requirements of children with special needs. Reading Recovery teachers participate in the special education referral process to provide early literacy intervention. Because of Reading Recovery, teachers in this school spend more time working together to plan instruction and review student progress. Reading Recovery monthly meetings (continuing contact) are effective and useful. Instructional materials (books, assessments, and other resources) needed to Implement our Reading Recovery program ate readily available. The faculty, staff, and administration believe that aH children can read at grade lev^ or above by the end of third grade. The Reading Recovery program is aligned with state and district reading and language arts standards. Percent Strongly Agree And Agree 100.00 95.45 95.45 95.45 86.36 54.55 59.09 77.27 63.64 86.36 100.00 100.00 81.82 77.27 86.36 81.82 100.00 100.00 63.64 100.00 1. 2. 3. 4. RRTQ Items To what degree did your school administration support your efforts as a Reading Recovery teacher? To what degree did your school Reading Coach support your efforts as a Reading Recovery teacher? Towhat degree (toes the district support your efforts asaReading Recovery teacher? To what degree did your schedule allow the time to routinely monitor first grade students' progress after they were discontinued from Reading Recovery tutoring? Percent Extensively 86.36 86.36 72.73 18.18 Percent Neutral 0.00 4.55 0.00 0.00 4.55 9.09 18.18 22.73 18.18 13.64 0.00 0.00 18.18 13.64 9.09 18.18 0.00 0.00 31.82 0.00 Percent Somewhat 13.64 4.55 27.27 72.73 Percent Disagree and Strongly Disagree 0.00 0.00 0.00 4.55 0.00 36.36 18.18 0.00 13.64 0.00 0.00 0.00 0.00 9.09 4.55 0.00 0.00 0.00 4.55 0.00 Percent Not at all 0.00 0.00 0.00 9.09 NOTE: Item percentages may not total 100% because of missing input from some respondents. 54Table 4, continued Reading Recovery Teacher Questionnaire (RRTQ): 2004-2005 N =22 Total Years of Experience in this School Less than 1 year 1-5 years 6-10 years 11-15 years More than 15 years Number of Respondents__________ Total Years of Experience in any School Less than 1 year 1 - 5 years 6-10 years 11-15 years More than 15 years Number of Respondents 0.00 36.36 31.82 18.18 9.09 22 0.00 0.00 18.18 13.64 68.18 22 How many years experiences have you had as a Reading Recovery teacher? Less than one year 1-5 years 6-10 years 11-15 years More than 15 years ________Number of Respondents ____________Educational Attainment Bachelors degree Masters degree Masters plus 20 hrs Education Specialists Doctoral ________Number of Respondents _______________Ethnicity/Race Asian or Pacific Islander American Indian or Alaskan Native Black, not of Hispanic origin Hispanic, regardless of race White, not of Hispanic origin Multi-racial / Other Number of Respondents 0.00 68.18 31.82 0.00 0.00 22 13.64 36.36 40.91 9.09 0.00 22 0.00 0.00 13.64 0.00 72.73 9.09 22 NOTE: Item percentages may not total 100% because of missing Input from some respondents. 55Table 5 Open-Ended Responses on the RRTQ Question Positive Comments 1. What are the strongest aspects of the Reading Recovery Program? 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Teacher Selection____________ __________Responses________________ One-on-one intervention/ Individualized lessons Bridging Achievement Gap by reaching lowest performing students Reading Recovery Teaching Strategies and Components of Instruction Early intervention Support from other RR teachers Continuing contact with students Professional training and development Addresses both reading and writing Increases students' confidence Collaboration with classroom teacher Close contact with parents Frequency Percent 12 10 4 3 3 2 2 2 1 1 1 57.1 47.6 19.0 14.3 14.3 9.5 9.5 9.5 4.8 4.8 4.8 Total Responses N = 21 Sample responses Working one-on-one with an at-risk child. Reading Recovery is an early intervention. The ability to identify and provide one-to-one instruction for a child is the strongest aspect. The child can be helped before bad habits are in place and self-esteem is an issue. Because it takes place in first grade the children are usually able to get on track with their classmates instead of falling behind more each day. Question Neutral or Neuative Comments What are the weakest aspects of the Reading Recovery Program? 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Teacher Selection __________Responses____________ Planning time Time for Lesson Cant offer RR to All who need it Too much Paperwork Doesnt work for all children Continuing Contact Need more training and professional development support Need more RR teachers at the school More phonological approach in reading instruction Special Education screening No weak aspects Frequency Percent 7 4 4 2 2 2 33.3 19.0 19.0 9.5 9.5 9.5 9.5 9.5 1.48 1.48 1.48 Total Responses N= 21 2 2 1 1 1 Sample responses Not any planning time. Need more ongoing training fro teachers, especially those who may be in their second or third year of Reading Recovery. 56Table 5, continued Open-Ended Responses on the RRTQ Teacher Selection Question Continuation Comments Do you think your school should continue the Reading Recovery Program? Responses Why? 1. Yes: Why? Helps struggling readers Can see the results in achievement Decreased the number of students in special ed. Increases parental support Boosts self-confidence of student See continued gains as children advance Frequency Percent 19 10 8 3 2 1 1 90.5 2. Yes, but.... Need better implementation 3. Yes, with no elaboration 4. No_____________________ Total Responses N= 21 1 1 4.8 4.8 0 Sample responses Yes. It has made a major difference in the number of children not on grade level and even the number of children referred for special education testing at my school. Yes: I believe that Reading Recovery has enabled many children to get a great foundation for reading on grade level or near grade level by the end of 2"^ or 3" grade. Reading Recovery Classroom Teacher Questionnaire (RRCTQ) Descriptive results. The results of the RRCTQ administered to K-3 teachers in the 15 experienced Reading Recovery schools are summarized in Table 6. For the classroom teachers, the three items on which there was 80% or higher agreement included: teacher support of the program (93.59%), positive impact on student achievement (87.82%), and improving achievement of African- American students (82.05%). The two items on which classroom teachers expressed the strongest disagreement or disagreement included: sufficient faculty and staff to fully implement the program (23.08%) and because of Reading Recovery, parents are more involved in the literacy program (14.74%). Demographic data. K-3 classroom teachers reported less teaching experience and education attainment than Reading Recovery teachers. Close to 30% (28.85%) reported less than one year to five years teaching experience in any school and 57.69% reported having a Bachelors degree. However, more classroom teachers reported their ethnicity as African-American (28.85%) than Reading Recovery teachers with the majority (68.59%) reporting ethnicity as white. 57Open-ended responses. In addition to the 13 items the teachers were asked to rate on the questionnaire, several open-ended questions were also asked of the classroom teachers. Teachers were asked to describe their understanding of the Reading Recovery program, relate the strongest aspects of the program, and discuss whether or not Reading Recovery should be continued at their school. Teachers responses were very informative and many listed multiple responses for each of the questions. A complete summary of responses can be found in Table 7, and a verbatim listing of teacher responses can be found in Appendix F. There were 151 teachers who described their understandin
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.