Program evaluations, Volume I

Box 5 LRSD 1) Program Evaluations, March 14, 2003, Volumes I - IVCollection Number $ S O S' Box of Collection Name: OC^ (V\ Q Coo PelsX ri X z a 2 VOLUME H P3 > SC X X RECEIVED MAR 1 4 2003 OFFICE OF DESEGREGATION MONITORING J 2 X >, 2 r e o e k/ o e z >6 O O oSOUTHWEST SF.m. PWXiKXM CHARTER SCHOOL EARl.Y LITERACYri ri LITTLE ROCK SCHOOL DISTRICT INSTRUCTIONAL RESOURCE CENTER 3001 PULASKI STREET LITTLE ROCK, AR 72206 TO: Board of Education ri FROM: T. Kenneth James, Superintendent of Schools PREPARED BY: J Linda Watson, Assistant Superintendent ^bBonnie A. Lesley, Associate Superintendent for Instruction DATE: October 24, 2002 SUBJECT: Approval of the Charter School Program Evaluation Background Information Dr. Linda Watson and Ms. Krishna Young, former director of the LRSD Charter School, presented to the Board of Education in June 2001 the program evaluation for the Charter School. That report was presented as information, but the Section 2.7.1 Compliance Plan requires that the Board formally approve each of the program evaluations listed on page 148 of the Final Compliance Report. The Charter School Program Evaluation was prepared by Dr. Larry McNeal, Professor at the University of Arkansas at Little Rock. Serving on the team with him were Dr. Linda Watson, Ms. Krishna Young, and Dr. Ed Williams, members of the LRSD staff. All of the Charter School teachers, grades 3-5, participated in administering the assessments: the Achievement Level Tests at grades 3-5, the SAT9 at grade 5, and the State Literacy and Mathematics Benchmark examinations at grade 4. and the Success for All quarterly assessments in reading. z The program evaluation included not only student achievement data, but also demographic data, student attendance rates, records of suspensions, student grades, and financial costs for the program. S 0p 99 Performance data for the program evaluation were not disaggregated by race. The student body, however, was 87 percent African American. Due primarily to budget constraints, the District eliminated funding for the Charter School in summer 2002 after two years of operation, so this program has now been abandoned. Recommendation That the Board of Education accept and approve the LRSD Charter School Program Evaluation for 2000-2001. BAL/adg Attachment 1 I kI k1 ku LITTLE ROCK SCHOOL DISTRICT 2000 - 2001 ELEMENTARY CHARTER SCHOOL EVALUATION p 9 9 9 9 9 I 2 IJ LITTLE ROCK SCHOOL DISTRICT CHARTER ELEMENTARY SCHOOL PI > -J > EVALUATION REPORT: 2000-2001 Prepared By Dr. Larry McNeal, Professor University of Arkansas at Little Rock Little Rock, Arkansas e 2 >2, t ! 0 0 c p 9 9 3 Executive Summary The twentieth centurys influence on school choice is reflected in the introduction of charter schools, school vouchers, magnet schools, academy schools, alternative schools, and a host of other specialty schools that focus on specific and often time unique student groups. The recent interest in school choices can be traced to the Nation At Risk report, which was published by the National Commission on Excellence in Education in 1983. The findings about the poor quality of public education alarmed the American public. The report concluded that America was losing its competitive edge in the global marketplace because of its educational system. The result was the passage of educational reform laws and legislation that promoted standards for students, teachers, and administrators. The American educational system continues to search for programs and services that meet the developmental needs of its students. Charter schools are the latest spin-off to a discontented American public looking for vast improvements in educational practices that will better equip its children with marketable skills. Arkansas has also been influenced by this trend. Arkansas modified its existing charter school law in 1999 (Arkansas Charter School Act of 1999) and as a result the charter school movement in Arkansas took off. The Little Rock School District Charter Elementary School opened its doors in the fall of 2000. The charter school offers a nontraditional innovative learning environment for students in the 3,4, and 5 grades. > Pl > < Mission The missions of the Little Rock School District and the Little Rock School District Charter Elementary School are mutually supportive and facilitative of student success. The intent of the missions of both are to provide students with programs and services that fit their educational and developmental needs, as well as prepare them with the knowledge and skills needed to compete in the marketplace. Z P O 9 9 P 9 9 Goal The goal of the Little Rock School District Charter Elementary School is to provide opportunities for students educational and behavioral needs while guiding them to adjust 1 4 their behavior and habits in such a manner that they will become successful, lifelong learners and productive citizens. The Little Rock School District Charter Elementary School accomplishes its goal through an alternative learning environment. Description of Need The description of need is an overview of the variables that influence student academic achievement and behavior at the elementary grade level. Collectively, these variables provide a richer understanding of what is happening at the elementary grade level to some students in the Little Rock School District and why it has become increasingly more difficult to educate all children in a regular classroom setting. Poverty and other factors present barriers for many children who reside in the city of ?5 > JC PI > < Little Rock from receiving the educational opportunities critical for their development. The inability to obtain preschool and other early-childhood education programs is evidenced in the Little Rock School Districts Early Childhood Program where an enrollment capacity of some 750 students leaves approximately 1,000 children on the programs waiting list each year. Once behind, children face a significant challenge of catching up in their school career and many of these children become potential school dropouts in later years. In the 1997-1998 school year, 921 students out of 10,628 students at the secondary level dropped out of school. This represents 8.7% of the Little Rock School District secondary student enrollment. Black students accounted for 69% of the dropout population, with black males representing 44% and black females representing 25.1% of the total. During the 1998-1999 school year the dropout rate increased to 9.7%. For many of these students, the barriers that they faced prior to elementary school were so overwhelming that it left them ill prepared for the challenges associated with schooling
therefore, less capable to take advantage of the educational opportunities at the secondary level. 9 2 2 > ii Iz* b 9 9 9 s 9 Student performance on standardized tests also served as an indicator of the extent of at-riskness for some students in the Little Rock School District. Scores on standardized tests showed percentile ranks across the District and the racial disparity in academics in all grades and subjects. Standardized test scores in reading for the 1997-1998 school year 2 5 I revealed that overall more than one-third of Little Rock School District students were in the lowest 25% nationally for reading skills, with elementary students at 38%, junior high students at 36% and high school students at 34%. Standardized test scores for the 1998- 1999 school year revealed that overall approximately two-thirds of the students (64% of the fifth grade students tested, 63% of the seventh grade students tested and 70% of the tenth grade students) tested were at or below the 50* percentile. The pattern of low performance on standardized tests can be contributed to several factors with the lack of availability of early childhood education programs and low social development being but two. Children who come to school ready and able to learn usually face fewer educational barriers than students who come to school not ready and able to learn. These students are less capable of performing at the levels necessary to acquire the knowledge and skills needed to take advantage of the opportunities of education. n > < Student success can also be influenced by the composition of the family unit. Family composition is related to income and poverty for youth, with two-parent families generally having greater income than families with only one parent. Students coming from families with higher incomes are usually better prepared at an earlier age for schooling. Nationally, in white families a single parent heads two out of every ten households, while single parents head five out of ten Afincan American households. In the city of Little Rock single parents head about six out of ten Afincan American householders. The majority of whom are single mothers. The relationship between income, poverty and academic success suggests that some students who are attending school in the Little Rock School District may need more support prior to the formal schooling experience and perhaps throughout their educational career in the public school system. Students who experience behavior problems often times are not benefiting as much from the academic aspects of schooling because usually they are removed from the educational setting. The number of students who were long-term suspended and/or expelled from the Little Rock School District increased during the 1992-93 to 1997-98 school years. The District revamped its alternative learning environments during the 1997-98 school year by enlarging the Alternative Learning Center and establishing an Accelerated Learning i ! X9 > z p 9 O o p 9 9 3 6 Center for secondary students. During the 1998-99 school year, the District experienced a decrease in the number of long-term suspensions and a significant decrease in the number of expulsions. At the beginning of the 1999-2000 school year, the District established four elementary alternative classrooms in four of its elementary schools. The fall 2000 opening of the Little Rock School District Charter Elementary School was a logical progression in the Districts goal to provide for the educational and social needs of all students, especially, those with behavioral problems. The continued development of alternative learning environments is representative of the Districts commitment to providing the proper match between programs and services and students. The needs of some students cannot be met within the regular classroom setting and so alternative learning enviroiunents that focus on academics and behavior modification can ensure that all students receive an education that will prepare them to be lifelong learners and productive members of society. >P5 2 5P!I > Governance Two years of exploration and planning took place prior to the establishment of the charter school by the Little Rock School District Charter School Planning Team. The comprehensive effort included teachers, parents, and community members. The charter school model developed by the team was based on twelve years of research and evaluations by parents and teachers of the Alternative Classroom Experience (ACE) program at Pfeifer Kiwanis Camp. The team reviewed data to determine where the greatest needs exist. After careful review, subcommittees made up of classroom teachers, specialists, parents, and administrators discussed the implementation of various programs and services. For instance, a subcommittee examined the Success For All program while the Mental Health/Parental Involvement Team subcommittee investigated possible programs and services to include parents, families, and the community. Reports from the subcommittees were made to the Planning Team. After careful consideration, the Planning Team delineated issues, concerns, and discussed subsequent proposals. The Planning Team was later changed into an advisory group (Little Rock School District Charter Elementary School Advisory Committee). 4 IZ* P 9 9 9 P 9 9 7 The Little Rock School District Charter Elementary School Advisory Committee met periodically to monitor the progress of the school as well as provide assistance and feedback to the charter school principal and staff. Little Rock School District personnel facilitated the planning and advisory process and provided technical assistance and support. District personnel were instrumental in supporting the charter through inservices on proven educational practices, data evaluation, and a review of the Title I plan. They also offered parental involvement suggestions as well as conducted research activities. D Instructional Program The Little Rock School District Charter Elementary School is located at the Badgett Elementary School. The Little Rock School District Elementary Charter School was modeled on proven methods and strategies used in the Alternative Classroom Experience at Pfeifer Kiwanis Camp. Those strategies include effective early intervention/prevention for academic underachievement, effective early intervention/prevention for socio-behavioral and personal failure, and an aggressive approach to rebuilding community and parent identity and support for public education. In addition to these methods and strategies, other innovations include Success For All, an Extended Day/After School Program, Gifted and Talented Program, Exceptional Children Services Program, Accelerated Reading Program, Read Across America Program, Media Center Program, Awards Programs, Incentive Programs and programming using computer technology, mathematics and science modules, field trips, the residential component located at Pfeifer Camp, music, experiential education, and physical education. Students also wore school uniforms. PI > 2< Pl I z 5 5 >1 21 fl z p 9 O e p 9 9 The charter school has six classrooms staffed with teachers and program assistants to teach each student. A gifted and Ulented teacher, resource teacher, reading specialist, reading tutor, media specialist, music teacher, speech therapist, occupational and physical therapist, counselor, nurse, school examiner, director, secretary, cafeteria personnel, and custodians provide other services. 5 8 I Student Demographics The Little Rock School District Charter Elementary School offers a nontraditional innovative learning environment for students in the 3*, 4*, and 5* grades. Students attending the school come from a host of elementary schools in the Little Rock School District, North Little Rock School District and Pulaski County Special School District
however, priority admission was given to students cunently enrolled in Baggett Elementary School. Other students were eligible for admission if they were academically functioning one or more grades levels below grade placement
had a pattern of recurring absenteeism
had poor social and interactive skills (displaying unacceptable patterns of behavior and failure to fit the social environment of the school)
were retained one or more grades: and, were from an unstable family structure. 2> P5 % > Once students were selected multiple assessment measures were used to assess them. The assessments included Success For All Baseline Assessment (which is reported in terms of grade level). District Achievement Level Test (RIT Score), State Benchmark Test (4 grade 2000 only) and Teacher Recommendations. The data generated from the assessments facilitated a better understanding of the needs of individual students. i 5X >1 2 Initially, 116 students made application to enrolled in the school
however, 17 students did not attend for various reasons including relocation out of the District, logistics (parents wanting their children to all attend the same school or schools closer to home). program fit (some parents concluded that the program and services offered were not appropriate for their child), and a few students withdraw prior to the start of the 2000- 2001 school year. Ninety-nine students attended the school at some period during the school year of which eighty-seven students completed the school year at the school. z p 0 p 9 9 The demographic data shows that overall the characteristics of the students enrolled were similar to other schools within the District
however, there were significant differences in some aspects. For instance, the school was different than most other elementary schools because of its concentration of students who were academically underachieving and 6 9 I I behaviorally and socially challenged. No other elementary school in the District had an exact similar student population. I I The demographics of this student population are highlighted in Tables 1, 2, and 3. The demographic data examined include racial makeup, grade level, attendance, absenteeism, and discipline. In addition, social economic statue and mobility are discussed. I I Table 1 STUDENT DEMOGRAPHICS Racial Make Up Black Female White Female Black Male White Male 3rd Grade 4 3 23 1 4th Grade 2 3 24 1 Sth Grade 10 1 23 4 Total 16 7 70 6 Percentage 16% 7% 71% 6% Total Student Count 31 30 38 99 100% > g Pl ? < 3 A review of the data shows that the majority of the students attending the school were black. The majority of the students were black males followed by black females. The number of white females and males attending was about the same. The overall gender and racial make up of the school does not reflect the gender and racial make up of the average elementary school in the Little Rock School District. The above data includes student suspension data. If the suspension data were subtracted from the data, the overall attendance rate would be 94.5%. 2 2 >, pj z p o p 9 9 In addition, an examination of student records shows that an overwhelming majonty of the students in attendance receive reduced or free lunch. Of the students attending the school, 91% receive reduced or free lunch. Again, this percentage is higher than the average elementary school in the District. The combination of gender, race, and reduced or free lunch provides a better understanding of the social-economic status of the student population at the Little Rock School District Charter Elementary School. 7 10 L Part of the criteria for admission to the school was a pattern of recurring absenteeism. On average absenteeism is lower at the elementary school level than at any other grade level
however, the absenteeism rate for students attending the charter school is higher than normal. The attendance history of the student body is presented in Table 2. Table 2 STUDENT ATTENDANCE BY QUARTER Grade Level 3rd Grade 4th Grade Sth Grade__ Qverall Average 1st Quarter 87% 84% 92% 88% 2nd Quarter 94% 83% 87% 88% 3rd 4th Quarter Quarter 84% 92% 89% 88% 83% 85% 82% 83% Average 87% 86% 88% 87% The overall average attendance rate for all grades was 87%, which means that the school had an absenteeism rate of 13%. The attendance percentage is lower than the District elementary school average of 95%. The attendance and absenteeism rates of students in attendance at the Little Rock School District Charter Elementary School are more similar to the attendance and absenteeism rates of other schools with similar students. In addition, the mobility rate of the school was 14%
however, the adjusted mobility rate was 4%. Mobility can be understood as the relationship between the number of students who started out the school year attending a particular school and the number of students who were still in attendance at that same school at the end of the school year. The number of student transfers and suspensions impacted the mobility rate at the Little Rock School District Charter Elementary School. When the impact of the transfers and suspensions are factored out, the mobility rate declines from 14% to 4%. A major criterion for admission to the school was a history of academic, social and behavior problems. Students attending the school usually had poor social and interactive skills along with imaccepUble patterns of behavior. As a result students who attended the 8 n >2 P3 ft < j: V. i XX >, ! Z p 9 e p9 0 11 school usually had experienced more instances of inappropriate behavior, which resulted in more suspensions than other students. Students with poor social and interactive schools came from all three of the school Districts. Table 3 reflects the disciplinary records of students. Table 3 STUDENT DISCIPLINE Total Number of Suspensions Total Enrollment________________________ Total Number of students with Suspensions Total Number of students with No Suspensions Percentage of student with Suspensions______ Percentage of students with NO suspensions 66 99 37 62 37% 63% P5 > 2 P5 r
< When considering the grade level of students being suspended the 3"* and 4* grades had 13 student receiving suspensions, and the 5* grade had 11 students receiving suspensions. Several of the students had multi-suspensions. In addition, some of the students who were suspended no longer attend the school. The majority of students attending the charter school had no suspensions. Furthermore, it is anticipated that suspensions will decrease in the 2001-2002 school year with the _. cih current 5 grade class leaving and with the 3"* and 4* grade students returning to an environment that advocates behavior and social management. Student Academic Achievement The Little Rock School District Charter Elementary School complies with all mandated state and federal testing, and other assessments based on student needs, which the teacher deems most appropriate. Student performance is measured in part by the same critena as for the rest of the schools in the Little Rock School District. The performance indicators include the Primary Benchmark Exam at Grade 4, Achievement Level Tests at grades 3-5, Success For All. quarterly grades, WRAT (pre 9 12 V 5 >1 2 z p 9 9 9 9 9 Iand post), and SAT 9 at grade 5. These performance indicators are examined in the tables and charts contained in this section. The data presented in Table 4 represents the 1999-2000 school year student literacy achievement as measured by the Primary Benchmark Exam at Grade 4 for current 5* graders at the Little Rock School District Charter Elementary School. Table 4 PRIMARY (GRADE 4) BENCHMARK EXAMINATION FOR LITERACY: SCHOOL YEAR 1999-2000 PREFORMANCE LEVEL _______ BELOW BASIC BASIC PROFICENT ADVANCED PERCENTAGE OF STUDENTS 86% 7% 7% 0% As illustrated in the table, only 7% of the then 4* grade students were proficient in literacy. The majority of students fall within the below basic range. This can be interpreted, as the overwhelming majority of 4* grade students who entered the Little Rock School District Charter Elementary School in the fall of 2000 could not read at grade level. An examination of the results of the 2000-2001 benchmark scores, as presented in Table 5, further illustrates the challenging task of improving student reading achievement. Table 5 PRIMARY (GRADE 4) BENCHMARK EXAMINATION FOR LITERACY: SCHOOL YEAR 2000-2001 PREFORMANCE LEVEL PERCENTAGE OF STUDENTS BELOW BASIC 83% BASIC 17% PROFICENT ADVANCED 0% 0% 10 13 Pl P>l 2 Z P 0 c p 9 9 STAFF Staff Title I Staff Residential Component TOTAL Materials & Equipment Computers Classroom equipment/ Furniture Printers Computer Tables Computer Software_______ Cabling________ Server(s) Hubs__________ Classroom supplies Office materials and supplies Repairs of equipment Textbooks Periodicals Audiovisual materials TOTALS CHARTER SCHOOL BUDGET 2000-2001 Proposed 725,000 96,000 821,000 53,100 35,000 7,000 8,500 1,500 20,000 10,000 1,000 2000-2001 Acutal 345,000 71,000 20,000 400 2,000 158,000 89,000 505,000 61,596 34,147 7,000 8,500 1,500 31,176 3,811 26,681 Source of Funds LRSD Title I Provided Provided 174,411 2001-2002 345,000 71,000 ADE Grant ADE 43,000 USDE 62,335 ADE 35,000 LRSD LRSD LRSD ADE 36,000 USDE 43,700 USDE USDE ADE LRSD LRSD LRSD 33,500 449,500 7,000 3,000 5,000 5,700 1,000 5,000 Provided Provided 26,700 Source of Funds LRSD Title I LRSD LRSD USDE LRSD LRSD LRSD LRSD LRSD LRSD LRSD 1 14 B > < d PI JB > J i 3 ?) z I* p 9 9 9 P 9 9fl fl fl fl fl fl fl fl nn HH fl fl Contractual 2000-2001 Proposed 5,000 2000-2001 Actual 10,000 Staff Development Training & Curriculum/ Improvement Success for All - Staff Development & Materials Travel Dues and Fees Telephone Postage Printing Utilities Maintenance Supplies & Equipment Construction & Repairs Transportation Other Support Services TOTAL GRAND TOTAL 50,000 49,135 40,000 10,000 1,200 500 500 500 30,000 3,500 100,000 (at least) 150,000 346,200 1325,700 9,751 1,000 38 144 Provided Provided 175,000 224,263 S09331 1,188,742 Source of Funds USDE 35,000 USDE 50,000 ADE 40,000 USDE 10,000 USDE LRSD LRSD LRSD LRSD LRSD LRSD LRSD 2001-2001 Proposed 50,000 4,700 10,000 2,500 1,200 500 300 500 Provided 3,500 250,000 5,000 328,200 804,400 Source of Funds USDE USDE LRSD USDE LRSD LRSD LRSD LRSD LRSD LRSD LRSD LRSD LRSD NOTE: The above budget included grant funds from the Arkansas Department of Education (ADE) and United States Department of Education (USDE). 1 15 P5 > r 5 Pl XX >1 2 z * p Q9 0 P 9 0 5 The number of students scoring below basic decreased from 86% to 83%. The largest increase occurred in the number of students rated basic. The number increased from 7% to 17%. Overall, the student of students who were rated as proficient readers decreased from 7% to 0%. The Little Rock School District Charter Elementary Schools emphasis on reading has not yet impacted student reading achievement. The data presented in Table 6 represents the 1999-2000 school year student mathematics achievement as measured by the Primary Benchmark Exam at Grade 4 for current 5*'' graders at the school. As previously mentioned he current 5* graders were not in attendance at the school during the 1999-2000 school year. ?5 > P5 < Table 6 PRIMARY (GRAPE 4) BENCHMARK EXAMINATION FOR MATHEMATICS^ SCHOOL YEAR 1999-2000 PREFORMANCE LEVEL _____ BELOW BASIC BASIC PROFICENT PERCENTAGE OF STUDENTS 86% 7% 7% 0% i >1 Seven percent of the then 4* grade students were proficient in mathematics. The overwhelming majority of students fall within the below basic range which indicates that students were not proficient in mathematics at grade level prior to attending the Little Rock School District Charter Elementary School in the fall of 2000. An examination of the results of the 2000-2001 benchmark scores, as presented in Table 7, further illustrates the challenging task of improving student mathematics achievement. P 0 0 P 0 0 t 0 11 16 Table 7 PRIMARY (GRADE 4) BENCHMARK EXAMINATION FOR MATHEMATICS
SCHOOL YEAR 2000-2001 PREFORMANCE LEVEL ______ BELOW BASIC BASIC PROFICENT ADVANCED PERCENTAGE OF STUDENTS 96% 0% 4% 0% The number of students scoring below basic increased from 86% to 96%, while the number of students rated basic and proficient decreased 7% to 0% and from 7% to 4% respectfully. Again, the Little Rock School District Charter Elementary Schools emphasis on mathematics has not yet impacted student reading achievement. > g W I In addition to the benchmark examinations, the Little Rock School District Elementary Charter School has an array of iimovative programs and services to offer students that focus on their academic development. A variety of means exist to measure the effectiveness of those programs, for instance, the grade level reading and mathematics tests. These tests are given in the fall and spring of the year. The grade level achievement tests highlight student progress. They measure students progress over the school year. The reason for measuring the rate of student growth is because it serves as an indicator of how much progress students are making compared to themselves over a defined period of time. I >, The data are presented in Charts 1 and 2. The data presented compare the growth rate of students in the Little Rock School District Charter Elementary School to similar students in the Little Rock School District and nation. In Chart 1 the grade level reading achievement growth rate is presented. Z P o c p 9 9 0 12 17E 14 12 Chart 1 READING ACHIEVEMENT LEVEL TEST FALL 2000 TO SPRING 2001 LU s *r P s z < 5 UJ S 10 I6.J' i5 1^5 J5.4
Charter
District j I National > ft < OJ' 3rd Grade 31- 4 Grade Sth Grade 8 6 4 1 ' ^5 I At the 3* and 4* grade levels, the growth rate exceeds those for students in the Little Rock School District and in the nation. The increases range from 4.2% at 3"* grade to 2.5% at 4* grade. The growth rate in both cases is significant. At the 5* grade level, the growth rate is equal to those of students in the Little Rock School District and only slightly below the national growth rate for similar students. 2 * 21 In Chart II the grade level mathematics achievement growth rate is presented. The data contained in the chart also indicate that the growth rate for students in the Little Rock * P 9 O Q P o 9 School District Charter Elementary School is either higher or slightly below that of the Little Rock School District and the nation depending on the grade level. 9 13 18E 16 14 12 10 Chart 2 MATH ACHIEVEMENT LEVEL TEST FALL 2000 TO SPRING 2001 111.9 U15| >n 2 P |9 8.7 s O Z. < uS 2 a' Charter District National 0-* JI 10 2 Si 3rd Grade 4th Grade Sth Grade ?! 8 6 4 2 w ( i 1 i + ! I 8 1^11 yi Similar statements can be made about student performance on the mathematics achievement level test. At the 3"* and 4* grade level the growth rates exceed those for students in the Little Rock School District and in the nation. The increases range from .1% at 3"* grade to 6.3% at 4* grade. The growth rate at 4* grade is significant. At the 5' grade level, the growth rate is 1% lower than the Little Rock School District and 2% lower the national growth rate for similar students. th p e o e p o 9 The Charter School also used the Success For All program. The program has been used in more than 1000 schools. Benefits are particularly strong for students who are at risk. This is especially the case for those students in the lowest 25th percentile of their class. 1 i 14 19 IM The data contained in Chart 3 show the progress of students at the Little Rock School District Charter Elementary School during the 2000-2001 school year. Charts SUCCESS FOR ALL READING BY GRADE LEVEL d ! fl u o < UJ u 100%/r 90% ** 80%'*' 70%-*' 60%** 50%-^ AS/? !56%r| BASELINE ' PI > Pl 40% 30% 20% 10% 132% :24'}^n ir^39?i38% llnl iQ FIRST SECOND IBTHIRD io FINAL A 3 pi third grade FOURTH GRADE FIFTH GRADE The 3"* grade reading scores have increased each quarter of the school year. The percentage of students reading at grade level in the 3" grade went from 0% to 56% over the course of the school year. The 4*" grade reading scores have remained fairly constant with 40% of the students reading at grade level at the beginning and end of the school year The 5* grade reading scores have also increased from 24% to 38% during the same time period. Student reading scores have increased as a result of the Success For All reading program and other such programming activities. Those other activities include a total of 1,776 hours of staff development in such areas as Success For All, active learning with technology, experiential education, Smart Step, and cooperative learning. Student grades are also indictors of achievement. Student report card grades are presented in Charts 4 and 5. In Chart 4 reading grades by grade level and quarter are presented 15 20 I IS > z r o e p d d d I _ I I I I X Chart 4 READING SCORES BY GRADE A 60/ u g H CZ3 b O u o < u CU 50--^' 40 30^ 20 10 J21 14 1616 2( JS11 28 124 1 24' X- 1ST QUARTER READING 2ND QUARTER READING 3rd QUARTER READING ' 4th QUARTER READING >n 2 P3 r<
GRADE The number of students receiving an A and B grade increased at the same time that the number of students receiving a C grade decreased in the 4* quarter. The number of students receiving a D grade also increased. Overall, 2 more students received an A grade by the 4'*' quarter and 31 more students received a B grade by the 4* quarter. The number of students receiving a C grade decreased by 17, while the number receiving a D grade increased by 1 in the 4* quarter. Reading grades have continued to rise with significant improvement occurring in the number of students receiving a B grade along with the decrease in the number of students receiving a C grade. fl I> Si P V- r n e a e u o o 16 2X <3 I I I t 1 B 0 A B C D Math scores have also risen with a significant increase in the percentage of students receiving a B grade. Chart 5 MATHEMATICS SCORES BY GRADE LEVEL 60/ 50-^ i43 UJ O CZ5 o UJ o UJ U 40-^ 30 i22!
28 28 20-'" 120B 10 1321 :23 H3B3 13 FIRST QUARTER MATH B SECOND QUARTER MATH J B THIRD QUARTER MATH i BFOURTH QUARTER MATH
P5 > g Pl 5C > r: I
37 I 1 I I 0 1 0 B A c GRADE s X X > F D The percentage of students receiving a B grade has increased over the course of the year, while the number of students receiving an A, C, D or F grade has decreased. Overall, the number of students receiving a B grade increased from 28 in the 1 quarter to 51 in the 4* quarter. The biggest decrease was in the number of students receiving a C grade. That number went from 43 to 23. There were also small decreases in the number of students receiving an A, D, or F grade. The numbers ranged from 2 to 3. The decrease in the number of students receiving an A, C, D, or F grade is offset by the increase in the number of students receiving a B grade. Z P o c p 9 9 < 17 22 Additional data about student performance at the Little Rock School District Charter School can be glanced from results of the Math WRAT assessment. Student performance on this examination is presented in Chart 6. The examination was administered both as a pretest and posttest to students in the 3"*, 4*, and 5* grades in the fall and spring of the 2000-2001 school year. Chart 6 g < ou u> < O s oo cU4J a 2 z 18 16 14 12 10 8 6 4 1 0 MATH WRAT ASSESSMENT 3RD GRADE 4TH GRADE STH GRADE PRE i I a post
9 i /i V 2 2 Student performance as measured by WRAT indicates an increase in the average number of correct responses at all three grade levels. In 3"* grade the average number of correct responses increased by 2.5. In 4* grade the average number of correct responses increased by 1.3. In 5* grade the average number of correct responses increased by 2.6. B z * p e c p e 0 The final academic assessment examined is the Stanford 9 Basic Battery for the 5 grade from fall 2000. As shown in Table 7, the examination provides another view on student performance. The test was administered to 5* grade students shortly after the start of the 2000-2001 school year. B The performance of the 5* graders reflects their prior academic preparation rather than their preparation at the Little Rock School District Charter Elementary School. The 18 23 a students had been attending the school for only several weeks before the test was administered. I 1 Chart 7 a STANFORD 9 NORM REFERENCE TEST NATIONAL PERCENTILE RANK BASIC BATTERY STH GRADE FALL 2000 1 a a a a a CZ3 LU Q H CZ2 LU O LU u u oi u Qb 100 90- 80 10 60 50 40 30 20 10 0 ,88 12 OE 0%-25% 26%-50% 51%-75% 76%-99% QUARTILES The test results show that 88% of the student population performed in the 0 - 25% quartile, while the remaining 12% of the students performed in the 26% - 50% quartile range. Student performance for this group represents past academic preparation efforts. This groups performance should be compared with that of the fall 2001 S* graders who were 4 .* graders during 2000-2001 school year. This kind of comparison would ascertain the affect of the charter school program on 4* grade students during the 2000-2001 school year. Academically, the students at the Little Rock School District Charter Elementary School have made progress. The arrays of assessments highlight this improvement in 19 24 Pl 2 Pl ft 5/5 I % > 2 7- C C o o t Ia mathematics and reading. The schools success is a measure of the effectiveness of the programs and services provided in the innovative alternative learning environment. Financial The program and services provided during the 2000-2001 school year by the Little Rock School District Elementary Charter School cost $1,182,842 (see attached budget). fl Even in an otherwise well-conceived and fairly funded charter school the need to purchase textbooks, instructional materials and supplies, computers and equipment as well as provide for the staff development training needs of staff can make it appear to be an expensive venture. Given the singular nature of several of the expenditures, the overall cost of operating the Little Rock School District Charter Elementary School should decrease during the remaining years of the charter school contract (see attached budget). W > 2 P5 > ft < fl fl Also, in comparing the start up cost of the charter school to similar schools a more reasonable comparison would be with other elementary schools in the Little Rock School District, North Little Rock School District, or Pulaski County Special School District that have opened within the past three years. Just like in these schools the initial cost is high but tends to decrease over a period of time. Another comparison could be made between other schools in the Little Rock School District that are serving students with special needs such as the magnet or incentive schools. 1 ?! y z ?! 2 2 > 2 Conclusion Virtually all charter schools have had to overcome obstacles during development and implemenUtion. The barriers range from lack of start-up funds, lack of planning time, inadequate operating funds, inadequate facilities, state or local board opposition, internal conflicts, hiring staff, acquiring enough students, determining the right mix of programs and services, accountability requirements, and community opposition. While not all of the barriers apply to the Little Rock School District Charter Elementary School, some did. They include barriers associated with sharing a building, acquiring enough students, and meeting accountability standards. In addition, to the biggest challenge of them all pi z p 9 0 P 9 9 / 20 25a which was meeting the needs of all of the students who were academically, socially, and behaviorally challenged through the right mix of programs and services. The ability of an organization to respond to these bamers can be somewhat quantified by examining student demographic data, student academic data, staff commitment to program goals, and stakeholders satisfaction. The demographic data indicates a student a a population that is highly challenged academically, socially, and behaviorally. The impact of this challenge is felt in the areas of student attendance, mobility, and discipline. The school can positively impact attendance and discipline, however, its ability to impact mobility is questionable. a a The academic data also illustrates a population that produces mixed results. Pre and post student achievement scores range from above average to below average. The school has the capacity to positively impact student achievement with its array of programs and services. The issue of matching students with the right program may still need to be refined. a P5 > P5 ft < I ifi Another way of quantifying the organizations ability to respond is by examining the diligent and commitment of staff to the goals of the Little Rock School District Charter Elementary School. The closer the alignment of diligent and commitment to goals the better the chance that the needs of all students will be met. The numerous hours devoted to staff development and volunteerism is an indicator of the staffs diligent and commitment to students. JO 2 > a Still another way to quantify the affect of the Little Rock School District Charter Elementary School is through the impact that it has on stakeholders. Those stakeholders are the parents, children, teachers, and staff of the school. Stakeholder satisfaction is crucial to the overall success of the program. The overall impact of the charter school to stakeholders can best be summarized by the responses attained from a survey given earlier in the school year to parents, students, and staff and from the comments of parents at the end of the school year. Stakeholders developed consensus on the importance of curriculum, instruction, sUff development, and parental involvement. These items were a z r c c e p 9 0 21 26a key elements in the program and related directly to successful student outcomes. The stakeholders felt these items would provide the foundation for long-term success. a In closing, the overall impact of the Little Rock School District Charter Elementary School is apparent not so much in the data presented but in the comment of one parent who perhaps speaks for all stakeholders. She comments, The benefits that my child received from being in the program can not be understated. He is better as a result of being at the charter school. a a > 2 r 5 P5 > < a a Vi 3 4 fl fl n 2 ?! 2 z r c p 9 9 a a 22 27EARLY LITERACY 1 Lisa 7 i J J '3 fl LITTLE ROCK SCHOOL DISTRICT INSTRUCTIONAL RESOURCE CENTER 3001 PULASKI STREET LITTLE ROCK, AR 72206 fl TO: Board of Education fl FROM: PREPARED BY: T. Kenneth James, Superintendent of Schools ^^onnie A. Lesley, Associate Superintendent for Instruction Pat Price, Director of Early Childhood and Elementary Literacy DATE: October 24, 2002 SUBJECT: Early Literacy Program Evaluations fl fl Background The Board of Education approved in its 1999-2000 program evaluation agenda the Early Literacy program, which began implementation in fall 1999. During July and August 2000 the Assistant Superintendent of PRE presented to the Board drafts of this evaluation, which the Board tabled in August 2000 pending completion. That early draft was never completed and was not again submitted to the Board of Education for review and approval. ys 23 I j: ri 2 During summer 2001 Dr. Bonnie Lesley, on behalf of the Early Literacy Program Evaluation team (Pat Price, Pat Busbea, Ann Freeman, Ed Williams, Ken Savage, Anita Gilliam, and Sharon Kiilsgaard) presented a completed 204-page program evaluation: Year 2 Evaluation: The Effectiveness of the PreK-2 Literacy Program in the Little Rock School District (1999-2000 and 2000-2001). This report was presented for information, but our Section 2.7.1 Compliance Plan now requires that all program evaluations be presented for Board acceptance and approval. z* p c p 9O Dr. Steve Ross of the University of Memphis had served as an external consultant to the team. He read both a near-complete draft and made several suggestions for its improvement, which were incorporated into the final draft. He also read the final draft and responded. 1, All grades K-2 teachers administered the assessments, both fall and spring, in all three years, 1999-2000, 2000-2001, 2001-2002. All elementary principals supervised both the fall and spring administrations of the Developmental Reading Assessment and the Observation Surveys and the Achievement Level Tests at grade 2. Central office Elementary Literacy staff conducted the training for the assessments, collected the answer documents, and participated in the analysis of data: Patricia Price, Pat Busbea, Judy Milam, Judy Teeter, Kris Huffman, and Ann Freeman. Both Dr. Ed Williams and 28 f D Board of Education - Memo October 24, 2002 Page Two II II Ken Savage assisted in the production and analysis of score reports. Anita Gilliam and Sharon Kiilsgaard assisted in checking the data tables for accuracy and in preparing the final reports. Copies of this program evaluation were provided to Mr. John Walker, to Ms. Ann Marshall at ODM, and to all elementary principals and elementary literacy staff. Executive summaries, including the program evaluation recommendations, were sent to all K-2 teachers with a cover memorandum congratulating them on their successes. The program evaluation was comprehensive, including the following: an introduction
a chapter on the literacy program design and its relationship to the Districts Strategic Plan and the Revised Desegregation and Education Plan
a description of all of the K-2 assessments used to measure student progress
a chapter aligning the program with national research studies on effective early literacy programs
numerous tables displaying the data in several different ways, disaggregated by grade level and race
an analysis of the results (based on student performance data)
an analysis of additional data relating to achievement gap among schools and the impact of professional development on student achievement: a chapter on findingsanswers to the six research questions originally posed
a bibliography: and tables of school-level data on each assessment for the two-year period. An important chapter of the program evaluation relating to Section 2.7.1 of the Revised Desegregation and Education Plan was the one on findings. Research Question 2 was as follows: Is the new program effective in improving and remediating the academic achievement of African American students? The discussion filled pages 81-96. The following paragraph includes the criteria that were used to determine effectiveness." To determine the effectiveness of the new program in improving and remediating the academic achievement of African American students, the District used the performance results of the Observation Survey and the Developmental Reading Assessment. The basic criterion established in determining program effectiveness for black students was that black student achievement would have to improve and then that growth over the two-year period of the programs implementation would need to be equal to, but preferably greater than, the growth of non-black students, (p. 81) II II n n H B n n n n H n 29I - 7 1 i Board of Education - Memo October 24, 2002 Page Three I The report included a detailed analysis of all available data: I I I - I The following findings based on Observation Survey, Developmental Reading Assessment, and Achievement Level Test results make it possible to conclude that the new early literacy program has so far been effective in improving and remediating the reading achievement of African American students, as well as all students. It is unusual in any District to find gains by both blacks and non-blacks over a two-year period on eight different measurements, as this study finds. Again, however, experts on program implementation advise that it takes approximately five years to determine program effectiveness, so this year 2 study at best establishes baseline and early trend data for comparisons in future years, (pp. 82-83) In this sections conclusions, the program evaluation included not only a summary of findings as they relate to the achievement gap, but also how they compare to the findings in recent national research on reading achievement among African American and white students: The results of two years of changes in the LRSD policies, programs, and procedures in grades PreK-2 indicate that both black and non-black children in the Little Rock School District are learning to read independently by grade 3 (see Section 5.2.1 of the Revised Desegregation and Education Plan). The findings and analyses in this report indicate trends in the opposite direction of the national research findings cited above and of scores of other similar studies. Instead of black students growing at slower rates than non-blacks, in most of the measurements the LRSD results indicate higher rates of growth of black students than non-blacks. Instead of the gap widening between grades 1 and 2 as it does in national studies, it narrows significantly in the LRSD by every one of the eight measures (fve sub-tests of the Observation Survey, the Developmental Reading Assessment, and two sub-tests of the Achievement Level Test), (p. 94) 3 fl fl i > {fl z p 9 e p 0 0 Pages 107-113 included recommendations for improvement in instruction, parent involvement, interventions, and professional development. Five schools were identified for improvement since they were the lowest performing schools in at least two of the three grades tested. Recommendations for the next program evaluation were also included. These recommendations were all considered by the program staff and by school-level staff and many were immediately implemented, as well as others identified in formative evaluations during year 3. The major recommendations made to principals for program improvement included (1) ensuring that all teachers are fully trained and are implementing the District program
and (2) adding Reading Recovery and literacy coaches wherever possible, since both of these actions in some schools had resulted in higher achievement. / I 30B Board of Education - Memo October 24, 2002 Page Four B B At the end of 2001-02 the staff decided that another comprehensive study was not necessary so early in the programs implementation (year 3). They, therefore, presented to the Board of Education in June 2002 an update that included all the 2001- 02 scores on the Observation Surveys and Developmental Reading Assessment, along with a summary of analysis of performance, especially comparisons of African American student achievement with other students. Those findings not only confirmed the findings of the 1999-2001 study, but the results were even stronger in year 3. At the end of year 3. African American students scores were at least 90 percent of other student scores on all five measures of the Observation Survey by the end of grade 2. In other words, the achievement gap was either closed on these measurements or almost closed, given the standard of 90 percent as an acceptable ratio. On the Developmental Reading Assessment, the most difficult of the measurements, the black to non-black ratio grew from 35 percent at the beginning of kindergarten in fall 1999 to 82 percent at the end of grade 2 in 2002. Deeper analysis also revealed that although many African American children from poverty were not learning to read in grade 1, they did successfully learn to read in grade 2, so they will most likely reach the goal of independent reading by grade 3, even though they began far behind their peers. Interestingly, the growth of other students generally exceeded African American student growth on the DRA in grade 1, but African American growth exceeded other student growth in grade 2. Copies of the program evaluation and the update are attached for Board members review. Recommendation That the Board of Education accept and approve, as submitted, the following: Year 2 Evaluation: The Effectiveness of the PreK-2 Literacy Program in the Little Rock School District. 1999-2000 and 2000-2001 Update on the Implementation of the PreK-2 Literacy Program, Little Rock School District, 1999-2000, 2000-01, and 2001-02 BAUadg Attachments B B B B B B B B n nnn H 31 i: k:t It- Ir b-' b- Year 2 Evaluation: The Effectiveness of the PreK-2 Literacy Program in the Little Rock School District 1999-2000 and 2000-2001 Presented to the Board of Education Little Rock School District October 2001 I fl 9 2 >2 Ir b^ pj z r p G G G y G G I Prepared by Dr. Bonnie A. Lesley Dr. Ed Williams Patricia Price Pat Busbea Ann Freeman Ken Savage Anita Gilliam Sharon Kiilsgaard 1 I 32 p I i p Year 2 Evaluation: The Effectiveness of the PreK-2 Literacy Program in the Little Rock School District 1999-2000 and 2000-2001 s Section I: Introduction I I 1 d i 1 Introduction During March 2000 the Little Rock School District provided to the Board of Education, the federal court, the Office of Desegregation Monitoring, and administrators an Interim Compliance Report, which included a status report on the implementation of the PreK-3 Literacy Program (pp. 93-105) relating to the Revised Desegregation and Education Plan (RDEP). In August 2000 the Planning, Research, and Evaluation (PRE) office provided to the Board and staff a draft copy of a program evaluation for the first year of implementation of the K-2 Literacy Program. At least two subsequent drafts were developed as more data became available, but these were not presented to the Board of Educationjust discussed among staff members. An implementation update was provided to the Board in January 2001 by the curriculum staff, on the status of program implementation and including an analysis of available data, along with an outline of next steps. Then in March 2001 the staff provided a summary evaluation in the Compliance Report (pp. 72-93) relating to the Revised Desegregation and Education Plan that was filed with the federal court and provided to members of the Board of Education. 3 d 5 5 fl fl I > The Board of Education approved on second reading in March 2001 a new policy on program evaluation. Policy IL: Evaluation oflnstructional Programs requires that the staff evaluate the instructional programs designated by the Board of Education in their annual approval of the program evaluation agenda. Each evaluation is to provide valuable insights into how programs arc operating, the extent to which they are serving the intended purpose of increasing student achievement, the strengths and weaknesses, the cost-effectiveness, and directions for the future. In August 1999, 2000, and 2001, the Board of Education included the PreK-2 literacy program on its approved research agenda for the following year. P c p 9 0 An interim program evaluation was provided to the Board of Education in June 2001, the first analysis of the scores on the Developmental Reading Assessment in grades K-2 for 1999-2000 and 2000-2001. At that time the scores were reported as the percent of students at each grade level, by race, who met the standard for readiness, the level that would predict success at the next grade level (level 2 at kindergarten
level 16 at grade 1
and level 24 at grade 2). Copies of that report, plus the summary and the slides were immediately sent via e-mail to principals to use in their own analysis and to provide to J 1 33 I' I' (' I. I' 1 E. Update on the Implementation of the PreK-3 Literacy Program Plan, Highlights of Grades K-2 Results: Developmental Reading Assessment, 1999-2000 and 2000-2001, and a copy of the slides for the June 2001 presentation to the Board of Education n I I B B B B B n n n 34 I ' Table of Contents Section I: Introduction Introduction Research Questions Methodology Outline of Program Evaluation Sections Outline of Appendices 1-6 1-2 3 3-5 5-6 6 I Section II: Background on Program Design Background on Program Requirements: Design of the PreK-3 Literacy Program Background on Program Requirements: LRSD Strategic Plan Background on Program Requirements: Revised Desegregation and Education Plan 7-13 7-8 8 8-13 Section III: The Assessments The Assessments: Observation Survey The Assessments: Developmental Reading Assessment Defimtion of Readiness vs. Proficiency Reliability and Validity: National Study Reliability and Validity: LRSD Study Developmental Appropriateness of Testing Instruments The Assessments: Achievement Level Tests in Reading and Language Usage 14-25 14-15 15-21 16-19 19-20 20-21 21-23 23-25 V: I JC I Section IV: Alignment with National Research on Early Literacy 26-29 Section V: Description of Tables Table 1: Kindergarten, 1999-2000, Fall to Spring Black and Non-Black Performance Table 2: Kindergarten, 2000-01, Fall to Spring Black and Non-Black Performance Table 3: Grade 1,1999-2000, Fall to Spring Black and Non-Black Performance Table 4: Grade 1,2000-01, Fall to Spring Black and Non-Black Performance Table 5: Grade 2,1999-2000, Fall to Spring Black and Non-Black Performance 30-42 31 31 32 33 33 35 [fl r G C> c p 9 9 I I I Table 6
Grade 2,2000-01, Fall to Spring Black and Non-Black Performance Table 7
Cohort 1, Kindergarten Fall 1999 and Grade 1 Spring 2001 Table 8
Cohort 2, Grade 1 Fall 1999 and Grade 2 Spring 2001 Table 9: Grades K-2,1999-2000, Fall to Spring Performance, All Students Table 10: Grades K-2,2000-01, Fall to Spring Performance, All Students Table 11: Percent of Maximum Scores, Kindergarten Black Students Table 12: Percent of Maximum Scores, Kindergarten Non-Black Students Table 13: Percent of Maximum Scores, Grade 1 All Students Table 14: Percent of Maximum Scores, Grade 1 Black Students Table 15: Percent of Maximum Scores, Grade 1 Non-Black Students Table 16: Percent of Maximum Scores, Grade 1 All Students Table 17: Percent of Maximum Scores, Grade 2 Black Students Table 18: Percent of Maximum Scores, Grade 2 Non-Black Students Table 19: Percent of Maximum Scores, Grade 2 All Students Table 20: Cohort 1All Students, Kindergarten Fall 1999 and Grade 1 Spring 2001 Table 21: Cohort 2All Students, Grade 1 Fall 1999 and Grade 2 Spring 2001 Table 22: Percent Readiness, DRA, Black and Non-Black Students Table 23: Percent Readiness, DRA, All Students Table 24: Grade 2 Reading, ALT, Black and Non-Black Comparisons Table 25: Grade 2 Reading, ALT, All Students Table 26: Grade 2 Language Usage, ALT, Black and Non-Black Comparisons Table 27: Grade 2 Language Usage, ALT, All Students 34 35 35 36 37 37 38 38 38 38 39 39 39 39 40 40 41 41 41 42 42 42 36 i- I i 3 Section VI: Analysis of Results, 1999-2000 and 2000-2001 Letter Identification Word Test Concepts about Print Writing Vocabulary Hearing and Recording Sounds Developmental Reading Assessment 43-67 43-46 47-51 52-54 55-59 60-63 64-67 f Section VII: Additional Data Achievement Gap Among Schools Impact of Professional Development 68-71 68-70 70-71 I I Section VIII: Program Evaluation Findings and Recommendations for Improvement Research Question 1^Program Effectiveness Research Question 2Achievement Disparities Research Question 3Professional Development Research Question 4Four Literacy Models Research Question 5^Program Strengths and Weaknesses Research Question 6Cost Effectiveness Recommendations for Improvement Instruction Parent Involvement Interventions Professional Development Schools Identified for Improvement Year 3 Program Evaluation 72-113 72-80 81-96 96-100 100-103 103-105 105-106 106- 107-109 109-110 110-112 112 112 112-113 ( pl V! pl 9 IB2 > Section IX: Bibliography 114-116 Section X: School-Level Data Letter Identification, Kindergarten Word Test, Kindergarten Concepts about Print, Kindergarten Writing Vocabulary, Kindergarten 117-205 119-122 123-126 127-130 131-134 135-138 pj z r p 0 c Hearing and Recording Sounds, Kindergarten Developmental Reading Assessment, Kindergarten 139-142 Letter Identification, Grade 1 Word Test, Grade 1 Concepts about Print, Grade 1 Writing Vocabulary, Grade 1 Hearing and Recording Sounds, Grade 1 143-146 147-150 151-154 155-158 159-162 I 37 Developmental Reading Assessment, Grade 1 163-166 Word Test, Grade 2 Writing Vocabulary, Grade 2 Hearing and Recording Sounds, Grade 2 Developmental Reading Assessment, Grade 2 167-170 171-174 175-178 179-182 Cohort 1^Letter Identification, Black and Non-Black Cohort 1Word Test, Black and Non-Black Cohort 1Concepts about Print, Black and Non-Black Cohort 1Writing Vocabulary, Black and Non-Black Cohort 1Hearing and Recording Sounds, Black and Non-Black Cohort 1^Developmental Reading Assessment, Black and Non-Black 183 184 185 186 187 188 Cohort 2Word Test, Black and Non-Black Cohort 2Writing Vocabulary, Black and Non-Black Cohort 2^Hearing and Recording Sounds, Black and Non-Black Cohort 2^Developmental Reading Assessment, Black and Non-Black 189 190 191 192 Percent Readiness, Developmental Reading Assessment, K-2 Percent Readiness, DRA, Rank Order, K-2 Percent Readiness, DRA, Black and Non-Black 193-195 196-198 199-201 ! Grade 2 ALT, ReadingAll Students Grade 2 ALT, Reading, Black and Non-Black Grade 2, ALT, Language Usage, All Students Grade 2, ALT, Language Usage, Black and Non-Black 202 203 204 205 Appendices A. B. C. D. PrcK-3 Literacy Program Plan Section 5.2.1 of the Revised Desegregation and Education Plans March 2000 Interim Compliance Report Section 5.2.1 of the Revised Desegregation and Education Plans March 2001 Compliance Report Presentation to the Board of Education, January 2000 (update on program implementation and early results) 38 I teachers and parents. (See Appendix E.) Elementary principals used these materials in their August 2001 preschool inservice sessions. I I This Year 2 Evaluation of the Effectiveness of the PreK-2 Literacy Program in the Little Rock School District builds on the information provided in all earlier reports. It is intended to meet the requirements specified in Policy IL for the 2000-01 school year, as well as to fulfill the requirements in Section 2.7.1 of the Revised Desegregation and Education Plan for the PreK-3 Literacy Program Plan. The grade levels evaluated include only grades kindergarten through grade 2. Another report will include grades 3 through 5. I I The curriculum staff received from PRE on July 19,2001, the report on the mean scores for K-2 students on both the Observation Survey and the Developmental Reading Assessment for 2000-01. Achievement Level Test data were available earlier, but they had not yet been disaggregated by race. This program evaluation, therefore, differs from, but builds upon, the evaluation report that was presented to the Board of Education in June. It includes a much more detailed analysis of data
it includes the results of the five sub-tests of the Observation Survey
and it includes the average performance scores for each school on each sub-test^not just the percent of students meeting the standard. It also includes the results of the grade 2 Achievement Level Tests in reading and language usage. The new data permit the staff to calculate and analyze the scores in a different way (mean performance vs. percent readiness), and they permit the calculation of a black to non-black student ratio so that the degree to which the achievement gap in narrowed can be measured, as well as how the gap has changed over the two years of program implementation. One caution in comparing the 1999-2000 and 2000-01 pre-test scores on the Observation Survey and the Developmental Reading Assessment is that some schools did not complete their fall testing by the deadline in 1999 and so their pre-test scores were higher than they would have been had the testing been done in a timely manner. There were instances when there were several weeks difference in the test date, so this variance would affect the pre-test scores. The kindergarten pre-test scores in fall 2000, for instance, were generally lower than those for fall 1999, for both black and non-black students. These differences do not necessarily indicate that this past years kindergarten class was that much weaker than the one the year beforeespecially when this past years end-of-year scores were higher than the previous classs end-of-year scores. The third and fourth tests administered are the Achievement Level Tests in reading and language usage that are given in spring of grade 2. Those scores, combined with the results of the Observation Survey and the DevelopmenUl Reading Assessment, enable the District to assess the effectiveness of the early literacy program in LRSD, including its impact on the improvement of the academic achievement of Afiican American children. 2 39 3 1 fl dX V. fl *I pzl pr c 0 ep 0 8 r* 1 Research Questions Using the obligations set forth in the Revised Desegregation and Education Plan (RDEP), the Boards Strategic Plan, and the Boards Policy IL, the following research questions were estabhshed to guide this study: 1. 2. 3. 4. 5. 6. Are the new curriculum standards/benchmarks, instructional strategies, and materials effective in teaching primary grade students how to read independently and understand words on a page? (See Section 5.2.1a of RDEP and Strategy 2 of the Strategic Plan.) Is the new program effective in improving and remediating the academic achievement of African American students? (See Section 2.7 of RDEP.) Is there a relationship between teacher participation in professional development and student achievement? (See Policy IL expectation to examine cost effectiveness and Strategy 7 of the Strategic Plan.) Is there evidence of success in each of the four Uteracy models in useEarly Literacy Learning in Arkansas (ELLA) only
ELLA and Reading Recovery
Success for All
and Direct Instruction? (See Section 2.7 of RDEP.) What are the programs strengths and weaknesses? (See Policy IL.) Is the program cost effective? (See Policy IL and Strategy 3 of the Strategic Plan.) Methodology An interdisciplinary team was assembled to prepare the program evaluation for the PreK- 2 literacy program for Year 2. Several staff members provided assistance and support in the construction of 27 separate tables of district-level data to display not only the mean scores for each sub-test, by race and for all students, on the Observation Survey and the Developmental Reading Assessment, but also to display the percent who scored at or above the readiness level on the Developmental Reading Assessment and the median RIT score on the sub-tests of the Achievement Level Tests. Calculations were verified three times by separate staff members to ensure the highest possible degree of accuracy. Among the calculations that were made to assist in the analysis of data were numbers of points of growth from fall to spring for each of the two years, spring to spring, and fall of one grade to spring of the following grade (for a two-year growth). Black to non-black ratios were calculated to determine the degree to which black students were attaining essential knowledge and skill at the same level as non-black students. Growth ratios were also determined^the degree to which growth in a given year by black students was at the same level or higher than that of non-black students. The percent of growth for one year of instruction and then two years of instruction in the program was calculated for each level and each sub-test, although these calculations were not used in the section on findings or in the recommendations made for improvement. And, finally, the mean 3 40i I score on the Observation Survey and the Developmental Reading Assessment was divided by the maximum possible score to determine the average percent for each score. An additional table was constructed to display the achievement gap between/among schools for each sub-test at each grade level. i The Districts statistician conducted three statistical studies that informed the study: one of the average number of days of teacher participation in professional development on the implementation of ELLA, by program model, and another of descriptive statistics between teacher participation in professional development on ELLA implementation and student achievement. A third study was conducted to determine the validity of the Observation Survey and the Developmental Reading Assessment in relationship to the Achievement Level Tests. fI 5 Finally, 87 tables of school-level data were constructed to add to the study and to provide the critical information for school-level staff members to conduct their own analyses at the school level. Throughout the writing of this report individual staff members, both program staff and assessment specialists, were interviewed and queried in order to clarify issues of program implementation, testing administration, instructional procedures, and data interpretation. Their assistance was invaluable. The research studies which guided the initial design of the PreK-3 Literacy Program Plan were again reviewedespecially the research on the identification of prerequisite knowledge and skills that children must acquire on their pathway to learning to read. These findings were once again mapped with the implementation plan for LRSD, as well as the assessment instruments to ensure ongoing alignment. Serendipitously, the National Center for Education Statistics published a report in July 2001 entitled Educational Achievement and Black-White Ineoualitv. which proved to be very helpful in interpreting Little Rock results in a national context, and which is cited in this program evaluation, along with other external studies. Multiple strategies to analyze the data were employed so as to establish as thoroughly and comprehensively as possible a basis for determining the programs quality. The detailed analysis is found in Section VI. No attempt was made in this study to analyze the results for limited-English proficient children since that program is evaluated separately. It is important to note, however, that the scores of limited-English proficient students are included in each schools results. The District requires them to take the tests so their progress in learning English, as well as in learning to read, may be monitored. And, finally, credible research studies were consulted, as were informed staff, in the determination of recommendations for improvement or determining next steps in becoming even more effective. 4 41 J I fl 4 CZ! fl 2 s 2 ?5 Z e P 0 0 < Before the program evaluation was published, it was reviewed by many individuals, including Dr. Steve Ross of the University of Memphis, and groups, including the Early Literacy program staff, PRE staff representatives, and School Services staff. The District is grateful to all who offered feedback and suggestions for the improvement of this report. To the best of the writers ability, the suggestions for improvement were incorporated into the draft. Others were added to recommendations for the Year 3 study. Outline of Program Evaluation Sections This report is organized into ten sections: 1. Section I includes the Introduction, as well as a delineation of the Research Questions for the study and a description of the methodologies employed. 2. Section n provides background information on the program design and its relationship to the Strategic Plan and the Revised Desegregation Plan. 3. Section HI describes the selection of appropriate assessments for grades K-2 and the processes by which readiness standards were established for each grade level for the Developmental Reading Assessment. It also includes information on national and local validation studies of the Observation Survey and the Developmental Reading Assessment, as compared to the Achievement Level Test. 4. The literacy plans design in relationship to the findings in national research studies on early literacy is described in Section IV. This section also includes an alignment of the research with the assessments selected by the District. 5. Three major sections on data analysis follow. Section V is a description of each of the tables that was constructed from the data reports to assist the writers of this report and its readers in analyzing the results on the eight measurements: the five sub-tests on the Observation Survey (OS)
the Developmental Reading Assessment (DRA)
and the reading and language usage sub-tests of the Achievement Level Tests (ALTs). 6. Section VI is a detailed analysis of the data in each table and a comparison of 1999-2000 and 2000-01 data, by race. 7. Additional data are provided in Section VII on the achievement gap among schools and on some statistical studies that were conducted relating to program effectiveness and the relationship between teacher participation in professional development and the achievement of their students. 8. Following the data analysis is Section VIH that summarizes the program strengths and weaknesses and specifies the implications for instruction, with specific recommendations for improvements in 2001-2002. 5 414 1 9. Section IX is the Bibliography for the study. 10. Section X includes 87 tables of school-level data. Those interested in individual school performance or comparisons are encouraged to use the model in this report for data analysis at the District level to conduct similar analyses at the school level. Behind Section X are appendices A-E for more background and further reference: A. PreK-3 Literacy Program Plan e B. Section 5.2.1 of the Revised Desegregation and Education Plans March 2000 Interim Compliance Report I C. Section 5.2.1 of the Revised Desegregation and Education Plans March 2001 Compliance Report D. Presentation to the Board of Education, January 2000 (update on program implementation and early results) E. Update on the Implementation of the PreK-3 Literacy Program Plan, Highlights of Grades K-2 Results: Developmental Reading Assessment, 1999-2000 and 2000-01, and a copy of the slides for June 2001 presentation to the Board of Education 6 z 0 5 ?! *5 43 4 I > J r C e p o e 4II. Background on Program Design Background on Program Requirements: Design of the PreK-3 Literacy Program During early fall 1998 a committee was formed in the Division of Instruction of the Little Rock School District to design a new elementary literacy program, with an emphasis on the primary grades of PreK-3. The processes and ultimate design of that plan are described in the PreK-3 Literacy Program Plan in Appendix A. All elementary schools in the Little Rock School District are expected to teach the same curriculum standards and grade-level benchmarks, regardless of the instructional strategies and/or materials that are selected according to the various implementation models. Twenty-seven of the Districts 35 schools are implementing the Early Literacy Learning in Arkansas (ELLA) instructional strategies that are the content of the professional development program for PreK-2 teachers. This model was developed through a collaborative effort that included the Reading Recovery Training Center at the University of Arkansas at Little Rock, the Arkansas Reading Recovery teacher leaders, and the Arkansas Department of Education. Nine schools are implementing the Reading Recovery program, a first-grade intervention, developed by Marie Clay. Seven schools are implementing the Success for All model that was developed at Johns Hopkins University. Little Rock schools receive their training for this program from the University of Memphis. Both ELLA and Success for All training are designed fixjm the same research base on early literacy
they differ in implementation strategies and materials. One school is implementing Direct Instruction through an approved waiver from the District program. Both the Success for All schools and the Direct Instruction school are supplementing their programs, in some cases, with ELLA strategies for greater effectiveness. According to Busbea (2000), In ELLA the importance of helping students feel like readers and writers on the first day of school is stressed. In order to achieve such a goal, teachers must provide students with the needed materials and opportunities for literacy activities. A balanced literacy approach is used to give students these opportunities. The children are engaged in whole text, but they are given formal instruction based on their strengths and needs (30-31). The literacy components taught in the ELLA professional development program, again according to Busbea, are as follows: Read aloud. Shared reading. Guided reading. Familiar reading. Modeled writing or shared writing. Interactive writing. I 7 441 Writing aloud. Revising and editing. Independent writing and conferencing. Phonetic skills. Classroom management. r Each school is required to dedicate a two and one-half hour block of uninterrupted time daily for literacy instruction. :9 3 ' Background on Program Requirements: LRSD Strategic Plan The District adopted its Strategic Plan in 1996, and it was updated in fall 1998. Three of the eleven strategies were important in the development of the PreK-2 Literacy Program Plan: Strategy 2: In partnership with our community, we will establish standards in the core curriculum (reading/language arts, mathematics, science, and social studies) at each appropriate level, as well as develop the means of assessing whether students have met these standards. Strategy 3: We will develop and implement a broad range of alternatives and interventions for students scoring below the Sd'' percentile on standardized tests or who are at serious risk of not achieving District standards in the core curriculum. I z 0 .*) z3 Strategy 7: We will design a comprehensive staff development system to best achieve the mission and objectives in the Strategic Plan. Background on Program Requirements: Revised Desegregation and Education Plan The charge to the design committee of the PreK-3 Literacy Plan included three major sections of the Revised Desegregation and Education Plan that was approved by the federal court in February 1998: Section 2.7, Section 2.7.1, and Section 5.2.1. The first of these sections (2.7) establishes the obligation to improve the achievement of students, especially those who are African American. I> 2 Section 2.7: LRSD shall implement programs, policies, and/or procedures designed to improve and remediate the academic achievement of African- American students, including but not limited to Section 5 of this Revised Plan. zr ec 0P 9 0 On January 21,1998, Mr. John Walker, on behalf of the Joshua Intervenors, signed an agreement with the Little Rock School District that was filed with the federal court, which included the following statement: < With regard to the achievement disparity, the January 16 Revised Plan recognizes that the only legitimate means to eliminate the racial disparity in achievement is by improving African-American achievement (2). 9 8 45 To that end and to address the obligation in Section 2.7, the staff made a conscious decision to emphasize designed to improve... the academic achievement of African- American students, rather than to remediate that achievement, given the failure of most remediation efforts not only in Little Rock, but across the country. This is not to say that the District abandoned its remediation efforts. It did not. Re-teaching, tutoring, Title I programs, computer-assisted instruction, inter-sessions in the Extended Year schools, after-school programs, summer school, and Reading Recovery (first-grade intervention in some schools) continued as much as ever, but as supplemental to the efforts going on in every classroom to prevent as much failure as possible, rather than try to correct failure after it had occurred. These remediation efforts are documented in the schools School Improvement Plans and their Title I Plans. And, of course, the Success for All program implemented in seven LRSD elementary schools and Direct Instruction at Washington Magnet can be described as both preventative and remedial in nature. This decision to emphasize prevention of failure vs. remediation is supported in the published work of the National Research Council, Preventing Reading Difficulties Among Young Children (1998)
the research in scores of studies sponsored by the International Reading Association
and from Marie Clay, who developed the Reading Recovery program. The National Research Council concluded in their massive study the following: The majority of reading problems faced by todays adolescents and adults are the results of problems that might have been avoided or resolved in their early childhood years. It is imperative that steps be taken to ensure that children overcome these obstacles during the primary grades (5). Marie Clay writes the following
Teachers and parents of 11- to 16-year olds often believe that schools have done nothing for the reading difficulties of the young people they are concerned about. Yet the older child has probably been the focus of a whole sequence of well-intentioned efforts to help, each of which has done little for the child. This does not mean that children do not sometimes succeed with a brilliant teacher, a fantastic teacher-child relationship, a hard-working parent-child team. What it does mean is that the efforts often fail (15). Dorothy Strickland makes a similar finding: Historically, educators focused their attention on remediation, allowing children to fail before help was given. The importance of intervening early and effectively is well established among educators and social service providers (325). I t She explains that the cycle of failure often starts early in a childs school career and that there is a near 90% chance that a child who is a poor reader at the end of grade 1 will remain a poor reader at the end of grade 4. Therefore, as the child continues to experience failure and defeat, he/she becomes likely to drop out of school (326). Also, 9 - 46 I' r she states that supplementary remedial programs such as Title I and replacement programs that substitute for regular, in-class instruction have had mixed results over the years (326). She concludes: sr* Those who have turned their attention to early intervention state that it is ultimately less costly than years of remediation, less costly than retention, and less costly to students self-esteem. This final point may be the most compelling of all because the savings in human suffering and humiliation is incalculable. Teachers in remedial programs often observe that students who feel they are failures firequently give up and stop trying to learn despite adequate instructional opportunities (326). Linda Dom (1998), Reading Recovery Teacher Leader Trainer and developer of the Arkansas Early Literacy and Literacy Coach model, and her colleagues French and Jones explain this shift in understanding about teaching as follows: i I7 Recently, Linda asked a group of teachers in a college course how they taught reading to their lowest achieving children. From their responses, it was clear that their theory was a deficit one guided by their concern about how much the children did not know. Traditionally, we have tested children to identify their weak areas and then designed instruction based on what they do not know. This theory of learning is in direct opposition to what research tells us about how the brain acquires information and then organizes related information into larger networks. 0 I ... instruction that is based on inadequate background is grounded in a deficit model, which may force young learners to rely on low-level processes (24-25). 9 I 3 In their summary of Chapter 1, they wrote: Prevention of reading problems must begin in the early grades. If children are not reading on grade level by the end of third grade, their chance of success in later years is minimal. One significant characteristic of problem readers is their lack of literacy experiences during their preschool years. Schools must compensate by providing the children with rich literacy classroom programs and supplemental literacy services that focus on early intervention (15). z p c c c p 0 0 In other words, those who persist in insisting on remediation of learning as the primary emphasis for the lowest-achieving children doom those children to lessons that never get beyond the rote memorization of basic information, and those children will never have an opportunity to understand anything well, much less apply higher-order thinking skills. Dom, si si (1998) urge teachers, therefore, to identify the strengths of young children and use this information as the basis for designing rich learning experiences that emphasize problem-solving (p. 25). In these ways, schools can prevent failure. I 10 47 It should be noted that the District sees its HIPPY and expansive pre-kindergarten program as a part of its overall prevention-of-failure efforts. (See Compliance Report of March 2001, pp. 72-73, for a break-down of the 1312 youngsters involved in early childhood education during 2000-01.) The second section fiom the Revised Plan (2.7.1) requires the District to conduct annual assessments of English language arts and mathematics in order to determine their effectiveness in improving the achievement of Afiican American studentsand then to take appropriate action if the program is not effective by either modifying the programs implementation or replacing it. Section 2.7A: LRSD shall assess the academic programs implemented pursuant to Section 2.7 after each year in order to determine the effectiveness of the academic programs in improving African-American achievement. If this assessment reveals that a program has not and likely will not improve African- American achievement, LRSD shall take appropriate action in the form of either modifying how the program is implemented or replacing the program. ! Prior to fall 1999 there was not in place a reading assessment (except the eight-week assessments in the Success for All schools) that measured student progress in their acquisition of leaming-to-read skills in the early grades. For a time the SAT9 was administered in grades 2-3, but it was not used to drive instructional practice as much as it was used to identify students for the gifted/talented program. The Literacy Benchmark examination required by the State of Arkansas in grade 4 was the first formal assessment of whether students could read independently. The design committee believed strongly that to comply with the Revised Plan and also, importantly, to be able to diagnose potential reading difficulties, as well as to identify progress and growth of individual students, classrooms, schools, and the District, an annual assessment would be required. The District could not afford to wait until grade 4 to find out whether every student had learned to read independently, a goal established in the Revised Plan. After a review of the available literacy assessments for young children and after consulting with the experts involved in the Early Literacy Learning in Arkansas (ELLA) professional development program and with specialists at the Arkansas Department of Education, District staff decided to adopt two sets of measurements-the Observation Survey of Early Literacy Achievement developed by Marie M. Clay and the Developmental Reading Assessment developed by Joetta Beaver. Subsequently, because of a need to have a measurement for the identification of students for the grade 3 gifted/talented program, the Achievement Level Test developed by the Northwest Evaluation Association in collaboration with LRSD teacher teams was added to the assessment plan for grade 2. The results of these data would be the primary basis for evaluating program effectiveness. 11 48 i The third section (5.2.1) of the Revised Desegregation and Education Plan establishes several curriculum, instruction, professional development, assessment, and parental involvement obligations
[ I 1 IIf Readim/Lansuaee Arts Section 5.2.1: Primary Grades. LRSD shall implement at least the following strategies to improve the academic achievement of students in kindergarten through third grade: a. b. c, d. e. g-h. i. j-k. I. Establish as a goal that by the completion of the third grade all students will be reading independently and show understanding of words on a page
Focus teaching efforts on reading/language arts instruction by teaching science and social studies through reading/language arts and mathematics experiences
Promote thematic instruction
Identify clear objectives for student mastery of all three reading cueing systems (phonics, semantics, and syntax) and of knowing-how-to-learn skills
Monitor the appropriateness of teaching/leaming materials to achieving curricular objectives and the availability of such materials in all classrooms
Establish uninterrupted blocks of time for reading/language arts and mathematics instruction
Monitor student performance using appropriate assessment devices
Provideparents/guardians with better information about their childs academic achievement in order to help facilitate the academic development of the students
Provide pre-kindergarten, kindergarten, andfirst grade learning readiness experiences for students who come to school without such experiences
Train teachers to manage successful learning for all students in diverse, mainstreamed classrooms
Use the third and/or fourth grade as a transition year from focused reading/language arts and mathematics instruction to a more traditional school day
and Provide opportunities for students to perform and display their academic training in a public setting. Rather than repeat in this program evaluation the information provided in a number of earlier reports, the relevant pages from those earlier reports are included in the appendices. The document in Appendix E entitled Update on the Implementation of the PreK-3 Literacy Program Plan includes the following list of initiatives that have been implemented from the PreK-3 plan and which require emphasis (pp. 2-3): 12 49 7 0 3 fl 3 CZ! fl 2 2 z r p c a c p 9 9 9 Title I programming was restructured and aligned with the Districts program. A moratorium was placed on adding any new supplemental reading/ language arts programs. Some programs in previous use were abandoned. A waiver was granted to Washington Magnet to keep its Direct Instruction program. Cumculum standards, instructional strategies, instructional materials, assessments, and professional development were tightly aligned. Each school established a sacred, uninterrupted, two and one-half hour daily block for the teaching of reading/language arts. A new English-as-a-Second Language program was implemented that is also tightly aligned with the Districts general education program. New assessments that are developmentally appropriate and aligned with the curriculum and instructional program were implemented. Animated Literacy, a phonemic awareness program, was implemented in kindergarten. Early Literacy Learning in Arkansas (ELLA) was implemented in grades K-2, with Pre-ELLA added in fall 2000 for prekindergarten students. More than $35O,(XX) was expended in the purchase of reading and other cumculum support materials during the past two years. 9 A committee has almost completed work on a new elementary report card. Most primary teachers experienced a minimum of one week of ELLA training, with follow-ups as necessary and appropriate (See Compliance fiSgOU in Appendix C for lists of professional development sessions.) The Parent-School Compact was revised, and the Student Academic Improvement Plan (SAIP) was developed and implemented. The Parent Program was restructured in May 2000. An ESL Parent Coordinator was employed in spring 2001. 13 50 f 3 in. The Assessments 1 ' - -Jg h H I I i Marie Clay makes the point repeatedly in her book, An Observation Survey of Early Literacy Achievement (1993), that no one observation task is satisfactory on its own when one needs to make important instructional decisions for children (p. 20). She would find strong support from Grant Wiggins, who is a national expert in assessment. In his book, Assessing Student Performance: Exploring the Purpose and Limits of Testing (1993), Wiggins wrote: One test signifies nothing, let us emphatically repeat, but five or six tests signify something. And that is so true that one might almost say, It matters very little what the tests are so long as they are numerous (13). In the Little Rock School District, the tests are numerous. The Assessments: Observation Survey Below is siunmary information about what the five sub-tests in the Observation Survey measure. Letter Identification This sub-test answers the following questions: What letters does the child know? Which letters can he/she identify? All letters, lower and upper case, are tested. The observation includes an analysis of the childs preferred mode of identifying letters
the letters a child confuses
and the unknown letters. (Clay, p. 43) The maximum score is 54. This test is administered in grades K-1. 5 I ?! V. Word Test The student is tested over the most fi-equently occurring words in whatever basic reading texts are being used. Scores on this measure are useful in determining a childs readiness to read. (Clay, p. 53) The maximum score is 20. This test is administered in grades K-2. Concepts about Print This sub-test (5-10 minutes) includes testing whether the student knows the front of the book, that the print (not the picture) tells the story, that there are letters, that are clusters of letters called words, that there are first letters and last letters in words, that you can choose upper or lower case letters, that spaces are there for a reason, and that different punctuation marks have meanings. Scores on this measure have proven to be a sensitive indicator of behaviors that support reading acquisition. (Clay, p. 47) The maximum score is 24. This test is administered in grades K-1. Writing Vocabulary The student is asked to write down in ten minutes all the words he/she knows how to write, starting with his/her own name and making a personal list of words 14 51 9 I {fl z r p c p 0 9he/she has managed to learn. There is no maximum score. This test is administered in grades K-2. Hearing and Recording Sounds in Words The teacher asks the child to record a dictated sentence. The childs performance is scored by counting the childs representation of the sounds (phonemes) by letters (graphemes). The maximum score is 37 at grades K-1 and is 64 at grade 2. This test is administered in grades K-2. The Assessments: Developmental Reading Assessment The Developmental Reading Assessment is a one-on-one assessment of reading skills primarily accuracy of oral reading and comprehension through reading and re-telling of narrative stories. The assessment consists of stories that increase in difficulty. Factors which contribute to the gradient of difficulty of the stories include the number of words on a page, complexity of vocabulary, length of the stories, degree of support from the pictures, as well as complexity of sentence and story structure. The assessment formats are as follows: Levels A-2 (Kindergarten Grade Level), 7-8 minutes 1. 2. 3. 4. 5. 6. Teacher selects book Teacher introduces text Teacher reads one or two pages Child points and reads rest of story
teacher takes running record Teacher asks print questions Teacher asks preference questions Levels 3-16 (First Grade Level), 10-15 minutes 1. Teacher selects book 2. Teacher introduces text 3. Child looks at pictures
tells what is happening 4. 5. 6. 7. Child reads story aloud
teacher takes running record Child retells story Teacher asks response questions Teacher asks preference questions Levels 18-44 (Second Grade Level), 15-20 minutes 1. Teacher selects range of three texts 2. Child previews and chooses one 3. Teacher introduces text 4, 5. 6. 7. 8. 9. Child reads first 2-4 paragraphs aloud Child predicts what will happen in story Child reads complete story silently in another location Child retells story Teacher asks response questions Child reads selected portion of text
teacher takes ruruiing record 15 52r 10. Teacher asks preference questions 11. Teachers asks one or two inference questions (Levels 28-44). 3 3 Readiness levels for the Little Rock School District have been established as follows: KindergartenLevel 2 Grade 1Level 16
and Grade 2Level 24. 4e 5 The explanation below (developed in summer 2000) on Definition of Readiness vs. Proficiency is a delineation of the Districts efforts to define appropriate cut scores for each grade level so that a determination could be made of the percent of students who are achieving a standard of readiness for success at the next grade level. I' I Definition of Readiness vs. Proficiency The Arkansas Department of Education has defined performance at four levels: Below Basic, Basic, Proficient, and Advanced for the Benchmark examinations that are administered at grades 4, 6, and 8 and the end-of-level examinations for designated high school courses. Proficient is the performance standard that all students should achieve. The ADE definition follows: Proficient students demonstrate solid academic performance for the grade tested and are well-prepared for the next level of schooling. They can use Arkansas established reading, writing, and mathematics skills and knowledge to solve problems and complete tasks on their own. Students can tie ideas together and explain the ways their ideas are coimected. The Developmental Reading Assessment allows teachers to assess reading levels of students through a one-on-one test reading conference between teacher and student. Teachers observe student performance during the test, make notes on reading behaviors, and score the performance as they go along. V! 2J 5 fl fl I> 2 r I The desire was to establish appropriate cut points that would define proficient performance. To gauge which level is equivalent to how Arkansas defines proficiency, the staff used national reading standards for each grade level as defined in Reading and Writing Grade bv Grade: Primary Literacy Standards from Kindergarten through Third Grade (New Standards Primary Literacy Committee, National Center on Education and the Economy and the University of Pittsburgh, 1999). The staff then identified the DRA level that corresponds to that specific performance. Standards and DRA equivalents by grade level follow: fl z r p c e p 9 0 f I 16 53 Grade Level Kindergarten Grade I I I Grade 2 Reading Standards Children at the end of kindergarten should understand that every word in a text says something specific. They can demonstrate this competence by reading Level B books that they have not seen before, but that have been previewed for them, attending to each word in sequence and gening most of them correct. By the end of the year, we expect first-grade students to be able to
read Level 16 books that they have not seen before, but that have been previewed for them, with 90 percent or bener accuracy of word recognition (self-correction allowed). When they read aloud, we expect first graders to sound like they know what they are reading. Fluent readers may pause occasionally to wo^ out difficult passages. By the end of the year, we expect first-grade students to be able to independently read aloud from Level I books that have been previewed for them, using intonation, pauses and emphasis that signal the structure of the sentence and the meaning of the text. By the end of the year, we expect second-grade students to be able to independently read aloud unfamiliar Level 24 books with 90 percent or bener accuracy of word recognition (selfcorrection allowed). DRA Level_______________________________________ Assessment texts A through 2 consist of a repeated word or sentence panem with natural language structures. The simple illustrations include animals and objects familiar to primary children and highly support the text. One or two lines of text appear on the left page and are large and well spaced so that children can point as they read. The number of words in the texts ranges fiom ten to thirty-six._______________________ Assessment texts 16 through 28 are stories with beginnings, middles, and ends, throughout which problems are presented and resolved. The characters are either imaginary (giants and elves) or animals with human characteristics. The content begins to move beyond children's personal experiences and builds a basis with which to compare and contrast other stories. Literacy language structures are integrated with natural language. Some description of characters and setting is included. Illustrations provide moderate to minimum support. The text may be three to twelve lines above or beneath the illustrations, or a full page. The number of words in these texts starts at 266 and increases with each level of difficulty. Assessment texts 16 through 28 are stones with beginnings, middles, and ends, throughout which problems are presented and resolved. The characters are either imaginary (giants and elves) or animals with human characteristics. The content begins to move beyond children's personal experiences and builds a basis with which to compare and contrast other stories. Literacy language structures are integrated with natural language. Some description of characters and sening is included. Illustrations provide moderate to minimum support. The text may be three to twelve lines above or beneath the illustrations, or a full page. The number of words in these texts starts at 266 and increases with each level of difficulty.___________________________________________ The staff also considered the work of others who use the DRA in their determination of appropriate cut points to define proficiency at each grade level. Several sUtes and many school districts have adopted the DRA for early literacy assessment. One example is the chart establishing proficiency levels developed by the East Baton Rouge Parish School System in Louisiana. They have determined that On Grade Level is defined by a kindergarten students performance at levels 1,2 on the DRA
grade 1 is levels 16,18
and grade 2 is levels 24,28. Above Grade Level is defined as levels 3-14 at kindergarten
levels 20-28 at grade 1
and levels 30-38 at grade 2. In Lindsay, California, the Approaching Proficiency levels are defined similarly: level 2 at kindergarten
levels 10-12 at gr^e 1
and level 24 at grade 2. 17 54 3 5^^ IS fl 1 I 4 A program evaluation conducted by the Austin, Texas, Independent School District indicates that the Grade Level performance on the DRA was defined level 2 at kindergarten
level 16 at grade 1
and levels 24-28 at grade 2. as The State of Ohio defined Success Indicators for reading for each grade level. These can be compared to the national standards developed by the National Center for Education and the Economy: At the end of kindergarten, children should be able to write in a left to right/top to bottom manner, have a firm grasp of letters and their sounds, and recognize a few simple words. By the end of first grade, students should be using and integrating phonics and reading strategies as they read, writing simple stories, reading independently, and demonstrating comprehension of stories through drawing, writing, discussion, and dramatization. By the end of second grade, students should be reading silently for extended periods and reading orally with appropriate use of punctuation. They should demonstrate that they can gather information by reading, predict how stories will end, compare and contrast story elements, sequence evens from a story, retell a story, and relate what they read to their lives. The State of Connecticut uses the Developmental Reading Assessment as a part of their state accountability system in grades 1-3. Grade 1 students who perform at or below level 10 and grade 2 students who perform at or below level 16 at the end of the year are identified as substantially deficient. Such students then receive a personal or individual reading plan that outlines additional instructional support and monitors student progresssimilar to the Districts Student Academic Improvement Plan (SAIP). Although Connecticut does not identify grade-level proficiency levels, they have established the literacy standard for LEP students to exist the bilingual program: at kindergarten the student must perform at level 2
at grade 1 level 16
and at grade 2 level 28. Vermont, likewise, uses the DRA in their state assessment program and has established similar levels of proficiency. 7- 9 5 I Vi 9 I > i Joetta Beaver, the developer of the Developmental Reading Assessment (published by Celebration Press in 1997), suggests that districts should define proficiency levels so that students performing below those levels receive necessary interventions and remediation. Her recommended proficiency levels are levels 1-2 for kindergarten
levels 16-18 at grade 1
and levels 24-28 at grade 2. All these efforts to define proficiency are either exactly aligned with the decisions made by LRSD sUff or are very close. 18 55 pi y
r 9 C P 9 9 <Given, however, the difficulty of establishing with confidence an equivalent defimtion of proficiency that would predict achievement on the grade 4 Benchmark examination. District staff members have made the decision to use what in their best judgment are the appropriate cut scores (based on all the research cited), but to use the term Readiness to define the desired perfonnance. VA/MAM T\a>A_^A 1_____1x* Ma When the District has multiple years of data and when the 1999-2000 kindergarten students take the Grade 4 Benchmark examination in spring 2004 then the staff can do some stotistical calculations that will enable the District set cut scores that reliably predict Proficient performance on the grade 4 Benchmark. to I 1 I I Reliability and Validity: National Study The development of the Developmental Reading Assessment began in 1988 by a team of teacher-researchers. According to the national validation study, the purpose of the assessment was to guide teachers ongoing observations of student progress over time withm a hterature-based reading program (p. 2). Over the next six years ere were numerous revisions in response to teacher feedback. In spnng 1996 the first formal validation study was conducted. Seventy-eight teachers fi-om various parts of the United States and Canada participated, (p. 3) The results of the study were very positive, and where the correlations were not as strong as they possibly could be, revisions to the instrument were made to strengthen validity. In summary, the I Iz A W M TAI Am. t* M M . _ 1 - J ..._________ * M - * DRA was found to be a valid assessment. Teachers found it very helpful in deterluirung individual students instructional text reading level
describing his/her performance as a reader
selecting appropriate interventions and/or focus for instruction
and identifying students who may be reading below proficiency (11). A reliability study of the Developmental Reading Assessment was conducted in spring 1999 by Dr. E. Jane Williams. In this study eighty-seven teachers from ten states participated. All had prior experience in administering the DRA. The findings were at both the inter-rater reliability and the internal consistency of the test were strong to very strong (6). The construct validity of the DRA was also established through an additional study. Construct vahdity ensures that the test measures what was intended that it measure. The statistics for this study were done using DRA individual student scores compared to individual scores on the Iowa Test of Basic Skills. They correlated positively, and for the ITBS Total Readmg subscale, very positively. The conclusion, then, was that the DRA validly measures a childs ability to decode and understand/comprehend what he/she has read (6). Of importance to the LRSD was another conclusion to this study: It should be noted that a major purpose of the DRA is to help guide instruction. Ninety-eight percent of the teachers and raters agreed or strongly agreed to the 4 s a 19 56I-I I .< I statement that the information gained about the reader during the DRA conference helped them better identify things that the child needed to do or learn next ( 9). 5 3 < d It was the intent of the design committee and is the intent of the curriculum staff that the multiple assessments selected for grades K-2 be used to drive instructionfor the data gathered from those assessments to be used to assist teachers in deciding what to do next for each individual child. LRSD embraces the joint position statement of the International Reading Association and the National Association for the Education of Yoimg Children that was adopted in 1998: [ Throughout these critical years accurate assessment of childrens knowledge, skills, and dispositions in reading and writing will help teachers better match instruction with how and what children are learning. However, early reading and writing cannot be measured as a set of narrowly defined skills on standardized tests. These measures often are not rehable or valid indicators of what children can do in typical practice, nor are they sensitive to language variation, culture, or the experience of young children. Rather, a sound assessment should be anchored in real-life writing and reading tasks... and should support individualized diagnosis needed to help young children continue to progress in reading and writing (20). i Reliability and Validity: LRSD Study The following correlational matrix constructed by the Districts statistician in spring 2001 displays the relationships between the scores on the Achievement Level Tests (ALTs) and the Observation Survey and Developmental Reading Assessment scores. 9 3 I ?! ?! CorrelatioBal Matrix* Spring 2001 ALT Reading RTF, ALT Reading Goal RTTi, Olmrvation Survey, and DRA Scores e I> Goal 1: Word Meaniog Goal 2: Literal Comprebeu SMB Goal 3: Goal 4: Interpretive Evaluative Compreheo Coapreben Observation Survey: Word Test SIM sion Observation Survey: Writing Vocabulary Observation Survey: Dictation DRA Reading RTF Skore Goal 1: Word Meaning Goal 2: Literal Comprehension 0.937 0.940 0.839 0.922 0.805 0.823 0.917 0.815 0.822 0.280 0.255 0.223 0.467 0.438 0.418 0.638 0.602 0.577 0.788 0.733 0.12A Goal 3: Interpretive Comprehension Goal 4: Evaluative Comprehension Observation Survey: Word Test Observation Survey: Writing Vocabulary ObMrvatiM Survey: Dictation__________ 0.795 0.199 0.207 0.410 0.413 0.276 0.535 0.574 0.351 0.442 0.696 0.719 0.360 0.478 0.683 P c p 9 9 AU conclahons arc significant at the .05 level < N's range from 1577 to 1684 While all the relationships are significant at the .05 level, some relationships are stronger than others. All of the ALT scores relate strongly to the DRA, with values of .696 to 20 57 .788. Only Hearing and Recording Sounds (Dictation) on the Observation Survey has value above .50-.683. Also, within the Observation Survey correlational values are a lower. The staff anticipated this result since the Observation Survey measures leaming- how-to-read skills, and the Developmental Reading Assessment measures more difficult comprehension skills. The large sample size gives power to this matrix and contributes to significance at apparent low correlational values. 1 The statistician subsequently ran a statistical test called Cronbachs Alpha, which is a reliability test for internal consistency of an assessment. Reliability is a measure of a tests stability
that is, if one gives the same test more than once, a reliable test would produce a similar or same result. A test with an acceptable Alpha indicates that the variability in scores is a result of the test taker, while a low Alpha indicates that the variability in scores is a result of a poorly designed or inconsistent test. A test with an Alpha of .60 and greater is usually considered to be internally consistent. The Alpha coefficients for the Observation Survey and the DevelopmenUl Reading Assessment for both fall and spring administrations are as follows: 1 Fall K Grade 1 Grade 2 .63 .66 .74 Spring .85 .62 .65 Therefore, both the Observation Survey and the Developmental Reading Assessment appear to have stability and are internally consistent. The Alpha for the spring grade 2 Achievement Level Test is .97. What these data are indicating is that the Developmental Reading Assessment is a valid and reliable test The lower correlation values of the Observation Survey are more likely a product of these tests measuring pre-reading knowledge and skills, as opposed to the reading comprehension skills measured on the grade 2 Achievement Level Test Developmental Appropriateness of Testing Instruments Both the sub-tests on the Observation Survey and the Developmental Reading Assessment are admimstered one-on-one by the classroom teacher to the student. The teacher scores the students performance, based upon rubrics and scoring instructions provided to the teacher in a mandated training session and in writing. The teacher then bubbles in on each childs answer sheet his/her level of performance and sends those answer sheets to the Director of Early Literacy for processing and the compilation of scoring reports. One caution, therefore, in interpreting the data is that the teacher has scored his/her own students performance, and bias may be possible. The District has conducted a procedure to verify the accuracy of the spring scoresthose most likely to be influenced by bias. Students spring scores are matched with their fall scores the following year, and then if there is a wide discrepancy, that score can be flagged. When there is a pattern of significantly higher spring scores from one teacher than the next years fall scores, then an investigation must be conducted. One school with suspiciously high spring scores was flagged for review in fall 2000. However, when 21 58* 5 the match of scores was run, the staff found absolutely no evidence of cheating. The fall 2000 scores were closely in line with those of the previous spring, even though the children in the fall were in several different schools, and there were more than seven teachers administering the fall tests. The steff also has collected some anecdotal evidence that a few teachers may, in fact, be under-reporting student achievement rather than overreporting, due to their own low expectations. I a 3 To avoid even the appearance of bias, some would recommend that the District use a standardized examination with individual students writing their own answers and then the answer sheets scored by machine. The problem with this approach is that the results would likely be even more questionable than the ones produced through one-on-one testing. Experts in early literacy and in early education have developed strongly stated positions against the use of standardized tests for young children, ages 3 through 8. For example, a position statement, Learning to Read and Write: Developmentally Appropriate Practices for Young Children, was issued in 1998 by the International Reading Association (IRA) and the National Association for the Education of Young Children (NAEYC). The section on assessment follows: 1 Group-administered, multiple-choice standardized achievement tests in reading and wnting skills should not be used before third grade or preferably even before fourth grade. The younger the child, the more difhcult it is to obtain valid and reliable indices of his or her development and learning using one-time test administrations. Standardized testing has a legitimate function, but on its own it tends to lead to standardized teachingone approach fits allthe opposite of the kind of individualized diagnosis and teaching that is needed to help young children continue to progress in reading and writing (11). A 1987 position paper by NAEYC, Standardized Testing of Young Children 3 Through 8 Years of Age, is even more explicit: Young children are not good test takers. The younger the child, the more inappropriate paper-and-pencil, large group test administrations become. Standards for administration of tests require that reasonable comfort be provided to the test taker (AERA, APA, & NCME, 1985). Such a standard must be broadly interpreted when applied to young children. Too often, standardized tests are administered to children in large groups, in unfamiliar environments, by strange people, perhaps during the first few days of school or under other stressful conditions. During such test administrations, children are asked to perform unfamiliar tasks, for no reason that they can understand. For test results to be valid, tests are best administered to children individually in familiar, comfortable circumstances by adults whom the child has come to know and trust and who are also qualified to administer the tests (5). In conclusion, therefore, the staff made the determination that the Observation Survey and the Developmental Reading Assessment met all the criteria for selecting good assessment instruments for the children in K-2 classrooms. They were closely aligned 22 59 c 3 I fl 3 cz: fl flI> Z j* C c p e 0 4 }| with the curriculum and teaching strategies that were to be used by teachers
they measured the leaming-to-read skills that were essential for children becoming independent readers
they provided teachers with necessary diagnostic and sununative dau
they were developmentally appropriate
their administration procedures met test administration standards for young children
and their results were much likely to be valid and reliable than if a standardized test was used. The Assessments: Achievement Level Tests in Readmg and Language Usage The Achievement Level Test (ALT) at grade 2 in reading and language usage was first administered in spring 2000. The ALTs are a series of tests that are aligned with the Little Rock School District curriculum and the Arkansas state standards. Because the scores are along one continuum over the grade levels, they allow staff and others who are interested to calculate the amount of growth for individual students, classrooms, schools, and the District as a whole fi-om year to year. With the ALTs, students take tests at a level that matches their current achievement level. The test should be challenging, but neither too difficult nor too easy. Because the tests match the achievement level of the student, teachers receive accurate information that helps them to monitor each students academic growth. ALTs are not timed, and they take about one hour per subject for most students. The District scores the ALTs, and the results are returned to the schools as quickly as possible, sometimes within 48 hours. Any retesting that is necessary is completed, so school reports carmot be printed until all testing is finished, and district reports cannot be completed until all schools finish their testing. Reports are also produced for parents, teachers, and administrators. Once a student has been through two administrations of the ALTs, a trend report is produced for parents that allows them to monitor the growth of their child compared to the growth of the District and the growth of the national group that takes the test. Student progress is reported in a scale score called the Rasch Unit (RTT). It is an equal interval measure. It can be compared to measuring a childs physical growth in inches and then comparing it to an expected growth chart. The test measures achievement growth with a RIT scale and compares the growth to an expected national growth chart. By monitoring the growth of students, staff can pinpoint areas where individual students might need extra help or attention. District staff and Campus Leadership Teams use the information to make data-driven decisions about school improvement plans, curriculum and instructional changes, and professional development needs. The scores are also used in program evaluations. There are four goals/standards that are measured on the reading sub-test: 1. Word Meaning A. Phonetic skills B. Context clues C. Synonyms, antonyms, homonyms D. Component structure (prefix, suffix, origin, roots) E. Multiple meanings 23 604 I 2. 3. I I 4. Literal Comprehension A. Recall/identify significant details B. Identify main idea C. Locate information D. Follow directions E. Sequence details Interpretive Comprehension A. Inference B. Identify cause and effect C. Authors purpose D. Prediction E. Summarize F. Identify literacy elements (character, plot, setting, theme, etc.) Evaluative Comprehension I A. B. C. D. E. Evaluate conclusions, validity (supporting context) Identify fact and opinion Identify literary techniques (figurative language, mood, tone, etc.) Distinguish text forms Identify bias, stereotypes. 1 Three goals/standards are tested on the Language Usage sub-test: 1. Writing Process A. B. C. D. E. F. Prewriting skills Drafting and revising- Editing/proofreading Choosing appropnate format Sentence choice appropriate to purpose Paragraph skills (topic and concluding sentences, indenting, etc.) z 93 fl W! fl 9 I >s 2 2. 3. Grammar and Usage A. Sentence patterns B. Phrases and clauses C. Noun forms D. Verb usage: tenses, irregular verbs, subject-verb agreement E. Adjective forms F. Adverb forms G. Pronoun forms H. Pronoun-antecedent agreement I. Negative forms Mechanics A. End punctuation B. Commas C. Apostrophes D. Enclosing punctuation E. Titles F. Beginning capitalization G. Proper nouns and adjectives pi r p c p 90 24 61 H. Capital I The staff made a deliberate decision to delay the use of this formal, group-administered test until the end of second grade. Even then, many teachers, principals, central office staff, and parents question its usefulness in measuring leaming-to-read skills and knowledge. The data are included in this program evaluation because they exist and because they provide another measurement of student achievement that may be used to inform decision-making about the program. I 25 62I IV. Alignment with National Research on Early Literacy Background on the Context: National Research on Early Literacy A publication of the National Research Council (1998), Preventing Reading Difficulties in Young Children, is nationally recognized, and it was used to a high degree in the design of the LRSD PreK-2 literacy program. Below is a short summary of the reports recommendations for early learners: Prekindergarten: Preschool programs ... should be designed to provide optimal support for cognitive, language, and social development, within this broad focus. However, ample attention should be paid to skills that are known to predict future reading achievement, especially those for which a causal role has been demonstrated. 1 Kindergarten: Kindergarten instruction should be designed to stimulate verbal interaction
to enrich childrens vocabularies
to encourage talk about books
to provide practice with the sound structure of words
to develop knowledge about print, including the production and recognition of letters
and to generate familiarity with the basic purposes and mechanisms of reading. Beginning readers need explicit instruction and practice that lead to an appreciation that spoken words are made up of smaller units of sounds, familiarity with spelling-sound correspondences and common spelling conventions and their use in identifying printed words, sight recognition of fi'equent words, and independent reading, including reading aloud. Fluency should be promoted through practice with a wide variety of well- written and engaging tests at the childs own comfortable reading level. ^5 2 I pl (/) pl > 2 Children who have started to read independently, typically second graders and above, should be encouraged to sound out and confirm the identities of visually unfamiliar words they encounter in the course of reading meaningful texts, recognizing words primarily through attention to their letter-sound relationships. Although context and pictures can be used as a tool to monitor word recognition, children should not be taught to use them to substitute for information provided by the letters in the word. z r p p e a Because the ability to obtain meaning from print depends so strongly on the development of word recognition accuracy and reading fluency, both of the latter should be regularly assessed in the classroom, permitting timely and effective instructional response when difficulty or delay is apparent. Beginning in the earliest grades, instruction should promote comprehension by actively building linguistic and conceptual knowledge fl 26 63in a rich variety of domains, as well as through direct instruction about comprehension strategies such as summarizing the main idea, predicting events and outcomes of upcoming texts, drawing inferences, and monitoring for coherence and misunderstandings. This instruction can take place while adults read to students or when students read themselves. Once children learn some letters, they should be encouraged to write them, to use them to begin writing words or parts of words, and to use words to begin writing sentences. Instruction should be designed with the und
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.