Evaluation Report: Timeline

Page 1 of 1 Dejarnette, Karen From: Sent: To: Dejarnette, Karen Tuesday, October 17, 2006 1:50 PM Roberts, Olivine
Hattabaugh, Hugh
Brooks, Roy G
HELLER@fec.net Subject: timeline for evaluation reports As you know three of the draft evaluation reports were submitted yesterday to the court. Final drafts are to be submitted by November 17. I need your assistance to define a timeline for Board members to review the draft and provide feedback to evaluators so they can produce final drafts for submission to the court on Nov. 17. Based on feedback from board members attending last weeks board meeting they seem to want hard copies of lengthy reports. Therefore, I am printing copies of the reports this afternoon and tomorrow and will bring enough copies to Beverly tomorrow for Board members. I will also send copies of each report to Cabinet members via interdistrict mail as soon as they are printed. I need your assistance to complete the timeline below, see number 4: 1. Copies of draft reports delivered to evaluation team members, Cabinet and Board members as soon as possible, or by Friday. 2. Evaluation teams will meet at the IRC (with evaluators on conference call) to give feedback on October 25 (Read 180 and 21 Century) and October 30 (A+). 3. Cabinet members to provide feedback by or during October 30*^ Cabinet meeting. 4. Board members to provide feedback on ??? Questions-Will a special board meeting be called so board members can provide feedback before November O? Or, will board members provide feedback on November 9 (NOTE this would only leave 1 week for evaluators to edit and finalize the reports before they are due to the Court and I am not sure this is enough time.) Also,do you want the evaluators to be invited to present their reports to board members or answer questions? Thanks for assisting. 10/17/2006Page 1 of 1 Dejarnette, Karen From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Cc: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true. Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) jim.wohlleb@lrsd.org 10/17/2006Notes of discussion about the second annual survey for the monitoring report October 2, 2006 Attending: Mr. Hugh Hattabaugh, Dr. Olivine Roberts, Mr. Joe Mittiga, Dr. Karen DeJamette, Ms. Maurecia Robinson, Dr. Ed Williams, and Mr. Jim Wohlleb Olivine summoned Jim and Karen to the administration building for a conference, and Karen invited Ed and Maurecia. PRE Department members thought it would be about the authority of PRE to determine what it assesses. Instead, Dr. Roberts led discussion about this years survey of teachers, parents, and students for the monitoring report. Mention of on-line surveys by Education for the Future (EFF), ready for administration now, was met by Mr. Hattabaughs declaration that any services by outside organizations require RFPs. Dr. DeJarnette noted that EFF is so busy it does not consider RFPs. In her opinion, LRSD is turning away from the best methods and services and instead using its own unvalidated measures. Given this, she prefers that Mr. Mittigas office rather than PRE conduct the survey for the monitoring report. All agreed on a more attractive survey instrument than last years and distribution by some means other than USPS. Showing high priority to the survey is a way to increase participation. For ES parents, teachers can ask them to answer questionnaires at the start of conferences with teachers. This might not work so well with parents of middle & high school students. There was agreement by both Dr. Roberts and Mr. Mittiga that last years questions were ambiguous. They also endorsed stakeholder participation in the design. Martha Hill was mentioned as a good participant. Afterwards, Mr. Wohlleb sent copies of the four EFF questionnaires to Mr. Mittiga, and he sent around a copy of the survey he designed with UALR but did not administer due to lack of funds. Comments sent with the EFF documents noted the intended application of them in PREs assessment of the teacher performance challenge as step 2 evidence.Page 1 of 1 Dejarnette, Karen From: Sent: To: Cc: Roberts, Olivine Thursday, October 12, 2006 2:01 PM Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: Mr. Wohlleb, I do not know what was told to the PRE staff regarding the purpose of the meeting, but my sole intended purpose was to discuss the monitoring report. Please correct the minutes to reflect that. Thank you. Olivine ()(h'i.ne 'Rolnrts, TiCl). ..Associate Super mt eiutent, Tducat tonal Services fittie Rock ScHool 'Dblriti 300
.s'. 'Puluslii St. Little Rock, S\R 72206 Jfione: soi.447.ss^o fa.v: 501.4473321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true. Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) jim.wohlleb@lrsd.org 10/17/2006Page 1 of 2 Dejarnette, Karen From: Sent: To: Cc: Wohlleb, Jim Thursday, October 12, 2006 2:16 PM Roberts, Olivine: Hattabaugh, Hugh
Mittiga, Joseph Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis. Darral Subject: RE: climate survey Certainly, Ill correct my notes. They are silent on the matter of not proceeding with the survey prepared by EFF. Do you recall whether it was resolved during that discussion? Thanks. From: Roberts, Olivine Sent: Thursday, October 12, 2006 2:01 PM To: Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE
Mr. Wohlleb, I do not know what was told to the PRE staff regarding the purpose of the meeting, but my sole intended purpose was to discuss the monitoring report. Please correct the minutes to reflect that. Thank you. Olivine Oliyine 'Roberts, 'i'lL'D. SAssociale Superinleyhleiit, TdiK ulloiiul Services Lit lie 'R/)ck School 'Jlistrict _so()i S. 'PuUisHi St. Little Ri)ck, .AR 72200 Phone: 501.447.3320 fax: 501-447.3321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true. Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 10/17/2006Page 2 of 2 501/447-7609 (office fax) 501/680-9244 (mobile) jim.wohlleb@lrsd.org 10/17/2006Page 1 of2 Dejarnette, Karen From: Sent: To: Cc: Roberts, Olivine Thursday, October 12, 2006 2:18 PM Wohlleb, Jim
Hattabaugh, Hugh
Mittiga. Joseph Dejarnette. Karen
Robinson. Maurecia
Williams. Ed
Paradis. Darral Subject: RE: climate survey The group agreed to use a locally developed instrument. 06VIne Hofierly, Id.'l). .'Associate Supennlendent, Educational Services Eittle 'Jiock .School 'District 3001 S. '.Pulaski St. Tittle 'Hack, yX'H 7.. J^TOiie: 51)1.44/ .-5 2(i(i '20 Tax: 501.44^.5521 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 2:16 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Certainly, Ill correct my notes. They are silent on the matter of not proceeding with the survey prepared by EFF. Do you recall whether it was resolved during that discussion? Thanks. From: Roberts, Olivine Sent: Thursday, October 12, 2006 2:01 PM To: Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: Mr. Wohlleb. I do not know what was told to the PRE staff regarding the purpose of the meeting, but my sole intended purpose was to discuss the monitoring report. Please correct the minutes to reflect that. Thank you. Olivine Olh'ine 'Roherts, 'EcL'D. .Associate Superhdeiulen.t, 'Educationa.l Services Eittte 'Rock School 'DistricI 3001 S. 'Eutaski St. Eittte 'Rock, .A'R. 7'2206 'Phone: 501.447- '33'2o 'Fax: 501.447.3321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph 10/17/2006Page 2 of 2 Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true, Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock. AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) jim.wohlleb@lrsd.org 10/17/2006Page 1 of2 Dejarnette, Karen From: Sent: To: Cc: Dejarnette, Karen Thursday, October 12, 2006 2:43 PM Wohlleb, Jim
Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Jim, I understood there were two purposes for the meeting. See the following email I received from Dr. Roberts on Friday September 29*^ at 9:29a.m: Lets meet on Monday following Cabinet to discuss the Monitoring Report and the Climate Survey. Please ask Jim to attend. Thank you. Olivine 'Roberts, id.J). .'Associate Superintendent, Lducalionai Services Little Rocfi SeboolDistrict 30111 S. Puiasbi SI. Little 'Rock. LVR 72206 'Rhone: so 1.447.3320 Lax: 501.447.'3321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 2:16 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Certainly, I'll correct my notes. They are silent on the matter of not proceeding with the survey prepared by EFF. Do you recall whether it was resolved during that discussion? Thanks. From: Roberts, Olivine Sent: Thursday, October 12, 2006 2:01 PM To: Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: Mr. Wohlleb, I do not know what was told to the PRE staff regarding the purpose of the meeting, but my sole intended purpose was to discuss the monitoring report. Please correct the minutes to reflect that. Thank you. Olivine OLivine Roberts, Rd/D. .'Associate Superintendent, 'Rdiicational Services Little .Scfiooi''District 10/17/2006Page 2 of 2 S. 'Pulaski St. PitlCe 'Rocd, .:AR ^2206 Th m ic>: 501.447.33^ < > fax: 501.447.3321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true. Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb. Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) ji m. wohl leb@ Irsd. org 10/17/2006Page 1 of2 Dejarnette, Karen From: Sent: To: Cc: Roberts, Olivine Thursday, October 12, 2006 3:07 PM Dejarnette, Karen
Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Karen, you are right. That is why it was a part of the discussion. Olivine 'Rolwrls, I'cl.l). .'Associate Superinienclint. 'Lducat ional Service.^ fill Ie Jioik Scliool'Distcicl sooz S. J'ul'aski SI. fit tie 'Rock. fXR ~2^(>6 'Rhone: sc)i.447..s:-i2() lux: s(.it.447cfS'2i From: Dejarnette, Karen Sent: Thursday, October 12, 2006 2:43 PM To: Wohlleb, Jim
Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Jim. I understood there were two purposes for the meeting. See the following email I received from Dr. Roberts on Friday September 29*^ at 9:29a.m: Let's meet on Monday following Cabinet to discuss the Monitoring Report and the Climate Survey. Please ask Jim to attend. Thank you. Olivine RoRerLs, fd.'D. .'Associate Superintendent, 'f ducal ional Services fit tie 'Rock School District sooi S. 'Rulaski St. fit lie 'Rock, .'A'R -2206 Rli 01 le: 501.44/.3420 J'ax: 501.447.34'21 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 2:16 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: climate survey Certainly, Ill correct my notes. They are silent on the matter of not proceeding with the survey prepared by EFF. Do you recall whether it was resolved during that discussion? Thanks. From: Roberts, Olivine Sent: Thursday, October 12, 2006 2:01 PM 10/17/2006Page 2 of 2 To: Wohlleb, Jim
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: RE: Mr. Wohlleb, I do not know what was told to the PRE staff regarding the purpose of the meeting, but my sole intended purpose was to discuss the monitoring report. Please correct the minutes to reflect that. Thank you. Olivine Olivhie 'IWberl.'!, Td. J). .'/Xssoeiale Superinlendeiil, Tducaliorml Services U.t(e 'Ruck Sebool 'DLslrix I 30(11 S. 'Ililasl^i SI. RillLe 'Rock, .'A'R 72206 'Rhone: 301.447.3320 fax: 501.447.3321 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE won't go forward with the climate" survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If that's true. Id like to inform EFF that LRSD won't follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) jimw0hlleb@lrsd.9rg 10/17/2006Page 1 of 1 Dejarnette, Karen From: Sent: To: Cc: Hattabaugh, Hugh Monday, October 16, 2006 7:11 PM Wohlleb, Jim Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral
Roberts, Olivine
Mittiga, Joseph Subject: I stated that a RFP needed to be processed through LRSD Procurement, if we proceed with EFF. It was stated that EFF is to busy to be subjected to the RFP process. If a RFP for the survey instrument and services is not processed, your conclusion is correct. Sincerely, Hugh E. Hattabaugh, Deputy Superintendent Little Rock School District 810 West Markham Street Little Rock, AR 72210 (W) 501-447-1009 (C) 501-580-6815 (FAX) 501-447-1159 From: Wohlleb, Jim Sent: Thursday, October 12, 2006 12:52 PM To: Roberts, Olivine
Hattabaugh, Hugh
Mittiga, Joseph Cc: Dejarnette, Karen
Robinson, Maurecia
Williams, Ed
Paradis, Darral Subject: Mr. Hattabaugh & Dr. Roberts, My conclusion from our discussion October 2 was that PRE wont go forward with the climate survey designed for LRSD by EFF. Instead, there will be another survey for the monitoring report. If thats true. Id like to inform EFF that LRSD wont follow through with a contract for the survey. Was my conclusion correct? Attached are my notes from the meeting. Thanks very much. Jim Jim Wohlleb, Statistician Planning, Research, & Evaluation Dept Little Rock School District 3001 South Pulaski Little Rock, AR 72206-2873 501/447-3381 (office voice) 501/447-7609 (office fax) 501/680-9244 (mobile) jim.wohlleb@lrsd.org 10/17/2006Page 1 of 1 Dejarnette, Karen From
Sent: To: Cc: Dejarnette, Karen Tuesday, October 17. 2006 2:35 PM Hattabaugh, Hugh
Mittiga, Joseph
Roberts, Olivine Wohlleb, Jim
Williams, Ed
Robinson, Maurecia Subject: superintendent's report PRE will attend the meeting about the Superintendents monitoring report on Thursday afternoon. However I will not be bringing a detailed budget to the meeting. I cannot create such until I have the details of the project. At this time, I am unclear on the number of questionnaires, cover letters, and how they will be administered. Will all questionnaires be administered by hard copy? Or, will any surveys be mailed? If questionnaires will be administered to all parents (26,000), most students (3'^^ -12*^ would be about 20,000), all teachers (2000) and all community partners (200) then the printing part of this project will likely be large enough to go through the bidding process. We are talking about almost 50,000 questionnaires. And, I am assuming you will want cover letters to go with each questionnaires so that means about 100,000 total pages printed. Last year only 12,000 pages (questionnaires and letters) were printed. If you are planning to include open response items on each questionnaire then there will be need to be discussion about who will transcribe the written comments, likely a group of consultants will need to do this. Last year Metros print shop printed the questionnaires and many parents, staff and students complained that the forms were too hard to read, bubbles printed so lightly they could not see which bubble to fill in. The questionnaires may need to be in two colors (not just black and white) so they are more easily readable. For example, bubbles can be printed in light blue for more easy reading and scanning. Also, last year Metro printed many unusable/unscannable questionnaires, their registration on printing was off. These are just some of the points to be discussed before a budget can be detailed. As you know PRE worked with Dr. Bernhardt last year to draft questionnaires. However, Dr. Bernhardts group does not recommend administering any hard copy questionnaires. They do however have an online system that will provide questionnaires to respondents, quantify the responses as they are collected, and provide a report at any time during or after administration. The cost for online administration of parent, student and staff questionnaires to all LRSD is 45,000 total (about 900 per school site). 10/17/2006Page 1 of 123 Margie From: To: Sent: Attach
Subject
"Robinson, Maurecia" <Maurecia.Maicolm@lrsd.org> <mqpowell@odmemaii.com> Monday. October 16, 2006 10:17 PM image002.jpg
image004.png
image006.png
imageOOS.png
imageOIO.png
image012.png
image014.png
imageOIO.png
imageOlS.png
image020.png
image022.png
image024.png
image026.png
image028.png
imageOSO.png
image032.png
image034.png
image036.png
image038.png
image040.png
image042.emz
image043.pcz
image045.wmz
image046.pcz
image047.wmz
image048.pcz
oiedata.mso RI 80_Draft_Final_Report_ Hi margie, Let me know if this works. Maurecia CREP Little Rock School District (.'enter for Research in Edneational Policy Read 180 Evaluation rhe IJnivcTsity of Memphis 525 Browning Hall Memphis, rennes.sce .58152 Foil f ret: l-8h6-670-6147 DRAFT TECHNICAL REPORT 10/17/2006 Page 2 of 123 CREP CenttT for Rescartb in Educational Policy Hie University ofMempliis 325 Browning Hall Memphis, I cnnessee 38152 Toll Free: 1-866-670-6147 Little Rock School District Read 180 Evaluation DRAFT TECHNICAL REPORT October 2006 Clif Mims, Ph. D. Deborah L. Lowther, Ph.D. J. Daniel Strahl, M.S. Center for Research in Educational Policy John Nunnery, Ph.D. Old Dominion University 10/17/2006Page 3 of 123 READ 180 Little Rock School District (LRSD) Draft Executive Summary This report summarizes the evaluation study results of the Little Rock School Districts (LRSD) 2005-2006 READ 180 program. The overall purpose of the evaluation was threefold: 1) to assess the effects of READ 180 on improving and remediating the academic achievement of African-American students, 2) to examine READ 180 implementation processes and practices, and 3) to document the perceptions of students, teachers, principals, and district and school personnel involved with READ 180 regarding strengths, weaknesses, and needed improvements of the program. Research Questions Primary Evaluation Question Has the READ 180 program been effective in improving and remediating the academic achievement of African-American students? Supplemental (Qualitative/Step 2) Evaluation Questions What are the quality and level of implementation of READ 180 at the schools implementing it in 2005-2006? What is the level of participation in READ 180 by African-American students relative to other ethnic groups at the school? What are the perceptions of READ 180 teachers regarding program implementation, impacts, strengths, and weaknesses? What are the perceptions of other teachers in the school regarding program implementation, impacts, strengths, and weaknesses? What are the perceptions of parents/guardians of students participating in READ 180 regarding program impacts, strengths, and weaknesses? Evaluation Design and Measures Participants. LRSD identified 5 middle schools and 5 high schools to participate in the evaluation. Collectively, the evaluation participants included approximately 1000 Read 180 students and 23 Read 180 teachers. Design. The evaluation utilized a mixed-method design. Both quantitative and qualitative data were collected from the participating schools by trained external researchers. The researchers observed classrooms, administered surveys for teachers, students and parents, conducted teacher and student focus groups, and interviewed school principals. Instrumentation. Five measurement strategies were used to collect the evaluation data: direct classroom observations, surveys, focus groups, interviews, and assessment of student academic achievement. Following are descriptions of the assessment instruments. Direct Classroom Observations. Three instruments were used to collect observation data
1) School Observation Measure - used to record the use or nonuse of 24 target strategies
2) READ 180 Quality Assessment - used to document READ 180 implementation practices
3) READ 180 Survey of Computer Use - used to record 10/17/2006Page 4 of 123 student use of READ 180 software. Surveys. Four surveys were administered to the following groups to collect perceptions of the READ 180 program: 1) Read 180 Teachers
2) Non-RAD 180 Teachers
3) READ 180 Students
and 4) READ 180 Parents Focus Groups. READ 180 teacher and student focus groups were conducted to solicit impressions about READ 180. Principal Interview. The interview focused on principal impressions of Read 180 implementation, how it meets the learning needs of African-American students, and how READ 180 could be improved. Student Achievement. ITBS Total Reading NCE from 2005 was used as a student matching variable and pretest covariate in all analyses. ITBS Vocabulary, Comprehension, Total Reading, and Revised Writing NCE scores from 2006 were employed as outcome variables, as were 2006 Literacy Scale Scores and Proficiency Levels from the Arkansas Benchmark examinations. Procedure and Data Sources All data were collected during the spring of 2006 by external researchers. Direct observations were conducted in 17 randomly selected READ 180 classrooms across all 10 schools participating in this evaluation study. These observations each covered a full (90- minute) class period. Seventeen teachers participated in focus groups at seven randomly selected schools, 38 students participated in focus groups at eight randomly selected schools, and all 10 principals were interviewed. Surveys were administered to all Read 180 students, teachers, and parents and yielded the following: students n = 579
teachers n = 269
parents n = 164. Results Direct Observation School Observation Measure. Observation results from 17 Read 180 classrooms, revealed that teachers most frequently used direct instruction, higher level questioning, and acted as a coach, or facilitator. Students were most frequently engaged in reading, writing, or student discussion. Students were observed using computers to complete Read 180 activities in 94.1% of the observations. Overall, the observers reported that the Read 180 classes were always highly focused on learning and that the students were highly engaged all or nearly all of the time (76.5% extensively
23.5% frequently). Read 180 Quality Assessment. There was a low occurrence of teachers utilizing fluency, vocabulary, text comprehension, or writing strategies recommended by Read 180. However, the learning environments were observed to be conducive to cooperative interactions, effective classroom management, and active teacher monitoring, while slightly less were found to be conducive for Read 180 rotation. Only 62% substantially adhered to the recommended 90-minute cycle. Read 180 Survey of Computer Use. The observed classes were comprised of 212 African American and 19 non-African American students. Most classrooms had 8-10 up-to- date computers. Students used Read 180 software in 15 of the 17 classes and primarily worked on reading comprehension, vocabulary, and spelling activities. All African-American students demonstrated a high level of attention, interest and engagement when using the READ 180 software. The non-African American students, present in slightly over half of the classes, demonstrated slightly lower overall levels of attention, interest and engagement. 10/17/2006Page 5 of 123 Surveys READ 180 Teacher Questionnaire. Eighteen of the 23 READ 180 teachers (Caucasian = 61.1%
African-American = 33.3%) completed the questionnaire. Approximately 80% of the teachers indicated they adhered to the 90-minute Read 180 schedule, while all agreed that they routinely used Read 180 data to customize activities to meet student needs. However, almost half indicated that class by ethnicity reports were only used on a monthly basis. All teachers indicated that their computer skills were adequate and most felt they had received enough training to effectively utilize READ 180 resources. Less than half (44.4%) of the teachers indicated their school had formal guidelines for placing students in READ 180. All but one teacher indicated that the program should be continued. Non-Read 180 Teacher Questionnaire. A total of 269 non-Read 180, grades 6-9 teachers (Caucasian = 62.5%
African American = 28.3%) completed the survey. Most agreed that they understood the Read 180 program goals and how the classes are structured. However, 66.9% were not able to identify students who were taking or who had taken READ 180 classes. Of those who were able to identify Read 180 students, about 60% indicated that the students demonstrated improved vocabulary, literacy and comprehension skills. There was less agreement that Read 180 students showed more interest in learning, changed their classroom behavior or submitted work that reflected better writing. Only about half felt the program should be continued. Read 180 Student Questionnaire. A total of 579 READ 180 grade 6-9 students completed the Questionnaire (62.8% of 921 total Read 180 students). Of these, most (88.3%) were African American, nearly half were in the 9**^ grade and 75% were in their first year of READ 180. Nearly three-fourths of the students agreed that their reading skills and slightly more than one-half agreed that their writing skills had improved due to READ 180. Approximately 60% indicated that they learned a lot from computers
teacher directed small groups, reading by themselves, and teacher instruction at the beginning of the class. Nearly a third reported they did not learn from reading with other students. Read 180 Parent Survey. Over three-fourths of 164 parents completing the survey had African-American children. Most parents were aware that their children were participating in the Read 180 program, with 44.9% indicating it had helped them a lot, or some (43.5%). Specifically parents indicated that the program improved their childrens reading grades (68.7%)
interest in learning (67.3%). interest in reading (58.5%), and time spent reading (51.0%). Almost all parents indicated that they felt Read 180 was an important part of their childrens education. Focus Groups Read 180 Teachers. A total of 17 Read 180 teachers from seven randomly selected schools participated in the focus groups. The teachers reported the following as overall strengths of READ 180: students like and are motivated by the program, it supports progress and success, student reading has increased, repetition provides practice and increases comprehension, and rotation of activities. Suggested improvements included
reduce technical difficulties, increase class time, create more user-friendly reports, and reduce class size. All teachers wanted the READ 180 program to be continued. Read 180 Students. A total of 38 students from eight randomly selected schools participated in the focus groups. The students reported the following as overall strengths of READ 180: increased time spent reading and improving reading skills, using the computer 10/17/2006Page 6 of 123 (16%), and working in small groups. In contrast, the students reported that the following were areas in need of improvement: increase READ 180 time, reduce computer and CD problems, and improve book collections. All of the student wanted the program to be continued because it improved their reading and spelling skills, increased their desire to read, and was fun. Principal Interviews The ten principals from the schools participating in this READ 180 program evaluation were interviewed to examine their impressions of the program. The majority of these principals (60%) were administrators of schools that were using READ 180 for the second year. Nearly all of the principals indicated a positive overall impression of the READ 180 program while one reported that the benefits were dependent on the classroom teacher. Three of the principals indicated that their faculty liked READ 180 and two reported that they personally believe it is beneficial. One principal described that the program is wonderful because it restores confidence in the students. Student Achievement Sixth grade. ITBS 2006 subtests. MANCOVA indicated no statistically significant multivariate main effects for program or for school X program interaction. Benchmark Literacy. ANCOVA revealed no statistically significant main effects for program or program X school interaction. The percentages of students obtaining proficiency on the Benchmark Literacy exam were nearly equal between Read 180 and Control groups. Seventh grade. ITBS 2006 subtests. MANCOVA indicated a statistically significant multivariate main effect for program (p = .03), with no school X program interaction effect. Follow-up univariate tests showed that Read 180 students performed significantly lower than the Control groups for Reading Comprehension (p = .001) and Total Reading (p = .006). Benchmark Literacy. ANCOVA revealed no statistically significant main effects for program or program X school interaction. A lower percentage of Read 180 vs. Control students achieved proficiency at Southwest, Henderson, and Cloverdale, whereas a higher percentage achieved proficiency at Mabelvale. Eighth grade. ITBS 2006 subtests. MANCOVA indicated a statistically significant multivariate main effect for program (p = .04), with no school X program interaction effect. Follow-up univariate tests showed that Read 180 students performed significantly lower than the Control groups for Revised Writing (p = .001). Benchmark Literacy. ANCOVA revealed no statistically significant main effects for program, but a significant program X school interaction effect (p = .04) was observed. Follow-up tests revealed a statistically significant positive effect at Mabelvale (ES = +0.38) and a statistically significant negative effect at Henderson (ES = - 0.29). Nearly equal percentages of Read 180 vs. Control students achieved proficiency at Henderson, Cloverdale, and Mabelvale, but lower percentage at Southwest. Ninth grade. ITBS 2006 subtests. MANCOVA indicated a statistically significant multivariate main effect for program (p = .02), with no school X program interaction effect. Follow-up univariate tests showed that Read 180 students performed significantly lower than Control groups on Vocabulary (p = .01), Reading Comprehension (p = .004) and Total Reading (p=.002). Student Achievement Summary. ITBS 2006 subtests. The mean effect size estimates for all four ITBS subtests were statistically significantly less than zero, indicating overall negative effects of Read180. Benchmark Literacy. Overall, Read180 students were less likely to obtain proficiency on the Arkansas Benchmark Literacy examination. Conclusions 10/17/2006Page 7 of 123 Primary Evaluation Question Has the READ 180 program been effective in improving and remediating the academic achievement of African-American students? The preponderance of evidence suggests that the Read 180 program has not been effective in improving or remediating the academic achievement of African American students. Relative to Control students who were individually matched on the basis of prior achievement, sex, race, special education status, and free or reduced-price lunch status. Read 180 students consistently performed lower on both ITBS Reading subtests and the Benchmark Literacy exam. The only exception to the general pattern of Read180 students performing at equal or lower levels to Comparison students was that eighth grade Read180 students at Mabelvale performed significantly higher than their matched Control counterparts, with an effect size of +0.38. The design employed to assess Read 180 effects, while quite rigorous, cannot rule out the possibility of selection effects because students were not randomly assigned to treatment conditions. Supplemental (Qualitative/Step 2) Evaluation Questions What are the quality and level of implementation of READ 180 at the schools implementing it in 2005-2006? Overall, the observers reported that the Read 180 classes were always highly focused on learning and that the students were highly engaged all or nearly all of the time. Although 60% of the teachers reported use of the READ 180 professional modules, teacher infrequent use of targeted literacy strategies indicates that additional teacher professional development focused on implementing these strategies is needed. Also needed is a modified class schedule and increased technical support to ensure students spend the recommended time completing Read 180 computer activities. What is the level of participation in READ 180 by African-American students relative to other ethnic groups at the school? The 2005-2006 Read 180 program was implemented in 13 LRSD schools with student populations comprised of approximately 90% African American students. Of the 231 students observed during direct observation of Read 180 classes, all of the 212 African American students were rated as having a high level of attention, interest, and engagement, while the ratings of the 19 non-African American students were distributed across High, Moderate, and Low. Approximately 90% of the Read 180 teachers agreed that Read 180 was valuable for improving the achievement of African-American students in reading and literacy. In addition, many of the Read 180 students, agreed that the program had increased their reading (70%) and writing (56%) skills. Approximately 75% of the 164 parents who responded to the survey were parents of African American students in Read 180, and nearly all parents felt the program was an important part of their childs education. School principals were also in agreement that Read 180 met the needs of African American students by providing individualized literacy instruction that was highly motivating and used hands-on, practical approaches to assist students with low reading abilities to achieve greater learning. What are the perceptions of READ 180 teachers regarding program 10/17/2006Page 8 of 123 implementation, impacts, strengths, and weaknesses? There was an general consensus among the Read 180 teachers that the program had a positive impact on students by improving students' literacy skills, overall quality of work, achievement and engagement in learning. Key strengths reported were that the program motivated students to learn and the repetition increased comprehension and reading skills. In contrast, the teachers reported that Read 180 needed to increase technical support for computer problems
increase and/or better distribute time
create more user-friendly reports, decrease class size
and establish formal guidelines for student placement into Read 180. Some teachers reported a need for more Read 180 professional development (PD), yet concern was raised as to the quality of the Read 180 PD modules. Teachers agreed that Read 180 was supported and liked by school principals, other teachers, parents, and the students and all but one teacher agreed that the READ 180 program should be continued. What are the perceptions of other teachers in the school regarding program implementation, impacts, strengths, and weaknesses? Of the 269 non-READ 180 teachers that completed a survey, most were aware of Read 180 and understood the program goals and class rotation structure. However, two-thirds indicated that they were not able to identify students who were taking or who had taken READ 180 classes. Therefore, data reflecting non-Read 180 teacher perceptions of the program are limited to 81 teachers. Of these, about 60% indicated that Read 180 students demonstrated improved written, oral vocabulary, and literacy skills, increased reading comprehension, and were more willing to read in class. While, only about half of the non-Read 180 teachers thought that Read 180 students showed more interest in learning, changed their classroom behavior, or submitted work that reflected better writing. Similarly, only half of the r\on-READ 180 teachers felt the program should be continued. What are the perceptions of parents/guardians of students participating in READ 180 regarding program impacts, strengths, and weaknesses? Over 75% of the 164 parents responding to the survey represented African American students enrolled in Read 180 classes. Nearly all of the parents responded that they were aware of and supportive of their childs participation in the Read 180 program and believed that the program was an important part of their child or childrens education. Most of the parents thought Read 180 had helped or somewhat helped improve their childs reading grades
interest in learning and in reading. Slightly fewer parents agreed that the program increased the amount of time that their son or daughter spent reading. Overall, the parents agreed that it is beneficial fortheir children to participate in the Read 180 program because of its positive impact on their reading and overall learning. 10/17/2006Page 9 of 123 READ 180 Little Rock School District (LRSD) DRAFT FINAL REPORT INTRODUCTION This report summarizes the evaluation study results of the Little Rock School Districts (LRSD) 2005-2006 READ 180 program. The overall purpose of the evaluation was threefold
1) to assess the effects of READ 180 on improving and remediating the academic achievement of African-American students, 2) to examine READ 180 implementation processes and practices, and 3) to document the perceptions of students, teachers, principals. and parents involved with READ 180 regarding strengths, weaknesses, and needed improvements of the program. READ 180 \sa reading intervention program that is aimed at assisting low performing adolescent readers. The program provides adaptive instructional software, high-interest literature, and direct instruction in reading, writing, and vocabulary skills. Each READ 180 class is designed for a 90 minute time block that is divided into three primary components. The class begins with 20-minutes of Whole Group Direct Instruction during which the teacher generally lectures and provides instructions for the remaining activities. Next is the 60-minute Small Group Rotations, in which small groups of students rotate through each of three 20- minute modules. The three modules include small group direct instruction from the teacher. modeled and independent reading and computer time using the READ 180 software. The class concludes with the final component, known as the Whole Group Wrap-up. During this final 10 minutes the teacher leads the students in the lessons conclusion. Currently, five middle schools and eight high schools in LRSD use this program. Students are targeted to participate in the program based on results from the Arkansas Benchmark Exam. EVALUATION QUESTIONS This evaluation was structured around one over-arching, primary question concerning 10/17/2006Page 10 of 123 the impact of READ 180 on student achievement, and five supplemental questions that addressed contextual factors related to implementation of the READ 180 program. Primary Evaluation Question 1. Has the READ 180 program been effective in improving and remediating the academic achievement of African-American students? Supplemental (Qualitative/Step 2) Evaluation Questions 1. What are the quality and level of implementation of READ 180 at the schools implementing it in 2005-2006? 2. What is the level of participation in READ 180 by African-American students relative to other ethnic groups at the school? 3. What are the perceptions of READ f 80 teachers regarding program implementation, impacts, strengths, and weaknesses? 4. What are the perceptions of other teachers in the school regarding program implementation, impacts, strengths, and weaknesses? 5. What are the perceptions of parents/guardians of students participating in READ 180 regarding program impacts, strengths, and weaknesses? EVALUATION DESIGN AND MEASURES The following section describes the participants, the student achievement sample. design, instrumentation and procedures utilized for this evaluation. Participants The Little Rock School District has 7 middle schools, 8 high schools and 1 alternative high school. Of these, 5 middle and 8 high schools use the READ 180 program. LRSD identified 5 middle schools and 5 high schools as participants in the 2005-2006 READ 180 evaluation. The ten schools implementing Read 180 collectively served grades 6*^ through 12**^. However the Read 180 program was only implemented in 6**^ through 9**^ grades. Collectively these schools had an enrollment of 9884 students and employed approximately 750 classroom teachers. There were approximately 1000 total Read 180 students in the 10/17/2006Page 11 of 123 program and 23 Read 180 teachers. All Read 180 schools schedule classes in blocks, so they all scheduled Read 180 classes in approximate 90-minute blocks. Student Achievement Sample According to district records, 921 students in ten schools participated in the Read 180 program. Participation by school ranged from a low of n = 18 at Parkview Arts & Science Magnet School, to a high of n = 149 at Cloverdale Magnet Middle School. Read 180 students performed significantly and substantially lower than other students in the same schools on 2005 ITBS Reading normal curve equivalent (NCE) scores, with a mean NCE of 30.48 for program participants versus a mean NCE of 48.71 for non-participants (f = 23.32, df= 4918, p < .001). Given that NCE scores have a standard deviation of 21.06, participants scored 0.87 standard deviation units lower than all non-participants attending the same schools. District enrollment records with basic demographic information were available for 910 of the 921 participants, yielding a match rate of 98.8%. Compared to students attending the same schools. Read 180 participants were more likely to be male (53.6% vs. 47.6%), African American (91.1% vs. 70.3%), free lunch recipients (69.7% vs. 47.5%), and special education students (17.5% vs. 9.0%). Participants were about equally as likely as non-participants to have limited English proficiency (1.6% vs. 1.2%). Design The evaluation, which utilized a mixed-method design, was conducted during the 2005- 2006 academic year. Both quantitative and qualitative data were collected from the participating schools by trained external researchers (e.g., university faculty and staff). The researchers observed classrooms, administered surveys for teachers, students and parents. conducted teacher and student focus groups, and interviewed school principals. Table 1 provides a description of the evaluation instruments and a summary of the participants and data sources, presented with associated research questions. 10/17/2006Page 12 of 123 Table 1. Summary of Instruments, Participants, and Data Sources by Evaluation Question Evaluation Questions Primary Question: Has the READ 180 program been effective in improving and remediating the academic achievement of African-American students? Participants Data Sources All READ 180 students ITBS and Benchmark Exam Supplemental Questions: What are the quality and level of implementation of READ 180 at the . schools implementing it in 2005- , 06? READ 780 teachers READ 180 students Principals at READ 180 schools READ 780 Observations (17 90- minute observations): SOMIREAD 180 Quality Assessment/RHAD 780 SOU School level READ 180 reports READ 780 Teacher Questionnaire READ 180 Student Questionnaire (all READ 180 students) READ 780 Teacher Focus Groups READ 180 Student Focus Groups (random sample) Principal Interviews What is the level of participation in READ 180 by African American students relative to other ethnic groups at the school? All READ 180 schools School level READ 180 reports READ 180 Observations (17 90- minute observations): SOM/READ 780 Quality Assessment/READ 780 SOU What are the perceptions of READ 780 teachers regarding program implementation, impacts, strengths, and weaknesses? READ 180 teachers READ 780 Teacher Questionnaire READ 780 Teacher Focus Groups What are the perceptions of other teachers in the school regarding program implementation, impacts, strengths, and weaknesses? Non-READ 780 Teachers at schools using the program tion-READ 180 Teacher Questionnaire (random selection of teachers) What are the perceptions of parents/guardians of READ 180 students regarding program impacts, strengths, and weaknesses? Parents of READ 180 students READ 180 Parent Survey 10/17/2006Page 13 of 123 Instrumentation Five measurement strategies were used to collect the evaluation data: assessment of student academic achievement, direct classroom observations, surveys/questionnaires, focus groups, and interviews. Following are descriptions of the evaluation instruments. Student Academic Achievement ITBS Total Reading NCE from 2005 was used as a student matching variable and pretest covariate in all analyses. ITBS Vocabulary, Comprehension, Total Reading, and Revised Writing NCE scores from 2006 were employed as outcome variables, as were 2006 Literacy Scale Scores and Proficiency Levels from the Arkansas Benchmark examinations. Observations Observation data were collected with three measures: Read 180 School Observation Measure, Read 180 Quality Assessment, and the Read 180 Survey of Computer Use. READ 180 School Observation Measure. The School Observation Measure (SOM) was developed to determine the extent to which different common and alternative teaching practices are used throughout an entire school (Ross, Smith, & Alberg, 1999). The target strategies include traditional practices (e.g., direct instruction and independent seatwork) and alternative, predominately student-centered methods associated with educational reforms (e.g., cooperative learning, project-based learning, inquiry, discussion, using technology as a learning tool). The strategies were identified through surveys and discussions involving policy makers, researchers, administrators, and teachers, as those most useful in providing indicators of schools instructional philosophies and implementations of commonly used reform designs (Ross, Smith, Alberg, & Lowther, 2001). Using the SOM, the observer examined classroom events and activities descriptively, not judgmentally. Notes were taken relative to the use or nonuse of 24 target strategies. The observer completed a SOM every 10 minutes throughout the class time. At the conclusion of the 90-minute visit, the observer summarized the frequency with which each of the strategies was observed across all 9 of the completed SOMs on a data summary form. The frequency is 10/17/2006Page 14 of 123 recorded via a 5-point rubric that ranges from (0) Not Observed to (4) Extensively. To ensure the reliability of data, observers receive a manual providing definitions of terms, examples and explanations of the target strategies, and a description of procedures for completing the instrument. After receiving the manual and instruction in a group session, each observer participates in sufficient practice exercises to ensure that his/her data are comparable with those of experienced observers. In a 2004 reliability study reported by Sterbinsky, Ross & Burke, pairs of trained observers were within one category for 96% of the whole-school observations and for 91% of the targeted observations. READ 180 Quality Assessment. The READ 180 Quality Assessment (QA) was designed to document the processes and practices used to implement READ 180 in classrooms during the approximately ninety minute observation period. The instrument was used to record the meaningfulness of the following instructional components: fluency. vocabulary, text comprehension, writing and the learning environment. The data were recorded every 10 minutes for the duration of the observation. READ 180 Survey of Computer Use. The READ 180 Survey of Computer Use (SCU) was designed to document the processes and practices used to implement the READ 180 computer program in classrooms. The instrument was used to record the number of students in each READ 180 class by ethnicity (African-American and Non-African-American) and computer configuration data (e.g., number, type, and working condition of the computers). Data were also recorded regarding student use of READ 180 software including: subject area of the activities, teacher/student interactions during READ 180 use, and level of African- American and non-African-American student engagement/interest. The data were recorded during two 10 minute time slots of the Small Group Rotations component of the class. The SCU data was summarized on a data summary form at the end of the entire observation period. Surveys READ 180 Teacher Questionnaire. The READ 780 Teacher Questionnaire is a three- part instrument used to collect teachers perceptions of the READ 180 program. In the first 10/17/2006Page 15 of 123 section, teachers rate their level of agreement with 21 statements regarding six program-related areas: compliance with READ 180 guidelines, impact on instruction, impact on students, readiness to teach READ 180, overall support for READ 180 and technology support. Items are rated with a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Two primary questions are asked in the second section. The first asks teachers to rate the frequency of their use of different READ 180 reports as daily. weekly or monthly. Next, teachers indicate the number of minutes they spend on the following READ 180 activities: whole class direct instruction, small group direct instruction, READ 180 software, modeled/independent work and whole group wrap-up. The third section asks teachers to comment on the strengths and weakness of the program. The final question asks teachers about changes they would recommend for the READ 180 program. The Non-READ 180 Teacher Questionnaire. The Non-RAD 180 Teacher Questionnaire is a two-part instrument designed to ascertain perceptions of the READ 180 program held by teachers at schools that offer the program but are not involved in teaching READ 180 classes. In the first section, teachers rate their level of agreement with 10 statements regarding their own understanding of the program and their perceptions about READ 18Os influence on their students literacy, vocabulary, writing and behavior. In the second section, teachers are asked to comment on the strengths and weakness of the program. Teachers are also asked about changes they would recommend for the READ 180 program. The final question asks teachers if they think the program should be continued. The READ 180 Student Questionnaire. The READ 180 Student Questionnaire is a three-part instrument used to collect students perceptions of the READ 180 program. In the first section, students rate their level of agreement with 10 statements regarding their progress related to reading and writing skills, their interest in learning, and their value and enjoyment of READ 180. Items are rated with a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Two primary questions are asked in the second section. The first asks students to rate the frequency that they work on reading, writing and vocabulary activities in their READ 180 class. Students also rate how much they learn from the following 10/17/2006Page 16 of 123 class activities: teacher instruction at the beginning of class, computer activities, teacher directed small group work, reading by yourself and reading with another student. In the third section students are asked to comment on the strengths and weakness of the program. The final question asks students about changes they would recommend for the READ 180 program. The READ 180 Parent Survey. The READ 180 Parent Survey was designed to ascertain parent awareness and perceptions regarding their childs participation in the program. They were asked to complete one survey per household, but indicate the number and ethnicity of their school-aged child/children. If the parent or guardian was aware of the READ 180 program, they were asked five general questions regarding student attitudes about READ 180 and the value of the program. The final section of the survey consisted of three open-ended items to record parents perceptions of the best and worst aspects of their childs/childrens use of READ 180 and what changes they recommended. Focus Groups Teacher Focus Groups. The Teacher Focus Group Protocol solicited teachers impressions about the following components of the READ 180 program: whole-group direct instruction, computer activities, small-group direct instruction, and independent reading/reading with another student. Focus group participants were asked what was the best part of each of these components and how could each be improved. In closing, the teachers were asked what they believed to be the strongest and weakest aspects of the overall READ 180 program and whether they advised continuing the program. Student Focus Groups. The Student Focus Group Protocol solicited students impressions about the following components of the READ 180 program: whole-group direct instruction, computer activities, small-group direct instruction, and independent reading/reading with another student. Focus group participants were asked what was the best part of each of these components and how could each be improved. In closing, the students were asked what they believed to be the strongest and weakest aspects of the overall READ 180 program and whether they advised continuing the program. 10/17/2006Page 17 of 123 Interviews Principal Interviews. The principal interview was designed for principals of schools that were using READ 180. Interviewees were asked how many years the school been using READ 180. Interview questions examined each principals impressions about the following: how the program is implemented, degree to which the READ 180 program meets the learning needs of African-American students, overall impressions of the program and how his/her schools use of READ 180 could be improved. PROCEDURE The ten data collection measures are summarized in Table 2 by type of measure. instrument, number completed and the data collection procedure. 10/17/2006Page 18 of 123 Table 2. Data Collection Summary Timeline: 2005-2006 Number of Schools = 10 Read 180 schools selected to participate in the study Type of Measure Observations Instrument SOM Number Collected 17 QA 147 SCU 17 Surveys Teacher 18 Focus Groups Interviews Non-Read 180 Teacher Student Read 180 Parent Read 180 Teacher Read 180 Student Principal 269 579 164 7 groups, total number of teachers = 17 8 Focus groups with a total number of students = 38 10 REPORT OF THE FINDINGS Description Prearranged 90 minute sessions in which teachers were observed following the Read 180 rotation protocol. Note forms were completed every 10 minutes of the lesson and summarized on a Data Summary Form. Prearranged 90 minute sessions in which teachers were observed following the Read 180 rotation protocol. Instruments were completed every 10 minutes of the lesson Prearranged 90 minute sessions in which teachers were observed following the Read 180 rotation protocol. Note forms completed every 10 minutes during the 60 minutes of student computer rotation only. The six notes forms from each visit were summarized on a data summary form. Teacher surveys distributed to each of the 23 Read 180 teachers. Eighteen completed surveys were returned to the evaluators. Non-Read 180 Teacher surveys distributed to all non-Read 180 teachers. Distributed to all Read 180 students during class by teacher for completion. Completed surveys were forwarded to the evaluators. Distributed by Read 180 teachers to all students for delivery home. All parents were requested to complete the survey. Completed surveys were returned to the school and forwarded to the evaluators Researchers conducted teacher focus groups with all Read 180 teachers at 7 randomly selected schools. Each focus group interview lasted approximately 30-45 minutes Researchers conducted student focus groups at 8 randomly selected schools. The focus groups at each school consisted of 4 to 5 students randomly selected from students with signed parent consent forms. Each focus group interview lasted approximately 30-45 minutes Researchers individually interviewed each principal. Each interview lasted approximately 60 minutes. 10/17/2006Page 19 of 123 The results of the study are presented below by measurement strategy: observations, surveys, focus groups, interviews, and student achievement. In the Discussion and Conclusions section, the findings are synthesized across instruments to address each research question. Observation Results All the observation results {READ 180 SOM, READ 180 Qualitative Assessment and READ 180 SCU) reflect data collected during observations conducted in 17 randomly selected READ 180 classrooms across all 10 schools participating in this evaluation study. READ 180 School Observation Measure. In observations of 17 Read 180 classrooms, observers found that the main instructional orientation was towards direct instruction, or lecture. Nearly two-thirds of the observations found this kind of teaching to be occurring (41.2% frequently: 23.5% extensively). The most common type of instructional strategy being used, among those considered. was the use of higher level questioning. This was used occasionally in 47.1% of the cases, and frequently in 23.5% of the cases. Another often-used strategy was for the teacher to act as a coach, or facilitator, which happened occasionally 41.2% of the time and frequently 11.8% of the time. One strategy that was used by some, and not by others, was to give higher-level instructional feedback to enhance learning. Over 40% (41.2%) were found to never use this strategy and 29.4% to rarely use it
but 29.5% were found to use it at least occasionally. The observers also recorded the frequency that students engaged in particular activities. Of those reported, the most frequent student activity was sustained reading, which was observed to happen frequently or extensively in 53% of the cases and occasionally in 41.2% of the cases. Two other popular strategies were to engage the students in sustained writing (17.7% frequently and 35.3% occasionally) or student discussion (23.5% frequently and 17.7% occasionally). Rarely was independent seatwork, such as self-paced worksheets or individual assignments, used (35.3% rarely, 41.2% never). In areas of technology use, computers were used for instructional delivery of the Read 10/17/2006Page 20 of 123 180 software in 94.1% of the observations. Nearly 30% (29.4%) were found to use technology in this way frequently or extensively, while 64.7% used the Read 180 software occasionally. The observers did not observe any of the classes using technology as a learning tool (e.g., use of word processing, spreadsheets, or conducting Internet searches). In areas of assessment, the observers rarely found examples of student selfassessment (5.9%) and no cases of performance assessment strategies. Overall, the observers reported that the Read 180 classes were always highly focused on learning and that the students were highly engaged all or nearly all of the time (76.5% extensively
23.5% frequently). The results from these observations are reported in Table 3. 10/17/2006Page 21 of 123 Table 3. READ 180 School Observation Measure (SOM) Results N = 17 READ 180 Classrooms The extent to which each of the following was observed in the classroom. None JOE _______Percent Observed__________________ Rarely Occasionally Frequently Extensively Hi (2) Hi Mean SD Instructional Orientation Direct instruction (lecture) Team teaching Cooperative/collaborative learning Individual tutoring 0.0 82.4 100.0 76.5 17.7 5.9 0.0 17.7 17.7 0.0 0.0 0.0 41.2 5.9 0.0 5.9 23.5 5.9 0.0 0.0 2.7 0.5 0.0 0.4 1.1 1.2 0.0 0.8 Classroom Organization Ability groups Multi-age grouping Work centers (for individuals or groups) 100.0 100.0 5.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 23.5 0.0 0.0 70.6 0.0 0.0 3.5 0.0 0.0 1.0 Instructional Strategies Higher level instructional feedback (written or verbal) to enhance student learning Integration of subject areas Project-based learning Use of higher-level questioning strategies Teacher acting as a coach/facilitator Parent/community involvement in learning activities 41.2 29.4 11.8 17.7 0.0 1.1 1.1 94.1 94.1 11.8 17.7 100.0 0.0 0.0 17.7 29.4 0.0 0.0 0.0 47.1 41.2 0.0 0.0 5.9 23.5 11.8 0.0 5.9 0.0 0.0 0.0 0.0 0.2 0.2 1.8 1.4 0.0 1.0 0.7 1.0 1.0 0.0 Student Activities Independent seatwork (self-paced worksheets, individual assignments) Experiential, hands-on learning Systematic individual instruction Sustained writing/composition (self-selected or teachergenerated topics) Sustained reading Independent inquiry/research on the part of students Student discussion 41.2 94.1 100.0 35.3 5.9 0.0 11.8 0.0 0.0 11.8 0.0 0.0 0.0 0.0 0.0 0.9 0.1 0.0 1.0 0.2 0.0 Technology Use Computer for instructional delivery Technology as a learning tool or resource Assessment Performance assessment strategies Student self-assessment (portfolios, individual record books) Summary Items High academically focused class time High level of student attention, interest, engagement 29.4 5.9 88.2 29.4 17.7 0.0 5.9 29.4 35.3 41.2 0.0 17.7 17.7 41.2 5.9 23.5 0.0 11.8 0.0 0.0 1.4 2.5 0.2 1.4 1.1 1.0 0.8 1.2 5.9 100 0.0 0.0 64.7 0.0 23.5 0.0 5.9 0.0 2.2 0.0 0.8 0.0 100.0 94.1 0.0 0.0 Note. Item percentages may not total 100% because of missing data 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.0 0.0 0.0 0.1 0.2 0.0 0.0 0.0 0.0 0.0 23.5 100.0 76.5 4.0 3.8 0.0 0.4 10/17/2006Page 22 of 123 READ 180 Quality Assessment. The Quality Assessment instrument was used to record the occurrence of processes and practices used to implement Read 180. The observations were recorded every 10 minutes, typically resulting in nine observations in a 90-minute class period. The observations were made in five different categories: fluency, vocabulary, text comprehension, writing, and learning environment. Overall, as seen in Table 4, there was a low occurrence of teachers utilizing fluency. vocabulary, text comprehension, or writing strategies recommended by Read 180. Specifically, observers reported about 20% frequent or extensive evidence of student work on reading fluency (18.4% for modeling fluent oral reading
19.1% for students reading orally). In vocabulary, observers reported frequent or extensive evidence of introducing or reviewing key vocabulary words only 13.6% of the time and explicit vocabulary instruction only 4.8% of the time. In text comprehension, there were only two strategies that were observed frequently or extensively in approximately 20% of the observations: higher order questioning (20.4%) and interactive discussion (18.4%). Techniques for writing were the least observed strategy when compared to techniques for fluency, vocabulary, or for text comprehension. On a positive note, the learning environments observed in the Read 180 classrooms were frequently to extensively observed to be conducive to cooperative interactions (98.6%), have students actively engaged (98.0%), have effective classroom management (93.2%), and to have teachers actively monitoring (98.6%). Slightly less classes were found to be set up in a manner conducive for Read 180 rotation (78.2%) and only 62% substantially adhered to the recommended 90-minute cycle. 10/17/2006Page 23 of 123 Table 4. READ 180 Quality Assessment Results N= 156 The extent to which each of the following was observed in the classroom. None ((0) Rarely or Frequently or Occasionally Extensively 112). (3,4) Mean Standard Deviation Fluency Models fluent oral reading Has students read/re-read orally 80.3 78.2 1.4 2.7 18.4 19.1 0.4 0.4 0.8 0.8 Vocabulary Introduces or reviews key vocabulary words Explicit vocabulary instruction 83.7 94.6 2.7 0.7 13.6 4.8 0.3 0.1 0.7 0.4 Text Comprehension Explicit comprehension strategy instruction Makes connection to prior knowledge Ask students for predictions Uses higher order questioning Guides visual imaging Guides interactive discussion 89.1 86.4 92.5 72.8 91.8 81.6 7.5 2.0 4.1 6.8 1.4 0.0 3.4 11.6 3.4 20.4 6.8 18.4 0.1 0.3 0.1 0.5 0.2 0.4 0.4 0.7 0.4 0.8 0.5 0.8 Writing Instructs letter formation, handw/riting Explains the writing process Conducts language mechanics lesson 100.0 90.5 91.8 0.0 3.4 4.1 0.0 6.1 4.1 0.0 0.2 0.1 0.0 0.5 0.4 Learning Environment Conducive to cooperative interactions Students are actively engaged Effective classroom management Teacher actively monitors The room is set up conducive to the Read 180 rotation The teacher substantially adheres to the 90 min. cycle Read 180 supporting material are available for student use 0.7 0.0 0.0 0.0 0.0 6.8 0.0 0.7 2.0 6.8 1.4 21.8 31.3 100.0 98.6 98.0 93.2 98.6 78.2 62.0 0.0 2.0 2.0 1.9 2.0 1.8 1.6 2.0 0.2 0.1 0.3 0.1 0.4 0.6 0.0 READ 180 Survey of Computer Use. A total of 17 6**^-9** grade READ 180 classes were observed. These classes were comprised of 212 African American students and 19 non-African American students. The majority (88.2%) of these classrooms had 8-10 computers and all the computers in the classes observed were up-to-date. Students were observed using the Read 180 software in 15 (88.2%) of the 17 classes. The students were observed rarely asking questions related to the use of READ 180 software or the computer equipment (Table 5). There was a high level of academic focus during the Small Group Rotations component of the READ 180 classes, when the students were working on the computer modules. The 10/17/2006Page 24 of 123 students were observed predominately spending computer time working on Read 180 reading comprehension (82.4% of time), vocabulary (76.5%) and spelling (70.6%) activities. They were not observed asking any content related questions while using the software. All African-American students demonstrated a high overall level of attention, interest and engagement when observed using the READ 180 computer program. The Non-African American students, present in slightly over half (52.9%) of the classes, demonstrated lower overall levels of attention, interest and engagement. While data were colleted about the types of instruction teachers provided specific to students use of READ 180 computer program, analysis indicates that in over half (52.9%) of the classes, students received no instructions from the teacher. When instruction was provided, the most frequently observed type was related to using the computer, which was observed rarely to occasionally in nearly 30 percent of the classes (29.4%). Other types of instruction seen were related to classroom rules (17.7%), Read 180 software (11.8%), and only 5.9% related to the Read 180 subject-area content. 10/17/2006Page 25 of 123 Table 5. READ 180 Survey of Computer Use Results N= 17 READ 780 Classrooms School Cloverdale Middle Cloverdale Middle Central High Central High Hall High Hall High Henderson Middle Henderson Middle J.A. Fair High Mabelvale Middle McClellan High McClellan High Parkview High Pulaski Middle Pulaski Middle Southwest Middle Southwest Middle TOTAL Grade Observed 6 7 9 9 9 9 7 8 9 8 9 9 9 6 6 6 8 NA Number of students by ethnicity African American 14 14 11 12 15 19 5 13 13 13 13 13 12 14 14 10 7 212 Non-African American Number of classes observed Computer Configuration and Use How many computers were available for Read 180? Only one 2-4 5-7 8-10 11 or more % Observed 0.0 0.0 11.8 88.2 0.0 Never How frequently did malfunctions occur on computers used for Read 180? 76.5 Computers used for Read 180 most frequently had: No Headphones 5.9 Mostly Non-functional Headphones and Microphones used for Read 180 were
11.8 0 0 1 1 3 2 1 0 2 1 0 0 3 0 0 2 3 19 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 17 Most computers used for Read 180 were: Up to date Aging but adequate Outdated/Limited Capacity % Observed 100.0 0.0 0.0 Rarely 11.8 Occasionally Frequently Extensively 5.9 0.0 0.0 Headphones with no microphones 5.9 Headphones and Microphones 88.2 Displayed signs of disrepair 0.0 All in good working order 82.4 Read 180 Computer Activities In which subject areas did students complete Read 180 computer work? Reading comprehension Vocabulary Spelling % of time 82.4 76.5 70.6 10/17/2006Page 26 of 123 Table 5. Continued Items________________________________________________ What was the level of academically focused time while students were using the computer for Read 180? What was the overall level of African-American student attention, interest, and engagement while using the Read 180 computer program? Not Applicable 17.6* 11.8* Low 0.0 0.0 Moderate 5.9 0.0 High 76.5 88.2 What was the overall level of NON African-American students attention, interest, and engagement while using the Read 180 computer program? 52.9 11.8 5.9 29.4 Students did not use computers
"Class did not have any NON-African American students Types of Questions Students Asked While Using the computer Read 180 %Not Observed % Rarely % Occasionally % Frequently % Extensively Content area (e.g. how to solve a problem, the meaning of a word). Software use (e.g. how to log in
how to move to the next section
how to take a test) Computer use (e.g. how to get the mouse or keyboard to work properly) Non-Read 180 questions (e.g. Do I have to sit next to John? Can I go to the restroom?) The teacher provided the following types of instruction specifically for student use of Read 180 computer activities: Content area (e.g. reading, vocabulary) Software use (e.g. how to log in
find correct lesson) Computer use (e.g. locate software, use mouse) Classroom behavior njies No instructions were given 100.0 88.2 70.6 94.1 % Not Observed 94.1 88.2 70.6 82.4 52.9 0.0 0.0 0.0 0.0 11.8 29.4 5.9 % Rarely 0.0 11.8 23.5 11.8 11.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 % Occasionally 5.9 % Frequently % Extensively 0.0 0.0 0.0 5.9 5.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 35.3 Read 180 Time-on-Task. As seen in Table 6, the Read 180 teachers whose classes were observed spent close to the recommended amount of time for each of the five Read 180 activities. However, when examining a district provided sample of Read 180 computer program time-on-task reports from six of the 10 schools, the results revealed that an average Read 180 computer session lasted from 9.3 to 13.6 minutes per student (Table 7). Students completed an average of 0.2 to 2.0 sessions per week for up to 31 weeks. Thus, the mean overall time that students spent 10/17/2006Page 27 of 123 working on Read 180 computer activities during the 2005-2006 academic year ranged from 2.2 hours to 16.1 hours, with the average being 10.2 hours per student. Read 180 as the name implies, recommends that students spend 20 minutes per day throughout the academic year. When computing this number with the required 180 days of school attendance, the total time equals 60 hours. The LRSD Read 180 students worked less than 20% of the recommended time completing the instructional activities presented by Read 180 software. Table 6. Observed vs. Recommended time per Read 180 Activity Whole Class Instruction Computer Small Group Independent Reading Whole Total Group Wrap Time Average Time Observed* Read 180 Recommended Time 19.4 19.4 19.4 18.2 8.2 84.7 20 20 20 20 10 90 W = 17 observations at 10 Read 180 schools Table 7. Read 180 Computer Program Report of Student Time-on-Task N = 6of the 10 Read 180 schools participating in the study Schools HaifH^ McClellan HS Parkview HS Pulaski Heights MS Fair HS Southwest Alt. Center Average Weeks of Time 31.0 24.6 30.3 21.9 28.0 20.3 26.0 Mean Sessions Per Week 2.0 1.6 1.4 Mean Session Time 12.3 10.1 12.5 Mean Total Sessions 78.7 71.9 57.2 Mean Total Time (Min) 968 728 717 Mean Total Time (Hrs) 16.1 12.1 12.0 1.4 1.2 0.2 1.3 12.6 9.3 13.6 11.8 54.4 46.9 9.7 53.1 688 437 132 612 11.5 7.3 2.2 10.2 Survey Results READ 180 Teacher Questionnaire. The READ 780 Teacher Questionnaire was completed by 18 of the 23 READ 180 teachers, representing a 78.2% return rate. The respondents were primarily Caucasian (61.1%), and secondarily African-American (33.3%), with most being female (94.4%). Nearly two-fifths (38.9%) of the respondents were first year READ 180 teachers while 22.2% were 10/17/2006Page 28 of 123 teaching their second year with the program and 38.9% were in their third. The respondents represented ail READ 180 grade levels, with sixth and ninth grade teachers (33.3% each) more heavily represented than seventh and eight grade teachers (22.2% each). As shown in Table 8, the first 21 items assessing teachers perceptions of the READ 180 program were indicative of a relatively high level of teacher approval for the program (as measured by a combination of the categories Strongly Agree and Agree. Almost 90% (88.9%) of respondents strongly agreed or agreed that the program had a positive impact on students, improving their overall quality of work, achievement and engagement in learning. All teachers were in agreement that they routinely customized READ 180 activities to meet the instructional needs of students, while over four fifths (83.3%) reported that they modified READ 180 on the basis of report feedback. Only whole class and individual reports were used on a daily basis, but the majority of teachers indicated they used these reports at least weekly. Almost half (44.4%) of respondents indicated they used the class by ethnicity and class by gender reports on a monthly basis and never on a daily basis. Over four-fifths (83.3%) of teachers indicated that they adhered to the full 90 minute implementation schedule. This time was fairly evenly split among the required learning activities. Most teachers (94.4%) used 20 minutes for small group discussion and 20 minutes for READ 180 software. Additionally, 83.4% of teachers spent between 15-20 minutes on whole class or group discussion. Whole group wrap up was the activity teachers reported spending the least amount of time on, although 27.8% of teachers did report spending 15-20 minutes on wrap up. All teachers indicated that their computer skills were adequate to effectively utilize READ 180 resources. While nearly four-fifths of teachers felt they had received enough training, 16.7% indicated that they could use more training in order to address students learning needs. Interestingly only 61.1% used the READ 180 professional modules to enhance their own effectiveness as READ 180 teachers. This may be related to the 66.7% agreement that the READ 180 resources enabled teachers to effectively implement the program according to 10/17/2006Page 29 of 123 recommended guidelines. In addition less than half the respondents strongly agreed that their school has a well-developed plan to guide the READ 180 program. Nearly two-fifths (38.9%) of respondents were neutral as to the schools well-developed plans and 16.7% disagreed that there were well-developed plans at all. In addition, less than half (44.4%) of the respondents indicated their school had formal guidelines in place for determining which students should participate in the READ 180 program. The same percentage (44.4%) actually disagreed or strongly disagreed that their school used the Scholastic Reading Inventory to place READ 180 students. In fact, two thirds (66.6%) reported that the schools administration did not routinely use READ 180 reports to monitor and adjust implementation practices. However, these shortcomings in formal guidelines do not indicate lack of support. In fact. 83.3% of respondents agreed or strongly agreed that the school administration fully supports READ 180, with no disagreements. Similarly, two-thirds (66.7%) reported that their colleagues were generally supportive of the READ 180 program, with almost the same number (61.1%) reporting support from parents. All but one teacher indicated that the program should be continued. When asked to describe Read 180 program strengths, teachers listed individualized and small group instruction, the teaching materials, and that the program was motivating, assisted students in achieving rapid progress, and that they received positive feedback about the program (Appendix B). The primary weaknesses noted were technology problems, not enough time, limited selection of books, inflexible structure, and lack of organization in the Read 180 materials. The teachers offered the following recommendations for improving the program: better student screening, correct technical problems, train new teachers, include higher-level questions on the handouts, and provide materials that are more engaging and books that are more interesting for the students. 10/17/2006Page 30 of 123 Table 8. READ 180 Teacher Questionnaire Results N= 18 Read 180 Teacher Questionnaire items % Strongly Agree and Agree (4,5) % Neutral (3) % Strongly Disagree and Disagree ____tL21____ Mean Standard Deviation mpact on Students 'he use of the Read 180 has increased the level of student attention, interest and engagement in learning. tead 180 has had a positive impact on student sarning and achievement. Overall, the Read 180 program seems valuable for Tiproving the achievement of African-American tudents in reading and literacy. 'he use of Read 180 has improved the quality of itudent work. mpact on Instruction frequently use the Read 180 professional levelopment modules to enhance my effectiveness as I Read 180 teacher. routinely customize Read 180 activities to meet the nstructional needs of students. routinely modify my Read 180 instructional practices lased on Read 180 report feedback. he design of the Read 180 resources enables me to jffectively implement the Read 180 program according 0 recommended guidelines. Compliance with READ 180 Guidelines adhere to the Read 180 recommended 90 minute Tiplementation schedule. Jy school has formal guidelines for placing students in he Read 180 program. 4y school has formal guidelines for determining when I student no longer needs the Read 180 program. school uses the Scholastic Reading Inventory (SRI) 0 place students in the Read 180 program. teadiness to Teach READ 180 have received enough training to address student sarning needs through the use of Read 180 resources. computer skills are adequate to effectively utilize tead 180 resources. 88.9 88.9 88.9 88.9 61.1 100.0 83.3 66.7 83.3 44.4 22.2 38.9 77.8 100.0 11.1 5.6 5.6 0.0 22.2 0.0 11.1 16.7 5.6 22.2 27.8 16.7 5.6 0.0 0.0 5.6 5.6 5.6 16.7 0.0 5.6 16.7 5.6 33.3 50.0 44.4 16.7 0.0 4.2 0.6 4.3 4.2 4.2 3.7 4.6 4.1 3.8 4.2 3.1 2.6 2.9 3.9 4.7 0.8 1.0 0.8 1.2 0.5 0.8 1.4 0.8 1.2 1.3 1.3 1.1 0.5 10/17/2006Table 8. Continued iead 180 Teacher Questionnaire Items % Strongly Agree and Agree (4,5) % Neutral (3) % Strongly Disagree and Disagree ____(12)____ Mean Page 31 of Standard Deviation 123 echnology Support can readily obtain answers to questions about Read 80. ?1ost of our school computers that are used for Read 80 are kept in good working condition. }verall Support for READ 180 Ay school's administration fully supports the Read 180 irogram. 3ur school has a well developed plan that guides the tead 180 program. Ay school's administration routinely uses the Read 180 eports to monitor and adjust program implementation iractices. "eachers in this school are generally supportive of the lead 180 program. arents and community members support our school's ise of Read 180. 83.3 83.3 83.3 44.4 27.8 66.7 61.1 16.7 16.7 11.1 38.9 33.3 33.3 38.9 0.0 0.0 0.0 16.7 33.3 0.0 0.0 4.2 0.7 4.1 0.7 4.4 0.7 3.5 2.9 3.9 3.8 1.0 1.2 0.8 0.7 How routinely do you use the following Read 180 performance report formats? Whole class Class by ethnicity Class by gender Individual student % Daily 16.7 0.0 0.0 11.1 % Weekly 61.1 11.1 5.6 72.2 % Monthly 22.2 44.4 44.4 16.7 Ini fol wt Sit Re Me Wt Teachers who feel the Read 180 program should be continued. Yes No % 88.9 5.6 Respondents teaching at each grade level 6*^ Grade 7*^ Grade 8*^ Grade 9*^ Grade % 33.3 22.2 22.2 33.3 Ethnicity Caucasian African-American Hispanic Asian Multi-Ethnic % 61.1 33.3 0.0 0.0 0.0 Gender Male Female % 0.0 94.4 low long have you taught a Read 180 class? I year i years ! years 38.9% 22.2% 38.9% Non-READ 180 Teacher Questionnaire. The Non-/?AD 180 Teacher Questionnaire is a two-part instrument designed to ascertain perceptions of the READ 180 program held by teachers at schools that offer the program but are not involved in teaching READ 180 classes. As seen in Table 9, of the 269 10/17/2006Page 32 of 123 respondents, 62.5% were Caucasian, 28.3% were African American, with less than 3% representing other races. Most respondents (44.4%) were 9*^ grade teachers, with the remaining grades being represented fairly evenly at around 20%. Nearly three-fourths (73.6%) were female. Reports from colleagues of READ 180 teachers revealed that most (86.4%) agreed or strongly agreed that they understood the Read 180 program goals, and nearly as many (79%) indicated they understood how READ 180 classes are structured. However, two-thirds (66.9%) of the teachers indicated that they were not able to identify students who were taking or who had taken READ 180 classes. The non-Read 180 teachers (30.1%) who were able to identify Read 180 students were generally positive with regard to the impact of READ 180 on students. Specifically, 63.0% indicated that Read 180 students demonstrated improved written and oral vocabulary skills. continuous improvement in literacy skills (61.7%), increased comprehension of assigned reading (60.5%), and were more willing to read in class (59.3%). However, there was less agreement among the non-Read 180 teachers that Read 180 students showed more interest in learning (55.6%), changed their classroom behavior (51.9%) or submitted work that reflected better writing (50.6%). Only about half (49.8%) of these r\on-READ 180 teachers felt the program should be continued. Responses of teachers who could identify the Read 180 students responded to open- ended comments are located in Appendix C. When asked to describe strengths of Read 180, the most common responses were that the program improved student reading skills, grades. and tests. Also mentioned was the individualized approach to instruction and use of a variety of instructional delivery methods. The most frequent response to program weakness and areas of needed improvement was that access to the program was limited - that it is not reaching all the students with low reading ability. The teachers also indicated that there were not enough books or computers to adequately support implementation of Read 180. There were 99 of the 269 non-Read 780 teachers who supported continuation of the program because it improved student reading skills and performance and provided good teacher and 10/17/2006student feedback. Page 33 of 123 10/17/2006Page 34 of 123 Table 9. Non-/?AD 180 Teacher Questionnaire Results A/=269 ^on-Read 180 Teacher Questionnaire Items have an understanding of the Read 180 program goals. % Strongly Agree and Agree (4,5) 86.4 % % Strongly Disagree and Neutral Disagree 13L 7.4 1121 4.9 Mean 4.2 Standard Deviation 0.8 have an understanding of the Read 180 program classroom Tiplementation (how the classes are structured). 79.0 12.3 7.4 4.0 0.9 'he students in my class who are taking or have taken Read 180 classes lemonstrate improved written and oral vocabulary skills. 63.0 30.9 3.7 3.8 0.9 'he students in my class who are taking or have taken Read 180 classes lemonstrate continuous improvement in literacy skills. 61.7 30.9 4.9 3.8 0.9 'he students in my class who are taking or have taken Read 180 classes ihow increased comprehension of assigned reading. 60.5 34.6 2.5 3.8 0.8 'he students in my class who are taking or have taken Read 180 classes
how more willingness to read aloud in class. 59.3 30.9 6.2 3.7 0.9 'he students in my class who are taking or have taken Read 180 classes :how increased attention and interest in learning. 55.6 35.8 6.2 3.7 0.9 'he students in my class who are taking or have taken Read 180 classes lave improved classroom behavior. 51.9 37.0 8.6 3.6 0.9 'he students in my class who are taking or have taken Read 180 classes iubmit work that reflects improved writing. 50.6 40.7 3.7 3.7 0.9 Teachers able to identify students who are taking or have taken Read 180 classes. Yes No % 30.1 66.9 Teachers who feel the Read 180 program should be continued. Yes No % 49.8 0.7 Respondents teaching at each grade level 6*^ Grade 7*** Grade % 19.3 18.6 8*^ Grade 9*^ Grade 16.4 44.6 Ethnicity Caucasian African- American Hispanic Asian Multi-Ethnic % 62.5 28.3 0.7 0.4 1.9 Note: Item percentages may not total 100% because of missing input from some respondents. Gender Male Female % 24.5 73.6 10/17/2006Page 35 of 123 READ 180 Student Questionnaire. There was a total of 579 students involved in READ 180 classes that completed the READ 180 Student Questionnaire (Table 10). This number represents 62.8% of the total 921 Read 180 students. Of these, most (88.3%) were African American and nearly half (45.9%) were in the 9*^ grade, all other grades being represented fairly equally. This sample is representative of the whole population as 45.2% of READ 180 participants were in the 9*^ grade and all other grades are also represented fairly equally. Additionally, nearly half of the respondents were female (44.6%) and slightly over half (52.5%) were male. Three-fourths of the students were in their first year of READ 180, while most of the remaining students (23.7%) were in their second year of the program. READ 180 was implemented in 6**^ through 9^*^ grades. While the overall means of responses on this 5-point Likert-type survey were generally lower than scores reported by teachers, overall student response to the READ 180 program tended to be positive. Nearly three-fourths (74.1%) of the students agreed that their reading skills and slightly more than one-half (55.4%) agreed that their writing skills had improved due to the READ 180 program. Additionally, 48.4% of the students also felt as if their overall schoolwork had improved due to READ 180. Student attitudes were generally more positive than perceptions of increased ability. Approximately 70% (69.6%) of students agreed that their READ 180 classes were well- organized, while 58.9% looked forward to their READ 180 classes and 57.7% agreed that they were more interested in learning in general due to READ 180. Interestingly this is slightly higher than the same perception of increased interest by non-READ 180 teachers. Only about half (52.0%) of the students agreed that READ 180 was the best reading class that they had ever taken or that they learned more than in their other classes (47.2%). Even less, or nearly a third of students (31.8%) reported not wanting to repeat the program. READ 180 activities were broken down into reading, writing and vocabulary and students were asked to indicate how much they engaged in each of these (not at all, a little, or a lot) during their 90-minute Read 180 classes. Nearly 70% of the students indicated that they 10/17/2006Page 36 of 123 worked on reading (69.8%) and writing (69.6%) a lot, while one fourth (25.6%) reported that they only worked on these areas a little (reading = 25.6%
writing = 24.4%). The Read 180 students reported working on vocabulary to a lesser degree, as a lot was reported by 57.0% and a little by 32.5%. Students were asked to respond to closed- and open-ended items to indicate how much they learned from Read 180 activities (see Appendix D). The closed-ended items revealed that over 60% (63.7%) learned a lot from computers. Nearly as many indicated that they learned a lot from teacher directed small groups (59.4%), reading by themselves (58.7%), and teacher instruction at the beginning of the class (57.7%). Nearly a third (30.6%) reported they did not learn from reading with another student. The open-ended responses revealed spelling, pronunciation, reading and Reading Zone as most frequently cited activities that helped students to learn the most. Whereas, Reading was also cited along with Vocabulary Zones as activities that do not help students learn. When asked what would make the Read 180 better, most students wanted the program to last longer, more activities, videos, and centers, and more time on the computer. 10/17/2006Page 37 of 123 Table 10. READ 180 Student Questionnaire Results N = 57Q tead 180 Student Questionnaire Items______ /y reading skills have improved because of Read 180. ^y Read 180 class is well organized. lead 180 has made me want to get better grades. look forward to my Read 180 class. lead 180 has made me more interested in learning. 4y writing has improved because of Read 180. lead 180 is the best reading class I have ever taken, would like to be in the Read 180 class again next year, ity schoolwork is better because of what I have learned 1 Read 180. learn more in the Read 180 class than in my other lasses. % Strongly Agree and Agree (4,5) 74.1 69.6 62.5 58.9 57.7 55.4 52.0 50.4 48.4 47.2 In your Read 180 class, how often do you work on the following activities: Reading? Writing? Vocabulary? In your Read 180 class, how much do you learn from the following activities: Computer activities? Teacher directed small group work? Reading by yourself? Teacher instruction at the beginning of class? Reading with another student? % A lot (3) 69.8 69.6 57.0 % A lot (3) 63.7 59.4 58.7 57.7 28.3 Respondents at each grade level 6*'^ Grade 7** Grade 8*^ Grade 9* Grade % 20.0 19.5 14.5 45.9 % Neutral (3) 16.4 16.4 23.0 23.8 25.2 20.4 24.4 16.9 28.5 24.2 % A little (2) 25.6 24.4 32.5 % A little (2) 24.7 30.2 31.1 32.1 36.8 Ethnicity Caucasian African-American Hispanic Asian Multi-Ethnic % Strongly Disagree and Disagree (1,2) 9.2 13.1 13.5 16.8 16.2 23.5 23.0 31.8 22.1 Mean 3.9 3.9 3.8 3.6 3.6 3.5 3.5 3.3 3.4 St Dev 1.0 1.2 1.1 1.2 1.1 1.3 1.3 1.5 1.2 27.8 3.3 1.3 % Not at all (1) 2.8 3.5 7.1 % Not at all (1) 9.0 6.9 6.2 7.3 30.6 Mean 1.7 1.7 1.5 St Dev 0.5 0.5 0.6 1.6 1.5 1.6 1.5 1.0 0.7 0.6 0.6 0.6 0.8 % 2.9 88.3 2.8 0.2 4.3 Gender Male Female % 52.5 44.6 How long have you been in a Read 180 class? 1 year 2 years 3 years 74.3 23.7 0.7 READ 180 Parent Survey. The Read 180 Parent Survey was completed by 164 parents (Table 11). Over three- fourths of these parents had African-American children (75.6%), while 7.3% had Caucasian children, and the remainder represented Hispanic, Asian, or multi-ethnic backgrounds. Nearly 40% of the children (39.6%) whose parents completed this survey were in ninth grade while 26.8% were in sixth grade, 20.7% were in eighth grade, and 11% were in seventh grade. The vast majority of the parents (89.6%) were aware that their children were 10/17/2006Page 38 of 123 participating in the Read 180 program, with over half (53.7%) having learned this from their children. Other parents learned about the Read 180 program from teachers (28.6%) or from the schools (16.3%). Nearly all of the parents felt that the Read 180 program had improved their sons/daughters reading skills, with 44.9% indicating it had helped them a lot, and 43.5% indicating it has helped them some. None of the parents felt the program had not helped their children at all, but 8.8% were still not sure. To understand how these parents felt the Read 180 program helped their children, the parents were asked to what level they agreed with statements regarding the programs impact on their childrens reading ability. The most positive response was that the program improved their childrens achievement or reading grades (68.7% indicated yes
23.1% indicated somewhat). Parents also felt that the Read 180 program improved their childrens interest in learning (67.3% indicated yes
25.2% indicated somewhat) and their interest in reading (58.5% indicated yes
26.5 indicated somewhat). Fifty-one percent of parents agreed that the program improved the time their children spent reading, while 34.7% somewhat felt that it did. Similar results are seen in the parents responses to the open-ended item asking them to describe the best thing about their son/daughter being in a Read 180 class (Appendix E). The most frequent comments were that the program improved their childs reading skills and interest in reading. The parents indicated that the worst aspects of the program were that it did not encourage students to read at home, computer time was too limited, the activities were too challenging, there were too many computer problems, and it did not seem that students were learning anything new and did not have Read 180 homework. To summarize their support for the program, 89.1% of parents indicated that they felt Read 180 was an important part of their childrens education while 4.8% somewhat felt this way and less than one percent (0.7%) did not. Table 11. READ 180 Parent Survey Results N= 164 Do you think Read 180 has increased your sons/daughters: Interest in reading. Yes 58.5 Somewhat 26.5 No 6.1 Not sure 6.1 Mean 1.6 Std. Dev. 0.9 10/17/2006Page 39 of 123 Interest in learning Achievement or grades in reading. Time spent reading. 67.3 68.7 51.7 25.2 23.1 34.7 1.4 2.0 8.8 2.7 2.7 2.7 1.4 1.4 1.6 0.7 0.7 0.8 Do you think Read 180 is an Important part of your son/daughters education? Yes 89.1 Somewhat 4.8 No 0.7 Not sure 3.4 Mean 1.2 Std. Dev. 0.6 To what degree has the Read 180 improved your son's/daughter's reading skills? A lot 44.9 Some 43.5 Not at all 0.0 Not Sure 8.8 Parents who are aware of the Read 180 program son/daughter Is taking. Yes No 89.6 10.4 How did you learn about the reading (Read 180) class? School Teacher My kids Other parents/friends % 16.3 28.6 53.7 0.0 Grade level(s) of your son/daughter. 6* Grade 7* Grade 8** Grade 9** Grade 26.8 11.0 20.7 39.6 Ethnicity Caucasian African-American Hispanic Asian Multi-Ethnic 7.3 75.6 2.4 0.6 3.7 % % % Focus Group Results Teacher Focus Groups. A total of 17 Read 180 teachers from seven randomly selected schools participated in the focus groups. The teachers were asked to respond to questions related to implementing the Read 180 curriculum. Full results of the Teacher Focus Group are in Appendix F. Whole group direct instruction. The teachers reported that the best part about the teacher whole-group instruction at the beginning of class was that it provided an opportunity too explain the lesson, address the students all at once, and set the focus and tone for the day (82% of responses). When asked to describe strategies for improving this component of the READ 180 classroom, approximately one-fourth (24%) did not feel any changes were needed. while the same percent of teachers responded the time format should be revised (24%), that smaller classes would be beneficial (24%). Computer module. When describing the strengths of the READ 180 computer activities the teachers explained that it provides new subject matter in a format (computer, video, etc.) that keeps students interested (48%) while also addressing students individual needs and 10/17/2006Page 40 of 123 abilities (24%). The teachers also reported that the computer activities provide repetition and fluency practice (16%) and gave students and teachers immediate feedback (8%). The teachers suggested that overcoming technical issues should be the primary means of improving this READ 180 module. The teachers reported technical failures sometimes resulting in the loss of students work (43%) and equipment issues with headphones, software CDs, etc. (21%) as the major areas for improvement. Small group direct Instruction module. The majority of the responses (55%) indicated that the teachers valued this one-on-one time with students allowing them to interact with and focus on individual students. The teachers explained that during this time individual needs can be met (23%) and that the small groups allowed ease in monitoring (9%). When asked to describe changes that should be recommended the teachers explained that more time (23%) and more space and smaller groups (23%) would be beneficial. The teachers also reported difficulties in being available to students in the other modules (15%) and classroom distractions (8%) as areas for improvement. Independent reading. The teachers reported that independent reading time provides students with an opportunity for quiet reading and practice (21%) and that this activity can be individualized for students reading levels (16%). The teachers also noted that the ability to retake quizzes and get immediate feedback (11%), listen to audio books (10%), allow students to choose their own books to read (11%) as strengths of the independent reading time. While describing strategies for improving this module the teachers reported that the reading collection needed to be updated and increased to include more variety (38%) and that strategies for keeping students engaged for the duration of this module were needed (25%). The teachers explained that an aide/assistant would be helpful (13%) as would smaller group sizes (13%) as this was a time during the READ 180vjhen trouble-makers act up (6%). Modeled reading. Nearly two-fourths (38%) of the teachers described that the best part of students reading in pairs was the opportunity for modeling. The teachers reported that students enjoy reading, sharing, and discussing books using this strategy (23%). However, 15% of the teachers explained that modeling is rarely or never done in their READ 180 10/17/2006Page 41 of 123 classrooms. One-third (33.3%) of the responses suggest that no changes are needed for this module. Other suggestions included more time for peer reading (17%), smaller class size (17%), and smallergroup size (17%). Overall. The teachers reported the following as overall strengths of the READ 180 program: students like and are motivated by the program (16%), the program supports progress and success (16%), students are better able to focus and stay on-task (11%), student reading has increased (11%), repetition provides practice and increases comprehension (11 %), and rotation of activities (11 %). In contrast, the teachers stated that the following were areas in need of improvement: too many technical difficulties (16%), the need for more and better distributed time (16%), more user-friendly reports, and the need for smaller groups of students (10%). All of the teacher responses suggest that the READ 180 program should be continued. They explain that student improvement and success are notable (23%), that students enjoy the program (17%), and an increase in student reading comprehension and skills (17%), and an increase students self-worth from their own accomplishments (10%). Student Focus Groups. A total of 38 students from eight randomly selected schools participated in the focus groups. Full results of the Student Focus Group are in Appendix G. Whole group direct instruction. The students reported that the best part about the teacher instruction at the beginning of class was that they are provided with an overview of what is to be expected (40% of responses). The students also liked that it provided an opportunity forthem to receive study questions, practice sheets and a review of previous work (13%). When asked to describe strategies for improving this component of the READ 180 classroom the majority of the student responses (78%) stated that nothing needed to be changed. The only suggestions given were allowing students more time to complete their work during this component before moving on to the small group rotations (11%) and providing students with a better explanation of expectations for the rest of class (11 %). Computer module. When describing the strengths of the READ 180 computer activities 10/17/2006Page 42 of 123 the students explained that they enjoyed the Spelling Zone (23%), learning new vocabulary in the Word Zone (20%), reading (20%) and videos (11%). Half of the student responses (50%) indicate that no changes to this module are needed. Suggestions that were made included increasing the time allowed for the computer module (17%) and overcoming computer malfunctions (17%). Small group direct instruction module. The student responses indicate that they valued the following as strengths of small group interactions with the teacher: discussion and helping each other (30%). writing (18%), one-on-one attention (12%), and reading and writing essays (12%). The majority of the student responses (67%) indicate that no changes to this module of the READ 180 program are needed. Those suggestions that were given are linked. It was suggested that more time is needed during this module (22%) and that group sizes should be smaller (11). It was explained that if five students working at varying skill/ability levels each need individualized help/instruction, then 20 minutes goes by too quickly for the teacher to assist each student and teach a lesson. Independent reading. The students reported that their favorite aspects of the independent reading time included reading (27%), exploring new books and progressing to more advanced books (27%) and working on vocabulary words and questions (12%). While describing strategies for improving this module the students reported that the reading collection needed to be increased and include more variety, better books, longer books, and more audio books (39%). The students also suggested that this module needed to be more fun (6%) and incorporate opportunities for discussion (6%). Modeled reading. While describing the strengths of reading in pairs the students reported that they enjoyed partner work by modeling reading (25%), that it was more interesting and fun (19%) and helped them focus on emphasis and comprehension (19%). It was also reported that this module was not used in some classes (19%). When asked to suggest improvements for this module 67% reported that no changes were needed and 33% reported a desire to read in pairs more often. Overall. The students reported the following as overall strengths of the READ 180 10/17/2006Page 43 of 123 program: increased time spent reading and improving reading skills (26%), the computer component (16%), working in small groups (14%), and the whole experience (14%). In contrast, the students reported that the following were areas in need of improvement: increase allotment of time (11%), too many computer difficulties (11%) and CD/disk problems (11%), as well as improvements to the book collections (11%). All of the student responses suggest that the READ 180 program should be continued. They reported that improves reading and increases the desire to read (17%), learn more (13%), improves spelling skills (7%) and it is fun (7%). It was also explained that the READ 180 program helped in other classes (7%) which all can result in students feeling more successful (4%). Interview Results Principal Interviews. The ten principals from the schools participating in this READ 180 program evaluation were interviewed to examine their impressions of the program. The majority of these principals (60%) were administrators of schools that were using READ 180 for the second year, 30% were at schools using it for the third year and one school was using the program for the fourth year. Full results of the Student Focus Group are in Appendix H. Overall Impressions. Nearly ail of the principals indicated a positive overall impression of the READ 180 program while one reported that the benefits were dependent on the classroom teacher. Three of the principals indicated that their faculty liked READ 180 and two reported that they personally believe it is beneficial. One principal described that the program is wonderful because it restores confidence in the students. Meeting needs. When asked to evaluate how READ 180 meets the learning needs of African-American students 50% of the responses described that it improves reading abilities and comprehension. One principal indicated that it helps individualize instruction while others mentioned that it provided a practical instructional model for teachers to use. One principal explained that READ 180 is helpful for African-American students with deficient reading, comprehension and analytical skills. While, another indicated that the quality of the teacher 10/17/2006Page 44 of 123 impacted student ability to achieve success. Implementation. The principals were asked to describe how their Read 180 programs were implemented with regard to student selection, role of the Read 180 course (replace or supplement Standard English course), student graduation from Read 180, and use of tracking elements. Responses revealed that students were primarily from 9^*^ grade (25%), but also included grades 6-8 students. Students were primarily selected on the basis of Benchmark scores (77%), while one indicated automatic enrollment for all resource students. one used transcript data, and one staff recommendations. The Read 180 program was equally used to replace or to supplement Standard English courses. Principals indicated that students typically (60%) graduate at the end of the school year. Reporting and tracking elements. Four of the ten principals reported that the tracking forms included in the READ 180 software were used for program assessment. Three interviewees indicated that READ 180 in their schools used the software for pre- and postassessment of student performance. One principal further described that students were assessed at the beginning of the year, quarterly and at the end of the year using these tools. Strengths. The principals interviewed were asked what they considered to be the best aspects or strengths of the READ 180 program. The ability to work in small groups and individualize instruction was reported in 21% of the responses while 16% of the responses focused on the technological aspects of the program. One principal explained that the faculty believes that the READ 180 program meets the students at their reading level and helps them progress from there. It was also reported that the program encourages the use of a variety of teaching strategies (11%), that students, as well as teachers, can track student progress (11%) and that READ 180 classes holds students attention (11%). One principal commented that the faculty loves the program because of the hands-on approach. Another stated that READ 180 improved students vocabulary, reading and comprehension abilities. Improvements. The following were suggested as areas of improvement in the READ 180 program by the interviewees
overcoming technical issues (30%), bad marketing/image of the READ 180 program (20%) and costs of the program (20%). One principal reported that it 10/17/2006Page 45 of 123 was also difficult to ensure that everyone is maintaining data regarding students progress within the READ 180 program. Another indicated that as a result of the programs success the faculty at his/her school would like to offer the program to more students. However, the programs expense made this an unlikely possibility. Student Achievement Analyses and Results Student Achievement Analyses For all tests of program effects, a matched-samples design was employed. Where possible, each Read 180 student was individually matched to another student within the same school, at the same grade level, with the same 2005 ITBS Reading NCE score and demographic characteristics. In 13 of the 15 school/grade level combinations in which the program was implemented, this process resulted in nearly perfect matching based on a comparison of Read 180 and non-Read 180 students 2005 ITBS Reading NCE scores. Independent samples t-tests were performed on 2005 ITBS Reading NCE scores within each grade and school, and in 12 of 15 cases the resulting p-value was at or above 0.95, where 1.00 indicates a perfect match (i.e., exactly the same pretest mean
see Table 12). The least efficient matching occurred at Central High and J.A. Fair High, where the p-values were 0.481 and 0.494, respectively. In these two instances, a preponderance of students with low pretest scores participated in the program, so it was not possible to match each program student to a control student on a one-to-one basis. Nevertheless, the results of the pretest comparisons indicate that there were no statistically significant differences between program and control students within any grade level or school. 10/17/2006Page 46 of 123 Table 12 Mean 2005 ITBS Reading NCE Scores (Pretest) by School, Grade, and Treatment Condition: Matched Samples School Central High Grade 9 Hall High Parkview High Treatment Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 J.A. Fair High Pulaski Heights Middle Not Read 180 Read180 Not Read 180 Read180 Southwest Middle McClellan High Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Henderson Middle Cloverdale Middle Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Mabelvale Middle Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Not Read 180 Read180 Mean 29.89 28.94 37.88 37.84 41.67 41.67 28.78 27.68 29.36 28.76 24.32 24.24 28.00 28.27 24.27 24.41 27.15 27.18 28.90 28.57 40.11 40.00 29.36 29.14 26.10 26.27 25.91 26.11 36.92 36.92 38.61 38.74 36.68 36.63 SD 10.169 12.172 11.244 11.261 12.228 12.228 11.741 13.643 13.003 13.989 13.431 13.627 14.386 14.109 13.854 13.727 9.144 14.011 14.734 15.148 11.985 11.995 15.237 15.207 13.757 12.642 12.094 11.979 14.866 14.855 20.043 20.409 12.679 12.691 n 64 64 51 51 15 15 51 103 25 25 25 25 22 22 22 22 39 68 21 21 35 35 28 28 30 44 44 44 25 25 31 31 38 38 t .481 .018 .000 .494 .157 .021 -.063 -.033 -.009 .072 .040 .053 .056 -.080 .000 -.025 .018 .632 .990 1.00 .620 .876 .983 .950 .974 .993 .943 .968 .958 .956 .937 1.00 .980 .986 9 9 9 6 6 7 8 9 7 8 6 7 8 6 7 8 A total of 661 Read 180 (72% of participants) had matching 2005 ITBS, district enrollment, program enrollment records, and 2006 ITBS scores. The matched comparison sample was comprised of a total of 566 students. The overall number in the comparison sample was smaller due to the lack of sufficient one-to-one matches at Central and J.A. Fair. In addition to the near exact matches on pretest scores, program students and comparison 10/17/2006 Page 47 of 123 students were quite similar on other demographics: 52.6% vs. 51.8% male, 93.5% vs. 92.4% African American, 71.4% vs. 71.2% eligible for free lunch, and 19.1% vs. 17.5% special education students for Read 180 and the control group, respectively. For each grade 6-9, a multivariate analysis of covariance (MANCOVA) was performed with program (Read 180, Control) and school serving as independent variables
2006 ITBS Vocabulary, Reading Comprehension, Total Reading, and Revised Writing NCE scores as outcomes
and 2005 ITBS Reading NCE, student sex, special education status, and free or reduced-price lunch status as covariates. Wilks lambda was used as the criterion of multivariate significance and alpha was set at .05. When MANCOVA indicated a significant multivariate effect, univariate follow-up tests were conducted using a sequential Bonferroni approach
i.e., employing an alpha of .0125 to adjust for the fact that four outcome measures were being analyzed. Effect size estimates were computed for each outcome by subtracting the covariate-adjusted Control mean from the Read180 adjusted mean, and dividing the difference by 21.06 (the national norm standard deviation for NCE scores). Arkansas Benchmark Literacy scale scores were available for grades six through eight. For these grades, analyses of covariance were performed with program (Read 180, Control) and school serving as independent variables
Literacy scale scores as outcomes
and 2005 ITBS Reading NCE, student sex, special education status, and free or reduced-price lunch status as covariates. Effect size estimates were computed by subtracting the covariate- adjusted Control mean from the Read180 adjusted mean, then dividing the difference by the total standard deviation for each respective grade level. The percentages of students obtaining proficiency were computed for each combination of grade level, school, and treatment. Student Achievement Results Sixth grade ITBS 2006 subtests. MANCOVA indicated no statistically significant multivariate main effects for program (X, = 0.98, = 1.11, p = .36) or for the school X program interaction effect (X. = 0.92, F.,2 493 = 1.31, p = .21). Directionally, results tended to favor Control students 10/17/2006Page 48 of 123 (see Table 13 and Figures 1-4), particularly at Cloverdale and Mabelvale. At Pulaski Heights and Southwest, effect size estimates across subtests ranged from -0.11 to +0.05, showing essentially no differences between ReadlSO and Control students. At Mabelvale, moderately large negative effects were observed on Vocabulary (-0.39), Reading Comprehension (-0.21), and Total Reading (-0.33
see Table 12). Mean NCE scores for all groups across all subtests were quite low, ranging from 17.96 on Total Reading for the Read 180 group at Southwest Middle, to a high of 32.28 on Vocabulary for the Control group at Mabelvale Middle. 10/17/2006Table 13 Page 49 of 123 Sixth Grade 2006 ITBS Means, Adjusted Means, and Effect Size Estimates^ by Treatment Condition and School Vocabulary: NCE Treatment Control School Pulaski Heights Middle Mean 27.65 Adjusted Mean 27.47 Effect Size N 23 1 Read180 Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle 20.60 27.50 32.28 27.63 19.80 24.46 24.56 22.38 27.82 29.81 28.31 22.31 24.80 21.51 0.04 0.00 -0.14 -0.39 25 28 25 24 25 28 25 Reading Comprehension: NCE Reading Total: NCE Revised Writing: NCE Control Read180 Control Read180 Control Read180 Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Pulaski Heights Middle Southwest Middle Cloverdale Middle Mabelvale Middle Effect size estimates are all based on the NCE standard deviation of 21.06. 29.43 26.00 28.61 31.04 27.58 22.96 24.61 27.00 26.74 21.52 26.43 30.64 25.79 17.96 22.79 24.24 31.87 25.20 28.96 29.44 28.58 22.04 23.61 30.92 29.29 28.17 28.73 27.53 28.99 26.65 24.86 23.01 26.55 23.84 26.69 27.12 27.03 21.60 23.13 20.11 32.07 26.19 28.90 26.03 30.55 27.19 23.62 26.16 23 25 28 25 -0.01 -0.07 -0.18 -0.21 0.02 -0.11 -0.17 -0.33 -0.07 0.05 -0.25 0.01 24 25 28 25 23 25 28 25 24 25 28 25 23 25 28 25 24 25 28 25 10/17/2006Page 50 of 123 Means by School and Treatment. Control Readl80 10/17/2006 Page 51 of 123 30- 29- in c S 28- s re E 27- ra s o 01 t# 26- E 25- 111 24- T T T Pulaski Heiglits Southwest Cloverdale Middle -------- Control -------- Readl80 23- Middle Middle Mabelvale Middle E n T Figure 2. Sixth Grade 2006 ITBS Reading Comprehension NCE Adjusted Means by School and Treatment. 10/17/2006 28- Coiitiol Page 52 of 123 (A 26- 01 5 ra c '5 n 24- Z o 01 I# E 22- lU 20- T T T T Pulaski Heiglits Southwest Mabelvale Middle Cloverdale Middle - Read180 Middle Middle Figure 3. Sixth Grade 2006 ITBS Total Reading NCE Adjusted Means by School and Treatment. 10/17/2006 32- lA n soot Sw c 528- k. n s D 01 Io 26- E <0 LU 24- --------Control Readl80 22 Pulaski Heiglits Southwest Middle Middle Cloverdale Middle I Mabelvale Middle Page 53 of 123 T T T Figure 4. Sixth Grade 2006 ITBS Revised Writing NCE Adjusted Means by School and T reatment. Benchmark Literacy. ANCOVA revealed no statistically significant main effects for program (F.^ = 3.23, p = .07), and no program X school interaction effect F3193 = -11. P = .96). Directionally, results consistently favored the Control group, with effect size estimates ranging from -0.27 at Pulaski Heights to -0.12 at Cloverdale (see Table 14 and Figure 18). The percentages of students obtaining proficiency on the Benchmark Literacy exam were nearly equal between Readl80 and Control groups at Southwest (0.0% vs. 4.0%), Cloverdale (14.3% vs. 14.3%), and Mabelvale (16.0% vs. 20.8%), but only half the percentage of Read 180 students achieved proficiency relative to Control students at Pulaski Heights (12.0% vs. 24.0%
see Table 15). Table 14 10/17/2006 Page 54 of 123 2006 Benchmark Literacy Scale Score Means, Adjusted Means, and Effect Size Estimates by Grade Level, School, and Treatment Grade School Treatment Mean Adjusted Mean Effect Size N 6 Pulaski Heights Middle Control 534.04 539.07 25 7 8 Read180 487.24 501.06 -0.27 25 Southwest Middle Cloverdale Middle Mabelvale Middle Southwest Middle Henderson Middle Cloverdale Middle Mabelvale Middle Southwest Middle Henderson Middle Cloverdale Middle Mabelvale Middle Control 434.60 451.97 25 Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 Control Read180 373.00 518.54 505.61 537.71 518.84 465.18 450.91 550.67 541.62 565.17 522.50 600.97 629.57 620.57 581.59 733.66 707.26 594.95 595.40 649.82 704.84 431.13 523.56 506.76 490.71 462.69 498.85 479.84 550.10 576.35 584.53 533.80 555.98 570.30 651.44 647.78 690.90 652.18 629.81 629.24 629.56 680.53 -0.15 -0.12 -0.20 -0.12 0.16 -0.31 0.09 -0.03 -0.29 0.00 0.38 25 28 28 24 25 22 22 21 21 30 44 31 30 21 22 35 35 44 43 38 37 Note. Total standard deviations by grade were 142.00,163.52, and 135.24 for grades six, seven, and eight, respectively. 10/17/2006Page 55 of 123 Control Readl80 Figure 18. Sixth Grade Mean Adjusted 2006 Arkansas Benchmark Literacy Scale Scores by School and Treatment. 10/17/2006 Page 56 of 123 Table 15 2006 Arkansas Benchmark Literacy Proficiency Categories by School and Treatment Condition, Sixth Grade School Treatment Control Read180 Pulaski Heights Middle Proficiency Category Not Proficient 19 22 Southwest Middle Proficiency Category Proficient Not Proficient % within treatment condition % within treatment condition 76.0% 88.0% 24.0% 12.0% 24 25 n n n 6 3 Proficient % within treatment condition 96.0% 100.0% n 1 0 Cloverdale Middle Proficiency Category Not Proficient Proficient Mabelvale Middle Proficiency Category Not Proficient Proficient % within treatment condition % within treatment condition % within treatment condition % within treatment condition % within treatment condition 4.0% 24 85.7% 14.3% 19 79.2% 20.8% .0% 24 85.7% 14.3% 21 84.0% 16.0% 10/17/2006 n n 4 4 n n 5 4Page 57 of 123 Seventh grade ITBS 2006 subtests. MANCOVA indicated a statistically significant multivariate main effect for program (A = 0.95, F^ 205 = 2.81, p = .03), with no school X program interaction effect (A, - 0.92, F^2,543 .37, p = .18). Follow-up univariate tests showed a significant program effect for Reading Comprehension {F.^ 208 = 10-59, p = .001) and Total Reading {F.^ 208 ^-^0, p = .006). In both cases, the overall adjusted mean for Read 180 students was significantly lower than that of the Control group = 30.96 vs. Mq = 36.55 for Reading Comprehension
M.^qq = 28.84 vs. Mq = 33.64 for Total Reading). As shown in Figures 5-8 and on Table 16, Reading Comprehension and Total Reading scores consistently favored Control students at all schools, with effect size estimates ranging from -0.44 at Cloverdale Middle on Total Reading to -0.13 at Mabelvale Middle on Total Reading. Mean NCE scores across subtests for all groups, while somewhat higher than sixth grade, were still quite low. ranging from M = 23.5 on Vocabulary for the Southwest Middle Control students, to M = 43.43 on Reading Comprehension for the Mabelvale Control group (see Table 16). 10/17/2006Page 58 of 123 Table 16 Seventh Grade 2006 ITBS Means, Adjusted Means, and Effect Size Estimates^ by Treatment Condition and School Vocabulary: NCE Treatment Control School Mean Adjusted Mean ES N 1 Read 180 Southwest Middle Henderson Middle Cloverdale Middle Mabelvale Middle Southwest Middle Henderson Middle Cloverdale Middle 23.50 25.82 22 31.14 34.27 39.87 26.43 23.90 25.41 31.67 36.82
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resoources.