Oral deposition of Victoria Bernhardt, Ph.D.

Deposition taken at John W. Walker Law Firm, Little Rock, Arkansas
This transcript was created using Optical Character Recognition and may contain some errors.
ORIGINAL IN THE UNITED STATES DISTRICT COURT EASTERN DISTRICT OF ARKANSAS WESTERN DIVISION LITTLE ROCK SCHOOL DISTRICT, PLAINTIFF vs No.4:82-CV-866 PULASKI COUNTY SPECIAL SCHOOL DISTRICT, ET AL, DEFENDANTS MRS. LORENE JOSHUA, ET AL KATHERINE KNIGHT, ET AL INTERVENORS INTERVENORS DEPOSITION OF DR. VICTORIA BERNHARDT DATE: January 10, 2007 TIME: 5:59 p.m. PLACE: The John W. Walker Law Firm 1723 Broadway Little Rock, AR 72206-1220 APPEARANCES RECEIVE JAN 19 2007 OFACEOF DESEGREGATION MONllORI G On Behalf of the Plaintiff: Christopher Heller, Attorney Friday, Eldredge & Clark On Behalf of the Defendants: 400 West Capitol, Suite 2200 Little Rock, AR 72201-3493 John W. Walker, Attorney The John W. Walker Law Firm 1723 Broadway Little Rock, AR 72206-1220 ALSO PRESENT Joy Springer, Legal Assistant, The John W. Walker Law Firm Cobb Court Reporting P. O. Box i4, Sweet Horne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 AGREEMENT OF COUNSEL. SWEARING OF THE WITNESS. I N D E X EXAMINATION OF DR. VICTORIA BERNHARDT By Mr. Walker. By Mr. Heller. COURT REPORTER'S CERTIFICATION. * * * * * * * * * Cobb Court Reporting ORIGINAL P. o . Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria . cobb@cobbcourtreporting.com 2 PAGE . 3 . 3 .3-38 38-67 . 71 I I 1 2 - 3 4 I 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 3 ORIGINAL STIPULATIONS The deposition of Dr. Victoria Bernhardt, produced, sworn and examined at the John W. Walker Law Firm, 1723 Broadway, Little Rock, AR 72206-1220, commencing at 5:59 p.m., on January 10, 2007 in the captioned cause at the instance of counsel for the Plaintiff, and said deposition being taken according to the terms and provisions of the Arkansas Rules of Civil procedure. It is stipulated and agreed all forms and formalities in the taking, transcribing, forwarding and filing of said deposition, are hereby waived by the parties, the right being expressly reserved to object to the testimony of the witness at the time of trial as to incompetency, irrelevancy and immateriality, other than those with respect to the form of questions as propounded to the witness. * * * * * P R O C E E D I N G THEREUPON, DR. VICTORIA BERNHARDT having been called for examination by counsel for the plaintiffs, and having been first duly sworn, was examined and testified as follows: DIRECT EXAMINATION Questions by Mr. Walker: Q. Can you hear me fairly well, doctor? Cobb Court Reporting P . 0 . Box #4 , Sweet Home , AR 72164- 0004 Phones : (501) 490 - 0066 - Off
(501) 590 - 0975 - Cell (501) 490 - 0926 - Facs imile e - mail : gloria . cobb@cobbcourtreporting . com 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ORIGINAL A. Yes, I can hear you
thank you. Q. All right
this is John Walker. Have you ever given a deposition before? A. No, I haven't. Q. All right. First of all if any question that I or Mr. Heller ask you is unclear or you do not understand it, would you be kind enough to let us know that you do not. A. I do. 4 Q. And ah, I'll try not to talk too fast or interrupt you but I would like for you to be able to give me a full answer and I'll try not to interrupt you and Mr. Heller and I hopefully will not talk at the same time. If there's some reason that you need to interrupt the deposition just feel free to just say it on the record and then we'll stop for a while. I don't expect this deposition to be lengthy at least from my part but Mr. Heller has a right to ask you questions as well. I'm prepared to begin--is there any prelimi nary statement you want to make Chris? HELLER: No. Q. Dr. Bernard--- SPRINGER: Bernhardt. Q. Bernhardt, I called you Bernard
I'm sorry. Dr. Bernhardt when did you first meet Dr. Karen DeJarnette? A. Oh, wow. I'm trying to place the timing
it was before she was working at Little Rock School District so it was ah, she Cobb Court Reporting P. O. Box 14 , Sweet Home, AR 72164 - 0004 Phones : (501) 490-0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e -mail: gl oria .cobb@cobbcourtreporting . c om 5 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 OR\Glt~AL was also working with-- I was working .as controller with the Gala foundation as well so it put it in ah- - oh , like either late 1999 I think . Q. All right . Are you familiar with her qualifications? A. Very much so . Q. I see. How would you describe her qualifications to 5 perform the tasks of professional evaluations and assessments? A. Ah, from my knowledge of Karen's work ah , I would say she is probably the strongest professional program evaluator that I know that's very much research based, she is extremely strong in the profession and ah, I have never known her to cut corners or do something less than full, full on
I don't know how to describe it any other way. She has a lot of personal and professional integrity and I've always admired her work. Q. All right. What is your highest degree, doctor? A. PhD. Q. From what institution? A. University of Oregon. Q. In what discipline? A. Educational psychology research and measurement so that's basically statistics like to psychology in education . Q. All right. Are you currently on the faculty of any university? A. Yes, I'm a professor at a California State University Chico also I'm on leave to direct the education for the teacher Cobb Court Reporting P . o . Box f4 , Sweet Home, AR 72164 - 0004 Phones: (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e -mail : gloria.cobb@cobbcourtreporting.com 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ORlGll~AL 6 initiative that I've been directing since 1991. Q. I see. With all due respect I have to ask you this since the court will not get to hear you directly. Would you tell me whether you have received acclaim for work as a person who engages in providing evaluations and assessments of school programs? A. Have I received consulting fees? Q. No, ma'am
have you received acclaim? A. A claim? Q. Yes, ma'am
honors and awards and things like
distinctions? A. Ah, yes
yes, I have. Q. I see. And are you published? A. Yes. Q. Have you published a book or is it journal articles? A. Ah, both. I have published eleven books
I have two more in press at the moment. Q. I see. A. And t hen several journal a r ticles also. Q. I see. WALKER: We already have, I think we have access t o your vita and we intend Mr. Heller, to make that an exhibit at trial unless there's some objection. HELLER: WALKER: In the fourth quarterly report? Yes, it is. Cobb Court Reporting P . O. Box #4, Sweet Horne , AR 72164- 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting.com 3 5 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 7 HELLER: Submitted to the court. WALKER: All right. Q. Now what was your work with the Little Rock School District and when did you begin it to your recollection? A. Ah, to my recollection ah, the work began in early 2005 and that started out with conversations with Dr. DeJarnette about how we were going to work together to ah, embed assessments throughout the district. So it started out with conversation, written proposal and then I believe my first visit with that proposal would have been ah, the spring of 2005. Q. I see. Did you have occasion to meet with her staff? A. Yes. Q. Are you aware of Dr. Ed Williams? A. Yes. Q. Did you ever have any one-on-one meetings with Dr. Williams? A. One-on-one, no. Q. Were you ever in a position to make an assessment of Dr. Williams' qualifications? A. Ah, yes--- Q. When did--- A. ah, Dr. Williams was apart of the team that came to, a four person team that came to our data analysis summer institute in Chico, California and ah, they were among fifty participants that worked with me and my staff for that, for a week, the last Cobb Court Reporting P. o. Box f4, Sweet Home, AR 72164-0004 Phones : ( 501) 490-0066 - Off
( 501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 3 5 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ORIGINAL 8 weekend in July 2005 and then I've ah, seen some of his work when I visited PRE. Q. What did you understand his responsibility to be? A. I understood Dr. Williams' responsibility to be a statistician ah, somebody that helped PRE gather data, supports from them evaluations, assessment in the schools and probably a staff member that did anything that was needed to support the data needs of schools
that's typically what a staff member in a program evaluation and research job would do I would say in a district. Q. I see. Did you have occasion to meet with the staff on more than several occasions? A. Every time I was in Little Rock I met with the staff ah, so I know April 2006, the August / September 2005 then of course it was basically the PRE staff without Dr. DeJarnette in Chico at our institute the last of July in 2005. Q. Did you discern at that time any conflict among the staff members? A. Absolutely not. Q. All right. Now with respect to your task did you have occasion to read Judge Wilson's compliance remedy? A. I d i d quite a while ago. Q. You did. In that compliance remedy he makes reference to the concept -of deeply embedding the assessment and evaluation process into the curriculum. Do you recall that? That's at Cobb Court Reporting P. o. Box 14 , Sweet Home , AR 72 164 - 0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (501) 490-0926 - Fa csimile e-mail : glo ria .cobb@cobbcourtreporting .com 1 2 3 4 5 6 7 I 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 OR\G\~lAL 9 Part B, Page 6-3 of the draft quarterly report update as of September 1, 2006--you may not have that before you but ah, that's the section which is entitled devise and embed a comprehensive program assessment process
do you recall having read that? A. Yes, and more than anything discussing how a district will go about doing that with Dr. DeJarnette. Q. All right. Could you tell me what is meant by first of all a comprehensive assessment process? A. Collect--- HELLER: John--- WALKER: Yes, Chris. Did you say something Chris? HELLER: Ah, ah, go ahead. Q. What is meant by a comprehensive assessment process? A. Well, to me a comprehensive assessment process for a district would mean that we're looking at the data that would tell us about the system ah, about its effectiveness, efficiency of operating, and would tell the district what's working, what's not working within the system, and how to close achievement gaps ah, how to measure to know if there are gaps or if elements of the system are not working, the way it actually gets implemented could be very different in any particular district but basically it means to me looking at every little nook and cranny to understand efficiency and Cobb Court Reporting p , o. Box i4 , Sweet Home , AR 72164- 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Fac s i mile e - mail : gl oria.cobb@cobbcour t r eporting. com 2 3 4 5 6 7 " 8 9 II 10 II 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 10 effectiveness and then to do something about it. Q. Could you explain what is meant by--well, first of all are you familiar with the term deeply embedding a process? A. Well, to me that means making it a part of the way you do business and making it a part of the daily work. Q. Is that sometimes referred to as the culture of the situation? A. It could be the culture, it could mean sustaining or truly implementing. Q. Are you familiar with any literature regarding that subject? A. Yes. Q. Is this a term that is in your opinion, one that is not capable of being easily understood? A. I think it's easy to understand on a surface level to actually do the implementation is really hard. Q. All right. On the surface level you've just given me a description of what is involved? A. Uh-huh. Q. And in terms of implementing it would you explain what some of the difficulties are in embedding the process deeply? A. Ah, the hardest thing is really cheap. You mentioned something earlier, culture ah, the hardest thing with the embedding new practices in a system, an organization, any organization is getting people to work differently because Cobb Court Reporting P. O. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 2 3 4 5 6 7 8 9 10 11 12 I 13 14 15 16 17 18 19 20 21 22 23 24 25 OR\G\NAL 11 culture is really about--work culture is about how we do work and in order to get people in this case using data on an ongoing basis in a way that maybe they haven't done it before there has to be new structures put in place for how we work, how we look at data, when we look at data, and probably the hardest piece and the piece that makes it happen is strong leadership and in a district it would be from the top all the way down ah, principals has to be a part of this vision, everybody has to have the same vision first of all of what it would look like kind of like if we're really doing this and then how we're going to carry it out and then that piece has to be monitored and measured as well as everything else to make sure that we are working differently and doing ah, really embedding work differently. Q. All right. So a monitoring component is essential or if the process is to be is to work and become deeply embedded? A. Absolutely. Q. Do you have an opinion as to whether simple promulgation of a regulation announcing that you are going to do a certain thing is sufficient to constitute being a program evaluation p r ocess being deeply embedded into the curriculum of a school d i strict? A. Ah, just announcing that things are going to happen is not enough. Q. Does the announcement itself have value? Cobb Court Reporting P. O. Box 14 , Sweet Borne , AR 72164 - 0004 Phones : ( 501) 490 - 0066 - Off
( 501) 590- 0975 - Cell (50 1 ) 490 - 0926 - Facsimile e -ma i l : glor ia . cobb@c obbc ou rtreporting .com 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 12 A. The announcement has value
one probably wouldn't. Continuous announcements would in addition to training and ah, clarification of what we're talking about in terms of describing a vision. Q. I see. A. And for example? Q. Yes. A. And providing visual or structure to do the work. Q. I see. Is the importance of the process governed to an extent or determined at least to an extent by the relationship in the organizational chart of the PRE department to the top administration of the district? A. Okay, could you repeat that again? Q. Yes. What is the relationship between having the PRE department relate to the highest echelon of the district and the concept of the process being embedded into the curriculum? A. Okay, if for instance if a department like program evaluation research was trying to implement to embed assessment throughout the district I would say there's no way on earth that that assessment could ever be embedded without the support from the highest positional in a district. Q. All right. Are you familiar with the Little Rock School District? A. Q. Yes. Have you ever had any substantive meeting with the members Cobb Court Reporting P. o. Box 14, Sweet Borne, AR 72164-0004 Phones: ( 501) 490-0066 - Off
( 501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria . cobb@cobbcourtreporting . com 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 of the Board of Education of the Little Rock School District? A. No, I have not. Q. Have you ever had any substantive meeting with the current superintendent of School, Dr. Roy Brooks, of the Little Rock School District? A. Ah, not to my recollection. Q. Have you ever had any substantive meeting with the Deputy Superintendent of Schools, Mr. Hugh Hattabaugh? A. Substantive, no. Q. I see. Have 3/0U had any substantive meeting with Dr. Olivine Roberts with respect to embedding the evaluation and assessment process into the curriculum of the district? A. Yes. Q. Can you tell me how many such conversations you have had with her? A. Ah, I would think probably two occasions I'm thinking, you know, when I'm in Little Rock I'm usual ly there for a week so I know that I had conversations with her in April 2006 when I was there and t hen in August / September 2005 when I was there. Whether i t was one, two, or three conversations I don't know
she a l so a t tended some of the training that I did with the principals and her staff as well
yeah. Q. I see. Well, were you familiar with her qualifications in general? A. Ah, very generally I did work with her ah, on a leadership Cobb Court Reporting P . o . Box 14, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria .cobb@cobbcourtreporting .com 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ORIGlr,.JAL conference when she was in Orlando. Q. And when she was in Orlando did she have a senior level administrative position? 14 A. I'm not sure what Orlando would consider senior level and I wasn't, I know that her position was ah, I think she's the director of Professional Development in Orlando but I don't know if it was for one area or if it was for the whole district. Q. Is that what she told you? A. That's just my recollection at this point. Q. I see. A. Yeah, she had to ah, she contacted her office and made the arrangements. Q. I understand. Were you aware that her primary qualifications were in the field of mathematics? A. No. HELLER: Object to that question
misstates states the premise. WALKER: Would you mind stating for the record Chris what's wrong with that? HELLER: Assumes the fact that's not in evidence and not true. WALKER: Did she not indicate that her degrees were basically in mathematics and her experience mathematics? BERNHARDT: Are you asking me? Cobb Court Reporting P . o. Box 14, Sweet Home, AR 72164-0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (50 1 ) 490- 0926 - Facsimile e - mail : glo r i a.cobb@cobbcourtreporting . com 1 3 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 WALKER: I was asking Chris. HELLER: Are you asking Dr. Bernhardt? WALKER: No, Chris Heller. HELLER: No, no, she didn't. WALKER: I see
all right. 15 Q. Now did you ever interact with her and Dr. Karen DeJarnette and the PRE team together on a substantive basis at any time while in Little Rock? A. I--yes, mostly in a training scenario. Q. I see. Was that on more than one occasion? A. Yes. Q. All right. What did you understand her role in that training to be? A. Ah, my understanding was that she also ah, she and Dr. DeJarnette were responsible for me being there and that Dr. Roberts' position is the--was/is supervisor of Dr. DeJarnette and one of the sessions--well, okay I'll just stop there. Q. All right. What happened during one of the sessions that you were about to report? A. Ah, well one of the sessions was the training with principals but another session ah, Dr. Roberts was in and Dr. DeJarnette and also the PRE staff in addition to some other administrators and principals did an assessment on a continuous improvement continuums of the district. And so there was substantive conversation about how the district gathers data, Cobb Court Reporting P. o. Box J4, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria . cobb@cobbcourtreporting.com 5 6 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 16 ORIG\\lAL uses data now , have about a vision it's something that we took a great deal with, with a lot of districts around the country. Q. Is that the occasion during which you discussed the need for having a comprehensive data retention source? A. I don't--we certainly did discuss that at that point and time but I don't think that was the first time it was discussed. Q. I see. Did you make recommendations to LRSD regarding embedding a comprehensive assessment process into the curriculum? A. Yes, I did . Q. Did that involve or include development of a data warehouse? A. Absolutely
a way to pull out the data and make it accessible throughout the district, yes. Q. Did you have that discussion with Dr. Roberts as well as Dr. DeJarnette? A. I believe so. Q. Now what did you feel was needed that Little Rock did not already have? A. Ah, Little Rock like a lot of districts around the country it's very, very, very hard to pull the data together. Ah, there are data sources, there are data elements that are in nooks and crannies around the district and what they don't have it one way to pull it all together and be able to cross all Cobb Court Reporting P. O. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 2 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 that information to get much better information. Also, they don't have tools for a quick analysis and they didn't have a way for teachers or administrators, principals ah, district administrators, staff members to all pull the same data, all pull from the same data source any time they needed it or wanted it. 17 Q. How did you happen to arrive at these conclusions? Was it based on what you were told or did you otherwise learn that? A. Ah, it was part--it was in part what I was told and it was also in part from ah, PRE trying to put together data like ah, the institute that I mentioned that was at Cal-State Chico the last week in July in 2005 that the team attended ah, that's all about pulling that data together and understanding different report~, different analysis and I know from observing that the team had a very, very difficult time pulling that data t ogether, getting it reformatted, cleaning it and ah, trying to pull it into one source. Q. Now did you make a recommendation as to how they may go about pul l ing it into one source? A. Yes. Q. What was that recommendation? A. Oh, I recommended that they would get data warehouses and different warehousing companies, I also showed them the one that I use which is ah, it's called Easy Data Analyzer by Tetradata Corporation and ah, that warehouse would do just Cobb Court Reporting P . 0. Box #4, Sweet Borne , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria . cobb@cobbcourtre porting . com 5 6 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 II 24 25 everything that we wanted it to do to ah, for embedding assessments across the district. Q. Was Dr. Olivine Roberts privy to that recommendation? A. Yes, she was. Q. Did she voice any objection to it? A. Not in my presence. Q. I see. Now did anyone, Dr. Williams or anyone voice objection to it? 18 A. Not that I know of. When I saw them viewing the product they were ah, the PRE staff I should say ah, when I saw them viewing the product they were pretty much blown away with what they could do if they had such a tool for helping them with the data. Q. I see. I have heard and there's a writing and there are comments that ah, you had a conflict of interest in that you recommended that enterprise and the conflict was because you were a member of the board that developed and marketed Tetradata. Would you react to that, please? A. Yeah, I'll react to that. Let me just tell you this: Since 1991 all I had been doing was working with school and school districts on data analysis and this kind of work. And actually befor e that I was also doing it but not as intensively as I started in August of 1991. And at that point in time one of the, the major issue that came up in every single situation was I think that the schools and districts were willing to use Cobb Court Reporting P. o. Box 14, Sweet Horne , AR 72164 - 0004 Phones: (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e -mail : gloria . cobb@cobbcourtreporting . com I II II I n n 5 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 19 the data it's just they didn't have the tools to do that. So what I did in 1991 was go beat on the doors of data warehousing companies all around the country and I have billet in California so that's most of them are in Silicon Valley and at that point in time there weren't very many companies out there and they all had business solutions and I tried to talk with them about adapting their product so that it could work for schools and ah, every data warehousing company that I spoke with, with the exception of one ah, pretty much slammed the door in my face and laughed at me saying there's no future-there's no money in schools. And I said there's a future in schools, this is our future, this is what's going to, you know, get you your next engineers to keep your company going. Well, no existing data warehous.ing company was interested in working with us with the exception of Rio and basically they just said let's just give you and application and get out of my way and ah, so we used their product on a demo basis or a pilot basis I should say and we gave them all kinds of information back, we set up pilot schools all over the place, I was so excited that we were finally getting tools that would help us all do what takes months, months, and months for us to do by hand that we could do it very, very quickly. We gave them recommendations and they were like yeah, okay, just get out of my way and I, I was very, very discouraged. In late 1991 the CEO of Tetradata, Mark Bertoski called me and said we are a brand new company Cobb Court Reporting P . o . Box f4 , Sweet Horne , AR 72164 - 0004 Phones: (50 1 ) 490 - 0066 - Off
(50 1 ) 590 - 0975 - Cell (501) 490 - 0926 - Fa csimi l e e - mail : gl oria . cobb@cobbcourtrepor ting. c om 3 5 8 l 10 11 12 I 13 14 15 16 17 18 19 20 21 22 23 24 25 20 there's only three owners and we're trying to build a data warehouse to do everything that's in your data analysis book. And I almost hung up on the guy because I didn't know how to take the call and, and I spoke with him about what they were trying to do and I said to him the minute your product can do six things I will come to South Carolina where they're located at my expense and consult with you for a couple of days
and I still didn't know if it was for real and that's basically how the arrangement started. I went back for two days to work with them on what I would want in a data warehouse and I stayed an entire week because it was so exciting because they were able t o do things that I've never, I've never, I never before that point and time was able to do electronically and quickly and easily. So I became their main advisor and they did want to pay me but I refused to take payment because I wanted ah, I wanted to be able to tell districts that this is the product, the only one out on the market that will do all of these things that we talk about in comprehensive data analysis work. I'm currently the chair of their advisory committee and ah, I basically I use their product and ah, mostly the advice, I don't know I advise them when they have a question or they're moving in development ah, I don't know how that's a conflict of interest for the district because I'm not an employee of the district. There are other companies that have asked me to work with them and they are not committed to the kind of development Cobb Court Reporting P. o. Box 14 , Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590 - 0975 - Cell (501) 490- 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting . com II 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ORl GI l~AL 21 work that this kind of analysis would require and therefore I'm not working with them. advise. So Tetradata is the only company that I Q. Are you a stock owner of Tetradata? A. Ah, I have some stock options--no, I currently am not. Q. All right. A. At one point they talked about it. Q. All right. Are you on their board of directors? A. No. I'm just on their advisory committee. Q. All right. Now do you perceive a conflict of interest in recommending that a program that you have setup be utilized for the purpose of program evaluation? A. I do not see that as a conflict of interest because it's ah, I know that this program does what we want to have happen in the embedding of assessment and ah, so it would be a likely recommendation. Q. All right. A. I'm not going to recommend something that doesn't do what we're talki ng about doing. Q. All right. Are you familiar with Business Solutions? A. Is it a company? Q. Ah, I think that you mentioned it earlier
it may have been called Crystal Objects or Business Solutions, Business Objects? A. Ah, I am familiar with Crystal--- Q. Crystal Reports--- cobb Court Reporting P . o. Box 14 , Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Fac simile e-mail : gloria . cobb@cobbcourtre porting.com 3 4 5 6 7 ll 8 9 ll 10 ll 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 A. Reports. Q. yes. 22 A. Ah, when I was speaking about business solutions earlier I was talking about the generic term. Q. All right. What is your experience with Crystal Objects? A. My experience in other states, I didn't experience it in Little Rock but my experience in other states is that schools and districts ah, how do I say it nicely, swear at it ah, because it's very, very cumbersome, hard to use, and you can't cross the variables that you want to cross it's very, very hard, hard to use, there's a steep learning curve and ah, it's very time consuming, it takes a long, long time to do a few analysis so nobody like, I haven't seen schools or districts really do the comprehensive analysis that we'd like to see done with the footwork. Q. Did you learn that there was opposition to using Tetradata in the Little Rock School District? A. Ah, just recently. I didn't, I mean opposition, yeah. I do know that one of the times that I was, at one time when I was there probably the April time we talked about Tetradata and show them Tetradata, the next time I'm there I heard from Dr. Roberts oh, they were going to get Crystal Reports. So I just figured it was through the analysis and they're figuring out what was going to work for them and that's not unusual but I didn't hear that there was a big opposition. Cobb Court Reporting P. O. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail : gloria.cobb@cobbcourtreporting.com 3 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Q. I see. Did Dr. Roberts discuss with you the reasons for using business objects or crystal objects? A. Crystal Reports? Q. Crystal Reports, yeah. A. Yes, we just had a brief conversation about that. Q. What did she explain as the reason? A. Ah, I didn't--when she was talking to me about it in conjunction wi th purchasing another product that I think was going to do assessment for them so I understood it to be more o f a financial situation than anything and I also know what happens typically in districts is when they buy something like Crystal Reports which is really kind of like a software program that ultimately or either two years down the road they're going to re-buy
they will dump or drop that and buy the more robust engineered warehouse. So it didn't, I just thought that's whe re t hey were at that time. Q. I see. So they did not ask your opinion regarding the u t ility or r elative utility of us i ng Tetradata as over against what is it? A. Crystal Reports. Q. Crystal Reports? A. No. Q. I see. Did you include in your recommendation to Little Rock for embedding the comprehensive assessment process, the development of school portfolios? Cobb Court Reporting P . O. Box 14 , Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(50 1 ) 590- 0975 - Cell (501) 490- 0926 - Facsimile e - mail: gloria . cobb@cobbcourtreporting.com 3 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 A. Ah, yes
well yes, uh-huh. Q. Would you explain what that recommendation really was about? 24 A. Yeah ah, whether or not a good school portfolio that would be in quote or not what this means to me is that for every school and across the whole district there would be data collected on at least four categories of data and ah, a comprehensive analysis that would describe who they have as students, how that population has changed any over time so they get to predict who, how the population was going to change in the future and see the achievement results were processed like curriculum instruction assessment, you know, how do we teach kids, how do we assess them ah, how will we know that they're meeting the standards and what are we going to do when they're not meeting the standard and production which would be basically questionnaires of students/parents and then what that data would do when they do the comprehensive data analysis to really understand where they are right now in any school or the district is then division the curriculum, the instruction, the assessment that are the process is basically this ah, that are put together to get the outcome would be developed around what we know from the data and monitored and measured over time and then a part of that is also once a school gets it, a vision then we'd have to figure out what is the leadership structure for us to within every school not just across the district, but Cobb Court Reporting P. O. Box 14 , Sweet Home, AR 72164 - 0004 Phone s : (501) 490 - 0066 - Off
( 501) 590 - 0975 - Cell (501) 490 - 0926 - Facsimile e -mail : gloria . cobb@ c obbcour t reporting.com 3 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 i) R \ G I '\ I {\ L \,J \ \ Ir\ 25 within every school to implement that vision what professional development is acquired ah, how we going to include parents, community and business so that we can make sure that kids can meet those standards and how we can evaluate on a continuous basis to know what we're doing is making the difference that we want. Q. All right. Do you know whether Little Rock ever developed school portfolios across the district? A. I know that Little Rock developed the district portfolio which is never totally done and it, well at least we got a start on that I should say
I think they did the demographics and the achievement piece for the district. As far as the schools are concerned I do not think they developed those or got them started. Q. Can you embed the comprehensive assessment process into the curriculum of the district without development of school portfolios? A. Yes. Q. How can you do that? A. You can do it by doing all of the pieces. I mean whether or not you do a portfolio I mean it's a framework for looking at your data to know what you need to do ah, for getting a v i sion, a plan and evaluating it on an ongoing basis. So if they could do those pieces it could be embedded without them. Q. In doing those pieces does that require a lot more time and Cobb Court Reporting P . o. Box 14 , Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(50 1 ) 590-0975 - Cell ( 501) 490- 0926 - Fac s imil e e -mail : gloria . cobb@c obbcou rtreporting . com n n D I I I I 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1JRIGII I/\L a lot more effort or is it basically the same? A. Oh, it would be basically the same. Q. So the portfolio were not absolutely necessary but just ah--how would you characterize them? 26 A. Well, the portfolio is just the framework and ah, what we were trying to do with it with schools which is what we have found in other districts is if we call it, if we're trying to develop a product it give more concreteness than saying oh, we're going to do comprehensive data analysis then we're going to get a vision, we're going to build a plan and figure out the professional development ah, school are more readily, usually more readi l y open to putting together the product so they have something to show for it but so that they have pieces all in one place that they can see the relationship of the pieces to one another. Q. Is that a correlation between the district portfolios and a data warehouse? A. Ah, absol--there's a relationship of the district portfolio and the data warehouse and that is the data warehouse would make the putting together of the data in a district portfolio easier and simplify it and ah, there's ah, this whole piece about everybody getting or using the same sources of information. Q. Have you any familiarity with the IT department of the Little Rock School District? Cobb Court Reporting P . o . Box 14 , Sweet Home , AR 72 164 - 0004 Phones : (50 1 ) 490 - 0066 - Off
(501) 590- 0975 - Cell ( 501) 490- 0926 - Facsimile e -mail : gloria.cobb@c obbcourtreporting . com II n n I I I I 3 5 6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 A. Not really. Q. Did you ever meet with anyone from the IT department of the Little Rock School District with respect to the development of a data warehouse or the development of school portfolios? A. Not that I remember. One or two of them could have been in one of the meetings or the training but I couldn't say for sure. Q. What has been your experience with IT departments with respect to dealing with program evaluation and assessment and embedding that into the curriculum? A. In general? Q. Yes. A. Ah, in general the IT guys always say that they can develop it themselves and specifically I have found that in most districts ah, if they start trying to develop a data warehouse by themselves two years down the road and millions of dollars later they're not done and--well, there's just a lot I could tell you but here's the bottom line is that while they're working on the warehouse they're not doing the work and if they're doing the work they can't work on the work that they're hired to do and if they're unless you hire lots of extra staff it usually doesn't work for the district to design their own warehouse plus now we have the tools out there, we have warehouses available that you can purchase that companies have devoted their development dollars to and it's much easier or Cobb Court Reporting P . o. Box 14, Sweet Home, AR 72164 - 0004 Phones : (50 1 ) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria.cobb@cobbcourtreporting . com JI 11 ll n I I I I 1 3 5 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 28 it's much better if districts buy a product and use their IT staff to make it work within the district and ah, they have the IT staff help with ah, the programming of specific reports or standard reports that the district or schools would want and need. Q. Did you enter into a contract with the Little Rock School District? A. Did I enter into a contract
yes. Q. What was the purpose of the contract and what were its terms? A. Ah, the purpose of the contract was to lay out the scope of work and to ah, purpose how much it would cost to do the scope of work and to get the agreement on both side that we would do this work for this much money. Q. Was the work that was recommended by you completed? A. No. Q. Can you tell me what proportion of the work remains incomplete? A. I just think we got started
so two / thirds to three/quarters of the work still needs to be done. Q. Can it be said that the comprehensive program assessment process cannot be or has not been embedded into the curriculum of the district? A. Absolutely. Q. Would you explain that? Cobb Court Reporting P. o. Box 14 , Sweet Home, AR 72164-0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting . com 1 2 3 4 5 6 7 8 9 q 10 n 11 12 n 13 14 q 15 I 16 17 n 18 19 20 21 22 23 24 25 A. Ah, I would say that the structures for gathering data across the district within every school and ah, have not 29 changed since I started working there ah, so I think the use of data probably hasn't changed much with the exception of maybe student achievement assessments that Dr. Roberts was working on with her staff and visited school teams. Q. That Dr. Roberts was working on outside of PRE? A. Yes. Q. So are you saying that there was a separate assessment process underway in the Little Rock School District to your observation? A. Well I think so but this would be assessment of the curriculum and you know like ongoing assessments that would hopefully predict the assessment to us at the end of the year and in that that's not unusual that an assessment tool would be setup and curriculum instruction and assessment but that's not of comprehensive data analysis system. Q. Oh, the one that she has is not comprehensive data ana l ysis? A. It would be an element of it. Q. All right. Does it appear to you that that is at odds with the court directive remedy that all the PRE be in one source and t hat PRE coordinates the program evaluation and assessment process? A. No, that--- cobb Court Reporting P . O. Box i4 , Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimil e e - mail : gloria . cobb@cobbcourtreporting .com q u q q I n d 3 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 30 HELLER: I object to the form of that question as well. WALKER: All right
I'll change the question. Q. Is there a utility in having all the persons engaged in program evaluation work together? A. Yes. Q. And cooperatively? A. Of course. Q. And without dissidence? A. Yes. Q. And conflict? A. Correct. WALKER: Just a moment. I'm going to take a moment, Doctor you can stay on the line ah, Chris I'm afraid to hang up so I'm going to step out for a moment and speak with Mrs. Springer and then I'll be right back to see if I have any more questions before I turn her over to you. HELLER: Okay. WALKER: Excuse me, Dr. Bernhardt. BERNHARDT: Okay. [A recess was taken at 6:48 p.m., proceedings resumed at 6:49 p.m., to-wit:] WALKER: Dr. I'm back on the phone
Mr. Heller I'm back on the phone
are you there? HELLER: Yeah, I'm here. Cobb court Reporting P. O. Box #4, Sweet Home, AR 72164- 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting . com 5 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 BERNHARDT: And I'm here. DIRECT EXAMINATION CON'T Questions by Mr. Walker Can't: 31 Q. Doctor were you aware that with respect to data collection and data presentation to the outside evaluators that there was some problem with the accuracy of the data being assembled and passed on to the evaluators? HELLER: I object to that question as well
assuming facts not in evidence. Q. Now you may still answer, doctor. A. Ah, I don't know other than what I've heard just recently about ah, dirty data if you will or bad data being sent to the external evaluator. I do know that when we were pulling the data together for the district portfolio or the district data source book ah, that a lot of that data was not clean and ah, yeah. Q. How do you know that? A. Ah, it's very easy to see dirty data in districts
it's not uncommon, in fact every single district has dirty data which means that if you don't have a system for establishing procedures for entering data into any system that you have or in multiple systems that you have if you don't monitor it's probably going to be corrupt in some way. So like for example if a principal went into the student information system and wrote or entered some information about a student and--well Cobb Court Reporting P. o. Box 14, Sweet Borne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 3 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 n, i p I,, ....... II1 .1_ /\, -...,,\ \_.1 I 32 like for your name they could, he could ah, the principal could put in John Walker, a teacher could write in J. Walker ah, another teacher could write in some information and code some data in there with J. Walker and somebody else Walker, J., and you would come up with four different kids. And so what it takes is a structure, a process, procedurally setup so that it's clear who's entering data and it's clear how they're going to enter the data and then it's monitored to make sure that it's done correctly. It's very, very easy to see when you start pulling together a portfolio of any sort it's very easy to see that there's dirty data even if you're not apart of the district. Usually what happens one of the first quarries that I do in a district might be looking at enrollment longitudinally or over time and, you know, I'll look at that data and go how could you have twice as many students in 2005 than you do this year? It's really is obvious when there's dirty data. If you haven't used the data ever I can guarantee it was dirty. Q. I see. And does that cause a flawed ultimate result? A. Absolutely, absolutely and there's a whole campaign nationally to get states to insure the quality of the data in all of their districts and schools because that is it's such an issue there's some data at the national level and at state levels we don't even know that have been reported for years, we don't know how inaccurate it is
it's very common. Cobb Court Reporting P. o. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 3 4 5 6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 33 Q. In your professional opinion what percentage of data is acceptable to be dirty and ah yes, is there a percentage or a threshold of acceptable dirty data? A. I don't know that anybody has ever put a percentage on there but ah, I just, if I could just beat around the bush to get to an answer to that. My latest book called Data Warehousing For Improving Teaching and Learning ah, I sent the first draft of the book out to twenty-five reviewers and these were reviewers in data warehousing companies, they are teachers, principals, administrators in the U.S., Canada, Australia. And I was physically asking that question, I made a statement in one chapter in the book about dirty data and how you've got to clean all the data and some of the people came back and asked the same question what is the percentage and they said they have in their experience ever never e xperienced a hundred percent clean data and so we should accept as a rule 95 percent clean data and being close enough with an error band. Q. All right. Do you discuss this in the book that you sent out for review? A. Yes. Q. What i s the title of that book? A. Data Warehousing To Improve Teaching and Learning. Q. Have you provided a copy of that to Dr. DeJarnette? A. No, I have not. Cobb Court Reporting P . O. Box 14 , Sweet Home, AR 72164 - 0004 Phones: (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Facsimile e-mail : gloria . cobb@cobbcourtreporting . com 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 34 Q. Would you ask your secretary to mail me a copy with a bill if you happen to talk to her in the next day or two? A. If--- Q. Well, that's all right
I can ask her that myself. A. Well, I can ask her this but ah, if you want the--it's in draft form. Q. That's all right, the draft form is fine. A. We're still working on it. Q. Well, it will be useful even though it's in draft form. A. Okay. Q. And I'm sure Mr. Heller would like to have a copy as well. A. Okay, it's still a mess but it's a lot of work to do on it yet. Q. Well, I'm more concerned about the area of your book that discusses this particular problem. A. Yeah, except you're not going to see it in the version that's available right now. It's ah, because that came from handwri tten notes, some of it came from handwritten notes from the reviewer that I'm incorporating into it right now. Here I'm in Hawaii and I'm looking at it right now, I'm just writing in what the reviewer are talking to me about so you would not see that. If fact what the reviewers had asked is that I add a chapter devoted entirely to the, on cleaning the data so that piece isn't available. Q. All right
well, that's fine. Well, let me see ah, it is a Cobb Court Reporting P . o. Box f4 , Sweet Home , AR 72164- 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (5 01) 490- 0926 - Facsimile e-mail : gloria .cobb@c obbcourtreport i ng.com 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 serious problem you're saying when data are overly dirty? A. Sure. 35 Q. That's just life though isn't it for most people whatever the subject is, if a thing is pretty dirty it shouldn't be used. That's a commonsense approach to the use of data and everything I would think wouldn't you? A. If the data is dirty yeah, I'd be cleaning it and setting up procedures and policies to make sure that it's entered correctly and then monitor it. Q. I see. Are you familiar with the questionnaires used in the process of the Little Rock PRE? A. Ah, I know that there are some, I personally have not seen the final version of what they're using or I don't remember them
I have seen them. Q. Did you assist in the development of the questionnaires? A. I assisted with the first version and after that my staff member worked with them directly. Q. What's the importance of questionnaires in the comprehensive assessment process? A. Questionnaires add that wind from students, staff and parents to find out how programs, schools, districts can improve basically and if there's--it can tell about climate, it can talk about leadership, how students are treated, how staff are treated, it covers usually or we would want them to cover whether or not there's vision, a plan in place, and ah, get the Cobb Court Reporting P . o. Box 14 , Sweet Home , AR 72 164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Fac simile e -mail: gloria.cobb@cobbcourtreporting . com 3 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 36 system setup for success. Q. All right. Can questionnaires sometimes be referred to as surveys? A. Yes. Q. All right. And can questionnaires or surveys or by whatever name those type things may be called can they be subjected to professional validation? A. Yes. Q. How would you go about doing that? A. Ah, well there's two things you'd probably want to assess on a questionnaire and one is validity and the other is reliability. And validity is about are we asking the right questions, are we asking the questions that will get to what it is you want to know and that would be content validity. And to get to content validity you basically have to administer the questionnaire to many people of background somewhere to ah, who you really want to use it with and ah, basically have them review it, take it ah, talk to them ah, that's one way of getting the content validity. And ah, reliability says that if you give the same instrument over and over you're bound to get the same results. In other words everybody, if one person reads the question they will read it in the same way so that it's clear, it's not misunderstood, it wouldn't vary over time. And so if they're looking for reliability when you give the questionnaire over time you be getting essentially the same Cobb Court Reporting P. o. Box 4, Sweet Borne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria . cobb@cobbcourtreporting.com 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 results. Q. Were you aware of a group called the Public Education Foundation in Little Rock? A. In Little Rock? Q. Yes. A. No. 37 Q. Did anyone inform you that there were program evaluations or assessments being undertaken by persons other than Dr. Roberts and Dr. DeJarnette? A. Ah, if they did it didn't register. Q. Is a formative assessment a part of program assessment and evaluation? A. When I think of a formative assessment I would be thinking of assessments that teachers would use in the classroom with kids and it would be apart of curriculum instruction and assessment, certainly program evaluation and research should have access to that data and should work along with them in a typical district. Q. I see. Should the information that's assembled with respect to formative assessments be included in a data warehouse? A. I certainly would add it into a data warehouse . Q. And would this require therefore that all of the persons who are involved in any kind of assessment or evaluation process would at least have to know what each other is doing? Cobb Court Reporting P. O. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria . cobb@cobbcourtreporting . com II q 1 2 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 38 A. Somebody would have to know what everybody is doing
yes. WALKER: doctor and I think you. I see. I have no more questions, CROSS-EXAMINATION Questions by Chris Heller: Q. Dr. Bernhardt this is Chris Heller, I'm a lawyer for the Little Rock School District and I've got a number of questions for you based on what you just told Mr. Walker. First of all, where are you located at the moment? A. I'm in Hilo, Hawaii. Q. Okay. And is there anyone there with you? A. No. Q. What did you do to prepare for this deposition? A. Ah, put my other work aside and I went back to look at ah, my proposal, the proposal for Little Rock School District and looked at some of what I did in the last year and a-half with Little Rock. Q. Anything else? A. Ah, I looked at some of Dr. DeJarnette's notes, most recent notes ah, but I, frankly I'm here to work so I haven't had a lot of time t o study them. Q. Anything else that you did to prepare for this deposition? A. Not that I can think of. Q. What notes do you have from Dr. DeJarnette? A. Ah, mostly there's a memo to you I think these were mostly Cobb Court Reporting P . O. Box #4, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria .cobb@cobbcourtreporting . com 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 things that were in report or ah, quarterly reports which I just skimmed and ah, there's a status, there are status reports--- Q. How did you get that--- 39 A. there's one that ah, I think they're probably attachments that you or Mr. Walker was referring to earlier about the status of the hiring of PRE staff and ah, mostly our proposals. Q. How did you get those notes? A. Ah, they were faxed to me. Q. When? A. Yesterday. Q. By whom? A. By my secretary when we--I was flying all day yesterday all day so I didn't talk to anybody they just were here when I got here--- Q. When did you first receive those documents--- A. except that I knew from my secretary, from a call with my secretary that there would be a fax with some notes. Q. When did you first receive those notes? A. When I came. Q. These were not notes that you already had in your office in California? A. Ah, somewhere like the proposal that we developed
the proposal, the contract that we had and that's all I was really interested in refreshing my memory on what we, our role was. Cobb Court Reporting p , o. Box 14, Sweet Borne, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell ( 501 ) 490- 0926 - Facsimile e -mail : gloria . cobb@cobbcourtreporting . com q Q Q Q II n n 3 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 40 Q. Well, are there some documents you have with you today that you reviewed to prepare for this deposition that you have not previously seen before you got to Hawaii? A. Oh, yeah. Q. What are those? A. Ah, well the status of the September 1st , 2006 where PRE is so I think you probably have a copy of, there was a compliance remedy memo to Beverly Griffin from Karen ah, there's a memo to Chris Heller about the needs for a data warehouse or a data base to look at demographic student learning perceptions and financial data, and school process data and ah, basically that one. HELLER: I want to make all of those documents an exhibit to your deposition so would you keep the entire set of documents that you got from your secretary that you reviewed for this deposition and either ah--- BERNHARDT: That's fine. HELLER: one of you ah--- BERNHARDT: She'll have better copies than I have. HELLER: We'll make arrangements so that they can be properly appended to your deposition. Q. When is the last time you spoke with Dr. DeJarnette? A. When did I speak with Dr. DeJarnette ah, last week because ah, to try to look at my calendar to see if I could do the Cobb Court Reporting P . o. Box #4, Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail: gloria . cobb@cobbcourtreporting . com 5 6 7 u 8 9 ll 10 n 11 12 n 13 14 15 16 17 18 19 20 21 22 23 24 25 deposition. Q. Did you call her or she called you? A. To see if I could come to Little Rock. Q. Pardon? A. Or if I would be willing to do such a thing. Q. And she called you to ask you to give a deposition? A. Ah, she, I don't know--yeah, I guess so
she was asking if I would be willing to or interested in or available to talk about the school portfolio and our proposal and what we had intended to do. Q. How many times have you talked to Dr. DeJarnette in the last three months? A. Ah, that was the first time--I think that was the first time in that three months. Q. Have you communicated with her by e-mail or any other means during those three months? A. Well e-mail occasionally but I'm on the road 170 days usually a year so, you know, I don't get into the small talk. Q. When is the last time you spoke with anyone else from the Little Rock School District? A. Probably when I was there last which according to my records would have been April. Q. Did I hear you correctly that you've only been to the Little Rock School District twice, August 2005 and April 2006? A. That's what I saw on my calendar when I quickly looked at Cobb Court Reporting P. o. Box 14, Sweet Borne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting . com a n u n n I 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 42 it this morning but that doesn't sound right, I think I've been there. Maybe it's only twice but I was trying to, it felt like I should have had a third date with them but I couldn't find it, so I didn't prepare for the--- Q. But did you discuss with Dr. DeJarnette any of the issues that Mr. Walker just questioned you about during your last conversation? A. During our last conversation we didn't have a lot of time because I had an appointment and ah, like within fifteen minutes of when we were talking and we were also looking at the calendar more than anything. Ah, I just asked what kind of information you would want in a deposition from me and how I could prepare and ah, she basically said that you would want to know about the school portfolio process and data and data warehousing and what I know about that. Q. So between April of 2006 and this latest conversation with Dr. DeJarnette you haven't had any substantive communication with anyone from LRSD? A. We would have I think ah, after April, after I was there in April we had a long conversation about our contract and the proposal and what was happening in the district as far as ah, our work and the trainings that we had proposed and mostly Brad Guise of my office had longer conversations with the PRE team about working with them on their questionnaires. Q. What work did you or--were your, your ah--- cobb Court Reporting P. o . Box #4 , Sweet Horne, AR 72164 - 0004 Phones: (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (50 1 ) 490 - 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting.com 2 3 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 43 A. Brad. Q. right, your ah, I don't know--what track of entity is education for the future? A. Education for the future is a not for profit organization that operates out of the California State University Foundation. Q. What's your position in that organization? A. I'm the executive director. Q. What work did you or EFF actually perform for the Little Rock School District? A. Ah, during the weeks of August 30th to September 2, 2005 I worked with PRE staff--it seems like there should be another date before that but I just couldn't find it today but I worked with PRE staff on their portfolio and ah, and did some training sorted or--- Q. What portfolio did you work on with PRE staff in August or September? A. District so there actually their district data as far or first book that was put together during the institute in Chico at the end of July of 2005. And then in August/September I did training with principals on looking at data on how to analyze and use data and what data and--- Q. How many principals were tracked? A. Umm, I don't know for sure. Q. Is it fair to say that you and your team at EFF trained Cobb Court Reporting P. o. Box #4, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com q q q n I II 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 three or four representatives of PRE in how to develop a district portfolio? A. Ah, yes. Q. And that was done in July of 2005? A. Yes, and actually were doing it. Q. And actually the port folio itself was developed in part during the work at EFF in July of 2005? A. Correct. 44 Q. And then you trained principals in August and September of 2005 in the use of data? A. Yes. Q. For what purpose? A. Ah, to understand how to analyze it and how to look at all data, to look at their system and to get them setup to create their own portfolios for their schools. Q. Okay. And then what was done in April of 2006? A. Ah, from what I remember ah, April would have been the time ah, I worked with PRE to set up the understanding about the company and this is the work that we're going to do and I believe it was in April that we did ah, that I worked with like Dr. Roberts and Dr. DeJarnette and a lot of people I can't name right now, administrators and some of the schools in doing an assessment of the district on a continuous improvement continuum and I also facilitated some training done. Q. Now you said to Mr. Walker that--well, first tell me what Cobb court Reporting P. 0. Box #4, Sweet Home, AR 72164-0004 Phones: ( 501) 490-0066 - Off
( 501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com I u n II II n 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 45 training did you facilitate then? A. A big picture of conference data analysis, what data are important, why that data is important and then how that could lead to student achievement improvements in the district. Q. Okay. Do you recall even by category of people who was trained in those things? A. Ah, I would not be the best one to tell you about the positions of the people. But ah, I think it was Olivine's staff, Karen's staff, some principals and I know there were other people from the district office but I couldn't tell you what their positions were. Q. What's the last date you have any direct knowledge of the content of the Little Rock School District's portfolio, the district's portfolio? A. The l ast time I saw it would have been probably September 2nd_ Q. And why did you see it on September 2nd? A. Because it was September--oh no, that wouldn't be true. It would be April, the last date I was there which would have been April 2006 because it was ah, I saw it because the PRE staff was really, really excited to show me the bound version of it and they wanted me to go through the data and I did an analysis of the data and ah, told them what I saw in the data, some of the things that I saw in the data. Q. Okay. And in fact you made certain data statements for Cobb Court Reporting P . O. Box #4, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(50 1 ) 590- 0975 - Cell (501) 490 - 0926 - Fa c simile e -mail : gloria.cobb@cobbcourtreporti n g . com 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 example such as ah, observations about the composition of special education and discipline and things like that? A. Yeah. Q. And you said you have not seen LRSD district portfolio since April of 2006? A. No. 46 Q. You don't know what the contents of that district portfolio are today? A. No, I don't. I understood that they were updating it but I don't know if it did happen or I haven't seen it. Q. And is the work you did or you do probably reviewing the EFF ah, reflected in written agreements contracts? A. Ah, yes. Q. For the work we just talked about how was EFF compensated? A. Ah, a daily rate and ah, is that what you mean? Q. Pardon? A. Are you asking about ah--- Q. What basis, is it a proposal for a particular project or do you recall how much EFF--- A. It was a proposal that started I believe September 2005-- I'm not looking at the first date ah, 2005 through the spring of 2006 and it was for training, analyzing data ah, daily rate of air, hotels, per diems, ground transportation and you asked about ah, the second part of the question? Q. Oh, just whether, I think you've answered it, whether there Cobb Court Reporting P. o. Box 14, Sweet Borne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 3 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 47 was a specific fee for a specific proposal or whether it's done on some other basis such as a daily rate? A. It was a daily rate and we considered these proposals ah, that's what we think is going to happen but we still only charge what really did happen. Q. Is there any work, other work that you've proposed but your proposal was not accepted by the district? A. No, this proposal was accepted, the one that I'm referring to was accepted. Q. Okay. So are you saying that there was nothing else that you proposed to do that the district declined to accept? A. No. We work together to create it, talked about what they wanted to have happen in the end and then work their way backwards into thinking about what it would take to get there and from what I understood from our conversations about what was possible in other words could be pull out teams of teachers or would we just have to work with principals ah, I created the proposal and sent it to them and maybe we refined it at the time back and forth especially dates--- WALKER: Chris let me interrupt you for a minute. Ah, I think it would be useful for us to have maybe a court reporter break for maybe five minutes. HELLER: Okay. WALKER: Thank you. That you, doctor. HELLER: Dr. Bernhardt if you could just--- cobb court Reporting P . O. Box t4, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (50 1 ) 490- 0926 - Facsimil e e -mail: glori a . cobb@cobbcourtre porting . com 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 i
. f Jr\'~- 48 WALKER: You can just stay on the line, you can just leave the phone up doctor--- HELLER: Okay. WALKER: And go about your business and we'll come back on after Ms. Cobb has a few minutes to take a breather. BERNHARDT: Okay. WALKER: Maybe you could use one too Doctor. BERNHARDT: Yes, I could. WALKER: All right then. BERNHARDT: Okay, I'll be back. [A recess was taken at 7:24 p.m., proceedings resumed a t 7:29 p.m., to-wit:] WALKER: We're back in. Dr. Bernhardt are you there? you a r e. BERNHARDT: Yes, I am. WALKER: All right. We're ready to resume if REPORTER: Thank you. HELLER: Okay. CROSS-EXAMINATION CON'T Questions by Mr. Heller Can't: Q. Ah, is there any work Dr. Bernhardt that you or EFF performed for the Little Rock School District that we haven't discussed? Cobb Court Reporting P. o. BOK 14, Sweet Borne, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e-mail : gl oria . cobb@cobbcourtreporting.com 5 6 7 8 9 10 Q 11 12 II 13 14 n 15 n 16 17 18 19 20 21 22 23 24 25 A. No
I don't think so. Q. All right
so if there was an answer I didn't hear that. A. Oh no, I said no, not to my knowledge. Q. How many school districts are you familiar with Dr. Bernhardt across the country that have in place school portfolios right now? 49 A. I would have to take a guess at that because a lot of my training would be in sometimes whole states or regions where there are multiple districts that come to it and then and there might be three hundred people in the room and I don't know exactly what they implemented ultimately but I would say there's, well at least two hundred. There are districts that are also putting together portfolios based on publications that I maybe haven't met since in national conferences people come up and tell me that they're doing it and I wouldn't know any other way. Q. How many districts that you've worked with that have planning, research and evaluation department of the size and quality of the one in LRSD? A. Ah, I don't even know how to estimate this
let me just give you some background. Q. Okay. A. In 2006 I worked in 21 states and was on the road 151 days
now that's lower than the previous fifteen years. And a lot of these might not be on an ongoing basis
I see lots of districts Cobb Court Reporting P . o. Box #4 , Swee t Borne , AR 7216 4- 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : glor ia .cobb@cobbcourtreporting . com 1 2 3 4 5 6 7 I 8 9 JI 10 JI 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 so and I've never counted but there are districts, large districts like Little Rock used to have program evaluation and research departments or bigger ones maybe ten years ago but now they're probably dwindling across the country it depends on where you are and what state you're in I'd say the size of the program evaluation district department and whether or not assessment is included in it or separate. Q. Is it fair to say in your experience that most of the districts which might have a planning research and evaluation department with the, of the size and resources of the one in Little Rock would be districts larger than LRSD? A. What's the enrollment of LRSD right now? Q. About twenty six thousand students. A. Yeah, about thirty thousand ah, again that just depends on state, region, but I would say it would be average or maybe even bigger than some. Q. Are there any other school districts in Arkansas to your knowledge implementing school portfolios? A. At this point in time to my knowledge, no. I have worked in Arkansas before and did overviews of school portfolios but it wasn't a long-term engagement to know if they ever did anything with it. Q. Okay. Do you know whether any other school districts in Arkansas have data warehouses? A. Ah, I don't know. Cobb Court Reporting P. o. Box f4 , Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(50 1 ) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria. cobb@cobbcourtreporting . com 10 11 12 13 14 11 15 11 16 17 18 19 n 20 21 22 23 24 25 nR10111A i 1..j i 11...J 1 L 51 Q. Is it common for districts around the country to send their data on a regular basis, by that I mean every week or every other week to some central location within a state education department for storage? A. For storage? Q. Well, for--- A. No. Q. in other words the, whatever repository of the data would be at the state level rather than at the district level? A. No. And ah, where most districts the size of Little Rock are going right now is to data warehousing that would upload data every night from the student information system. Q. Can you give me an example of another district the size of Little Rock that would be doing that ah, be engaged in that process at the moment? A. Yeah, one in Grand Juncti on, Colorado
San Jose Unified in San Jose, California ah--- Q. How many students in San Jose--- WALKER: Well, let he finish the answer Chris, she was giv ing--- HELLER: Okay. WALKER: ah, she's mentioned two, Grand Junction and San Jose. A. I know that there are lots of districts in North Carolina and South Carolina that are--oh, there's lots of district that Cobb Court Reporting P . o. Box #4 , Sweet Horne, AR 72164-0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490-0926 - Fac s imile e - mail : gloria . cobb@cobbcourtreporting .com 3 5 7 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 rJ R I G ! i\J A L 52 are, these are the ones, I'm just mentioning ones that I have worked with. Q. About how large is the San Jose, California? A . Ah , 3 2 , 5 0 0 . Q. Geographically as far as you know what would be the closest district to Little Rock that is ah, has school portfolios in place? A. That have school portfolios in place? Ah, there'd be lots of districts in Missouri. Q. Okay. A. In fact I've been working in the southeast corner of Missouri the last couple of years
I'll be there next month. WALKER: Is that what they call Little Dixie? BERNHARDT: That's not the way I've heard it described. WALKER: Well, they have, it's a little name up there up near Blytheville. BERNHARDT: Oh, this is by Cape Girardeau. WALKER: Yeah ah. BERNHARDT: They call below Cape Girardeau the boot hill. WALKER: No, that's over closer to Fayetteville. BERNHARDT: Oh, that's not it
oh no, it's different
yeah. HELLER: I think she's right and you're wrong Cobb Court Reporting P . o. Box f4, Sweet Home , AR 72164 - 0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e-ma il : gloria . cobb@cobbcourtreporting . com 3 5 6 7 u 8 p 9 10 JI 11 12 i 13 14 15 16 17 18 19 20 21 22 23 24 25 53 John, Cape Girardeau is right up on the Mississippi River. WALKER: Okay, well I'm wrong
I'm glad to be. Q. Dr. Bernhardt have you ah--- A. Oh, Houston--oh, that's not close to Little Rock. Go ahead. Q. Did you have the opportunity to review any of the program evaluations that were done for the Little Rock School District by Dr. Ross and Dr. Katiyar? A. No, I did not. Q. You know Dr. Katiyar? A. Personally, no. Q. Does EFF perform program evaluations of academic programs? A. We do but what we--we do but we like to do is work with districts to do their comprehensive data analysis and build their capacity to automate evaluations of different programs. Q. Now we've, we've--do you see any distinction between the t erm evaluation and the term assessment? A. Yes. Q. Okay, what's the difference between those two things? A. Ah, assessment to me means more like doing assessments of student learning or that's what comes to my mind when we talk about assessment. Evaluation, program evaluation would be to me talking about or doing the work of looking at the effectiveness of different programs and if they're meeting their intention. Cobb Court Reporting P. o . Box 14, Sweet Home , AR 72164 - 0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (501) 490-0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting . com 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Q. So is there a different then between what I'd call a formal program evaluation and the general use of data on a regular basis in a school district to improve student learning? A. Would there be a distinction, is that your question? Q. Yes. A. I would make a distinction. I would hope that the comprehensive data analysis work would include the evaluation so that there's ongoing ah, there are ongoing measurement of quality. There's ah, there's also room for program evaluation to always be done by a third party, somebody that's objective and looking in instead of--looking from the outside in instead of the inside out. Q. But it's possible to do a program e valuation for e xample without having school portfolios in place? A. Yeah, the term school portfolio is just the framework basically and ah, yeah, you could do an evaluation without any, without schools ever looking at the data. Somebody could do an evaluation of someone without the schools having looked at it. The difference with the por tfolio framework is that schools are looking at, building the story of the school with data and looking at adjustments
in a sense they're doing program evaluation when they do that work themselves. Q. Well, that was going to be the other side of the point. You can have valuable information from having some type of school portfolios in place and without ever doing a formal Cobb Court Reporting P. o. Box i4, Sweet Home, AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590 - 0975 - Cell (501) 490 - 0926 - Facsimile e-mail : gloria . cobb@cobbcourtreporting . com 5 6 7 8 9 10 II 11 12 i 13 14 15 16 17 18 19 20 21 22 23 24 25 program evaluation of the way a program operates across the district? 55 A. Yes, as long as everybody knows how to do it, why they're doing it, and are really looking at all the data
yes. Q. Right. So program evaluations can exist without portfolios and portfolios can provide valuable information that doesn't necessarily amount to what you would call a program evaluation as the program is implemented in schools across the district? A. Yes, except to that last piece
I'm not quite sure if I heard that correctly. Q. Well, actually I'm talking about something that's in our compliance remedy where our program evaluations are supposed to look at the programs as they're being implemented in schools across the district. So--- A. You can look at it from inside or you can look at it--- Q. that's why I, I, I, real l y my point is and tell me if you agree that ah, school portfolios don't necessarily do that for you
they don't do that at all
they don't provide you program evaluation of for example early childhood on a district-wide basis? A. They would provide--if, if done well a portfolio would have al l the data necessary to do the evaluation. Q. And so would a data warehouse? A. Absolutely. Q. Okay. So ah--- Cobb court Reporting P. o. Box 4 , Sweet Rome, AR 72164-0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting . com 3 6 7 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 A. There's an element of somebody looking, analyzing and knowing what they're looking for and what they're looking at that would be the piece missing. 56 Q. Right. And you said several times that school portfolios is just a framework and really a school portfolio is a type of school improvement plan is it not? A. I'd say it's a process and it's a product and it's a framework
yeah. Q. And there are other types of school improvement plans that might use data to improve student achievement that might not follow exactly your outline of the school portfolio? A. That's correct. Although a good school improvement process should have a lot of the same elements of a vision, plan, a profess i onal development leadership to prepare you--- Q. Would you say that the requirements in Arkansas for the development of improvement plans at each school? A. I'm sor ry
I didn't hear all of that. Q. Are you familiar with the requirement in Arkansas for t he development of school improvement plans at each school? A. Ah, yes. Q. Do you know what type of information is required of each school as a part of that requirement? A. Ah, I ' d have to say yes, I do and I'd also have to say I would want to look at it again. Q. Have you heard the acronym ACSIP, A-C-S-I-P, plan? Cobb Court Reporting P. o. Box 14, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590-0975 - Cell (501) 490- 0926 - Facsimil e e-mail : gloria . cobb@cobbcourtreporting . com 3 5 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 A. Yes. Q. Do you know what those are? A. Ah, yes, generally. Q. What are they? 57 A. Aren't those the plans that the state are requiring of the schools? Q. Right. And do you know whether or not those plans are required to contain the same type of information that you would recommend in a school portfolio? A. Yes. What, in fact I would say that school portfolio would go above and beyond that because the school portfolio are the framework of what we're talking about for data analysis is usually a lot more comprehensive than what states minimally require. Q. Do you know specifically what's required in Arkansas? A. Not right now. Q. Okay. Let's talk generally then. These types of schools improvement plans the states require would contain at least some of the information that you would like to see in a school portfolio? A. Right. Q. Most typically student achievement, demographic information and maybe some perception of that? A. Yes
correct. Q. And that information itself if properly analyzed can be Cobb Court Reporting P . o . Box 14 , Sweet Borne, AR 72164 - 0004 Phones : (501 ) 490-0066 - Off
(501) 590- 0975 - Cell (5 01) 490 - 0926 - Facsimil e e-mail: gloria.c6bb@cobbcourtrepor ting .com 5 6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 useful in improving student achievement? A. Yes. A I Q. And are you familiar with the acronym SOAR in the Little Rock School District, S-O-A-R, student online assessment reports? A. Yes. 58 Q. Is that what you were referring to when you talked about Dr. Roberts being involved in other types of formative stuff? A. If that was what was just developed this last year because I know she was working with the pre-helm committee this last year. Q. And let me ask you if that process is to do quarterly assessments that are aligned with the state benchmarking plan to determine individual student problem, teacher problems, school or district-wide problems ah, is that the type of thing you would recommend that school districts do? A. Absolutely
yes. Q. And isn't that a good example of using data to improve student achievement? A. It is--yes. It is not a com--that is an example of using data to improve student achievement. Q. And do you know whether or not that SOAR is currently being implemented in every school of the Little Rock School District? A. No, I do not. Q. Have you provided any written reports to the Little Rock Cobb Court Reporting P. o. Box i4 , Sweet Home, AR 72164-0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490-0926 - Facsimile e - mail : gloria . cobb@cobbcourtreporting.com 3 7 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 r-,,. I 0" I I ' ) l ~ l l J l l J r/\\, .._'. _ 59 School District concerning any of your recommendations? A. To like the board? Q. To anyone? A. Ah, not to my knowledge other than the proposal at the very beginning. Q. But for example regarding any process necessary to embed an assessment process are any other recommendations you may have made none of those would be written recommendations? A. Well, in the proposal on setting it up ah, essentially that's what our proposal or contract is all about is, you know, what I though after discussing with PRE what it would take to embed a comprehensive assessment process. Q. Okay. How much time did it take to develop a school portfolio? A. Ah, about a year . Usually it all depends upon how available the data are. Q. And what process should be followed to develop a school portfolio? A. Ah, I'd say the first--well, if you're thinking from my prospective or from a school prospective ah, let's me just say then ah, the first thing I do is collect the data to find out where are we right now and ah, that's the piece that takes the longest. There are a few things that can be done simultaneously ah, not everybody at a school or a district needs to be going out collecting the data
you just need a Cobb Court Reporting P. o. Box 14, Sweet Home, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting.com 3 5 6 7 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 60 '\ small team working on that. At the same time you could be creating this vision at a school and the district if necessary and ah, and then development of a plan that's based on the vision and the data. Q. And so would the school portfolio be different for each school in the district? A. It should be because the data are different, they have different kids and the vision--the structure should be the same, the contents would vary depending upon each school
who they have as students, who they have as teachers, how they line up the teachers to students, their vision, the plan that they put together, how they work together. Q. So isn't it possible that some districts may be concerned about some issues more than other schools, I mean some schools may be concerned about particular issues than other schools and that may result in a difference even in the types of data they maintain in their school portfolio? A. That's true although there are guidelines for what data we ought to be collecting at least on an overview look and then as there are issues that come up definitely the data ought to be getting deeper and deeper, dig deeper into that forest or into what's going on
find out what's really happening to get to the root cause. Q. I asked you before and I think you said you or your company had done program evaluations, have you published any work on Cobb Court Reporting P. o. Box 14, Sweet Home, AR 72164-0004 Phones : ( 501) 490-0066 - Off
( 501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail: gloria.cobb@cobbcourtreporting .com 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 61 the subject of formal program evaluation? A. Ah, I just covered it a little bit in my books in terms of if you do this comprehensive data analysis work you would be setup for program evaluation needs assessment. Q. Right. And so is that what you say the relationship is between the comprehensive assessment work and a program evaluation that the comprehensive assessment work that's done puts you in a position to do a program evaluation? A. Absolutely. Q. And then part of your work with LRSD was not to perform any program evaluations was it? A. No, but in a ah, by setting up a comprehensive data analysis they should be setup to do their own program eval uation
yes. WALKER: Thank you, Chris. Q. The, you worked with PRE and other I think to develop questionnaires for the Little Rock School District
is that correct? A. Yes, but mostly my staff, I know Brad Guise worked with them on the content. Q. And you talked with Mr. Walker about the content validity and reliability. Ah, have those questionnaires been tested for content validity? A. The ones that Little Rock revised? Q. Well, let's say for example the ones that are used as Cobb Court Reporting P . o . Box 14 , Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Facs imile e -mail : gloria . cobb@cobbcourt reporting . com 2 3 5 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 examples in your book? A. Yes, they are
they have been. Q. And how was that determined? 62 A. Well, we took about ten thousand students, for example the student questionnaire administered the questionnaires over time to get the reliability and validity ah, interviewed students. Q. So was that a determination that was made by EFF rather than some outside evaluation of the content validity? A. Right. Q. Is the same true with respect to reliability? A. Yes. Q. Now you work with I guess government entities such as school districts more than any other type of entity don't you? A. Yes, also I would say regional offices of that in the last few years
regional offices where districts come to come together, yeah. I would say there would be probably our biggest contractor. Q. Have you seen a problem in districts maybe changing administrators or changing boards or the election changing the course of a school district with respect to how it approaches its curricular or evaluation or assessment type issues? A. Yes. Q. You understand it's fairly difficult for a government or entity to change and therefore the direction the entity goes will change as well? Cobb Court Reporting P . O. Box 14 , Sweet Home, AR 72164 - 0004 Phones : (50 1) 490 - 0066 - Off
(501) 590-0975 - Cell (50 1 ) 490- 0926 - Facsimile e-mail: gloria .cobb@ c obbcourtreporting . com II - 5 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 63 A. Absolutely and that's one of the reasons I have a hard time telling you how many districts in our country would be using portfolios right now because I might be working with an administrative staff that is very gung-ho portfolio, school portfolio, district portfolios and then the superintendent changes and next year there is no portfolio or it might be just the opposite. leaders change. Yes, we see all kinds of changes happen when the Q. In Little Rock I mean you made the statement that generally assessment can't be embedded without the highest level of support
do you know whether in Little Rock the board supports the process of program assessment? WALKER: Now let me object to that for the record of course. We're talking about two different boards, one that was Mr. Heller's board and one that's not Mr. Heller's board. So the board we are talking about has to be identified. Q. Well, at any point in time Dr. Bernhardt do you have any reason to believe that the Little Rock School District Board of Directors doesn't support program assessments? A. No. Q. Any reason to believe that the superintendent doesn't believe in program assessments? A. No. Q. You testified that you made recommendations to LRSD about deeply embedding a process, is that correct? Cobb Court Reporting P . O. Box 14 , Swee t Home , AR 7216 4-0004 Phones : (501) 490 - 0066 - Off
(50 1 ) 590- 0975 - Cell (501) 490 - 0926 - Facs i mile e -mail : gloria.cobb@cobbcour t r e porting . com 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 64 A. Yes. Q. I think we've established that those were except as to what's contained in your proposal the recommendations wouldn't be written? A. Ah, I would say not but this is the clearest ah, the clearest documentation at the time. Q. And what do you say, I take it that you have your proposal in front of you, what do you say there that you propose to do for LRSD? A. Ah, I have it right in front of me. That I was going to do some training with school teams, district team on the portfolio and comprehensive data analysis and ah, we were going to work with each school in the district on questionnaire services and ah, a lot of it the training and follow-up training on the first part of the change was of course understanding how to look at data, what data, would that data would lead to continuous improvement ah, other parts of the training would be about getting a vision and plan and ah, and then just having the follow-up sessions to monitor and continue with the training with the school team. Q. Is there anything else in that proposal
any other substantive aspects of that proposal? A. When I look at it those are the biggest pieces of work for me to do. Q. Okay. When you made that proposal were you aware of the Cobb Court Reporting P . o. Box 14, Sweet Home, AR 72 164- 0004 Phones : ( 501) 490 - 0066 - Off
( 501) 590- 0975 - Cell (501) 490 - 0926 - Facsimile e - mail : gloria . cobb@ c obbcourtreporting . com 3 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 I I ('1Dif'.:'f f !\ i t requirements in the compliance remedy concerning a comprehensive program assessment process? A. Yes. Q. And so that's the only writing we have that reflects something you said to the district about what saw would be necessary for deeply embedding an assessment process? A. I believe so. 65 Q. Now tell us about certain things that you felt were needed by LRSD ah, a way to pull data together, tools for quick analysis, ways for teachers, administrators and others to get data from the same source and you said this was based, verbally this assessment of need based firmly on what you were told. Can you tell me who told you there was a need for any of those things? A. That there was a need for any of those things? Q. Right, I mean you believe that--- A. Or a need for the tools? Q. You know whether or not LRSD currently has all of those things in place? A. In ah, in working with the staff at PRE as we talked about putting together--well, one of the proposals and I think this was done in person that's why there's not a lot in writing, as we're talking about how we would go about getting all of the data together this is a very common discussion as I work with districts. Ah, one approach would be . to have schools go out Cobb Court Reporting P. o. Box #4, Sweet Horne, AR 72164-0004 Phones: (501) 490-0066 - Off
(501) 590-0975 - Cell (501) 490-0926 - Facsimile e-mail : gloria.cobb@cobbcourtreporting.com 3 5 6 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 66 and try to pull the data together or to have the district pull all the data together and in ah, provide you know the district do it first and then provide some data for the schools what they can pull together which I would highly recommend because it's much better to have few people pulling it, trying to pull together all of the data and then supporting the schools that way than having schools try to go get the same information from one person or four people. And in those conversations we talked about okay, what about your demographics, your student information system rollup in other words the student information system is the transactional database that would be in every single school and we would want the school's data to roll up into one district student information system and so we were already talking, we were already talking this talk with the districts and what about perceptual data and what about student achievement data it's out of the same location is that accurate is the demographics on the student achievement in the student achievement files accurate ah, so that we could do desegregations easily and--- Q. You know--- A. and--- Q. I'm sorry, I didn't mean to interrupt you. A. ah, you know what kind of program evaluations, we talked all these things all the time and they, there were data pieces all over the place or in different l ocations and that's how we Cobb court Reporting P. o. Box 14, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell ( 501) 490- 0926 - Facsimile e-mail : gloria. c obb@cobbcourtreporting .com 3 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 talked about warehousing. WALKER: Now let me interrupt again. I think we're going to loose the court reporter because she's gone beyond the time that we promised, we had said was two hours. How much longer do you have Chris? 67 HELLER: Probably no more than 30 minutes. WALKER: Doctor, can you be available for thirty minutes tomorrow? BERNHARDT: No, I cannot I am--unless what time are we talking here? WALKER: Late at night, you know, any time at your convenience. BERNHARDT: I'm working with a group of schools tomorrow until about 3 o'clock--- HELLER: There's a school board meeting here, John. BERNHARDT: I wouldn't be available 3 p.m., so--- WALKER: Pardon? HELLER: 3 o'clock Hawaii time puts us into a school board meeting here tomorrow night. BERNHARDT: But I wouldn't be available until 4 o'clock tomorrow. WALKER: I think we're going to--Chris can you try to compress, I mean, you know I mean this is a trial for Cobb Court Reporting P . o. Box 14, Sweet Borne , AR 72164 - 0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (50 1 ) 490 - 0926 - Facsimile e-mail : gloria . cobb@cobbcourtre porting . com 3 5 6 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 68 all practical purposes this is a trial deposition so could you, I mean do what you want to do but ah, I think Ms. Cobb is going to have to go
she's out of it. things as well. She's had a long day and other HELLER: Well, let me just ask Dr. Bernhardt what we can do, I mean what would be, I mean is there a convenient time we can finish this up for her before we get to trial? BERNHARDT: And when is the trial? HELLER: The 20th WALKER: Well, you're not going to be a part of his case Doctor, so it could be, it will be before the 22 nd--HELLER: Well, I need to know, I need to know what she says. deposition. WALKER: We may not choose to use Dr. Bernhardt. HELLER: I still want to complete the WALKER: Well, I understand but you know, she has at least until the 26th or 21th--2sth or 26th . BERNHARDT: The earliest I could do this would be Saturday / Sunday or Monday. WALKER: Well, the judge is, we think we can do it, I think I'm available on Saturday or Sunday. the 14 th . HELLER: I can do it Saturday the 13th or Sunday Cobb Court Reporting P. o. Box f4 , Sweet Borne, AR 72164 - 0004 Phones : (501) 490- 0066 - Off
(501) 590- 0975 - Cell (501) 490- 0926 - Facsimile e -mail : gl ori a . cobb@cobbcourtreporting . com 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 I 69 BERNHARDT: I have to move locations on Saturday ah, so it, I'd have to do it either before or after. WALKER: Well Chris I mean I think that you've gotten to the point of where--well, I don't know why don't you take another 10 minutes and see if you got some more real questions. HELLER: Well, we're not, I'm just barely getting into some of the stuff that you asked John, I mean it's going to be a solid 30 minutes, I can promise you that. I don't want to mislead you. WALKER: Well, we're looking at maybe an hour and a-half then if you say a solid 30 minutes, so why don't we try to resume it for Sunday the 13th? t he v ictim. different? BERNHARDT: Saturday the 13th? WALKER: This coming Sunday. HELLER: Sunday is the 14th . WALKER: The 14th HELLER: What time? WALKER: Dr. Bernhardt sets t he time since she's Dr. Bernhardt? BERNHARDT So ah, are you talking four hours WALKER: Yes, ma' am. BERNHARDT: Ah, on Sunday I could do it at 9 a.m., I could do it anytime on Sunday. Cobb court Reporting P. O. Box #4, Sweet Horne , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590 - 0975 - Cell (501) 490 - 0926 - Facsimile e - mail: gloria . cobb@cobbcourtreporting . com I I I I I 1 3 4 5 7 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 70 HELLER: Would that be best 9 a.m., your time, I suppose it would have to be. So 1 o'clock our time, John. WALKER: 1 o'clock our time will be fine. BERNHARDT: Okay. HELLER: Can we arrange this the same way
I don't how you two got this--- WALKER: Yes, we'll arrange it the same way. HELLER: All right, great
I'll be here. WALKER: That will give you an advantage so you have some better, some more questions. HELLER: It will give us all an advantage because I can ah, hopefully streamline things a little bit. Bernhardt. WALKER: Thank you very much
than you Dr. BERNHARDT: Okay. HELLER: All right
thank you Dr. Bernhardt. BERNHARDT: You're welcome. HELLER: Thanks John. WALKER: Bye-bye. BERNHARDT: Bye. [A recess was taken at 8:11 p.m., January 10, 2007 to resume at 1:00 p.m., on Sunday, January 14, 2007.J * * * * * Cobb Court Reporting P . 0 . Box 14, Sweet Home , AR 72164 - 0004 Phones : (501) 490 - 0066 - Off
(501) 590- 0975 - Cell (501) 490 - 0926 - Fa c simile e - mail : gloria . cobb@cobbcourtreporting . com 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 I 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 I I I 71 REPORTER'S CERTIFICATE STATE OF ARKANSAS ss. 429-84-1622 COUNTY OF PULASKI I, Gloria Y. Cobb, A Certified Court Reporter and Notary Public in and for the aforesaid County and state, do hereby certify that the witness, DR. VICTORIA BERNHARDT, was duly sworn by me prior to the taking of testimony as to the truth of the matters attested to and contained therein
that the testimony of said witness was taken by me, a voice writer, and was thereafter reduced to typewritten form by me or under my direction and supervision
that the foregoing transcript is a true and accurate record of the testimony given to the best of my understanding and ability. I FURTHER CERTIFY that I am neither counsel for, related to, nor employed by any of the parties to the action in which this proceeding was taken
and, further, that I am not a relative or employee of any attorney or counsel employed by the parties hereto, nor financially interested, or otherwise, in the outcome of this action
and that I have no contact with the parties, attorneys, or persons with an interest in the action that affects or has a substantial tendency to affect impart i ality, that requires me to relinquish control of an original deposition transcript or copies of the transcript before it is certified and delivered to the custodial attorney, or that requires me to prov ide any service not available to all parties to the act. 1 ./ _j/. /. 1 My Commi ssion Expires: January 16, 2007 , $!') \, (, ," J 1-1 /lli-&u c~ . Gloria Y. p::obb, Notary Pul::!lib
This project was supported in part by a Digitizing Hidden Special Collections and Archives project grant from The Andrew W. Mellon Foundation and Council on Library and Information Resources.