Hi Measures Team!  Just wanted to provide an update after receiving some information from Hope Street Group.  I will be posting this as a blog and a discussion thread so don't think you are confused if you end up reading the same message twice!  We have noticed that blogs posted do not show up under recent content on the measures home page and can be missed while navigating the site, whereas discussion threads show notification all over the site! 

We received notification last night that we are to present the recommendations Monday evening on a conference call with HSG and the other team leaders.  We are excited to share all of the great team thinking and research, but we have a ways to go in completing the templates and we would really appreciate feedback about whether you think the recommendations are comprehensive and inclusive.  PLEASE read over the recommendations posted by Sherman two weeks ago and those posted this week.  We are looking for comments, additions to the discussion that led up to the recommendations, and input about the research, innovative practices, policy implications etc.  There are sections of the template that can be filled in directly.  On our last message we mentioned downloading the attachment, editing and then reposting.  I don't think that is necessary- I think if you click on the edit document tab under actions you can edit the document and add any pieces you would like to contribute and then push publish and it will automatically repost the edited version.  In this manner if we all contribute we should be able to complete the templates with no problem.  Again, the recommendations are due Monday at 5:00 EST so we are definitely down to the wire!  I know how incredibly busy everyone is, but please take a bit of time to read through the recommendations and add your input.

Thanks so much- Darcy and Dina

Good Afternoon Measures Team!

Hope everyone is doing well.  Well, this is it... crunch time!  This is the week that we get recommendations in to Hope Street Group.  First, we'd like to thank Sherman VERY much for the recommendations posted.  I have been out of town at a family reunion with very sporadic internet service and was unable to get feedback through (don't ever under-appreciate hot water or internet on demand).  Thanks so much for your work on that!  HSG said not to worry about breaking apart- just leave as is.  For the rest of us please post feedback about Sherman's draft and help fill in the portions of the template where he asked for additional references.  You can add as a separate blog or download the document and paste in additions and then repost for the group to continue working on.

We will be posting some additional ROUGH drafts of recommendations today and tomorrow.  These are general/umbrella recommendations based on the discussions held by the team and the research presented.  PLEASE post feedback and add to the documents attached and then repost them.  We want to be inclusive of everyone's view points and reflect all of the reading and research that's been done up to this point.  We need full team participation in order to meet the deadline and get these recommendations completed and in to HSG over the next week.  Thanks so much for your efforts on this... we are in the home stretch!

As always please don't hesitate to call or email if you have any questions, concerns, or comments.  We are looking forward to viewing your input and seeing these recommendations move from draft versions to actual versions!

Thanks- Darcy and Dina

Hello Measures Team- We are getting closer to the September 14th date for getting recommendations in to Hope Street Group.  This blog is a discussion about getting to the next step.  I’ll apologize up front for the length, but we have a great team with members at all different points in the process (welcome those that joined today)!  So…this is our attempt to get everyone on the same page and move ahead!

We know that what is needed is a credible system for measuring the quality of teachers' work, a comprehensive evaluation system. We all seem to agree that this system will need to address the quality of instruction as well as results of student achievement.  How can we get our arms around this idea and make some contributions as a team? We've been working in sub-teams to discuss and research these questions:

1. What student-level measures (test scores, other student work) would be useful? Why?? Are there any innovative practices in this area?

2. What classroom-level measures (observations, etc.) would be useful? Why?? Are there any innovative practices in this area?

3. What school-level data would be useful? Why?? Are there any innovative practices in this area?

There has been a great deal of discussion and research shared and I will summarize some of it in a bit (please add to the blog anything you would like included).  The next step will be for us to take this discussion about the problems and possible solutions for measuring at the student, classroom, school levels, and craft policy recommendations to be shared with stakeholders and decision makers.  We do not have to solve every foreseeable issue or provide all of the minute details, but this is our opportunity to share what we think should be included in the process and guidelines so that when the decision makers get down to the details within a context of a specific situation our recommendations set the backdrop.  See the template attachment for an example (Policy Recommendations Template ).  Remember our audience- many of the people these recommendations will be going to are not familiar with the problems or the important things to keep in mind while setting policy- the very issues that you are all experts in.  Hope Street Group wants to “be able to present policy recommendations and say educators, researchers, other professionals (us) agree that this is where we need to be going.”

At this point a good approach may be to look at this issue through a couple of lenses:

1.   What is effective teaching and how do we evaluate it?

2.   Problems with the current system(s) and possible solutions.

We are thinking that by looking through these two lenses the policy recommendations will jump right out at us and practically write themselves!  Maybe not quite that easy but… our task is to think about what would be pivotal and essential to building an effective teacher evaluation system, specifically with regard to measures.

So…What is an effective teacher?  This was one of the original discussion threads and I will copy the summary of the groups’ thoughts that Sarah posted last week:

The overall consensus is that “no teacher is effective unless students are growing.” Effective teachers can specifically contribute to this growth by:

Teaching their students to problem solve, perform research and wrestle with the challenges of discovery;

Adapt their teaching to the needs of students;

Spend time developing the strengths of individual students;

Have high expectations for students and believe in finding the right balance between pushing students too far and nudging them to go beyond their comfort zone to the next level of learning;

Use engagement strategies and stimulating approaches to motivate and excite students to learn;

Use techniques to bridge the distance between students and teacher;

Respect that the unique and individual relationship between teacher and student determines the learning dynamic.

Encouraging students to have fun and be comfortable with learning and taking risks, and are able to apply what they’ve learned

What measures come to mind?  What discussions that we have had address these “contributions?”

And… problems with the current system?  I will not take the space to include all of the issues being discussed, but will include some of the overarching issues.

Measures are currently based on:

·      student achievement data from standardized test scores (Do you want me to list the possible equity issues with class size, high demand schools, special needs students?)

·      percentage of students at mastery of state standards (inconsistency of state to state and state to national)

·      credentials earned by teachers (research shows no correlation between highly credentialed teachers and student achievement)

·      “Drive-by” observations (or if teaching long enough NO observation) of practice that may rate things as arbitrary as start time, appearance, etc

·      evaluations provided by administrators with little knowledge of “best practices”

·      evaluations without reflection or feedback (no pathway for improving practice)

·      limited knowledge of teacher impact on student growth

So…keeping those two lenses in mind

1.   What is effective teaching and how do we evaluate it?

2.   Problems with the current system(s) and possible solutions.

What are some possible “solutions” or recommendations for how to approach this idea of measuring effective teaching.

I am going to list (randomly) some of the ideas being discussed.  Please think about these discussions you’ve been having.  Do you see some of these as possible recommendations or several of them combined as a policy recommendation?  Our next job will be to distill down our discussions and craft policy recommendations that will include any related innovative practices and related research that can back up the recommendation or provide a credible reference.  Please think through the information in this blog and be ready to discuss those close to your heart!

Ideas being discussed:

·      Evaluations including student achievement and quality of instruction data

·      the need for multiple measures

·      the need for varied measures

·      the need for multiple evaluators

·      improving inter-rater reliability- the need for consistently trained evaluators

·      including student work

·      Different approaches to value-added measures

·      Using teaching standards to evaluate instruction (consistency in definition of quality teaching)

·      including teacher reflections about student work and/or instructional moves

·      portfolios- teacher

·      portfolios- student

·      Smart Goals

·      Data Folders

·      Comprehensive checklist or rubric for teacher observation/evals

·      Videos

·      Essays

·      Evaluations to strengthen teaching (paths for improving- coaching, peer review, etc)

·      Student surveys

·      Parent surveys

·      Teacher surveys

·      Trained district evaluation teams

·      Using consistent measures but altering expectations about how different groups perform in different situations (not perfect but addresses some of the equity issues)

·      Documentation of varied teaching techniques to accommodate students- differentiation in general

·      Authentic assessments versus or in addition to standardized testing

·      Weighting results

·      Master/mentor teachers to build capacity and evaluating all involved

·      Staffing and training to meet increased needs and responsibilities of a comprehensive evaluation system (implementation issue?)

·      Electronic portfolios

·      Professional growth plans

We encourage you to start writing or typing.  Please think about how you see these recommendations being organized and record ideas around at least two recommendations to be shared during our conference call.

Recommendation:

Related innovative practices:

Related research:

We look forward to talking with you soon!  Darcy and Dina

Great work, everyone!

I wanted to bring your attention to what is (I think) a good quick synopsis  from Rob Manwaring of some of the research about using value-added student achievement data to evaluate teachers. Normally I wouldn't block quote, but since the Quick and the Ed (the blog where this is posted) is changing, I want to make sure we capture it:

There’s a great deal of attention being given to using test scores to measure teacher performance these days, recent announcements from the Gates foundation ensure this will be high on the national agenda in coming years. But recent studies show that the value-added measures contain significant amount of error. Which raises questions: how can imperfect measures be incorporated into high-stakes decisions like teacher pay? How good is good enough?

Reformers have been waiting for longitudinal data systems to be implemented to provide value-added data to support improvements to compensation and retention decisions. The data is now there in several states, but the quality of the new information may not be as good as many of us had hoped. Just before Thanksgiving, two new studies were released that show the lack of stability of value-added measures of teacher effectiveness over time. The first by Dan Goldhaber looks at North Carolina data to see if pre-tenure teacher effectiveness (measured by the value-added gain of a teacher’s students) is a good predictor of their effectiveness post-tenure (here). The study showed that a teacher ranked in the bottom quintile of teacher effectiveness has a 32 percent chance of being in the bottom quintile post-tenure. While this is better than random (random would be around 20 percent), it is not much better than random. At the same time, 11 percent of the poor performers pre-tenure (bottom quintile) end up being in the top quintile post-tenure. The measure is a little more consistent at identifying top performers – 46 percent of top performers are top performers post-tenure (see Table 1 for all measures of pre and post value-added effectiveness). Goldhaber also looked at using the first 3-years of data to predict outcomes, and the predictive power does not change much.

Table_1a.pdf

A second paper by Tim Sass shows similar results from California and Florida studies. (here)
This paper focuses on whether value added measures of teacher quality are stable enough to use for compensation decisions. It shows similar results as the Goldhaber study over time. The lack of stability over time may not be surprising given the group of students a teacher gets each year is random. The data can not measure whether a teacher had a particularly disruptive class in the first year, and a better group of students the next. So, the randomness of classroom make-up may have a lot to do with these results. The Sass study shows that while measurable, student characteristics explain some of the differences in value-added effectiveness, but most of the differences across time are unexplained (See Table 2 for complete effectiveness measures).

figure_2.pdf

The part of the Sass study that caused me the greatest concern was how inconsistent these value-added measures are across tests. Students in Florida take two tests annually. They take a low stakes norm-referenced test and a high stakes standard-aligned test. Sass looks at how stable these value-added measures are across these two tests. So for this comparison, the random draw of students is the same for any given teacher. While these results look a little more stable (43% of bottom quintile teachers remain in the bottom quintile on the other exam), they are not as stable as you would hope. If just switching the exam moves 5 percent of teachers from the bottom of the distribution to the top, it would likely make teachers question the validity of the measure reflecting true effectiveness.

These papers and a few others suggest that value-added measures are not very consistent over time, and may not be the panacea for which some reformer have been hoping.

How Good is Good Enough? Now you would think that the bar for improving teacher compensation and tenure decisions would be pretty low. The current compensation structure is based almost exclusively on a teacher’s years of experience and college credits/advance degrees. Advance degrees have been consistently shown to have no impact of teacher effectiveness. For experience, teachers appear to improve their craft slightly over the first two to three years, but additional experience does not seem to have any impact. Clearly moving to value-added compensation could more accurately reward effective teachers than the current system. However, if a compensation system were based partially on these value added measure, I think that teachers would perceive the outcomes above as too arbitrary. It also makes me think that principals and mentor teachers could do a better job of predicting effectiveness than last year’s test results. (See Brian Jacobs on this question – principals seem to do pretty well on the identifying teachers at the top and bottom of the distribution, but their measurement is less predictive than prior year’s value-added (here). Of course this is not an either or choice. Can principals armed with value-added test results do an even better job than either one alone? What about a combination of principal evaluations, mentor teacher evaluations and value-added? Are there more rigorous evaluation methods like those of the Teacher Advancement Program or others better predictors than the value-added measures? (See Ed Sector Report on Teacher Evaluation here) As with all good research, it leads to more research. And with Gates interested in these topics, more research is likely to be on its way.

Hi Raquel, Rachel,JoAnn, Connie, Matthew and Sherman,

It's time to get started and delve into researching specific quesions, which will help all of us (other teams included) in this process.  Darcy and I will break you up into groups of two and then assign each group a question to research and discuss.  We would like for you, at this point, to start logging your thoughts, questions and comments right to the measures discussion area.  We will start a new discussion called..."Let's Talk Measures..." and that can be our forum for now to share your ideas etc... with everyone.  Darcy and I will assign the questions by tomorrow, so please be on the lookout in your task box to see who your partner is and the question you have been given..

Here are the three questions to review... If you have preference, please let us know by the end of today and we will do our best to assign it to you.  We can't wait to get some research done, and start our discussions.  We hope that you all feel comfortable with the site and if you have any questions, comments, ways that Darcy and I can assist you in this process, please either write to us via our personal emails... rockd@agnon.org and dmoody12@cox.net.

1. What student-level measures (test scores, other student work) would be useful?  Why?  Are there any innovative practices in this area?

2. What classroom-level measures (observations etc...) would be usefull?  Why? Are there any innovative practices in this area?

3.  What school-level data would be useful?  Why?  Are there any innovative practices in this area?

Also, it is very important,we feel, to interact with the other teams as you see fit.  We know some of you are on two teams and we would love to have integrated discussions from all teams.

More to follow tomorrow and in the few weeks to come.  Please note, that our final draft for our recommendations is due Sept. 14th and so that leaves us but a few short weeks to upack all of this and design goals and recommendations... thanks for all of your support, and interest.

We will be contacting you tomorrow...

Dina and Darcy

Hi Everyone!

Just wanted to let you know that Darcy and I are in the process of setting goals and gathering some task ideas to assign to the group and we will have some of that ironed out in the next day or so.. We will post a blog and our goals/tasks by Thursday or Friday of this week.  Until then, thanks for all of your posts in our discussion thread... we are very excited about working together and we look forward to unpacking this topic of measurement as the weeks go on!    While you are reading any articles that are posted...or any others pertinent to the topic, keep this thought in mind, if you will... we are trying to assess teacher effectiveness based on student success via assessments, evaluations and the impact a teacher has on a student's success...on and off the paper...

Thanks for all of your participation!

Dina and Darcy