Advances in Teacher Value-Added Modeling Measures

In 2011, the Florida legislature passed a law requiring that
teachers in Florida be evaluated in part using student achievement
data.  Teachers are measured based on a statistical inference
of how much they impacted a student’s learning over the course of
the year.  This is done using a method called “value-added
modeling”.

In value-added modeling a formula is used to predict how well a
student would be expected to perform on an assessment, based on
previous assessment performances and student, class & school
characteristics.  That prediction is then compared with how
well the student actually performs.  The difference between
how the student would have been predicted to do based on past
performance and specific characteristics alone and how he or she
actually performed is considered the “value-added” by the teacher
to that student’s learning.  So if a child exceeds
expectations, that is considered evidence of positive value-added
to the student’s learning by the teacher that year.

The Florida formula for calculating predicted student
achievement is a complicated one, but takes into account the
following factors:

  • The number of subject-relevant courses in which the student is
    enrolled
  • Two prior years of achievement scores
  • Students with Disabilities (SWD) status
  • English language learner (ELL) status
  • Gifted status
  • Attendance
  • Mobility (number of transitions)
  • Difference from modal age in grade (as an indicator of
    retention)
  • Class size
  • Homogeneity of entering test scores in the class

In essence, the model estimates the impact of these factors by
looking at how growth was affected across all students with any of
these characteristics, and then adjusting the expected performance
of any student with that characteristic accordingly.  (A full
technical explanation of the model is available here.)

Using enough student data, these models can actually predict
well how a student with specific performance history and
characteristics would be expected to perform. 

The key issues are making sure the model correctly accounts for
everything it needs to, and that the data is all accurate. 
One of the common concerns about value-added modeling is that there
are multiple different ways the models can be constructed. 
For example, should you control for student poverty or not? Is it
better to use three years of prior data than two, or does it not
make a difference?  With each potentially important element
that is added or not added, a change is being made that could
ultimately affect some individual teachers differently. 

Last week, the Institute of Education Sciences and Michigan State
University’s Education Policy Center
held a conference bringing
together some of the top value-added modeling researchers,
education policy leaders and educators from around the country to
share ideas on the latest advances in not only statistical designs
but also practical implementation strategies. 

We attended to learn more about the advances being made that
Duval County and Florida can learn from to continue trying to make
sure we have the most comprehensive, accurate and fair teacher
accountability measures possible. 

Some of the key issues raised at the conference that Florida may
want to consider as part of ongoing efforts to ensure the accuracy
of our teacher evaluation models:

  • Improved roster verification processes: One of
    the biggest challenges for many states is making sure the right
    students are matched to the right teachers in the data. 
    Because students move between teachers, schools and districts
    regularly, sometimes data records are not completely up-to-date in
    student-teacher assignments when VAM scores are calculated. 
    Processes for having teachers personally check, and principals
    cross-check, which students they taught in which subject areas were
    shown to significantly reduce the risk of teachers being
    incorrectly classified (as “Highly effective”, “Effective” or
    “Needs Improvement”) in their final results.
  • Improved ‘dosage’ verification processes:
    Related to correct student/teacher/course associations was the
    issue of how much a teacher had a student for any individual
    lesson.  For example, in some states elementary school
    students are assigned to their homeroom teacher in the data for all
    their primary subject areas (i.e., Reading, Math, etc.).  But
    many of those schools actually use “team teaching” or other
    instructional models that have students going to different teachers
    for all or some of their instruction.  As part of the roster
    verification process mentioned above, some places ask teachers to
    assign a percentage of the students instruction in each course area
    that they are responsible for.  For example, 100% if they are
    the students only Reading teacher, 50% if the student spends equal
    time with two different teachers for Reading instruction, etc..
    Again, in these models principals are often asked to verify and
    correct when student’s assignment numbers between teachers do not
    add up to 100%.
  • Multiple model formulas: Many states use a
    single, uniform statistical formula for measuring
    teacher-value-added effect across all grade levels. But the way a
    student’s instruction is structured in elementary school is
    typically quite different from the way it is in high school. 
    For example, some researchers at the conference presented evidence
    on the impact of sorting tracks in the upper grades (college-prep,
    career and technical ed, other) on teacher value-added estimates
    when not accounted for in the model.  This raises the
    possibility that, rather than a single formula for teachers at all
    grade levels, perhaps a formula with multiple variations
     specifically-tailored to the grade-levels being measured
    would be more appropriate.
  • Improving school effect attribution: One of
    the core functions of any value-added model is to separate what
    impacts the school environment as a whole has on achievement from
    what impacts the teacher specifically has within the
    classroom.  (For more explanation, see here.)  But what to do with that is
    another question.  In Florida, 50% of the school effect is
    credited to the teacher, on the thinking that all teachers share at
    least partial responsibility for the overall school
    environment.  But by assigning 50% of the school effect to
    each teacher, that is telling the model that each individual
    teacher is single-handedly responsible for half of the impact of
    the entire school.  In other words, this might make sense if
    there were only two teachers at a school, but in general it’s
    assigning too much credit and blame to each teacher for whatever
    else is happening in the school as a whole.  Many researchers
    and policy makers talked about options they are using that makes
    more efficient and appropriate use of the teacher and school
    components together.
  • Better connecting research to policy: Some of
    the advanced VAM researchers at the conference lamented the
    difficulty of getting their more complex, advanced (and often more
    appropriate) models for measuring teacher value-added effect
    adopted by politicians or education policy makers who may feel
    constrained by the need to be able to explain an idea simply to
    their constituents.  In other cases, limitations placed on
    models prior to developing them were also cause for concern among
    some researchers.

Measuring teacher effectiveness is a critical component for
ensuring all students have access to highly effective teachers,
capable of accelerating their learning at every grade level. 
But it is also a highly complicated undertaking.  Value-added
modeling is a highly useful method for measuring teacher’s impacts
on student learning when statistical models are specified well and
results are interpreted strictly.  There are some tremendous
advances in value-added modeling happening around the country that
Florida should try to learn from and evaluate what elements are
most appropriate for here.

For more information about Florida’s value-added model formula
click here, or to view recordings of all the
presentations from the conference, click here.

Source Article from http://www.jaxpef.org/news/2013/10/advances-in-teacher-value-added-modeling-measures.aspx

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>