In 2011, the Florida legislature passed a law requiring that
teachers in Florida be evaluated in part using student achievement
data. Teachers are measured based on a statistical inference
of how much they impacted a student’s learning over the course of
the year. This is done using a method called “value-added
In value-added modeling a formula is used to predict how well a
student would be expected to perform on an assessment, based on
previous assessment performances and student, class & school
characteristics. That prediction is then compared with how
well the student actually performs. The difference between
how the student would have been predicted to do based on past
performance and specific characteristics alone and how he or she
actually performed is considered the “value-added” by the teacher
to that student’s learning. So if a child exceeds
expectations, that is considered evidence of positive value-added
to the student’s learning by the teacher that year.
The Florida formula for calculating predicted student
achievement is a complicated one, but takes into account the
- The number of subject-relevant courses in which the student is
- Two prior years of achievement scores
- Students with Disabilities (SWD) status
- English language learner (ELL) status
- Gifted status
- Mobility (number of transitions)
- Difference from modal age in grade (as an indicator of
- Class size
- Homogeneity of entering test scores in the class
In essence, the model estimates the impact of these factors by
looking at how growth was affected across all students with any of
these characteristics, and then adjusting the expected performance
of any student with that characteristic accordingly. (A full
technical explanation of the model is available here.)
Using enough student data, these models can actually predict
well how a student with specific performance history and
characteristics would be expected to perform.
The key issues are making sure the model correctly accounts for
everything it needs to, and that the data is all accurate.
One of the common concerns about value-added modeling is that there
are multiple different ways the models can be constructed.
For example, should you control for student poverty or not? Is it
better to use three years of prior data than two, or does it not
make a difference? With each potentially important element
that is added or not added, a change is being made that could
ultimately affect some individual teachers differently.
Last week, the Institute of Education Sciences and Michigan State
University’s Education Policy Center held a conference bringing
together some of the top value-added modeling researchers,
education policy leaders and educators from around the country to
share ideas on the latest advances in not only statistical designs
but also practical implementation strategies.
We attended to learn more about the advances being made that
Duval County and Florida can learn from to continue trying to make
sure we have the most comprehensive, accurate and fair teacher
accountability measures possible.
Some of the key issues raised at the conference that Florida may
want to consider as part of ongoing efforts to ensure the accuracy
of our teacher evaluation models:
- Improved roster verification processes: One of
the biggest challenges for many states is making sure the right
students are matched to the right teachers in the data.
Because students move between teachers, schools and districts
regularly, sometimes data records are not completely up-to-date in
student-teacher assignments when VAM scores are calculated.
Processes for having teachers personally check, and principals
cross-check, which students they taught in which subject areas were
shown to significantly reduce the risk of teachers being
incorrectly classified (as “Highly effective”, “Effective” or
“Needs Improvement”) in their final results.
- Improved ‘dosage’ verification processes:
Related to correct student/teacher/course associations was the
issue of how much a teacher had a student for any individual
lesson. For example, in some states elementary school
students are assigned to their homeroom teacher in the data for all
their primary subject areas (i.e., Reading, Math, etc.). But
many of those schools actually use “team teaching” or other
instructional models that have students going to different teachers
for all or some of their instruction. As part of the roster
verification process mentioned above, some places ask teachers to
assign a percentage of the students instruction in each course area
that they are responsible for. For example, 100% if they are
the students only Reading teacher, 50% if the student spends equal
time with two different teachers for Reading instruction, etc..
Again, in these models principals are often asked to verify and
correct when student’s assignment numbers between teachers do not
add up to 100%.
- Multiple model formulas: Many states use a
single, uniform statistical formula for measuring
teacher-value-added effect across all grade levels. But the way a
student’s instruction is structured in elementary school is
typically quite different from the way it is in high school.
For example, some researchers at the conference presented evidence
on the impact of sorting tracks in the upper grades (college-prep,
career and technical ed, other) on teacher value-added estimates
when not accounted for in the model. This raises the
possibility that, rather than a single formula for teachers at all
grade levels, perhaps a formula with multiple variations
specifically-tailored to the grade-levels being measured
would be more appropriate.
- Improving school effect attribution: One of
the core functions of any value-added model is to separate what
impacts the school environment as a whole has on achievement from
what impacts the teacher specifically has within the
classroom. (For more explanation, see here.) But what to do with that is
another question. In Florida, 50% of the school effect is
credited to the teacher, on the thinking that all teachers share at
least partial responsibility for the overall school
environment. But by assigning 50% of the school effect to
each teacher, that is telling the model that each individual
teacher is single-handedly responsible for half of the impact of
the entire school. In other words, this might make sense if
there were only two teachers at a school, but in general it’s
assigning too much credit and blame to each teacher for whatever
else is happening in the school as a whole. Many researchers
and policy makers talked about options they are using that makes
more efficient and appropriate use of the teacher and school
- Better connecting research to policy: Some of
the advanced VAM researchers at the conference lamented the
difficulty of getting their more complex, advanced (and often more
appropriate) models for measuring teacher value-added effect
adopted by politicians or education policy makers who may feel
constrained by the need to be able to explain an idea simply to
their constituents. In other cases, limitations placed on
models prior to developing them were also cause for concern among
Measuring teacher effectiveness is a critical component for
ensuring all students have access to highly effective teachers,
capable of accelerating their learning at every grade level.
But it is also a highly complicated undertaking. Value-added
modeling is a highly useful method for measuring teacher’s impacts
on student learning when statistical models are specified well and
results are interpreted strictly. There are some tremendous
advances in value-added modeling happening around the country that
Florida should try to learn from and evaluate what elements are
most appropriate for here.