Why Two Trained Observers Can See the Same Lesson Differently

Two administrators walk into the same classroom. They have been trained on the same evaluation framework, understand the same rubric, and observe the same lesson from beginning to end.

You might expect them to leave with the same evaluation of what they saw. But that doesn't always happen.

One observer may focus on the questions the teacher asks. Another may notice how students respond to one another. One may see a classroom full of students participating and identify strong engagement. Another may notice that the teacher is still doing most of the thinking.

Neither observer necessarily misunderstood the rubric. They may simply have noticed, prioritized, and interpreted different evidence.

Observation Is More Complicated Than Watching a Lesson

Classrooms are busy places. During even a short observation, an administrator may listen to teacher questions, watch student responses, note instructional strategies, look for evidence of differentiation, consider classroom management, and connect it all to multiple indicators on an evaluation rubric.

That's a lot to process in real time.

Research on classroom observation has identified the observer as an important potential source of variation in evaluation scores. Cohen and Goldhaber (2016) note that observers can struggle to keep multiple dimensions of instructional quality in mind at the same time and that rater differences can contribute substantially to variation in observation scores. 

Training helps observers understand what the rubric asks them to evaluate. But training cannot make a classroom less complex.

What Did You Notice?

Imagine two observers watching a middle school discussion. Several students are eagerly raising their hands. The teacher calls on students, asks follow-up questions, and keeps the conversation moving.

One observer records strong student engagement because students are attentive and participating. The other notices that the teacher asks most of the questions, calls on the same few students repeatedly, and does most of the talking.

Same lesson. Same rubric. Different evidence captured.

That difference becomes important when observation notes turn into scores. Classroom observation systems depend on observers applying evaluation criteria accurately and consistently; otherwise, scores can reflect differences among raters as well as differences in teaching practice (White, 2018). 

The issue isn't that observers should record identical notes. Different people will naturally notice different things. The concern is whether the evidence being used to make a professional judgment provides a consistent and defensible picture of what actually happened in the classroom.

Consistency Starts With the Evidence

When two observers arrive at different scores, the natural response may be to revisit the rubric or provide another round of training. Both may sometimes be appropriate. But there is another question worth asking first:

What evidence did each observer use to arrive at the score?

Comparing scores tells us that observers disagree. Comparing the evidence behind those scores can help us understand why.

This is one of the problems Evidence-First™ was developed to address. Rather than starting with a rating and working backward, Evidence-First focuses more on observable evidence of classroom practice and the connections observers make between that evidence and professional expectations.

The goal isn't to make every classroom look the same—or to expect every observer to write identical notes. It is to create a stronger, more consistent foundation for the professional judgments that follow.

Because when two trained observers see the same lesson differently, the most useful conversation may not begin with “Who scored it correctly?”

It may begin with “What did each of us actually see?”

References

Cohen, J., & Goldhaber, D. (2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387. https://doi.org/10.3102/0013189X16659442

White, M. C. (2018). Rater performance standards for classroom observation instruments. Educational Researcher, 47(8), 492–501. https://doi.org/10.3102/0013189X18785623