Two Trained Evaluators. Same Classroom. Same Rubric. Different Conclusions.
Evidence-First™ Series | Part 1 of 10
Two trained evaluators observe the same classroom. They watch the same lesson. They use the same rubric. Yet they reach different conclusions. Why?
One evaluator sees highly effective questioning. Another sees questioning that is merely effective. One notices strong student agency in the classroom; the other focuses on limited differentiation. When this happens, we often assume the problem lies with the rubric, the rater, or the need for additional calibration training.
But what if the disagreement began before either evaluator ever considered a score?
Research on classroom observation suggests that trustworthy conclusions depend on much more than a well-designed rubric. Ratings can be influenced by observers, lessons, observation length, scoring design, and other contextual conditions (Mashburn et al., 2014; Weston et al., 2021).

What Happens Before the Score?
A classroom observation requires an evaluator to make numerous decisions in real time:
What deserves attention?
Which teacher and student actions are significant?
Which professional expectation does it represent?
What rating does the accumulated evidence support?
These judgments can occur within seconds. The result is what we call cognitive compression: combining observation, evidence recognition, interpretation, evaluation, and recommendation into one rapid professional judgment process.
The challenge may not be that evaluators lack expertise. The challenge may be that we ask them to use too much of that expertise simultaneously.
The Rating May Be Where We Discover the Problem, not Where It Begins
Consider two evaluators who assign different scores to the same lesson. A traditional calibration conversation might begin:
“Why did you give a particular rubric category a 4 instead of a 3?”
But perhaps there are more important questions to ask first:
What did each evaluator notice?
What evidence did each evaluator consider important?
Did they interpret the same classroom events in the same way?
Only then should we ask how each person applied the rubric. A final score tells us that disagreement occurred; it does not necessarily tell us where that disagreement began.
This distinction matters because observers can agree on a final rating while attending to very different evidence, or disagree on a rating even when they observed many of the same classroom events. Research has similarly cautioned that rater agreement by itself does not fully establish the quality or defensibility of observational conclusions (Hill et al., 2012)
What If We Reversed the Process?
This question is at the center of Evidence-First™.
Evidence-First does not replace Danielson, Marzano, InTASC, state-approved teacher evaluation frameworks, locally developed rubrics, or candidate-observation instruments. Instead, it asks organizations to reconsider the sequence through which professional judgment occurs.
Rather than beginning with the rubric as the primary lens for interpreting the lesson, Evidence-First establishes observable evidence markers first and applies the evaluative framework afterward.
The purpose is not to eliminate professional judgment or claim that evaluator bias can disappear. Rather, the goal is to make the path from classroom observation to professional conclusion more visible, disciplined, and open to examination.
Looking at What Happens Before the Score
This post begins a conversation about an often-overlooked question: What actually happens between entering a classroom and assigning a rubric score?
Over the next several weeks, we will explore issues such as the cognitive demands placed on observers, how early impressions can influence later judgments, and why conventional calibration may begin too late. We will also examine how structured evidence can provide organizations with much deeper information than an overall rubric score.
Most importantly, we will consider a different possibility:
The next major improvement in teacher evaluation and candidate observation may not require another rubric. It may require us to rethink what happens before the score.
That is the conversation we hope you will join.
References
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64.
Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C. (2014). The effect of observation length and presentation order on the reliability and validity of an observational measure of teaching quality. Educational and Psychological Measurement, 74(3), 400–422.
Weston, T. J., Hayward, C. N., & Laursen, S. L. (2021). When seeing is believing: Generalizability and decision studies for observational data in evaluation and research on teaching. American Journal of Evaluation, 42(3), 377–398.




Comments