Instructional Rounds & Inter-Rater Reliability

Radiologists are trained to look at very specific sources of data— MRIs, CT scans, and x-rays, among others—and provide accurate judgments about what they see.

As instructional leaders, we aspire to do the same in classroom observations. But can we?

If you show the same image to half a dozen radiologists, there should be substantial inter-rater reliability on the question of whether:

  • Everything is normal, or
  • Something is definitely wrong, or
  • More information is needed due to ambiguities

In radiology, measuring inter-rater reliability in radiology is both possible and essential. But achieving rock-solid inter-rater reliability isn’t always possible.

Is the same true in instructional leadership and supervision? The increasingly popular practice of “instructional rounds” provides some critical clues.

Rounds Isn’t Radiology

Can we achieve inter-rater reliability in teacher observations? This is an issue of growing importance, because many school districts are convening principals—as a form of professional development—for instructional rounds visits focused on calibrating observation ratings.

Other districts are using non-school-based observers to provide a second rating when a teacher receives an unfavorable review—much like a hospital showing an MRI to another radiologist for second opinion.

If we’re going to conduct high-stakes teacher evaluations, the thinking goes, they need to be valid and reliable, and inter-rater reliability is a strong indicator that an evaluator is being fair.

How is it working for our profession? Is the “rounds” model holding up? Does the “second opinion” add value?

First, we need to understand what medical rounds are. When conducting rounds, doctors discuss all of the available information about the patient, not just a single source of data like an MRI.

Second, while I’m no doctor, I suspect that if you asked a group of physicians how well their rounds achieve inter-rater reliability, you’d be met with a puzzled look. An accurate diagnosis leading to an effective course of treatment—not inter-rater reliability—is the goal of medical rounds.

And yet in the instructional rounds process, we’ve been stretching one-shot classroom observations beyond what they’re capable of telling us.

In short, we’ve been treating classroom observations like MRIs.

Inter-Rater Reliability and Instructional Leadership’s “MRI”

The “MRI” for teaching is the classroom observation. There’s no doubt—when leaders convene to hold instructional rounds, the “image” they’re looking at is a lesson or part of a lesson.

Typically, instructional rounds works like this: A group visits a classroom, observes for a while, then departs to discuss what they saw (with or without looping in the teacher they observed).

There are variations on the process, but at the core of rounds is a classroom observation by a group of outsiders, escorted by the teacher’s supervisor.

We’d like to believe that a single observation can yield precise insights about a teacher’s strengths and weaknesses, especially if we put our heads together. We’d like to believe that we’re radiologists looking at an MRI. Because we all have evidence, and we can compare notes, we think we’re like radiologists all looking at the same CT scan.

Does the instructional rounds process turn us into the educational equivalent of radiologists?

No—or if it does, only in a very narrow sense. Observing a lesson is simply the tip of the iceberg of teaching practice.

Missing Context

When we attempt the rounds process, we’re typically working with relatively little information—the information we can ascertain from observing a lesson.

For certain topics, this isn’t a problem. If you want to see how well the teacher uses a certain questioning strategy, or how he handles student behavior, and if the lesson gives you the opportunity to see what you came to see, rounds can be productive.

If you all observe the teacher using a questioning strategy, it’s reasonable to work toward some degree of consensus about the effectiveness of the practice you observed.

But it’s entirely unreasonable to expect to come to consensus about the teacher’s overall effectiveness. It’s simply too broad a question for the narrow data available. Bringing along more observers doesn’t help.

To make broader judgments, we need richer data. High-performance instructional leaders have more information, which isn’t readily available to a team of outsiders. They’re in classrooms daily, and in every classroom every two weeks. This provides enormously helpful context for what happens in an observed lesson, and makes the available evidence much more useful.

If you’re doing rounds, make sure you aren’t mining your observations for insights they can’t provide. And if you’re using outside observers, make sure they spend enough time in the classroom to have meaningful context for what they see during formal observations.

A Better Goal

Inter-rater reliability is only a useful construct when each of the raters have sufficient information. Otherwise, they’re achieving precision without accuracy—producing tightly clustered ratings that, while close to each other, miss the mark.

But what can be gained from having half a dozen or more observers discuss what they see in a lesson? Plenty.

Even with incomplete information, a rounds team can benefit from using the language of their shared instructional framework to discuss the teaching practice they observed.

The problem? The team won’t always get the information they hope to gather on a specific topic.

If no students misbehave (good!), they won’t see how the teacher responds to misbehavior. If the teacher doesn’t use the preferred questioning strategy during the observed lesson, there’s simply no evidence to discuss.

With those factors in mind, here are four recommendations for getting more out of instructional rounds—instead of trying to achieve inter-rater reliability.

Four Ways To Get More From Instructional Rounds

1. Go in with open eyes
Don’t expect to see a particular strategy at a particular time, unless it’s a strategy that should be used every single day in every single lesson.

Instead, stay attuned to what the teacher is trying to accomplish with the lesson. That’s a much fairer basis for judging the effectiveness of a lesson, and will lead to much more relevant discussions.

2. Record evidence in the language of your instructional framework
The more familiar you are with your framework, the more you’ll be able to capture salient points in your notes. Better evidence will make for a better discussion afterward.

3. Choose a focus with plenty of evidence
Don’t start your discussion with “warm” and “cool” feedback. Unless it was an unmitigated disaster, you probably have very little basis for drawing an overall conclusion. The effectiveness of a lesson depends on what happened before and after the lesson, which you don’t get to see in a brief visit.

Instead, start your discussion by looking closely at your instructional framework. What elements seems most salient, given what you observed? Where do you have the most—or the most interesting—evidence to discuss?

4. Look for descriptors of practice

This is where the “calibration” discussion can become productive. Once you’ve decided what you actually have enough evidence on, you can seek to align that evidence with your evaluation rubric or instructional framework.

In the better frameworks, like Danielson’s, you’ll find leveled descriptors of each practice, so you’ll have a reference point other than personal opinions.

Remember, you’re calibrating the way you match evidence to the rubric, not calibrating your judgments of the teacher’s overall performance. You didn’t see the teacher’s overall performance—this is radiology, not true medical rounds.

Toward Better Professional Development for Instructional Leaders

The rounds process can be powerful if we keep our focus on the right goals. It should be a part of every principal’s professional development.

But even more important is getting administrators into their own teachers’ classrooms more often. That’s why I created the 21-Day Instructional Leadership Challenge, which to date has helped more than 3,200 administrators in 50 countries develop the daily habit of providing evidence-rich feedback to their teachers.

If you’re interested in bringing the Challenge to your district, please get in touch. It’s a free program that anyone can join at any time, and if you’d like to bring me out to kick things off, contact me for rates and availability.

Share:Email this to someoneTweet about this on TwitterShare on Google+Share on FacebookShare on LinkedIn

Key Practices of High-Performance Instructional Leaders

What does it mean to be an instructional leader?

Too many definitions contrast instructional leadership with “mere” management, as if the management work that administrators do isn’t related to teaching and learning. I believe that it’s all connected, and that an effective instructional leader is also an effective building manager who ensures that everyone has what they need to succeed in teaching and learning.

Instructional leadership is more than being a school leader who focuses on teaching and learning—though that’s certainly crucial. We’ve all known administrators who focused too sharply on instruction, and lacked the leadership perspective necessary for leading a healthy school culture. Effective leaders are balanced in their focus, but there’s more to consider.

As administrators, we’re not the only leaders in our schools—far from it.

Formal staff leaders, like teachers who serve on committees, are leaders, but so are teachers who strive to make things better, and who take initiative even if it’s not noticed.

Students, too, can be instructional leaders, setting goals, working toward standards, and contributing to a culture of excellence.

We need to define instructional leadership broadly enough to include all of its manifestations in our schools.

Even so, what do high-performance instructional leaders actually do?

Three Key Practices

High-performance instructional leaders—whether they’re formal leaders like administrators, informal teacher leaders, or students—engage in three key practices.

Think about what each of these practices might look like for administrators, teachers, other staff, and students:

  • Listening with the Language of Learning
  • Making Decisions in Dialogue
  • Building Systems for High Performance

Let’s take a closer look at each of these practices.

Listening with the Language of Learning

As administrators, we’re trained to tell—to give feedback, to set expectations, to delegate tasks, to communicate a vision—but we’re not conditioned to listen.

As anyone who’s ever been in a relationship can affirm, listening matters. It serves as a sign of caring, and as the saying goes:

Nobody cares how much you know, until they know how much you care.
—Attributed to Teddy Roosevelt

But listening also matters in a very practical sense, because it provides information. Leaders who insulate themselves from critical information quickly start to make bad decisions.

And yet it’s not enough to listen to whatever anyone has to say, because we’ll hear a great deal that could take us off-course.

High-performance instructional leaders listen with a particular set of ears, attuned to a specific, shared framework.

One such framework is your teacher evaluation and growth model, whether it’s based on Charlotte Danielson’s work, a state framework, or a model developed locally.

A shared instructional framework identifies effective practice and distinguishes it from ineffective practice, as clearly and specifically as possible.

When you have a shared vocabulary for professional practice, you can listen for and respond to that vocabulary in a way that advances practice.

This is very different from using jargon as a way of implying consensus. Phrases like “what’s best for kids” and “research-based practice” are often used to end discussions and silence disagreement, without offering any substance or doing anything to build instructional leadership capacity.

For students, listening with the language of learning involves knowing and working toward standards, whether those are state standards, content-area standards specific to a course, or other agreed-upon aims. When students know what they’re working toward, they’re able to take greater ownership of the learning process, and more able to contribute to the organization’s instructional leadership capacity.

Making Decisions In Dialogue

Leaders are called upon to make decisions all day, every day. And yet, the leader who makes decisions autocratically will soon burn through their staff’s goodwill and trust.

There is no single best type of decision-making that should be used in all circumstances. Some situations require speed and decisiveness—how to handle a safety crisis, for example—while others demand careful discussion, deliberation, and consensus-building.

Still other decisions fall somewhere in between on the spectrum from autocratic to inclusive. If there’s no single best approach, what should leaders do?

High-performance instructional leaders operate with an extraordinary degree of transparency about decision-making. They develop, with staff, a matrix delineating the method by which different types of decisions will be made, and specifying whether and how everyone will be involved, consulted, or informed.

Wise leaders also recognize that some situations that appear to call for a decision are in fact polarities, in which there exist healthy, dynamic tensions that need to be managed carefully. This, too, calls for dialogue and transparency in order to create clarity.

Then, once the situation has been grasped, managed, and communicated, high-performance instructional leaders keep solved problems solved by creating systems that ensure effectiveness, efficiency, and consistency.

Building Systems for High Performance

Sustained high performance involves three factors:

  • Strategy determines our effectiveness
  • Tools increase our efficiency
  • Habits create consistency and build the muscle we need for high performance

I call this the High Performance Triangle:

While implementing sound strategy consistently is an obvious goal, we often ignore the question of efficiency. If we’re willing to work hard, why bother thinking about efficiency?

At its core, inefficiency is waste—wasted resources, and wasted opportunities.

If teachers are scoring formative assessments by hand instead of taking advantage of their BYOD program to allow technology to help, they’re missing crucial opportunities to obtain timely feedback about their teaching.

If a principal is taking handwritten notes on a legal pad, and not keeping organized, digital documentation of classroom visits, the observation and evaluation process will achieve less than it otherwise could.

But when sound strategies are articulated clearly, implemented efficiently with the right tools, and installed as personal or organizational habits, the capacity for high-performance instructional leadership emerges.

It’s a story that has played out in countless schools around the world. It’s a story that can unfold in your school or district.

If you’re interested in bringing this message to your instructional leaders through an onsite or virtual presentation, please contact me to discuss how we might work together.

Share:Email this to someoneTweet about this on TwitterShare on Google+Share on FacebookShare on LinkedIn

Why Instructional Leaders Must Stop Propagating Fear of Accountability

Teachers are the backbone of the education profession, and we need talented, committed people to work with our students. So I was dismayed to hear Global Teacher Prize winner Nancie Atwell advising creative, smart young people to avoid the teaching profession. From a CNN interview: “Public school teachers are so constrained right now by the […]

Continue reading...

Dan Ariely’s Reddit AMA—My Favorite Quotes

Cognitive psychologist Dan Ariely is one of my favorite decision scientists, and he’s recently taken an interest in time management—one of the most challenging decision-making issues we all face each day. So I was delighted to see that he did an “Ask Me Anything” (AMA) on the social network Reddit recently. Here are my favorite […]

Continue reading...

Look for New Articles at

I’ve decided to move the majority of my future writing to, which we’ve been updating this year and hope to have finished in the next month or so. Most of this change has been behind-the-scenes since February (when we switched from Drupal to WordPress), but you’ll see more visible changes at The Principal Center […]

Continue reading...

Dear Michael Hyatt: Lean In

Last week, author and entrepreneur Michael Hyatt made a surprising announcement: he’s offering a high-end coaching “mastermind” program for a select group of 15 people. That’s not the surprising part; many consultants and thought leaders offer these types of groups, usually for an annual fee in the low five figures, and including a combination of […]

Continue reading...

The First App To Buy After Upgrading to iOS 8

When the email came in, I actually did a little happy dance. Forgive me for being excited, but it’s a big moment for iPad and iPhone users. Why? There are lots of great things about iOS 8, but one that can’t be overemphasized is support for custom 3rd-party keyboards, which means TextExpander Touch 3 now […]

Continue reading...

How To Use Your iPad Without Being Antisocial

Do I want to be a productive weirdo, or actually pay attention to the people around me? It’s not a fun dilemma, yet it’s probably one you face. If you’re among the majority of school leaders who have iPads, you probably hoped the device would make a difference in your productivity throughout the day. More […]

Continue reading...

Why Behavior Trumps Attitude

We’ve always known the importance of “getting the right people on the bus,” to borrow a phrase from Jim Collins. As leaders, we each have in mind a picture of the “right” kind of person for our school—the right attitude, the right work ethic, the right interest in collaborating, the right coachable mindset, the right […]

Continue reading...

Go and See: Classroom Walkthroughs as Genchi Genbutsu

As leaders, we need to spend our time where the work is done. We need to understand that work deeply, so we can provide the kind of leadership the organization needs. As instructional leaders, that means we need to be in classrooms. At Toyota, this concept is called Genchi Genbutsu, which conveys the idea of […]

Continue reading...

Is A Paperless Office Possible?

The dream of having a paperless office has been around for decades, yet it seems that as each year passes, we end up with more paper, not less. Case in point: when my district went to an all-online job application system, instead of getting 250 hardcopy pages in the mail, I’d have to print 500 […]

Continue reading...

Real Firefighting: Leadership as Creative and Reactive Work

I love being part of the profession of school administration, but the word “administrator” is a little too paper-pusher-ey for my tastes. That’s why, like many people, I prefer the term “school leader.” Leadership is inherently creative work. We’re not just pushing the buttons that someone else in our bureaucracy told us to push. We’re […]

Continue reading...

Should You Turn This Year’s Regrets Into Next Year’s Agenda?

One of the most critical starting points for high-performance instructional leadership is having a focused leadership agenda. If you don’t have a well-defined sense of what you’re focused on, everything that you’re not focused on will gradually creep in. The principalship—and the work of schools in general—is subject to the Ratchet Effect: once the “ratchet” […]

Continue reading...

How (Not) To Differentiate Your Leadership with Staff

As leaders, we achieve most of our results indirectly. I don’t teach reading or math or art; I ensure that reading and math and art are taught well. We work through relationships and systems to turn our daily work into results for students. But that impact isn’t distributed evenly. Teachers are not mere conduits for […]

Continue reading...

The $10 Experiment for Giving Something Up

Scope creep. “Other duties as assigned.” Coverage. Little tasks and duties get added to our calendars, and over time, this can take up quite a bit of a principal’s week. We’re all team players and don’t want to ask our staff to do anything we wouldn’t do ourselves…but sometimes we end up spending our time […]

Continue reading...