How Premiership officials review their performance

In one of Twickenham’s plush corporate boxes, a screen and a projector has been set up. Welsh will lead this particular seminar, which features three other referees alongside [Christophe] Ridley.

“After every Premiership weekend, we go into a review session,” Welsh says. “I normally do two games every weekend in person. Typically, after a Saturday game, I will review it in full on Sunday. Even if I’m at the game. Christophe will do the same. Then, today, we look at the common bits. The coaching, the technical stuff.

“Sitting behind everything we do is a system called AMS. An online system where we record everything from every match; recording the referee’s accuracy. It’s a comprehensive review process.”

AMS is a refereeing goldmine. It is a (strictly confidential) database where referees can access a treasure trove of data about their refereeing. How many penalties for not rolling away has Ridley awarded this season? And how does it compare to his peers? He can view the minutiae of his officiating at the push of a button. Ridley likes to look at his average ruck speed as it presents a simple way of identifying how well he manages a number of factors at the breakdown.

The detail is staggering. It is all completely transparent, too; Ridley can access exactly the same data for any of his colleagues. “There’s no hiding,” he says. “And the really clever thing about our system is I can view every single decision I’ve made. If I click on ‘not rolling away’ here, I’ll download a movie which is a montage of every decision I’ve made for that.”

“There’s a massive amount of data that sits behind every game,” Welsh continues. “We look at ‘no decisions’ [decisions he should have made but he didn’t]; and errors [where he did make a decision, but it was inaccurate],” Welsh continues. “They’re the two main things.

“We also look at coaching outputs: communication, management, values or issues in the game – if there are problems with players in the game. It’s both the technical and management side.”

Ridley shows one of his clips, where he awarded a mark for a team even though the ball was touched in flight. “I gave it because everyone stopped – but that was wrong. In law, it’s play on, but I wasn’t certain it had been touched,” Ridley says.

“In these instances, you’re better off making no decision than making a wrong decision. We would rather a referee played through if they missed something, than guess and get it wrong.”

Welsh adds: “That’s an error in law. But the error rate we have from the referees is really, really low. They are 90 per cent plus accurate in the decisions they make. It’s the ‘no decisions’ that there are more of, in terms of things that they didn’t see or do.”
'We identify trends to prepare for the next game'

Refereeing decisions are rarely black and white. Even this small group of elite referees will often not be unanimously in agreement with the independent assessor. Often, the aim is more exploratory. What decisions could have been made here?

“People might think we come to a definitive answer on everything,” Welsh says. “We don’t, but we do try and come to some sort of consensus on most things.”

When there is no consensus, a handy tool can be the external review, carried out by an independent assessor, outsourced to a high-performance consultancy. These reviews are important, but they are of particular significance when the room is split, with the assessment offering a third, impartial voice. The teams have access to this independent data, but they prefer engaging with the officials directly, Ridley says.

“It’s not used as a coaching tool,” Welsh adds. “It’s just for the raw data.”

Today, it is the internal review that is discussed at length, however. The emphasis with these is always on improving.

The internal assessment outlines six main areas of the game: safety, speed, space, scrum, line-out and maul. Both referee and reviewer leave their comments on a summary sheet – and most of the time they correlate.

“We’re trying to put an overview in there of the game,” says one of Ridley’s refereeing colleagues. “It’s useful, often, for anyone refereeing those teams next week. We can look at the main, ‘flair’ points in each of those five areas. And see any trends to prepare for the next games.”

With all of this preparation, how difficult is it for the referees to not prejudice themselves for the following weekend, arriving with preconceived notions about how a team or a player behaves?

“It’s more about how you would deal with a situation,” says another officiating peer. “If a tighthead the week before fell on his knee on five or six scrums, we would use that information to have a strategy of how to deal with it, rather than expect it to happen again. We would talk to him before the game, too, and explain what we’ve seen and that it might need to be a little bit better this week.”

Ridley adds: “We have conversations with the two coaches before the game, too. But a good example might be Faf de Klerk. When he defends, he often starts from five metres behind the defensive line and comes flying up. If you’re not prepared for him to do that, you’re probably going to wrongly penalise him for offside. He might be anyway, but if we’re aware he’s doing it then we can judge it better and get it right. If you’re not prepared, as a referee, for those pictures, you’re going to get it wrong.

“The reason why we prepare like this is so we can reward teams when they get it right.”

How do we 'sell' a decision to fans?

Ridley has prepared 10 incidents that he wants to share with the group. They will be discussed and debated and the more contentious among them will be discussed in tomorrow’s wider group session. The subject matter will be deliberately grey – “I’m not showing a missed knock-on, because it’s just a missed knock-on, we know that and no one in this room cares,” Ridley says.

“We look at coaching things,” Welsh says. “There are times when we’ll say: ‘Christophe, that was world-class – and that’s your template for going forward.’ We might say: ‘That was an ok decision, but how do we make it world-class? How do we “sell” it to the fans and the stakeholders, to make them understand it?’

“We might say: ‘Christophe, you got that wrong.’ Of course, it’s not about you getting it wrong – it's about why you got it wrong.”

Ridley wants a “sense check” on his first clip. He is concerned that he might have been too harsh on a tackler not rolling away – an area that referees have clamped down on this year, deeming the swift removal of the tackler to be vital in order to reward a jackaling player. The debate ensues. Adam Leal says that Ridley was probably right but maybe for the wrong reasons. “The tackler is not blocking a player clearing out but he is preventing the ball-carrier from presenting,” he says. So, a fair call.

The next is a “cheap shot” and some retaliation that was dismissed as handbags on the day, with which Welsh agrees. “We look at context here, too,” he says. “If this game had been a s--- fight for 25 minutes already then you’d need to do something – otherwise you’ll get anarchy. But there had been nothing preceding this. This was a judgment call and, in terms of the way the game had gone, it was correct. You brushed it under the carpet because there was nothing sinister there.”

The conversation moves to a specific club with whom Christophe has had a specific problem with back-chat and challenging calls. One of the other referees says he had no problem the week before while another says it might be a particular issue with Ridley, given that he has spent a bit of time in these clubs’ training sessions, advising as a referee. They know him, and there is a concern that that overfamiliarity is extending to the pitch – where relationships must remain entirely neutral.

The group reinforces that under no circumstances will incidents be reviewed just because a player has appealed or requested such – but there is an exception, for allegations of biting, gouging or testicle grabbing.

'That scrum half has to be fair game'

The finale of this session becomes heated. Ridley first of all, has his knuckles mildly rapped by Welsh for saying “too bad” to a player after awarding a penalty, an offence which requires contextual judgment of the progression of the match.

“‘Too bad’ – we don’t want to be doing that,” says Welsh. “And you know that. You can be better than that. It doesn’t sound like a bad comment but it was almost like saying ‘tough s—’. It’s not language we want to use.”

Ridley adds: “Yeah, I can be better than that. If I want to be the best referee in the world and I’m refereeing New Zealand against South Africa, would I look the All Blacks captain in the eye and say, ‘too bad’?”

But one of Ridley’s colleagues spots something. He vehemently disagrees with the decision that led to the “too bad” comment. It leads to a philosophical debate on how invincible the scrum-half has become at rucks.

Ridley: “That nine has to be fair game. There is no one else at the ruck. Otherwise, you’re saying to the defending team here: ‘You are not allowed to do anything.’ He can’t put his hands on the ball, because it’s not out.”

Ref 1: “I’m not sure that’s fully consistent with how everyone would referee that.”

Ref 2: “We would 100 per cent give that as playing the nine. 100 per cent. You can’t say you want to protect the nine on the other one – this goes completely against that.”

Ridley: “What I didn’t want to do in this scenario is penalise the team who had numbers and reward a team with no numbers at the breakdown. You’re rewarding a team who’ve failed.

Ref 1: “I understand and that’s nice philosophically.”

Ridley: “That’s how the teams will look at it. How does the other team compete for the ball without touching the nine?”

Ref 2: “I don’t see the nine over the ball and looking to engage with the ruck.”

Ridley: “I think that’s a real ‘referee’ way of looking at it. I don’t think anyone else – the coaches, the players, the fans – would understand that. If that was a loosehead prop standing there, in the same position as the nine, then we wouldn’t even be having this conversation. We’re only having this conversation because it’s the nine standing there. I don’t doubt that if we showed this to a bunch of referees they would say ‘it’s playing the nine’ but explain that to Premiership directors of rugby would have a different outcome.”

Unanimity evades the group. As Ridley finishes, he receives a text from Barnes. The two will be working together the following weekend, so the veteran asks his colleague to come and review his clips with him, so that they are both singing off the same hymn sheet. It means Ridley has to duck out of another review, but anything particularly challenging will be brought up in the wider review session later. The format is identical to the smaller seminar; also identical, is that the room is split when Ridley presents his scrum-half clip and Welsh’s earlier words echo around the room.

“People might think we come to a definitive answer on everything – we don’t.”