Judging in figure skating has long been controversial, especially around the 2002 scandal where French judge Marie-Reine Le Gougne was accused of promising to give Elena Berezhnaya and Anton Sikharulidze of Russia higher marks than Jamie Salé and David Pelletier of Canada regardless of how each pair performed. That scandal led to media outrage, to Salé and Pelletier being awarded golds as well (seen above), to suspensions for Le Gougne and French federation head Didier Gailhaguet, and to a whole new international judging system.
Since then, there’s been a whole lot of talk about how clean figure skating is, and how the judging is now fair. But a new BuzzFeed News analysis by John Templon and Rosalind Adams found that there’s still a whole lot of home-country bias, with a statistical analysis of scoring data covering over a year and over 200 judges finding that 27 (7.4 percent) of those judges consistently gave higher scores to skaters from their country than the average of scores from other judges. And of those 27 judges, 16 are in PyeongChang, comprising 33 percent of the 48 judges at these Winter Olympics.
To prevent false positives, this study only flagged judges whose scores for skaters from their country exceeded the average by such a margin that the odds of that happening randomly are less than 1 in 100,000. So this isn’t even necessarily every indication of home-country bias; it’s only naming the names of judges whose apparent home-country biases stand up to a significant degree of statistical scrutiny. The data also suggests that this is a notable effect, as shown by the chart seen at right below; that shows the average boost given by many top countries’ judges to their own skaters.
That isn’t necessarily from those judges consciously choosing to boost their own country. The piece notes that this statistical analysis can’t determine an individual’s motivation, and that experienced skaters, coaches and judges have suggested some more favorable reasons why that could happen, such as additional familiarity with their country’s skaters or a preference for the styles favored in their country. Unconscious biases matter too, and we’ve seen that with data-driven studies of other officiating, including those revolving around home-team bias in everything from soccer to basketball to the NFL.
It should be noted the current scoring system uses a “trimmed mean” procedure, where the highest and lowest scores are discarded and the other seven are averaged. That’s not a bad step in helping to minimize home-country bias. And the International Skating Union, the sport’s governing body, has its own algorithm to try and find problematic results. So those two things can perhaps catch anything that’s particularly over the top. However, there’s an opportunity for more subtly-problematic boosts. As per BuzzFeed’s analysis, the ISU algorithm only flags around one percent of all scores on the technical side, and even less on the artistic side.
This system also doesn’t compare a judge’s scores over time, but treats each event’s scores in isolation. And the elimination of the highest and lowest scores also doesn’t solve the problem if the home-country judges make sure to stay within the realms of their colleagues, but on the upper range of those scores. So if a judge consistently gives scores to skaters from their country that are higher than those of many judges, but close enough to at least one other judge, they’d be impacting the results without being caught by this system. And that’s what many who spoke to BuzzFeed mentioned:
“The corridor is so big that it’s almost impossible to get flagged,” said a former ISU technical committee member.
“The judges really know how do it — how to play within that band,” said another former ISU technical committee member.
“It’s quite a huge margin that the ISU allows,” said a former high-level member of a national federation.
So, this is an issue, and it’s something that could possibly impact who winds up with figure skating medals in these Olympics. And it shows that the new system hasn’t necessarily fixed everything. Some aspects have been improved over time, of course, from the higher focus on technical judging (which has more stringent and repeatable criteria) to the 2016 decision to again publish judges’ scores (making this sort of analysis possible). But there are further steps that could be taken, from having judges not judge their country’s own skaters to expanding the ISU algorithm to evaluate data over time and wider the range of results it flags, and maybe that would improve things.
There’s always liable to be some level of bias in a judged sport, but it’s worth considering where that bias comes from and how it can be reduced. And this analysis is certainly helpful for spotlighting some of the issues that are still present, and suggesting that the current system is far from perfect.
