The BIGGEST problem with Artistic Gymnastics judging is the Execution Score range between best and worst.
That range is too small — therefore it’s mostly Difficulty Score that decides ranking.
This article doesn’t address the BIG issue, but it is interesting to see how FIG is evaluating their judges.
Dr Hugues Mercier speaks exclusively to Olympics.com about his 10-year project that evaluates judges’ performance in disciplines such as artistic and rhythmic gymnastics. …
“Judging a routine in pommel horse is intrinsically much more difficult than judging a routine in vault, and it’s very important for the tools we provide to be fair. So, I always tell the judges that the mathematical tools that we use take into account that judging in pommel horse is difficult.
“So, on a 7.5 routine, which is a good routine on pommel horse, a judge who makes an error of, let’s say 0.3, 0.4 is completely normal. In vault, giving 9.5 instead of 9.1, it’s a very, very, very large error… it’s an outrageous difference. So, this is all scaled, so that when we analyse the accuracy of judges, we take into account the apparatus on which they judge.” …
Olympics.com
Gymnastics Coaching.com 