Respected expert Dr. William Sands responds to Andrew Thorton’s conclusion that at the execution score of the best gymnasts in the world has been in decline since the start of the open code. And, partly as a result, that execution scores of the top group of gymnasts in the world are ‘boxed’ between 8.5 and 9.0.
While the trends appear obvious, be aware that there is a fundamental flaw in evaluating judges – the lack of a “gold-standard” with which to compare. People have tried for decades to establish an approach to evaluate judges. Gymnastics judging immediately faces four paired problems: reliability and validity – and – cheating and incompetence. Some have argued that the actual score doesn’t matter much as long as the judges get the athletes in the right order (a sort of validity). Others have studied scores to see if the judges agree among themselves (inter-rater reliability). I would argue that the first is the most important, that the right athlete wins. The second, inter-rater reliability, is also important because when judges agree there is a tacit understanding that they’re seeing the same thing.
Unfortunately, what you often find is that people can rationalize athlete placements just about anyway they want. Both the advocate of a score and the detractor’s arguments rapidly collapse to circularity because neither has a gold-standard and must rely largely on opinion. The problem of inter-rater reliability is the very old game of “stay in range.” I think judging rules also have to combat the second pair of problems: cheating and incompetence. Fundamentally, these are unlikely to be changed by statistics.
If judges are supposed to be basically like court stenographers, then I would suggest that the “stream” of deductions and other information be recorded in terms of time on their computers. Clearly the technology now exists for this, and has existed for some time. Moreover, the tenets of systematic observation have been well known in scientific circles for decades. In this way, judges not only have to stay in range with their total score, they also have to be deducting for the same things (as seen in their time-based stream of data). I think editing should be allowed, in other words, the judge should be able to go back in the stream of his/her writing and add-in or change things that he/she couldn’t write fast enough or he/she simply changed his/her mind at the end. The original data stream is always preserved however, so that any changes are recorded as changes. However, once completed by the judge and scores are “locked” then the overall analysis of judges can proceed in several dimensions simultaneously and could serve as both an evaluative and educational tool for judges. For fans, the stream could be displayed in real time and finally reduce some of the mystery of “how on earth did they get that score” for fans.
I realize that to think this is likely to be implemented is naive. However, if you check the judging literature, you’ll rapidly find that these same issues have plagued judging since at least the 1951 (in my personal library), and probably before that.
All the best.
Wm A. Sands, PhD, FACSM, C-ARS, NR/WEMT
Bill’s now Director at the Monfort Family Human Performance Research Laboratory in Colorado.
On Vault I think we should judge exactly like they do in Diving. No paperwork. One number for execution is flashed immediately and transparently in real time.
On the other apparatus it might be possible to use an iPad or keyboard to record in real time. I recall an American MAG judge who used a keyboard to both record skills and execution on Horizontal Bar. It looked very accurate to me.
Bill’s citations on the gymnastics judging problem(s) are posted in the comments.
… “The gross motor development skills, that’s what we do,” said Ben Fox of Bart Conner, who, along with Ann Goff, gym teacher at Lincoln, originated the idea to offer these free classes to Norman Public Schools’ students who are identified on the autism spectrum.
“It’s great for motor and language skills, and they’re having fun while learning. That’s what kids like to do … run … jump,” Fox said, motioning toward one of the students, Isaac, as he jumped in circles on the trampoline …
Online I’m seeing a lot more Women’s Collegiate gymnastics content, especially videos from training. Interest is picking up in advance of the season start in January.
Andy Thorton has been holding fire to the feet of FIG judges better than anyone else.
He’s just posted some disturbing statistics on Execution score trends at the the international level. Here are a few of the highlight quotes from his analysis:
scores seem to be “trapped” between an 8.5 and a 9.0 regardless of the performance …
With the exception of men’s vault, it would appear from the numbers that gymnasts in general are anywhere from three to seven tenths sloppier today than they were in 2006. Is this a fact, or a function of something else going on? …
Judging in general has become much more harsh, much more unreasonable …
Men’s high bar judging has perhaps become the most outrageous and unpredictable; sometimes the cleanest routine receives an 8.7 and sometimes the sloppiest routine receives an 8.9, but the rule is no one gets above a 9.0. I miss the days even four years ago when 9.5’s and 9.6’s were given to clean routines …
As we strive for a resurgence of artistry, stricter rules are not the solution; in fact, they’re part of the problem. Today’s execution standards have not created less subjectivity in our sport; they’ve created MORE subjectivity …
I miss the days when judges felt free to throw out a 9.8, a 9.9, or even a 10.0 when a gymnast was magnificent …
He doesn’t mention this time, but has in the past, that one of the main causes of “boxed scores” (lowest Execution score too close to highest) is that judges fear being out-of-range.
It used to be that Women’s Gymnastics was far more guilty. (They’ve always listed many subjective deductions that are near impossible to evaluate consistently.) But what’s going on in Men’s Gymnastics? … Andy’s stats show the MAG execution scores dropping even more precipitously than WAG.
I have no confidence that either FIG Technical Committee has the leadership to fix boxed scores. That means, in most cases, the highest difficulty score will continue to win. There’s no incentive to try to improve execution.
Next? … I’d love to see Andy or THE ALL AROUND do a more detailed statistical analysis on this trend.
It’s a nice summary, MAG and WAG Junior Olympics as well as mention of the easier WAG PREP-OP divisions in his region.
In the USA Gymnastics Junior Olympic program, there are set rules throughout the country for each level.
For the lower levels (2-6 for girls and 4-7 for boys), there are specific routines set by USAG, so every kid in the country will be doing the same routines at these levels (though the boys program allows certain bonus skills to be performed in some of the compulsory routines for a couple extra tenths).
For the upper levels (7-10 girls, 8-10 for boys), gymnasts have their own individual routines. For girls, these levels still have very specific requirements, while the boys’ upper-levels are much more open-ended. …
But it can cause problems at the gym. Especially if both kids are competitive gymnasts.
Do you have a policy on that? … If so, leave a comment.
Coach Sasha on Make it or Break It will kick the girls out of the club for dating. At Altadore, years ago, I recall we once had a rule where all boys had to stay physically on one side of the long pit. The girls on the other side. … Like that worked.
With the teenage girls I sometimes invite the boyfriend to come to workout. Once. Then give him so much volunteer work to do while he’s there … that he never comes back.
🙂
Gymnastics Coaching.com 