The report gives percentage scores for the accuracy of 21 news, sport and discussion programs which were captioned live. The scores were obtained using the NER Model developed by Pablo Romero Fresco of Roehampton University and Juan Martinez of the University of Barcelona.
As levels of live captioning have increased in recent years, in Australia and internationally, finding an accurate and efficient method for measuring its quality has become increasingly important. It’s a task that is problematic because the seriousness of errors varies so greatly. Minor errors may have little effect on the comprehensibility of the captions, while a single major error can change the meaning of a story completely. Live captioners also routinely modify language when creating captions, removing unnecessary repetitions and cleaning up grammar to make the captions more readable.
The NER Model takes these factors into account, making a distinction between appropriate errors introduced by captioners and genuine errors (which are divided into serious, standard and minor). The model was last year adopted by the UK’s communications regulator, Ofcom, which currently requires broadcasters to submit six-monthly reports on the quality of a sampling of live-captioned programs. Ofcom recently released the results of the first of these reports.
Red Bee found that most of the 21 programs it reviewed using the model scored accuracy rates of 99%. The lowest score, 96.88%, was for a sports program. In its report, Red Bee Media notes that live captioning of sport is different to captioning for other types of programs. This is because captioners routinely omit some of the commentary because what is being described is evident from watching the vision, and therefore superfluous.
The report concludes by stating that Red Bee Media’s “experience so far with the NER Model has shown it to be an effective tool for providing realistic accuracy assessments on a range of outputs”. And, while preparing the reports is too time consuming for them to be done for the bulk of its output, the company’s “intention going forward is to conduct quarterly reviews using the NER Model, with monthly spot checks, all conducted randomly across a range of outputs”.
Earlier this year, Media Access Australia produced a white paper, Caption Quality: International approaches to standards and measurement, which provides detailed information about the quality measurement models that have been developed, and the approaches taken to caption quality in different countries.
Top of page