Broadcasters are required to measure quality in sample of programs from three genres: news, entertainment and chat shows. The dimensions of quality to be measured are:
- The average speed of captioning
- The time-lag (or latency) between the audio and captions appearing
- The number and type of errors
Ofcom chose the NER Model as the one that broadcasters should use to measure caption quality. Developed by Pablo Romero-Fresco and Juan Martinez of Roehampton University, the model is specifically designed for live captioning, and divides errors into three categories: serious, standard and minor. Captions which achieve 98% accuracy or higher using the model are considered to be of acceptable quality. For Ofcom’s measurement exercise, the results obtained by the broadcasters using the model are checked by a team from Roehampton University.
The main results from Measuring live subtitling quality: Results from the first sampling exercise (which looked at samples of programs from October-November 2013) are as follows:
- Accuracy for news programs was generally higher than 98%, however it was often significantly lower for chat shows.
- The average time lag for captions was 5.6 seconds, which is higher than Ofcom’s recommended target of 3 second, but could go as high 24 seconds.
- The average speed of captions was generally within Ofcom’s guideline of 160–180 words per minute.
Ofcom has also encouraged broadcasters to use block captions as much as possible, especially when programs which have been captioned live are repeated. (Block captions appear as one or two lines of text, rather than one word at a time, and are much easier for viewers to read). The report found that all the major networks – the BBC, ITV, Channel 4, Channel 5 and Sky – are willing to use block captions when possible, and in many cases are already doing so.
The findings of Ofcom’s first quality exercise support the recommendations made in Media Access Australia’s white paper, Caption quality: International approaches to standards and measurement, which was released in March 2013. The white paper examined various models which have been developed to measure caption quality, and identified the NER Model as a potentially promising one, although more testing of it needed to be done. It also advocated the use of block captions whenever possible.
“This Ofcom report is hugely encouraging,” said Media Access Australia’s Project Manager for television and author of the white paper, Chris Mikul. “There has been endless argument around the world about whether it’s feasible or practical to measure the quality of live captioning in a meaningful way. This shows that the NER model is a robust one, and 98% accuracy is a reasonable target for broadcasters to aim for. Ofcom should be applauded for launching such a useful initiative, and its report is essential reading for anyone interested in the subject of caption quality.”
Top of page