Presentation: Does measuring caption quality really make any difference?

Thursday, 22 November 2012 16:33pm

Our CEO, Alex Varley, is presenting at the Languages and the Media Conference in Berlin this week. Below are the slides and summary of his presentation questioning whether measuring caption quality leads to a better television experience for viewers.


SlideShare includes an auto-generated text transcript, though it should be noted that this does not include alternative text for images within slides. If you require an accessible version of the presentation please contact us.

Presentation summary

The numbers of the points below correspond with the slides.

  1. Alex Varley is the CEO of Media Access Australia, the independent not-for-profit organisation that develops access to media using mainstream technology.
  2. In the future will subtitle (caption) quality be a fully automated process?
  3. Will decisions be made in a cold, robotic way?
  4. The calculation of quality would be reduced to pure numbers.
  5. The punishment for breaches of quality would also be automated.  Is this a vision of the future that people really want?
  6. It is so much easier if you measure things.  Numbers are exact and can be defined clearly. You can have legal tests for what exactly things mean.  It is easy to understand because you don’t have a need for interpretation. You know when you get it wrong.  Regulation can be a simple “yes I met the standard” or “no I didn’t meet the standard”. The rules are clear for everybody, aren’t they?
  7. The problem with this approach is that for the supplier, broadcaster and regulator it becomes just a process. The point behind the process, which is to provide a good service to a viewer, becomes lost. Surely with television, the viewer is the most important person involved?
  8. One of the problems with measuring things for subtitling is that there are different rules and different interpretations in different countries.  Subtitling companies operate in lots of different countries and they have to track what the local rules and preferences are.
  9. Some rules are pretty much universal. Accuracy includes spelling; display issues; punctuation.
  10. Interestingly, the range of accuracy written into rules is quite wide. For non-live the accuracy is usually 100% but the moment it is deemed live that can drop as low as 75% (which is for French Canadian live subtitling). The norm in English-speaking countries seems to be around 95%. What does this really mean? Do you refer to the soundtrack and count everything that is missing? Or is that clever reduction by the subtitler to make it easier to understand? Does a spelling error actually destroy understanding? “Angela Murkel” is wrong but in the context of the story, does the viewer understand that it is referring to the German Chancellor and therefore less of an impact?  Having a scrambled word that is key to the understanding can be more damaging to a viewer’s experience, even if that one error keeps the entire program within the stated accuracy requirement.
  11. Time delay is recognised as a major issue. Having synchronous subtitles is the best.  As many studies presented at this conference have said, eye tracking research shows that up to 85% of viewing time can be taken up reading a scrolling, non-synchronous subtitle. Some countries have opted for delaying some broadcasts to cover this (Holland and Norway?); others say that it would cause legal problems to do so. The reality is that a scrolling, live subtitle is going to be more like 5-7 seconds behind the soundtrack.
  12. What is a live program? This depends on how it is defined. Typically, broadcasters and suppliers want to define a live program as far from broadcast time as possible.  This extends up to 96 hours before broadcast (Canada again). The problem for the viewer is that it is cheaper to provide live subtitles than prepared subtitles (usually) so the market will encourage the maximisation of live subtitling, even for programs which could be prepared. Is it also fair to treat a small regional broadcaster that might have one or two live segments in its news each year, in the same way as a national broadcaster with many hours each day? That is about flexibility of supply arrangements. 
  13. Reading speed had its origins in the supposed ability of deaf people to process information and that “professional” narrators spoke at 180 wpm. The reality is that verbatim is the most popular and actually captures everything that is being said, allowing the viewer to decide what to read or not. We did some anecdotal studies of sports programs subtitled at different speeds about 10 years ago and the conclusion was that slowing it up or going verbatim (event at 280 wpm) made little difference to viewer understanding and preferences. 
  14. This is the problem, because this is what the broadcasters and their suppliers will get judged by and that is what they focus on – it’s human nature.
  15. In Australia there is a slightly different approach that is being taken. The approach is to focus on viewer comprehension and less on measurements. The regulator, the Australian Communications and Media Authority will be releasing its final version in early December.
  16. The key factor is that most complaints are around live subtitling.  So the approach is to minimise the amount of live subtitling, even during “live” programs! This recognises that some live programs have elements that prepared and can be subtitled prior to broadcast and cued in a synchronous way.
  17. It is also about expectations and reality. So live sport should have live scrolling subtitles.
  18. Live news, however, could have a mix of live and synchronous. The Seven Network in Australia provides news subtitles this way and the deaf consumers frequently cite this as being their preference for news subtitling.
  19. What stops this from happening? Training is a major factor. Can a well-trained subtitler make errors at the required rate or do they subtitle at a consistent level (yes) and the training should focus on ensuring that level is high? If you don’t have preparation time then accuracy will suffer. Is the output purely for broadcast or is it going online as well?  Is this the only live program or does the supplier/broadcaster do many? A regulator could forgive a late-arriving program being subtitled live on one occasion for a small regional broadcaster, but for a large network, different expectations are reasonable. What approach does the regulator take? Are they consistent and clear about the breaches? Are they quick in decision making? Do they identify what caused the problem, rather than there just being a problem? Finally, has the market got to a point where a line is drawn on how low pricing can go?
  20. What can a pro-active regulator do?  Expectations – should a viewer see scrolling on live sport and a mix on news?  Are broadcasters learning and fixing repeated errors or is there no process for this?  Is there consistency in supplier methodology and basic standards?  Does the regulator act clearly and consistently and publically?  Is there a process to identify systemic issues and work on solutions to fix them?  Does the market work properly, or is a cost-cutting, poor quality operator still given work?  Do the broadcasters understand the rules and the impact of not meeting them?
  21. A movie was live subtitled in Australia, but it was clearly a one-off problem.  There was a lot of focus on it and didn’t really achieve anything in terms of dealing with bigger issues in live subtitling. 
  22. This is the most cited reason in the industry for more live subtitling, the race to the bottom and quality suffering.  The only way is for the regulator to make it clear that this will not be tolerated and to punish swiftly and publically.  Then the suppliers will have a baseline that cannot be undermined.  We will wait and see if this really happens.
  23. This presentation was never intending to totally discard measurements, but really to get you thinking about the shortcomings of focussing on them alone.  The real message is that the ultimate quality judge is the viewer.  We shall see whether the implementation of the new system in Australia makes any difference – stay tuned.  Thank you.

Read Alex's interview on measuring caption quality with the organisers of the Languages and the Media conference.

Top of page