Captionbot is an application that attempts to identify the content of an uploaded image and provide a description in the form of a caption. It uses a combination of existing image tags and metadata (information about the image and its context that is invisible to the viewer but attached to the image) with an emotion recognition system and sampling from databases to create a best guess of what the image is.
A more ambitious project named Seeing AI is being worked on by Microsoft developer Saqib, who is blind. This technology, which consists of special glasses and linked software, aims to identify objects and people in real-time and includes additional information such as their emotive state by analysing clues such as facial expressions.
The audio described version of the video shows the developer using the Seeing AI technology on his way to work and in his workplace.
This is an example of wearable technology to assist people with disabilities. Early examples included Google Glass which was demonstrated displaying captions and also identifying text and reading that out to blind people.
According to Media Access Australia CEO Alex Varley, these technologies show good promise, but are not foolproof enough yet for safe everyday use.
“The sophistication of these products is amazing when you look at the difficult tests being used, such as trying to read a person’s emotions in a split second. However, I would be very wary of using this in any critical situations. Imagine the consequences of getting the wrong response when you are trying to decide if somebody is happy or upset with what you have just said.”
Varley noted that these new developments are making accessibility more automated and easier, just like the automatic captioning on YouTube has helped people by providing captions that can be corrected and included on more videos.
“Whilst the products are far from perfect, they can speed up the process of creating accessible documents, images and videos and are very useful tools if people use them while understanding their limitations.”
Top of page