A new method can automatically identify memes and make them intelligible via existing assistive technologies, researchers report.
People with visual impairments use social media like everyone else, often with the help of screen reader software. But that technology falls short when it comes to memes, which don’t include alternate text, or alt text, to describe what’s depicted in the image.
Memes are images that are copied and then overlaid with slight variations of text. They are often humorous and convey a shared experience, but “if you’re blind, you miss that part of the conversation,” says Cole Gleason, a PhD student in Carnegie Mellon University’s Human-Computer Interaction Institute (HCII.)
“Memes may not seem like the most important problem, but a vital part of accessibility is not choosing for people what deserves their attention,” says Jeff Bigham, an associate professor in the HCII. “Many people use memes, and so they should be made accessible.”
Memes largely live within social media platforms that have barriers to adding alt text. Twitter, for example, allows people to add alt text to their images, but that feature isn’t always easy to find. Of 9 million tweets the researchers examined, one million included images and, of those, just 0.1% included alt text.
Gleason says basic computer vision techniques make it possible to describe the images underlying each meme, whether it’s a celebrity, a crying baby, a cartoon character, or a scene such as a bus upended in a sinkhole. Optical character recognition techniques can decipher the overlaid text, which can change with each iteration of the meme. For each meme type, it’s only necessary to make one template describing the image, and the overlaid text can be added for each iteration of that meme.
But writing out what the meme is intended to convey proved difficult.
“It depended on the meme if the humor translated. Some of the visuals are more nuanced,” Gleason says. “And sometimes it’s explicit and you can just describe it.” For example, the complete alt text for the so-called “success kid” meme states “Toddler clenching fist in front of smug face. Overlaid text on top: Was a bad boy all year. Overlaid text on bottom: Still got awesome presents from Santa.”
The team also created a platform to translate memes into sound rather than text. Users search through a sound library and drag and drop elements into a template. This system was made to translate existing memes and convey the sentiment through music and sound effects.
“One of the reasons we tried the audio memes was because we thought alt text would kill the joke, but people still preferred the text because they’re so used to it,” Gleason says.
Deploying the technology will be a challenge. Even if it was integrated into a meme generator website, that alt text wouldn’t be automatically copied when the image was shared on social media.
“We’d have to convince Twitter to add a new feature,” Gleason says. It could be something added to a personal smartphone, but he notes that would put the burden on the user. The researchers are currently working on related projects, including a browser extension for Twitter that attempts to add alt text for every image and could include a meme system. Another project seeks to integrate alt text into the metadata of images that would stay with the image wherever it was posted.
The researchers presented their work this year at the ACCESS conference in Pittsburgh.
Additional researchers involved in the project are from Carnegie Mellon University and Columbia University.
Source: Carnegie Mellon University
