Adrian Watkins

Google announced the creation of Meena in a new research paper, claiming that it is the most advanced, chatbot ever built.

According to Google, Meena can carry on a conversation at a level that feels far more like talking to another person than any existing chatbot. To prove it, Google’s researchers also created an entirely new test for measuring how much sense chatbots make in their conversation.

SOCIAL MEDIA NEURAL NET

The scientists behind Meena built the chatbot to be responsive to people’s messages, to stay on topic, and to behave as much like another human being as possible. To reach those goals, Meena was built as an open-domain chatbot. Unlike voice assistants such as Google Assistant and Amazon’s Alexa, Meena can theoretically talk about anything, not just the topics already programmed into it. The chatbot was built with Google’s Evolved Transformer program, using 2.6 billion parameters to make its way through 341 gigabytes of online text from social media networks. Meena was ultimately trained on 40 billion words over 30 days.

To actually determine how well Meena performs relative to other AI chatbots, Google created the Sensibleness and Specificity Average (SSA). The SSA is a test for charting how comprehensible the chatbot is and how well it stays on topic.

Meena scored 79%, which is barely below the human score of 86%. Other AI developers have their own ways of measuring performance, but Meena crushed open-domain chatbots like Mitsuku and Cleverbot in the SSA test.

It’s not a comprehensive or fully authoritative exam though. The samples provided show Meena to be capable of conversation, but it’s hardly a scintillating discussion. The conversation sounds like two people meeting at a networking event and running out of things to say so that they scramble for any topic.

ADDING HUMANITY TO VOICE ASSISTANTS

A chatbot that can carry on an open-ended conversation is potentially very powerful. Google’s investment in Meena is only one of many efforts. Building a social AI that can hold up a conversation is the impetus for the Alexa Prize. The idea is that by making voice assistants better at conversation, people will be more likely to integrate them into their lives, buying and using smart products.

And there is evidence that people want a voice AI that reflects their own style of communication. A recent Apple study found that people are more likely to trust a voice AI that can mimic them. Combining chatbot AI and audio cues to imitate the speaker should make for a voice assistant people are very comfortable with. Even with their current limitations, voice assistants have already been shown to reduce the loneliness of older people, according to a British study.

Google isn’t likely to start integrating Meena into Google Assistant without some more work on how it could be used, but Meena’s model could help future variations of the voice assistant converse more like humans. The pun about cows going to Hayvard for college wasn’t a good joke, but it certainly felt human.