Facebook's AI Invents New Language. End Times? Hardly.
We interrupt my series on AI ethics to talk about a little bit of clickbait I came across this morning. I conjecture what may have happened in Facebook’s AI system, and why we shouldn’t be concerned.
This morning I saw that numerous news organizations have reported about Facebook’s AI Research group (FAIR) have shut down their new AI negotiation system because it “invented a new language”. Here’s a link to one such article. I think it’s irresponsible that news outlets are reporting this as if it’s some sort of profound new ability that AI systems have, when in reality it’s just a run-of-the-mill failure of machine learning. So despite not having access to a complete description of their architecture, I wanted to speculate what may have happened, and why we shouldn’t be alarmed.
Facebook is working on a platform that can allow users to interact with businesses on Facebook messenger via chatbot. The chatbots will need to do some sort of negotiating, so Facebook started doing experiments with chatbot-human and chatbot-chatbot negotiation.
While it doesn’t look like Facebook has released a detailed description of its architecture beyond the blog post that this article references, here’s my attempt to explain what probably happened in the absence of all the facts.
Facebook AI Research (FAIR) is well-known for its use of neural networks (the head of the department, Yann LeCun, is kind of a neural network rock star). A neural network is just a simple function approximator. As you might recall from grade 11 math, a function is just something that takes inputs and gives back outputs (usually, the inputs and outputs are numbers or vectors). Neural networks are just sets of neurons, which are simple functions. You hook the neurons up to each other in layers, so that all the neurons in one layer receive different weighted sums of the outputs of the previous layer as input. Here’s the key: the weights are just numbers, and they’re tunable. Hook up all these neurons together, and you can compute one big, complicated function. Which function you compute depends on the weights. As a result, you can train a neural network to do all sorts of interesting things, if you do it right. One of the most basic forms of training is called backpropagation. You give the neural network a desired input, and it computes an output according to its current weights. If you know what the desired output is, you can compute the difference, and then move all the weights a little bit in the direction that reduces that difference. Do this for long enough, and your neural network can do things like compute whether there is a dog in a given picture.
Conventional neural networks compute a function where the output depends only on the current input. To simulate something like dialogue, you need something like a recurrent neural network (RNN), which can store a little bit of information about previous inputs. RNNs usually do this by allowing some neurons to receive their own previous inputs/outputs as input. Because RNNs can store some information about previous inputs/outputs, you can use them to create whole sequences of outputs. This is why you can use them for something like dialogue, and this is why I think that FAIR used RNNs for their negotiation task.
Keep in mind that RNNs don’t really understand English words. To be used by an RNN, the English words have to be converted into vectors (lists of numbers). The neural network can then output other vectors, which we then turn back into English words. It’s as if someone just told you, “When I say ‘glip’, you say ‘glop’”. You have no idea what “glop” means, but you can effectively compute what you need to compute.
So it sounds to me like FAIR trained their RNNs on human-human negotiations. They gave them sequences of words to reproduce, like “I can only give you four pieces of string”. They trained them using a version of backpropagation that works for RNNs, called backpropagation through time. At the end of this process the AI was able to say the sorts of things that people say when negotiating, but wasn’t capable of effectively using them for bargaining. If you train it on enough real text and you give a long enough time to train, these RNNs can sound somewhat coherent (although when I tried to train an RNN on the speeches of Mormon leaders, the results were only semi-coherent).
They also did something new, which they call dialogue rollouts. Now that your chatbots can mimick the English language a bit, you can try a bit of reinforcement learning (RL). The gist is that the programmer designs some sort of “reward function”, and then the AI takes sequences of actions in an environment so as to maximize their long-term sum of rewards. In this case, the programmers designed reward functions based on what sorts of things the AI “receives” at the end of the negotiation, where each item has a certain value to the AI. The actions that the AI performs are the bits of dialogue.
In dialogue rollouts, the AI generates a few potential things to say using its RNN. For each of those “candidate utterances”, it then takes the perspective of its conversation partner and does the same thing. In this way, it generates a tree of possible conversations, each going all the way to the end of the conversation. It then computes how much reward it will receive at the end of each of these potential conversations, and then chooses the utterance that will maximize the expected reward. The conversation partner does the same thing with its reward function. At the end of the conversation, both negotiators receive an actual reward (just a number), and can use those rewards to train their RNNs to be more likely to say sequences of words that lead to a good outcome (using, presumably, a sort of backpropagation through time). FAIR could use this method to train its AI with other similar AIs.
The problem with training your AI in conversations with similar AIs is that the AI is just learning to produce sequences of words that will ultimately maximize its rewards, and the only thing it’s able to change is the weights of its RNN. The ability of the RNN to “speak English” is quite fragile, and if the weights move around too much, the result will become incoherent. So if you have two AIs “negotiating” with each other, and the only thing that matters is what things they get to take at the end of the conversation, little bits of incoherence that happen to lead to good outcomes wil be magnified.
Imagine you’re in the situation that one of these AIs was in. You’ve been learning a new language (say, Vietnamese) by mimickry alone. Maybe you’ve just been listening to hours and hours of Vietnamese music, and you can kind of say things that sound Vietnamese (even though you don’t really know what they mean). Now you’re ready to start practicing your Vietnamese negotiation skills in preparation for that big business trip to Vietnam. You find yourself a negotiation partner, and you start speaking your mimicked Vietnamese to them, still not really knowing what you say. You occasionally stumble. Unbeknownst to you, your conversation partner also doesn’t really speak Vietnamese. They’re also just faking it. So when you say “glip glop gloop” and your conversation partner doesn’t seem to notice a difference, at the end of the conversation you get that spatula you really want, you start to think to yourself, “maybe saying ‘glip glop gloop’ helped me get that spatula”. And meanwhile your partner is doing the same thing, and before too long you’re just saying gibberish that happens to get you both decent outcomes. After thousands and thousands of these conversations, all the negotiators are speaking what might be considered a new language, and they’re pretty good at using that new language to get what they want. Maybe they even implicitly represent concepts like numbers in that new language! But they don’t really have any understanding of what the words and phrases of that new language mean. All they know is that saying “glip glop gloop” really helped them get that spatula, and that swapping out “gloop” for “glorp” can get them TWO spatulas.
This is more or less what happened with FAIR’s RNNs. They tried to overcome this problem by continuing to train the AIs on the human dialogue at the same time as they had them negotiate with the other AIs, in the hope that the result would still sound like English. This is like listening to a song in Vietnamese between each of your negotiations. The hope is that you’ll keep speaking mock-Vietnamese AND speak it in a way that gives you good stuff, but there’s no guarantee that these two goals will come together in quite the right way. It sounds like they didn’t in this case.
So did the AIs invent their own language? In a manner of speaking, I suppose they did. They didn’t know they were inventing it, and it’s not something that they can use to rise up and destroy us. No amount of gibberish negotiation will get you to a point where you’re suddenly talking about assassinating the Prime Minister of Vietnam. FAIR did turn off the project, but not because it was potentially dangerous. They just turned it off because the weights weren’t converging to anything resembling English, so they couldn’t use it to communicate with real users. You can bet that they’ll keep working on this project, trying to get around this problem, and not because they’re immoral. This is not a moral problem. This is just a temporary machine learning setback.