Language Revolution

21/10/2020 38

OpenAI was founded in 2015, which was an important milestone among others, for the exponential advance of artificial intelligence in these last years. OpenAI was created as a non-profit organization, with the mission of sharing and extending the knowledge of artificial intelligence. One of the founders is Elon Musk, who pursues the goal that the development of general artificial intelligence benefits all of humanity, that is why knowledge is shared and open.

One of the first developments in which OpenAI has worked is in the research of NLP (natural language processing). With these researches they intend to improve the capacity of interpreting the natural language, used by people, and that machines can understand it better and better. This will allow important advances in the area of neural networks, as well as provide important improvements to applications such as chatbots, among others.

Its first advances with language were the launch of GPT (Generative Pretrained Transformer), based on generative neural networks that have been trained to predict the next word, within a sequence of words, in an unsupervised way. This is a big change, since current NLP systems require supervised training, i.e. texts that are annotated.

A so-called semi-supervised model, consisting of unsupervised training and fine-tuning done in a supervised manner, is used for the development of GPT.

GPT-2 was the next version announced in January 2019, with 10 times more training data as well as 10 times more network parameters, it generates automatic texts, from a text written by the user. The application is also capable of generating summaries and answering questions about the content of the text. Although the model is quite accurate 95%, in comparison to traditional models, the first doubts of the founders of OpenAI arise here.

First of all, it does not have enough precision to work in an autonomous way. Secondly, it can be used to generate unsuitable texts that contribute to hatred or certain terrorist actions or to the generation of false news (usually called fake news). Finally, language models can be biased and this adds a new challenge to maintaining the fairness of artificial intelligence.

It is for all these reasons that OpenAI decided not to release the whole GPT-2 model initially, and to create responsible publishing standards together with the scientific community of AI. Another of the advances in the treatment of the language, was last year when Blender was announced. Blender is a new Facebook chatbot, which promises a better user experience, since it is able to talk about any topic in a pleasant way.

Blender’s ability comes from the immense amount of data used for its training, 1500 million Reddit conversations. He then perfected himself to understand emotions in conversations, and to establish conversations between people with different personalities to teach him about personality.

The model proved to be much more efficient than those designed so far, but still does not come close to the conversational capabilities of a human being. So far, it has been tested with short conversations; researchers suspect that in longer sentences the conversation would no longer make sense. Since the training is also done with data sets usually from social networks, they may also have problems with bias or abusive language.

In 2020 GPT-3, the third version of GPT, is announced, with 175 billion parameters, when GPT-2 used 1.5 billion. OpenAI published the specifications in May but it was not until early July that it opened the platform for developers to test it. The tests have revealed how it is capable of writing sonnets, creating texts for web pages and summarizing research articles. The big leap is that he is also capable of programming basic code or translating texts into legal language, or doing sums, the most striking thing is that these are tasks for which he was not intended.

In short, the program knows that it has to give an answer to what it is asked for, and it looks for an answer until it finds it. Experts are not surprised, since the potential of neural networks is very high. If we think of neural networks as an attempt to emulate human reasoning, which does not need to see an object thousands of times to classify it, with this example it seems that progress is being made.

In short, when GPT-3 receives an entry that it does not know, it goes to its huge memory to find an appropriate answer, but it is also true that if at any time the text loses consistency or sense, it will not know it.

GPT-3 is undoubtedly a great advance in terms of NLP and neural networks, and even today the impact it can have is not known. The major technologies continue to research this technology and spend millions on its improvements. Currently the pattern is available, everyone can use it, but the difference is the computing capacity available, which not every user has. Only to train GPT-3 they spent about 5 million dollars.

The development of GPT-3 is a revolution for the language, who knows if in the future journalists will receive texts written by this technology that they will only have to retouch, will the same happen with books, will it also be a support for programmers, helping them to develop faster and more efficient code.

We cannot forget either, that with this technology you can invent stories, contribute to deepfakes or increase biases. Today, researchers do not know how far the scope of this technology will go, but it is clear that a balance is still needed between what technology can do and what technology should do.

The recent agreement between OpenAI and Microsoft for the development of this technology as a preferred partner is surprising. Let’s not forget that the emergence of OpenAI, had the objective of making AI available to everyone to benefit humanity and now the benefit has been put in the hands of a single company.

It is also true that research in the field of artificial intelligence still requires a lot of investment and it seems that Microsoft is willing to do so. Now more than ever, it is important that OpenAI maintains its mission, that the development of general artificial intelligence benefits everyone.