GPT3 has millions of labels. Every vocabulary term is a label. It’s equivalent to supervised learning in architecture. The “self-supervised” business is mostly spin to make it sound a bit more novel. People have been predicting the next word for ages (Turing did this).
Input: <previous words of article>
Label: <next word>
Your point is well taken that the number of input data points is also important when considering the complexity of the problem. In this case however the number of data points more or less exactly equals the number of labels.
(About Me: the first year+ of my PhD was focused on large scale language modelling, during which transformers came out.)
Input: <previous words of article>
Label: <next word>
Your point is well taken that the number of input data points is also important when considering the complexity of the problem. In this case however the number of data points more or less exactly equals the number of labels.
(About Me: the first year+ of my PhD was focused on large scale language modelling, during which transformers came out.)