Reading the section on [synthetic data](https://huggingfacefw-blogpost-fineweb-v1.static.hf.space/di...) was eye-opening for me. The hockey-stick growth of words associated with common ChatGPT output in the common crawl database over the past ~18 months is worrying.
It might be worrying but they also point out that the quality seems to go up. Perhaps people think that random web scrapes are way better than they really are, and so expect ChatGPT output to worsen corpuses on average rather than improve them...
I am by no means a data scientist, but if, as a large language model, ChatGPT was trained to optimize the same "quality" metrics that are used to evaluate the models trained on these random web scrapes, and now ChatGPT output has a larger proportion of the random web scrapes, wouldn't the measured "quality" increase as a result? It all seems intertwined.
Again, I could be totally ignorant on how these things work. (edited to add key words associated with ChatGPT output in order to increase the quality of my comment :))
