That's very interesting, although I'm still curious about the training resource usage -- not "only" inference. I wonder what is the relative importance of training (i.e., what percentage of the resource usage was in training vs. inference)
One random preprint I found (https://arxiv.org/html/2505.09598v2) estimates inference is 90% of the total energy usage. From some googling around, everything seems to agree that inference dominates (or is at least comparable to) training on the large commonly-used models.
I was less surprised that inference dominates training after I read that chatgpt is serving billions of requests per day.
Many papers on large models make claims about the energy used. Google has consistently said that training and inference use about the same fraction of their data center resources.
Is this just meant to be dismissive, or just like a kind of non-answer? That it could be said for anything doesn't make asking it of this specific thing unimportant or uninteresting.
Right, and the scraping, the extra storage required, the manufacture of all the extra GPUs, etc. This is them hoping people don't understand that the query is only one part of it and aren't curious enough to ask further questions.
