In my quick experiment (asking a question that would naturally lead to content on my own site) it is not doing a real time request to the site in question. Its answer included links back to my site (and relevant summaries), but there was no requests for those pages while it was generating its answer. So it's clearly drawing from info that has already been scraped at some earlier point. And given that I see Claudebot routinely (and politely) crawling the site I'd guess it's working from it's own scraped copies (because why use someone else's if you've got your own....)
Major AI players don’t want to use someone else web index as they may cut it off or jack up the prices etc. major players want to build their own web index
And this is why we see our logs overloaded with ABot BBot CBot etc, every single "AI" company makes their own bot and they all crawl the same pages over and over.
