Using the fetch MCP tool(which is very basic) works fairly well for me, though v...

MattSayar · 2025-03-21T15:14:34 1742570074

> though various sites will block it from time to time.

The page itself describes a --ignore-robots-txt and customizing the user agent. Guess we can just all copy OpenAI and continue to make SourceHut's life miserable /s

This is a cool tool, thanks for sharing

knowaveragejoe · 2025-03-22T02:34:58 1742610898

For what it's worth, it's fetch under the hood. More akin to curl than automated scraping.