> though various sites will block it from time to time. The page itself describe... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		MattSayar 11 months ago \| parent \| context \| favorite \| on: Claude can now search the web > though various sites will block it from time to time. The page itself describes a --ignore-robots-txt and customizing the user agent. Guess we can just all copy OpenAI and continue to make SourceHut's life miserable /s This is a cool tool, thanks for sharing

knowaveragejoe 11 months ago [–]

For what it's worth, it's fetch under the hood. More akin to curl than automated scraping.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact