curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that.

[1] https://github.com/lwthiker/curl-impersonate

Oh wow, thank you! I had no idea something like this existed. I was manually compiling against BoringSSL.

For a while now I can't even access WhatsApp Web [1] through Firefox. It gets stuck on the loading page and keeps refreshing itself forever. I have to resort to Chrome whenever I need WhatsApp on my laptop.

[1] https://web.whatsapp.com

If you time, please file a bug report on Mozilla’s webcompat.com with the steps you took and a screenshot. Mozilla’s webcompat developers will try to debug the website, reach out to the website’s web developers with a proposed fix, and until the website is fixed, add a site-specific intervention or UA override in Firefox itself, if possible.

Looks like there are already a couple bug reports about whatsapp.com, but not the forever-refreshing page you see:

https://webcompat.com/issues?page=1&per_page=50&state=open&s...

I don't think I can be of much help but WhatsApp web has been working for me on Ubuntu with all Firefox versions since WhatsApp web has been a thing. Did you try to start Firefox in safe mode, no addons and/or a fresh profile?

If anyone finds a solution it would be greatly appreciated. I have been cut off from a huge number of sites due to infinitely refreshing cloudflare.

It's absurd that this is even possible. They should at least provide troubleshooting advice for end users.

I think they have different plans and configurations for their anti-bot service [1]. I'm not sure though because I'm not using their services.

[1] https://developers.cloudflare.com/bots/get-started/

They don't block you completely, just present you with a JS challenge that delays your access to the site. A browser, even if behind a MITM proxy, would be able to solve this challenge.

I hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.

There was a conversation on their mailing list contemplating dropping NSS support. https://curl.se/mail/lib-2022-01/0120.html If you have a use case for NSS in curl, you may want to speak up. Perhaps "I want curl to look exactly like a browser" is a significant use case?

Agreed, it is very important to bring this up on the mailing list. It might also be plausible to make curl look like Chrome if curl had BoringSSL support.

I will try to impersonate Chrome next, However, I suspect this is going to be more challenging. Chrome uses BoringSSL, which curl does not support. So it means either enforcing curl to compile with BoringSSL or modifying NSS to look like BoringSSL.

Thanks for the suggestion, I had no idea ESR was a thing. I've just added support for Firefox ESR 91 (it was pretty similar and required adding one cipher to the cipher list and changing the user agent).