Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Cloudflare blocks an almost 2 Tbps multi-vector DDoS attack (cloudflare.com)
185 points by sendilkumarn on Nov 13, 2021 | hide | past | favorite | 92 comments


There is a truly excellent video on Mirai's (the botnet or atleast code in question) origin. It was created in the Minecraft server community by teenagers. The botnet was huge to a point where Akamai had to get help from Google to mitigate an attack on krebs' security blog. It also was used to attack Dyn, the infrastructure provider, and resulted in a huge outage affecting Netflix, Twitter etc.

Sadly it's only in German, but if you are on desktop, you can auto-translate the subtitles.

https://www.youtube.com/watch?v=uletKRPMnuo


I've read that Cloudflare also hosts a lot of DDoS-for-hire services. That seems like a conflict of interest.


This is 2021, where almost everyone creates a global problem, then makes money off of being the one to "mitigate the problem"... The people dedicated to not creating new problems, but trying genuinely to fix problems simply fail and/or run out of money are increasingly ignored because they don't have the biggest marketing budgets. Honesty isn't making money any more... A huge problem.

The absence of any real accountability, and admiration of hypocrisy, is what threatens us most heading into the future.


“It's a gigantic social phenomenon. People find ways of getting money by impeding society. Once they can impede society, they can be paid to leave people alone.”

— Richard Stallman, 1986 https://www.gnu.org/gnu/byte-interview


By that logic (abuse of) the global internet is a problem, but the underlying technology isn’t, if it were localized.


I am not convinced, do you have any sources that prove your conspiracy?


I don't see any mention of conspiracy. I see a (colorfully hyperbolic) description of systemic problems.

And there are plenty of them out there. Look at the opioid epidemic, where a pain-relieving drug creates pain when you try to stop it. Look at Facebook, which simultaneously creates loneliness [1] and purports to offer its cure. To say nothing of more traditional addictive substances, like nicotine and alcohol, which create problems for users that more consumption temporarily ameliorates.

Then we could look at more subtle, multi-agent problems. For example, consider the way the US's incarceration rate is 5-10x peer countries. [2] Why is that? There are many factors, but look at the way for-profit prisons and prison guard unions are big spenders on influencing politicians to be "tough on crime". Look at the media that profitably generates fear about crime. The way police are not incentivized to reduce crime, but just to performatively fight it. This of course takes money away from schools and social services. And all of that creates disruption in communities that ensure the supply of criminals necessary to keep this going.

Is there any conspiracy there? I doubt it. One of the miracles of free-market systems is the extent to which conspiracy is unnecessary. All you need is networks of agents with aligned incentives and you get very robust, persistent systems. There's no conspiracy to get lovely fresh produce in my grocery store the year round; there's no need of one. But markets are morally neutral, so we always have to use POSIWID [3] thinking to keep an eye out for pernicious systems.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7820562/

[2] https://en.wikipedia.org/wiki/Comparison_of_United_States_in...

[3] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...


Oh No... No... Not me!... :P

Not really a conspiracy theory... Just a personal opinion.

These days sharing "conspiracy theories" get people banned online and worse...

Just made as a statement in reply to the parent comment, but if you watch the commercials during television news, you might perhaps wonder how "Restless Leg Syndrome" became a real thing, and why there's now how conveniently there is a drug that claims to "fix it" if you're willing to sacrifice diarrhea for in exchange for the pill's implied benefits.


Your ignorance of a neurological disorder before you watched a commercial about it doesn't imply it's an invention. Restless leg syndrome has been described for centuries.


Dude, Cloudflare is not encouraging ddos to then benefit from it, it existed and will exist with or without them.


Capitalism incentivizes selling a pill to cure something instead of other solutions, I'll grant you that.

However, I have RLS to the point that I'll kick my wife awake at night. I have found that certain foods trigger this, and avoid those foods. Search for "IBS RLS" if you don't believe me.

I guess what I mean is, don't let the existence of hucksters for a problem's cure convince you that the problem doesn't exist.


I didn't intend to mock the syndrome... It was moreso about a company's attempt to classify it as any kind of rapid leg movement that can be fixed with a pill that potentially causes diarrhea that made me roll my eyes during the commercial... Then they played a "Reverse Mortgage" commercial just after it, which also causes diarrhea..



If I'm understanding this correctly, then what Cloudflare is doing is hosting websites of DDoS services rather than hosting DDoS attacks themselves.


Yes, that's right. I don't think anyone here has been claiming otherwise.


Thanks. Just clarifying for some of us (including myself) who tend to jump to the most exciting possible conclusion.


Don't understand why anyone bothers to accuse cloudflare of encouraging ddos by simply hosting a website

I assumed that they actually did the ddos


2015. not saying that anythings changed but worth noting.

quote from said article for perspective:

The Web site crimeflare.com, which tracks abusive sites that hide behind CloudFlare, has cataloged more than 200 DDoS-for-hire sites using CloudFlare. For its part, CloudFlare’s owners have rather vehemently resisted the notion of blocking booter services from using the company’s services, saying that doing so would lead CloudFlare down a “slippery slope of censorship.”

As I observed in a previous story about booters, CloudFlare CEO Matthew Prince has noted that while Cloudflare will respond to legal process and subpoenas from law enforcement to take sites offline, “sometimes we have court orders that order us to not take sites down.” Indeed, one such example was CarderProfit, a Cloudflare-protected carding forum that turned out to be an elaborate sting operation set up by the FBI.


I think this an uncharitable simplification of a complex issue. Cloudflare tries to balance itself between censorship and overreach of what their customers are doing with their service (booting off Parlor earlier this year for example) as well as what the law-enforcement legally requires them to do. If Al Queda hosts a website on AWS, the problem is exactly the same.

And now, we have people essentially conspiring that Cloudflare creates their own DDoS attacks just so to prevent it based on a glib oversimplification.


They certainly don't host DDoS network ops. What you're talking about is hosting web pages.


They're not just "web pages". They're a key part of the financial infrastructure sustaining the problem that Cloudflare gets paid $600m/year to fight.

Does that imply that Cloudflare is intentionally boosting the problem? No. But let's be clear here: anything that makes DDOS attacks less of a problem means less money for Cloudflare. So whatever their intent, Cloudflare is helping to support the problem that they owe their existence to. It's very much a conflict of interest.


But then DDOS providers give away their IP addresses and traffic to Cloudflare, which can be valuable data.


That doesn't sound very plausible to me. Your theory is that there are criminal gangs sophisticated enough to create large DDOS attacks but so clueless that won't use a cheap virtual server and a VPN when setting up their public intake?


And even if it worked for Cloudflare, it's not like they're shutting down the DDOS services they're tracking. The services could still go out and attack non-Cloudflare customers. So even if you were right, it wouldn't be exculpatory.


You’ve figured us out! Damn it; it would have been such a great plan if we hadn’t decided to give DDoS mitigation services away for free. Dagnabit!

https://blog.cloudflare.com/unmetered-mitigation/


I mean, I don't buy the conspiracy theory at all, but I would expect it to still be true that you benefit from DDoS attacks even if you offer protection for free. As the need for DDoS protection increases, you convert non-users to users, moving them one step closer to paying customers.

I'm sure that the fact that it's highly illegal and unethical are reason enough for Cloudflare to not sell DDoS capacity, but the perverse incentive is still there.


You want everyone to believe that none of us are capable of seeing the first derivative of an action and reaction?

Cloudflare facilitates DDoS, yet Cloudflare "mitigates" for free. "How could this EVER be a business model?", you disingenuously ask.

Simple - if DDoS are common, then more and more people and companies become afraid of them. After a while, everyone wants DDoS mitigation. More and more people move to Cloudflare.

Whether paid or not, you now control more people. Duh.


I always thought there should be more terabit attacks with the level of home connections nowadays


I would imagine ISPs have some sort of bot prevention measures that would get triggered if you went all out on using a home connection.


A good mitigation strategy is giving people 1Gbps down, over DOCSIS 3.1, that nobody can ever actually hit, and overselling significantly on top of that. Then, doing the same with upload, but only offering around 30Mbps up.

At least that’s how it feels in the U.S.


It's my understanding that 1000/30 isn't an artificial limitation. The coax lines have limited bandwidth such that 1000/1000 per customer just isn't possible. They could split it different ways, of course, but since historically most customers download far more than they upload the 1000/30 became standard among consumer ISPs.

Not that ISPs aren't evil. They were paid to run fiber everywhere, such that everyone would have 1000/1000 fiber links by now. But such as it is.


DOCSIS is asymmetrical, but my understanding is that 3.1 could theoretically handle 10000/1000 with all channels. I’m sure the infrastructure in many places wouldn’t be able to do that, but I have a feeling they could do better than 30.


Lol???

5mbps x 200,000 subscribers is already 1 tbps

We all need faster speeds at home, not slower.

Counter suggestion: make fcc regulate iot, whenever a person's appliance enters a botnet, suspend his connection until said appliance is removed and fine the person if the device wasn't fcc aproved.

There, no more botnets inside the US. The rest of the world to go


The FCC as regulator is an interesting idea.

Appliances sold in the US already have to prove they don't create harmful EMF emissions. It wouldn't be much of a stretch to add minimum security requirements to avoid harmful "data emissions" to that same certification process.


So you can't make your own devices anymore?


You could say you should take care of making them right. And add a few safety rules if you want to sell them.


sure you can. but the instant they're part of a botnet attacking someone, you, it's owner, should have to do something about it. We have fire code to regulate what people build so they're not a death trap and this wouldn't be so different.


How would you certify that a windows PC won't join a botnet?


I wouldn't. But when the device, which happens to be running windows, takes part in a DDOS attack, I wish we could do something about that, rather than have to buy our way out of the problem by having a bigger pipe and sinking traffic, because it means that you have to be blessed by the powers that be of the Internet(Cloudflare, AWS, GCP, etc) in order to stay online in the face of a DDOS attack.


At my last apartment it was gigabit. And it was definitely gigabit speeds


Coax cable is limited to 10 Gbps (DOCSIS 3.1) and is shared with many houses/apartments (can easily be a few hundred modems) in a neighborhood. Theoretically only 10 people can use 1 Gbps at any one time, in practice probably even less.


There are at least 65 million homes in the US.


Certainly even 50/500, or 100/500 would be a better split.


In general, no. Unless you start affecting their internal network. If you keep the traffic rather moderate a home connection can spew traffic for months on end.


They do! I have a fast fiber connection. I have had an ISP sec/ops guy literally call me and ask about my traffic patterns. He was more curious than anything -- but they do monitor strange patterns. I agreed to turn off my crawlers and explained it wasn't a botnet.


Some have, but it's usually signature based. If a customer has an infection with a known worm (all I've seen were windows based) it's matched by some signature and the connection is isolated. From then on all web traffic is redirected to the ISPs service portal helping the customer install an antivirus solution.

Never seen it applied to DDoS kind of things.


I assume you live outside the US? Our home connections here haven’t improved in two decades.


Assuming this is about telnxy outages this week and their migration to cloudflare. https://status.telnyx.com/

Maybe premature for cloudflare to be declaring victory?


Whilst I'm a big fan of people updating status pages, copy/pasted updates really rub me up the wrong way.


Can’t they try take the bots offline? Do the bots hide their IP address or could they not start contacting the owners of said ip addresses and tell them they need to remove the infected device from the internet? I know it wouldn’t be that easy but is there nothing they can do to fight back and start getting rid of these bots?


> Do the bots hide their IP address

For this attack and many like it, yes, the bots hide their IP.

Per the article, this attack was a combination of DNS amplification and UDP flood. UDP packets don't use a connection like TCP (where the recipient verifies it can talk back to the sender); instead, the packet just declares where it came from, and the recipient fires-and-forgets a response to that IP, blindly assuming that IP is actually the sender.

So for the UDP flood portion, the victim receives a packet with a fraudulent source IP and no way to tell where it really came from.

For the DNS amplification part (also done over UDP), the attacker finds an open DNS resolver online, sends it a request to resolve a record, and fakes the UDP source IP, telling the DNS server to send the response to the attack victim. Not only does this mean the DDoS packets aren't coming directly from the attacker, but DNS responses can easily be much larger than DNS requests, so an attacker multiplies how many gigabits of traffic they hit the victim with, versus just sending UDP packets directly to the victim.

Here's Cloudflare's primer on DNS amplification attacks: https://www.cloudflare.com/en-gb/learning/ddos/dns-amplifica... and UDP floods: https://www.cloudflare.com/en-gb/learning/ddos/udp-flood-ddo...

As far as solutions go, the answers are broadly 1) get everyone in the world to stop putting up UDP services that send large responses to unverified requests (this attack used DNS, but this happens with other protocols too), and 2) convince ISPs everywhere to deny outbound UDP packets which claim a source IP from outside that ISP's network. Since this is one of those "you have to be perfect, but the attacker only has to find one weakness" scenarios, these sorts of attacks will keep happening until it becomes impractical to find enough abusable networks/services to mount high-volume attacks.


How do you spoof the source IP? If these attacks mainly originate from IoT devices in home/business networks shouldn’t the ISP block outgoing packets which have a source IP which does not match the IP of the home/business routers interface?

I haven’t tried but I would hope my ISP would drop any packets I send out from my home network which do not have the public IP address of my router, but I haven’t tested it.


You're mainly right about the solution - really only the ISPs can do much, and they tend not to.

CF actually wrote a pretty nice article about challenges in doing so - https://blog.cloudflare.com/the-root-cause-of-large-ddos-ip-...


The second point sounds like something that can be fixed with regulation and/or fines.


Regulation and fines could help, but that's hard to organize globally.


In past, they have taken bots offline (mainly by taking over the Command/Control server) but most of these "bots" are just malware infected connected devices operated by clueless average folks - hard to update, hard to take down.


The article mentions that these were UDP attacks... which are usually reflections based on spoofed IP addresses. So who should Cloudflare contact? In the meantime another few hundred small attacks arrive. It's more constructive to improve the capability to mitigate attacks as they and other network providers have agency over that.


The UDP packets still have to pass through the network and networks can attach all kind of tracking headers to these packets. So you should be theoretically able to track down the sources of long running attack if every network provider along the line cooperates.


UDP doesn't have a notion of key/value headers of arbitrary data (like HTTP does). This is all the metadata that UDP packets include: https://en.wikipedia.org/wiki/User_Datagram_Protocol#IPv4_ps...

If cooperation of intermediary networks is assumed, these attacks can be crippled by convincing ISPs to deny outbound UDP packets claiming source IPs from outside their networks.


> This is all the metadata that UDP packets include: https://en.wikipedia.org/wiki/User_Datagram_Protocol#IPv4_ps...

That is explicitly a simplified representation used only to compute the checksum of the UDP package. It doesn't even include the full IP header, nor does it touch any of the protocols the IP package would be encapsulated in at all. Network tagging and other fun things happen as low as the Ethernet layer.

> these attacks can be crippled by convincing ISPs to deny outbound UDP packets claiming source IPs from outside their networks.

Not sure this would be enough, I think ISPs generally have complete ranges of IP addresses so it would be trivial for an attacker to create a list of "valid" IPs to use.


> Network tagging and other fun things happen as low as the Ethernet layer.

I see the IP packet ID field (which appears to have this very use prohibited?) and the 802.1Q VLAN tag on Ethernet frames (a 32-bit value). Is that what you're referencing? Does that mean the idea is each network tagging traffic during transit within their network, with a process for downstream entities to request logged tracking data? I got the impression you meant for the end recipient to receive the intermediate tracking markers alongside the sender's original data, but maybe I misunderstood :)

> Not sure this would be enough, I think ISPs generally have complete ranges of IP addresses so it would be trivial for an attacker to create a list of "valid" IPs to use.

It would prevent the reflection portion of these attacks (a bot could only reflect traffic back into its own IP block, not towards an arbitrary global target), and knowing which networks originated the traffic would enable other countermeasures.


It was more of a half thought out idea than something concrete, currently drawing a blank on how the network would communicate the tracking information outside. I also seem to have misremembered various things about Ethernet tagging. :(

> It would prevent the reflection portion of these attacks (a bot could only reflect traffic back into its own IP block, not towards an arbitrary global target), and knowing which networks originated the traffic would enable other countermeasures.

That makes sense.


Limited anti-spoofing that only allows spoofing within the ISPs ranges is sufficent to stop reflection attacks targeting IPs outside that range, which is usually enough.

It doesn't help much with direct volumetric attacks, but it would potentially make it easier to track (hey ISP, we're getting a lot of traffic evenly divided over your IP ranges, and they can confirm it's coming from their network and maybe figure out where it originates)


Come to think of it, it’s a bit odd routers don’t in force this by default.


Routers are optimized to know where to send packets given a destination address, not to know what source addresses are valid given a packet is received.

In some cases, it's simple, one address/subnet per port, would be 'easy' to enforce; this is often the case for normal residential connections and commercial users that didn't bring their own IPs. In other cases, networks are connected to networks and what to send there and what is ok to receive may not be the same and may also be dynamic.


Routers know how to figure out what route to use for a given IP.

If they apply this algorithm to the source IP and find that the optimal route to the source is a different interface than it was received on, that’d potentially be a red flag. But if the optimal route to the source is the same as the optimal route to the destination, that’d be a huge red flag.


That really only works if you assume routing symmetry and a single optimal route.


I was mainly thinking of ISP routers for residential/small businesses which seemingly make up a large portion of DDoS traffic.


How long does it take to contact thousands and thousands of IP owners looking for infected device? Many of which are behind NAT devices which require even further tracing.

What about the ones overseas that just don't care?


What is with ddos these days?

Are they doing it for money ?

It just seems silly with services like cloud flare


If you have that much computing power at your disposal, you might as well just mine cryptocurrency, right?


Not necessarily as this is Mirai, a botnet of IoT devices. There’s probably not much else you could do with them.


You can probably do both at the same time, as cryptocurrency uses a lot of CPU/GPU/memory/... but little bandwidth, and DDoS typically uses bandwidth but little CPU/GPU/memory/...


You can hire a ddos agaisnt your across the street competitor. It could be the other pizzeria, the other hardwareshop. Use your imagination


That used to be a thing but is it anymore?

There is so much mitigation so it's pretty much ineffective


It's even worse nowadays than it used to be, due to "Serverless" and "Infinite Scalability"/"Auto-scaling".

One of the most fascinating things I've read recently is the rise of "Denial-of-Capital" attacks.

Essentially, you DDoS a competitor, but not directly in the interest of just taking them offline.

Instead (hopefully) running up a massive cloud bill and putting them out of business. Or a similarly critical financial hit.

If you don't have billing limits enforced for all of your services, and you run auto-scale/serverless workloads in any part -- if someone can pass enough traffic to your services they can cause you potentially incredible financial grief.

Most recent (publicized) one I can think of is this one. Fathom Analytics attacks:

https://news.ycombinator.com/item?id=25194795

There was an initial cloud bill, but now they're paying $3,000/mo for AWS to have a Cloud Protection team on standby for them.

  "$36,000 & my call with Fola"

    "I don’t know anybody who has signed up for this $3,000/month service from AWS… it’s called AWS Shield Advanced. The big value of this service to us is that we have access to some of the world’s best DDoS mitigation experts. In the event of an attack, we can page them, and they’ll help us mitigate the attack, creating firewall rules, identifying bad actors, and offering advice. So instead of just two of us responding to DDoS attacks, we have genius engineers we can speak with, and that feels good."
Ouch.


no such thing as billing limit in Azure, anyway.


As if anyone signing up for Azure care about pricing.


Not everyone has mitigation. If you know your competitor is hosted by a small hosting outfit you can get them banned from their webhost by directing a DOS attack at them.


Ineffective? It fuels cloudflare's business model.


what business model? cloudflare basic ddos protection is free


Cornering the whole internet is their business model. Besides, advanced DDoS protection is paid.


> The entire attack lasted just one minute.

Did the attack last one minute because Cloudflare 'mitigated' it after that, or because the attackers stopped?


Botnets tests their capabilities all the time. This could have been a command and control test, a test to see what they could muster, or a demonstration.

When testing they seldom run for a long time.

Cloudflare's mitigation would've dropped in on the metals and still been visible to Cloudflare's monitoring... so the attackers stopped after a minute.


So those nice graphs on Cloudflare's blog are exactly the information the attackers wanted? If that's the case, by publishing such detailed post-morterms, Cloudflare is just inviting future test attacks.


I used to run the servers for a popular website. It was common to get DDoSed targeting our servers (or more frequently, just a single one out of the group) for exactly 90 seconds (plus or minus a few systems that had poor ntp synchronization). Whether or not that took my servers down, the attack would stop.

To my knowledge, we never got any communication from the people behind the attack, seemed like people just kicking the tires on DDoS as a service. Ocassionally, we'd get a longer interval, sometimes 60 minutes.


I was responsible for a website (one of a many of this kind) that provided access to a niche auction platform. At some point in the beginning of 2010s it became a subject of a precisely coordinated series of timed attacks designed to disrupt bidding of one of our prominent clients in the specific auctions. It was enough to bring down the service for ~5 minutes to prevent the client from winning.

Eventually we migrated behind CF and the problem was solved but I couldn't help but wonder if there are some applications for which even a few seconds disruption (I assume that's the minimum time Cloudflare needs to begin effectively mitigate the attack of this scale) will be disastrous and what could possibly be done in this case?


If you can't handle a few seconds disruption, you really need actually private networking. Dedicated lines (or at least dedicated wavelength on shared fiber) and redundancy and very fast failover.

Volumetric udp reflection isn't really too bad to process anyway, as long as you've got the bandwidth --- fancy tricks get you from the UDP stack dropping useless packets to dropping useless packets without the UDP stack, possibly at the edge without using up nearly as much internal bandwidth.

Where it gets pretty hard to manage would be application level bursts, IMHO.


Stock trading comes to mind


CF: Would it be asking too much to have a date and time stamp on your blog posts somewhere?


It's right below the title, where you'd expect it.


Whatever, guys... Nothing, NOTHING will make me think better of CloudFlare. I won't forgive you, CF, for captcha, tracking and blocking me from accessing a critical server from an airport! Burn in hell!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: