It's actually interesting why this idea is so totally wrong.
So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically.
Global Crossing didn't have to accept the bad update. They generally trust updates from other organisations that are generally trustworthy. They apply checks and restrictions proportionate to the risks involved.
These mistakes happen rarely. If they were to happen more often, major operators would apply more checks and restrictions. If they were to stop happening, operators would apply less checks and restrictions, because they have a cost in manpower, complexity, and loss of flexibility.
That's how the internet works. You could almost say that's what the internet is--the idea of being actively managed by people who know what they're doing and are not bound by exhaustive predefined policies is defining of how the internet came about and how it came to be dominant.
If you want a network guaranteed to be resistant to this kind of f---up, build one. The internet is that network which does not work that way, which is flexible, expandable, mostly "good enough" but not ever designed for absolute reliability.
If anything, the outrage should be directed at Global Crossing/L3. They shouldn't be allowing some little ISP to screw up their route tables like that. Because this means that the Malaysian government could secretly announce just a few blocks, route them back out so they still work, and easily do MITM.
OTOH, so much in telecom is hung together on the assumption of basically good actors everywhere.
>It's actually interesting why this idea is so totally wrong.
No it is not. You accept their claims of "mistakes" I see no evidence of that - how, exactly, do you leak a full table by accident? and this is too big a security hole to leave to hackers. Leak a bunch of table, shut down an entire country.
full table leaks happen ALL the time. The reasons you dont notice or hear about it is:
- the providers which do this by accident are too small (multiple asn hops away from a major transit provider) to become the best choice for most people
- the small guy who does leak the full table to a major transit provider, is adequately filtered by the major transit provider by default
- the small guy who leaks to the medium transit provider might take an outage, but may not leak to his upstreams due to outbound filtering or the upstreams filtering
you would be surprised, BGP is an old protocol, has had very little serious security improvements. It currently works more or less based upon the goodwill and discipline of network engineers around the world, because if they screw it up, they usually end up offline and out of a job.
Even if was a mistake, that doesn't suddenly make it OK. People make mistakes, yeah. But this isn't a simple mistake, in fact this incident consists of multiple mistakes.
1) Someone wrote an incorrect config
2) They did not test it
3) They pushed it to production systems without testing it
4) They did not monitor their systems after pushing new configs
5) They took ages to fix the problem after it was detected.
It's a little difficult to test this kind of config without emulating the entire internet - which is quite clearly beyond the scope of all bar a very small number of organisations.
This isn't about how the internet works. This is someone messing up and breaking the internet.
If I'm driving a car without paying attention to the road and kill a bunch of kids, that's my fault.
If Telecom Malaysia pushes new configs without testing them and breaks the internet, that's their fault.
Obviously these two things aren't even remotely comparable in seriousness, but it's clear that in both cases the people messing up should be held responsible.
Excuse me Mr Lol, but since it happened, it would appear that it in fact is how the internet works.
If the postal service misdelivers some mail are there criminal charges?
I would suggest resisting the urge to hit every problem and mistake in the world with the law hammer; it is rarely productive, and generally isn't how the world works.
You are wrong because this IS about how the Internet works and less about how one person/company screwed up.
Apparently all it takes is one ISP misconfiguring something to break large swathes of the Internet. I believe the consensus on HN is that no one entity should have an Internet kill switch.
If someone managed to disrupt mail delivery on a global scale, people would be less concerned with THAT it happened than that it COULD HAVE happened in the first place. Why would global mail delivery be so not-fault-tolerant that one mistake brought it to a grinding halt for hours? Same deal here.
I don't think anyone is saying that the fact that this CAN happen isn't a huge problem, it most definitely is.
Thing is, everybody knows (and has always known) this can happen. Everybody also knows how to avoid it. Well-intentioned people tend to try to avoid breaking the internet.
That really doesn't make the negligence of Telecom Malaysia any more defensible.
I really don't see what's the point of defending Telecom Malaysia, a plenty of people manage to operate their equipment in a manner that doesn't break the internet.
Just because this was a mistake doesn't mean they should not be held responsible (and no, I'm not saying someone should go to prison.)
I really don't see what's the point of defending Telecom Malaysia, a plenty of people manage to operate their equipment in a manner that doesn't break the internet.
Perhaps the point is that Malaysia is part of the world, and we can't realistically expect to exclude them from the internet, any more than we could expect to exclude them from the commercial airline system. Their network people answer to their customers in Malaysia, not to us nerds on HN. (Also ISTM many Malaysians are more concerned about earthquakes caused by exhibitionist tourists [0] than about internet stability.) I guess in some way TM answer to their upstream in GLBX, and they could get "demoted" in some way, but GLBX isn't going to just walk away from an income stream.
Many times when one node is blamed for network-wide bad results, the nodes that connect it to the network might be blamed fairly, as well.
> Also ISTM many Malaysians are more concerned about earthquakes caused by exhibitionist tourists [0] than about internet stability.
Malaysian here. More Malaysians care about the 18 people who died on Mt Kinabalu from the earthquake, than about the antics of a few douchey tourists.
As for Telekom Malaysia, TM has the same reputation that BT has in the UK - shitty service, but Malaysians are stuck with them. I don't think it surprised anyone that TM caused this fuckup.
I'm definitely not saying we should disconnect Malaysia from the internet, that'd be terrible.
Despite the name, TM is a private company. I really wouldn't have any issues with (temporarily) disconnecting them, or alternatively fining them. GLBX should definitely be able to do both of those.
Of course best case scenario would involve Malaysian government intervention. (As unlikely as that sounds in a country that seems to be ran by people that believe in magic)
I really wouldn't have any issues with (temporarily) disconnecting them, or alternatively fining them. GLBX should definitely be able to do both of those.
If that language is in their contracts, then sure. Such penalties might not be made public, however, or there may be different enforcement mechanisms in place. The internet (and all global commerce, really) functions anyway.
Of course best case scenario would involve Malaysian government intervention.
Given how often "Malaysian government intervention" entails unconscionable violence, I cannot agree.
Based on previous experiences I'd imagine their contract would allow early termination in case of abuses such as this,but of course this is speculation.
I definitely didn't intend that anybody should be executed, but at most fined (or imprisoned for a reasonable amount of time if this was in fact intentional, but that's unlikely).
"with design to obstruct the correspondence, or to pry into the business or secrets of another, or opens, secretes, embezzles, or destroys the same" nope
So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically.
Global Crossing didn't have to accept the bad update. They generally trust updates from other organisations that are generally trustworthy. They apply checks and restrictions proportionate to the risks involved.
These mistakes happen rarely. If they were to happen more often, major operators would apply more checks and restrictions. If they were to stop happening, operators would apply less checks and restrictions, because they have a cost in manpower, complexity, and loss of flexibility.
That's how the internet works. You could almost say that's what the internet is--the idea of being actively managed by people who know what they're doing and are not bound by exhaustive predefined policies is defining of how the internet came about and how it came to be dominant.
If you want a network guaranteed to be resistant to this kind of f---up, build one. The internet is that network which does not work that way, which is flexible, expandable, mostly "good enough" but not ever designed for absolute reliability.