I would have like to see some technical notes in there. It almost comes across like a glorified sales pitch for two products!
First Joel says that they don't deliver static files and that edge caching won't help. And then they swap out some IPs, cuddle up to Akamai and bam -- suddenly the Internet delivers.
My impression is that Joel has no idea how this Akamai IPA thing works; he is treating it as magic. I would never buy a service that is advertised as magic, though.
Hiya. I'm one of the Copilot developers, so I thought I'd answer some questions.
The Akamai IPA thing is not "magic," and none of us think it is, but dumping a rambling technical diatribe onto Joel's blog would alienate a lot of its readership. We're talking on Hacker News now, though, so read on if you're curious what's going on and how Akamai's helping us.
There are two ways that an Internet connection can be slow: you can have insufficient bandwidth, and you can have bad lag. With Copilot, we're already doing just about all we can to minimize bandwidth with our raster-based format. We've got pretty fast compression algorithms, we dynamically pick compression levels on-the-fly, and because we're using existing VNC code under-the-hood, we do a good job sending across just the differences in your screen. Switching the protocol to be command-based (like X, or RDC) is the only real way left to improve bandwidth--and with our current developer resources, that's simply not feasible. The good news is that this discussion mostly ends up being a non-issue: relatively few customers actually have bandwidth saturation issues, and internet connection speeds are steadily increasing, so bandwidth's not generally a big concern.
That leaves lag. Most of the time, when customers complain Copilot is "slow," they're complaining about lag. Copilot gets lag in two places: on the network connection itself, and on the glorified echo-server (which we call the Reflector) that we use when none of our firewall bypass tricks work. We try really hard to ensure you connect directly, but sometimes you simply can't, so it's very important that the Reflector be very efficient.
About a year ago, I did some analysis to see how we could speed things up, and found that our Reflector introduced nearly a quarter second of lag in some circumstances--not terribly surprising, since it was literally the very first thing I wrote at Fog Creek, back when I was an intern. The design used one dedicated thread per connection with blocking I/O. On Windows, that type of design is very slow. I took a few weeks out and rewrote the Reflector to use overlapping I/O with thread pools. The resulting Relector now introduced virtually no lag, and as an ancillary benefit, used a fraction of the CPU and proved far easier to maintain. For the time being, our customers were really happy.
Unfortunately, over the last year, customers have complained that our performance is still sub-optimal. The problem's been compounded because firewalls are getting smarter, which is making our firewall bypass tricks fail more and more often. Even if the Reflector truly introduced no lag whatsoever, you're still gaining a lot of lag when you have a reflected connection, simply because your data has to make a full round-trip from you to the reflector to the other party.
That's where Akamai comes in. The main thing that Akamai's doing for us is dramatically reducing lag between our data center and our customers. That means your keystrokes get to the other side faster, the machine starts responding faster, and you get the graphical updates faster. Even though bandwidth hasn't increased, your Copilot experience suddenly becomes far more pleasant, because the machine feels much more responsive.
We're still monitoring our clients to see how much of a difference Akamai's making, but in general, what we're seeing is great. The bitrates for Akamai-enabled connections has gone way up, since the higher response rate allows our users to get more done in less time, and we've been able to pull of some stunts--like watching YouTube videos via Copilot--that were previously impossible over reflected connections.
Akamai's not a panacea, and what they're doing isn't magic, nor is it useful in all circumstances, but for a real-time application such as Copilot, it can yield a pretty dramatic improvement in exchange for very little developer time.
[EDIT: Fixed some snafus I caught when re-reading.]
Excellent! Thanks for taking the time to write up a more technical look at the changes you guys made.
I'm still not clear on how Akamai helps speed things up. I understand how CDNs work: they move static files to edge servers that are physically closer to customers, which mean less hops that data has to move through. But this isn't static stuff.
Let me ask: is the general concept this -- prior to Akamai, users would have to hop through a dozen servers but now they only have to go through 2 or 3 because Akamai uses its own network? (Essentially bypassing the general Internet and using a type of private internet.)
Basically, yes. We're hopping off the Internet at the nearest Akamai hub, bypassing the normal backbone, and then hopping back on close to the endpoint. I'm in the office at the moment, so I can't test this for obvious reasons, but you should see the effect by doing a traceroute to webnlbcopilot.fogcreek.com and comparing it to web1copilot.fogcreek.com. The former should go through Akamai, the latter will route directly to us.
I actually ran the traceroute. To the webnlbcopilot server, it's 18 hops, and a few of the machines along the way are named like "deploy.akamaitechnologies.com". (So I assume this is the Akamai route.) The latency on this route is 38ms or so.
The other route is only 14 hops, but the ping time is 54ms.
So yeah, interesting. More hops, but measurably faster.
So to cut a long story short, there are now two Internets. Or many Internets, if you like.
There's the default basic one, which is the slowest and laggiest "consumer grade" Internet.
Then you can pay more to a company like Akamai to have your traffic go over a different network that is properly designed and provisioned for less data contention.
You're right that there are many Internets, but there isn't a "default basic internet" any more than there's a "default basic airplane route"
The distributions of Airlines and Bandwidth Providers are fairly similar in number and size, and the customer patterns are quite similar too. Very few people buy directly from the providers, they go through retail intermediaries to get the lowest prices (booking agents/ISPs). The people who do buy directly either have some sort of 'relationship' with the provider, or are massive enough to avoid buying anything at the retail level. The hub-and-spoke networks with routes merging mid-stream are also quite similar topographically (though not geographically).
Akamai's network isn't any more "properly designed", they're just willing to pay the actual Bandwidth Providers more, and more importantly willing to pay up front for guaranteed bandwidth. Peering bandwidth is normally billed by volume used, with a fee based on your maximum thouroughput over the billing period.
Anyone could always buy point to point IP links with lower contention ratios, but this is the first time I've seen a company (Akamai) with enough of them internationally to have a serious attempt to have full Internet coverage.
The design used one dedicated thread per connection with blocking I/O. On Windows, that type of design is very slow. I took a few weeks out and rewrote the Reflector to use overlapping I/O with thread pools.
How'd you become enlightened as to the terribleness of your original design?
From what I can tell "overlapping I/O" appears to be windows-ese for good old poll(2) -- non-blocking asynchronous I/O with level-triggered notification. Does Windows not have a way to do edge-triggered notifications like Linux's epoll(7)?
Also, is there some incentive on Windows for using a thread pool instead of a handful of independent processes per box? Realistically you'll have the server running on multiple machines just for sanity's sake, and you'll have a way to distribute clients across them with failover, so why have two levels of indirection when one will do?
FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible. How terrible it is is a function of address-space starvation from too many threads (a non-issue in 64-bit code), and the overhead of the OS thread scheduler. In a green threading model, I/O can be implemented behind the scenes as non-blocking I/O, but still give user code the ease of use of blocking I/O.
Back to Windows: overlapped I/O is not equivalent to poll, it's much better than that. It is edge-triggered, but Windows manages the threads itself, and tries to ensure 100% CPU utilization in user code, rather than in thread switching logic.
Windows overlapped I/O is effectively a way to write efficient I/O intensive applications in continuation-passing style. You don't need to manage the threads or dispatch loop yourself.
As to why use threads rather than processes, the issue is largely the awkwardness of duplicating the connection handle for a child thread. In Posix land, there's the fork function that makes handing off the handle fairly easy. WSADuplicateHandle can create a new handle that can be used by a child process, but you need to know the child process's ID first, so you need to perform IPC to just to send off the handle.
On the other hand, shared memory concurrency has its advantages too.
FWIW, it's not obvious that the one-thread one-connection blocking I/O design is terrible.
Any time you're using a 'thread' idiom, even if they are green, something has to be maintaining their individual execution state (stack frames, thread-local address space, etc.). I like to think of it as being analagous to unoptimized tail-recursion, with a purely event-driven callback model being like tail-call optimization.
Thanks for explaining Windows' overlapped I/O, it's way better than I thought it was from reading the documentation. I didn't realize that it does all the thread-pooling stuff all by itself -- I guess it makes the most sense not to fight it, and just accept the free utilization (even if it does give a slight latency hit).
No - not tail call - event driven callback model I/O is continuation-passing style (CPS). The two models are equivalent. Code written in the thread idiom can, in principle, be mechanically transformed into CPS and thereby use the event-driven callback model.
In the event callback style, the state that would otherwise be on the stack is held in the callback closure.
In the .NET space, F# does this transformation automatically with what it calls asynchronous workflows (http://blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f...). It uses the 'let!' assignment as a dividing point between the call to the asynchronous method and converting the remainder of the method body into a continuation that gets passed to the asynchronous call.
Er, I'm not sure what you were reading, but Windows has a poll function. It's called poll. Overlapped I/O is the edge-trigged notifications you want, and correspond to epoll/kqueues. You make a function call, which returns immediately. You then get a callback (either literally, if you're on .NET, or as a Windows event otherwise) when data is available. Overlapped I/O automatically uses the Windows thread pools, so making my own wouldn't make sense.
Native Windows overlapped I/O can give you a callback without needing to use an event: WriteFileEx (or WSASend) accepts a callback routine, etc. Also, the OVERLAPPED structure requires you to provide the event handle, and it's optional, so you don't need to use that kernel resource if you don't want to.
The "poll" function you speak of - I presume you mean Socket.Poll - is spelled "select" in native code, and is equivalent to the BSD sockets version.
We're getting a bit knit-picky, but yes, you can get a callback directly--which is in fact what we're doing.
Vista and later (which includes Windows 2k8 Server) actually introduces a proper WSAPoll function, so you don't need to use the godawful select() anymore unless being masochistic really is just your thing.
So, I suppose the next step would be to deploy multiple of your reflector servers at different network locations on the internet to improve even more? :)
What media are you trying to deliver? Traditional CDN are very expensive -- like cell phone minutes, you pay up-front for X number of GB. If you buy 1000GB and burn 20GB, you still pay your $10k/month bill.
Amazon is making a big play in this space, and doing a 'pay for what you eat'. Hopefully this will encourage CDNs to take a look at their pricing schedules.
Not CDN service or delivering media. More like this proxy-type service he describes where you cut out a lot of middle-tier routing and pay for better backbone transit.
For personal use, this would be a great way to proxy your own traffic when using the Internet overseas, where accessing US-based sites can be very slow. Or even for setting up a faster coast-to-coast VPN within the US.
Akamai either owns the lines on the "superfast superclean superhighway" (Joel's words) between the Akamai nodes, or they have purchased preferential traffic treatment from someone who owns those lines.
"superfast superclean superhighway" means "we have a private network." There's nothing special going on. Call up your telco and say you want a 10 gigabit connection from new york to LA and they'll be happy to oblige.
Just because two computers in the world are talking doesn't mean they use the big-i Internet.
Such a purchased private network is what I meant by "owns the lines" (even if in practice such connections are rented from telcos).
Which leads me back to my main point: why should 'neutrality' regulations care whether a telco offers a 10 gbps dedicated line, or preferential routing for 10 gbps worth of traffic on their shared lines?
And if the motivation for buying either of those is the exact same value -- snappier service for your end users -- that Akamai sells, why is Akamai's service celebrated, but alternate services based on traffic shaping considered a 'neutrality' violation?
Because when your local ISP does it, they are abusing their monopoly.
If it was actually an open market with real competition, then I wouldn't care about net neutrality, because there would be plenty of options for non-broken internet access. However, if I can only get access through one or two providers, then those providers have an incentive to extort money out of websites by threatening to degrade access to their sites.
I think the real fix to the net neutrality problem is to open up local access so that the phone/cable company only provides the wire, and an unlimited number of ISPs compete to provide the net access. The phone/cable company probably shouldn't be allowed to provide the internet access, since they will inevitably abuse their monopoly (and necessary monopolies should be kept as limited as possible).
Agreed 100% that local competition is the best fix, either by enforced sharing of the scarce last-hop wires, or introduction of new paths (new wires or wireless technology).
Enforced 'neutrality' could push real competition further away, by limiting the kinds of offerings and profits that attract new entrants.
Also, for a sense of proportion: even though local access in the US is usually a cable/dsl duopoly, and total speeds lag some other national markets, prices continue to fall and speeds continue to increase. No last-mile ISP has 'extorted' websites with threats of degraded access.
So there's no urgent need for legislators and regulators to start dictating how service is delivered, as if there will never be competition. If they want to help, they should adopt policies that spur new entrants, not tie the hands of current providers.
Historically we've had real neutrality without any regulation. In the last few years the local monopolies have made a number of moves to turn that neutrality off, and in some cases there have been serious abuses; and they've also started trying to persuade politicians that neutrality is a bad idea. Regulation that maintains the status quo of network neutrality is going to be a lot easier to achieve than regulation repeal to strip local phone and cable providers of their legal regulated monopolies.
...is almost always a bad idea. We need new services, new investment, new entrants.
Opnnness can defend itself -- it got this far. If any of the alleged 'abuses' truly became 'serious' that would give a marketing coup to competitors and spur to new entrants.
The idea that regulating the existing monopolies is 'easier to achieve' than true competition is a self-fulfilling prophecy. New regulations entrench present-day services, and flatter regulators into thinking only they, and not competition, can protect customers.
If a shortage of competition is the real problem -- and it is -- policies should address that, not enshrine the current lack of competition in regulations.
Does RON == Resilient Overlay Networks? And do you think Akamai's selling points and pricing indicate they're improving end-to-end peformance without buying any special paths?
Yes and yes. My understanding of Akamai's philosophy is cheap redundant hardware plus cheap (often free) redundant dumb bandwidth plus genius software.
First Joel says that they don't deliver static files and that edge caching won't help. And then they swap out some IPs, cuddle up to Akamai and bam -- suddenly the Internet delivers.