I promise you weren't actually doing your costing correctly to arrive at that conclusion. Engineers always badly mis-underestimate the costs of things and "rack & stack data center management" is way more costly than you are actually accounting for. Especially in terms of opportunity cost and, well, just wasted resources that aren't actually adding value to the company.
There is way, way, way, way, way more to a running a successful business than "saving money".
The savings enabled us to hire more people than it took to run. After the upfront setup (Xen and Ansible), it was pretty painless, had much of the same flexibility as cloud options, and better performance. For bulk storage, we used S3.
Did your devs have the same turnaround time for new equipment and services as AWS? When they had problems, did they have the same volume of searchable material to help them repair? Did your services evolve and adapt as quickly as AWS? Does your networking connectivity survive DDoS attacks fairly transparently? Do you have DCs all around the part of the world that’s relevant to you? Can you survive multiple DC outages?
I was a cloud skeptic and ran Tech Ops (including our DCs) for years. About 5 years ago, it dawned on me that even owning the whole budget for Tech Ops, that I wasn’t capturing the full costs of trapping my org onto our in-house solutions.
At tiny, small, and medium scale, cloud is obviously the way to go, IMO. At large and huge scale, I think letting some hybrid leak in where systems change rarely and cloud costs are WAY out of line (DropBox storage, Netflix CDN, etc) makes sense.
We did have quick turnaround - we were running everything on Xen VMs on LLVM internally, it was trivial to throw up new instances, snapshot them, etc. Our demands weren’t changing that quickly, so there wasn’t that much need to bring on new hardware all the time. A small number of dual socket machines can go a very long way these days. We didn’t have the same kind of multi-DC redundancy that AWS can give you, no, but over those years, many AWS based services were down multiple times due to being focused in Virginia, so it’s not an automatic win for AWS. The amount of time we would have spent getting that running and maintaining automatic multi-zone failover on AWS likely would have swamped the benefits for us, though, and I think are usually overkill for a small company. YMMV.
Majority of businesses aren't running insane computations at huge scales...
Most are running a few small internal-facing servers hosting some internally developed apps, and need very little resources.
Just run ESXi, XenServer, Xen or something, and spin up a few VM's on a few thousand dollars of hardware, get a couple people to maintain it, and be done.
Even at large scales, like Lyft, having your own internal team and hardware is going to save money. Amazon is profiting off your instances... which leaves room for you to do it for less. Maybe not $7mm less monthly, but even a $1mm savings is significant... but likely a lot more.
They're making $200 million / month based on their S-1 filing. I'm sure they've done the math and the effort to cost savings here simply isn't worth it to them.
Or, more likely, in the beginning AWS is what their developers knew and were familiar with, and then as time went on it was easy and convenient to just spin up "one more instance".
Fast forward to today, and now it would be a serious undertaking with serious risks to move off AWS, not to mention the costs of building up the staff and assets to reimplement their requirements in parallel of AWS until reasonably confident they can flip the switch and still have an operating company afterward.
So, they're probably stuck - beholden to Amazon's whims and pricing mood of the day. They've bought convenience from Amazon in trade for massive technical debt, one which may be even more costly to get out of... Or impossible.
AWS isn't going to get any cheaper in the future..
Those free AWS credits Amazon gives students really pay dividends.
But AWS will get cheaper in the future, if historical trends continue. To the best of my knowledge, Amazon has never increased pricing on a service. They do, however, routinely (but unpredictably) drop prices, either directly (https://aws.amazon.com/blogs/aws/category/price-reduction) or indirectly (every new generation of EC2 instance is marginally cheaper and/or marginally more performant than the previous generation).
The “whims” of Amazon’s pricing are no more unpredictable than the pricing “mood” of your colo or your hardware vendor.
What I'm saying is that <5% of revenues on technical infrastructure for a technology company at Lyft's stage doesn't seem that crazy to me.
I agree that it would be a serious undertaking to move off AWS today. But it's probably also going to provide marginal benefit. No one on the finance side of their business is probably losing sleep over it. If/once it makes sense to move off then the finance dept will tell the eng dept they need to reign in infrastructure cost...and eng will do that.
There is way, way, way, way, way more to a running a successful business than "saving money".