I just wanted to say that while we’re sad to shut down and definitely didn’t accomplish everything we wanted to we are deeply grateful for having had the chance to build something significant and get past a 1.0.
Our original crowdfunding round and supporters - almost all of whom found us on HN, Y Combinator, and our seed investors helped make that possible (plus of course all our open source contributors, alpha and beta users, early customers, and even the people who filed GitHub issues panicked because their sites had just gone down).
Most of our team worked together on Tent (protocol for decentralized social networking) before this which no VC would get behind. So we know how much it sucks to have a great idea that no one will pay for you to build.
We’re very grateful to HN for the opportunity to build something so cool.
Neither of those projects would have been possible without HN. I know for a fact that we only got meetings, term sheets, into YC, and more on the strength of the HN comments and later the GitHub stars.
Before we had a working product we could at least say, “hey, these people all say they want what we’re trying to build” and that got us in the door.
We had lots of different proof points to show investors and even customers, but the most important thing was the number of upvotes and enthusiasm of the comments.
Thank you all for being so open minded and supporting our work. We literally wouldn’t have been allowed to do it without you.
Hope to have something even better to share with you all one day.
We’re surprised and flattered at the interest and discussion.
For those wondering why we shut down here’s a brief summary:
We created Flynn to be Heroku that you could run on your own infrastructure (cloud or otherwise). We started Flynn as a crowdfunded project with support mainly from other startups, we imagined it would be non-commercial and community-driven.
After the prototype it became clear that we would need a full time team to develop and maintain the project. When we couldn’t find anyone who wanted to fund the project long term we applied to YC and then raised a seed round which let us build a 1.0.
Unfortunately we were never able to raise more VC so we spent several years trying to build a business that would allow us to keep developing the project.
We spent the last four years with a skeleton team and while we were able to build a business doing “ops as a service” for other startups we ultimately couldn’t generate enough revenue to cover both development/features and the 24/7 on call team we needed. While we were able to support our paying customers we couldn’t deliver the features we wanted to build or support unpaid open source users.
We also take stability and support really seriously. We were extremely uncomfortable with the fact that people were running Flynn but didn’t know enough to run it safely. So we’d get panicked emails and github issues from users who couldn’t pay us but needed help. That’s a terrible situation for everyone involved.
The ecosystem has come a long way since we started (we were one of the first Docker powered projects and started about a year before kubernetes was announced). But we still feel like there’s a need for something like Flynn :(
Unfortunately it’s expensive to develop full-featured platforms and we could never get VCs interested in a Series A.
The 24/7 on call lifestyle got to be too much for our team, especially during COVID so with no prospects of things changing we decided to call it quits.
Just a hypothetical question I want to know as a FOSS dev, but someone who's never used Flynn before:
If someone came to you right now and offered an amount to keep going, no strings attached, enough to support the applications which remain hosted on your platform after those who have already lost their confidence leave, would you consider keeping it running?
What amount, in whatever currency you prefer, would make you consider it?
I've no horse in this race, but I'd like to get at least one datapoint.
This is not the first time I see a post like this, and it always makes me wonder if you could've gotten enough funds if youd gone out and said, "Look, Flynn is going to shut down if we don't get X amount minimum to support us. If this platform means anything to you, now is your chance to save it."
Interesting question. I think there are two angles:
1. Having already shut down and the team gone separate ways and onto new projects how much would you need to get back on the horse so to speak
2. Are there still things you want to accomplish and what would it take to make them succeed
Personally I’d happily take a check to do more development. There are a lot of Flynn 2.0 features that were in various stages of completion, if someone was willing to fund development to complete them up front that’s great.
However I wouldn’t want to do sales for this/run a b2b company around this again.
So if one of the companies that was using Flynn called and said they wanted to pay our development team to keep building, that’s great. But if a VC called and said “wait! Let’s finally make this company take off” there would have to be a decent sized bonus check attached. Mostly that’s just because after years on a ramen profitable startup as a founder your finances aren’t awesome so saying “yes” means turning down higher paying offers.
The biggest problem we had financially was being able to develop features before our customers needed them. We knew what to build but didn’t have enough budget to ship them. Any offer would have to come with the guarantee that we could focus on development for day 6 months before starting sales back up.
At the end of the day there’s work we left unfinished and if someone was willing to fund that in advance I’d generally be up for it. But if someone just wanted to keep the business wheels turning that’s not super attractive at this point, so it would need to be a https://levels.fyi salary rather than a startup founder salary.
If you don't mind me asking, what are some features you think are most desirable for you to develop, and what would be your time budget for developing them, if it is more than 6 months?
And which title from levels.fyi do you think is most applicable to the person or people who would be working on that feature?
Happy to — it’s a great question! I want all software to be open source but we really haven’t figured out how to fund many types of open source projects yet.
- Database appliances (RDS) for major DBs
- turnkey security, especially for compliance (SOC2, HIPAA, ISO 27001)
- close the loop on development environments (vscode)
- CICD
That’s pretty close to our series A roadmap. Mythical man months aside, I’d say all of that could be production-ready in around 18 months with a few engineers per bullet point.
In terms of salaries it’s a little tricky for a number of reasons (people are willing to work for less for a startup and/or on open source plus we hired around the world), but for many of the developers I’d say equivalent to L4-5.
The important thing to remember about PaaS is multi-node vs single node are different worlds. There are great tools like dokku that were designed only for single node operation, then there are things like Flynn or k8 for many nodes. As soon as you’re doing many nodes you’re in distributed systems territory where engineers have to manage much more complete systems design and as a result are more expensive and in more demand. That’s not saying anything bad about simpler tools, they’re great and important, it’s just a lot harder to design around distributed systems where failure modes, consensus, partitions, etc are a lot more complicated. A lot of the challenges when we started were figuring out what the problems were and how to solve them rather than implementation time. In the last 5 years there has been a lot more written so less research would be needed to implement some of these today.
Our overall goal was to automate anything in the devops lifecycle that could be automated, so that’s a never ending process. Unfortunately it means that the more you scale the more technology you have to make, so Flynn for smaller companies with fewer users is less expensive to design and build than Flynn for huge companies with lots of traffic. (Again, distributed systems are hard) so knowing the scale of your prospective users is a big part of the cost equation.
For me as a totally uninitiated, the database appliances sounds like the lowest hanging fruit with the fewest unknown unknowns and the biggest payoff, is that correct?
At what you described, for 6 months, that's about 200K USD per dev, so it would cost about 600K USD to develop the database appliances "superfeature"?
Do you think this is a realistic figure? And how realistic do you think it is to get that much money together from all the users?
$180k base salary is appropriate but most of those people are expecting stock options as well (compare to Levels) so the fully loaded price* is closer to 300-500k/year x 3 people, so $1-1.5mm
We’ve already done most of the conceptual and architectural work for Postgres, MySQL, and mongo (and redis but not highly available). Much of that was inspired by the Manatee project at Joyent. So (though I haven’t looked at this I a while and the DBs may have changed in a way that becomes more challenging) this is probably a case of put money in, get value out.
However I can assure you you aren’t going to get even 600k out of the community.
We worked really hard to get the first 120k and twisted a lot of wrists to get there. (Of course if you know someone who wants to write 600k checks for open source stuff please have them call me)
The 1-1.5mm number is about what a standard YC company gets at demo day. So that’s the path I’d recommend. Add in another 500k for overhead and biz dev people and that’s a pretty solid startup. Honestly if we had just focused on that we would probably have got a series A.
One of the great things about open source is that you create 100x+ the amount of value that you capture. The dark side is that no one wants to contribute cash as users to make that happen (I’m thinking more of companies than individuals - lots of individuals will give a few dollars)
So, unfortunately, and as always, it’s easier to raise VC than sponsorships for open source.
That being said I strongly encourage anyone who’s interested to pick up where we left off. Either as a community project or a funded startup this would be a huge benefit to the ecosystem.
* of course you can find people who are cheaper and passionate about the project to do it for less, maybe much less. But if someone called me and said for what price can you guarantee this could be done, I’d say 1-1.5mm USD.
Cool. We got the first 120k from I think around 12 companies. It followed close to a power law distribution, very long tail.
Our strategy was to encourage developers to talk to their bosses rather than contribute individually which we think but can’t prove worked well. Basically we tried to get on calls with CTOs and CEOs and explain that this would help the company.
Is there anything you would've done differently during that campaign?
Did the money come from the companies' tech budgets?
Was it a one lump sum pre-payment or periodic payments? Were there any strings like deliverables attached?
Hope this is not too much to ask, I'm really interested in the process.
I am developing something almost completely the opposite -- a website platform designed only for small-scale deployments by individuals and small communities, with purposeful mechanisms to limit/control growth, and for now deliberately without any finances, fregan.
Recently I have been coming to terms with the idea that I may get a lot further if I attract outside help, but currently I don't have much to pay with, and I've not made it easy for someone to just jump in and start contributing.
Honestly there aren’t a lot of products that were funded this way so I can’t point you to specific resources.
In our case the funding came prepaid with no strings attached. Can’t speak to where in their budgets the funds were allocated from.
I think we should build an ecosystem where companies support open source projects that benefit their companies with great ROI. Unfortunately I don’t know how we get there yet.
There are so many different funding models now for different things (https://humanipo.app/ for example) that there must be a good answer. We should all work together and try harder to find what the best option is.
Other cofounder here. I just wanted to add that we still have some t-shirts and stickers left. So if you want some defunct project/startup swag, we'd love to send it to you! https://shop.flynn.io
Still wear my flynn shirt from 2015 when you kindly did a YC practice interview with us! Thanks for that BTW, y'all captured the intensity of it very well :)
(For those who don’t know, all the partners in a YC interview ask questions of everyone simultaneously— this is complicated at the best of times and totally nuts when you’re trying to answer technical questions about distributed systems and questions about how you’ll build the business, but it’s a great mental workout!)
Thanks, we are doing well - never did get into YC but it was a valuable process every time we applied - gave us time when we're not occupied with the day-to-day operating to think about our strategy.
We're back in NZ and focusing solely on the education market which really accelerated our growth. Not sure when we'll be able to travel to the US again but would be great to catch up when that happens!
I think that’s part of it but it’s also the natural consequence of having several people on one side of the table trying to find out everything they can from the founders in a very short period of time and being genuinely interested if you and what you’re building.
Nice! Picked up a shirt and some stickers :) Sorry to see you guys go. Flynn was one of the first things that made me legitimately excited about the future of Docker. Thanks for all your great work.
Sorry to hear the bad news, John and Daniel. Thanks for all your great work, especially in promoting TUF. Hope you guys have found greener pastures now.
24x7x365 is an expensive support option, and not all customers need/want it. I found that if I budget staffing for a manager and 7 employees, create "slots" for access at different tiers (segment by business hours in different time zone ranges, response time, and support contract duration), price access to 24x7x365 slots accordingly, the price-first customers will sort themselves out. Good support that performs properly as a function of the continuous development cycle is staffed by highly-technical and high-soft-touch skill sets, and it is expensive to find and retain those people. Don't shortchange them or yourself, and charge the market accordingly.
3-7 years is about the maximum I would expect an unsustainably-structured support organization to last before people burn out, so you had a good run.
Sorry to hear about the lack of VC interest. I'm the founder at Render (https://render.com) and would love to chat and compare notes. Email in profile.
technically illitare here, pardon the stupid question. I hear so much buzz around k8s/docker/AWS/GCP/Azure and all that but simply don't understand what they actually do. if you go with the extra effort to run your product on you own infrastructure, what do you need Heroku/Flynn et al for? isn't the idea of PaaS to take care of server/database/hardware management etc for you, so you can focus on developing your product?
Great question. We just consider ourselves to be a normal company that happens to build open source tools. We also try to make money in ways that don't interfere with the open nature of what we build.
There are lots of different kinds of companies whose engineers primarily work on open source projects. In most cases where their core "product" is an open source technology they also sell commercial support, usually to large companies who need an SLA. Others (like us) sell managed services as well.
We made the decision before we started the company to make anything we ever asked users to run on their own machines permissively licensed open source and just be careful with our trademark (like Mozilla is[1]). That leaves the door open for closed-source SaaS in the future but gives our users (and investors) a clear set of expectations about how and what we'll build.
I think it's important for companies to be as clear as possible about how and why they license their code, what choices they'll make in the future, and what happens when the company changes hands. It would be great to see something like an "open source pledge" where founders/the company could contractually commit to either a final open source release before closing (like Parse did[2]) or staying open after being acquired (like Sun didn't after being acquired by Oracle [3]).
Thanks! Flynn is designed to be cloud-independent so you're not locked in to any cloud, or even cloud in general. You can also run Flynn on your own servers.
Flynn also allows portability between clouds. Since we don't use any cloud-specific features (like, say RDS) you can backup a Flynn cluster (including database appliances) from one cloud and deploy onto a new cluster on an entirely different cloud.
Thanks for the great question. I'm the CEO & co-founder of Flynn.
tldr: we're open to making Flynn a foundation/community project if we could find the right kinds of partners and once the core innovation is complete.
This is something we've thought a lot about over the past few years. Everyone on our team is a hardcore open source believer and contributor and the long term success and sustainability of open source projects and Flynn in particular are something we care deeply about.
When we started working on Flynn (with just a crowdfunding page posted to HN) we had no intentions of commercializing it. Before asking HN to fund the project we reached out the founders of many growing startups (many of which are now unicorns) who we saw as the core users. Our goal was to agree on an architecture and set of APIs, then have each company "own" one component of Flynn, with one engineer at each working on their component full time.
Everyone we talked to said they loved the idea of an open source Heroku that could run anything (not just 12 factor apps) and run anywhere from AWS to their own colos and datacenters, but none had the staff to spare on infrastructure engineering (which is why everyone wanted a PaaS in the first place). Several, like Shopify, contributed to Flynn financially so we could hire a dedicated team to build Flynn for everyone.
So a community/industry collaboration was absolutely our goal from the start, but several things made us question that route. Not only were our first choices for collaborators and partners unable to participate directly, but we spent a lot of time talking to companies in the then-thriving OpenStack community as well as other foundation/industry run infrastructure projects. They shared a number of horror stories of how the most promising new features and capabilities were killed by cross-vendor compatibility concerns and how specs were limited by legacy vendors on committees.
Committees and bureaucracies are great at keeping something alive and maintaining the status quo, but not at innovation.
We also met executives from very large companies in regulated sectors like banking and telecommunications. They were extremely excited about Flynn but had a completely different set of requirements than the younger companies like startups whose needs we knew best, and had experienced ourselves. It was clear that we would have to build a very different project to serve the largest of enterprise customers. Crucially, foundations/industry collaborations on infrastructure attract the largest of vendors who have these mega-companies in their sights as the real customers.
If you want to see what a PaaS built to their (imagined) needs are, look at CloudFoundry built by Pivotal, which is currently owned in part by Dell/EMC, VMWare, GE, Microsoft, Ford or at OpenShift, which was built as a startup but acquired and retooled by RedHat.
While we think Flynn has a lot to offer today and in the future to these large institutional users, they have not been our focus.
Flynn today has the minimum feature set we consider necessary to be useful. Looking towards the future we want it to encompass a lot more. We'll be publishing a comprehensive roadmap/master plan soon, but most of that innovation would be harder or impossible to accomplish if we were locked into a foundation today. Development would be slower, changes would have to work their way through approval to consensus, and many partners would object to new features competing with their own established products.
We've been very careful both to limit external dependencies and to provide end-to-end integration between all our components. As a result we're able to move very quickly when changing APIs, implementation details, even major pieces of architecture, all without impacting customers. This gives us the freedom to stay nimble in a changing landscape, but partners might be more attached to technologies in which they already had a large institutional investment.
Infrastructure companies today are already a world apart from where things were in 2008 when Heroku upended PaaS. The default is now open source, making lock-in (historically the bigger concern) far rarer. There are few companies among our peers battling to become the next Oracle.
Additionally there's a tremendous amount of portability both among unified PaaS's and build-your-own infrastructure because of technologies like buildpacks and containers. As a result most apps that run well on one platform can be moved to another. There are of course features, like Flynn's database appliances, that are not available on most others.
We've made other choices like a small codebase with few external dependencies that make managing Flynn's development and maintenance easier, whether it's our team or yours doing that maintenance.
That being said, we're entirely committed to Flynn's future personally. While we'd love to see it become a huge company, that's not necessary for us personally. We're a team of five, and speaking for at least my cofounder any myself, we're not going anywhere, regardless of the VC market.
We work on Flynn (for several years and many long hours now) because we believe it needs to exist in the world, not because we think it'll make us rich. We're committed to keeping Flynn alive as long as that stays true (basically until it becomes unnecessary or there's something else that does everything we want but better). Since our 1.0 launch we've had huge growth and adoption and are currently fundraising. We're single digit new customers away from default alive with the current team. Whether we become a small, sustainable company to selling to/partnering with a large enterprise vendor, to creating a foundation, everyone on the team is 100% committed to a permanent, sustainable future for the product, with or without venture capital, unicorns, or IPOs.
Thank you for the detailed response - much appreciated. You have tempted me to look into Flynn further.
The headline feature that attracts me is HA postgresql - that has instant relevance. My greatest concern is stability - I'm not seeing many success stories but lots of problems. I want to hear that Flynn is "the one bit of my stack I don't have to worry about". I hope you can publish some case-studies (both successes and post-mortems) including discussion of the attempted architectures.
A fundamental philosophical worry I have is that it is a "wrapper tech". I increasingly stay away from these - the promise is that they make the underlying tech easier/quicker to use but they make assumptions that don't hold for you in the long-term, they serve too many unrelated use cases so become bloated and poorly fitted to your own needs and they reduce the features, flexibility and stability of the tech they wrap. The cost in fixing & working-around the wrapper can quickly exceed the cost/value of building your own glue based on the underlying tech because it also gives you a greater depth of knowledge in the fundamentals and more power.
Heroku is a great product and it going down permanently would be a huge loss to the community.
However there are a number of open source alternatives with very similar user experiences that you can run yourself in a variety of environments including:
From what kind of scale would you consider running flynn (I'm a fan since the early days) or deis (scoped it as well) a pragmatic step compared to heroku?
Absolutely. We're putting together a list right now, we just don't have permission from everyone to talk about it publicly yet. Hopefully we'll have quotes and case studies to post on the website soon.
We're very active about providing as much free support as possible to users over email, github, and IRC. We've collected some great quotes from users about how impressed they are with our responsiveness.
Unfortunately we're a small team and frequently travel to events, etc. So it's expensive for us to guarantee that someone will be around. Obviously we'd like to provide support to as many people as possible at the lowest possible rate, so as we have more people paying for support and economies of scale kick in we hope to be able to drop those prices. In the meantime we need to cover our costs or development will stall
edit: our current prices are also based on the level of support that our most active beta users needed. Now that we're past 1.0 hopefully users will need less help which will also help lower prices. We have a very high bar for support quality and don't want to overextend ourselves to the point where we can't deliver
We designed Flynn to be easy to install and use and for it to take care of itself as much as possible. Users generally get up and running and manage issues on their own.
Most current users (from individual developers to large organizations) haven't needed or wanted commercial support contracts. The offering is there for organizations who need significant help and/or an absolute guarantee that they'll be able to reach us at any hour of the day, every day of the year.
I use Cloud 66, so I'll use that as a comparison in terms of price. Their price is $19/mo for the first server + $9/mo for additional servers. It's a little bit of work to get set up and running.
To compare with Flynn, their current pricing is $74-75/node, which is quite a bit more. However, Flynn appears to offer more 1 on 1 support whereas Cloud66's is extra.
Nonetheless, there's quite a bit of difference. I think the range of $20/node may be reasonable. Anything less, the company would be unable to grow. In fact, especially for early stage startups, I'd probably err on the side of a higher pricing (eg. $30-40).
The paid plan is not for the side project but for funded projects or ones that already have money coming in.
Flynn is also a whole lot easier to get set up. Again, just what I've observed based on an earlier release.
BOSH operates at a lower level, working with VMs. Flynn is higher level and allows deployment and management of applications and databases in containers.
Instead of relying on a lower layer like BOSH, Flynn is self-bootstrapping and self-hosting. It runs everything except the container runner in containers managed by the platform. The same APIs and tools are used to manage user applications and Flynn components.
You can bring your own lower level infrastructure, like IaaS or bare metal. There is a cloud installer that can spin up a cluster to try out on EC2, Azure, or Digital Ocean, but most production users are using whatever management tools they are comfortable with to deploy and scale Flynn hosts out.
Dokku is explicitly single-host. Flynn can run in a single-host configuration but typically deployed in a highly available configuration with three or more hosts.
Flynn is designed to tackle some much bigger problems and supports running and managing highly available databases inside the platform out of the box in addition to stateless web apps.
As Titanous said, BOSH operates at a lower level than Flynn. You could use BOSH to deploy Flynn, hypothetically. A more direct comparison would be Cloud Foundry.
Kubernetes didn't exist when we started, so we ended up building our own scheduling system and management APIs. So it wouldn't really be "adding" support, instead we'd need to switch several layers of the system to be Kubernetes instead of our code.
We're watching Kubernetes, it's possible that doing that change will make sense at some point in the future.
Is there a document describing the architecture of a running system?
I didn't get from reading the documentation whether I'm spawning processes, containers, or affecting full servers.
What comprises a cluster?
Can I choose to dedicate a host to run a db, and at the same time run serveral mini-processes without launching every one in a container?
The front that flynn exposes is pleasently simple, but I also want to know what implies every command and how it affect the running environment.
Please let me know if anything specific is missing so that we can add it.
> Can I choose to dedicate a host to run a db, and at the same time run serveral mini-processes without launching every one in a container?
You can dedicate specific hosts to specific apps and process types using node tagging, though it looks like we are missing documentation for that. I'll make sure it gets added soon.
You are free to fork multiple processes inside of containers you deploy, there doesn't need to be a 1:1 mapping of container to process.
Thanks, that detail was what I was looking for.
I also wanted to understand the flexibility and options that Flynn offers to modify the running environment.
We view the work of the Kubernetes (and Mesos, Docker Swarm, and other scheduler projects) team as incredibly valuable to Flynn.
We try to solve problems and create value at a the platform layer. Schedulers are one technology we use, but aren't at the core of the value that we (or most other platforms) deliver. We want to make development and management of production apps simple and invisible. For most users that should mean never needing to know about or interact with a scheduler directly.
We closely monitor progress on the major scheduler projects and look forward to a day when we can utilize one of them in Flynn instead of maintaining our own. As these technologies mature we'll be able to deliver even more value on top of them, while saving users from interacting with lower level components.
I'm interested in different schedulers. Is the Flynn scheduler documented anywhere? Anything you want to say about it and how it differs from the other offerings out there?
We've basically implemented the minimum viable system for our use case, and only add features as needed. The main thing that's missing right now is awareness of resource constraints and usage and then balancing based on that. There is a patch in progress that adds awareness of host-specific persistent volumes for our database appliances.
As an aside, when we started Flynn, the only open source scheduler that was available was Mesos and it didn't meet our criteria for integration so we ended up writing our own code. That code has since been rewritten approximately twice.
I just wanted to say that while we’re sad to shut down and definitely didn’t accomplish everything we wanted to we are deeply grateful for having had the chance to build something significant and get past a 1.0.
Our original crowdfunding round and supporters - almost all of whom found us on HN, Y Combinator, and our seed investors helped make that possible (plus of course all our open source contributors, alpha and beta users, early customers, and even the people who filed GitHub issues panicked because their sites had just gone down).
Most of our team worked together on Tent (protocol for decentralized social networking) before this which no VC would get behind. So we know how much it sucks to have a great idea that no one will pay for you to build.
We’re very grateful to HN for the opportunity to build something so cool.
Neither of those projects would have been possible without HN. I know for a fact that we only got meetings, term sheets, into YC, and more on the strength of the HN comments and later the GitHub stars.
Before we had a working product we could at least say, “hey, these people all say they want what we’re trying to build” and that got us in the door.
We had lots of different proof points to show investors and even customers, but the most important thing was the number of upvotes and enthusiasm of the comments.
Thank you all for being so open minded and supporting our work. We literally wouldn’t have been allowed to do it without you.
Hope to have something even better to share with you all one day.