I wonder if this is an unpopular opinion, but I just don't get why so many people in the industry started using containers where using, say, jars in the JVM would have worked perfectly fine.
It seems so much is lost, for no apparent reason. We used to have clean interfaces, architecture independence, pretty good performance, low startup times, low memory usage, and heaps of experience with the JVM or .NET for years.
We now seem to have regressed to modularising software into separate unix processes with less strict interfaces, extremely complex dependency chains, and many more moving parts.
Obviously, this is all fine if you are actually building systems at enormous scale, but I see this move to containers in all fields, even for applications that run on a single CPU where performance or scalability is not an issue.
What would be the reason for this? Is it simply that younger generations need to relearn from their own mistakes? Or is this part of the micro service hype?
(Of course I'm not ignorant to some positive aspects of containers, especially for interfacing with databases or web servers, and other unix services.) My problem is mostly with using containers to split even the most straightforward applications, where existing techniques were (at least IMHO) vastly superior.)
Every programming language had its own version of jars, dependency management, etc. that you then need to reproduce everywhere you want those things to run. Containers mostly abstract that away - you just have a container image filled with what you want and you execute it. It also provides a consistent means of isolation. Thats the benefit I see.
You don't need to split an application into containers, you can have your full app in a container no problem. You can have all your jars in the container. Using containers doesn't mean you know have to split your apps in different ways, it means you ship the runtime, compiled code, whatever libraries are required and all that in a bundle.
Some applications cannot be deployed as a monolith. Maybe they mix native and managed dependencies. Maybe they rely on external software (like, say, ffmpeg) being present. Maybe this external software depends on specific system library versions.
Yes, you can make all of this work without containers. They make all this very easy and reliable though.
So your container has the right Java Runtime Environment and your app inside. I don't even need to know it's Java!
To be fair, there are still things it doesn't catch; I've hit containers that needed an update to the host to run. Too old of a kernel, maybe? I'm afraid I forget the details.
I remember the good old days of having to script my own deployments on our own on-prem servers, with linter free wild west testing. It went to crap pretty easily because we were always customizing something, automating something, customizing the automating of something. Dependency checking was also very manual.
Nowadays we work with way more traffic that incentivizes strict bin packing of traffic within the tightest bounds of the resource pool, with the fastest actuation time. VMs won't give you that. Containerization just makes that so much easier and snappier for large amounts of traffic. But then we had to go global, and orchestrating containers across continents is a PITA. So there's value in k8s for dealing with that. These tools vastly reduce the complexity with regards to managing globally distributed services. But as usual, we start customizing these tools, leading to more complexity.
With that said, I believe startups need to focus on the fastest development and deployment times. So stuff like Docker, Kubernetes or Terraform are entirely unnecessary. As I commented elsewhere before, no one with a total QPS of 10 needs anything beyond GCE or EC2 or on-prem servers. People born in the Cloud age seem to have forgotten that they can just run the cheapest Linux machine with HAProxy and httpd behind a static IP. The focus on always going straight to Cloud complexity, if anything, smells more like engineers in on some VC grift.
Side note: It's hard to hire senior/motivated Java developers. It became a self-fulfilling cycle. Happened somewhere around 2011-2014, no one other than fresh grads graduating from legacy courses still used Java or .NET.
I found personally a lot of use for k8s in the "10 QPS" bracket, especially if you can afford to use GKE & CloudRun (but recently I managed similar setup with single hetzner dedicated server and k3s).
Specifically, the simplicity of managing things once you go through the initial pain of setting it up (much reduced with GKE), cost savings from bin packing your apps (not just your product but also anything else you need for your company that you can't afford SaaS for [1]). You can share servers for dev/test/prod till you get enough clients to migrate up. And as much as I love HAproxy, configuring it is way more annoying than dealing with some basic ingress controller + certmanager (+ optional external-dns, not needed if you just set a wildcard).
Right now I have a super cheap (compared to cloud) hetzner server which provides (using k3s) services to multiple clients, ranging from just VPN (using headscale) to handling git etc. All with SSO, low maintenance, etc.
[1] costs of SaaS service per employee hit differently when you're not VC backed and you're not based in the United States.In my experience it's often cheaper to get another full time engineer to handle some self hosting than pay for various cloud/SaaS/PaaS.
It complicates it downwards, but simplifies it upwards. For example, containers generalizes the workload in a Kubernetes world, where the JVM would not be a general abstraction. This mainly matters on a scale where Kubernetes or similar solutions are motivated.
I also didn't like to resolve issues with different versions of log4j, and OSGi didn't make things a lot easier either. Then there's classpath misery, gradle, and IDE incompatibilities.
But I'd still rather live in a world where we'd try to fix things, instead of moving to completely new ideas. Probably easier to change my yearnings than the world of software development.
its the ability to lower operating cost with less skilled staff.
No need to worry about some some other app using port 8080, dont need to ensure your scripts do a ps and filter out other users stuff, all those kinds of things.
Then theres libraries and vulnerability management which is easier to manage where the blast radius is just a single container.
> dont need to ensure your scripts do a ps and filter out other users stuff
Your broader point still stands, but I just wanted to say that finding processes to signal (presumably to kill or restart?) via `ps` is insane. Linux has really good process management tools available, whether service management via the init system (systemd, OpenRC, GNU Shepherd, etc.) or some other process management tool (supervisord, daemon tools, runit, etc.). Even beyond that, there's good ol' nohup and pidfiles. All of those things are well established and easy to do. Filtering ps output and using it to decide what to do is indeed painful, but it's painful because it's the wrong approach for Unix, not because Unix multitenancy is more painful than containers.
Containers are nice but it sounds like maybe you have some trauma from bad ops mixed into your assessment!
It seems you have been fortunate to not deal with "enterprise grade" vendor software with scripts that assume they are the only thing running on the host.
Indeed I have! My current job is at a big corp but so far I've not yet encountered something that heinous. I'm sure that we deal with vendors that bad in some parts of this massive org, though. The quality of tools we have to work with seems to vary a lot from department to department.
Why do businesses spend insane amounts of money and go through painfully slow and laborious sales and purchase processes to buy software that is utter garbage?
Is it most likely to happen when their needs are highly specialized, so there's just not much competition?
I know that enterprise software often has terrible or dated UX, and I understand the reasons for that. But beyond the UI, why is software that costs $5 million more likely to be shoddy and carelessly made than software that costs $50?
We already tried "JAR containers" for the JVM with this was called WAR and EAR files. We also add clustered servers to which you could dynamically deploy these files and scale up. These most popular "Application Servers" like IBM WebSphere and BEA/Oracle WebLogic could theoretically handle everything for you.
The only problem is that it all didn't work.
First, we've go the "worked on my machine" type of problem: Conflicting dependencies? Different version of the JVM? Native libraries?
Then you had the issues with the application servers themselves. All the configuration was usually done in some tedious GUI environment. Lost your server? Good luck. Want to automate something? Good luck. Want to use CLI? You're not a proper enterprise developer, who hired this crazy Linux hacker, fire him before he'll try to Open Source something. To be serious again, it was just downright uncomposable.
Lastly we had all the devops issues: How do you automate a deployment of a new version?
For classic PHP or cgi-bin apps you'd just upload everything to an FTP server (well, it's not as perfect as classic PHP fanboys make it, you do have to handle deleted files - rsync would still be safer).
For containerized apps, you just create a manifest file (or a Helm chart or whatever suits your fancy) and hit deploy. Kubernetes will take care of everything, including upgrading the container image if necessary.
It's the stuff in the middle that gives me nightmare. It's not enough to just dump a new JAR file on a server. You've got to stop the server and restart it using the new version. At the very least you'll need to have provisions for backup and quick rollback, and probably also a blue-green environment to avoid lost connections. In more critical services you'll want to have canary releases and a pipeline that carries your server through multiple environment.
This is why containers are unavoidable for most kinds of networked service. It's the deployment story. That's why you see people wrapping single-binary go apps in containers, even these binaries are completely static and often don't even depend on libc.
Are containers an overkill sometimes? Yes. They are far from ideal for CLI apps. That's where you want fast startup times, quick download, better package management and better integration with your host environment. But most CLI apps that I've seen using docker went that route because their installation process was too troublesome to bother with. You'd see that a lot with Python apps, which are quite hard to package. But even Java apps could be a hit or miss if you don't have the right JVM.
I am well aware of the JAR hell, but I foresee a not so distant future where we'll be in "container hell". Reproducing images seems fine now, but wait until all the utilities and the enormous dependency surface gets unmaintained.
Again, if your systems are complex enough and span multiple programming languages, then containers may be great. Thing is, I see teams of n Java developers splitting their application in n+1 services, each with their own container, and thinking they're on the right track.
I used to feel like docker wasn't worth the complexity but I've come around. It strikes a pretty dang good balance. Is kubernetes worth learning if you don't use microservices at work?
Using Kubernetes is a skill worth learning and is pretty straightforward to pick up.
Running a Kubernetes cluster, in a shared environment (e.g. not just a hobbyist or single-purpose cluster), is much more involved. I'd never say developing a skill isn't worth it, but it is a deep subject with lots of unintuitive and frustrating corners, and you won't pick it up in a day or without scars. There's a reason lots of small- and medium-sized business just write a check to their cloud provider.
> it is a deep subject with lots of unintuitive and frustrating corners, and you won't pick it up in a day or without scars.
Do you have any examples of a corner that might give you a scar? I’ve heard this a lot but haven’t experienced it myself.
I’ve spent a number of years working primarily with cloud foundry. Started working with k8s in the past year and I’ve mostly found it straight forward. But I’m far from an expert yet.
I'd say so. I've been using k8s as a basis for all my projects for years. It's the best solution I've seen to the problem of packaging and deploying apps. I have a kubespray cluster at home that runs dozens of services flawlessly. Flux is the easiest devops deployment system I've ever had.
For work, having minikube + skaffold is perfect for local dev of complex apps. And it's easy to use production deployment manifests with it. It's also easy to use Kind for running integration and e2e tests. The ecosystem is second to none.
I don't think so. It's overkill for most stuff and probably not particularly worth learning until you /need/ to.
If you're curious about orchestration concepts in general, something like nomad is a nice way to wade into the pool rather than the whirlpool of obscenity that is k8s.
+1, Nomad is fantastically simple (but still flexible), and gives you the proper level of insight into the nature of orchestration and distributed container workloads without bogging you down with arbitrary figments of "because Kubernetes" complexity.
After spending a day with Nomad, I feel I fully understand how this could get nightmarishly complicated in a Kubernetes way, and what that would look like, and why someone would want that in some situations. All the understanding without pointless but exacting detours.
I fully support this! Nomad has a much smaller API surface compared to Kubernetes, at least by default. Expanding Nomad is also much more straight forward.
> Is kubernetes worth learning if you don't use microservices at work?
Maybe. K8s makes orchestration better. (If you disagree look at the last time marathon/mesos/dcos released and propose a better replacement.) Do you need container orchestration? That's a much better question. If you do, reach for k8s, if you don't, don't.
If you’re interested enough to be curious, it’s worth sitting down for a day with a mini kubernetes distribution and working through the basics of pods, deployments, and services. They are relatively simple conceptually (the hard part is grokking the kubernetes way of doing things) but they give you a feel for the way Kubernetes works.
I'd say yes, worth it, just to be aware about its capabilities and caveats, strong and weak sides.
At one point or another someone will say something about K8s, and it never hurts to know more than just the peachy marketing posters from the large cloud providers, so you can have an opinion. What it realistically can and cannot do for you - this kind of stuff.
It greatly frustrates me that Docker Swarm fell out of favour leaving a void between single-host Docker and Kubernetes. It makes self hosting docker with a small amount of resilience and orchestration much more challenging.
Agreed, and we're actually trying to bridge this gap at our company Kurtosis! Docker is great for doing dev - fine-grained control over the container, strong predictability guarantees, fast, and relatively lightweight (in comparison with Kubernetes), but you can't run Prod on it unless you start rebuilding Kubernetes primitives like Services from scratch.
Our approach has been "build an abstraction over Docker and Kubernetes that can provide the predictability of Docker when you need it (Dev & Test), and the scalability and resilience of k8s when you need it (Prod)", and then handle compiling down to actual Docker & Kubernetes commands. The goal is for the user to have a single environment definition all the way from Dev to Prod, so you don't have to do the dance of "Docker Compose for Dev, Bash on top of Compose for test, and Helm for Prod"
He was there at the very start at DotCloud and worked on Dockers predecessor and then Docker itself. Early adopter is really underselling how much he contributed.
This guy has pretty comprehensive and well rated courses for docker and kubernetes - keep an eye out for sales with huge value discounts on his courses...
It seems so much is lost, for no apparent reason. We used to have clean interfaces, architecture independence, pretty good performance, low startup times, low memory usage, and heaps of experience with the JVM or .NET for years.
We now seem to have regressed to modularising software into separate unix processes with less strict interfaces, extremely complex dependency chains, and many more moving parts.
Obviously, this is all fine if you are actually building systems at enormous scale, but I see this move to containers in all fields, even for applications that run on a single CPU where performance or scalability is not an issue.
What would be the reason for this? Is it simply that younger generations need to relearn from their own mistakes? Or is this part of the micro service hype?
(Of course I'm not ignorant to some positive aspects of containers, especially for interfacing with databases or web servers, and other unix services.) My problem is mostly with using containers to split even the most straightforward applications, where existing techniques were (at least IMHO) vastly superior.)