I followed the call of an employee with 10yrs more experience than I have and he proposed we move our infrastructure towards docker containers for easier deployment / management.
Generally containerization of our app was a good step at this point, but we made the mistake to also put redis / postgresql into docker, which obviously is an stupid idea if you're trying to scale things. On top of that, docker networking was a science on itself, and we didn't use any proper automation tools.
Docker containers kept running out of disk space / memory and it took a lot of effort to figure out where to place apps. In the end we ended up in the same situation as before, that some containers were on dedicated machines and the rest had a kind of hard limit on the max. # of instances per machine.
Our deployment process went from taking 2 minutes to ~10-15 minutes because of docker registry and extensive jenkins build scripts. It also needed lots of network bandwith to be transferred to the actual hosts. We had to regularly remove unused images because the docker registry or our docker hosts ran out of disk space.
Finally we changed our deployment process to incorporate docker hub because it was advertised as the premium docker building service. We paid for that service, but it took between 20 and 40 minutes to build our images. This was unacceptable for our use case because we needed to fix bugs near-instantly. Customer support from docker the company and their unresponsiveness towards our latency problems was another problem.
After the engineer in question left towards a very well-known european music streaming startup, we finally ditched docker and moved everything to AWS beanstalk. It does proper version management, config management, auto scaling and can be defined with a simple json file and some some straightforward cli tools.
I'm really not the biggest fan of docker, but it seems like everything y'all did was wrong: improper image caching, so you wouldn't have to continually build the same base images for your services; putting databases in the container, instead of somewhere else; not using cgroups/docker fundamentals to restrict space on the container level; storing state (something) on disk within a container.
You're right, there was a lot of PEBKAC involved. But it was not very straightforward to do the right thing in that situation.
The engineer in question was very excited about the technology, so we went with it. In the end it was a great learning on many levels, and I can really appreciate the kind of problems that AWS solves for you.
Generally containerization of our app was a good step at this point, but we made the mistake to also put redis / postgresql into docker, which obviously is an stupid idea if you're trying to scale things. On top of that, docker networking was a science on itself, and we didn't use any proper automation tools.
Docker containers kept running out of disk space / memory and it took a lot of effort to figure out where to place apps. In the end we ended up in the same situation as before, that some containers were on dedicated machines and the rest had a kind of hard limit on the max. # of instances per machine.
Our deployment process went from taking 2 minutes to ~10-15 minutes because of docker registry and extensive jenkins build scripts. It also needed lots of network bandwith to be transferred to the actual hosts. We had to regularly remove unused images because the docker registry or our docker hosts ran out of disk space.
Finally we changed our deployment process to incorporate docker hub because it was advertised as the premium docker building service. We paid for that service, but it took between 20 and 40 minutes to build our images. This was unacceptable for our use case because we needed to fix bugs near-instantly. Customer support from docker the company and their unresponsiveness towards our latency problems was another problem.
After the engineer in question left towards a very well-known european music streaming startup, we finally ditched docker and moved everything to AWS beanstalk. It does proper version management, config management, auto scaling and can be defined with a simple json file and some some straightforward cli tools.