On a related note, is anyone aware of any tool(s) to declaratively describe an application architecture and automate its deployment on something like AWS or in a single virtual machine + docker? For example "I want 10 replicated microservice1 processes and 5 replicated microservice2 processes in front of a nginx load balancer. Also, throw in a Redis and a PostgreSQL database." This would be immensely useful for (one man show) developers who don't necessarily have the time or experience to "properly" setup and fine tune infrastructure. I think what I'm describing is known as "Infrastructure as code" in DevOps parlance.
Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.
You download a BOSH release for your infrastructure, tweak the manifest for your chosen cloud, and push the deployment out. If you make a change, you just push it out.
Cloud Foundry itself (the elastic runtime) is a BOSH application that includes an HAPRoxy load balancer, dynamic router, and allows you to easily push/scale the stateless micro services and declaratively wire them up with other services you've deployed (with CF or Bosh).
It can run on AWS, vSphere, OpenStack, or on your own Vagrant VM in containers (bosh-lite). Or you can use Pivotal Web Services public cloud: http://run.pivotal.io
BOSH requires some wrestling early on but I find it to be a fascinating project.
It uses libvirt for provisioning, and Puppet for configuration. The coupling is extensive but not tight. It's also coupled to a specific way of deploying Java apps. There's a sort of backdoor that would let you deploy other kinds of apps by writing some Puppet code.
I do think the basic ideas are good, though. Possibly more of an object of study than a tool to actually use!
> Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.
Cloud66 can handle most of what you describe using manifest files. You can have a relatively simple yml file describing what types of servers exist in your stack - something like "I need Postgres servers, Redis, a RabbitMQ server, oh and here's the 3 different types of process servers I have and also how my main web server functions."
There's still a point where it's beneficial to split an application into seperate stacks, in which case the manifest concept breaks down and you're stuck doing a lot of work yourself, but they can take you a fairly long way.
> There's still a point where it's beneficial to split an application into seperate stacks
Do you mean like having a different "infrastructure manifest" for each micro-service? In that case, couldn't each micro-service specify which other micro-service it depends upon? For example, I could have a central "API gateway" service which would specify all the other micro-services it depends upon. This reminds me a bit of a Gemfile/Package.json but for processes instead of libraries. On the other hand, I can easily see how this could turn into a dependency hell.
Docker with CoreOS and fleet will get you pretty far on that front. You can just specify that you want 6 of container x running across the cluster, and 3 of container y.
Supplementing this with Hashicorp tools like Terraform and Consul can get you to the last step.
Salt/salt-cloud operates as you describe. It allows you to define any number of virtual machines associated to a cloud provider 'profile', i.e., instance type, size, platform, etc.
> Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.
Depending on your concept of "deploy", I can point you to two options, both produced by the company I work for.
If you think in terms of controlling from the VM up, look at BOSH[1]. It's an IaaS deployment / management / update tool which has been used in production for several years.
You create two yaml files: a release manifest (describing the software components of your system) and a deployment manifest (describing the computers and networks of your system and how to map components to them).
This separation means that a single release manifest can be applied to many different actual deployments. At my day job I deploy a large PaaS release into a single virtual machine. My colleagues deploy exactly the same release into AWS with thousands of machines.
Systems defined using BOSH can be deployed to AWS, vSphere, OpenStack or Warden (a containerisation system).
For local development, the Warden backend enables "BOSH lite"[2] -- releasing and deploying into a virtual machine that hosts a local cloud of containers.
If you want to go the next level up to a proper PaaS, you can try Cloud Foundry[3]. It's defined as a BOSH release[4], which is how at my day job my co-workers and I are able to work both locally and at scale on identical software.
In Cloud Foundry we distinguish between stateless (apps) and stateful (services) software, and we provide a simple way to connect them. The easiest way to play with it is to log into Pivotal Web Services. IBM Bluemix is the same software and is also open to the public.
So for example, with Cloud Foundry, if you want ten copies of your app, you do this:
Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.