Since this is from CloudFormation team, what can we expect about the performance vs calling the APIs directly? CloudFormation is not known for its speed.
Each API still needs to have its own custom body, with specific mandatory keys and values. The name of the verb is just another key and value ... so how is this different from existing APIs? Not different much.
The only addition introduced here is the support for third-party CloudFormation resources. So the same AWS APIs can now be used to directly manipulate third party resources without using CloudFormation at all.
Cloud Control provides a direct API to the primitive functions that underpin CloudFormation resources. So, if you look closely at what it's doing (hint: "desired state" parameter), Cloud Control is declarative on the resource side.
I don't think that replaces other AWS resource schedulers like Terraform and Pulumi, but it could, uh, embrace them into a more CloudFormation-oriented ecosystem, which is clever.
Pulumi lets me create custom providers, leverage providers for other clouds/ products, and build code-based abstractions. I see no reason why I'd stop using it.
Not really a moat, at least for HashiCorp. Consider that Pulumi was able to get off the ground by using Terraform providers, including HashiCorp's AWS provider, behind the scenes.
In addition to the other replies, Terraform has used ARM on Azure for a while now which is similar to this a unified API for all Azure Resource Management (hence ARM) and this hasn't caused any issues for them there.
In most places I've been, Terraform was used more for documentation than mobility. Sure, you don't need to unlearn Cloudformation if you switch to GCP, but you have a ton of AWS-specific stuff in your TF config that's going to need to be ported to the appropriate counterparts in your next cloud. But having a documented change history in Git for (almost) everything was much more useful than theoretical mobility.
I think this is really similar to the "we use kubernetes so we don't get locked into AWS" argument; true in theory, never pans out in practice.
I'd hazard that the number of engineering orgs using a single Terraform "stack" (repository? domain? whatever) to manage resources across multiple providers is in the single-digit percentages.
The ability to transfer knowledge has non-zero benefit, but the reality is; that knowledge is 10% "Terraform & HCL", 90% "AWS Resource Configuration & Best Practices". 90% of it doesn't transfer, and most of the remaining 10% could be learned quickly enough.
Maybe it enables you to transfer CI/tooling between cloud providers should your org switch. Probably the most reasonable argument.
I think the core of what you're saying is actually right, for the same reasons that the kubernetes statement above is right; AWS is really becoming its own isolated world. Terraform on AWS makes less and less sense, as their in-house options like CF, CDK, SAM, etc evolve maturity beyond what TF is capable of. TF (and Pulumi) (like Kubernetes) is for everything else.
Terraform has multi-cloud support but it is not cloud agnostic. You have to rewrite your terraform for the things that you want to move to another provider.
Moving to other providers is unfortunately rarely easy except for all but the most basic infrastructure.
I think that the main usage pattern for a tool like terraform is be able to version control your config, and , on the way, apply the git flow/git ops to infra changes (infra as code).
I don't think the (theoretical) ability to move to another infrastructure provider is the main motivation to use those tools. It might be a supporting motivation. But definitely not the real reason why they're using it. Hardly anybody actually changes their infrastructure provider anyway
I'll honestly just say it outright: this is horrible advice.
People using this argument has clearly never needed to be multi-cloud. If you can ever, ever, ever, avoid being absolutely forced into this, you should. Being able to move fast on infrastructure has such big implications, and being multi-cloud basically means that you can only rely on an extremely minimal subset of features that completely align across the clouds (hint: the less managed stuff).
You'd pretty much have to start from scratch switching between clouds even with Terraform. All the providers map pretty closely to the underlying nature of the resources defined in other clouds.
In what world does TF or pulumi code written for Aws runs on gcp or azure? I mean, maybe you can cook something with ts and pulumi but it is not realistic
I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward.
Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools.
But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model.
With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates.
At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!
Now if we could just use cfn-signal (or something like it) when setting up an EC2 auto-scaling group without having to create a Cloudformation stack behind the scenes as the Pulumi awsx package does.
Pardon my density, but isn't that the function of the health checks built into the ASG, which offers both EC2 and ELB health checking to take a bogus instance out of service?
Or do you mean "operation A --> ASG --> (wait $foo time for 'healthy') --> operation B" type thing?
Will it? Given how Pulumi can use Terraform providers, I'd say the hard work there isn't the connectors (which this change makes easy) but the framework and the orchestration bits. These aren't made any easier by AWS standardizing their APIs.
I wonder how long before this becomes a standard like swagger. People will create a spec for a well-known endpoint and probably call it CURDL that allows APIs to be self documenting and automatically compatible with a general universal API client. No one will have to maintain their own clients any more and everyone will wonder why it took so long to think of this...
There's Smithy, which tries to help design API operations around a resource in a CRUDL-like fashion - I think that's what you're getting at? https://awslabs.github.io/smithy/
I don’t understand how this changes something? Now the type of resource you want to create and many different bodies depending on that will be inside the CreateResource instead of whatever function the specific AWS API had.
The important difference is that the wait semantics are built in, meaning that each operation no longer needs custom code. This is a boon for things like Pulumi and Terraform providers.
I highly doubt that tools like Terraform/Pulumi will not have custom code for each type of resource. Each type still needs to have its own custom Body with specific data types, some mandatory, some optional. Even if TF supports Cloud Control as yet another resource, using it is going to be a huge mess of untyped/unchecked/undocumented bodies.
The bodies are specified in a schematised way, so all of this code can be generated. This is not hypothetical - Pulumi’s Kubernetes support, Azure (“nextgen”) support and GCP Native providers are all built like this.
It’s quite possible that the provided schema may need some augmentation for some resources (for example, semantic diff for IAM policies) to get the best plan-time experience, but this is a different world from having to hand write schema and CRUD logic for every resource type per the last 7 years of Terraform AWS support.
I feel like the more complex corners of the AWS resource model (e.g. offer-accept stuff between accounts, in TGW for example) are going to get over simplified behind this CRUDL API.
My use case is for mostly read-calls. I just tried it for DynamoDB Table, SQS Queue, SNS Topic, and EC2 Instance, Kinesis Stream. With these resources, only Kinesis supports List call. SQS Queue & SNS Topic does not even support READ. It might be useful for resource creation, but I would go nuts to discover what resource and what APIs are supported.
It is super interesting to me that they now have essentially a declarative API in CloudFormation and now this corresponding imperative version with the same format. I'm not quite sure why you would need the imperative version. Anyone have any ideas? Couldn't Pulumi have just compiled their code into CloudFormation and deployed it?
CloudFormation is really three separate things rolled up into one: (1) a resource model for AWS, (2) a deployment orchestration engine and (3) a syntax for specifying desired state as CloudFormation templates.
Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine.
Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more.
Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.
As a concrete example of what this enables, the Pulumi Automation API lets you embed IaC straight into a bigger program. Folks have used this to create infrastructure-oriented SaaS products, self-service portals, and new higher level abstraction frameworks and tools, for instance, often spanning multiple cloud resource types (AWS, Azure, Kubernetes, Cloudflare, others) -- https://www.pulumi.com/automation/. Transpiling to YAML and handing it over to CloudFormation is clumsy and wouldn't work for these cases among others.
Imperative version can fill more dynamic use cases, like spinning up entire servers in response to something (eg user wants to create a game server at the click of a button).
Not something you can easily do with the declarative version.
Yo more complexity !
They(AWS) should hire the UX team from "Hetzner Cloud Services" - sure it doesn't have the same features, but dammit it's super easy to navigate.
What does this change compared to the current status ? Terraform/Pulumi/AWS CDK already use AWS API. Is it just about having consistent API calls naming ?
The Cloud Control API is an API to the resource model. It unlocks the ability for AWS teams to contribute to the resource model without having to worry about the rest of what Cloudformation does. As a service owner interested in contributing to other systems like TF and Pulumi, now I just have to contribute to one thing. In the medium/long term it's going to be a lot more efficient way to evolve how we all use AWS resources in our favorite Infrastructure-as-code systems.
Disclosure: Work at AWS (but not on the CloudFormation team)
Just buy Pulumi already and have them do it the way it's supposed to be. It's ridiculous that AWS documentation usually contains dozens of error prone manual steps.
Not sure why you are getting downvoted, this was the first thing that sprang to mind for me.
"Oh, yes, another option." On top of terraform, pulumi, CloudFormation, the AWS SDKs, the CLI, and the CDK.
I mean, I appreciate them trying to push things forward and realize all options aren't all equivalent, but where's the deprecation to make things simpler?
(and why 'cloudcontrol' for that matter, not top-level CRUD)
Awscli loves its arguments. Several people are going to point me at autocompletions, tell me zsh has this really great feature where .. all fine. But why have this 'problem' to solve in the first place? I don't get it.
For non-interactive use, sure, I don't care how verbose it is, it's even a sort of self-documenting feature. Is that it? They just really want all interaction via the web console?
All AWS CLI commands follow the format 'aws [service] [action] [flags...]'. To my knowledge, there are exactly zero instances where positional arguments are used outside of naming the service and action; its always flags.
Whether this is best or not is a different discussion; I'm simply pointing out the irony of suggesting the CLI API for a service designed to make their APIs more consistent, should be inconsistent with the other CLI APIs customers are used to.
Yes I know, I was sort of using it (new CLI designed to be a better interface) to say why is it so.
Apparently I hold a minority opinion that's so offensively contrary it deserves downvotes though!
I maybe didn't make it clear enough, but I think it's somewhere between 'don't care either way' and 'great' for non-interactive/scripted use. I just don't like the UX at all for using it interactively, I find it extremely slow and cumbersome.
This looks to be a big improvement, but - to me - just leaves such easy obvious trivial stuff on the table.
The AWS CLI is old and for the most part auto-generated from the API definitions. I actually prefer the first example to the second one, but in general I’m happy that it’s consistent. Even if it’s sometimes consistently annoying (help, not —help).
What do you mean other than? I removed it, that is the (entire) difference.
> Warp it in a shell script
As I said, for non-interactive use of course I don't care. But trying to actually do/check/test anything with awscli interactively/'manually' is pretty painful IMO.
I disagree here. The first example is much more self-describing and follows the same paradigm that the rest of the AWS CLI already follows. I'd prefer erbosity here rather than the AWS CLI trying to figure out what I'm doing.
I'm of the same opinion - I'd rather not have to consider the argument order, and instead just be able to append flags to an interactive command in any order I see fit as I'm refining queries.
That is a solved issue . It is the standard way of command line syntax and provides a common way of parsing command line for many OS. You can certainly do things “another” way but all it will do is break decades worth of work for standardization and good old fashioned common sense. Make things modular so you can expand in different ways.
Well that's not really true, or at least incomplete. GNU & BSD have different styles; I believe both do positional/unnamed arguments.
It's true that awscli goes for all --named=args currently, but that's not decades of it not happening in general, and I just commented on wishing it weren't the case here.
I think this is a great simplification/abstraction, and I don't see why it couldn't go further and just be `aws <crud> <type> <resource>`. But that's apparently highly disagreeable even as a mere thought or suggestion, so I'll shutup.
Such verbosity (and other bloat and complexity in how AWS does things) is why I could never get excited about S3, EC2 et seq. when these services were first announced. Rightly or wrongly, I believe it is because most of AWS was originally implemented in Java. Programming in Java requires a certain comfort with verbosity and clunkiness. That showed through in early AWS interfaces and documentation. Indeed, it has alway been painful to interact with manually.
