 |
Today, I am happy to announce the availability of AWS Cloud Control API a set of common application programming interfaces (APIs) that are designed to make it easy for developers to manage their AWS and third-party services.
AWS delivers the broadest and deepest portfolio of cloud services. Builders leverage these to build any type of cloud infrastructure. It started with Amazon Simple Storage Service (Amazon S3) 15 years ago and grew over 200+ services. Each AWS service has a specific API with its own vocabulary, input parameters, and error reporting. For example, you use the S3 CreateBucket API to create an Amazon Simple Storage Service (Amazon S3) bucket and the Amazon Elastic Compute Cloud (Amazon EC2) RunInstances API to create an EC2 instances.
Some of you use AWS APIs to build infrastructure-as-code, some to inspect and automatically improve your security posture, some others for configuration management, or to provision and to configure high performance compute clusters. The use cases are countless.
As applications and infrastructures become increasingly sophisticated and you work across more AWS services, it becomes increasingly difficult to learn and manage distinct APIs. This challenge is exacerbated when you also use third-party services in your infrastructure, since you have to build and maintain custom code to manage both the AWS and third-party services together.
Cloud Control API is a standard set of APIs to Create, Read, Update, Delete, and List (CRUDL) resources across hundreds of AWS Services (more being added) and dozens of third-party services (and growing).
It exposes five common verbs (CreateResource, GetResource, UpdateResource, DeleteResource, ListResource) to manage the lifecycle of services. For example, to create an Amazon Elastic Container Service (Amazon ECS) cluster or an AWS Lambda function, you call the same CreateResource API, passing as parameters the type and attributes of the resource you want to create: an Amazon ECS cluster or an Lambda function. The input parameters are defined by an unified resource model using JSON. Similarly, the return types and error messages are uniform across all verbs and all resources.
Cloud Control API provides support for hundreds of AWS resources today, and we will continue to add support for existing AWS resources across services such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3) in the coming months. It will support new AWS resources typically on the day of launch.
Until today, when I want to get the details about an Lambda function or a Amazon Kinesis stream, I use the get-function API to call Lambda and the describe-stream API to call Kinesis. Notice in the example below how different these two API calls are: they have different names, different naming conventions, different JSON outputs, etc.
aws lambda get-function –function-name TictactoeDatabaseCdkStack { “Configuration”: { “FunctionName”: “TictactoeDatabaseCdkStack”, “FunctionArn”: “arn:aws:lambda:us-west-2:0123456789:function:TictactoeDatabaseCdkStack”, “Runtime”: “nodejs14.x”, “Role”: “arn:aws:iam::0123456789:role/TictactoeDatabaseCdkStack”, “Handler”: “framework.onEvent”, “CodeSize”: 21539, “Timeout”: 900, “MemorySize”: 128, “LastModified”: “2021-06-07T11:26:39.767+0000”, … aws kinesis describe-stream –stream-name AWSNewsBlog { “StreamDescription”: { “Shards”: [ { “ShardId”: “shardId-000000000000”, “HashKeyRange”: { “StartingHashKey”: “0”, “EndingHashKey”: “340282366920938463463374607431768211455” }, “SequenceNumberRange”: { “StartingSequenceNumber”: “49622132796672989268327879810972713309953024040638611458” } } ], “StreamARN”: “arn:aws:kinesis:us-west-2:012345678901:stream/AWSNewsBlog”, “StreamName”: “AWSNewsBlog”, “StreamStatus”: “ACTIVE”, “RetentionPeriodHours”: 24, “EncryptionType”: “NONE”, “KeyId”: null, “StreamCreationTimestamp”: “2021-09-17T14:58:20+02:00” } }
In contrast, when using the Cloud Control API, I use a single API name get-resource, and I receive a consistent output.
aws cloudcontrol get-resource –type-name AWS::Kinesis::Stream –identifier NewsBlogDemo { “TypeName”: “AWS::Kinesis::Stream”, “ResourceDescription”: { “Identifier”: “NewsBlogDemo”, “Properties”: “{“Arn”:”arn:aws:kinesis:us-west-2:486652066693:stream/NewsBlogDemo”,”RetentionPeriodHours”:168,”Name”:”NewsBlogDemo”,”ShardCount”:3}” } }
Similary, to create the resource above I used the create-resource API.
aws cloudcontrol create-resource –type-name AWS::Kinesis::Stream –desired-state “{“Name”: “NewsBlogDemo”,”RetentionPeriodHours”:168, “ShardCount”:3}”
In my opinion, there are three types of builders that are going to adopt Cloud Control API:
Builders
The first community is builders using AWS Services APIs to manage their infrastructure or their customer’s infrastructure. The ones requiring usage of low-level AWS Services APIs rather than higher level tools. For example, I know companies that manages AWS infrastructures on behalf of their clients. Many developed solutions to list and describe all resources deployed in their client’s AWS Accounts, for management and billing purposes. Often, they built specific tools to address their requirements, but find it hard to keep up with new AWS Services and features. Cloud Control API simplifies this type of tools by offering a consistent, resource-centric approach. It makes easier to keep up with new AWS Services and features.
Another example: Stedi is a developer-focused platform for building automated Electronic Data Interchange (EDI) solutions that integrate with any business system. “We have a strong focus on infrastructure as code (IaC) within Stedi and have been looking for a programmatic way to discover and delete legacy cloud resources that are no longer managed through CloudFormation – helping us reduce complexity and manage cost,” said Olaf Conjin, Serverless Engineer at Stedi, Inc. “With AWS Cloud Control API, our teams can easily list each of these legacy resources, cross-reference them against CloudFormation managed resources, apply additional logic and delete the legacy resources. By deleting these unused legacy resources using Cloud Control API, we can manage our cloud spend in a simpler and faster manner. Cloud Control API allows us to remove the need to author and maintain custom code to discover and delete each type of resource, helping us improve our developer velocity”.
APN Partners
The second community that benefits from Cloud Control API is APN Partners, such as HashiCorp (maker of Terraform) and Pulumi, and other APN Partners offering solutions that relies on AWS Services APIs. When AWS releases a new service or feature, our partner’s engineering teams need to learn, integrate, and test a new set of AWS Service APIs to expose it in their offerings. This is a time consuming process and often leads to a lag between the AWS release and the availability of the service or feature in their solution. With the new Cloud Control API, partners are now able to build a unique REST API code base, using unified API verbs, common input parameters, and common error types. They just have to merge the standardized pre-defined uniform resource model to interact with new AWS Services exposed as REST resources.
Launch Partners
HashiCorp and Pulumi are our launch partners, both solutions are integrated with Cloud Control API today.
HashiCorp provides cloud infrastructure automation software that enables organizations to provision, secure, connect, and run any infrastructure for any application. “AWS Cloud Control API makes it easier for our teams to build solutions to integrate with new and existing AWS services,” said James Bayer – EVP Product, HashiCorp. “Integrating HashiCorp Terraform with AWS Cloud Control API means developers are able to use the newly released AWS features and services, typically on the day of launch.”
Pulumi’s new AWS Native Provider, powered by the AWS Cloud Control API, “gives Pulumi’s users faster access to the latest AWS innovations, typically the day they launch, without any need for us to manually implement support,” said Joe Duffy, CEO at Pulumi. “The full surface area of AWS resources provided by AWS Cloud Control API can now be automated from familiar languages like Python, TypeScript, .NET, and Go, with standard IDEs, package managers, and test frameworks, with high fidelity and great quality. Using this new provider, developers and infrastructure teams can develop and ship modern AWS applications and infrastructure faster and with more confidence than ever before.”
To learn more about HashiCorp and Pulumi’s integration with Cloud Control API, refer to their blog post and announcements. I will add the links here as soon as they are available.
AWS Customers
The third type of builders that will benefit from Cloud Control API is AWS customers using solution such as Terraform or Pulumi. You can benefit from Cloud Control API too. For example, when using the new Terraform AWS Cloud Control provider or Pulumi’s AWS Native Provider, you can benefit from availability of new AWS Services and features typically on the day of launch.
Now that you understand the benefits, let’s see Cloud Control API in action.
How It Works?
To start using Cloud Control API, I first make sure I use the latest AWS Command Line Interface (CLI) version. Depending on how the CLI was installed, there are different methods to update the CLI. Cloud Control API is available from our AWS SDKs as well.
To create an AWS Lambda function, I first create an index.py handler, I zip it, and upload the zip file to one of my private bucket. I pay attention that the S3 bucket is in the same AWS Region where I will create the Lambda function:
cat << EOF > index.py heredoc> import json def lambda_handler(event, context): return { ‘statusCode’: 200, ‘body’: json.dumps(‘Hello from Lambda!’) } EOF zip index.zip index.py aws s3 cp index.zip s3://private-bucket-seb/index.zip
Then, I call the create-resource API, passing the same set of arguments as required by the corresponding CloudFormation resource. In this example, the Code, Role, Runtime, and Handler arguments are mandatory, as per the CloudFormation AWS::Lambda::Function documentation.
aws cloudcontrol create-resource –type-name AWS::Lambda::Function –desired-state ‘{“Code”:{“S3Bucket”:”private-bucket-seb”,”S3Key”:”index.zip”},”Role”:”arn:aws:iam::0123456789:role/lambda_basic_execution”,”Runtime”:”python3.9″,”Handler”:”index.lambda_handler”}’ –client-token 123 { “ProgressEvent”: { “TypeName”: “AWS::Lambda::Function”, “RequestToken”: “56a0782b-2b26-491c-b082-18f63d571bbd”, “Operation”: “CREATE”, “OperationStatus”: “IN_PROGRESS”, “EventTime”: “2021-09-26T12:05:42.210000+02:00” } }
I may call the same command again to get the status or to learn about an eventual error:
aws cloudcontrol create-resource –type-name AWS::Lambda::Function –desired-state ‘{“Code”:{“S3Bucket”:”private-bucket-seb”,”S3Key”:”index.zip”},”Role”:”arn:aws:iam::0123456789:role/lambda_basic_execution”,”Runtime”:”python3.9″,”Handler”:”index.lambda_handler”}’ –client-token 123 { “ProgressEvent”: { “TypeName”: “AWS::Lambda::Function”, “Identifier”: “ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9”, “RequestToken”: “f634b21d-22ed-41bb-9612-8740297d20a3”, “Operation”: “CREATE”, “OperationStatus”: “SUCCESS”, “EventTime”: “2021-09-26T19:46:46.643000+02:00” } }
Here, the OperationStatus is SUCCESS and the function name is ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 (I can pass my own name if I want something more descriptive 🙂 )
I then invoke the Lambda function to ensure it works as expected:
aws lambda invoke –function-name ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 out.txt && cat out.txt && rm out.txt { “StatusCode”: 200, “ExecutedVersion”: “$LATEST” } {“statusCode”: 200, “body”: “”Hello from Lambda!””}
When finished, I delete the Lambda function using Cloud Control API:
aws cloudcontrol delete-resource –type-name AWS::Lambda::Function –identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 { “ProgressEvent”: { “TypeName”: “AWS::Lambda::Function”, “Identifier”: “ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9”, “RequestToken”: “8923991d-72b3-4981-8160-4d9a585965a3”, “Operation”: “DELETE”, “OperationStatus”: “IN_PROGRESS”, “EventTime”: “2021-09-26T20:06:22.013000+02:00” } }
Idempotency
You might have noticed the client-token parameter I passed to the create-resource API call. Create, Update, and Delete requests all accept a ClientToken, which is used to ensure idempotency of the request.
- We recommend always passing a client token. This will disambiguate requests in case a retry is needed. Otherwise, you may encounter unexpected errors like ConcurrentOperationException or AlreadyExists.
- We recommend that client tokens always be unique for every single request, such as by passing a UUID.
One More Thing
At the heart of AWS Cloud Control API source of data, there is the CloudFormation Public Registry, which my colleague Steve announced earlier this month in this blog post. It allows anyone to expose a set of AWS resources through CloudFormation and AWS CDK. This is the mechanism AWS Service teams are now using to release their services and features as CloudFormation and AWS CDK resources. Multiple third-party vendors are also publishing their solutions in the CloudFormation Public Registry. All resources published are modelled with a standard schema that defines the resource, its properties, and their attributes in a uniform way.
AWS Cloud Control API is a CRUDL API layer on top of resources published in the CloudFormation Public Registry. Any resource published in the registry exposes its attributes with standard JSON schemas. The resource can then be created, updated, deleted, or listed using Cloud Control API with no additional work.
For example, imagine I decide to expose a public CloudFormation stack to let any AWS customer create VPN servers, based on EC2 instances. I model the VPNServer resource type and publish it in the CloudFormation Public Registry. With no additional work on my side, my custom resource “VPNServer” is now available to all AWS customers through the Cloud Control API REST API. Not only, it is also automatically available through solutions like Hashicorp’s Terraform and Pulumi, and potentially others who adopt Cloud Control API in the future.
It is worth mentioning Cloud Control API is not aimed at replacing the traditional AWS service-level APIs. They are still there and will always be there, but we think that Cloud Control API is easier and more consistent to use and you should use it for new apps.
Availability and Pricing
Cloud Control API is available in all AWS Regions, except China.
You will only pay for the usage of underlying AWS resources, such as a CloudWatch logs or Lambda functions invocations, or pay for the number of handler operations and handler operation duration associated with using third-party resources (such as Datadog monitors or MongoDB Atlas clusters). There are no minimum fees and no required upfront commitments.
I can’t wait to discover what you are going to build on top of this new Cloud Control API. Go build!
