In this blog post, I will cover two scenarios where I had to “extend” the CloudFormation support. I could achieve this by defining Custom Resources in the CloudFormation template, which are backed by Lambda functions.

Those constraints might not exist in future, as AWS rapidly implements new features, but the main takeaway is to understand that whenever you find a limitation, most likely you can overcome it by using Custom Resources backed by Lambda.

1. Background

I was working recently on a project, automating the entire CI/CD release pipeline and all the underlying infrastructure in AWS. I used CloudFormation heavily, and I was very happy with the result. In a matter of minutes, I’ve built from scratch a self contained ready to go environment for the developers, which included:

  • VPCs
  • CI/CD Pipeline (CodePipeline, CodeBuild)
  • Building the Docker images whenever a change is detected in source control
  • ECR and S3 repositories to store the Docker container images and artifacts
  • CI,DEV,TEST Stages with auto-scaling groups using ECS (Container service)

It was all good, until we got the requirement to be able to create non-persistent stacks. That means that we should be able to fully teardown the stacks, making sure that all the resources created by CloudFormation and also artifacts created by the stack are deleted. CloudFormation handles that for you when you delete the stack. However, there are some caveats, and that’s what we will be covering in this blog post.

2. The constraints

We found two limitations in CloudFormation when deleting the CloudFormation stack, which are well documented, and depending on the point of view, it can be considered a constraint by design:

  • Does not delete a S3 bucket if there are objects in the bucket
  • Does not delete an ECR repository if there are images in the repository

This is the error message you will get if you try to delete a CloudFormation stack, which tries to delete an ECR repository containing images:

Those two constraints are not an issue for long living stacks, and it’s actually a good thing to “protect” repositories on deletion (there are other ways to protect though, by using DeletionPolicy).

But in our case, we do want to be able to delete those repositories on stack deletion event.

3. Let’s take a look at a simple CloudFormation Template

In my use case, I have 8 CloudFormation templates building the different layers. For the scope of this Blog post, I will focus on the creation of one S3 bucket and one ECR repository only.

I will be using CloudFormation yaml (rather than JSON), to keep the examples shorter.

The following CloudFormation template will create one S3 bucket and one ECR repository.

A few things to note:

  • Resource name: I haven’t specified a name for the S3 Bucket, nor for the ECR repository. In this case, the resource name will be prefixed with your CloudFormation name, and will be suffixed by a random string generated by CloudFormation
  • Region: The resources will be created in the same region as you are creating your CloudFormation template

If you delete your CloudFormation stack, and both the S3 Bucket and ECR repository are empty, the stack will be successfully deleted.

However, if there are objects in either of the S3 bucket or ECR repository, the deletion will fail, as we explained earlier.

What if you want to keep those resources on stack deletion?

Let’s say you have a CloudFormation template, which creates a bunch of resources, and you have an S3 and an ECR repository. When you delete the stack, you might want to delete all resources, except the S3 bucket and the ECR repository. You can easily achieve that by using the DeletionPolicy attribute and set the value to Retain.

These are the available DeletionPolicy values, which you can specify:

  • Delete (default, if not specified)
  • Retain – Keep the resources intact. Can be used for any resource type
  • Snapshot – Takes a snapshot before deleting it (EC2 Volume, RDS Instance, ElasticCache, Redshift)

By default, if not specified, the DeletionPolicy value is Delete.

This is what our CloudFormation template would look like:

We will cover how to delete those resources in the next section.

4. Using CloudFormation Custom Resources to invoke a Lambda function

As a workaround to the current CloudFormation constraints, you can write a Lambda function in one of the supported languages (Python, NodeJS, Java or .NET). The Lambda function will then handle deleting the resources that cannot be deleted from CloudFormation out of the box.

However, you don’t need to do any external invocation to the Lambda function. It can be invoked from CloudFormation itself, as a Custom Resource.

In order to have the solution self-contained without any external dependencies, the CloudFormation template will have additional resources with the following responsibilities:

  • Create the Lambda function, retrieving the code from an S3 bucket
  • Create an IAM role which will be assumed by the Lambda function
  • Invoke the Lambda function

4.1 Let’s write some code to be invoked by CloudFormation

In this section, we will be writing some simple code, which will be invoked by CloudFormation.

At the end, we will be packaging them together and make it available in our S3 bucket.

4.1.1 Generic function to create the response to notify CloudFormation

First we will create a Python file called cloudformationnotify.py, which has a single function called send.

This function will receive the following arguments from another Python code, which will contain the logic to delete the resources, and will be the main function, which will be invoked by CloudFormation:

  • event – the original event received in your Lambda function. That’s essentially the entire input payload
  • context – the original context received in your Lambda function
  • responseStatus – either FAILED or SUCCESS. If it’s FAILED, it will tell the CloudFormation stack to fail and interrupt the process.
  • responseData – The payload returned to CloudFormation
  • physicalResourceId – The Physical resource Id for your custom resource. Since in our context we are going to use the Custom resource on stack deletion only, it’s not particularly relevant. However, if you are dealing with stack updates, you might need to handle defining different physicalResourceIds to let CloudFormation know when there is a resource replacement, for example.

4.1.2 Delete S3 bucket content Lambda function

We will create a Python file called: delete-s3-bucket-content.py

This Lambda function will essentially delete the content from the S3 bucket, once it receives a Delete stack event from CloudFormation.

These are important aspects of this function:

  • Import send function from cloudformationnotify.py Python file, as discussed in the previous section
  • If the CloudFormation event=’Delete’, then deletes the content from the bucket
  • Invokes the send() function, specifying if it was SUCCESS or FAILED, and then returns the response to ClouFormation

Note: If your stack is in progress for a long time, and it seems to be stuck, there is a good chance that your Custom Resource didn’t send the notification to CloudFormation. Make sure that you handle the exceptions, and always return the notification to CloudFormation.

4.1.3 Delete ECR repository Lambda function

We will create a Python file called: delete-ecr-content.py

This Lambda function will essentially delete the ECR repository, once it receives a Delete stack event from CloudFormation. We could just delete the ECR images, as our main requirement is to make sure that the repository is empty for CloudFormation to delete the empty repository. However, in order to keep it simpler I’m deleting the repository itself, in order to avoid additional edge cases to handle.

These are important aspects of this function:

  • Import send function from cloudformationnotify.py Python file, as discussed in the previous section
  • If the CloudFormation event=’Delete’, then deletes the ECR repository. force=true makes sure that it will delete the repository even if the repository is not empty.
  • Invokes the send() function, specifying if it was SUCCESS or FAILED, and returns the response to CloudFormation

4.1.4 Packaging the Python code and distributing into a S3 bucket

In order to have CloudFormation consuming the Python code to create the Lambda function, we will store a package in a S3 Bucket.

We need to perform the following steps:

  • Create a zip file called lambda-utilities.zip, which will contain the following 3 files:
    • cloudformationnotify.py
    • delete-s3-bucket-content.py
    • delete-ecr-content.py
  • Create a S3 bucket in our AWS account. Let’s call this bucket as mybucket-test-delete-resources (note that the bucket name is globally unique for all the AWS accounts, so you will need to chose a different name)
  • Copy the lambda-utilities.zip file to your S3 bucket

4.2 CloudFormation Templates

We will now create a CloudFormation template to create an S3 bucket and an ECR repository. As discussed, the CloudFormation template will also handle the deletion of resources. In order to keep it simpler, I created two separate CloudFormation templates. They follow pretty much the same structure.

4.2.1 CloudFormation Template for the S3 Bucket

Let’s call the CloudFormation template file cf-custom-s3.yaml

This is a brief description of the main sections:

  • Parameters
    • S3BucketUtilities: This is the parameter which you should provide the bucket where you’ve placed the lambda-utilities.zip. Note that the default value will be used in case you don’t provide it. You can override it when you create the cloudFormation stack
    • S3KeyUtilityPackage: The  name of the zip file which contains the python code, packaged in previous section
  • Resources
    • ArtifactBucket – Defines the S3 Bucket to be created
    • DeleteBucketContentFunction – Creates the Lambda function, retrieving the package from S3, and assigns an IAM role defined in DeleteBucketContentLambdaExecutionRole
    • DeleteBucketContentLambdaExecutionRole – Creates an IAM role, which will allow to be assumed by the Lambda function. The IAM role allows logs to be created in Cloudwatch Logs, as well as listing the S3 bucket and deleting its content.
    • DeleteArtifactBucketContent – Invokes the Lambda function to delete the S3 content. Note that it DependsOn ArtifactBucket, therefore on create will be invoked after the creation. On delete, will be invoked in the reverse order, just before deleting the S3 bucket by CloudFormation. It will provide BucketName as a parameter

 

4.2.2 CloudFormation Template for the ECR Repository

Let’s call the CloudFormation template file cf-custom-ecr.yaml

This is a brief description of the main sections:

  • Parameters
    • S3BucketUtilities: This is the parameter which you should provide the bucket where you’ve placed the lambda-utilities.zip. Note that the default value will be used in case you don’t provide it. You can override it when you create the cloudFormation stack
    • S3KeyUtilityPackage: The  name of the zip file which contains the python code, packaged in previous section
  • Resources
    • Repository – Defines the ECR Repository to be created
    • DeleteRepositoryFunction – Creates the Lambda function, retrieving the package from S3, and assigns an IAM role defined in DeleteRepositoryLambdaExecutionRole
    • DeleteRepositoryLambdaExecutionRole – Creates an IAM role, which will allow to be assumed by the Lambda function. The IAM role allows logs to be created in Cloudwatch Logs, as well as deleting the ECR repository created by this CloudFormation template.
    • DeleteRepository – Invokes the Lambda function to delete ECR repository. Note that it DependsOn Repository, therefore on create will be invoked after the creation of the ECR repository. On delete, will be invoked in the reverse order, just before deleting the ECR repository by CloudFormation. It will provide as parameters:
      • RegistryId=AWS account ID
      • RepositoryName=Repository name generated by CloudFormation which was assigned to your ECR repository

4.3 CloudFormation Custom Resource Payloads

Request Payload sample from CloudFormation -> Lambda function:

Response Payload Lambda -> CloudFormation

 

5. Validating the solution end to end

For the purpose of this blog post, I will show how to create/delete a CloudFormation stack based on the cf-custom-s3 template.

5.1 Creating the CloudFormation stack

  • In your AWS console, navigate to Services -> CloudFormation -> Create Stack
  • Then select the cf-custom-s3.yaml template, as follows, and then click on Next

  • Stack Details
    • On the stack name, you can type the name for your CloudFormation stack. I will name ours as poc5
    • Type the name of your S3 bucket, where you copied the lambda-utilities.zip, on the S3BucketUtilities parameter
    • Click on Next

  • Options – Don’t make any change, just click on Next
  • Make sure that you select the checkbox for CloudFormation to create IAM resources, and click on Create

  • Once the stack is created, you will see a CREATE_COMPLETE status as follows

5.2 Copy a few files to the bucket created by your CloudFormation template

  • Navigate to Services -> S3
  • You should see a bucket which prefix is: poc5-artifactbucket
  • Copy a few files into that bucket

5.3 Deleting the CloudFormation stack

  • In your AWS console, navigate to Services -> CloudFormation
  • Select the poc5 CloudFormation stack, and click on Delete Stack, as follows:

The stack should be deleted successfully

 6. Conclusion

In this Blog post we covered an approach for overcoming constraints in CloudFormation, by using Custom Resources backed by Lambda Functions.

It is a bit of an inconvenience having the constraints mentioned about S3 bucket and ECR repository deletion. Those constraints might change in future, but the main takeaway is knowing how to extend CloudFormation to work around these constraints.

Have you come across anything similar? If so, how did you handle the constraints?

0 Comments
Join the conversation

Your email address will not be published. Required fields are marked *

Leave a Reply