AWS

Dealing with AWS Secrets Manager managed credential rotation

How to run arbitrary code the Secrets Manager rotates.


First things first, how to rotate.

When creating a secret you can toggle or not the credential rotation.

The specifics of credential rotation are out of scope of this article, although guidance can be found here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html

Using terraform

I generally use this snippet to rotate credentials on any secret, just provide its ARN and forget about it.

resource "aws_secretsmanager_secret_rotation" "example" {
  secret_id = <arn>
  rotation_rules {
    automatically_after_days = 180
  }
}
 

If more complex configuration is required please take a look at: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret_rotation

The 'Right Way', using AWS SDK.

Short answer: use the SDK. Why does that solve the issue with credential rotations ? If your application fetches the credentials programmatically it will always have the latest version available, and you will have no risk of leaking it with hardcoded credentials hanging around.

But there is no single right way of handling credential rotations, however we do recommend that your application is AWS-aware and is able to fetch its credentials directly from AWS itself. To achieve this you need to install the AWS SDK and issue the get-secret-value call.

AWS itself recommends handling it this way, when you are about to create your secret on the console AWS shows you a list of languages (Java, Javascript, C#, Python3, Ruby, Go and Rust) and an example of a snippet to retrieve the secret directly from AWS. They even let you download the SDK right then and there.

Now what happens if you cannot install the SDK for some reason ? There's a way to run arbitrary code when the rotation ends.

Dealing with applications that cannot use AWS SDK.

If you are using a lambda function to rotate your secret you obviously don't need a way of running code after that rotation because you can run any kind of code at rotation time. As an example, imagine you already have a lambda that regenerates a password according to company policy, now imagine that a new requirement arrived and that you now have to issue an API (Or an e-mail for that matter) call to one of your vendors when that happens.

In this case you already have the lambda, the process of including a snippet before or after (or both) is fairly simple. However, what happens when you don't have the lambda and the secret is rotated automatically by a third party service ? That's exactly the case when using RDS managed secrets (and a fairly common use case).

Short answer: Use EventBridge rules to trigger a lambda.

Long answer: The goal here is to catch the event  Rotation Succeeded from secrets manager and invoke a lambda to handle whatever we need it to handle.

When building the rule use the following pattern:

{
  "detail": {
    "additionalEventData": {
      "SecretId": ["arn"]
    },
    "eventName": ["RotationSucceeded"],
    "eventSource": ["secretsmanager.amazonaws.com"]
  },
  "source": ["aws.secretsmanager"]
}
 

Just insert your secret arn and your rule is almost ready. Add the desired lambda arn as a target and the rule is ready. You can also add multiple targets, meaning you can reduce the surface for code errors by adding multiple services.

Here is an example of a lambda function and a SNS topic as targets:

resource "aws_rds_cluster" "default" {
  cluster_identifier              = "id"
  engine                          = "aurora-mysql"
  availability_zones              = local.azs
  database_name                   = "example"
  manage_master_user_password     = true # <-- very important
  master_username                 = "coolmasterusername"
  network_type                    = "IPV4"
  port                            = 3306
  storage_encrypted               = true
 
  # Networking
  vpc_security_group_ids = [
    aws_security_group.rds.id
  ]
  db_subnet_group_name = module.vpc.database_subnet_group_name
 
  # Deletion protection
  deletion_protection = true
  lifecycle {
    prevent_destroy = true
    ignore_changes = [
      availability_zones,
    ]
  }
}
 
# Use the output of the `master_user_secret` object, which includes `secret_arn`,
# to manage the rotation rules.
resource "aws_secretsmanager_secret_rotation" "example" {
  secret_id = aws_rds_cluster.default.master_user_secret[0].secret_arn
 
  rotation_rules {
    automatically_after_days = 180
  }
}
 
resource "aws_cloudwatch_event_rule" "rotation" {
  name        = "name"
  description = "Capture RDS key rotation"
 
  event_pattern = jsonencode({
    "source" : ["aws.secretsmanager"],
    "detail" : {
      "eventSource" : ["secretsmanager.amazonaws.com"],
      "eventName" : ["RotationSucceeded"],
      "additionalEventData" : {
        "SecretId" : [aws_rds_cluster.default.master_user_secret[0].secret_arn]
      }
    }
  })
}
resource "aws_sns_topic" "rotation" {
  name              = "name"
  kms_master_key_id = "alias/aws/sns"
}
resource "aws_cloudwatch_event_target" "sns" {
  rule      = aws_cloudwatch_event_rule.rotation.name
  target_id = "SendToSNS"
  arn       = aws_sns_topic.rotation.arn
}
resource "aws_sns_topic_policy" "rotation" {
  arn = aws_sns_topic.rotation.arn
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Allow EB to publish"
        Effect = "Allow"
        Action = "SNS:Publish"
        Principal = {
          Service = "events.amazonaws.com"
        }
        Resource = [aws_sns_topic.rotation.arn]
      },
    ]
  })
}
resource "aws_cloudwatch_event_target" "lambda" {
  arn  = lambda_arn
  rule = aws_cloudwatch_event_rule.rotation.name
}
 

Don't forget to add permissions on your lambda function to allow events.amazonaws.com to invoke it.

Conclusion

This should now be the default way of handling the credential rotation. Again, ideally your application should fetch the credentials directly from AWS. If that's not an option, then use the methods described above.

Similar posts

Get notified on new marketing insights

Be the first to know about new B2B SaaS Marketing insights to build or refine your marketing function with the tools and knowledge of today’s industry.