Introduction to Data Lifecycle Manager – Create EBS Snapshot/AMI periodically

2023-03-192024-04-21

Introduction to Data Lifecycle Manager – Create EBS Snapshot/AMI periodically

Data Lifecycle Manager (DLM) can be used to automate instance and EBS backups.

You can use Amazon Data Lifecycle Manager to automate the creation, retention, and deletion of EBS snapshots and EBS-backed AMIs.
Amazon Data Lifecycle Manager

This page uses DLM to periodically create AMI and EBS snapshots of the instance.

Environment

Diagram of Introduction to Data Lifecycle Manager - Create EBS Snapshot / AMI periodically.

Create an EC2 instance and EBS for verification purposes.

Create three different DLM policies for the above two resources.

Policy for periodically creating AMIs of EC2 instances
Policy for periodically creating snapshots of EC2 instance volumes
Policy for periodically creating snapshots of EBS

In common, set up a lifecycle policy so that an AMI/snapshot is created every hour and the two most recent ones remain.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL

https://github.com/awstut-an-r/awstut-soa/blob/main/02/003/soa-02-003.yaml

Explanation of key points of template files

DLM

Create three DLM policies to create snapshots of AMI, EC2 instance volumes, and EBS.

Policy for AMI

Resources:
  AmiLifecyclePolicy:
    Type: AWS::DLM::LifecyclePolicy
    Properties:
      Description: !Sub "${Prefix}-LifecyclePolicy-AMI"
      ExecutionRoleArn: !GetAtt AWSDataLifecycleManagerDefaultRoleForAMIManagement.Arn
      PolicyDetails:
        Parameters:
          NoReboot: false
        PolicyType: IMAGE_MANAGEMENT
        ResourceTypes:
          - INSTANCE
        Schedules:
          - CreateRule:
              Interval: 1
              IntervalUnit: HOURS
              Times:
                - "20:00"
            DeprecateRule: # AMI only.
              Count: 1
            Name: test-ami-policy
            RetainRule:
              Count: 2
        TargetTags:
          - Key: !Ref DLMTagKey1
            Value: !Ref DLMTagAmiValue
      State: ENABLED
Code language: YAML (yaml)

The PolicyDetails property is used to configure the specific settings of the DLM policy.

The Parameters property allows you to set optional parameters for the DLM policy.
In the case of a policy for AMI, you can set whether or not the instance should be restarted at the time of AMI creation.
For this policy, specify “false” to not restart the instance.

The PolicyType property specifies the resource to be created by the policy.
To create an AMI, specify “IMAGE_MANAGEMENT” for this property.

To create a DLM policy for AMI, specify “INSTANCE” for the ResourceTypes property.

The Schedules property sets the conditions for when AMIs are created and when old AMIs are deleted.

Set the timing for creating AMI with the CreateRule property.
In this case, set it to work under the following conditions.

An AMI will be created every hour.
AMI creation starts at 20:00 (UTC).
Delete old AMIs so that the number of AMIs is two.
When deleting AMIs, delete one at a time.

In the TargetTags property, specify the target instance for which the AMI is to be created.
In this case, we will target instances with the following tags

Tag name: dlm1
Tag Value: ami

Policy for instance volume snapshots

Resources:
  InstanceLifecyclePolicy:
    Type: AWS::DLM::LifecyclePolicy
    Properties:
      Description: !Sub "${Prefix}-LifecyclePolicy-Instance"
      ExecutionRoleArn: !GetAtt AWSDataLifecycleManagerDefaultRole.Arn
      PolicyDetails:
        Parameters:
          ExcludeBootVolume: false
        PolicyType: EBS_SNAPSHOT_MANAGEMENT
        ResourceTypes:
          - INSTANCE
        Schedules:
          - CreateRule:
              Interval: 1
              IntervalUnit: HOURS
              Times:
                - "12:40"
            Name: test-ami-policy
            RetainRule:
              Count: 2
        TargetTags:
          - Key: !Ref DLMTagKey2
            Value: !Ref DLMTagInstanceValue
      State: DISABLED
Code language: YAML (yaml)

Generally the same as the policy for AMI.
The main differences are discussed below.

To create a snapshot of the instance’s volume, specify “EBS_SNAPSHOT_MANAGEMENT” for the PolicyType property.

This time, specify the TargetTags property as follows

Tag Name: dlm2
Tag Value: instance

Policy for EBS snapshots

Resources:
  EBSLifecyclePolicy:
    Type: AWS::DLM::LifecyclePolicy
    Properties:
      Description: !Sub "${Prefix}-LifecyclePolicy-EBS"
      ExecutionRoleArn: !GetAtt AWSDataLifecycleManagerDefaultRole.Arn
      PolicyDetails:
        PolicyType: EBS_SNAPSHOT_MANAGEMENT
        ResourceTypes:
          - VOLUME
        Schedules:
          - CreateRule:
              Interval: 1
              IntervalUnit: HOURS
              Times:
                - "12:50"
            Name: test-ami-policy
            RetainRule:
              Count: 2
        TargetTags:
          - Key: !Ref DLMTagKey3
            Value: !Ref DLMTagEbsValue
      State: DISABLED
Code language: YAML (yaml)

Generally the same as the aforementioned policy.
The main differences are discussed below.

To create a snapshot of the instance’s volume, specify “VOLUME” for the ResourceTypes property.

This time, specify the TargetTags property as follows

Tag Name: dlm3
Tag Value: ebs

IAM Roles for DLM

IAM Role for AMI Policy

Resources:
  AWSDataLifecycleManagerDefaultRoleForAMIManagement:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - dlm.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRoleForAMIManagement
Code language: YAML (yaml)

The permissions required to execute the policy for AMI are provided as the AWS Management Policy AWSDataLifecycleManagementServiceRoleForAMIManagement.

Create an IAM role with this AWS management policy attached.

IAM role for instance volumes and EBS

Resources:
  AWSDataLifecycleManagerDefaultRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - dlm.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole
Code language: YAML (yaml)

The permissions required to run the policy for instance volumes and EBS are provided as an AWS management policy AWSDataLifecycleManagerServiceRole.

Create an IAM role with this AWS management policy attached.

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.

https://awstut.com/en/2021/12/11/cloudformations-nested-stack

After reviewing the resources in each stack, information on the main resources created in this case is as follows

EC2 instance: i-068b544013614d289
EBS: vol-0daa7dfef543973b8
DLM policy for AMI: policy-0ea4b29c277e69b8a
DLM policy for instances: policy-04c02847b362179d3
DLM policy for EBS: policy-00eceb6c181303e2a

Check various resources from the AWS Management Console.

Check the DLM policy for AMI.

You can see that the three policies have been successfully created.

Check the EC2 instance and EBS.

You can see that both have been successfully created.
You can see that there are two tags set for the instance and one for the EBS for the DLM policy.

Operation Check

DLM Policy for AMI

1st

Confirm the operation from the policy for AMI.
Wait for a while.

About an hour after the policy was created, the first AMI was created.
You can see that a snapshot associated with the AMI was also created.
Since this instance has an EBS attached, in addition to the snapshot of the instance volume, a snapshot of the EBS was also created.

2nd

Wait another hour or so.

A second AMI has been created.
The total number of snapshots is now four.

3rd

Wait another hour or so.

A third AMI and snapshot have been created.

The policy is set to keep the two most recent, but there are three left.
It seems that immediately after the DLM policy is activated, there may be more AMI/snapshots stored than the specified value.

After 20-30 minutes, check the AMI/snapshot again.

There are now two AMIs.
A lifecycle policy has been implemented so that the two newest remain.

The snapshot was not linked to the AMI and remained for some time.

DLM Policy for Instance

1st

Activate the DLM policy for the instance and wait a moment.

The first snapshot was created approximately one hour after the policy was activated.
You can see that two snapshots were created at once.
Since this instance has an EBS attached, in addition to the snapshot of the instance volume, a snapshot of the EBS was also created.

2nd

Wait another hour or so.

A second snapshot has been created.
The total number of snapshots is now four.

3rd

Wait another hour or so.

The third snapshot has been created.
The policy is set to keep the two most recent snapshots, but three snapshots are being kept at a time.
After all, it seems that immediately after the DLM policy is activated, snapshots that exceed the specified value may be saved.

After 20 to 30 minutes, check the snapshot again.

Snapshots are now two at a time.
Lifecycle policies were enforced so that the two most recent remain.

DLM Policy for EBS

1st

Activate the DLM policy for EBS and wait for a while.

The first snapshot was created approximately one hour after the policy was activated.

2nd

Wait another hour or so.

A second snapshot has been created.

3rd

Wait another hour or so.

The third snapshot has been created.
Three snapshots are left, although the policy is set to keep the two most recent.
It seems that immediately after the DLM policy is activated, more snapshots than the specified value may be saved.

After 20 to 30 minutes, check the snapshot again.