SAA_EN

Create S3 bucket with cross-region replication enabled using CFN

スポンサーリンク
Create S3 Bucket with Cross-Region Replication enabled using CloudFormation SAA_EN
スポンサーリンク
スポンサーリンク

Use CloudFormation to create S3 bucket with cross-region replication enabled

One of the topics of the AWS SAA is about designing a elastic architecture.

One of the features provided by S3 is replication.

Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets.

Replicating objects

The replication destination bucket can be specified not only in the same region, but also in a different region.
This is called cross-region replication (CRR).

This time, we will use CloudFormation to build a CRR environment.

Environment

Diagram of create S3 Bucket with Cross-Region Replication enabled using CloudFormation

Create S3 buckets in two regions.

Enable replication in one bucket.

The other bucket is the destination bucket for replication.

In this configuration, the two buckets will be located in the following regions

  • Source bucket: ap-northeast-1
  • Destination bucket: us-east-1

CloudFormation template files

Build the above configuration with CloudFormation.
The CloudFormation templates are placed at the following URL

awstut-saa/01/003 at main · awstut-an-r/awstut-saa
Contribute to awstut-an-r/awstut-saa development by creating an account on GitHub.

Explanation of key points of the template files

Destination bucket

Resources: ReplicationBucket: Type: AWS::S3::Bucket Properties: AccessControl: Private BucketName: !Ref Prefix VersioningConfiguration: Status: Enabled
Code language: YAML (yaml)

The key to creating a destination bucket is the versioning configuration.

Both source and destination buckets must have versioning enabled.

Requirements for replication

Enable versioning configuration according to the above.

Versioning is configured with the VersioningConfiguration property.
For more information, please refer to the following page

Source Bucket

Resources: Bucket: Type: AWS::S3::Bucket Properties: AccessControl: Private BucketName: !Ref Prefix ReplicationConfiguration: Role: !GetAtt S3BucketReplicationRole.Arn Rules: - Destination: Bucket: !Sub "arn:aws:s3:::${ReplicationBucketName}" Status: Enabled VersioningConfiguration: Status: Enabled
Code language: YAML (yaml)

Versioning must also be enabled in the source bucket.

Configure replication settings in the ReplicationConfiguration property.

Specify the destination bucket in the Rules property.

In the Role property, specify an IAM role with the privileges required for replication.

Resources: S3BucketReplicationRole: Type: AWS::IAM::Role DeletionPolicy: Delete Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - s3.amazonaws.com Policies: - PolicyName: !Sub "${Prefix}-S3BucketReplicationPolicy" PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:GetReplicationConfiguration - s3:ListBucket Resource: - !Sub "arn:aws:s3:::${Prefix}" - Effect: Allow Action: - s3:GetObjectVersionForReplication - s3:GetObjectVersionAcl - s3:GetObjectVersionTagging Resource: - !Sub "arn:aws:s3:::${Prefix}/*" - Effect: Allow Action: - s3:ReplicateObject - s3:ReplicateDelete - s3:ReplicateTags Resource: - !Sub "arn:aws:s3:::${ReplicationBucketName}/*"
Code language: YAML (yaml)

The policy was set up with reference to the following official AWS website

Setting up permissions - Amazon Simple Storage Service
Explanation of necessary permissions for replication.

Architecting

Use CloudFormation to build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

In this case, we will create stacks in two regions with the following commands

$ aws cloudformation create-stack \ --stack-name saa-01-003-2 \ --template-url https://[bucket-url]/saa-01-003-2.yaml \ --capabilities CAPABILITY_IAM \ --region us-east-1 $ aws cloudformation create-stack \ --stack-name saa-01-003-1 \ --template-url https://[bucket-url]/saa-01-003-1.yaml \ --capabilities CAPABILITY_IAM \ --region ap-northeast-1
Code language: Bash (bash)

After checking the resources in each stack, the following is the information on the main resources created in this case.

  • Source S3 bucket: saa-01-003-1
  • Destination S3 bucket: saa-01-003-2

Check each resource from the AWS Management Console.

Detail of S3 1.

Two buckets have indeed been created.
You can also see that both buckets were created in different regions.

Check the destination bucket.

Detail of S3 2.

Versioning is enabled.

Check the source S3 bucket.

Detail of S3 3.

This bucket is indeed versioning enabled as well.

Check the replication settings.

Detail of S3 4.

You can see that the bucket created in us-east-1 is set as the destination.

Checking Action

Now that everything is ready, upload the object to the source bucket.
Use the AWS CLI to upload objects.

$ echo 'hogehoge' > sample1.txt $ aws s3 cp sample1.txt s3://saa-01-003-1/ upload: ./sample1.txt to s3://saa-01-003-1/sample1.txt
Code language: Bash (bash)

Check the source bucket.

Detail of S3 5.

Sure enough, an object has been placed.

Check the details of this object.

Detail of S3 6.

Replication Status shows “COMPLETED”.
This means that replication has been successfully completed.

Check the destination bucket.

Detail of S3 7.

Object is placed.
It has the same name as the object placed in the source bucket.

Check the details of this object.

Detail of S3 8.

Replication Status shows “REPLICA”.
This means that this object is a replicated object.

In this way, replication can be performed between buckets in different regions.

(Reference) Versioning of objects

Since versioning was enabled for both buckets at the time of setup, we will check if the older version will be replicated.

First, update the object.

$ echo 'fugafuga' >> sample1.txt $ aws s3 cp sample1.txt s3://saa-01-003-1/ upload: ./sample1.txt to s3://saa-01-003-1/sample1.txt
Code language: Bash (bash)

Object has been updated.

Check the version information of this object in the destination bucket.

$ aws s3api list-object-versions \ --bucket saa-01-003-2 \ --prefix sample1.txt { "Versions": [ { "ETag": "\"304693ccbc28bbeac4689b24bda76e0e\"", "Size": 18, "StorageClass": "STANDARD", "Key": "sample1.txt", "VersionId": "X53EqnDUikg9zw827hBU0J5pr6BpmZS.", "IsLatest": true, "LastModified": "2022-12-28T07:01:41+00:00", "Owner": { "DisplayName": "[owner-name]", "ID": "[owner-id]" } }, { "ETag": "\"d9a3fdfc7ca17c47ed007bed5d2eb873\"", "Size": 9, "StorageClass": "STANDARD", "Key": "sample1.txt", "VersionId": ".fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c", "IsLatest": false, "LastModified": "2022-12-28T06:58:55+00:00", "Owner": { "DisplayName": "[owner-name]", "ID": "[owner-id]" } } ] }
Code language: Bash (bash)

You can see that it is indeed replicated for the version.
The second data is the older version of the object.

Download the older version of the object from the source bucket.

$ aws s3api get-object \ --bucket saa-01-003-2 \ --key sample1.txt \ --version-id .fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c \ sample1_replica.txt { "AcceptRanges": "bytes", "LastModified": "2022-12-28T06:58:55+00:00", "ContentLength": 9, "ETag": "\"d9a3fdfc7ca17c47ed007bed5d2eb873\"", "VersionId": ".fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c", "ContentType": "text/plain", "Metadata": {}, "ReplicationStatus": "REPLICA" } $ cat sample1_replica.txt hogehoge
Code language: Bash (bash)

The contents are pre-updated.
Indeed, we were able to retrieve the older version of the object from the source bucket.

Summary

We have seen how to build a CRR environment using CloudFormation.

タイトルとURLをコピーしました