Use CloudFormation to create S3 bucket with cross-region replication enabled
One of the topics of the AWS SAA is about designing a elastic architecture.
One of the features provided by S3 is replication.
Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets.
Replicating objects
The replication destination bucket can be specified not only in the same region, but also in a different region.
This is called cross-region replication (CRR).
This time, we will use CloudFormation to build a CRR environment.
Environment
Create S3 buckets in two regions.
Enable replication in one bucket.
The other bucket is the destination bucket for replication.
In this configuration, the two buckets will be located in the following regions
- Source bucket: ap-northeast-1
- Destination bucket: us-east-1
CloudFormation template files
Build the above configuration with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-saa/tree/main/01/003
Explanation of key points of the template files
Destination bucket
Resources:
ReplicationBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Ref Prefix
VersioningConfiguration:
Status: Enabled
Code language: YAML (yaml)The key to creating a destination bucket is the versioning configuration.
Both source and destination buckets must have versioning enabled.
Requirements for replication
Enable versioning configuration according to the above.
Versioning is configured with the VersioningConfiguration property.
For more information, please refer to the following page
Source Bucket
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Ref Prefix
ReplicationConfiguration:
Role: !GetAtt S3BucketReplicationRole.Arn
Rules:
- Destination:
Bucket: !Sub "arn:aws:s3:::${ReplicationBucketName}"
Status: Enabled
VersioningConfiguration:
Status: Enabled
Code language: YAML (yaml)Versioning must also be enabled in the source bucket.
Configure replication settings in the ReplicationConfiguration property.
Specify the destination bucket in the Rules property.
In the Role property, specify an IAM role with the privileges required for replication.
Resources:
S3BucketReplicationRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- s3.amazonaws.com
Policies:
- PolicyName: !Sub "${Prefix}-S3BucketReplicationPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:GetReplicationConfiguration
- s3:ListBucket
Resource:
- !Sub "arn:aws:s3:::${Prefix}"
- Effect: Allow
Action:
- s3:GetObjectVersionForReplication
- s3:GetObjectVersionAcl
- s3:GetObjectVersionTagging
Resource:
- !Sub "arn:aws:s3:::${Prefix}/*"
- Effect: Allow
Action:
- s3:ReplicateObject
- s3:ReplicateDelete
- s3:ReplicateTags
Resource:
- !Sub "arn:aws:s3:::${ReplicationBucketName}/*"
Code language: YAML (yaml)The policy was set up with reference to the following official AWS website
https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page
In this case, we will create stacks in two regions with the following commands
$ aws cloudformation create-stack \
--stack-name saa-01-003-2 \
--template-url https://[bucket-url]/saa-01-003-2.yaml \
--capabilities CAPABILITY_IAM \
--region us-east-1
$ aws cloudformation create-stack \
--stack-name saa-01-003-1 \
--template-url https://[bucket-url]/saa-01-003-1.yaml \
--capabilities CAPABILITY_IAM \
--region ap-northeast-1
Code language: Bash (bash)After checking the resources in each stack, the following is the information on the main resources created in this case.
- Source S3 bucket: saa-01-003-1
- Destination S3 bucket: saa-01-003-2
Check each resource from the AWS Management Console.
Two buckets have indeed been created.
You can also see that both buckets were created in different regions.
Check the destination bucket.
Versioning is enabled.
Check the source S3 bucket.
This bucket is indeed versioning enabled as well.
Check the replication settings.
You can see that the bucket created in us-east-1 is set as the destination.
Checking Action
Now that everything is ready, upload the object to the source bucket.
Use the AWS CLI to upload objects.
$ echo 'hogehoge' > sample1.txt
$ aws s3 cp sample1.txt s3://saa-01-003-1/
upload: ./sample1.txt to s3://saa-01-003-1/sample1.txt
Code language: Bash (bash)Check the source bucket.
Sure enough, an object has been placed.
Check the details of this object.
Replication Status shows “COMPLETED”.
This means that replication has been successfully completed.
Check the destination bucket.
Object is placed.
It has the same name as the object placed in the source bucket.
Check the details of this object.
Replication Status shows “REPLICA”.
This means that this object is a replicated object.
In this way, replication can be performed between buckets in different regions.
(Reference) Versioning of objects
Since versioning was enabled for both buckets at the time of setup, we will check if the older version will be replicated.
First, update the object.
$ echo 'fugafuga' >> sample1.txt
$ aws s3 cp sample1.txt s3://saa-01-003-1/
upload: ./sample1.txt to s3://saa-01-003-1/sample1.txt
Code language: Bash (bash)Object has been updated.
Check the version information of this object in the destination bucket.
$ aws s3api list-object-versions \
--bucket saa-01-003-2 \
--prefix sample1.txt
{
"Versions": [
{
"ETag": "\"304693ccbc28bbeac4689b24bda76e0e\"",
"Size": 18,
"StorageClass": "STANDARD",
"Key": "sample1.txt",
"VersionId": "X53EqnDUikg9zw827hBU0J5pr6BpmZS.",
"IsLatest": true,
"LastModified": "2022-12-28T07:01:41+00:00",
"Owner": {
"DisplayName": "[owner-name]",
"ID": "[owner-id]"
}
},
{
"ETag": "\"d9a3fdfc7ca17c47ed007bed5d2eb873\"",
"Size": 9,
"StorageClass": "STANDARD",
"Key": "sample1.txt",
"VersionId": ".fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c",
"IsLatest": false,
"LastModified": "2022-12-28T06:58:55+00:00",
"Owner": {
"DisplayName": "[owner-name]",
"ID": "[owner-id]"
}
}
]
}
Code language: Bash (bash)You can see that it is indeed replicated for the version.
The second data is the older version of the object.
Download the older version of the object from the source bucket.
$ aws s3api get-object \
--bucket saa-01-003-2 \
--key sample1.txt \
--version-id .fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c \
sample1_replica.txt
{
"AcceptRanges": "bytes",
"LastModified": "2022-12-28T06:58:55+00:00",
"ContentLength": 9,
"ETag": "\"d9a3fdfc7ca17c47ed007bed5d2eb873\"",
"VersionId": ".fp.Pb7WSzlbBjsXStbPbIE63uGDF9_c",
"ContentType": "text/plain",
"Metadata": {},
"ReplicationStatus": "REPLICA"
}
$ cat sample1_replica.txt
hogehoge
Code language: Bash (bash)The contents are pre-updated.
Indeed, we were able to retrieve the older version of the object from the source bucket.
Summary
We have seen how to build a CRR environment using CloudFormation.