Backup to S3 every time you push to CodeCommit
The following official AWS page shows how to backup CodeCommit contents to S3.
The significance of this structure is explained as follows.
This pattern describes how to automatically back up a CodeCommit repository to an Amazon Simple Storage Service (Amazon S3) bucket after a change is made to the repository. If the CodeCommit repository is later deleted, this backup strategy provides you with a point-in-time recovery option.
Automate event-driven backups from CodeCommit to Amazon S3 using CodeBuild and CloudWatch Events
This page presents a configuration based on the above.
Environment
Use CodeCommit as the source code repository.
In CodeBuild, clone the source code in CodeCommit.
ZIP the retrieved file and save it to an S3 bucket.
The trigger for CodeBuild to run is EventBridge.
Set up an EventBridge rule to run CodeBuild every time the master branch of CodeCommit is updated.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/136
Explanation of key points of template files
S3 bucket
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref Prefix
AccessControl: Private
Code language: YAML (yaml)S3 bucket to be used as a backup destination for source code.
No special settings are required.
CodeCommit
Resources:
CodeCommitRepository:
Type: AWS::CodeCommit::Repository
Properties:
RepositoryName: !Ref Prefix
Code language: YAML (yaml)Source code repository.
No special settings are required here either.
CodeBuild
Resources:
CodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: NO_ARTIFACTS
Cache:
Type: NO_CACHE
Environment:
ComputeType: !Ref ProjectEnvironmentComputeType
EnvironmentVariables:
- Name: BRANCH_NAME
Type: PLAINTEXT
Value: !Ref BranchName
- Name: BUCKET_NAME
Type: PLAINTEXT
Value: !Ref BucketName
- Name: REPO_REGION
Type: PLAINTEXT
Value: !Ref AWS::Region
- Name: REPOSITORY_NAME
Type: PLAINTEXT
Value: !GetAtt CodeCommitRepository.Name
Image: !Ref ProjectEnvironmentImage
ImagePullCredentialsType: CODEBUILD
Type: !Ref ProjectEnvironmentType
PrivilegedMode: true
LogsConfig:
CloudWatchLogs:
Status: DISABLED
S3Logs:
Status: DISABLED
Name: !Ref Prefix
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: NO_SOURCE
BuildSpec: !Sub |
version: 0.2
phases:
install:
commands:
- pip install git-remote-codecommit
build:
commands:
- env
- git clone -b $BRANCH_NAME codecommit::$REPO_REGION://$REPOSITORY_NAME
- dt=$(date '+%d-%m-%Y-%H:%M:%S');
- echo "$dt"
- zip -yr $dt-$REPOSITORY_NAME-backup.zip ./
- aws s3 cp $dt-$REPOSITORY_NAME-backup.zip s3://$BUCKET_NAME/
Visibility: PRIVATE
Code language: YAML (yaml)For basic information on CodeBuild, please refer to the following pages.
We will cover three key points.
The first point is the Artifacts property.
This time there are no artifacts, so specify “NO_ARTIFACTS” for this property.
The second point is the EnvironmentVariables property.
You can set environment variables within CodeBuild.
In this case, we will specify the aforementioned S3 bucket name, CodeCommit repository name, etc.
The third point is the BuildSpec property.
This property allows you to directly describe the contents of BuildSpec.yml.
This is described here with reference to the following page.
After installing git-remote-codecommit, clone the source code from CodeCommit.
Zip the acquired code and upload it to an S3 bucket.
Below are the IAM roles for CodeBuild.
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: CodeBuildExecutionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${BucketName}/*"
- Effect: Allow
Action:
- codecommit:GitPull
Resource:
- !GetAtt CodeCommitRepository.Arn
Code language: YAML (yaml)Authorization to retrieve source code from CodeCommit and upload objects to S3.
EventBridge
Resources:
EventsRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.codecommit
detail-type:
- CodeCommit Repository State Change
resources:
- !GetAtt CodeCommitRepository.Arn
detail:
event:
- referenceCreated
- referenceUpdated
referenceType:
- branch
referenceName:
- !Ref BranchName
Name: !Ref Prefix
Targets:
- Arn: !GetAtt CodeBuildProject.Arn
Id: !Sub "${Prefix}-CodeBuild"
InputTransformer:
InputPathsMap:
account: $.account
referenceType: $.detail.referenceType
referenceName: $.detail.referenceName
region: $.region
repositoryName: $.detail.repositoryName
InputTemplate: |
{"referenceType":"<referenceType>","region":"<region>","repositoryName":"<repositoryName>","account":"<account>","referenceName":"<referenceName>"}
RoleArn: !GetAtt EventsRuleRole.Arn
Code language: YAML (yaml)When a change is made to the master branch of CodeCommit, this triggers CodeBuild to run.
We also referred to the following pages for descriptions of rules and input transformers.
The following are the IAM roles for EventBridge.
Resources:
EventsRuleRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: PipelineExecutionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- codebuild:StartBuild
Resource:
- !GetAtt CodeBuildProject.Arn
Code language: YAML (yaml)Authorization to start CodeBuild.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- S3 bucket: fa-136
- CodeCommit: fa-136
- CodeBuild: fa-136
- EventBridge: fa-136
Check the created resource from the AWS Management Console.
Check CodeCommit.
Indeed, a repository has been created.
Check EventBridge.
CodeCommit creation/update is set as the trigger condition.
You can also see that CodeBuild is specified as the target.
Check CodeBuild.
You can see that it is configured as specified in the CloudFormation template.
Operation Check
We are ready.
First, pull CodeCommit.
$ git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/fa-136
Cloning into 'fa-136'...
warning: You appear to have cloned an empty repository.
Code language: Bash (bash)An empty repository has been pulled.
Add the test file to the repository.
touch ./fa-136/test.txt
Code language: Bash (bash)Push the test file to CodeCommit.
(master) $ git add .
(master) $ git commit -m "first commit"
[master (root-commit) 5a16cf1] first commit
...
1 file changed, 9 insertions(+)
create mode 100644 test.txt
(master) $ git push
...
* [new branch] master -> master
Code language: Bash (bash)I was able to push successfully.
Check CodeCommit again.
Sure enough, the test file is pushed.
The file push should trigger EventBridge to run and start CodeBuild.
Wait for a while and then check CodeBuild.
Indeed, the CodeBuild run completes successfully.
Check the S3 bucket.
Sure enough, a ZIP file has been uploaded.
Download and review the contents.
$ aws s3 cp s3://fa-136/25-05-2023-11:40:34-fa-136-backup.zip .
download: s3://fa-136/25-05-2023-11:40:34-fa-136-backup.zip to ./25-05-2023-11:40:34-fa-136-backup.zip
$ unzip -l 25-05-2023-11:40:34-fa-136-backup.zip
Archive: 25-05-2023-11:40:34-fa-136-backup.zip
Length Date Time Name
--------- ---------- ----- ----
0 05-25-2023 11:40 fa-136/
0 05-25-2023 11:40 fa-136/.git/
0 05-25-2023 11:40 fa-136/.git/branches/
73 05-25-2023 11:40 fa-136/.git/description
0 05-25-2023 11:40 fa-136/.git/hooks/
478 05-25-2023 11:40 fa-136/.git/hooks/applypatch-msg.sample
896 05-25-2023 11:40 fa-136/.git/hooks/commit-msg.sample
4726 05-25-2023 11:40 fa-136/.git/hooks/fsmonitor-watchman.sample
189 05-25-2023 11:40 fa-136/.git/hooks/post-update.sample
424 05-25-2023 11:40 fa-136/.git/hooks/pre-applypatch.sample
1643 05-25-2023 11:40 fa-136/.git/hooks/pre-commit.sample
416 05-25-2023 11:40 fa-136/.git/hooks/pre-merge-commit.sample
1374 05-25-2023 11:40 fa-136/.git/hooks/pre-push.sample
4898 05-25-2023 11:40 fa-136/.git/hooks/pre-rebase.sample
544 05-25-2023 11:40 fa-136/.git/hooks/pre-receive.sample
1492 05-25-2023 11:40 fa-136/.git/hooks/prepare-commit-msg.sample
2783 05-25-2023 11:40 fa-136/.git/hooks/push-to-checkout.sample
3650 05-25-2023 11:40 fa-136/.git/hooks/update.sample
0 05-25-2023 11:40 fa-136/.git/info/
240 05-25-2023 11:40 fa-136/.git/info/exclude
0 05-25-2023 11:40 fa-136/.git/refs/
0 05-25-2023 11:40 fa-136/.git/refs/heads/
41 05-25-2023 11:40 fa-136/.git/refs/heads/master
0 05-25-2023 11:40 fa-136/.git/refs/tags/
0 05-25-2023 11:40 fa-136/.git/refs/remotes/
0 05-25-2023 11:40 fa-136/.git/refs/remotes/origin/
32 05-25-2023 11:40 fa-136/.git/refs/remotes/origin/HEAD
23 05-25-2023 11:40 fa-136/.git/HEAD
260 05-25-2023 11:40 fa-136/.git/config
0 05-25-2023 11:40 fa-136/.git/objects/
0 05-25-2023 11:40 fa-136/.git/objects/pack/
0 05-25-2023 11:40 fa-136/.git/objects/info/
0 05-25-2023 11:40 fa-136/.git/objects/e6/
15 05-25-2023 11:40 fa-136/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391
0 05-25-2023 11:40 fa-136/.git/objects/ac/
159 05-25-2023 11:40 fa-136/.git/objects/ac/24cc10720d0e07a8d3fbd16767ef704663d5ee
0 05-25-2023 11:40 fa-136/.git/objects/5e/
53 05-25-2023 11:40 fa-136/.git/objects/5e/fb9bc29c482e023e40e0a2b3b7e49cec842034
114 05-25-2023 11:40 fa-136/.git/packed-refs
0 05-25-2023 11:40 fa-136/.git/logs/
0 05-25-2023 11:40 fa-136/.git/logs/refs/
0 05-25-2023 11:40 fa-136/.git/logs/refs/remotes/
0 05-25-2023 11:40 fa-136/.git/logs/refs/remotes/origin/
179 05-25-2023 11:40 fa-136/.git/logs/refs/remotes/origin/HEAD
0 05-25-2023 11:40 fa-136/.git/logs/refs/heads/
179 05-25-2023 11:40 fa-136/.git/logs/refs/heads/master
179 05-25-2023 11:40 fa-136/.git/logs/HEAD
137 05-25-2023 11:40 fa-136/.git/index
0 05-25-2023 11:40 fa-136/test.txt
--------- -------
25197 49 filesCode language: JavaScript (javascript)You can see that the contents of the repository, including the test file, have indeed been backed up.
Summary
We have identified a way to back up to S3 each time we push to CodeCommit.