AWS_EN

Backup to S3 every time you push to CodeCommit

Backup to S3 every time you push to CodeCommit

The following official AWS page shows how to backup CodeCommit contents to S3.

Automate event-driven backups from CodeCommit to Amazon S3 using CodeBuild and CloudWatch Events - AWS Prescriptive Guidance

The significance of this structure is explained as follows.

This pattern describes how to automatically back up a CodeCommit repository to an Amazon Simple Storage Service (Amazon S3) bucket after a change is made to the repository. If the CodeCommit repository is later deleted, this backup strategy provides you with a point-in-time recovery option.

Automate event-driven backups from CodeCommit to Amazon S3 using CodeBuild and CloudWatch Events

This page presents a configuration based on the above.

Environment

Diagram of backup to S3 every time you push to CodeCommit.

Use CodeCommit as the source code repository.

In CodeBuild, clone the source code in CodeCommit.
ZIP the retrieved file and save it to an S3 bucket.

The trigger for CodeBuild to run is EventBridge.
Set up an EventBridge rule to run CodeBuild every time the master branch of CodeCommit is updated.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL

awstut-fa/136 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of template files

S3 bucket

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref Prefix
      AccessControl: Private
Code language: YAML (yaml)

S3 bucket to be used as a backup destination for source code.

No special settings are required.

CodeCommit

Resources:
  CodeCommitRepository:
    Type: AWS::CodeCommit::Repository
    Properties: 
      RepositoryName: !Ref Prefix
Code language: YAML (yaml)

Source code repository.

No special settings are required here either.

CodeBuild

Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties: 
      Artifacts:
        Type: NO_ARTIFACTS
      Cache: 
        Type: NO_CACHE
      Environment: 
        ComputeType: !Ref ProjectEnvironmentComputeType
        EnvironmentVariables:
          - Name: BRANCH_NAME
            Type: PLAINTEXT
            Value: !Ref BranchName
          - Name: BUCKET_NAME
            Type: PLAINTEXT
            Value: !Ref BucketName
          - Name: REPO_REGION
            Type: PLAINTEXT
            Value: !Ref AWS::Region
          - Name: REPOSITORY_NAME
            Type: PLAINTEXT
            Value: !GetAtt CodeCommitRepository.Name
        Image: !Ref ProjectEnvironmentImage
        ImagePullCredentialsType: CODEBUILD
        Type: !Ref ProjectEnvironmentType
        PrivilegedMode: true
      LogsConfig: 
        CloudWatchLogs:
          Status: DISABLED
        S3Logs:
          Status: DISABLED
      Name: !Ref Prefix
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Source: 
        Type: NO_SOURCE
        BuildSpec: !Sub |
          version: 0.2
          
          phases:
            install:
              commands:
                - pip install git-remote-codecommit
            build:
              commands:
                - env
                - git clone -b $BRANCH_NAME codecommit::$REPO_REGION://$REPOSITORY_NAME
                - dt=$(date '+%d-%m-%Y-%H:%M:%S');
                - echo "$dt" 
                - zip -yr $dt-$REPOSITORY_NAME-backup.zip ./
                - aws s3 cp $dt-$REPOSITORY_NAME-backup.zip s3://$BUCKET_NAME/
      Visibility: PRIVATE
Code language: YAML (yaml)

For basic information on CodeBuild, please refer to the following pages.

We will cover three key points.

The first point is the Artifacts property.
This time there are no artifacts, so specify “NO_ARTIFACTS” for this property.

The second point is the EnvironmentVariables property.
You can set environment variables within CodeBuild.
In this case, we will specify the aforementioned S3 bucket name, CodeCommit repository name, etc.

The third point is the BuildSpec property.
This property allows you to directly describe the contents of BuildSpec.yml.
This is described here with reference to the following page.

Automate event-driven backups from CodeCommit to Amazon S3 using CodeBuild and CloudWatch Events - AWS Prescriptive Guidance

After installing git-remote-codecommit, clone the source code from CodeCommit.
Zip the acquired code and upload it to an S3 bucket.

Below are the IAM roles for CodeBuild.

Resources:
  CodeBuildRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codebuild.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: CodeBuildExecutionPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                Resource:
                  - !Sub "arn:aws:s3:::${BucketName}/*"
              - Effect: Allow
                Action:
                  - codecommit:GitPull
                Resource:
                  - !GetAtt CodeCommitRepository.Arn
Code language: YAML (yaml)

Authorization to retrieve source code from CodeCommit and upload objects to S3.

EventBridge

Resources:
  EventsRule:
    Type: AWS::Events::Rule
    Properties: 
      EventPattern:
        source:
          - aws.codecommit
        detail-type:
          - CodeCommit Repository State Change
        resources:
          - !GetAtt CodeCommitRepository.Arn
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType:
            - branch
          referenceName:
            - !Ref BranchName
      Name: !Ref Prefix
      Targets:
        - Arn: !GetAtt CodeBuildProject.Arn
          Id: !Sub "${Prefix}-CodeBuild"
          InputTransformer:
            InputPathsMap: 
              account: $.account
              referenceType: $.detail.referenceType
              referenceName: $.detail.referenceName
              region: $.region
              repositoryName: $.detail.repositoryName
            InputTemplate: |
              {"referenceType":"<referenceType>","region":"<region>","repositoryName":"<repositoryName>","account":"<account>","referenceName":"<referenceName>"}
          RoleArn: !GetAtt EventsRuleRole.Arn
Code language: YAML (yaml)

When a change is made to the master branch of CodeCommit, this triggers CodeBuild to run.

We also referred to the following pages for descriptions of rules and input transformers.

Automate event-driven backups from CodeCommit to Amazon S3 using CodeBuild and CloudWatch Events - AWS Prescriptive Guidance

The following are the IAM roles for EventBridge.

Resources:
  EventsRuleRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: PipelineExecutionPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - codebuild:StartBuild
                Resource:
                  - !GetAtt CodeBuildProject.Arn
Code language: YAML (yaml)

Authorization to start CodeBuild.

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • S3 bucket: fa-136
  • CodeCommit: fa-136
  • CodeBuild: fa-136
  • EventBridge: fa-136

Check the created resource from the AWS Management Console.

Check CodeCommit.

Detail of CodeCommit 1.

Indeed, a repository has been created.

Check EventBridge.

Detail of EventBridge 1.
Detail of EventBridge 2.

CodeCommit creation/update is set as the trigger condition.
You can also see that CodeBuild is specified as the target.

Check CodeBuild.

Detail of CodeBuild 1.

You can see that it is configured as specified in the CloudFormation template.

Operation Check

We are ready.

First, pull CodeCommit.

$ git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/fa-136
Cloning into 'fa-136'...
warning: You appear to have cloned an empty repository.
Code language: Bash (bash)

An empty repository has been pulled.

Add the test file to the repository.

touch ./fa-136/test.txt
Code language: Bash (bash)

Push the test file to CodeCommit.

(master) $ git add .

(master) $ git commit -m "first commit"
[master (root-commit) 5a16cf1] first commit
...
 1 file changed, 9 insertions(+)
 create mode 100644 test.txt

(master) $ git push
...
 * [new branch]      master -> master
Code language: Bash (bash)

I was able to push successfully.

Check CodeCommit again.

Detail of CodeCommit 2.

Sure enough, the test file is pushed.
The file push should trigger EventBridge to run and start CodeBuild.

Wait for a while and then check CodeBuild.

Detail of CodeBuild 2.

Indeed, the CodeBuild run completes successfully.

Check the S3 bucket.

Detail of S3 1.

Sure enough, a ZIP file has been uploaded.

Download and review the contents.

$ aws s3 cp s3://fa-136/25-05-2023-11:40:34-fa-136-backup.zip .
download: s3://fa-136/25-05-2023-11:40:34-fa-136-backup.zip to ./25-05-2023-11:40:34-fa-136-backup.zip

$ unzip -l 25-05-2023-11:40:34-fa-136-backup.zip
Archive:  25-05-2023-11:40:34-fa-136-backup.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  05-25-2023 11:40   fa-136/
        0  05-25-2023 11:40   fa-136/.git/
        0  05-25-2023 11:40   fa-136/.git/branches/
       73  05-25-2023 11:40   fa-136/.git/description
        0  05-25-2023 11:40   fa-136/.git/hooks/
      478  05-25-2023 11:40   fa-136/.git/hooks/applypatch-msg.sample
      896  05-25-2023 11:40   fa-136/.git/hooks/commit-msg.sample
     4726  05-25-2023 11:40   fa-136/.git/hooks/fsmonitor-watchman.sample
      189  05-25-2023 11:40   fa-136/.git/hooks/post-update.sample
      424  05-25-2023 11:40   fa-136/.git/hooks/pre-applypatch.sample
     1643  05-25-2023 11:40   fa-136/.git/hooks/pre-commit.sample
      416  05-25-2023 11:40   fa-136/.git/hooks/pre-merge-commit.sample
     1374  05-25-2023 11:40   fa-136/.git/hooks/pre-push.sample
     4898  05-25-2023 11:40   fa-136/.git/hooks/pre-rebase.sample
      544  05-25-2023 11:40   fa-136/.git/hooks/pre-receive.sample
     1492  05-25-2023 11:40   fa-136/.git/hooks/prepare-commit-msg.sample
     2783  05-25-2023 11:40   fa-136/.git/hooks/push-to-checkout.sample
     3650  05-25-2023 11:40   fa-136/.git/hooks/update.sample
        0  05-25-2023 11:40   fa-136/.git/info/
      240  05-25-2023 11:40   fa-136/.git/info/exclude
        0  05-25-2023 11:40   fa-136/.git/refs/
        0  05-25-2023 11:40   fa-136/.git/refs/heads/
       41  05-25-2023 11:40   fa-136/.git/refs/heads/master
        0  05-25-2023 11:40   fa-136/.git/refs/tags/
        0  05-25-2023 11:40   fa-136/.git/refs/remotes/
        0  05-25-2023 11:40   fa-136/.git/refs/remotes/origin/
       32  05-25-2023 11:40   fa-136/.git/refs/remotes/origin/HEAD
       23  05-25-2023 11:40   fa-136/.git/HEAD
      260  05-25-2023 11:40   fa-136/.git/config
        0  05-25-2023 11:40   fa-136/.git/objects/
        0  05-25-2023 11:40   fa-136/.git/objects/pack/
        0  05-25-2023 11:40   fa-136/.git/objects/info/
        0  05-25-2023 11:40   fa-136/.git/objects/e6/
       15  05-25-2023 11:40   fa-136/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391
        0  05-25-2023 11:40   fa-136/.git/objects/ac/
      159  05-25-2023 11:40   fa-136/.git/objects/ac/24cc10720d0e07a8d3fbd16767ef704663d5ee
        0  05-25-2023 11:40   fa-136/.git/objects/5e/
       53  05-25-2023 11:40   fa-136/.git/objects/5e/fb9bc29c482e023e40e0a2b3b7e49cec842034
      114  05-25-2023 11:40   fa-136/.git/packed-refs
        0  05-25-2023 11:40   fa-136/.git/logs/
        0  05-25-2023 11:40   fa-136/.git/logs/refs/
        0  05-25-2023 11:40   fa-136/.git/logs/refs/remotes/
        0  05-25-2023 11:40   fa-136/.git/logs/refs/remotes/origin/
      179  05-25-2023 11:40   fa-136/.git/logs/refs/remotes/origin/HEAD
        0  05-25-2023 11:40   fa-136/.git/logs/refs/heads/
      179  05-25-2023 11:40   fa-136/.git/logs/refs/heads/master
      179  05-25-2023 11:40   fa-136/.git/logs/HEAD
      137  05-25-2023 11:40   fa-136/.git/index
        0  05-25-2023 11:40   fa-136/test.txt
---------                     -------
    25197                     49 filesCode language: JavaScript (javascript)

You can see that the contents of the repository, including the test file, have indeed been backed up.

Summary

We have identified a way to back up to S3 each time we push to CodeCommit.

タイトルとURLをコピーしました