AWS_EN

All patterns of server-side encryption of S3 buckets – SSE-S3/SSE-KMS/SSE-C

All patterns of server-side encryption of S3 buckets – SSE-S3/SSE-KMS/SSE-C

There are three patterns of server-side encryption for S3 buckets

Server-side encryption with Amazon S3 managed keys (SSE-S3)

Server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS)

Server-side encryption with customer-provided keys (SSE-C)

Protecting data using server-side encryption

We will use CloudFormation to experience the encryption of all the above patterns.

Environment

Diagram of all patterns of server-side encryption of S3 buckets - SSE-S3/SSE-KMS/SSE-C

Create three different S3 buckets.
The breakdown is as follows

  • Bucket with SSE-S3 configured
  • Bucket configured for SSE-KMS
  • Bucket configured for SSE-C

In addition, SSE-KMS will create the following two patterns

  • Use an AWS Managed Key.
  • Use a Customer Managed Key (CMK).

In SSE-C, a key is prepared on the user side.
In this case, a random string is automatically generated when a secret is created in the Secrets Manager, and this is used as the key.

Each bucket will be accessed using the AWS CLI.
The buckets for SSE-C will also be accessed through Lambda functions.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL

awstut-fa/125 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of template files

S3 bucket

SSE-S3

Resources:
  Bucket1:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub "${Prefix}-sse-s3"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - BucketKeyEnabled: false
            ServerSideEncryptionByDefault: 
              SSEAlgorithm: AES256
Code language: YAML (yaml)

SSE-S3 is enabled with the ServerSideEncryptionByDefault property.
It is enabled by specifying “AES256” for the SSEAlgorithm property.

SSE-KMS(AWS Managed Key)

Resources:
  Bucket2:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub "${Prefix}-sse-kms-aws-managed"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - BucketKeyEnabled: false
            ServerSideEncryptionByDefault: 
              KMSMasterKeyID: !Sub "arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3"
              SSEAlgorithm: aws:kms
Code language: YAML (yaml)

Enabling SSE-KMS is the same as before.
Specify “aws:kms” for the SSEAlgorithm property.
When using KMS for encryption, you can choose between AWS Managed Key and Customer Managed Key.
This bucket selects the former by specifying the alias of the managed key for S3 in the KMSMasterKeyID property.

SSE-KMS(Customer Managed Key)

Resources:
  Bucket3:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub "${Prefix}-sse-kms-cmk"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - BucketKeyEnabled: false
            ServerSideEncryptionByDefault: 
              KMSMasterKeyID: !Ref Key
              SSEAlgorithm: aws:kms
Code language: YAML (yaml)

In this bucket, specify the SSE-KMS customer-managed key.
Specify the following KMS key in the KMSMasterKeyID property.

Resources:
  Key:
    Type: AWS::KMS::Key
    Properties: 
      Enabled: true
      KeyPolicy:
        Version: 2012-10-17
        Id: !Sub "${Prefix}-cmk"
        Statement:
          - Effect: Allow
            Principal:
              AWS: "*"
            Action:
              - kms:Encrypt
              - kms:Decrypt
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:DescribeKey
            Resource: "*"
            Condition:
              StringEquals:
                kms:CallerAccount: !Ref AWS::AccountId
                kms:ViaService: s3.ap-northeast-1.amazonaws.com
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
            Action: "*"
            Resource: "*"
      KeySpec: SYMMETRIC_DEFAULT
      KeyUsage: ENCRYPT_DECRYPT
      MultiRegion: false
Code language: YAML (yaml)

For the key policy, we set up the key policy with reference to AWS Managed Keys for S3.

SSE-C

Resources:
  Bucket4:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Sub "${Prefix}-sse-c"
Code language: YAML (yaml)

Buckets for SSE-C. No special configuration is required.
However, refer to the following page to set the bucket policy.

Using server-side encryption with customer-provided keys (SSE-C) - Amazon Simple Storage Service
To use your own custom keys to encrypt the objects that you store on Amazon S3, use server-side encryption with customer-provided encryption keys (SSE-C).
Resources:
  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref Bucket4
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
              - s3:PutObject
            Effect: Deny
            Principal: "*"
            Resource:
              - !Sub "arn:aws:s3:::${Bucket4}/*"
            Condition:
              "Null":
                s3:x-amz-server-side-encryption-customer-algorithm: true
Code language: YAML (yaml)

Here is how this policy works, as quoted below.

bucket policy denies upload object (s3:PutObject) permissions for all requests that don’t include the x-amz-server-side-encryption-customer-algorithm header requesting SSE-C.

Requiring and restricting SSE-C

The key to be provided by the customer, this time a random string automatically generated by the Secrets Manager.

Resources:
  Secret:
    Type: AWS::SecretsManager::Secret
    Properties: 
      Description: Secret for S3 SSE-C.
      GenerateSecretString: 
        ExcludeCharacters: ""
        ExcludeLowercase: false
        ExcludeNumbers: false
        ExcludePunctuation: true
        ExcludeUppercase: true
        IncludeSpace: false
        PasswordLength: !Ref PasswordLength
        RequireEachIncludedType: true
      KmsKeyId: alias/aws/secretsmanager
      Name: !Ref Prefix
Code language: YAML (yaml)

For more information on how to generate random passwords with Secrets Manager, please refer to the following page.

We generated a random password under the following conditions, since it is a key for AES256.

  • Password length is 32 characters
  • Contains lowercase alphabets and numbers

As for SSE-C, objects are uploaded and downloaded not only from the AWS CLI but also from Lambda functions.

The code to be executed by the Lambda function in inline format.
For more information, please refer to the following page.

Create two functions.
The first is for uploading objects.

Resources:
  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          BUCKET_NAME: !Ref Bucket4
          OBJECT_KEY: !Ref TestObjectKey
          SECRET_ARN: !Ref Secret
      Code:
        ZipFile: |
          import base64
          import boto3
          import hashlib
          import os
          
          bucket_name = os.environ['BUCKET_NAME']
          object_key = os.environ['OBJECT_KEY']
          secret_arn = os.environ['SECRET_ARN']
          
          secretmanager_client = boto3.client('secretsmanager')
          s3_client = boto3.client('s3')
          
          object_body = 'awstut!'
          char_code = 'utf-8'
          content_type = 'text/plain'
          
          def lambda_handler(event, context):
            response = secretmanager_client.get_secret_value(
              SecretId=secret_arn
            )
            key = response['SecretString']
            key_base64 = base64.b64encode(key.encode()).decode()
            
            key_hash = hashlib.md5(key.encode()).digest()
            key_hash_base64 = base64.b64encode(key_hash).decode()
            
            response = s3_client.put_object(
              Bucket=bucket_name,
              Key=object_key,
              Body=object_body.encode(char_code),
              ContentEncoding=char_code,
              ContentType=content_type,
              SSECustomerAlgorithm='AES256',
              SSECustomerKey=key_base64,
              SSECustomerKeyMD5=key_hash_base64
            )
            return response
      FunctionName: !Sub "${Prefix}-function1"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)

The code is as follows.

  • After creating a client object for Secrets Manager, access the secret and obtain the key for encryption.
  • Calculate the Base64-encoded value of the key and the MD5-calculated hash value of the key, both encoded in Base64.
  • After creating the client object for S3, execute the put_object method.

The point is the argument of the put_object method.
When encrypting an object in SSE-C, three request headers must be used.

x-amz-server-side​-encryption​-customer-algorithm

x-amz-server-side​-encryption​-customer-key

x-amz-server-side​-encryption​-customer-key-MD5

Specifying server-side encryption with customer-provided keys (SSE-C)

Three parameters (SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5) are specified as the corresponding arguments.

Note that SSECustomerKeyMD5, as quoted below, will be automatically populated if not specified.

Please note that this parameter is automatically populated if it is not provided. Including this parameter is not required

put_object

The second is for downloading objects.

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          BUCKET_NAME: !Ref Bucket4
          OBJECT_KEY: !Ref TestObjectKey
          SECRET_ARN: !Ref Secret
      Code:
        ZipFile: |
          import base64
          import boto3
          import hashlib
          import os
          
          bucket_name = os.environ['BUCKET_NAME']
          object_key = os.environ['OBJECT_KEY']
          secret_arn = os.environ['SECRET_ARN']
          
          secretmanager_client = boto3.client('secretsmanager')
          s3_client = boto3.client('s3')
          
          char_code = 'utf-8'
          
          def lambda_handler(event, context):
            response = secretmanager_client.get_secret_value(
              SecretId=secret_arn
            )
            key = response['SecretString']
            key_base64 = base64.b64encode(key.encode()).decode()
            
            key_hash = hashlib.md5(key.encode()).digest()
            key_hash_base64 = base64.b64encode(key_hash).decode()
            
            response = s3_client.get_object(
              Bucket=bucket_name,
              Key=object_key,
              SSECustomerAlgorithm='AES256',
              SSECustomerKey=key_base64,
              SSECustomerKeyMD5=key_hash_base64
            )
            body = response['Body'].read()
            return body.decode(char_code)
      FunctionName: !Sub "${Prefix}-function2"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)

It is generally the same as the first function.
Execute the get_object method of the client object for S3 to download the uploaded object.
Three parameters for SSE-C are specified here as well.

Here are the IAM roles for the two functions

Resources:
  FunctionRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: FunctionPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                Resource:
                  - !Ref Secret
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                Resource:
                  - !Sub "arn:aws:s3:::${Bucket4}"
                  - !Sub "arn:aws:s3:::${Bucket4}/*"
Code language: YAML (yaml)

The contents of the authorization to save/retrieve objects to/from the target bucket.

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • Bucket 1: fa-125-sse-s3
  • Bucket 2: fa-125-sse-kms-aws-managed
  • Bucket 3: fa-125-sse-kms-cmk
  • Bucket 4: fa-125-sse-c
  • KMS key for CMK: f395ccae-b5d2-426e-8ad9-db852042cbc4
  • Secrets Manager secret: fa-125
  • Function 1: fa-125-function1
  • Function 2: fa-125-function2

Check various resources from the AWS Management Console.

Check the encryption settings for each bucket.

Detail of S3 1.
Detail of S3 2.
Detail of S3 3.
Detail of S3 4.

We can see that all four buckets have been successfully created.

For the SSE-C bucket (fa-125-sse-c), SSE-S3 is enabled even though it is not configured.
This is because SSE-S3 is now enabled by default, as quoted below.

Amazon S3 now applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for every bucket in Amazon S3. Starting January 5, 2023, all new object uploads to Amazon S3 are automatically encrypted at no additional cost and with no impact on performance.

Protecting data using server-side encryption

Check the KMS key.

Detail of KMS 1.

Key for CMK.
The key policy is also set correctly.

Check Secrets Manager.

Detail of Secrets Manager 1.

You can see that a random string is created as a customer-managed key.

Check the two Lambda functions.

Detail of Lambda 1.
Detail of Lambda 2.

This one is also created successfully.

Operation Check

Now that we are ready, we check the operation of each bucket.

Bucket 1 (SSE-S3)

Upload a test file from the AWS CLI.

$ touch test1.txt
$ aws s3 cp ./test1.txt s3://fa-125-sse-s3
upload: ./test1.txt to s3://fa-125-sse-s3/test1.txt

$ aws s3 ls s3://fa-125-sse-s3
2023-04-05 13:14:07          0 test1.txt
Code language: Bash (bash)

The upload was successful.
Thus, no special operation is required when uploading objects to a bucket with SSE-S3 enabled.

Also check the behavior when downloading files.

$ aws s3 cp s3://fa-125-sse-s3/test1.txt ./test1-download.txt
download: s3://fa-125-sse-s3/test1.txt to ./test1-download.txt
Code language: Bash (bash)

The download was successful.
Thus, no special operations are required when downloading objects from a bucket with SSE-S3 enabled.

Bucket 2 (SSE-KMS: AWS Managed Key)

Upload a test file from the AWS CLI.

$ touch test2.txt
$ aws s3 cp ./test2.txt s3://fa-125-sse-kms-aws-managed
upload: ./test2.txt to s3://fa-125-sse-kms-aws-managed/test2.txt

$ aws s3 ls s3://fa-125-sse-kms-aws-managed
2023-04-05 13:39:15          0 test2.txt
Code language: Bash (bash)

The upload was successful.
Thus, no special operations are required to upload an object to a bucket with SSE-KMS enabled using AWS Managed Keys.

Also check the behavior when downloading files.

$ aws s3 cp s3://fa-125-sse-kms-aws-managed/test2.txt ./test2-download.txt
download: s3://fa-125-sse-kms-aws-managed/test2.txt to ./test2-download.txt
Code language: Bash (bash)

The download was successful.
Thus, no special operations are required to download an object from an SSE-S3 enabled bucket using an AWS managed key.

Bucket 3 (SSE-KMS: CMK)

Upload a test file from the AWS CLI.

$ touch test3.txt
$ aws s3 cp ./test3.txt s3://fa-125-sse-kms-cmk
upload: ./test3.txt to s3://fa-125-sse-kms-cmk/test3.txt

$ aws s3 ls s3://fa-125-sse-kms-cmk
2023-04-06 11:39:01          0 test3.txt
Code language: Bash (bash)

The upload was successful.
Thus, no special operation is required to upload an object to a bucket with SSE-KMS enabled using CMK.

Also check the behavior when downloading files.

$ aws s3 cp s3://fa-125-sse-kms-cmk/test3.txt ./test3-download.txt
download: s3://fa-125-sse-kms-cmk/test3.txt to ./test3-download.txt
Code language: Bash (bash)

The download was successful.
Thus, no special operations are required to download objects from a CMK-enabled SSE-S3 bucket.

Bucket 4 (SSE-C)

Upload a test file from the AWS CLI.

Obtains a string corresponding to a customer key stored in Secrets Manager.

$ key=$(aws secretsmanager get-secret-value --secret-id fa-125 | jq -r .SecretString)
$ echo $key
hws7rqxtlcwd98e8nds94jf8al3yyi5t
Code language: Bash (bash)

Get the Base64-encoded value of the customer key.

$ key_encoded=$(echo -n $key | base64)      
$key_encoded
aHdzN3JxeHRsY3dkOThlOG5kczk0amY4YWwzeXlpNXQ=
Code language: Bash (bash)

Similarly, the MD5 hash value of the customer key is also obtained, encoded in Base64.

$ hash=$(echo -n $key | openssl md5 -binary | base64)
$ echo $hash
aLkI4bOu2sXS3Guq7vU37A==
Code language: Bash (bash)

We are ready.

First, check the behavior of uploading with the s3api command.

First, try uploading a test file without specifying a customer key or other information.

$ touch test4-1-1.txt
$ aws s3api put-object \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--body ./test4-1-1.txt
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Code language: Bash (bash)

Error.
It means it was rejected by the bucket policy.

Upload the test file while specifying the customer key and hash value again.

$ aws s3api put-object \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--body ./test4-1-1.txt \
--sse-customer-algorithm AES256 \
--sse-customer-key $key_encoded \
--sse-customer-key-md5 $hash
{
    "ETag": "\"f540442da544cc136f5ec839b1b7b1f4\"",
    "SSECustomerAlgorithm": "AES256",
    "SSECustomerKeyMD5": "aLkI4bOu2sXS3Guq7vU37A=="
}

$ aws s3 ls s3://fa-125-sse-c
2023-04-06 11:58:28          0 test4-1-1.txt
Code language: Bash (bash)

Uploaded successfully.

Next, check the behavior of uploading with the s3 command.

First, try uploading a test file without specifying a customer key or other information.

$ aws s3 cp ./test4-1-2.txt s3://fa-125-sse-c
upload failed: ./test4-1-2.txt to s3://fa-125-sse-c/test4-1-2.txt
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Code language: Bash (bash)

Error.
It means it was rejected by the bucket policy.

Upload the test file by specifying the customer key again.
In this case, it is not necessary to encode the customer key in Base64, nor is it necessary to specify a hash value.

$ aws s3 cp ./test4-1-2.txt s3://fa-125-sse-c \
--sse-c AES256 \
--sse-c-key $key
upload: ./test4-1-2.txt to s3://fa-125-sse-c/test4-1-2.txt

$ aws s3 ls s3://fa-125-sse-c
2023-04-06 11:58:28          0 test4-1-1.txt
2023-04-06 12:05:19          0 test4-1-2.txt
Code language: Bash (bash)

Also check the behavior when downloading files.

First, let’s check the behavior of the s3api command when no customer key or other information is specified.

$  aws s3api get-object test4-1-1-download.txt \
--bucket fa-125-sse-c \
--key test4-1-1.txt
An error occurred (InvalidRequest) when calling the GetObject operation: The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.
Code language: Bash (bash)

An error occurred because the customer key was not specified.

Try downloading again, specifying the customer key, etc.

$  aws s3api get-object test4-1-1-download.txt \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--sse-customer-algorithm AES256 \
--sse-customer-key $key_encoded \
--sse-customer-key-md5 $hash
{
    "AcceptRanges": "bytes",
    "LastModified": "2023-04-06T11:58:28+00:00",
    "ContentLength": 0,
    "ETag": "\"f540442da544cc136f5ec839b1b7b1f4\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "SSECustomerAlgorithm": "AES256",
    "SSECustomerKeyMD5": "aLkI4bOu2sXS3Guq7vU37A=="
}
Code language: Bash (bash)

I was able to download the file successfully.

Next, we will check the behavior of the s3 command when no customer key or other information is specified.

$ aws s3 cp s3://fa-125-sse-c/test4-1-2.txt test4-1-2-download.txt
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request
Code language: Bash (bash)

An error has occurred.

Try downloading again, specifying the customer key, etc.

$ aws s3 cp s3://fa-125-sse-c/test4-1-2.txt ./test4-1-2-download.txt \
--sse-c AES256 \
--sse-c-key $key
download: s3://fa-125-sse-c/test4-1-2.txt to ./test4-1-2-download.txt
Code language: Bash (bash)

I was able to download the file successfully.

Finally, in the Lambda function, upload/download files in SSE-C.

Execute the upload function (fa-125-function1).
The following is the log of the execution of this function.

Detail of Lambda 3.

The function has been successfully executed.
The test file should now be uploaded.

Check the S3 bucket.

Detail of Lambda 4.

Indeed, the file (test4-2) has been uploaded.

Then execute the function for download (fa-125-function2).
The following is the log of the execution of this function.

Detail of Lambda 5.

The function was successfully executed.
The contents of the test file (“awstut!”) were indeed confirmed.

In this way, the Lambda function can also be used to upload/download files in SSE-C.

Summary

All patterns of server-side encryption of S3 buckets (SSE-S3/SSE-KMS/SSE-C) are confirmed.

タイトルとURLをコピーしました