All patterns of server-side encryption of S3 buckets – SSE-S3/SSE-KMS/SSE-C
There are three patterns of server-side encryption for S3 buckets
Server-side encryption with Amazon S3 managed keys (SSE-S3)
Server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS)
Server-side encryption with customer-provided keys (SSE-C)
Protecting data using server-side encryption
We will use CloudFormation to experience the encryption of all the above patterns.
Environment
Create three different S3 buckets.
The breakdown is as follows
- Bucket with SSE-S3 configured
- Bucket configured for SSE-KMS
- Bucket configured for SSE-C
In addition, SSE-KMS will create the following two patterns
- Use an AWS Managed Key.
- Use a Customer Managed Key (CMK).
In SSE-C, a key is prepared on the user side.
In this case, a random string is automatically generated when a secret is created in the Secrets Manager, and this is used as the key.
Each bucket will be accessed using the AWS CLI.
The buckets for SSE-C will also be accessed through Lambda functions.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/125
Explanation of key points of template files
S3 bucket
SSE-S3
Resources:
Bucket1:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Sub "${Prefix}-sse-s3"
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: false
ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
Code language: YAML (yaml)SSE-S3 is enabled with the ServerSideEncryptionByDefault property.
It is enabled by specifying “AES256” for the SSEAlgorithm property.
SSE-KMS(AWS Managed Key)
Resources:
Bucket2:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Sub "${Prefix}-sse-kms-aws-managed"
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: false
ServerSideEncryptionByDefault:
KMSMasterKeyID: !Sub "arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/s3"
SSEAlgorithm: aws:kms
Code language: YAML (yaml)Enabling SSE-KMS is the same as before.
Specify “aws:kms” for the SSEAlgorithm property.
When using KMS for encryption, you can choose between AWS Managed Key and Customer Managed Key.
This bucket selects the former by specifying the alias of the managed key for S3 in the KMSMasterKeyID property.
SSE-KMS(Customer Managed Key)
Resources:
Bucket3:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Sub "${Prefix}-sse-kms-cmk"
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: false
ServerSideEncryptionByDefault:
KMSMasterKeyID: !Ref Key
SSEAlgorithm: aws:kms
Code language: YAML (yaml)In this bucket, specify the SSE-KMS customer-managed key.
Specify the following KMS key in the KMSMasterKeyID property.
Resources:
Key:
Type: AWS::KMS::Key
Properties:
Enabled: true
KeyPolicy:
Version: 2012-10-17
Id: !Sub "${Prefix}-cmk"
Statement:
- Effect: Allow
Principal:
AWS: "*"
Action:
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:GenerateDataKey*
- kms:DescribeKey
Resource: "*"
Condition:
StringEquals:
kms:CallerAccount: !Ref AWS::AccountId
kms:ViaService: s3.ap-northeast-1.amazonaws.com
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
Action: "*"
Resource: "*"
KeySpec: SYMMETRIC_DEFAULT
KeyUsage: ENCRYPT_DECRYPT
MultiRegion: false
Code language: YAML (yaml)For the key policy, we set up the key policy with reference to AWS Managed Keys for S3.
SSE-C
Resources:
Bucket4:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Sub "${Prefix}-sse-c"
Code language: YAML (yaml)Buckets for SSE-C. No special configuration is required.
However, refer to the following page to set the bucket policy.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html
Resources:
BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref Bucket4
PolicyDocument:
Version: 2012-10-17
Statement:
- Action:
- s3:PutObject
Effect: Deny
Principal: "*"
Resource:
- !Sub "arn:aws:s3:::${Bucket4}/*"
Condition:
"Null":
s3:x-amz-server-side-encryption-customer-algorithm: true
Code language: YAML (yaml)Here is how this policy works, as quoted below.
bucket policy denies upload object (s3:PutObject) permissions for all requests that don’t include the x-amz-server-side-encryption-customer-algorithm header requesting SSE-C.
Requiring and restricting SSE-C
The key to be provided by the customer, this time a random string automatically generated by the Secrets Manager.
Resources:
Secret:
Type: AWS::SecretsManager::Secret
Properties:
Description: Secret for S3 SSE-C.
GenerateSecretString:
ExcludeCharacters: ""
ExcludeLowercase: false
ExcludeNumbers: false
ExcludePunctuation: true
ExcludeUppercase: true
IncludeSpace: false
PasswordLength: !Ref PasswordLength
RequireEachIncludedType: true
KmsKeyId: alias/aws/secretsmanager
Name: !Ref Prefix
Code language: YAML (yaml)For more information on how to generate random passwords with Secrets Manager, please refer to the following page.
https://awstut.com/en/2023/04/15/use-secrets-manager-to-generate-random-password-en
We generated a random password under the following conditions, since it is a key for AES256.
- Password length is 32 characters
- Contains lowercase alphabets and numbers
As for SSE-C, objects are uploaded and downloaded not only from the AWS CLI but also from Lambda functions.
The code to be executed by the Lambda function in inline format.
For more information, please refer to the following page.
https://awstut.com/en/2022/02/02/3-parterns-to-create-lambda-with-cloudformation-s3-inline-container
Create two functions.
The first is for uploading objects.
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
BUCKET_NAME: !Ref Bucket4
OBJECT_KEY: !Ref TestObjectKey
SECRET_ARN: !Ref Secret
Code:
ZipFile: |
import base64
import boto3
import hashlib
import os
bucket_name = os.environ['BUCKET_NAME']
object_key = os.environ['OBJECT_KEY']
secret_arn = os.environ['SECRET_ARN']
secretmanager_client = boto3.client('secretsmanager')
s3_client = boto3.client('s3')
object_body = 'awstut!'
char_code = 'utf-8'
content_type = 'text/plain'
def lambda_handler(event, context):
response = secretmanager_client.get_secret_value(
SecretId=secret_arn
)
key = response['SecretString']
key_base64 = base64.b64encode(key.encode()).decode()
key_hash = hashlib.md5(key.encode()).digest()
key_hash_base64 = base64.b64encode(key_hash).decode()
response = s3_client.put_object(
Bucket=bucket_name,
Key=object_key,
Body=object_body.encode(char_code),
ContentEncoding=char_code,
ContentType=content_type,
SSECustomerAlgorithm='AES256',
SSECustomerKey=key_base64,
SSECustomerKeyMD5=key_hash_base64
)
return response
FunctionName: !Sub "${Prefix}-function1"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)The code is as follows.
- After creating a client object for Secrets Manager, access the secret and obtain the key for encryption.
- Calculate the Base64-encoded value of the key and the MD5-calculated hash value of the key, both encoded in Base64.
- After creating the client object for S3, execute the put_object method.
The point is the argument of the put_object method.
When encrypting an object in SSE-C, three request headers must be used.
x-amz-server-side-encryption-customer-algorithm
x-amz-server-side-encryption-customer-key
x-amz-server-side-encryption-customer-key-MD5
Specifying server-side encryption with customer-provided keys (SSE-C)
Three parameters (SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5) are specified as the corresponding arguments.
Note that SSECustomerKeyMD5, as quoted below, will be automatically populated if not specified.
Please note that this parameter is automatically populated if it is not provided. Including this parameter is not required
put_object
The second is for downloading objects.
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
BUCKET_NAME: !Ref Bucket4
OBJECT_KEY: !Ref TestObjectKey
SECRET_ARN: !Ref Secret
Code:
ZipFile: |
import base64
import boto3
import hashlib
import os
bucket_name = os.environ['BUCKET_NAME']
object_key = os.environ['OBJECT_KEY']
secret_arn = os.environ['SECRET_ARN']
secretmanager_client = boto3.client('secretsmanager')
s3_client = boto3.client('s3')
char_code = 'utf-8'
def lambda_handler(event, context):
response = secretmanager_client.get_secret_value(
SecretId=secret_arn
)
key = response['SecretString']
key_base64 = base64.b64encode(key.encode()).decode()
key_hash = hashlib.md5(key.encode()).digest()
key_hash_base64 = base64.b64encode(key_hash).decode()
response = s3_client.get_object(
Bucket=bucket_name,
Key=object_key,
SSECustomerAlgorithm='AES256',
SSECustomerKey=key_base64,
SSECustomerKeyMD5=key_hash_base64
)
body = response['Body'].read()
return body.decode(char_code)
FunctionName: !Sub "${Prefix}-function2"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)It is generally the same as the first function.
Execute the get_object method of the client object for S3 to download the uploaded object.
Three parameters for SSE-C are specified here as well.
Here are the IAM roles for the two functions
Resources:
FunctionRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: FunctionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource:
- !Ref Secret
- Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${Bucket4}"
- !Sub "arn:aws:s3:::${Bucket4}/*"
Code language: YAML (yaml)The contents of the authorization to save/retrieve objects to/from the target bucket.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.
https://awstut.com/en/2021/12/11/cloudformations-nested-stack
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- Bucket 1: fa-125-sse-s3
- Bucket 2: fa-125-sse-kms-aws-managed
- Bucket 3: fa-125-sse-kms-cmk
- Bucket 4: fa-125-sse-c
- KMS key for CMK: f395ccae-b5d2-426e-8ad9-db852042cbc4
- Secrets Manager secret: fa-125
- Function 1: fa-125-function1
- Function 2: fa-125-function2
Check various resources from the AWS Management Console.
Check the encryption settings for each bucket.
We can see that all four buckets have been successfully created.
For the SSE-C bucket (fa-125-sse-c), SSE-S3 is enabled even though it is not configured.
This is because SSE-S3 is now enabled by default, as quoted below.
Amazon S3 now applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for every bucket in Amazon S3. Starting January 5, 2023, all new object uploads to Amazon S3 are automatically encrypted at no additional cost and with no impact on performance.
Protecting data using server-side encryption
Check the KMS key.
Key for CMK.
The key policy is also set correctly.
Check Secrets Manager.
You can see that a random string is created as a customer-managed key.
Check the two Lambda functions.
This one is also created successfully.
Operation Check
Now that we are ready, we check the operation of each bucket.
Bucket 1 (SSE-S3)
Upload a test file from the AWS CLI.
$ touch test1.txt
$ aws s3 cp ./test1.txt s3://fa-125-sse-s3
upload: ./test1.txt to s3://fa-125-sse-s3/test1.txt
$ aws s3 ls s3://fa-125-sse-s3
2023-04-05 13:14:07 0 test1.txt
Code language: Bash (bash)The upload was successful.
Thus, no special operation is required when uploading objects to a bucket with SSE-S3 enabled.
Also check the behavior when downloading files.
$ aws s3 cp s3://fa-125-sse-s3/test1.txt ./test1-download.txt
download: s3://fa-125-sse-s3/test1.txt to ./test1-download.txt
Code language: Bash (bash)The download was successful.
Thus, no special operations are required when downloading objects from a bucket with SSE-S3 enabled.
Bucket 2 (SSE-KMS: AWS Managed Key)
Upload a test file from the AWS CLI.
$ touch test2.txt
$ aws s3 cp ./test2.txt s3://fa-125-sse-kms-aws-managed
upload: ./test2.txt to s3://fa-125-sse-kms-aws-managed/test2.txt
$ aws s3 ls s3://fa-125-sse-kms-aws-managed
2023-04-05 13:39:15 0 test2.txt
Code language: Bash (bash)The upload was successful.
Thus, no special operations are required to upload an object to a bucket with SSE-KMS enabled using AWS Managed Keys.
Also check the behavior when downloading files.
$ aws s3 cp s3://fa-125-sse-kms-aws-managed/test2.txt ./test2-download.txt
download: s3://fa-125-sse-kms-aws-managed/test2.txt to ./test2-download.txt
Code language: Bash (bash)The download was successful.
Thus, no special operations are required to download an object from an SSE-S3 enabled bucket using an AWS managed key.
Bucket 3 (SSE-KMS: CMK)
Upload a test file from the AWS CLI.
$ touch test3.txt
$ aws s3 cp ./test3.txt s3://fa-125-sse-kms-cmk
upload: ./test3.txt to s3://fa-125-sse-kms-cmk/test3.txt
$ aws s3 ls s3://fa-125-sse-kms-cmk
2023-04-06 11:39:01 0 test3.txt
Code language: Bash (bash)The upload was successful.
Thus, no special operation is required to upload an object to a bucket with SSE-KMS enabled using CMK.
Also check the behavior when downloading files.
$ aws s3 cp s3://fa-125-sse-kms-cmk/test3.txt ./test3-download.txt
download: s3://fa-125-sse-kms-cmk/test3.txt to ./test3-download.txt
Code language: Bash (bash)The download was successful.
Thus, no special operations are required to download objects from a CMK-enabled SSE-S3 bucket.
Bucket 4 (SSE-C)
Upload a test file from the AWS CLI.
Obtains a string corresponding to a customer key stored in Secrets Manager.
$ key=$(aws secretsmanager get-secret-value --secret-id fa-125 | jq -r .SecretString)
$ echo $key
hws7rqxtlcwd98e8nds94jf8al3yyi5t
Code language: Bash (bash)Get the Base64-encoded value of the customer key.
$ key_encoded=$(echo -n $key | base64)
$key_encoded
aHdzN3JxeHRsY3dkOThlOG5kczk0amY4YWwzeXlpNXQ=
Code language: Bash (bash)Similarly, the MD5 hash value of the customer key is also obtained, encoded in Base64.
$ hash=$(echo -n $key | openssl md5 -binary | base64)
$ echo $hash
aLkI4bOu2sXS3Guq7vU37A==
Code language: Bash (bash)We are ready.
First, check the behavior of uploading with the s3api command.
First, try uploading a test file without specifying a customer key or other information.
$ touch test4-1-1.txt
$ aws s3api put-object \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--body ./test4-1-1.txt
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Code language: Bash (bash)Error.
It means it was rejected by the bucket policy.
Upload the test file while specifying the customer key and hash value again.
$ aws s3api put-object \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--body ./test4-1-1.txt \
--sse-customer-algorithm AES256 \
--sse-customer-key $key_encoded \
--sse-customer-key-md5 $hash
{
"ETag": "\"f540442da544cc136f5ec839b1b7b1f4\"",
"SSECustomerAlgorithm": "AES256",
"SSECustomerKeyMD5": "aLkI4bOu2sXS3Guq7vU37A=="
}
$ aws s3 ls s3://fa-125-sse-c
2023-04-06 11:58:28 0 test4-1-1.txt
Code language: Bash (bash)Uploaded successfully.
Next, check the behavior of uploading with the s3 command.
First, try uploading a test file without specifying a customer key or other information.
$ aws s3 cp ./test4-1-2.txt s3://fa-125-sse-c
upload failed: ./test4-1-2.txt to s3://fa-125-sse-c/test4-1-2.txt
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Code language: Bash (bash)Error.
It means it was rejected by the bucket policy.
Upload the test file by specifying the customer key again.
In this case, it is not necessary to encode the customer key in Base64, nor is it necessary to specify a hash value.
$ aws s3 cp ./test4-1-2.txt s3://fa-125-sse-c \
--sse-c AES256 \
--sse-c-key $key
upload: ./test4-1-2.txt to s3://fa-125-sse-c/test4-1-2.txt
$ aws s3 ls s3://fa-125-sse-c
2023-04-06 11:58:28 0 test4-1-1.txt
2023-04-06 12:05:19 0 test4-1-2.txt
Code language: Bash (bash)Also check the behavior when downloading files.
First, let’s check the behavior of the s3api command when no customer key or other information is specified.
$ aws s3api get-object test4-1-1-download.txt \
--bucket fa-125-sse-c \
--key test4-1-1.txt
An error occurred (InvalidRequest) when calling the GetObject operation: The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.
Code language: Bash (bash)An error occurred because the customer key was not specified.
Try downloading again, specifying the customer key, etc.
$ aws s3api get-object test4-1-1-download.txt \
--bucket fa-125-sse-c \
--key test4-1-1.txt \
--sse-customer-algorithm AES256 \
--sse-customer-key $key_encoded \
--sse-customer-key-md5 $hash
{
"AcceptRanges": "bytes",
"LastModified": "2023-04-06T11:58:28+00:00",
"ContentLength": 0,
"ETag": "\"f540442da544cc136f5ec839b1b7b1f4\"",
"ContentType": "binary/octet-stream",
"Metadata": {},
"SSECustomerAlgorithm": "AES256",
"SSECustomerKeyMD5": "aLkI4bOu2sXS3Guq7vU37A=="
}
Code language: Bash (bash)I was able to download the file successfully.
Next, we will check the behavior of the s3 command when no customer key or other information is specified.
$ aws s3 cp s3://fa-125-sse-c/test4-1-2.txt test4-1-2-download.txt
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request
Code language: Bash (bash)An error has occurred.
Try downloading again, specifying the customer key, etc.
$ aws s3 cp s3://fa-125-sse-c/test4-1-2.txt ./test4-1-2-download.txt \
--sse-c AES256 \
--sse-c-key $key
download: s3://fa-125-sse-c/test4-1-2.txt to ./test4-1-2-download.txt
Code language: Bash (bash)I was able to download the file successfully.
Finally, in the Lambda function, upload/download files in SSE-C.
Execute the upload function (fa-125-function1).
The following is the log of the execution of this function.
The function has been successfully executed.
The test file should now be uploaded.
Check the S3 bucket.
Indeed, the file (test4-2) has been uploaded.
Then execute the function for download (fa-125-function2).
The following is the log of the execution of this function.
The function was successfully executed.
The contents of the test file (“awstut!”) were indeed confirmed.
In this way, the Lambda function can also be used to upload/download files in SSE-C.
Summary
All patterns of server-side encryption of S3 buckets (SSE-S3/SSE-KMS/SSE-C) are confirmed.