Upload/download files to/from S3 with Presigned URL
One of the features S3 offers is presigned URLs.
you can use a presigned URL to optionally share objects or allow your customers/users to upload objects to buckets without AWS security credentials or permissions.
Using presigned URLs
This time we will see how to upload/download files using presigned URL.
Environment
Create an S3 bucket.
Upload/download objects here.
Create two Lambda functions.
Both functions will do the work of creating presigned URLs.
One is for uploading the object and the other is for downloading.
The runtime environment for the functions is Python 3.8.
CloudFormation template files
The above configuration is built using CloudFormation.
The CloudFormation templates are located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/106
Explanation of key points of the template files
S3 bucket
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref Prefix
AccessControl: Private
Code language: YAML (yaml)No special configuration is required on the S3 bucket side to create presigned URLs.
In principle, setting the ACL setting to “Private” denies access to the bucket by others.
Lambda Function 1
Function to create presigned URLs for file upload.
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
BUCKET_NAME: !Ref S3BucketName
EXPIRE: !Ref S3PresignedUrlExpire
OBJECT_KEY: !Ref S3ObjectKey
REGION: !Ref AWS::Region
Code:
ZipFile: |
import boto3
import os
bucket_name = os.environ['BUCKET_NAME']
expire = os.environ['EXPIRE']
object_key = os.environ['OBJECT_KEY']
region = os.environ['REGION']
client_method = 'put_object'
http_method = 'PUT'
params = {
'Bucket': bucket_name,
'Key': object_key
}
s3_client = boto3.client('s3', region_name=region)
def lambda_handler(event, context):
url = s3_client.generate_presigned_url(
ClientMethod=client_method,
HttpMethod=http_method,
Params=params,
ExpiresIn=expire
)
return url
FunctionName: !Sub "${Prefix}-function1"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole1.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)This time, we will define the Lambda function in an inline format.
For more information on how to create a Lambda function using CloudFormation, please check the following page
To create presigned URLs, use the generate_presigned_url method.
Check the arguments.
When creating a URL for upload, specify “put_object” for the ClientMethod value, “PUT” for the HttpMethod value, and the S3 bucket or object name to upload to in Params.
Specify the time in seconds until the expiration date in ExpiresIn.
In this case, 3600 seconds (1 hour) is used.
To summarize the settings for the URL to be created, we will create a URL for uploading a file in the S3 bucket “fa-106” with the object name “sample.txt” for a limited period of one hour.
The following is the IAM role for this function.
Resources:
FunctionRole1:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: !Sub "${Prefix}-S3PutObjectPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${S3BucketName}/*"
Code language: YAML (yaml)The key point in creating presigned URLs is authorization.
AWS officially mentions the following
if you receive a presigned URL to upload an object, you can upload the object only if the creator of the presigned URL has the necessary permissions to upload that object.
Generating a presigned URL to upload an object
In other words, we need to give the Lambda function upload permissions in the form of IAM roles.
In this case, we will allow the “s3:PutObject” action for the target S3 bucket.
Lambda Function 2
Function to create presigned URLs for file download.
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
BUCKET_NAME: !Ref S3BucketName
EXPIRE: !Ref S3PresignedUrlExpire
OBJECT_KEY: !Ref S3ObjectKey
REGION: !Ref AWS::Region
Code:
ZipFile: |
import boto3
import os
bucket_name = os.environ['BUCKET_NAME']
expire = os.environ['EXPIRE']
object_key = os.environ['OBJECT_KEY']
region = os.environ['REGION']
client_method = 'get_object'
http_method = 'GET'
params = {
'Bucket': bucket_name,
'Key': object_key
}
s3_client = boto3.client('s3', region_name=region)
def lambda_handler(event, context):
url = s3_client.generate_presigned_url(
ClientMethod=client_method,
HttpMethod=http_method,
Params=params,
ExpiresIn=expire
)
return url
FunctionName: !Sub "${Prefix}-function2"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole2.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)The setup is almost the same as for the aforementioned function.
By setting “get_object” to ClientMethod and “GET” to HttpMethod, a URL for downloading is created.
To summarize the settings of the URL to be created, this function creates a URL to download the object name “sample.txt” located in the S3 bucket “fa-106” for one hour only.
The following is the IAM role for this function.
FunctionRole2:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: !Sub "${Prefix}-S3GetObjectPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- !Sub "arn:aws:s3:::${S3BucketName}/*"
Code language: YAML (yaml)Now we need to grant upload permissions to the Lambda function.
For the S3 bucket, configure it to allow the “s3:GetObject” action.
Architecting
Using CloudFormation, we will build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- S3 bucket: fa-106
- Lambda function 1 (creation of upload URL): fa-106-function1
- Lambda function 2 (creation of URL for download): fa-106-function2
Check each resource from the AWS Management Console.
Check the Lambda functions.
Two functions have indeed been created.
Checking Action
Now that everything is ready, upload/download files to the S3 bucket using the presigned URLs.
Preigned URL for upload
First, check the signed URL for upload.
Execute function 1.
After executing the function, a presigned URL is returned.
Attempt to upload a file to this URL using the curl command.
$ echo "hogehoge" > sample.txt
$ curl -X PUT --upload-file hogehoge.txt "https://fa-106.s3.amazonaws.com/sample.txt？AWSAccessKeyId=[AWSAccessKeyId]&Signature=[Signature]&x-amz-security-token=[x-amz-security-token]"Code language: PHP (php)Check the S3 bucket.
The file hogehoge.txt has been renamed sample.txt and placed as an object.
Using the presigned URL, the object can be uploaded to the S3 bucket.
Preigned URL for download
Next, check the presigned URL for download.
Execute function 2.
The function has been executed and the presigned URL has been returned.
Attempt to download the file to this URL using the curl command.
$ curl "https://fa-106.s3.amazonaws.com/sample.txt？AWSAccessKeyId=[AWSAccessKeyId]&Signature=[Signature]&x-amz-security-token=[x-amz-security-token]"
hogehogeCode language: JavaScript (javascript)The contents of the sample.txt file are output.
By using a presigned URL for downloading in this way, an object can be downloaded from an S3 bucket.
(Reference) Error message
Redirect
If an S3 bucket has just been created, the following message may appear when accessing a presigned URL.
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>TemporaryRedirect</Code>
<Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message>
<Endpoint>fa-106.s3-ap-northeast-1.amazonaws.com</Endpoint>
<Bucket>fa-106</Bucket>
</Error>
Code language: HTML, XML (xml)The AWS official explanation for this reason is as follows
After you create an Amazon S3 bucket, up to 24 hours can pass before the bucket name propagates across all AWS Regions. During this time, you might receive the 307 Temporary Redirect response for requests to Regional endpoints that aren’t in the same Region as your bucket.
Why am I getting an HTTP 307 Temporary Redirect response from Amazon S3?
Expired
If the deadline set when creating a presigned URL is exceeded, the following message will appear
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Request has expired</Message>
<Expires>2022-12-21T12:30:33Z</Expires>
<ServerTime>2022-12-22T12:55:32Z</ServerTime>
</Error>
Code language: HTML, XML (xml)Summary
We have reviewed how to upload/download files using presigned URLs.