AWS_EN

Upload/download files to/from S3 with Presigned URL

スポンサーリンク
Upload/Download files to/from S3 with Presigned URL. AWS_EN
スポンサーリンク
スポンサーリンク

Upload/download files to/from S3 with Presigned URL

One of the features S3 offers is presigned URLs.

you can use a presigned URL to optionally share objects or allow your customers/users to upload objects to buckets without AWS security credentials or permissions.

Using presigned URLs

This time we will see how to upload/download files using presigned URL.

Environment

Diagram of Upload/Download files to/from S3 with Presigned URL.

Create an S3 bucket.
Upload/download objects here.

Create two Lambda functions.
Both functions will do the work of creating presigned URLs.
One is for uploading the object and the other is for downloading.
The runtime environment for the functions is Python 3.8.

CloudFormation template files

The above configuration is built using CloudFormation.
The CloudFormation templates are located at the following URL

https://github.com/awstut-an-r/awstut-fa/tree/main/106

Explanation of key points of the template files

S3 bucket

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref Prefix
      AccessControl: Private
Code language: YAML (yaml)

No special configuration is required on the S3 bucket side to create presigned URLs.

In principle, setting the ACL setting to “Private” denies access to the bucket by others.

Lambda Function 1

Function to create presigned URLs for file upload.

Resources:
  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          BUCKET_NAME: !Ref S3BucketName
          EXPIRE: !Ref S3PresignedUrlExpire
          OBJECT_KEY: !Ref S3ObjectKey
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          import boto3
          import os
          
          bucket_name = os.environ['BUCKET_NAME']
          expire = os.environ['EXPIRE']
          object_key = os.environ['OBJECT_KEY']
          region = os.environ['REGION']
          
          client_method = 'put_object'
          http_method = 'PUT'
          params = {
            'Bucket': bucket_name,
            'Key': object_key
          }
          
          s3_client = boto3.client('s3', region_name=region)
          
          def lambda_handler(event, context):
            url = s3_client.generate_presigned_url(
              ClientMethod=client_method,
              HttpMethod=http_method,
              Params=params,
              ExpiresIn=expire
            )
            return url
      FunctionName: !Sub "${Prefix}-function1"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole1.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

This time, we will define the Lambda function in an inline format.

For more information on how to create a Lambda function using CloudFormation, please check the following page

To create presigned URLs, use the generate_presigned_url method.

Check the arguments.
When creating a URL for upload, specify “put_object” for the ClientMethod value, “PUT” for the HttpMethod value, and the S3 bucket or object name to upload to in Params.
Specify the time in seconds until the expiration date in ExpiresIn.
In this case, 3600 seconds (1 hour) is used.

To summarize the settings for the URL to be created, we will create a URL for uploading a file in the S3 bucket “fa-106” with the object name “sample.txt” for a limited period of one hour.

The following is the IAM role for this function.

Resources:
  FunctionRole1:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: !Sub "${Prefix}-S3PutObjectPolicy"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                Resource:
                  - !Sub "arn:aws:s3:::${S3BucketName}/*"
Code language: YAML (yaml)

The key point in creating presigned URLs is authorization.
AWS officially mentions the following

if you receive a presigned URL to upload an object, you can upload the object only if the creator of the presigned URL has the necessary permissions to upload that object.

Generating a presigned URL to upload an object

In other words, we need to give the Lambda function upload permissions in the form of IAM roles.

In this case, we will allow the “s3:PutObject” action for the target S3 bucket.

Lambda Function 2

Function to create presigned URLs for file download.

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          BUCKET_NAME: !Ref S3BucketName
          EXPIRE: !Ref S3PresignedUrlExpire
          OBJECT_KEY: !Ref S3ObjectKey
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          import boto3
          import os
          
          bucket_name = os.environ['BUCKET_NAME']
          expire = os.environ['EXPIRE']
          object_key = os.environ['OBJECT_KEY']
          region = os.environ['REGION']
          
          client_method = 'get_object'
          http_method = 'GET'
          params = {
            'Bucket': bucket_name,
            'Key': object_key
          }
          
          s3_client = boto3.client('s3', region_name=region)
          
          def lambda_handler(event, context):
            url = s3_client.generate_presigned_url(
              ClientMethod=client_method,
              HttpMethod=http_method,
              Params=params,
              ExpiresIn=expire
            )
            return url
      FunctionName: !Sub "${Prefix}-function2"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole2.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

The setup is almost the same as for the aforementioned function.
By setting “get_object” to ClientMethod and “GET” to HttpMethod, a URL for downloading is created.

To summarize the settings of the URL to be created, this function creates a URL to download the object name “sample.txt” located in the S3 bucket “fa-106” for one hour only.

The following is the IAM role for this function.

  FunctionRole2:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: !Sub "${Prefix}-S3GetObjectPolicy"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource:
                  - !Sub "arn:aws:s3:::${S3BucketName}/*"
Code language: YAML (yaml)

Now we need to grant upload permissions to the Lambda function.
For the S3 bucket, configure it to allow the “s3:GetObject” action.

Architecting

Using CloudFormation, we will build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

  • S3 bucket: fa-106
  • Lambda function 1 (creation of upload URL): fa-106-function1
  • Lambda function 2 (creation of URL for download): fa-106-function2

Check each resource from the AWS Management Console.

Check the Lambda functions.

Detail of Lambda 1.
Detail of Lambda 2.

Two functions have indeed been created.

Checking Action

Now that everything is ready, upload/download files to the S3 bucket using the presigned URLs.

Preigned URL for upload

First, check the signed URL for upload.

Execute function 1.

Detail of Lambda 3.

After executing the function, a presigned URL is returned.

Attempt to upload a file to this URL using the curl command.

$ echo "hogehoge" > sample.txt

$ curl -X PUT --upload-file hogehoge.txt "https://fa-106.s3.amazonaws.com/sample.txt?AWSAccessKeyId=[AWSAccessKeyId]&Signature=[Signature]&x-amz-security-token=[x-amz-security-token]"Code language: PHP (php)

Check the S3 bucket.

Detail of S3 1.

The file hogehoge.txt has been renamed sample.txt and placed as an object.

Using the presigned URL, the object can be uploaded to the S3 bucket.

Preigned URL for download

Next, check the presigned URL for download.

Execute function 2.

Detail of Lambda 4.

The function has been executed and the presigned URL has been returned.

Attempt to download the file to this URL using the curl command.

$ curl "https://fa-106.s3.amazonaws.com/sample.txt?AWSAccessKeyId=[AWSAccessKeyId]&Signature=[Signature]&x-amz-security-token=[x-amz-security-token]"
hogehogeCode language: JavaScript (javascript)

The contents of the sample.txt file are output.

By using a presigned URL for downloading in this way, an object can be downloaded from an S3 bucket.

(Reference) Error message

Redirect

If an S3 bucket has just been created, the following message may appear when accessing a presigned URL.

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>TemporaryRedirect</Code>
  <Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message>
  <Endpoint>fa-106.s3-ap-northeast-1.amazonaws.com</Endpoint>
  <Bucket>fa-106</Bucket>
</Error>
Code language: HTML, XML (xml)

The AWS official explanation for this reason is as follows

After you create an Amazon S3 bucket, up to 24 hours can pass before the bucket name propagates across all AWS Regions. During this time, you might receive the 307 Temporary Redirect response for requests to Regional endpoints that aren’t in the same Region as your bucket.

Why am I getting an HTTP 307 Temporary Redirect response from Amazon S3?

Expired

If the deadline set when creating a presigned URL is exceeded, the following message will appear

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Request has expired</Message>
  <Expires>2022-12-21T12:30:33Z</Expires>
  <ServerTime>2022-12-22T12:55:32Z</ServerTime>
</Error>
Code language: HTML, XML (xml)

Summary

We have reviewed how to upload/download files using presigned URLs.

タイトルとURLをコピーしました