Create yum Repository in S3 and Access from Private Subnet

Accessing original yum repository on S3 from Amazon Linux 2 in a private subnet

For Amazon Linux (2), yum can be run from a private subnet by referencing an Amazon Linux repositories built on an S3 bucket.

Amazon Linux repositories are hosted in Amazon Simple Storage Service (Amazon S3) buckets. To update and install packages on your instance without an internet connection, create an S3 Amazon Virtual Private Cloud (Amazon VPC) gateway endpoint.

How can I update yum or install packages without internet access on my EC2 instances running Amazon Linux 1 or Amazon Linux 2?

However, not all packages can be installed from the Amazon Linux repositories.
For example, to connect to Oracle or Microsoft SQL Server versions of RDS, a dedicated client is required, but these cannot be installed on EC2 without installing a NAT gateway or similar.

So this time, we will build our own yum repository on S3.
Client packages for the two database engines (Oracle and Microsoft SQL Server) are stored in that repository and installed on instances in the private subnet.


Diagram of create yum repository in S3 and access from private subnet.

Create two VPCs.
One VPC is for yum repository creation. Deploy the latest Amazon Linux 2 instance, create a repository for the clients, and upload it to the S3 bucket.
On the other VPC, we will deploy a verification instance for our home-grown repository. No Internet Gateway or NAT gateway will be placed in this VPC, and the S3 bucket will be accessed via the VPC endpoint for S3.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL.

Explanation of key points in the template files

Enable static website hosting functionality and make S3 bucket yum repository

Check the S3 bucket.
The key point is the settings related to static website hosting.

    Type: AWS::S3::Bucket
      BucketName: !Ref Prefix
      AccessControl: Private
        IndexDocument: index.html
Code language: YAML (yaml)

The WebsiteConfiguration property sets the static website hosting functionality. By enabling this function, HTTP communication from yum clients can be accepted.
The point to note is the IndexDocument property. Set “index.html” to this property. This is because the hosting function cannot be activated without setting this property. Therefore, although the index file is not actually set up, it is set as a formality.

Restricting access with bucket policy

Restrict access to the buckets you have created using bucket policies.
The key point is the conditions under which access is allowed. In this case, we will use the following policy to restrict access.

  1. allow access if the source IP address is an Elastic IP address assigned to the NAT gateway * for instances in VPC1
  2. If the source VPC is VPC2, allow access * For instances in VPC2
    Type: AWS::S3::BucketPolicy
      Bucket: !Ref Bucket
        Version: 2012-10-17
          - Action:
              - s3:*
            Effect: Allow
              - !Sub "arn:aws:s3:::${Bucket}/*"
                  - !Ref EIP
            Principal: "*"
          - Action:
              - s3:*
            Effect: Allow
              - !Sub "arn:aws:s3:::${Bucket}/*"
                  - !Ref VPC
            Principal: "*"
Code language: YAML (yaml)

Two policies are defined.
The first policy is to allow access from instance 1; the key is the Condition property; by setting the IpAddress property to “aws:SourceIp” and the Elastic IP address given to the NAT gateway, the policy will allow access from instance 2. The policy allows communication from the same address. When instance 1 accesses an S3 bucket, it goes through the NAT gateway and the source address is replaced with the Elastic IP address.
The second policy is to allow access from instance 2. The second policy is to allow access from instance 2, again by setting the Condition property to “aws:SourceVpc” and the ID of VPC2 in the StringEquals property.

Create repository with createrepo command and upload it to S3 bucket

To create a yum repository, check the contents to be executed on instance 1.

    Type: AWS::SSM::Association
      AssociationName: !Sub "${Prefix}-runshellscript-association1"
      Name: AWS-RunShellScript
          - "sudo yum update -y"

          - !Sub "mkdir ${Repository}"

           # download oracle client
          - !Sub "curl{oracle-instantclient19.14-basic-} --output ./${Repository}/#1"
          - !Sub "curl{oracle-instantclient19.14-sqlplus-} --output ./${Repository}/#1"

          # download mssql-tools
          - "curl -OL"
          - "sudo mv ./prod.repo /etc/yum.repos.d/msprod.repo"
          - !Sub "sudo yum install -y --downloadonly --downloaddir=./${Repository} mssql-tools"

          # create repository
          - "sudo yum -y install createrepo"
          - !Sub "createrepo ./${Repository}"

          - !Sub "aws s3 cp ./${Repository} s3://${Bucket}/ --recursive"
        - Key: InstanceIds
            - !Ref Instance1
      WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

This time, as part of the instance initialization process, the SSM document AWS-RunShellScript is executed to create a repository. For more information on the initialization process using this document, please refer to the following page.

Four ways to initialize Linux instance 【Four ways to initialize a Linux instance】 Consider how to perform the initialization process when an EC2 instance is started. We will cover the following ...

The commands property defines the actual commands to be executed. The commands are as follows

  • Preparation (updating packages, creating directories for storing packages)
  • Download the client (sqlplus) package for Oracle
  • Download the client (sqlcmd) package for Microsoft SQL Server
  • Install and execute the createrepo command to make the above directory a yum repository.
  • Upload repository data to S3 bucket using AWS CLI.

For the installation procedure of each client, we referred to the following official website.

VPC Endpoint for S3

Check VPC2.
This VPC does not create a point of contact with the Internet and accesses the S3 bucket through the VPC endpoint.

    Type: AWS::EC2::VPCEndpoint
        - !Ref PrivateRouteTable
      ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3"
      VpcId: !Ref VPC
Code language: YAML (yaml)

No special configuration is required.
Set S3 to the ServiceName property.
The VPC endpoint for S3 is a gateway type, so set the VPC2 ID and the route table associated with the subnet in the VPC.

Register original repository in yum

Check the contents to be performed on instance 2.
The key point is how to refer to the yum repository you have created.

    Type: AWS::SSM::Association
      AssociationName: !Sub "${Prefix}-runshellscript-association2"
      Name: AWS-RunShellScript
          - !Sub |
              sudo cat << EOF > /etc/yum.repos.d/${Repository}.repo
          - "sudo yum clean all"
        - Key: InstanceIds
            - !Ref Instance2
      WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

This one also executes the SSM document AWS-RunShellScript to perform the process for referencing your own repositories.
Generally, repositories to be referenced when yum is executed are defined in a configuration file under the /etc/yum.repos.d directory. In the same directory, create and set up a configuration file for your own repository.
This time, register the repository under the name “myrepo”.


Use CloudFormation to build this environment and check actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stack.
For information on how to create stacks and check each stack, please refer to the following page

CloudFormation’s nested stack 【How to build an environment with a nested CloudFormation stack】 Examine nested stacks in CloudFormation. CloudFormation allows you to nest stacks. Nested ...

After checking the resources in each stack, information on the main resources created in this case is as follows

  • S3 bucket: fa-031
  • Instance 1: i-04533bae0ba1dd468
  • Instance 2: i-054ec8995ebc83925

The resource creation status is also checked from the AWS Management Console.
First is the instance creation status.

Two instances are created.

Indeed, two instances have been created.

Next, we check the execution result of the SSM document on instance 1.

SSM document successfully executed.

You can check the execution result from Output.

Check the S3 bucket.

The yum repository file is uploaded.

You can see that packages, etc. for both clients have been uploaded to the bucket.

S3 static website hosting feature is enabled.

You can also confirm that the static website hosting function is enabled and ready to accept HTTP requests.

Check registration status of original repository

Now that everything is ready, access instance 2.
To access the instance, use SSM Session Manager.
Please refer to the following page for details.

Accessing Linux instance via SSM Session Manager 【Configure Linux instances to be accessed via SSM Session Manager】 We will check a configuration in which an EC2 instance is accessed via SSM Session Manag...
% aws ssm start-session \
--target i-054ec8995ebc83925

Starting session with SessionId: root-0ec77415ecf2fb260

sh-4.2$Code language: JavaScript (javascript)

Check the file for registering the original repository.

sh-4.2$ ls -l /etc/yum.repos.d/
total 12
-rw-r--r-- 1 root root 1003 Oct 26 17:55 amzn2-core.repo
-rw-r--r-- 1 root root 1105 Mar 16 01:52 amzn2-extras.repo
-rw-r--r-- 1 root root  105 Mar 19 02:02 myrepo.repo
Code language: Bash (bash)

The file (myrepo.repo) has indeed been created.
Check the contents.

sh-4.2$ cat /etc/yum.repos.d/myrepo.repo
Code language: Bash (bash)

Note the value of baseurl, which is the URL of the S3 website endpoint. This means that when referencing myrepo, the S3 bucket will be accessed.

Check the list of repositories recognized by yum.

sh-4.2$ yum repolist
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2 packages excluded due to repository priority protections
repo id                                                                repo name                                                                  status
amzn2-core/2/x86_64                                                    Amazon Linux 2 core repository                                             27418
amzn2extra-docker/2/x86_64                                             Amazon Extras repo for docker                                                 56
myrepo                                                                 myrepo                                                                       4+2
repolist: 27478
Code language: Bash (bash)

You can see that myrepo is recognized in addition to the default Amazon Linux repositories.

Check the packages that can be installed from myrepo.

sh-4.2$ yum list all | grep myrepo
msodbcsql17.x86_64                                myrepo
mssql-tools.x86_64                                myrepo
oracle-instantclient19.14-basic.x86_64                  myrepo
Code language: Bash (bash)

You can see that it is hosted in myrepo.

Client installation for Oracle

First, install the client for Oracle.

sh-4.2$ sudo yum install -y oracle-instantclient19.14-basic.x86_64
  oracle-instantclient19.14-basic.x86_64 0:


sh-4.2$ sudo yum install -y oracle-instantclient19.14-sqlplus.x86_64
  oracle-instantclient19.14-sqlplus.x86_64 0:

Code language: Bash (bash)

The installation was successful.

Check the installed packages.

sh-4.2$ yum list installed | grep oracle
Code language: Bash (bash)

The Oracle client was indeed installed from myrepo.

Check the version.

sh-4.2$ sqlplus -V

SQL*Plus: Release - Production
Code language: Bash (bash)

The version was successfully displayed.

Client installation for Microsoft SQL Server

First, install the client for SQL Server.

sh-4.2$ sudo yum install -y mssql-tools
  mssql-tools.x86_64 0:

Dependency Installed:
  libtool-ltdl.x86_64 0:2.4.2-22.2.amzn2.0.2               msodbcsql17.x86_64 0:               unixODBC.x86_64 0:2.3.1-14.amzn2


Code language: Bash (bash)

The installation was successful.

Check the installed packages.

sh-4.2$ yum list installed | grep mssql-tools
mssql-tools.x86_64                               @myrepo
Code language: Bash (bash)

Sure enough, the SQL Server client was installed from myrepo.

Check the version.

sh-4.2$ /opt/mssql-tools/bin/sqlcmd -?
Microsoft (R) SQL Server Command Line Tool
Version 17.9.0001.1 Linux
Copyright (C) 2017 Microsoft Corporation. All rights reserved.
Code language: Bash (bash)

The version was successfully displayed.


We have seen how to build our own yum repository on S3.
By referencing the original repository, we were able to install packages that do not exist in the Amazon Linux repositories on instances in our private subnet.