AWS_EN

Create yum Repository in S3 and Access from Private Subnet

スポンサーリンク
Create yum Repository in S3 and Access from Private Subnet AWS_EN
スポンサーリンク
スポンサーリンク

Accessing original yum repository on S3 from Amazon Linux 2 in a private subnet

For Amazon Linux (2), yum can be run from a private subnet by referencing an Amazon Linux repositories built on an S3 bucket.

Amazon Linux repositories are hosted in Amazon Simple Storage Service (Amazon S3) buckets. To update and install packages on your instance without an internet connection, create an S3 Amazon Virtual Private Cloud (Amazon VPC) gateway endpoint.

How can I update yum or install packages without internet access on my EC2 instances running Amazon Linux 1 or Amazon Linux 2?

However, not all packages can be installed from the Amazon Linux repositories.
For example, to connect to Oracle or Microsoft SQL Server versions of RDS, a dedicated client is required, but these cannot be installed on EC2 without installing a NAT gateway or similar.

So this time, we will build our own yum repository on S3.
Client packages for the two database engines (Oracle and Microsoft SQL Server) are stored in that repository and installed on instances in the private subnet.

Environment

Diagram of create yum repository in S3 and access from private subnet.

Create two VPCs.
One VPC is for yum repository creation. Deploy the latest Amazon Linux 2 instance, create a repository for the clients, and upload it to the S3 bucket.
On the other VPC, we will deploy a verification instance for our home-grown repository. No Internet Gateway or NAT gateway will be placed in this VPC, and the S3 bucket will be accessed via the VPC endpoint for S3.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL.

awstut-fa/031 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points in the template files

Enable static website hosting functionality and make S3 bucket yum repository

Check the S3 bucket.
The key point is the settings related to static website hosting.

Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: !Ref Prefix AccessControl: Private WebsiteConfiguration: IndexDocument: index.html
Code language: YAML (yaml)

The WebsiteConfiguration property sets the static website hosting functionality. By enabling this function, HTTP communication from yum clients can be accepted.
The point to note is the IndexDocument property. Set “index.html” to this property. This is because the hosting function cannot be activated without setting this property. Therefore, although the index file is not actually set up, it is set as a formality.

Restricting access with bucket policy

Restrict access to the buckets you have created using bucket policies.
The key point is the conditions under which access is allowed. In this case, we will use the following policy to restrict access.

  1. allow access if the source IP address is an Elastic IP address assigned to the NAT gateway * for instances in VPC1
  2. If the source VPC is VPC2, allow access * For instances in VPC2
Resources: BucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref Bucket PolicyDocument: Version: 2012-10-17 Statement: - Action: - s3:* Effect: Allow Resource: - !Sub "arn:aws:s3:::${Bucket}/*" Condition: IpAddress: "aws:SourceIp": - !Ref EIP Principal: "*" - Action: - s3:* Effect: Allow Resource: - !Sub "arn:aws:s3:::${Bucket}/*" Condition: StringEquals: "aws:SourceVpc": - !Ref VPC Principal: "*"
Code language: YAML (yaml)

Two policies are defined.
The first policy is to allow access from instance 1; the key is the Condition property; by setting the IpAddress property to “aws:SourceIp” and the Elastic IP address given to the NAT gateway, the policy will allow access from instance 2. The policy allows communication from the same address. When instance 1 accesses an S3 bucket, it goes through the NAT gateway and the source address is replaced with the Elastic IP address.
The second policy is to allow access from instance 2. The second policy is to allow access from instance 2, again by setting the Condition property to “aws:SourceVpc” and the ID of VPC2 in the StringEquals property.

Create repository with createrepo command and upload it to S3 bucket

To create a yum repository, check the contents to be executed on instance 1.

Resources: RunShellScriptAssociation1: Type: AWS::SSM::Association Properties: AssociationName: !Sub "${Prefix}-runshellscript-association1" Name: AWS-RunShellScript Parameters: commands: - "sudo yum update -y" - !Sub "mkdir ${Repository}" # download oracle client - !Sub "curl https://download.oracle.com/otn_software/linux/instantclient/1914000/{oracle-instantclient19.14-basic-19.14.0.0.0-1.x86_64.rpm} --output ./${Repository}/#1" - !Sub "curl https://download.oracle.com/otn_software/linux/instantclient/1914000/{oracle-instantclient19.14-sqlplus-19.14.0.0.0-1.x86_64.rpm} --output ./${Repository}/#1" # download mssql-tools - "curl -OL https://packages.microsoft.com/config/rhel/7/prod.repo" - "sudo mv ./prod.repo /etc/yum.repos.d/msprod.repo" - !Sub "sudo yum install -y --downloadonly --downloaddir=./${Repository} mssql-tools" # create repository - "sudo yum -y install createrepo" - !Sub "createrepo ./${Repository}" - !Sub "aws s3 cp ./${Repository} s3://${Bucket}/ --recursive" Targets: - Key: InstanceIds Values: - !Ref Instance1 WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

This time, as part of the instance initialization process, the SSM document AWS-RunShellScript is executed to create a repository. For more information on the initialization process using this document, please refer to the following page.

The commands property defines the actual commands to be executed. The commands are as follows

  • Preparation (updating packages, creating directories for storing packages)
  • Download the client (sqlplus) package for Oracle
  • Download the client (sqlcmd) package for Microsoft SQL Server
  • Install and execute the createrepo command to make the above directory a yum repository.
  • Upload repository data to S3 bucket using AWS CLI.

For the installation procedure of each client, we referred to the following official website.

Database Clientインストレーション・ガイド
RPMをダウンロードし、yum installコマンドを使用してOracle Instant Clientをインストールする方法を学習します。
Install SQL Server command-line tools on Linux - SQL Server
Learn how to install the SQL Server command-line tools, Microsoft ODBC drivers, and their dependencies on Linux.

VPC Endpoint for S3

Check VPC2.
This VPC does not create a point of contact with the Internet and accesses the S3 bucket through the VPC endpoint.

Resources: S3Endpoint: Type: AWS::EC2::VPCEndpoint Properties: RouteTableIds: - !Ref PrivateRouteTable ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3" VpcId: !Ref VPC
Code language: YAML (yaml)

No special configuration is required.
Set S3 to the ServiceName property.
The VPC endpoint for S3 is a gateway type, so set the VPC2 ID and the route table associated with the subnet in the VPC.

Register original repository in yum

Check the contents to be performed on instance 2.
The key point is how to refer to the yum repository you have created.

Resources: RunShellScriptAssociation2: Type: AWS::SSM::Association Properties: AssociationName: !Sub "${Prefix}-runshellscript-association2" Name: AWS-RunShellScript Parameters: commands: - !Sub | sudo cat << EOF > /etc/yum.repos.d/${Repository}.repo [${Repository}] name=${Repository} baseurl=${BucketWebsiteURL}/ gpgcheck=0 enabled=1 EOF - "sudo yum clean all" Targets: - Key: InstanceIds Values: - !Ref Instance2 WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

This one also executes the SSM document AWS-RunShellScript to perform the process for referencing your own repositories.
Generally, repositories to be referenced when yum is executed are defined in a configuration file under the /etc/yum.repos.d directory. In the same directory, create and set up a configuration file for your own repository.
This time, register the repository under the name “myrepo”.

Architecting

Use CloudFormation to build this environment and check actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stack.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created in this case is as follows

  • S3 bucket: fa-031
  • Instance 1: i-04533bae0ba1dd468
  • Instance 2: i-054ec8995ebc83925

The resource creation status is also checked from the AWS Management Console.
First is the instance creation status.

Two instances have created.

Indeed, two instances have been created.

Next, we check the execution result of the SSM document on instance 1.

SSM document successfully executed.

You can check the execution result from Output.

Check the S3 bucket.

The yum repository file is uploaded.

You can see that packages, etc. for both clients have been uploaded to the bucket.

S3 static website hosting feature is enabled.

You can also confirm that the static website hosting function is enabled and ready to accept HTTP requests.

Check registration status of original repository

Now that everything is ready, access instance 2.
To access the instance, use SSM Session Manager.
Please refer to the following page for details.

% aws ssm start-session \ --target i-054ec8995ebc83925 Starting session with SessionId: root-0ec77415ecf2fb260 sh-4.2$
Code language: JavaScript (javascript)

Check the file for registering the original repository.

sh-4.2$ ls -l /etc/yum.repos.d/ total 12 -rw-r--r-- 1 root root 1003 Oct 26 17:55 amzn2-core.repo -rw-r--r-- 1 root root 1105 Mar 16 01:52 amzn2-extras.repo -rw-r--r-- 1 root root 105 Mar 19 02:02 myrepo.repo
Code language: Bash (bash)

The file (myrepo.repo) has indeed been created.
Check the contents.

sh-4.2$ cat /etc/yum.repos.d/myrepo.repo [myrepo] name=myrepo baseurl=http://fa-031.s3-website-ap-northeast-1.amazonaws.com/ gpgcheck=0 enabled=1
Code language: Bash (bash)

Note the value of baseurl, which is the URL of the S3 website endpoint. This means that when referencing myrepo, the S3 bucket will be accessed.

Check the list of repositories recognized by yum.

sh-4.2$ yum repolist Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2 packages excluded due to repository priority protections repo id repo name status amzn2-core/2/x86_64 Amazon Linux 2 core repository 27418 amzn2extra-docker/2/x86_64 Amazon Extras repo for docker 56 myrepo myrepo 4+2 repolist: 27478
Code language: Bash (bash)

You can see that myrepo is recognized in addition to the default Amazon Linux repositories.

Check the packages that can be installed from myrepo.

sh-4.2$ yum list all | grep myrepo msodbcsql17.x86_64 17.9.1.1-1 myrepo mssql-tools.x86_64 17.9.1.1-1 myrepo oracle-instantclient19.14-basic.x86_64 19.14.0.0.0-1 myrepo 19.14.0.0.0-1 myrepo sh-4.2$
Code language: Bash (bash)

You can see that it is hosted in myrepo.

Client installation for Oracle

First, install the client for Oracle.

sh-4.2$ sudo yum install -y oracle-instantclient19.14-basic.x86_64 ... Installed: oracle-instantclient19.14-basic.x86_64 0:19.14.0.0.0-1 Complete! sh-4.2$ sudo yum install -y oracle-instantclient19.14-sqlplus.x86_64 ... Installed: oracle-instantclient19.14-sqlplus.x86_64 0:19.14.0.0.0-1 Complete!
Code language: Bash (bash)

The installation was successful.

Check the installed packages.

sh-4.2$ yum list installed | grep oracle oracle-instantclient19.14-basic.x86_64 oracle-instantclient19.14-sqlplus.x86_64
Code language: Bash (bash)

The Oracle client was indeed installed from myrepo.

Check the version.

sh-4.2$ sqlplus -V SQL*Plus: Release 19.0.0.0.0 - Production Version 19.14.0.0.0
Code language: Bash (bash)

The version was successfully displayed.

Client installation for Microsoft SQL Server

First, install the client for SQL Server.

sh-4.2$ sudo yum install -y mssql-tools ... Installed: mssql-tools.x86_64 0:17.9.1.1-1 Dependency Installed: libtool-ltdl.x86_64 0:2.4.2-22.2.amzn2.0.2 msodbcsql17.x86_64 0:17.9.1.1-1 unixODBC.x86_64 0:2.3.1-14.amzn2 Complete!
Code language: Bash (bash)

The installation was successful.

Check the installed packages.

sh-4.2$ yum list installed | grep mssql-tools mssql-tools.x86_64 17.9.1.1-1 @myrepo
Code language: Bash (bash)

Sure enough, the SQL Server client was installed from myrepo.

Check the version.

sh-4.2$ /opt/mssql-tools/bin/sqlcmd -? Microsoft (R) SQL Server Command Line Tool Version 17.9.0001.1 Linux Copyright (C) 2017 Microsoft Corporation. All rights reserved. ...
Code language: Bash (bash)

The version was successfully displayed.

Summary

We have seen how to build our own yum repository on S3.
By referencing the original repository, we were able to install packages that do not exist in the Amazon Linux repositories on instances in our private subnet.

タイトルとURLをコピーしました