SOA_EN

Create a custom patch baseline with SSM Patch Manager (Linux)

スポンサーリンク
SSM Patch Manager on Linux instance. SOA_EN
スポンサーリンク
スポンサーリンク

Configure SSM Patch Manager to create custom patch baselines and distribute patches

System Manager (SSM) Patch Manager is a function that automates the application of patches to EC2 instances.

This time, we will target an Amazon Linux 2 instance. In addition to the default AWS-AmazonLinux2DefaultPatchBaseline, we will create our own patch baseline (custom patch baseline) and run both to check the behavior of Patch Manager.

For more information on running SSM Patch Manager on Windows instances, please refer to the following page.

Terms related to SSM Patch Manager

First, we will briefly organize the terms related to SSM and Patch Manager. There are three terms we will cover.

Managed Instance

The first one is managed instance. A managed instance is an EC2 instance that is ready to be manipulated by SSM.

A managed instance is an Amazon EC2 instance that is configured for use with Systems Manager. Managed instances can use Systems Manager services such as Run Command, Patch Manager, and Session Manager.

Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?

Patch Baseline

The second is the patch baseline. A patch baseline is a resource that corresponds to a rule for delivering patches from Patch Manager. When you create a patch baseline, you specify the target OS for patch distribution and the conditions (classification and severity) of the patches to be approved.

Patch Group

The third is the patch group. A patch group is a framework that organizes the targets for patch distribution according to the patch baseline.

You can use a patch group to associate managed nodes with a specific patch baseline in Patch Manager, a capability of AWS Systems Manager. Patch groups help ensure that you’re deploying the appropriate patches, based on the associated patch baseline rules, to the correct set of nodes.

About patch groups

If you have multiple instances that you want to distribute patches with similar rules, you can distribute patches in a unified manner by having them belong to the same patch group.

Environment

Diagram of SSM Patch Manager on Linux instance.

Place two EC2 instances in a private subnet. Apply the default AWS-AmazonLinux2DefaultPatchBaseline to one, and apply a custom patch baseline to the other.
Save the execution log of Patch Manager in an S3 bucket.

CloudFormation templates

We will build the above configuration using CloudFormation. We have placed the CloudFormation template file at the following URL.

awstut-soa/04/001 at main · awstut-an-r/awstut-soa
Contribute to awstut-an-r/awstut-soa development by creating an account on GitHub.

Explanation of key points of the templates

We will cover the key points of each template file to configure this architecture.

Prepare bucket to store Patch Manager logs

Resources: S3Bucket: Type: AWS::S3::Bucket Properties: BucketName: !Ref Prefix AccessControl: Private
Code language: YAML (yaml)

Create an S3 bucket, which is used to store the logs of Patch Manager runtime. No special configuration is required.

Allow VPC endpoints to run Patch Manager on instances in private subnets

In order to run Patch Manager against an instance in a private subnet, there are two main types of VPC endpoints that are required.

Endpoint for SSM

The first is a VPC endpoint for SSM.

Resources: SSMEndpoint: Type: AWS::EC2::VPCEndpoint Properties: PrivateDnsEnabled: true SecurityGroupIds: - !Ref EndpointSecurityGroup ServiceName: !Sub "com.amazonaws.${AWS::Region}.ssm" SubnetIds: - !Ref PrivateSubnet VpcEndpointType: Interface VpcId: !Ref VPC EC2MessagesEndpoint: Type: AWS::EC2::VPCEndpoint Properties: PrivateDnsEnabled: true SecurityGroupIds: - !Ref EndpointSecurityGroup ServiceName: !Sub "com.amazonaws.${AWS::Region}.ec2messages" SubnetIds: - !Ref PrivateSubnet VpcEndpointType: Interface VpcId: !Ref VPC
Code language: YAML (yaml)

When creating VPC endpoints for SSM, at least two interface endpoints are required.

com.amazonaws.region.ssm: The endpoint for the Systems Manager service.

com.amazonaws.region.ec2messages: Systems Manager uses this endpoint to make calls from SSM Agent to the Systems Manager service.

Step 6: (Optional) Create a Virtual Private Cloud endpoint

Follow the above steps to create two endpoint resources.

Endpoint for S3

The second one is an endpoint for S3.

Resources: S3Endpoint: Type: AWS::EC2::VPCEndpoint Properties: PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: "*" Action: "*" Resource: "*" RouteTableIds: - !Ref PrivateRouteTable ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3" VpcId: !Ref VPC
Code language: YAML (yaml)

This is also considered a mandatory endpoint for manipulating instances with SSM.

com.amazonaws.region.s3: Systems Manager uses this endpoint to update SSM Agent, perform patching operations, and for tasks like uploading output logs you choose to store in S3 buckets, retrieving scripts or other files you store in buckets, and so on.

Step 6: (Optional) Create a Virtual Private Cloud endpoint

The specific buckets that need to be accessed in order to run Patch Manager are explained in the section on IAM roles to be applied to instances.

Specify patch group to which instance belongs with tag

First, define the EC2 instance itself. The point is to set the tags to be assigned to the instances.

Resources: Instance1: Type: AWS::EC2::Instance Properties: IamInstanceProfile: !Ref InstanceProfile ImageId: !Ref ImageId InstanceType: !Ref InstanceType NetworkInterfaces: - DeviceIndex: 0 SubnetId: !Ref PrivateSubnet GroupSet: - !Ref InstanceSecurityGroup Instance2: Type: AWS::EC2::Instance Properties: ... Tags: - Key: Patch Group Value: !Sub "${Prefix}-patch-group"
Code language: YAML (yaml)

Define two instances. The settings are exactly the same, except for the tag settings; only Instance2 will have a tag.
The patch group to which the EC2 instance belongs is specified by the tag.

Patch groups require use of the tag key Patch Group. You can specify any tag value, but the tag key must be Patch Group.

Working with patch groups

Following the above, specify “Patch Group” in the Key property and the patch group name in the Value property in the Tags property of Instance2. This time, we will use the built-in function Fn::Sub to set the patch group to belong to the “soa-04-001-patch-group”.

As for the base AMI for the EC2 instance, I deliberately chose a slightly older version instead of the latest version. This is because if you select the latest version of the AMI, the Patch Manager may create an instance to which the patches you plan to apply have already been applied.

Prepare permissions in IAM role to meet requirements for SSM managed instances

Next, we will define the IAM role to be assigned to the EC2 instance.

Resources: InstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - ec2.amazonaws.com Policies: - PolicyName: PutPatchBaselineLog PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:GetObject Resource: - !Sub "arn:aws:s3:::aws-windows-downloads-${AWS::Region}/*" - !Sub "arn:aws:s3:::amazon-ssm-${AWS::Region}/*" - !Sub "arn:aws:s3:::amazon-ssm-packages-${AWS::Region}/*" - !Sub "arn:aws:s3:::${AWS::Region}-birdwatcher-prod/*" - !Sub "arn:aws:s3:::aws-ssm-document-attachments-${AWS::Region}/*" - !Sub "arn:aws:s3:::patch-baseline-snapshot-${AWS::Region}/*" - !Sub "arn:aws:s3:::aws-ssm-${AWS::Region}/*" - !Sub "arn:aws:s3:::aws-patchmanager-macos-${AWS::Region}/*" - Effect: Allow Action: - s3:PutObject Resource: - !Sub "${S3BucketArn}/*" ManagedPolicyArns: - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Code language: YAML (yaml)

In order to be treated as a managed instance, the following conditions must be met

Have the AWS Systems Manager Agent (SSM Agent) installed and running.

Have connectivity with Systems Manager endpoints using the SSM Agent.

Have the correct AWS Identity and Access Management (IAM) role attached.

Have connectivity to the instance metadata service.

Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?

The above three conditions are particularly important.
Regarding the first one, the SSM agent, an Amazon Linux 2-based EC2 instance has the SSM agent installed and running by default. Therefore, it already meets the requirements.
As for the second endpoint, the VPC endpoint mentioned above will be installed, so this also meets the requirements.
As for the third IAM role, the IAM role that should be attached to the EC2 instance is mentioned in the SSM as follows

To make APIs calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore permissions policy to the IAM role attached to your instance.

Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?

It also says that you need to access a specific S3 bucket to run SSM.

In the course of performing various Systems Manager operations, AWS Systems Manager Agent (SSM Agent) accesses a number of Amazon Simple Storage Service (Amazon S3) buckets.

SSM Agent communications with AWS managed S3 buckets

From the above, create an IAM role that meets the requirements.

You can specify the AWS management policy in the ManagedPolicyArns property. As we saw earlier, specify “AmazonSSMManagedInstanceCore”.

Next, define the inline policies in the Policies property: the first policy allows you to retrieve objects for the S3 buckets that you access to run SSM. The second policy allows you to retrieve objects for the buckets that store the execution logs of Patch Manager. The second policy allows storing objects against the bucket that stores the execution log of Patch Manager.

OS, patch groups, and approval rules are key to creating custom baselines

Check the custom baseline.

Resources: MyPatchBaseline: Type: AWS::SSM::PatchBaseline Properties: ApprovalRules: PatchRules: - ApproveAfterDays: 7 ComplianceLevel: UNSPECIFIED EnableNonSecurity: false PatchFilterGroup: PatchFilters: - Key: CLASSIFICATION Values: - Security - Key: SEVERITY Values: - Critical #- Important #- ApproveAfterDays: 7 # ComplianceLevel: UNSPECIFIED # EnableNonSecurity: false # PatchFilterGroup: # PatchFilters: # - Key: CLASSIFICATION # Values: # - Bugfix Description: Test Patch Baseline. Name: !Sub ${Prefix}-MyPatchBaseline OperatingSystem: AMAZON_LINUX_2 PatchGroups: - !Sub ${Prefix}-patch-group
Code language: YAML (yaml)

There are various parameters to create a custom baseline, but there are three important ones.
The first one is the target OS. Set the OS in the OperatingSystem property. In this case, we will target an EC2 instance running on Amazon Linux 2, so specify “AMAZON_LINUX_2”.
The second is the patch group. Specify the managed instances to be patched in the patch baseline in the form of patch groups. Set the patch group in the PatchGroups property. This time, specify the same value as the tag set for Isntance2.
The third is the approval rule. Set the conditions for the patches to be delivered, the grace time for automatic approval, and so on. This time, we will create a custom patch baseline by referring to the default patch baseline AWS-AmazonLinux2DefaultPatchBaseline and changing some of the parameters.

The following image details the AWS-AmazonLinux2DefaultPatchBaseline.

Outline of AWS-AmazonLinux2DefaultPatchBaseline

In this case, we will create a custom patch baseline for only those patches that have a classification of Security and a severity of Critical.
The approval rules are defined in the ApprovalRules property. You can specify the number of days until the patches corresponding to this rule are automatically approved by using the ApproveAfterDays property.
The ComplianceLevel property allows you to specify the level to be assigned to the patches that correspond to this rule.
In the EnableNonSecurity property, you can specify whether to distribute non-security patches as well.
The PatchFilterGroup property is used to configure the classification and severity settings. You can specify the classification and importance in the PatchFilterGroup property of the same property. To define the classification, specify “CLASSIFICATION” for the Key property, and to specify the severity, specify “SEVERITY”. In this case, we will specify “Security” for the classification and “Critical” for the severity in the Values property.

Note that if you uncomment the settings, the configuration will be the same as AWS-AmazonLinux2DefaultPatchBaseline.

Running Patch Baseline

For the two instances, create an association with the SSM document and run the patch baseline.

Resources: AWSRunPatchBaselineAssociation: Type: AWS::SSM::Association Properties: AssociationName: !Sub "${Prefix}-AWSRunPatchBaselineAssociation" Name: AWS-RunPatchBaseline OutputLocation: S3Location: OutputS3BucketName: !Ref S3BucketName OutputS3KeyPrefix: !Sub "AWSRunPatchBaselineAssociation" Parameters: Operation: - Install Targets: - Key: InstanceIds Values: - !Ref Instance1 - !Ref Instance2 WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

In order to run the patch baseline, create an association with the SSM document AWS-RunPatchBaseline.
In the OutputLocation property, specify the bucket where the Patch Manager runtime logs are saved.
In the Paramaters, specify the parameters for running AWS-RunPatchBaseline. Specify Install in the Operation property, which is a required parameter. For more information, see About the AWS-RunPatchBaseline SSM document.
Specify the targets to which you want to associate the SSM document in the Targets property. In this case, we will specify the IDs of the two instances mentioned above.

Let’s organize the behavior when AWS-RunPatchBaseline is executed with this configuration.

  • Instance1: No tag (patch group) setting -> The default AWS-AmazonLinux2DefaultPatchBaseline will be executed.
  • Instance2: With tag (patch group) setting -> Custom patch group (MyPatchBaseline) will be executed.

Architecting

Using CloudFormation, we will build this environment and check its actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stack.
For more information on creating stacks and how to check each stack, please refer to the following page.

After checking the resources of each stack, the information of the main resources created this time is as follows

  • Name of the bucket for log storage: soa-04-001
  • ID of Instance1: i-07d21ade1467247e6
  • ID of Instance2: i-09327871f0830e91e
  • Custom patch baseline name: soa-04-001-MyPatchBaseline
  • ID of custom patch baseline: pb-0b520c1268ad72b19
  • Patch group for the custom patch baseline: soa-04-001-patch-group

Verify the custom patch baseline that has been created.

Details of the Custom Patch Baseline I created.

In the Patch groups section, we can also see that the target of this baseline is “soa-04-001-patch-group”.

Next, let’s check the tags assigned to the instances.

Instance1 is not configured with a tag for patch groups.
Instance2 is configured with a tag for the patch group.

Only Instance2 has a Patch Group tag. You can see that the value of this tag is the same name as the patch group in the custom patch baseline that we just checked. This means that Instance2 belongs to the patch group.

Check results of Patch Manager execution

Now that we are ready, let’s check the results of the Patch Manager run, which can be viewed on the SSM Run Command page.

Results of running SSM Patch Manager.

We can see that the Patch Manager has been successfully executed for two instances.

Next, let’s check the execution results for each instance. First, let’s check Instance1.

Result of running AWS-AmazonLinux2DefaultPatchBaseline.

You can see that the default AWS-AmazonLinux2DefaultPatchBaseline has been run and the 65 patches have been applied.

Next, we will check Instance2.

Result of running Custom Patch Baseline.

You can see that the custom patch baseline (soa-04-001-MyPatchBaseline) has been executed and the 9 patches have been applied.

Finally, check the execution log. The logs will be placed in the specified bucket for each instance ID.

The execution log of SSM Patch Manager is saved in the S3 bucket.

The above image shows the log of Instance1, but the execution log is placed as stdout.txt.

Let’s check the contents of the Instance1 execution log.

/usr/bin/python3 /usr/bin/python2.7 /usr/bin/python2 /usr/bin/python /usr/bin/yum Using Yum version: 3.4.3 Using python binary: 'python2.7' Using Python Version: Python 2.7.18 ... 01/21/2022 23:26:08 root [INFO]: [{'OperationStartTime': '2022-01-21T23:25:15Z', 'BaselineId': u'pb-00fda5699d1ae3942', 'InstalledPendingRebootCount': '0', 'FailedCount': '0', 'MissingCount': '0', 'NotApplicableCount': '825', 'RebootOption': u'RebootIfNeeded', 'OperationEndTime': '2022-01-21T23:26:08Z', 'SecurityNonCompliantCount': '0', 'PatchGroup': u'', 'ExecutionId': '4fbe03b6-0e37-4cbf-995a-0f13a99f952a', 'InstalledRejectedCount': '0', 'OtherNonCompliantCount': '0', 'InstalledOtherCount': '377', 'CriticalNonCompliantCount': '0', 'SnapshotId': u'5839c3cf-0024-4a17-bdb2-302c3606223a', 'InstalledCount': '65', 'OperationType': u'Install'}] ...
Code language: plaintext (plaintext)

You can check the patching process.

Summary

Using SSM Patch Manager, we confirmed that we can deliver patches to EC2 instances (Linux).

By creating and running a custom patch baseline in addition to the default AWS-AmazonLinux2DefaultPatchBaseline, we confirmed how to check the patch application results and the log output status.

タイトルとURLをコピーしました