Configure SSM Patch Manager to create custom patch baselines and distribute patches
System Manager (SSM) Patch Manager is a function that automates the application of patches to EC2 instances.
This time, we will target an Amazon Linux 2 instance. In addition to the default AWS-AmazonLinux2DefaultPatchBaseline, we will create our own patch baseline (custom patch baseline) and run both to check the behavior of Patch Manager.
For more information on running SSM Patch Manager on Windows instances, please refer to the following page.
Terms related to SSM Patch Manager
First, we will briefly organize the terms related to SSM and Patch Manager. There are three terms we will cover.
Managed Instance
The first one is managed instance. A managed instance is an EC2 instance that is ready to be manipulated by SSM.
A managed instance is an Amazon EC2 instance that is configured for use with Systems Manager. Managed instances can use Systems Manager services such as Run Command, Patch Manager, and Session Manager.
Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?
Patch Baseline
The second is the patch baseline. A patch baseline is a resource that corresponds to a rule for delivering patches from Patch Manager. When you create a patch baseline, you specify the target OS for patch distribution and the conditions (classification and severity) of the patches to be approved.
Patch Group
The third is the patch group. A patch group is a framework that organizes the targets for patch distribution according to the patch baseline.
You can use a patch group to associate managed nodes with a specific patch baseline in Patch Manager, a capability of AWS Systems Manager. Patch groups help ensure that you’re deploying the appropriate patches, based on the associated patch baseline rules, to the correct set of nodes.
About patch groups
If you have multiple instances that you want to distribute patches with similar rules, you can distribute patches in a unified manner by having them belong to the same patch group.
Environment
Place two EC2 instances in a private subnet. Apply the default AWS-AmazonLinux2DefaultPatchBaseline to one, and apply a custom patch baseline to the other.
Save the execution log of Patch Manager in an S3 bucket.
CloudFormation templates
We will build the above configuration using CloudFormation. We have placed the CloudFormation template file at the following URL.
https://github.com/awstut-an-r/awstut-soa/tree/main/04/001
Explanation of key points of the templates
We will cover the key points of each template file to configure this architecture.
Prepare bucket to store Patch Manager logs
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref Prefix
AccessControl: Private
Code language: YAML (yaml)Create an S3 bucket, which is used to store the logs of Patch Manager runtime. No special configuration is required.
Allow VPC endpoints to run Patch Manager on instances in private subnets
In order to run Patch Manager against an instance in a private subnet, there are two main types of VPC endpoints that are required.
Endpoint for SSM
The first is a VPC endpoint for SSM.
Resources:
SSMEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref EndpointSecurityGroup
ServiceName: !Sub "com.amazonaws.${AWS::Region}.ssm"
SubnetIds:
- !Ref PrivateSubnet
VpcEndpointType: Interface
VpcId: !Ref VPC
EC2MessagesEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref EndpointSecurityGroup
ServiceName: !Sub "com.amazonaws.${AWS::Region}.ec2messages"
SubnetIds:
- !Ref PrivateSubnet
VpcEndpointType: Interface
VpcId: !Ref VPC
Code language: YAML (yaml)When creating VPC endpoints for SSM, at least two interface endpoints are required.
com.amazonaws.region.ssm: The endpoint for the Systems Manager service.
com.amazonaws.region.ec2messages: Systems Manager uses this endpoint to make calls from SSM Agent to the Systems Manager service.
Step 6: (Optional) Create a Virtual Private Cloud endpoint
Follow the above steps to create two endpoint resources.
Endpoint for S3
The second one is an endpoint for S3.
Resources:
S3Endpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: "*"
Action: "*"
Resource: "*"
RouteTableIds:
- !Ref PrivateRouteTable
ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3"
VpcId: !Ref VPC
Code language: YAML (yaml)This is also considered a mandatory endpoint for manipulating instances with SSM.
com.amazonaws.region.s3: Systems Manager uses this endpoint to update SSM Agent, perform patching operations, and for tasks like uploading output logs you choose to store in S3 buckets, retrieving scripts or other files you store in buckets, and so on.
Step 6: (Optional) Create a Virtual Private Cloud endpoint
The specific buckets that need to be accessed in order to run Patch Manager are explained in the section on IAM roles to be applied to instances.
Specify patch group to which instance belongs with tag
First, define the EC2 instance itself. The point is to set the tags to be assigned to the instances.
Resources:
Instance1:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile: !Ref InstanceProfile
ImageId: !Ref ImageId
InstanceType: !Ref InstanceType
NetworkInterfaces:
- DeviceIndex: 0
SubnetId: !Ref PrivateSubnet
GroupSet:
- !Ref InstanceSecurityGroup
Instance2:
Type: AWS::EC2::Instance
Properties:
...
Tags:
- Key: Patch Group
Value: !Sub "${Prefix}-patch-group"
Code language: YAML (yaml)Define two instances. The settings are exactly the same, except for the tag settings; only Instance2 will have a tag.
The patch group to which the EC2 instance belongs is specified by the tag.
Patch groups require use of the tag key Patch Group. You can specify any tag value, but the tag key must be Patch Group.
Working with patch groups
Following the above, specify “Patch Group” in the Key property and the patch group name in the Value property in the Tags property of Instance2. This time, we will use the built-in function Fn::Sub to set the patch group to belong to the “soa-04-001-patch-group”.
As for the base AMI for the EC2 instance, I deliberately chose a slightly older version instead of the latest version. This is because if you select the latest version of the AMI, the Patch Manager may create an instance to which the patches you plan to apply have already been applied.
Prepare permissions in IAM role to meet requirements for SSM managed instances
Next, we will define the IAM role to be assigned to the EC2 instance.
Resources:
InstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- ec2.amazonaws.com
Policies:
- PolicyName: PutPatchBaselineLog
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- !Sub "arn:aws:s3:::aws-windows-downloads-${AWS::Region}/*"
- !Sub "arn:aws:s3:::amazon-ssm-${AWS::Region}/*"
- !Sub "arn:aws:s3:::amazon-ssm-packages-${AWS::Region}/*"
- !Sub "arn:aws:s3:::${AWS::Region}-birdwatcher-prod/*"
- !Sub "arn:aws:s3:::aws-ssm-document-attachments-${AWS::Region}/*"
- !Sub "arn:aws:s3:::patch-baseline-snapshot-${AWS::Region}/*"
- !Sub "arn:aws:s3:::aws-ssm-${AWS::Region}/*"
- !Sub "arn:aws:s3:::aws-patchmanager-macos-${AWS::Region}/*"
- Effect: Allow
Action:
- s3:PutObject
Resource:
- !Sub "${S3BucketArn}/*"
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Code language: YAML (yaml)In order to be treated as a managed instance, the following conditions must be met
Have the AWS Systems Manager Agent (SSM Agent) installed and running.
Have connectivity with Systems Manager endpoints using the SSM Agent.
Have the correct AWS Identity and Access Management (IAM) role attached.
Have connectivity to the instance metadata service.
Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?
The above three conditions are particularly important.
Regarding the first one, the SSM agent, an Amazon Linux 2-based EC2 instance has the SSM agent installed and running by default. Therefore, it already meets the requirements.
As for the second endpoint, the VPC endpoint mentioned above will be installed, so this also meets the requirements.
As for the third IAM role, the IAM role that should be attached to the EC2 instance is mentioned in the SSM as follows
To make APIs calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore permissions policy to the IAM role attached to your instance.
Why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?
It also says that you need to access a specific S3 bucket to run SSM.
In the course of performing various Systems Manager operations, AWS Systems Manager Agent (SSM Agent) accesses a number of Amazon Simple Storage Service (Amazon S3) buckets.
SSM Agent communications with AWS managed S3 buckets
From the above, create an IAM role that meets the requirements.
You can specify the AWS management policy in the ManagedPolicyArns property. As we saw earlier, specify “AmazonSSMManagedInstanceCore”.
Next, define the inline policies in the Policies property: the first policy allows you to retrieve objects for the S3 buckets that you access to run SSM. The second policy allows you to retrieve objects for the buckets that store the execution logs of Patch Manager. The second policy allows storing objects against the bucket that stores the execution log of Patch Manager.
OS, patch groups, and approval rules are key to creating custom baselines
Check the custom baseline.
Resources:
MyPatchBaseline:
Type: AWS::SSM::PatchBaseline
Properties:
ApprovalRules:
PatchRules:
- ApproveAfterDays: 7
ComplianceLevel: UNSPECIFIED
EnableNonSecurity: false
PatchFilterGroup:
PatchFilters:
- Key: CLASSIFICATION
Values:
- Security
- Key: SEVERITY
Values:
- Critical
#- Important
#- ApproveAfterDays: 7
# ComplianceLevel: UNSPECIFIED
# EnableNonSecurity: false
# PatchFilterGroup:
# PatchFilters:
# - Key: CLASSIFICATION
# Values:
# - Bugfix
Description: Test Patch Baseline.
Name: !Sub ${Prefix}-MyPatchBaseline
OperatingSystem: AMAZON_LINUX_2
PatchGroups:
- !Sub ${Prefix}-patch-group
Code language: YAML (yaml)There are various parameters to create a custom baseline, but there are three important ones.
The first one is the target OS. Set the OS in the OperatingSystem property. In this case, we will target an EC2 instance running on Amazon Linux 2, so specify “AMAZON_LINUX_2”.
The second is the patch group. Specify the managed instances to be patched in the patch baseline in the form of patch groups. Set the patch group in the PatchGroups property. This time, specify the same value as the tag set for Isntance2.
The third is the approval rule. Set the conditions for the patches to be delivered, the grace time for automatic approval, and so on. This time, we will create a custom patch baseline by referring to the default patch baseline AWS-AmazonLinux2DefaultPatchBaseline and changing some of the parameters.
The following image details the AWS-AmazonLinux2DefaultPatchBaseline.
In this case, we will create a custom patch baseline for only those patches that have a classification of Security and a severity of Critical.
The approval rules are defined in the ApprovalRules property. You can specify the number of days until the patches corresponding to this rule are automatically approved by using the ApproveAfterDays property.
The ComplianceLevel property allows you to specify the level to be assigned to the patches that correspond to this rule.
In the EnableNonSecurity property, you can specify whether to distribute non-security patches as well.
The PatchFilterGroup property is used to configure the classification and severity settings. You can specify the classification and importance in the PatchFilterGroup property of the same property. To define the classification, specify “CLASSIFICATION” for the Key property, and to specify the severity, specify “SEVERITY”. In this case, we will specify “Security” for the classification and “Critical” for the severity in the Values property.
Note that if you uncomment the settings, the configuration will be the same as AWS-AmazonLinux2DefaultPatchBaseline.
Running Patch Baseline
For the two instances, create an association with the SSM document and run the patch baseline.
Resources:
AWSRunPatchBaselineAssociation:
Type: AWS::SSM::Association
Properties:
AssociationName: !Sub "${Prefix}-AWSRunPatchBaselineAssociation"
Name: AWS-RunPatchBaseline
OutputLocation:
S3Location:
OutputS3BucketName: !Ref S3BucketName
OutputS3KeyPrefix: !Sub "AWSRunPatchBaselineAssociation"
Parameters:
Operation:
- Install
Targets:
- Key: InstanceIds
Values:
- !Ref Instance1
- !Ref Instance2
WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)In order to run the patch baseline, create an association with the SSM document AWS-RunPatchBaseline.
In the OutputLocation property, specify the bucket where the Patch Manager runtime logs are saved.
In the Paramaters, specify the parameters for running AWS-RunPatchBaseline. Specify Install in the Operation property, which is a required parameter. For more information, see About the AWS-RunPatchBaseline SSM document.
Specify the targets to which you want to associate the SSM document in the Targets property. In this case, we will specify the IDs of the two instances mentioned above.
Let’s organize the behavior when AWS-RunPatchBaseline is executed with this configuration.
- Instance1: No tag (patch group) setting -> The default AWS-AmazonLinux2DefaultPatchBaseline will be executed.
- Instance2: With tag (patch group) setting -> Custom patch group (MyPatchBaseline) will be executed.
Architecting
Using CloudFormation, we will build this environment and check its actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stack.
For more information on creating stacks and how to check each stack, please refer to the following page.
After checking the resources of each stack, the information of the main resources created this time is as follows
- Name of the bucket for log storage: soa-04-001
- ID of Instance1: i-07d21ade1467247e6
- ID of Instance2: i-09327871f0830e91e
- Custom patch baseline name: soa-04-001-MyPatchBaseline
- ID of custom patch baseline: pb-0b520c1268ad72b19
- Patch group for the custom patch baseline: soa-04-001-patch-group
Verify the custom patch baseline that has been created.
In the Patch groups section, we can also see that the target of this baseline is “soa-04-001-patch-group”.
Next, let’s check the tags assigned to the instances.
Only Instance2 has a Patch Group tag. You can see that the value of this tag is the same name as the patch group in the custom patch baseline that we just checked. This means that Instance2 belongs to the patch group.
Check results of Patch Manager execution
Now that we are ready, let’s check the results of the Patch Manager run, which can be viewed on the SSM Run Command page.
We can see that the Patch Manager has been successfully executed for two instances.
Next, let’s check the execution results for each instance. First, let’s check Instance1.
You can see that the default AWS-AmazonLinux2DefaultPatchBaseline has been run and the 65 patches have been applied.
Next, we will check Instance2.
You can see that the custom patch baseline (soa-04-001-MyPatchBaseline) has been executed and the 9 patches have been applied.
Finally, check the execution log. The logs will be placed in the specified bucket for each instance ID.
The above image shows the log of Instance1, but the execution log is placed as stdout.txt.
Let’s check the contents of the Instance1 execution log.
/usr/bin/python3
/usr/bin/python2.7
/usr/bin/python2
/usr/bin/python
/usr/bin/yum
Using Yum version: 3.4.3
Using python binary: 'python2.7'
Using Python Version: Python 2.7.18
...
01/21/2022 23:26:08 root [INFO]: [{'OperationStartTime': '2022-01-21T23:25:15Z', 'BaselineId': u'pb-00fda5699d1ae3942', 'InstalledPendingRebootCount': '0', 'FailedCount': '0', 'MissingCount': '0', 'NotApplicableCount': '825', 'RebootOption': u'RebootIfNeeded', 'OperationEndTime': '2022-01-21T23:26:08Z', 'SecurityNonCompliantCount': '0', 'PatchGroup': u'', 'ExecutionId': '4fbe03b6-0e37-4cbf-995a-0f13a99f952a', 'InstalledRejectedCount': '0', 'OtherNonCompliantCount': '0', 'InstalledOtherCount': '377', 'CriticalNonCompliantCount': '0', 'SnapshotId': u'5839c3cf-0024-4a17-bdb2-302c3606223a', 'InstalledCount': '65', 'OperationType': u'Install'}]
...
Code language: plaintext (plaintext)You can check the patching process.
Summary
Using SSM Patch Manager, we confirmed that we can deliver patches to EC2 instances (Linux).
By creating and running a custom patch baseline in addition to the default AWS-AmazonLinux2DefaultPatchBaseline, we confirmed how to check the patch application results and the log output status.