MENU
Architecting AWS with CloudFormation
  1. Home
  2. AWS_EN
  3. Install CloudWatch Agent on Linux and collect data

Install CloudWatch Agent on Linux and collect data

TOC

Install CloudWatch Agent on Linux instance and configure to collect logs and metrics

The CloudWatch Agent can be used to collect logs and metrics.

Collect internal system-level metrics from Amazon EC2 instances across operating systems…

Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.

Collecting metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent

This time, we will target a Linux instance (Amazon Linux 2).
For the procedure to install CloudWatch Agent on a Windows instance, please refer to the following page.

あわせて読みたい
Install CloudWatch Agent on Windows and collect data 【Install CloudWatch Agent on Windows instance and configure to collect logs and metrics】 The CloudWatch Agent can be used to collect logs and metrics. Coll...

Environment

Diagram of install CloudWatch Agent and collect data.

In this configuration, the CloudWatch agent will be installed on instances in a private subnet to collect data. To do so, we will access three different services (S3, System Manager, CloudWatch) via VPC endpoints.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL

https://github.com/awstut-an-r/awstut-fa/tree/main/015

Template file points

Authorization to install and use CloudWatch agent

Check the permissions to be granted to the EC2 instance.

Resources:
  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - ec2.amazonaws.com
      Policies:
        - PolicyName: AllowAccessToYumRepositoryInS3AndUseCloudWatchAgentPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource:
                  - !Sub "arn:aws:s3:::amazonlinux.${AWS::Region}.amazonaws.com/*"
                  - !Sub "arn:aws:s3:::amazonlinux-2-repos-${AWS::Region}/*"
              - Effect: Allow
                Action:
                  - cloudwatch:PutMetricData
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: "*"
        - PolicyName: DeliverLogPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:PutObjectAcl
                  - s3:ListBucket
                Resource:
                  - !Sub "arn:aws:s3:::${SSMLogBucket}"
                  - !Sub "arn:aws:s3:::${SSMLogBucket}/*"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicyCode language: YAML (yaml)

Regarding the permissions required to install the CloudWatch Agent using SSM, the AWS official mentions the following

In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.

To use Systems Manager to install or configure the CloudWatch agent, select the box next to AmazonSSMManagedInstanceCore.

Create IAM roles and users for use with the CloudWatch agent

As described above, specify two AWS managed policies in the ManagedPolicyArns property.

Since the instance to be created in this configuration is Amazon Linux 2, it is possible to run yum against the repository built in the S3 bucket. Please refer to the following page for details.

あわせて読みたい
yum/dnf on private subnet instances 【Configuration for running yum/dnf on instance in private subnet】 We will check how to run yum/dnf on an instance in a private subnet. In this case, the fo...

Grant three permissions (cloudwatch:PutMetricData, logs:CreateLogStream, logs:PutLogEvents) to the instance to deliver metrics and logs to CloudWatch via the CloudWatch agent The following is an example.

The execution of SSM documentation, which will be described later, can output the execution log to an S3 bucket. In order to use this in this case, access to the target bucket is permitted.

Register CloudWatch agent configurations in SSM Parameter Store

Define the configurations required to start the CloudWatch Agent in the SSM Parameter Store.

Resources:
  CloudWatchConfigParemeter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: AmazonCloudWatch-linux
      Type: String
      Value: !Sub |
        {
          "agent": {
            "metrics_collection_interval": 60,
            "run_as_user": "root"
          },
          "logs": {
            "logs_collected": {
              "files": {
                "collect_list": [
                  {
                    "file_path": "/var/log/messages",
                    "log_group_name": "${Prefix}",
                    "log_stream_name": "${Instance}"
                  }
                ]
              }
            }
          },
          "metrics": {
            "append_dimensions": {
              "ImageId": "${!aws:ImageId}",
              "InstanceId": "${!aws:InstanceId}",
              "InstanceType": "${!aws:InstanceType}"
            },
            "metrics_collected": {
              "cpu": {
                "measurement": [
                  "cpu_usage_idle",
                  "cpu_usage_iowait",
                  "cpu_usage_user",
                  "cpu_usage_system"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                  "*"
                ],
                "totalcpu": false
              },
              "disk": {
                "measurement": [
                  "used_percent",
                  "inodes_free"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                  "*"
                ]
              },
              "mem": {
                "measurement": [
                  "mem_used_percent"
                ],
                "metrics_collection_interval": 60
              }
            }
          }
        }
Code language: YAML (yaml)

The configuration for launching the CloudWatch Agent can be registered in the SSM Parameter Store, rather than locally on the instance where the Agent is installed.

If you plan to use the SSM Agent to install the CloudWatch agent on servers, after you manually edit the CloudWatch agent configuration file, you can upload it to Systems Manager Parameter Store.

Uploading the CloudWatch agent configuration file to Systems Manager Parameter Store

As for the contents of the configuration, see Manually create or edit the CloudWatch agent configuration file for more details, define the logs and metrics sections and configure them to collect logs and metrics for the instance. Specifically, configure the contents of /var/log/messages as logs and multiple items of CPU, disk, and memory as metrics to be collected.

Add instance IDs, etc. to the metrics to be retrieved, using notation such as “${aws:InstanceId}” in the append_dimentions option. Note that this notation is the same as the variable notation of CloudFormation’s built-in function Fn::Sub. In this case, we will use the same function to specify the log group name and stream name where the logs will be delivered. However, when the function is executed, the dimension values in the append_dimentions option mentioned above are also evaluated as the target and an error occurs. To avoid this, add “! to escape it.

To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. CloudFormation resolves this text as ${Literal}.

Fn::Sub

Install and start CloudWatch agent with SSM documentation

Run the two SSM documents to set up the CloudWatch agent.

AWS-ConfigureAWSPackage

First check the SSM documentation for installing the CloudWatch agent.

Resources:
  AWSConfigureAWSPackageAssociation:
    Type: AWS::SSM::Association
    Properties:
      AssociationName: !Sub "${Prefix}-aws-configure-aws-package-association"
      Name: AWS-ConfigureAWSPackage
      OutputLocation:
        S3Location:
          OutputS3BucketName: !Ref SSMLogBucket
          OutputS3KeyPrefix: !Sub "${Prefix}/aws-configure-aws-package-association"
      Parameters:
        action:
          - Install
        installationType:
          - Uninstall and reinstall
        name:
          - AmazonCloudWatchAgent
        version:
          - latest
      Targets:
        - Key: InstanceIds
          Values:
            - !Ref Instance
      WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

Execution of SSM documents can be accomplished by defining an AWS::SSM::Association resource.
Specify the SSM document to be executed in the Name property. The document that should be executed this time is defined as follows.

In the Command document list, choose AWS-ConfigureAWSPackage.

Download and configure the CloudWatch agent

In the S3Location property, specify the location for the log when the document is executed. In this case, we will specify the aforementioned bucket.
Specify the target instance to run the document in the Targets property. In this case, specify the instance ID.
Specify the parameters for executing the document in the Parameters property. The parameters to be passed can be confirmed by referring to the documentation. The following is a description of the AWS-ConfigureAWSPackage parameters.

{
  "parameters": {
    "action": {
      "description": "(Required) Specify whether or not to install or uninstall the package.",
      "type": "String",
      "allowedValues": [
        "Install",
        "Uninstall"
      ]
    },
    "installationType": {
      "description": "(Optional) Specify the type of installation. Uninstall and reinstall: The application is taken offline until the reinstallation process completes. In-place update: The application is available while new or updated files are added to the installation.",
      "type": "String",
      "allowedValues": [
        "Uninstall and reinstall",
        "In-place update"
      ],
      "default": "Uninstall and reinstall"
    },
    "name": {
      "description": "(Required) The package to install/uninstall.",
      "type": "String",
      "allowedPattern": "^arn:[a-z0-9][-.a-z0-9]{0,62}:[a-z0-9][-.a-z0-9]{0,62}:([a-z0-9][-.a-z0-9]{0,62})?:([a-z0-9][-.a-z0-9]{0,62})?:(package|document)\\/[a-zA-Z0-9/:.\\-_]{1,128}$|^[a-zA-Z0-9/:.\\-_]{1,128}$"
    },
    "version": {
      "description": "(Optional) The version of the package to install or uninstall. If you don't specify a version, the system installs the latest published version by default. The system will only attempt to uninstall the version that is currently installed. If no version of the package is installed, the system returns an error.",
      "type": "String",
      "default": ""
    },
    "additionalArguments": {
      "description": "(Optional) The additional parameters to provide to your install, uninstall, or update scripts.",
      "type": "StringMap",
      "displayType": "textarea",
      "default": {},
      "maxChars": 4096
    }
  }
}
Code language: JSON / JSON with Comments (json)

For the values that should be specified for each parameter, refer to Download and configure the CloudWatch agent.

AmazonCloudWatch-linux

Next, review the SSM documentation for starting the CloudWatch agent.

Resources:
  AmazonCloudWatchManageAgentAssociation:
    Type: AWS::SSM::Association
    DependsOn:
      - CloudWatchConfigParemeter
      - AWSConfigureAWSPackageAssociation
    Properties:
      AssociationName: !Sub "${Prefix}-amazon-cloudwatch-manage-agent"
      Name: AmazonCloudWatch-ManageAgent
      OutputLocation:
        S3Location:
          OutputS3BucketName: !Ref SSMLogBucket
          OutputS3KeyPrefix: !Sub "${Prefix}/amazon-cloudwatch-manage-agent"
      Parameters:
        action:
          - configure
        mode:
          - ec2
        optionalConfigurationSource:
          - ssm
        optionalConfigurationLocation:
          - AmazonCloudWatch-linux
        optionalRestart:
          - "yes"
      Targets:
        - Key: InstanceIds
          Values:
            - !Ref Instance
      WaitForSuccessTimeoutSeconds: !Ref WaitForSuccessTimeoutSeconds
Code language: YAML (yaml)

The overall notation is the same as before.
The document to run to launch the CloudWatch Agent is mentioned as follows

In the Command document list, choose AmazonCloudWatch-ManageAgent.

Start the CloudWatch agent

Follow the above and specify AmazonCloudWatch-ManageAgent in the Name property.
Specify the parameters to run this document in the Parameters property. The description of the parameters for AmazonCloudWatch-ManageAgent is quoted below.

{
  "parameters": {
    "action": {
      "description": "The action CloudWatch Agent should take.",
      "type": "String",
      "default": "configure",
      "allowedValues": [
        "configure",
        "configure (append)",
        "configure (remove)",
        "start",
        "status",
        "stop"
      ]
    },
    "mode": {
      "description": "Controls platform-specific default behavior such as whether to include EC2 Metadata in metrics.",
      "type": "String",
      "default": "ec2",
      "allowedValues": [
        "ec2",
        "onPremise",
        "auto"
      ]
    },
    "optionalConfigurationSource": {
      "description": "Only for 'configure' related actions. Use 'ssm' to apply a ssm parameter as config. Use 'default' to apply default config for amazon-cloudwatch-agent. Use 'all' with 'configure (remove)' to clean all configs for amazon-cloudwatch-agent.",
      "type": "String",
      "allowedValues": [
        "ssm",
        "default",
        "all"
      ],
      "default": "ssm"
    },
    "optionalConfigurationLocation": {
      "description": "Only for 'configure' related actions. Only needed when Optional Configuration Source is set to 'ssm'. The value should be a ssm parameter name.",
      "type": "String",
      "default": "",
      "allowedPattern": "[a-zA-Z0-9-\"~:_@./^(*)!<>?=+]*$"
    },
    "optionalOpenTelemetryCollectorConfigurationSource": {
      "description": "Only for 'configure' related actions. Use 'ssm' to apply a ssm parameter as config. Use 'default' to apply default config for amazon-cloudwatch-agent. Use 'all' with 'configure (remove)' to clean all configs for amazon-cloudwatch-agent. It does not support MacOS instance.",
      "type": "String",
      "allowedValues": [
        "ssm",
        "default",
        "all"
      ],
      "default": "ssm"
    },
    "optionalOpenTelemetryCollectorConfigurationLocation": {
      "description": "Only for 'configure' related actions. Only needed when Optional Configuration Source is set to 'ssm'. The value should be a ssm parameter name. It does not support MacOS instance.",
      "type": "String",
      "default": "",
      "allowedPattern": "[a-zA-Z0-9-\"~:_@./^(*)!<>?=+]*$"
    },
    "optionalRestart": {
      "description": "Only for 'configure' related actions. If 'yes', restarts the agent to use the new configuration. Otherwise the new config will only apply on the next agent restart.",
      "type": "String",
      "default": "yes",
      "allowedValues": [
        "yes",
        "no"
      ]
    }
  }
}
Code language: JSON / JSON with Comments (json)

Refer to Start the CloudWatch agent to set the values that should be specified for each parameter.

VPC Endpoints

In this configuration, the instances are located in a private subnet, so it is necessary to access various services via VPC endpoints.

Endpoint for yum execution

As mentioned earlier, Amazon Linux 2 allows yum to be executed against S3 buckets. To do so, create a VPC endpoint for S3 (com.amazonaws.${AWS::Region}.s3).

Endpoints for SSM

Create three endpoints.

  • com.amazonaws.${AWS::Region}.ssm
  • com.amazonaws.${AWS::Region}.ec2messages
  • com.amazonaws.${AWS::Region}.ssmmessages

The above two are required to run SSM documentation. In addition, a third endpoint must be created, this time using SSM Session Manager.

Endpoint for metrics delivery with CloudWatch agent

Create three endpoints.

  • com.amazonaws.${AWS::Region}.monitoring
  • com.amazonaws.${AWS::Region}.ec2
  • com.amazonaws.${AWS::Region}.logs

Architecting

We will use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.
For more information on how to create stacks and check each stack, please refer to the following page.

あわせて読みたい
CloudFormation’s nested stack 【How to build an environment with a nested CloudFormation stack】 Examine nested stacks in CloudFormation. CloudFormation allows you to nest stacks. Nested ...

After checking the resources for each stack, the information for the main resources created this time is as follows

  • S3 bucket name: fa-015
  • CloudWatch Logs log group name: fa-015
  • Instance: i-058d2a76a7fb6ac06

The AWS Management Console also checks the status of resource creation. First, check the execution status of the SSM document.

Two SSM documents were running.

Indeed, two commands were executed, indicating that they completed successfully.

Next, we access the S3 bucket and check the logs of the SSM Run Command execution.
First, here is the result of the execution of the CloudWatch Agent installation: the log is output to the S3 bucket.

Detail of SSM 2.

The contents of the file are as follows

Initiating arn:aws:ssm:::package/AmazonCloudWatchAgent 1.247350.0b251780 install
Plugin aws:runShellScript ResultStatus Success
install output: Running sh install.sh
create group cwagent, result: 0
create user cwagent, result: 0

Successfully installed arn:aws:ssm:::package/AmazonCloudWatchAgent 1.247350.0b251780
Code language: plaintext (plaintext)

You can see that it has been installed successfully.

Next is the result of running the CloudWatch Agent.

Detail of SSM 3.

The contents of the file are as follows

****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:AmazonCloudWatch-linux --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: ap-northeast-1
credsConfig: map[]
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-linux.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Valid Json input schema.
I! Detecting run_as_user...
No csm configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Code language: plaintext (plaintext)

You can see that it has been started successfully.

Access instance and check CloudWatch Agent is running

In the meantime, we will access the instance and check the activation status of CloudWatch Agent.
To access the instance, we will use SSM Session Manager.

% aws ssm start-session \
--target i-058d2a76a7fb6ac06

Starting session with SessionId: root-0e251c4db3512cf60
sh-4.2$
Code language: Bash (bash)

For more information on accessing instances using the SSM Session Manager, please refer to the following page.

あわせて読みたい
Accessing Linux instance via SSM Session Manager 【Configure Linux instances to be accessed via SSM Session Manager】 We will check a configuration in which an EC2 instance is accessed via SSM Session Manag...

Now that you have access, check the startup status.

sh-4.2$ sudo systemctl status amazon-cloudwatch-agent -l
● amazon-cloudwatch-agent.service - Amazon CloudWatch Agent
   Loaded: loaded (/etc/systemd/system/amazon-cloudwatch-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-03-05 06:32:47 UTC; 10min ago
 Main PID: 1344 (amazon-cloudwat)
   CGroup: /system.slice/amazon-cloudwatch-agent.service
           └─1344 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json -pidfile /opt/aws/amazon-cloudwatch-agent/var/amazon-cloudwatch-agent.pid

Mar 05 06:32:47 ip-10-0-1-55.ap-northeast-1.compute.internal systemd[1]: Started Amazon CloudWatch Agent.
Mar 05 06:32:47 ip-10-0-1-55.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1344]: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
Mar 05 06:32:47 ip-10-0-1-55.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1344]: Valid Json input schema.
Mar 05 06:32:47 ip-10-0-1-55.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1344]: I! Detecting run_as_user...
Code language: Bash (bash)

It says “active (running)”, which means it is working properly.

Check logs delivered by CloudWatch Agent

Check the data actually delivered. This time, since /var/log/messages is set to be delivered, we will check whether the data has really been delivered.
First, check the status of the stream in the log group.

A stream is created in CloudWatch Logs.

The log group name and stream name are as specified in the CloudWatch Agent configuration.
Next, check the logs delivered to the stream.

Logs are delivered to CloudWatch Logs.

You can see that /var/log/messages is being delivered normally.

Check metrics delivered by CloudWatch Agent

Finally, we will check the metrics.
In the AWS Management Console, access the following

CloudWatch > Metrics > All Metrics > CWAgent

Metrics are delivered to CloudWatch.

As shown above, the metrics have been delivered and the graph has been created.

Summary

We have confirmed the procedure to install CloudWatch Agent on Linux instances using SSM.

Verified that CloudWatch Agent can be used to deliver logs and metrics.

AWS_EN
CloudWatch Metrics_EN EC2(Linux)_EN S3_EN SSM Document_EN SSM Parameter Store_EN SSM Session Manager_EN
TOC