Introduction to AWS IoT Core using CloudFormation
This page deals with AWS IoT Core.
AWS IoT Core services connect IoT devices to AWS IoT services and other AWS services. AWS IoT Core includes the device gateway and the message broker, which connect and process messages between your IoT devices and the cloud.
Getting started with AWS IoT Core
Since this is an introduction to AWS IoT Core, we will proceed with the following official tutorial.
https://docs.aws.amazon.com/iot/latest/developerguide/creating-a-virtual-thing.html
https://docs.aws.amazon.com/iot/latest/developerguide/using-laptop-as-device.html
Specifically, we will see how to use an EC2 instance as an IoT device to send MQTT messages to AWS IoT Core.
AWS IoT Core is one of the most central services among the various IoT-related services offered by AWS, and a better understanding of AWS IoT Core will help you build IoT applications smoothly.
Environment
Create a public subnet within the VPC.
Place an EC2 instance in the same subnet.
Look at this instance as an IoT device (Thing).
This instance is Amazon Linux 2023.
Connect to AWS IoT Core from this device and send MQTT messages.
Use the AWS IoT Device SDK for Python to connect to AWS IoT Core and send messages.
Scenarios
This page confirms the following
- Thing objects
- AWS IoT Policy
- Client Certificate
- Connect to AWS IoT Core using AWS IoT Device SDK
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/151
Explanation of key points of template files
AWS IoT Thing means IoT Device
An object of things means a device to be managed by AWS IoT.
Devices connected to AWS IoT Core are represented by thing objects in the AWS IoT registry. A thing object represents a specific device or logical entity. It can be a physical device or sensor (for example, a light bulb or a light switch on the wall).
Create a thing object
Resources:
Thing:
Type: AWS::IoT::Thing
Properties:
ThingName: !Ref Prefix
Code language: YAML (yaml)In this case, we will consider an EC2 instance as an IoT device, so the object of this thing will mean an instance.
This time we will simply set only the object name of the thing.
The name is “fa-151”.
AWS IoT policies control the permissions of things
AWS IoT policies are tied to the certificates used by the device and define the actions that the device can perform for AWS IoT.
Devices use an X.509 certificate to authenticate with AWS IoT Core. The certificate has AWS IoT policies attached to it. These policies determine which AWS IoT operations, such as subscribing or publishing to MQTT topics, the device is permitted to perform. Your device presents its certificate when it connects and sends messages to AWS IoT Core.
Create an AWS IoT policy
Resources:
Policy:
Type: AWS::IoT::Policy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- iot:Publish
- iot:Receive
Resource:
- !Sub "arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/${TopicName}"
- Effect: Allow
Action:
- iot:Subscribe
Resource:
- !Sub "arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/${TopicName}"
- Effect: Allow
Action:
- iot:Connect
Resource:
- !Sub "arn:aws:iot:${AWS::Region}:${AWS::AccountId}:client/${Thing}"
PolicyName: !Ref Prefix
Code language: YAML (yaml)As you can see, AWS IoT policies are similar to IAM policies.
In this case, since this is a verification, we created it based on what is described in the following official tutorial.
The name of this policy is “fa-151”.
Set up an EC2 instance as an IoT device
As mentioned earlier, the role of the EC2 instance in this case is an IoT device.
User data is used to configure the various settings required on the IoT device side.
For basic information on user data, please refer to the following pages.
Resources:
Instance:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile: !Ref InstanceProfile
ImageId: !Ref ImageId
InstanceType: !Ref InstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: true
DeviceIndex: 0
GroupSet:
- !Ref InstanceSecurityGroup
SubnetId: !Ref PublicSubnet1
UserData: !Base64
Fn::Sub: |
#!/bin/bash -xe
dnf update -y
dnf install python3.11-pip -y
dnf install -y git
mkdir ~/certs
curl -o ~/certs/Amazon-root-CA-1.pem \
https://www.amazontrust.com/repository/AmazonRootCA1.pem
aws iot create-keys-and-certificate \
--set-as-active \
--certificate-pem-outfile "~/certs/device.pem.crt" \
--public-key-outfile "~/certs/public.pem.key" \
--private-key-outfile "~/certs/private.pem.key" > ~/output.txt
certificate_arn=`cat ~/output.txt | jq -r .certificateArn`
aws iot attach-thing-principal \
--thing-name "${Thing}" \
--principal "$certificate_arn"
aws iot attach-policy \
--policy-name "${Policy}" \
--target "$certificate_arn"
aws iot describe-endpoint \
--endpoint-type iot:Data-ATS > ~/endpoint.txt
endpoint=`cat ~/endpoint.txt| jq -r .endpointAddress`
python3.11 -m pip install awsiotsdk
cd ~ && git clone https://github.com/aws/aws-iot-device-sdk-python-v2.git
cd ~/aws-iot-device-sdk-python-v2/samples && python3.11 pubsub.py \
--endpoint "$endpoint" \
--ca_file ~/certs/Amazon-root-CA-1.pem \
--cert ~/certs/device.pem.crt \
--key ~/certs/private.pem.key \
--client_id "${Thing}" \
--topic "${TopicName}" \
--count 5
Code language: YAML (yaml)We will focus on the UserData property.
First, after updating dnf, use dnf to install Python 3.11 and Git.
These are used to get and run the AWS IoT Device SDK for Python.
Then use the curl command to obtain the Amazon CA certificate (AmazonRootCA1.pem).
This certificate is used for server proof to verify that the destination is correct when sending MQTT messages from the IoT device to AWS IoT Core.
Create the certificate with the aws iot create-keys-and-certificate command.
Generate the client certificate (device.pem.crt) that the thing will use to communicate with the AWS IoT Core.
The certificate is generated on the EC2 instance and also registered on the AWS IoT side.
The command will also retrieve the public key (public.pem.key) and private key (private.pem.key) used to create the certificate.
The private key in particular is used to send MQTT messages to AWS IoT Core.
Save the result of this command execution to a text file (output.txt).
This is to retrieve the ARN assigned to the certificate when it is registered with AWS IoT.
Use the jq command to retrieve this and store it in a variable.
By the way, the execution result of this command is returned in the following format.
{
"certificateArn": "arn:aws:iot:[region]:[account-id]:cert/[cert-id]",
"certificateId": "[cert-id]",
"certificatePem": "-----BEGIN CERTIFICATE-----\n[cert-pem]\n-----END CERTIFICATE-----\n",
"keyPair": {
"PublicKey": "-----BEGIN PUBLIC KEY-----\n[public-key]\n-----END PUBLIC KEY-----\n",
"PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n[private-key]\n-----END RSA PRIVATE KEY-----\n"
}
}
Code language: JSON / JSON with Comments (json)Execute the aws iot attach-thing-principal command to associate the thing with the certificate.
Specify the resource name of the aforementioned thing and the ARN of the certificate, respectively.
Execute the aws iot attach-policy command to associate the AWS IoT policy with the certificate.
Note that the object to be associated with the AWS IoT policy is the certificate, not the thing.
Specify the aforementioned policy name and the ARN of the certificate.
Run the aws iot describe-endpoint command to get the endpoints to which MQTT messages are sent.
There are two types of endpoints as follows.
Every customer has an iot:Data-ATS and an iot:Data endpoint. Each endpoint uses an X.509 certificate to authenticate the client. We strongly recommend that customers use the newer iot:Data-ATS endpoint type to avoid issues related to the widespread distrust of Symantec certificate authorities.
Connecting devices to AWS IoT
Specify “iot:Data-ATS” for the endpoint-type option according to the above.
Save the result of this command execution to a text file (endpoint.txt) and store the endpoint name in a variable.
Again, use the jq command to extract the endpoint names from the following JSON-formatted data.
{
"endpointAddress": "[endpoint-id]-ats.iot.[region].amazonaws.com"
}
Code language: JSON / JSON with Comments (json)Install the AWS IoT Device SDK for Python (awsiotsdk) using pip.
And use Git to also get the sample script for this SDK.
Finally, run the sample script pubsub.py.
This sample script sends a test message (Hello World!) to AWS IoT Core.
To run this script, we pass the endpoints we have obtained so far, the Amazon CA certificate, the client certificate, and the private key for the client certificate.
In addition, two parameters are specified.
The first is the client ID.
This is explained by the AWS official as follows
A typical device use case involves the use of the thing name as the default MQTT client ID. Although we don’t enforce a mapping between a thing’s registry name and its use of MQTT client IDs, certificates, or shadow state, we recommend you choose a thing name and use it as the MQTT client ID for both the registry and the Device Shadow service.
Managing devices with AWS IoT
Specify the name of the thing for this parameter according to the above.
The second is the topic name.
This is explained by the AWS official as follows
MQTT topics identify AWS IoT messages. AWS IoT clients identify the messages they publish by giving the messages topic names. Clients identify the messages to which they want to subscribe (receive) by registering a topic filter with AWS IoT Core. The message broker uses topic names and topic filters to route messages from publishing clients to subscribing clients.
MQTT topics
For this one, specify “test/topic”.
IAM roles for EC2 instance
Resources:
InstanceRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- ec2.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Policies:
- PolicyName: InstancePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- iot:AttachPolicy
- iot:AttachThingPrincipal
- iot:CreateKeysAndCertificate
- iot:DescribeEndpoint
Resource:
- "*"
Code language: YAML (yaml)As we have just confirmed, this EC2 instance is intended to operate as an IoT device.
To this end, we have described the necessary processing in the definition of the initialization process with user data.
This IAM role grants the instance the necessary permissions to perform those processes.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check resources
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
Check the various resources of AWS IoT.
Check AWS IoT Thing.
Certainly things are created.
Check the AWS IoT policy.
The necessary permissions are granted to run the sample script.
The topic name is “test/topic” and the client ID is “fa-151”.
These are the values that the EC2 instance is supposed to specify when executing the sample script.
Check the certificate.
The certificate has indeed been created.
This means that the aws iot create-keys-and-certificate command was successfully executed on the EC2 instance.
Also, if we look at the resources associated with this certificate, we can see the policy and thing we just checked.
This means that the aws iot attach-thing-principal and aws iot attach-policy commands were successfully executed on the same instance.
Check the endpoint name.
This is the AWS IoT endpoint assigned to my account.
The EC2 instance will send MQTT messages here.
Operation Check
Now that we are ready, we access the EC2 instance (i-045ca338a33b06229).
The SSM Session Manager is used to access the instance.
% aws ssm start-session --target i-045ca338a33b06229
...
sh-5.2$
Code language: Bash (bash)For more information on SSM Session Manager, please refer to the following page.
Check the execution status of processes defined in user data.
Use the tree command to check the contents of the /root directory.
sh-5.2$ sudo tree /root
/root
├── aws-iot-device-sdk-python-v2
│ ├── ...
│ ├── samples
│ │ │ ├── ...
│ │ │ ├── pubsub.py
│ │ │ └── ...
│ └── ...
├── certs
│ ├── Amazon-root-CA-1.pem
│ ├── device.pem.crt
│ ├── private.pem.key
│ └── public.pem.key
├── endpoint.txt
└── output.txt
Code language: Bash (bash)Indeed, the process defined in the user data is successfully executed and various files are located.
Run the journalctl command to check the instance logs.
sh-5.2$ sudo journalctl --no-pager
...
Feb 24 06:15:19 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: + python3.11 pubsub.py --endpoint a2oxckhng7gmur-ats.iot.ap-northeast-1.amazonaws.com --ca_file /root/certs/Amazon-root-CA-1.pem --cert /root/certs/device.pem.crt --key /root/certs/private.pem.key --client_id fa-151 --topic test/topic --count 5
...
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Connecting to a2oxckhng7gmur-ats.iot.ap-northeast-1.amazonaws.com with client ID 'fa-151'...
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Connected!
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Subscribing to topic 'test/topic'...
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Connection Successful with return code: 0 session present: False
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Subscribed with 1
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Sending 5 message(s)
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Publishing message to topic 'test/topic': Hello World! [1]
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Received message from topic 'test/topic': b'"Hello World! [1]"'
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Publishing message to topic 'test/topic': Hello World! [2]
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Received message from topic 'test/topic': b'"Hello World! [2]"'
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Publishing message to topic 'test/topic': Hello World! [3]
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Received message from topic 'test/topic': b'"Hello World! [3]"'
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Publishing message to topic 'test/topic': Hello World! [4]
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Received message from topic 'test/topic': b'"Hello World! [4]"'
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Publishing message to topic 'test/topic': Hello World! [5]
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Received message from topic 'test/topic': b'"Hello World! [5]"'
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: 5 message(s) received.
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Disconnecting...
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Connection closed
Feb 24 06:15:25 ip-10-0-1-45.ap-northeast-1.compute.internal cloud-init[1672]: Disconnected!
...
Code language: Bash (bash)We have extracted the part of the script that sends MQTT messages to the AWS IoT endpoint.
We can see that the sample script is executed to connect to AWS IoT and send 5 messages.
Finally, check the messages sent from the EC2 instance on the AWS IoT side.
To confirm, use the AWS IoT MQTT client in the AWS Management Console.
In the Subscribe to a topic tab, enter the topic name in the Topic filter and press Subscribe.
Since the EC2 instance sends messages to “test/topic”, specify this.
Message displayed.
It was indeed sent by the EC2 instance just now.
Detach the certificates before deleting the objects and AWS IoT policies!
After confirmation, you will probably want to delete the CloudFormation stacks, but there is a caveat.
That is, you need to unattach the Things and AW IoT policy resources from the certificate before deleting them.
Press “Detach” on both pages.
Please note that if you try to delete both resources before this is done, the deletion will fail.
Summary
We have identified how to use an EC2 instance as an IoT device to send MQTT messages to AWS IoT Core.
AWS IoT Core is one of the most central services among the various IoT-related services offered by AWS.
Better understanding of AWS IoT Core will help you build IoT apps smoothly.