Automate OpenSearch Serverless indexing using CFN custom resources
The following pages discuss OpenSearch Serverless.
In the above page, the index was created by manually executing a Lambda function.
On this page, we will consider how to automatically create an index.
Specifically, how to use CloudFormation custom resources.
Environment
Two main resources are created using CloudFormation.
The first is the OpenSearch Serverless collection.
The second is a Lambda function.
The runtime environment for the function is Python 3.8.
To associate this function with a CloudFormation custom resource and set this function to run when the stack is created.
The function’s action is to index into OpenSearch Serverless.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/148
Explanation of key points of template files
Lambda Function
Resources:
CustomResource2:
Type: Custom::CustomResource
DependsOn:
- Function2
Properties:
ServiceToken: !GetAtt Function2.Arn
Code language: YAML (yaml)The CloudFormation custom resource.
Specifies the Lambda function described below, which can be executed during CloudFormation stack operations.
For more information on CloudFormation custom resources, please see the following pages
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
COLLECTION_ENDPOINT: !Sub "${Collection}.${AWS::Region}.aoss.amazonaws.com"
OPENSEARCH_INDEX_NAME: !Ref OpenSearchIndexName
REGION: !Ref AWS::Region
Code:
ZipFile: |
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3
import cfnresponse
import os
host = os.environ['COLLECTION_ENDPOINT']
opensearch_index_name = os.environ['OPENSEARCH_INDEX_NAME']
region = os.environ['REGION']
service = 'aoss'
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region, service)
CREATE = 'Create'
response_data = {}
client = OpenSearch(
hosts=[{'host': host, 'port': 443}],
http_auth=auth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection,
pool_maxsize=20,
)
def lambda_handler(event, context):
try:
if event['RequestType'] == CREATE:
create_response = client.indices.create(
opensearch_index_name
)
print(create_response)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
FunctionName: !Sub "${Prefix}-function-02"
Handler: !Ref Handler
Layers:
- !Ref LambdaLayer
Runtime: !Ref Runtime
Role: !Ref FunctionRoleArn2
Timeout: 60
Code language: YAML (yaml)Lambda functions to be associated with CloudFormation custom resources.
Use if statements to set up the subsequent code to be executed when the CloudFormation stack is created.
Create a client object for OpenSearch and execute the object’s indices.create method to create the index.
The name of the index to create is “python-test-index”.
This name string is passed to the function as an environment variable.
To access OpenSearch Serverless from Python, use the package (opensearch-py) provided by AWS official.
This package is not included in the default Lambda function runtime environment.
This time we will create a Lambda layer and include it so that we can use this package.
For more information, please refer to the page mentioned at the beginning of this document.
Automatically create this Lambda layer using CloudFormation custom resources.
For more information, please see the following page.
Resources:
FunctionRole2:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: FunctionRole2Policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- aoss:APIAccessAll
Resource:
- !Sub "arn:aws:aoss:${AWS::Region}:${AWS::AccountId}:collection/*"
RoleName: !Sub "${Prefix}-FunctionRole2"
Code language: YAML (yaml)IAM role for Lambda function.
The contents allow all API access to OpenSearch Serverless.
(Reference) IAM User
Resources:
User:
Type: AWS::IAM::User
Properties:
LoginProfile:
Password: !Ref Password
Policies:
- PolicyName: AllAllowPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- aoss:*
Resource: "*"
UserName: !Sub "${Prefix}-user"
Code language: YAML (yaml)Dedicated user to manipulate the OpenSearch Serverless dashboard.
Inline policy to allow all actions of OpenSearch Serverless systems.
(Reference) OpenSearch Serverless
Resources:
Collection:
Type: AWS::OpenSearchServerless::Collection
DependsOn:
- EncryptionSecurityPolicy
Properties:
Name: !Ref CollectionName
StandbyReplicas: DISABLED
Type: TIMESERIES
DataAccessPolicy1:
Type: AWS::OpenSearchServerless::AccessPolicy
Properties:
Name: !Sub "${Prefix}-data-policy-01"
Policy: !Sub >-
[{"Description":"Access for cfn user","Rules":[{"ResourceType":"index","Resource":["index/*/*"],"Permission":["aoss:*"]},
{"ResourceType":"collection","Resource":["collection/${CollectionName}"],"Permission":["aoss:*"]}],
"Principal":["${UserArn}"]}]
Type: data
DataAccessPolicy2:
Type: AWS::OpenSearchServerless::AccessPolicy
Properties:
Name: !Sub "${Prefix}-data-policy-02"
Policy: !Sub >-
[{"Description":"Access for Function2","Rules":[{"ResourceType":"index","Resource":["index/${CollectionName}/${OpenSearchIndexName}"],"Permission":["aoss:CreateIndex"]}],
"Principal":["${FunctionRoleArn2}"]}]
Type: data
NetworkSecurityPolicy:
Type: AWS::OpenSearchServerless::SecurityPolicy
Properties:
Name: !Sub "${Prefix}-network-policy"
Policy: !Sub >-
[{"Rules":[{"ResourceType":"collection","Resource":["collection/${CollectionName}"]},
{"ResourceType":"dashboard","Resource":["collection/${CollectionName}"]}],"AllowFromPublic":true}]
Type: network
EncryptionSecurityPolicy:
Type: AWS::OpenSearchServerless::SecurityPolicy
Properties:
Name: !Sub "${Prefix}-encryption-policy"
Policy: !Sub >-
{"Rules":[{"ResourceType":"collection","Resource":["collection/${CollectionName}"]}],"AWSOwnedKey":true}
Type: encryption
Code language: YAML (yaml)Create OpenSearch Serverless and various policies.
For basic information on OpenSearch Serverless, please refer to the page mentioned at the beginning of this document.
The key point is the data access policy.
Create two.The key point is the data access policy.
Create two.
The first is a policy for dedicated IAM user.
The content allows all actions for the user.
The second is a policy for Lambda function.
This content allows the action (aoss:CreateIndex) required for index creation.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
This time, we will create an IAM resource (e.g., IAM user) to be named, so set the options as follows
$ aws cloudformation create-stack \
--stack-name [stack-name] \
--template-url https://[bucket-name].s3.ap-northeast-1.amazonaws.com/[folder-name]/fa-148.yaml \
--capabilities CAPABILITY_NAMED_IAM
Code language: Bash (bash)Check the OpenSearch Serverless collection from the AWS Management Console.
The OpenSearch Serverless collection has been successfully created.
Check the data access policy.
The first is for IAM users and the second is for Lambda function.
Actions are allowed for each use.
Check the Lambda function.
The Lambda function has been successfully created.
If we look at the Lambda layer, we can see that a layer has been created and is associated with this function.
This means that the Lambda layer was automatically created by the CloudFormation custom resource.
Operation Check
Now that we are ready, we access CloudWatch Logs to check the status of the Lambda function.
This Function is automatically executed.
This means that this function was automatically executed by the CloudFormation custom resource.
Log in to the OpenSearch Serverless dashboard with a dedicated IAM user.
Check the index from the dashboard.
We can see that “python-test-index” has indeed been created.
This shows that indeed a Lambda function was executed by the CloudFormation custom resource to create an index in OpenSearch Serverless.
Summary
We showed you how to use CloudFormation custom resources to automatically create indexes in OpenSearch Serverless.