Configuration Getting Started with OpenSearch using CloudFormation
OpenSearch is a search and analysis suite forked from Elasticsearch.
In this introduction, we will build a basic OpenSearch environment using CloudFormation.
Environment
Create an OpenSearch domain.
Replicate the configuration described in the official AWS tutorial.
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsgcreate-domain.html
CloudFormation template files
Build the above configuration with CloudFormation.
The CloudFormation template is located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/042
Explanation of key points of template files
Template for OpenSearch domain
Resources:
Domain:
Type: AWS::OpenSearchService::Domain
Properties:
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: "*"
Action: es:*
Resource: !Sub "arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${DomainName}/*"
AdvancedSecurityOptions:
Enabled: true
InternalUserDatabaseEnabled: true
MasterUserOptions:
MasterUserName: !Ref MasterUserName
MasterUserPassword: !Ref MasterUserPassword
ClusterConfig:
DedicatedMasterEnabled: false
InstanceCount: !Ref InstanceCount
InstanceType: !Ref InstanceType
WarmEnabled: false
ZoneAwarenessEnabled: false
CognitoOptions:
Enabled: false
DomainEndpointOptions:
CustomEndpointEnabled: false
EnforceHTTPS: true
TLSSecurityPolicy: Policy-Min-TLS-1-0-2019-07
DomainName: !Ref DomainName
EBSOptions:
EBSEnabled: true
VolumeSize: !Ref VolumeSize
VolumeType: gp2
EncryptionAtRestOptions:
Enabled: true
KmsKeyId: !Ref Key
EngineVersion: !Ref EngineVersion
NodeToNodeEncryptionOptions:
Enabled: true
Code language: YAML (yaml)Check the parameters.
The AccessPolicies property allows you to set restrictions on access to the OpenSearch domain.
Restrictions are set in the form of IAM policies.
In this case, we will create a master user and use the user information for authentication.
Therefore, this property will allow all operations on the resource to be created.
The AdvancedSecurityOptions property allows you to set fine-grained access control for security.
The tutorial shows how to create a master user in OpenSearch and use that user’s information for authentication.
This property is set in a similar manner.
Specifically, activate the Internal User Database and set the user name and password for the master user to be created.
The ClusterConfig property allows you to configure settings related to the specifications of the OpenSearch domain to be created.
You can set the type and number of instances that make up the OpenSearch infrastructure, and whether or not there is a dedicated master node.
Basically, configure as described in the tutorial, but this time the number of instances is set to one.
The DomainEndpointOptions property allows you to configure settings related to the OpenSearch domain to be created.
If you want to enable authentication, or fine-grained access control, using master user information, as in this case, HTTPS for the domain is a prerequisite.
Fine-grained access control requires OpenSearch or Elasticsearch 6.7 or later. It also requires HTTPS for all traffic to the domain, Encryption of data at rest, and node-to-node encryption.
Enabling fine-grained access control
Fine-grained access control requires OpenSearch or Elasticsearch 6.7 or later. It also requires HTTPS for all traffic to the domain, encryption of data in storage, and encryption between nodes.
https://docs.aws.amazon.com/ja_jp/opensearch-service/latest/developerguide/fgac.html#fgac-enabling
Enabling Granular Access Control
Configure the settings in a manner similar to the above requirements.
Specifically, enforce HTTPS communication with this property and enable encryption with the EncryptionAtRestOptions and NoDeToNodeEncryptionOptions properties.
Custom endpoints are not enabled according to the tutorial policy.
Set the name of the OpenSearch domain to be created in the DomainName property.
The EBSOptions property allows you to configure the storage settings used by the instance described above.
Follow the tutorial to set up 10GB of gp2 type EBS.
Set the version of OpenSearch to be created in the EngineVersion property.
This time, we set the version to “OpenSearch_1.2,” which is the latest version.
Set the KMS key used for encryption as follows.
Resources:
Key:
Type: AWS::KMS::Key
Properties:
Enabled: true
KeyPolicy:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: "*"
Action:
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:GenerateDataKey*
- kms:CreateGrant
- kms:DescribeKey
Resource: "*"
Condition:
StringEquals:
kms:ViaService: !Sub "es.${AWS::Region}.amazonaws.com"
kms:CallerAccount: !Ref AWS::AccountId
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
Action: "*"
Resource: "*"
- Effect: Allow
Principal:
Service: es.amazonaws.com
Action:
- kms:Describe*
- kms:Get*
- kms:List*
Resource: "*"
Code language: YAML (yaml)Created with reference to the AWS managed key used by default for OpenSearch.
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stack.
For information on how to create stacks and check each stack, please refer to the following page
The following is information on the main resources created in this case
- OpenSearch domain name: fa-042
- OpenSearch dashboard URL: https://search-fa-042-m2yc6j6kr63hqyhvqp6jwfim7a.ap-northeast-1.es.amazonaws.com/_dashboards
- OpenSearch domain endpoint URL: https://search-fa-042-m2yc6j6kr63hqyhvqp6jwfim7a.ap-northeast-1.es.amazonaws.com
- Master user name: test
- Master user password: p@ssw0rd
Check OpenSearch from the AWS Management Console as well.
It has been created as described in the CloudFormation template file.
Confirmation of operation
Follow the tutorial.
First, upload the sample data.
https://docs.aws.amazon.com/ja_jp/opensearch-service/latest/developerguide/gsgupload-data.html
Using the data from the tutorial as is, upload the data using the curl command.
The following command is used for the configuration we have built this time.
$ curl -XPUT -u 'test:P@ssw0rd' 'https://search-fa-042-m2yc6j6kr63hqyhvqp6jwfim7a.ap-northeast-1.es.amazonaws.com/fa-042/_doc/1' -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json'
{"_index":"fa-042","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
Code language: Bash (bash)Uploaded successfully.
Next is the data search.
https://docs.aws.amazon.com/ja_jp/opensearch-service/latest/developerguide/gsgsearch.html
Following the tutorial, execute the search using the curl command.
The following command is applied to the configuration we have built this time.
$ curl -XGET -u 'test:P@ssw0rd' 'https://search-fa-042-m2yc6j6kr63hqyhvqp6jwfim7a.ap-northeast-1.es.amazonaws.com/fa-042/_search?q=mars&pretty=true'
{
"took" : 45,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "fa-042",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.2876821,
"_source" : {
"director" : "Burton, Tim",
"genre" : [
"Comedy",
"Sci-Fi"
],
"year" : 1996,
"actor" : [
"Jack Nicholson",
"Pierce Brosnan",
"Sarah Jessica Parker"
],
"title" : "Mars Attacks!"
}
}
]
}
}
Code language: Bash (bash)The search was successfully executed.
A search for the word “mars” yielded the string “Mars Attacks!” with a _score of “0.2876821”.
Summary
As an introduction to OpenSearch, we built a basic environment using CloudFormation.