Failing over with Route53 and displaying error page

Configuring Route53 Failover Routing Policy to Display Error Page

This topic is related to high elasticity, which is also one of the questions in the AWS SAA: You can increase the elasticity of your system by choosing a Route 53 failover routing policy and displaying an error page in the event of a failure.

Route53 allows you to choose from a variety of routing policies, one of which is the failover routing policy.

Failover routing lets you route traffic to a resource when the resource is healthy or to a different resource when the first resource is unhealthy.
Failover routing

In this case, we will build a system with an EC2 instance under the ALB distribution to build the primary website. If an abnormality occurs on the primary side, we will configure the system to display an error page placed in the S3 bucket.

Environment

Diagram of failover with Route53 and display error page.

Configuration of the primary web site side

We will place one EC2 instance on each of the two private subnets. The instances will be based on the latest Amazon Linux 2.

To install Apache, we will create a VPC endpoint for S3.

Place the ELB in front of the EC2 instance, and use the ALB type.

Configuration of the error page side

Next, we will prepare the error page in an S3 bucket.

Enable the static website hosting feature of the bucket, and place the HTML file for the error page in the bucket.

Configuration of Route53

Finally, Route53, as mentioned earlier, we will configure an active-passive type failover routing policy. Specify ALB as the primary resource and S3 bucket as the secondary resource.

To check the failover behavior, follow the steps below.

Check the behavior of the primary resource when it is normal.
Stop the two EC2 instances, and reproduce the situation in which the primary resource has failed.
Fail over to the secondary resource, and confirm that the error page is displayed.
Launch the two EC2 instances, and reproduce the situation in which the primary resource has recovered from the failure.
Verify that traffic is routed to the primary resource again.

CloudFormation template files

We will build the above configuration using CloudFormation.

Place the CloudFormation template at the following URL.

awstut-saa/01/001 at main · awstut-an-r/awstut-saa

Contribute to awstut-an-r/awstut-saa development by creating an account on GitHub.

The domain used in this configuration is assumed to be a domain obtained from Route53, so no settings for the domain’s HostedZone are required. If you wish to use a domain obtained from a source other than AWS, please add the definition of the HostedZone resource to the above template.

Template file points

We will cover the key points of each template file to configure this environment.

Attaching instances in private subnets to ALB

First, we will build the primary resource with ALB and two EC2 instances.

There are two points.

The first point is that the instances are placed on a private subnet.

The subnet that can be associated with the ALB is a public subnet. Therefore, you need to prepare an empty public subnet for each AZ and access the instances in each private subnet through it.

Please refer to the following page for details.

The second point is that in order to install Apache, you need to run yum on an instance in a private subnet.

Normally, when you run yum, you need a path to the Internet. However, in the case of Amazon Linux, it is possible to run yum by accessing the yum repository hosted on the S3 bucket. Even in a private subnet where there is no route to the Internet, it is possible to securely access the S3 bucket by installing an S3 endpoint.

For more details, please refer to the following page

This time, we will create the user data by setting up the installation and startup of Apache using yum, and writing the instance ID in index.html.

The S3 bucket name where error page is placed should match domain name

Define the resources related to S3 in saa-01-001-s3.yaml.

Set up an error page in the S3 bucket and configure it to be published.

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref DomainName
      AccessControl: Private
      WebsiteConfiguration:
        IndexDocument: index.html
Code language: YAML (yaml)

Use the static website hosting feature of S3 to display error pages. For more information about this feature, please refer to the following page.

The key point in displaying the error page using the static web hosting feature is the name of the bucket.

The bucket must have the same name as your domain or subdomain. For example, if you want to use the subdomain acme.example.com, the name of the bucket must be acme.example.com.
Routing traffic to a website that is hosted in an Amazon S3 bucket

The bucket name is specified in the BucketName property, and it is necessary to specify the same string as the domain name used in the primary website in this property. In this case, we will use the built-in function Fn::Ref to refer to the domain name.

Use health checks to check if primary resource has failed

Define the resources related to Route53 in saa-01-001-route53.yaml.

First, configure Route53 to be able to check the failure status of the primary resource.

Resources:
  DnsHealthCheck:
    Type: AWS::Route53::HealthCheck
    Properties:
      HealthCheckConfig:
        Port: !Ref HTTPPort
        Type: HTTP
        ResourcePath: /
        FullyQualifiedDomainName: !Ref ALBDnsName
        RequestInterval: !Ref RequestInterval
        FailureThreshold: !Ref FailureThreshold
Code language: YAML (yaml)

The HealthCheckConfig property specifies the target and method of health check, and the conditions under which a failure is considered to have occurred.

FullyQualifiedDomainName property: Specify the target of the health check as the root page of the ALB.
Port and Type properties: The health check is performed using HTTP (80/tcp).
RequestInterval property: Health check is performed once every 30 seconds.
FailureThreshold property: If the HTTP status code is not “200” for three consecutive times as a result of the health check, a failure is considered to have occurred.

Route53 active and passive type failover configuration

Finally, define the record information for Route53.

Resources:
  AwstutNetDnsRecordGroup:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      HostedZoneName: !Sub "${DomainName}."
      RecordSets:
        # ALB
        - Name: !Ref DomainName
          Failover: PRIMARY
          HealthCheckId: !Ref DnsHealthCheck
          SetIdentifier: primary
          Type: A
          AliasTarget: 
            DNSName: !Ref ALBDnsName
            EvaluateTargetHealth: true
            HostedZoneId: !Ref ALBHostedZoneId
        # S3
        - Name: !Ref DomainName
          Failover: SECONDARY
          SetIdentifier: secondary
          Type: A
          AliasTarget: 
            DNSName: !Ref S3DnsName
            HostedZoneId: !Ref S3HostedZoneId
Code language: YAML (yaml)

In the HostedZoneName property, specify the domain name to register with Route53. Specify the same domain name as the S3 bucket name mentioned above.

In the RecordSets property, specify the resources associated with the domain name specified in the HostedZoneName property, where the first element is the ALB and the second is the S3 bucket. The first element is the ALB, and the second is the S3 bucket. There are three settings that are particularly important when configuring failover.

For the Failover property, specify “PRIMARY” since ALB is the primary resource, and “SECONDARY” since S3 is the secondary resource.
Specify the aforementioned health check resource in the HealthCheckId property on the ALB side.
Specify “true” for the EvaluateTargetHealth property in the AliasTarget property on the ALB side.

By configuring the above settings, if a failure occurs in the ALB, it will automatically fail over and the traffic will be routed to the S3 side.

As for the HostedZoneId property in the RecordSetGroup, there are certain values that should be set.

First, for the ALB side, refer to Elastic Load Balancing endpoints and quotas, and you will see that you need to specify “Z14GRHDCWA56QT” for the ap-northeast-1 region.

Next, for the value on the S3 bucket side, according to Amazon Simple Storage Service endpoints and quotas, we need to specify “Z2M4EHUR26P7ZW” in the ap-northeast-1 region.

Architecting

Using CloudFormation, we will build this environment and check its actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.

For more information on how to create stacks and check each stack, please refer to the following page.

After checking the resources for each stack, the following information is available for the main resources created this time.

Domain name: awstut.net
Instance ID of EC2 instance: i-032febbe1b5fd6a9e, i-00520c950f3fbcd97
ALB Name: saa-01-001-ALB
S3 Bucket Name: awstut.net
Route53 Health Check ID: d0d474a5-aaba-4cf3-b5c4-dc3075f56b49

First, check the configuration status of Route53.

You can see that ALB is registered as the primary and S3 bucket as the secondary as specified in the template file.

Set up error page in S3 bucket

Set up an error page in the S3 bucket.

$ echo "<html><head></head><body>Hello World form S3 Bucket.</body></html>" > ./index.html

$ aws s3 cp index.html s3://awstut.net
upload: ./index.html to s3://awstut.net/index.html
Code language: Bash (bash)

After creating the HTML file using the echo command, copy it to the bucket using the AWS CLI.

Place the HTML file in the S3 bucket of the failover destination.

If the ALB fails, it will fail over and this HTML file will be displayed.

Behavior check: normal

Now that we are ready, we will access the web app we have built and check its behavior.

$ curl http://awstut.net
instance-id: i-00520c950f3fbcd97

$ curl http://awstut.net
instance-id: i-032febbe1b5fd6a9e
Code language: Bash (bash)

The IDs of the two EC2 instances were returned alternately. This is because the ALB is routing traffic to the two instances in a round-robin fashion.

This indicates that the ALB and the EC2 instances under it have passed the health check and are operating normally as the primary resource.

Route53 health check status check: normal

In the meantime, we will also check the status of the health check under normal conditions.

The Status is “Healthy”. This shows that the health check has been successful.

Recreate failure by stopping EC2 instance

We have seen that the primary resource is running normally.

Now we will stop the two instances to create a situation where a failure has occurred intentionally.

$ aws ec2 stop-instances \
--instance-ids i-032febbe1b5fd6a9e i-00520c950f3fbcd97
Code language: Bash (bash)

Stop the instance to reproduce the failure.

The value of Instance state for both instances is “Stopped”, so they have been stopped.

Route53 health check status check: when failure occurs

After the instance is stopped, check the status of the health check again.

Route53 health check failed due to failure

The Status value is now “Unthealthy”. This means that the health check has failed.

This is because there are no more normal targets registered in the ALB due to stopping the instance.

Behavior check: when failure occurs

Now that we have confirmed that the health check has failed, we will access the web app again.

$ curl http://awstut.net
<html><head></head><body>Hello World form S3 Bucket.</body></html>
Code language: Bash (bash)

The HTML file placed in the S3 bucket has been returned.

This shows that Route53 failed over the traffic routing from ALB, the primary resource, to S3, the secondary resource, due to the failed health check to ALB.

Start instance and reproduce disaster recovery

We now know that the system automatically fails over when a failure occurs. Next, let’s check the behavior at the time of failure recovery.

In order to reproduce the failure recovery, we will re-launch the instance that was stopped earlier.

$ aws ec2 start-instances \
--instance-ids i-032febbe1b5fd6a9e i-00520c950f3fbcd97
Code language: Bash (bash)

Start the instance and reproduce the failure recovery.

The Instance Status value is now “Running”. This means that the instance is currently running.

Behavior check: during failure recovery

Once the instance is back, access the web app again to check the behavior.

$ curl http://awstut.net
instance-id: i-032febbe1b5fd6a9e

$ curl http://awstut.net
instance-id: i-00520c950f3fbcd97
Code language: Bash (bash)

We can see that the traffic is routed to the ALB side again. This is because the instance was re-recognized as the target of the ALB when it was launched, and Route53 successfully performed a health check on the ALB.

Route53 health check passed again due to disaster recovery.

As you can see, in Route53’s active-passive failover routing policy, when the primary resource recovers from a failure, the failover state will be automatically removed and traffic will be routed to the primary side again.

Summary

This time, through hands-on experience with Route53 active-passive type failover, we confirmed the behavior of routing during failure and disaster recovery.　