Monitoring external URLs using Amazon CloudWatch Synthetics
What is Amazon CloudWatch Synthetics
Amazon CloudWatch Synthetics allows us to monitor the application endpoints more easily. With this new feature, CloudWatch now collects canary traffic, which can continually verify the customer experience even when we don’t have any customer traffic on our applications, and help us to identify the issue before the customer notices the same. CloudWatch Synthetics supports monitoring of the REST APIs, URLs, and website contents, checking for unauthorized changes from phishing, code injection and cross-site scripting. CloudWatch Synthetics runs tests on our endpoints every minute, 24×7, and alerts us when our application endpoints don’t behave as expected.
How canaries work ?
Canaries are Node.js scripts. They create Lambda functions in the account that use Node.js as a framework. Canaries can use the Puppeteer Node.js library to perform functions on our applications. Canaries work over HTTP and HTTPS protocols.
Canaries check the availability and latency of the endpoints and can store load time data and screenshots of the UI. They monitor the REST APIs, URLs, and website contents, and they can check for unauthorized changes from phishing, code injection and cross-site scripting.
We can run the canary in two ways either once or on a regular schedule. The canaries can be scheduled to run 24 hours a day, as often we set once per minute.
How to create a canary
- Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
- In the navigation pane, choose Canaries.
- Choose Create Canary.
- Choose one of the following:
- Use a blueprint
- Upload a script
- Import from S3
- Under Name, enter the name for the canary. The name is used on many pages, so it recommends giving it a descriptive name that distinguishes it from other canaries.
- Under Application or endpoint URL, enter the URL that we want the canary to test. This URL must include the protocol (such as https://).
- To test an endpoint on a VPC, we must also enter information about the VPC later in this procedure.
- (Optional) Under Alarms, choose whether we want default CloudWatch alarms to be created for this canary.
Resources That Are Created for Canaries
To view canary details and the results of canary runs, we must be signed in as an IAM user who has either the CloudWatchSyntheticsFullAccess or the CloudWatchSyntheticsReadOnlyAccess attached.
To read all of the Synthetics data within the console, we also need the AmazonS3ReadOnlyAccess and CloudWatchReadOnlyAccess policies. To view the source code used by canaries, we also need the AWSLambdaReadOnlyAccess policy.
To create canaries, we must be signed in as an IAM user who has the CloudWatchSyntheticsFullAccess policy or a similar set of permissions. To create IAM roles for the canaries, we also need the following inline policy statement:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“iam:CreateRole”,
“iam:CreatePolicy”,
“iam:AttachRolePolicy”
],
“Resource”: [
“arn:aws:iam::*:role/service-role/CloudWatchSyntheticsRole*”,
“arn:aws:iam::*:policy/service-role/CloudWatchSyntheticsPolicy*”
]
}
]
}
CloudWatchSyntheticsFullAccess Policy
{
“Version”:”2012-10-17″,
“Statement”:[
{
“Effect”:”Allow”,
“Action”:[
“synthetics:*”
],
“Resource”:”*”
},
{
“Effect”:”Allow”,
“Action”:[
“s3:CreateBucket”,
“s3:PutBucketEncryption”,
“s3:PutEncryptionConfiguration”
],
“Resource”:[
“arn:aws:s3:::cw-syn-results-*”
]
},
{
“Effect”:”Allow”,
“Action”:[
“iam:ListRoles”,
“s3:ListAllMyBuckets”,
“s3:GetBucketLocation”
],
“Resource”:”*”
},
{
“Effect”:”Allow”,
“Action”:[
“s3:GetObject”,
“s3:ListBucket”
],
“Resource”:”arn:aws:s3:::cw-syn-*”
},
{
“Effect”:”Allow”,
“Action”:[
“s3:GetObjectVersion”
],
“Resource”:”arn:aws:s3:::aws-synthetics-library-*”
},
{
“Effect”:”Allow”,
“Action”:[
“iam:PassRole”
],
“Resource”:[
“arn:aws:iam::*:role/service-role/CloudWatchSyntheticsRole*”
],
“Condition”:{
“StringEquals”:{
“iam:PassedToService”:[
“lambda.amazonaws.com”,
“synthetics.amazonaws.com”
]
}
}
},
{
“Effect”:”Allow”,
“Action”:[
“iam:GetRole”
],
“Resource”:[
“arn:aws:iam::*:role/service-role/CloudWatchSyntheticsRole*”
]
},
{
“Effect”:”Allow”,
“Action”:[
“cloudwatch:GetMetricData”,
“cloudwatch:GetMetricStatistics”
],
“Resource”:”*”
},
{
“Effect”:”Allow”,
“Action”:[
“cloudwatch:PutMetricAlarm”,
“cloudwatch:DeleteAlarms”
],
“Resource”:[
“arn:aws:cloudwatch:*:*:alarm:Synthetics-*”
]
},
{
“Effect”:”Allow”,
“Action”:[
“cloudwatch:DescribeAlarms”
],
“Resource”:[
“arn:aws:cloudwatch:*:*:alarm:*”
]
},
{
“Effect”:”Allow”,
“Action”:[
“lambda:CreateFunction”,
“lambda:AddPermission”,
“lambda:PublishVersion”,
“lambda:UpdateFunctionConfiguration”
],
“Resource”:[
“arn:aws:lambda:*:*:function:cwsyn-*”
]
},
{
“Effect”:”Allow”,
“Action”:[
“lambda:GetLayerVersionByArn”,
“lambda:GetLayerVersion”,
“lambda:PublishLayerVersion”
],
“Resource”:[
“arn:aws:lambda:*:*:layer:cwsyn-*”,
“arn:aws:lambda:*:*:layer:Synthetics:*”
]
}
]
}
CloudWatchSyntheticsReadOnlyAccess
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“synthetics:Describe*”,
“synthetics:Get*”
],
“Resource”: “*”
}
]
}
Using Canary Blueprints
This section provides the small print about each of the canary blueprints and therefore the tasks each blueprint is best fitted to .
Blueprints are provided for the following canary types:
- Heartbeat monitor
- API Canary
- Broken link checker
- GUI Workflow
Heartbeat monitor
Hearbeat scripts load the specified URL and store a screenshot of the page and an HTTP archive file (HAR file). They also store logs of accessed URLs.
API Canary
API canaries can test the essential Read and Write functions of a REST API.
REST stands for representational state transfer and it is a set of rules that developers follow when creating an API
The REST API contains two parts on a URL, and they are requests and data.
Again we can classify the API components as follows.
The endpoint, which is the URL that you request.
The method, which is the type of request that is sent to the server. The REST APIs support GET (read), POST (write), PUT (update), PATCH (update), and DELETE (delete) operations.
The headers, which provide information to both the client and the server. They are used for authentication and providing information about the body content. For a list of valid headers, see HTTP Headers.
The data (or body), contains information to be sent to the server. This is used only for POST, PUT, PATCH, or DELETE requests.
The URL that you request.
Broken Link Checker
The broken link checker collects all the links inside the URL that we are testing by using document.getElementsByTagName(‘a’) It tests only up to the number of links that we specify, and the URL itself is counted because the first link
GUI Workflow Builder
The GUI Workflow Builder blueprint verifies that actions can be taken on a web page. For example, if we have a web page with a login form, the canary can populate the user and password fields and submit the form to verify that the web page is working correctly. We can specify the actions on this type; these are the actions available with this blueprint.
- Click
- Verify selector
- Verify text
- Input text
- Click with navigation
Security Considerations for Synthetics Canaries
The following sections explain security issues that should be considered when creating and running canaries in Synthetics.
- Use Secure Connections
Use https connection to add canneries to our environment.
- Canary Naming Considerations
The canary ARN includes the canary name. Choose canary names that don’t reveal the proprietary information.
- Secrets in Canary Code
It is recommend that don’t include the secrets, such as access keys or database credentials, in the canary source code
- Permissions Considerations
- Stack Traces and Exception Messages
- Scope the IAM Roles Narrowly
- Header Logging
Viewing Canary Statistics and Details
To view canary statistics and details follow the provided steps.
- Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
- In the navigation pane, choose Canaries.
- In the details about the canaries that have been created:
- Status visually shows how many of the canaries have passed their most recent runs.
- In the graph under Canary runs, each point represents one minute of the canaries’ runs. We can pause on a point to see details.
- To see more details about a single canary, choose a point in the Status graph or choose the name of the canary in the Canaries table.
- In the details about that canary:
- Under Canary runs, we can choose one of the lines to see details about that run.
- Under the graph, we can choose Screenshot, HAR file, or Logs to see these types of details.
- The logs for the canary runs are stored on both S3 buckets and CloudWatch Logs.
- Screenshots show how the customers view their web pages. We can use the HAR files (HTTP Archive files) to view detailed performance data about the web pages. We can analyze the list of web requests and catch performance issues such as time to load for an item. Log files show the record of interactions between the canary run and therefore the website and may be wont to identify details of errors.
We can create a canary to monitor endpoints in a VPC specific environment as well.