Using container insights to monitor containerized applications
What is Container Insights?
Container Insights is a service incorporated with the amazon cloud watch to get the metrics and monitor the containerized applications and microservices.
Use CloudWatch Container Insights to collect, aggregate, and summarize metrics and logs from the containerized applications and microservices. Container Insights is available on Amazon ECS, Amazon EKS, and Kubernetes platforms on Amazon EC2. Amazon ECS support includes support for Fargate.
CloudWatch automatically collects the metrics for many resources, like CPU, memory, disk, and network parameters. Container Insights also provides diagnostic information, like container restart failures, to help isolate the issues and resolve them quickly. We can also set CloudWatch alarms on metrics that Container Insights collects from a cluster. Also we can use this for the custom cloudwatch monitoring dashboards.
How does it work ?
Container Insights used to collect data as performance log events using an embedded metric style . These performance logs events are entries that use a structured JSON schema that enables high-cardinality data to be ingested and stored at scale. From this data, CloudWatch creates aggregated metrics at the cluster, node, pod, task, and service level as CW metrics.
In Amazon EKS and Kubernetes, Container Insights uses a containerized version of the CloudWatch agent to find out all of the running containers in a particular cluster. Then it collects the performance data at every layer of the performance stack.
Supported Platforms:
Container Insights is available for Amazon ECS, Amazon EKS, and Kubernetes platforms on Amazon EC2 instances.
The entire process for setting up the ContainerInsights on Amazon EKS or Kubernetes are:
Verify that you have the necessary prerequisites.
Before installing the Container Insights on Amazon EKS or Kubernetes, verify the following prerequisites are satisfied:
- We must have a functional Amazon EKS or K8s cluster with nodes attached in one of the regions that supports the Container Insights for Amazon EKS and Kubernetes.
- Ensure that kubectl installed and running
- Be sure that the Kubernetes cluster has enabled role-based access control (RBAC)
- The kubelet has enabled Webhook authorization mode
- Edit the cluster and add the parameters and run a rolling-update on kubernetes cluster.
kubelet:
anonymousAuth: false
authenticationTokenWebhook: true
authorizationMode: Webhook
Attaching the policy to the IAM role of your worker nodes
Follow these steps to attach the policy to the IAM role of the worker nodes. This will work for both the Amazon EKS clusters and the Kubernetes clusters outside the Amazon EKS.
Follow the provided steps to attach the necessary policy to the IAM role for the worker nodes
- Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
- We need to select one of the worker node instances and choose the IAM role in the description.
- On the IAM role page, choose Attach policies.
- From the list of policies, select the checkbox next to CloudWatchAgentServerPolicy. If it is necessary, use the search box to find this policy.
- Choose Attach policies.
If the Kubernetes cluster is outside the Amazon EKS, might not already have an IAM role attached to the worker nodes. If not, we must first attach an IAM role to the instance and then add the policy as explained in the previous steps.
If a Kubernetes cluster outside Amazon EKS and we want to collect EBS volume IDs in the metrics, we must add another policy to the IAM role attached to the instance. Add the following as an inline policy.
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: [
“ec2:DescribeVolumes”
],
“Resource”: “*”,
“Effect”: “Allow”
}
]
}
The container runtime should be ‘Docker’:
- Set up the CloudWatch agent as a DaemonSet on the Amazon EKS cluster or Kubernetes cluster to send metrics to CloudWatch, and found out the FluentD as a DaemonSet to send logs to CloudWatch Logs.
To deploy the Container Insights using the quick start, follow the below provided steps .
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | sed “s/{{cluster_name}}/cluster-name/;s/{{region_name}}/cluster-region/” | kubectl apply -f –
* modify the clustername and region .
How to deploy the cloudwatch agent on a kubernetes cluster
Step 1: Create a Namespace for CloudWatch
We need to follow the steps to create a Kubernetes namespace called amazon-cloudwatch for CloudWatch.
To create a namespace for CloudWatch
Enter the following command.
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
Step 2: We need to create a Service Account in the Cluster
By following the steps we can create a service account for the CloudWatch agent, if it is not created previously.
To create the service account for the CloudWatch agent on the cluster to be monitored.
Enter the following command.
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml
Step 3: We need to create a ConfigMap for the CloudWatch Agent
Following the steps provided below we can create a ConfigMap for the CloudWatch agent.
To create a ConfigMap for the CloudWatch agent
Download the ConfigMap YAML to your kubectl client host by running the following command:
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml
After downloading the file modify the cluster name and other required parameters and apply the same.
Kubectl apply -f cwagent-configmap.yaml
Step 4: Deploy the CloudWatch Agent as a DaemonSet on the cluster
To finish the installation of the CloudWatch agent and begin the collecting the container metrics from the cluster, use the following steps.
To deploy the CloudWatch agent as a DaemonSet
- If we do not need to use StatsD on the cluster, enter the following command.
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
- If we need to collect the StatsD ente the following command.
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml
Edit the file and modify the below.
ports:
– containerPort: 8125
hostPort: 8125
protocol: UDP
kubectl apply -f cwagent-daemonset.yaml
kubectl get pods -n amazon-cloudwatch
- We can perform these steps at once as part of the quick start setup, or do them separately.
- Need to set up the Amazon EKS control plane logging.(Optional)
- Need to set up the CloudWatch agent as a StatsD endpoint on the cluster to send StatsD metrics to CloudWatch.(Optional)
- Enable the App Mesh for Envoy Access Logs. (Optional)