Amazon CloudWatch Insights for Amazon EKS on EC2 using AWS Distro for OpenTelemetry Helm charts
AWS Architecture BlogAmazon CloudWatch Insights for Amazon EKS on EC2 using AWS Distro for OpenTelemetry Helm chartsby Vimala Pydi, Sanjukta Mukherjee, and Sarat Para | on02 DEC 2022| inAmazon CloudWatch,Amazon EC2,Amazon Elastic Kubernetes Service,Architecture,AWS Distro for OpenTelemetry|Permalink|ShareThis blog provides a simplified three-step solution to collect metrics and logs from anAmazon Elastic Kubernetes Service(Amazon EKS) cluster onAmazon Elastic Compute Cloud(Amazon EC2) using theAWS Distro for OpenTelemetry (ADOT) Helm charts repositoryand send them toAmazon CloudWatchLogs and Amazon CloudWatch Container Insights. The ADOT Helm charts repository contains Helm charts to provide easy mechanisms to set up the ADOT Collector and other collection agents likefluentbitto collect telemetry data such as metrics, logs and traces to send to AWS monitoring services.Amazon EKS is a managed Kubernetes service that makes it easy for organizations to run Kubernetes on AWS Cloud and on premises. Organizations use Amazon EKS to automatically manage the availability and scalability of the Kubernetes control plane nodes responsible for scheduling containers, managing application availability, storing cluster data, and performing other key tasks.ADOTis a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Applications can set up ADOT Collector and other collector agents only once to send correlated metrics and traces to multiple AWS and Partner monitoring solutions. Fluent Bit is an open-source log processor and forwarder that you can use to collect data such as metrics and logs from different sources.Helmdeploys packaged applications to Kubernetes and structures them into Helm charts.Solution overviewA high-level architecture diagram depicted in Figure 1 shows a simple solution for collecting metrics and logs to send to Amazon CloudWatch Container Insights by installing an ADOT Helm chart on your existing or new Amazon EKS cluster.Here are the steps to set up an ADOT and fluentbit collector:Set up your environment and install the necessary tools to connect to an existing or newly created Amazon EKS cluster.Configure the necessary roles for AWS Identity and Access Management (IAM) roles for service accounts and install Helm charts for ADOT, enabling fluentbit.Monitor logs, metrics, and traces from Amazon CloudWatch Logs and Container Insights.Figure 1. Architecture diagram for Helm chart installation of ADOT and fluentbit to an existing Amazon EKS clusterPrerequisitesExisting AWS account with access to AWS Management ConsoleIntermediate-level knowledge and understanding of Amazon EKSAn existing or new Amazon EKS clusterInstall the toolsIn this blog,AWS Cloud9is used as an environment to connect to the Amazon EKS cluster and install Helm charts. If you choose to use AWS Cloud9, follow the step-by-step instructions provided inCreating an EC2 Environment. Refer toGetting started with Amazon EKSfor additional instructions to install eksctl, create EKS clusters, and set up required IAM permissions for connecting to an EKS cluster.Log in to your Amazon EKS cluster and inspect the cluster. Select an EKS cluster in AWS Management Console. On theResourcestab, check theDaemonSets, as in Figure 2a.Figure 2a. EKS cluster DaemonSetsOpen Amazon CloudWatch and inspect the Log groups and Amazon CloudWatch Container Insights. Note that the Log groups and Amazon CloudWatch Container Insights in Figure 2b do not show any EKS cluster-specific logs.Figure 2b. Container Insights before ADOT and fluentbit collector installationInstall Helm and configure IAM rolesRun the following command to install Helm, verify the version, and configure Bash completion for the Helm command:curl -ssl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
helm version --short

helm completion bash >> ~/.bash_completion
. /etc/profile.d/bash_completion.sh
. ~/.bash_completion
source <(helm completion bash)Set up IAM roles for service accounts.Replace XXX in the following commands with your EKS Cluster name.eksctl create iamserviceaccount \
--name fluent-bit \
--role-name EKS-ADOT-CWCI-Helm-Chart-Role-CW \
--namespace amazon-cloudwatch \
--cluster XXX \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-only \
--approveeksctl create iamserviceaccount \
--name adot-collector-sa \
--role-name EKS-ADOT-CWCI-Helm-Chart-Role-METRICS \
--namespace amazon-metrics \
--cluster XXX \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--role-only \
--approveDeploy the ADOT Helm chart.Replace XXX in the following code with your EKS Cluster name.CWCI_ADOT_HELM_ROLE_ARN_CW=$(aws iam get-role --role-name EKS-ADOT-CWCI-Helm-Chart-Role-CW | jq .Role.Arn -r)
CWCI_ADOT_HELM_ROLE_ARN_METRICS=$(aws iam get-role --role-name EKS-ADOT-CWCI-Helm-Chart-Role-METRICS | jq .Role.Arn -r)
helm repo add adot-helm-repo https://aws-observability.github.io/aws-otel-helm-charts
helm install adot-release adot-helm-repo/adot-exporter-for-eks-on-ec2  \
--set clusterName=XXX --set awsRegion=us-east-1 --set fluentbit.enabled=true \
--set adotCollector.daemonSet.service.metrics.receivers={awscontainerinsightreceiver} \
--set adotCollector.daemonSet.service.metrics.exporters={awsemf} \
--set adotCollector.daemonSet.cwexporters.logStreamName=EKSNode \Run the following commands to validate the successful deployment.Verify that two new namespaces have been created.kubectl get nsThe result should be:$ kubectl get ns
NAME                STATUS           AGE
amazon-cloudwatch   Active           2d20h
amazon-metrics      Active           2d20hVerify that a fluentbit pod was enabled as part of the ADOT Helm Chart under theamazon-cloudwatchnamespace.kubectl get all -n amazon-cloudwatchThe result should be:kubectl get all -n amazon-cloudwatch
NAME                   READY   STATUS    RESTARTS   AGE
pod/fluent-bit-9lrnt   1/1     Running   0          2d20h
pod/fluent-bit-h9lvt   1/1     Running   0          2d20h
pod/fluent-bit-nbqjm   1/1     Running   0          2d20h

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGEVerify theadot-collector-podunder theamazon-metricsnamespace.kubectl get all -n amazon-metricsThe result should be:$ kubectl get all -n amazon-metrics
NAME                                 READY   STATUS    RESTARTS   AGE
pod/adot-collector-daemonset-6qcsd   1/1     Running   0          2d20h
pod/adot-collector-daemonset-f92fr   1/1     Running   0          2d20h
pod/adot-collector-daemonset-gmhbx   1/1     Running   0          2d20h

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/adot-collector-daemonset   3         3         3       3            3           <none>          2d20hValidate the installation through the Amazon EKS cluster.Go to the Amazon EKS cluster and select theResourcestab. UnderWorkloads, selectDaemonSets,and find the fluent-bit and adot-collector-daemonsets as demonstrated in Figure 3.Figure 3. DaemonSet under Amazon EKS cluster resourcesMonitor logs, metrics, and tracesMonitor the CloudWatch Logs and CloudWatch Insights.In theLogssection, chooseLog groupsto view Amazon EKS cluster log groups with a prefix of/aws/containerinsights, as in Figure 4a.Figure 4a. EKS cluster log groupsIn theInsightssection, chooseContainer Insightsto view all the resources within your Amazon EKS cluster, as in Figure 4b.Figure 4b. EKS cluster’s Container Insights resourcesOn theContainer Insightspage, selectContainer mapfrom the dropdown to check the container map for Amazon EKS clusters, as demonstrated in Figure 4c.Figure 4c. EKS cluster’s Container Insights container mapOn theContainer Insightspage, selectPerformance monitoringfrom the dropdown to view various performance metrics for Amazon EKS cluster, as demonstrated in Figure 4d.Figure 4d. EKS cluster’s Container Insights performance monitoringCleanupIf you are no longer using the resources discussed in this blog, remove the excess AWS resources to avoid incurring charges. After you finish setting up ADOT and fluentbit collectors to send logs and metrics to Amazon CloudWatch Logs and Container Insights, clean up resources byuninstalling the ADOT Helm chart,deleting IAM Rolescreated for the services,deleting CloudWatch Logs, anddeleting Container Insights.ConclusionIn this blog we walked through a simple three-step solution to set up Amazon EKS cluster logs and Container Insights using Helm charts. The Helm chart installs ADOT and fluentbit as a DaemonSet in the existing EKS cluster to collect and port logs, metrics, and traces to Amazon CloudWatch Logs and Container Insights. The Amazon CloudWatch Container Insights provide insights into resources, monitor performance, and container map of all the resources within the Amazon EKS cluster.Vimala PydiVimala Pydi is a Sr. Solutions Architect based out of Orlando, FL. She works with the World-Wide Public-Sector team as a migrations specialist. She is passionate about helping customers with their cloud migration and modernization journey. Prior to AWS, she was helping customers in various industries in their digital transformations and has over 20 years of experience in IT industry.Sanjukta MukherjeeSanjukta Mukherjee is a Solutions Architect at AWS based out of New York. She works with customers from various industries to help them accelerate their adoption of AWS Cloud. She is responsible for helping customers to innovate and modernize their applications using AWS.Sarat ParaSarat Para is a Sr. Solutions Architect at AWS based out of Atlanta, GA. He enjoys learning new technologies and helping customers in their cloud adoption journey. He brings to his role over 20 years of technology experience in software development, architecture, and IT management.
This blog provides a simplified three-step solution to collect metrics and logs from an Amazon Elastic Kubernetes Service (Amazon EKS) cluster on Amazon Elastic Compute Cloud (Amazon EC2) using the AWS Distro for OpenTelemetry (ADOT) Helm charts repository and send them to Amazon CloudWatch Logs and Amazon CloudWatch Container Insights. The ADOT Helm charts repository […]
Vimala Pydi
Amazon CloudWatch Amazon EC2 Amazon Elastic Kubernetes Service Architecture AWS Distro for OpenTelemetry