Field Notes: Launch Amazon Elasticsearch Service with Fine-Grained Access Control and Amazon Cognito
AWS Architecture BlogField Notes: Launch Amazon Elasticsearch Service with Fine-Grained Access Control and Amazon Cognitoby Steffen Grunwald | on19 MAR 2021| inAmazon Cognito,Amazon OpenSearch Service,Architecture,Technical How-to|Permalink|ShareSeptember 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service.See details.Originally published March 2020 to the Field Notes blog, and reviewed for technical accuracy in March 2021.To get started withAmazon Elasticsearch Service(Amazon ES), you must have a concept for authentication and authorization for your search cluster. In addition to an IP-based access policy or a proxy server to protect your search cluster, you can leverage AWS Identity and Access Management (IAM) withAmazon Cognito User Poolsto authenticate and authorize users.With fine-grained access control you further control permissions to dashboards or data in your Amazon ES cluster. You can configure this using the AWS Management Console or the AWS Command Line Interface (AWS CLI). More information on configuration can be found in theAmazon Cognito Authentication for Kibana documentation.An Infrastructure-as-Code approach allows you to complete deployment and configuration in a safe, repeatable manner, so you can build and rebuild your search cluster automatically.This post describes the structure of anAWS Cloud Development Kit(AWS CDK) template for the fully automated provisioning of Amazon ES withfine-grained access controland Amazon Cognito resources, as well as your first search index and Kibana dashboard.Get StartedDeploy the sample template from the AWS Serverless Application Repository:Thesource codeis available on GitHub.Template RequirementsLet’s dive deeper into the resources you’ll need to include in your template for a fully automated deployment. The resources are listed in the order of their deployment:AnAmazon Cognito user pool,which is a collection of users. You can compare this to an LDAP directory. In this setup users can sign-in with their email and password to access Kibana.AnAmazon Cognito user pool groupfor your search cluster’s admin users.AnAmazon Cognito user pool domain, which is used to host the sign-in webpages for the authentication experience.AnAmazon Cognito identity pool, which stores unique identities for your users and federates them with identity providers like theAmazon Cognito user pool(#1). For each identity you can obtain temporary, limited-privilege AWS credentials with permission to your search cluster. You don’t need to configure any identity providers in this case. This configuration will be done automatically during theAmazon Elasticsearch Service domain(#7) deployment.IAM rolesfor your identity pool. You define an IAM role either as a default role for authenticated or unauthenticated users or map it to an Amazon Cognito user group. When a user signs-in, Amazon Cognito generates temporary AWS credentials for the user. These temporary credentials are associated with this role. Map one role to all authenticated users with restricted access to the search cluster data and dashboard elements. Map a second role to the admin group (#2) that has unrestricted access (master user).AnAmazon Elasticsearch Service IAM rolethat grants permissions to configure the identity provider and user pool for authentication of users for Kibana and your search cluster.AnAmazon Elasticsearch Service domain, which is your search cluster. During deployment Amazon Elasticsearch Service assumes its IAM role (#6) to configure the authentication with theAmazon Cognito user pool(#1) and theAmazon Cognito identity pool(#4). The search cluster has the fine-grained access control configuration, granting permissions to the IAM roles (#5).  You should size the search cluster according to your specific workload. If you just want to experiment with Amazon Elasticsearch Service, start withfree tierconfiguration of a singlet3.small.elasticsearchinstance and 10GB of EBS storage (refer to theAmazon Elasticsearch Service pricingfor details). For fine-grained access control, the instance type has to support encryption of data at rest (readsupported instance types).Anidentity pool role attachmentlinks theauthenticated user IAM role(#5) to theAmazon Cognito identity pool(#4).AnAWS Lambda functionthat is used by anAWS CloudFormation custom resource(#10) to send requests to theAmazon Elasticsearch Service domain(#7) endpoint. The function needs to sign every request with the credentials of an IAM role that has unrestricted access to the search cluster (master user).AnAWS CloudFormation custom resourcewhich defines the requests that should be sent by theAWS Lambda function(#9) to configure fine-grained access control, and create a search index template and a Kibana dashboard.PermissionsThere are two ways to grant permissions to your search cluster via IAM policies: a resource-based policy associated with your search domain, or identity-based policies associated with your IAM users and roles.The following resource-based policy shows the recommended configuration for fine-grained access control. Don’t use this policy without fine-grained access control or without network-based restrictions as it otherwise allows anonymous access.const esDomain = new CfnDomain(this, "searchDomain", {
    [...]
    accessPolicies: {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "AWS": "*"
            },
            "Action":
              "es:ESHttp*",
            ],
            "Resource": "arn:aws:es:" + this.region + ":"
            + this.account + ":domain/" + applicationPrefix + "/*"
          }
        ]
    }
});Lock down the access to IP ranges byadding a conditionbased on theaws:sourceIpcontext key. If your search cluster runs inside of an Amazon Virtual Private Cloud (VPC), use a security group.Deploy your Search Index and the DashboardAfter your search cluster deployment is completed, you can configure resources in the Kibana user interface (the dashboard editor). You can also send HTTP requests to the search cluster’s endpoint from the Kibana dev tools or external systems. The sample template includes an AWS CloudFormation custom resource that sendssigned requeststo set up fine-grained access control resources, create a search index template (1) and import a dashboard (2) from a previous export:import path = require('path');
import fs = require('fs');
[...]
new CustomResource(this, 'esRequestsResource', {
  provider: esRequestProvider.serviceToken,
  properties: {
    requests: [
[…]
      {
        "method": "PUT", // (1)
        "path": "_template/example-index-template",
        "body": fs.readFileSync(path.join(__dirname, "index-template.json")).toString()
      },
      {
        "method": "POST", // (2)
        "path": "_plugin/kibana/api/saved_objects/_import?overwrite=true",
        "body": fs.readFileSync(path.join(__dirname, "dashboard.ndjson")).toString(),
            "securitytenant": "logs-tenant",
            "filename": "dashboard.ndjson"
      }
    ]
  }
});Leveraging the JavaScript librariesfsandpathallows you to store the request bodies in separate files so that the JSON body does not bloat the template and you can easily validate the JSON syntax. To create a file similar to thedashboard.ndjsonexample, retrieve an export of your saved objects from Kibana.Deploy the template from codeThe sample template’ssource codeis available on GitHub. The template provisions an Amazon Elasticsearch Service cluster in a fully automated way. The search cluster consists of a singlet3.small.elasticsearchinstance with 10GB of EBS storage. It is integrated with Amazon Cognito User Pools so you only need to add your user(s). The template also configures an example Kibana dashboard and an Amazon ES index template.The template prefixes the search domain and the Amazon Cognito Hosted UI with a string that you can define with theapplicationPrefixtemplate parameter.You deploy the template with CDK. You must install and configure theAWS CLIand theCDK.The CDK template is written in TypeScript. TypeScript sources must be compiled to JavaScript initially and after each modification. Open a new terminal and keep this terminal open in the background if you like to change the source files. Change the directory to the one wherecdk.jsonis and execute:npm install
npm run watchRead theCDK developer guidefor more information.Create or update the application withcdk deploy:cdk deploy -c applicationPrefix=<PREFIX>Access the Example DashboardAs soon as the application is deployed completely the outputs of the AWS CloudFormation stack provides the links for the next steps. You will find two URLs in the AWS CloudFormation console calledcreateUserUrlandkibanaUrl.Use thecreateUserUrllink from the outputs, or navigate to the Amazon Cognito user pool in the console to create a new user in the pool.Enter an email address as username and email.Enter a temporary passwordof your choice.Leave the phone number empty anduncheck the checkbox to mark the phone number as verified.If you like you can check the checkboxes to send an invitation to the new user or to make the user verify the email address.Then chooseCreate user.The user has access restricted to the logs-tenant. If you want to provide full access control, including security management permissions, add the user to thees-adminsgroup:Access the Kibana dashboard with thekibanaUrllink from the outputs, or navigate to the Kibana link displayed in the Amazon Elasticsearch Service console.In Kibana, go to the tenant selection by choosing the user menu on the top right. ChooseSwitchtenants:Choose thelogs-tenantthat has been created during the launch of the search cluster. ChooseConfirm.Choose the navigation menu on top left and chooseDashboard.Choose theExample Dashboard. The dashboard contains instructions to add new documents to the search index and to visualize the documents with the graph in the dashboard.Cleaning UpTo avoid incurring charges, delete the AWS CloudFormation stack when you are finished experimenting via cdk destroy in the directory wherecdk.jsonis:cdk destroyOr delete the AWS CloudFormation stack manually:Sign in to the AWS CloudFormation console and choose your stack.ChooseDeleteto delete all resources, including the search cluster and the Amazon Cognito user pool.ConclusionIn this post, we identified the resources necessary to bootstrap an Amazon Elasticsearch Service search cluster with fine-grained access control. As well as including a search index template and a Kibana dashboard protected by Amazon Cognito User Pools.To get started, launch the sample template from the AWS Serverless Application Repository. As a next step you can make the template your own by customizing it according to your needs and deploy it with CDK. You can submit enhancements to the sample template in the source repository.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Amazon Cognito,Amazon Elastisearch,Field NotesSteffen GrunwaldSteffen Grunwald is a Principal Solutions Architect at Amazon Web Services. He supports customers in solving their sustainability challenges through the cloud. Having a long software engineering background, he loves to dive deep into application architectures and development processes to drive sustainability, performance, cost, operational efficiency, and increase the speed of innovation.
September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. Originally published March 2020 to the Field Notes blog, and reviewed for technical accuracy in March 2021. To get started with Amazon Elasticsearch Service (Amazon ES), you must have a concept for authentication and authorization for your search cluster. In addition to […]
Steffen Grunwald
Amazon Cognito Amazon OpenSearch Service Architecture Technical How-to