Microservices discovery using Amazon EC2 and HashiCorp Consul
AWS Architecture BlogMicroservices discovery using Amazon EC2 and HashiCorp Consulby Marine Haddad, Tom Palmer, Tony Phan, and George Agiasoglou | on30 JUN 2023| inAmazon EC2,Amazon VPC,Architecture,AWS Systems Manager,AWS Transit Gateway|Permalink|ShareThese days, large organizations typically have microservices environments that span across cloud platforms, on-premises data centers, and colocation facilities. The reasons for this vary but frequently include latency, local support structures, and historic architectural decisions. However, due to the complex nature of these environments, efficient mechanisms for service discovery and configuration management must be implemented to support operations at scale. This is an issue also faced by Nomura.Nomurais a global financial services group with an integrated network spanning over 30 countries and regions. By connecting markets East & West, Nomura services the needs of individuals, institutions, corporates, and governments through its three business divisions: Retail, Investment Management, and Wholesale (Global Markets and Investment Banking). E-Trading Strategy Foreign Exchange sits within Global Markets, and focuses on all quantitative analysis and technical aspects of electronic FX flows. The team builds out a number of innovative solutions for clients, all of which are needed to operate in an ultra-low latency environment to be competitive. The focus is to build high-quality engineered platforms that can handle all aspects of Nomura’s growing 24 hours a day, 5-and-a-half days a week FX business.In this blog post, we share the solution we developed for Nomura and how you can build a service discovery mechanism that uses a hierarchical rule-based algorithm. We use the flexibility ofAmazon Elastic Compute Cloud(Amazon EC2) and third-party software, such as SpringBoot and Consul. The algorithm supports features such as service discovery by service name, Domain Name System (DNS) latency, and custom provided tags. This can activate customers with automated deployments, since services are able to auto-discover and connect with other services. Based on provided tags, customers can implement environment boundaries so that a service doesn’t connect to an unintended service. Finally, we built a failover mechanism, so that if a service becomes unavailable, an alternative service would be provided (based on given criteria).After reading this post, you can use the provided assets in the open source repository to deploy the solution in their sandbox environment. The Terraform and Java code that accompanies this post can be amended as needed to suit individual requirements.Overview of solutionThe solution is composed of a microservices platform that is spread over two different data centers, and a Consul cluster per data center. We use twoAmazon Virtual Private Clouds (VPCs) to model geographically distributed Consul “data centers”. These VPCs are then connected via anAWS Transit Gateway. By permitting communication across the different data centers, the Consul clusters can form a wide-area network (WAN) and have visibility of service instances deployed to either. The SpringBoot microservices use the Spring Cloud Consul plugin to connect to the Consul cluster. We have built a custom configuration provider that uses Amazon EC2instance metadata serviceto retrieve the configuration. The configuration provider mechanism is highly extensible, so anyone can build their own configuration provider.The major components of this solution are:Sample microservices built using Java and SpringBoot, and deployed in Amazon EC2 with one microservice instance per EC2 instanceA Consul cluster per Region with one Consul agent per EC2 instanceA custom service discovery algorithmFigure 1. Multi-VPC infrastructure architectureA typical flow for a microservice would be to 1/ boot up, 2/ retrieve relevant information from the EC2 Metadata Service (such as tags), and, 3/ use it to register itself with Consul. Once a service is registered with Consul it can discover services to integrate with, and it can be discovered by other services.An important component of this service discovery mechanism is a custom algorithm that performs service discovery based on the tags created when registering the service with Consul.Figure 2. Service discovery flowThe service flow shown in Figure 2 is as follows:The Consul agent deployed on the instance registers to the local Consul cluster, and the service registers to its Consul agent.The Trading service looks up for available Pricer services via API calls.The Consul agent returns the list of available Pricer services, so that the Trading service can query a Pricer service.WalkthroughFollowing are the steps required to deploy this solution:Provision the infrastructure using Terraform. The application .jar file and the Consul configuration are deployed as part of it.Test the solution.Clean up AWS resources.The steps are detailed in the next section, and the code can be found in thisGitHub repository.PrerequisitesInstallGitInstallTerraformInstallPackerInstallAWS CLIAnAWS accountAConsul Enterprise License– if you don’t have an Enterprise license, you can contact theHashicorp Support Teamto request a trial license.Deployment stepsNote: The default AWS Region used in this deployment is ap-southeast-1. If you’re working in a different AWS Region, make sure to update it.Clone the repositoryFirst, clone the repository that contains all the deployment assets:git clone https://github.com/aws-samples/geographical-hierarchical-service-lookup-with-consul-on-awsBuild Amazon Machine Images (AMIs)1. Build the Consul Server AMI in AWSGo to the~/deployment/scripts/amis/consul-server/directory and build the AMI by running:packer build .The output should look like this:==>  Builds finished. The artifacts of successful builds are:-->  amazon-ebs.ubuntu20-ami: AMIs were created:ap-southeast-1: ami-12345678910Make a note of the AMI ID. This will be used as part of the Terraform deployment.2. Build the Consul Client AMI in AWSGo to~/deployment/scripts/amis/consul-client/directory and build the AMI by running:packer build .The output should look like this:==>Builds finished. The artifacts of successful builds are:-->amazon-ebs.ubuntu20-ami: AMIs were created:ap-southeast-1: ami-12345678910Make a note of the AMI ID. This will be used as part of the Terraform deployment.Prepare the deploymentThere are a few steps that must be accomplished before applying the Terraform configuration.1. Update deployment variablesIn a text editor, go to directory~/deployment/Edit the variable filetemplate.var.tfvars.jsonby adding the variables values, including the AMI IDs previously built for the Consul Server and ClientNote: The key pair name should be entered without the “.pem” extension.2. Place the application file.jarin the root folder~/deployment/Deploy the solutionTo deploy the solution, run the following commands from the terminal:export  VAR_FILE=template.var.tfvars.jsonterraform init && terraform plan --var-file=$VAR_FILE -out plan.outterraform apply plan.outValidate the deploymentAll the EC2 instances have been deployed withAWS Systems Manageraccess, so you can connect privately to the terminal using theAWS Systems Manager Session Managerfeature.To connect to an instance:1. Select an instance2. Click Connect3. Go to Session Manager tabUsing Session Manager, connect to one of the Consul servers and run the following commands:consul membersThis command shows you the list of all Consul servers and clients connected to this cluster.consul members -wanThis command shows you the list of all Consul servers connected to this WAN environment.To see the Consul User Interface:1. Open your terminal and run:aws ssm start-session --target <instanceID> --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["8500"],"localPortNumber":["8500"]}' --region <region>WhereinstanceIDis the AWS Instance ID of one of the Consul servers, and Region is the AWS Region.UsingSystem Manager Port Forwardingallows you to connect privately to the instance via a browser.2. Open a browser and go tohttp://localhost:8500/ui3. Find the Management Token ID inAWS Secrets Managerin theAWS Management Console4. Login to the Consul UI using the Management Token IDTest the solutionConnect to the trading instance and query the different services:curl http://localhost:9090/v1/discover/service/pricercurl http://localhost:9090/v1/discover/service/static-dataThis deployment assumes that the Trading service queries the Pricer and Static-Data services, and that services are returned based on an order of precedence (see Table 1 following):ServicePrecedenceCustomerClusterLocationEnvironmentTRADING1ACMEALPHADC1DEVPRICER1ACMEALPHADC1DEVPRICER2ACMEALPHADC2DEVPRICER3ACMEBETADC1DEVPRICER4ACMEBETADC2DEVPRICER5SHAREDALPHADC1DEVPRICER6SHAREDALPHADC2DEVSTATIC-DATA1SHAREDSHAREDDC1DEVSTATIC-DATA2SHAREDSHAREDDC2DEVSTATIC-DATA2SHAREDBETADC2DEVSTATIC-DATA2SHAREDGAMMADC2DEVSTATIC-DATA-1STARKALPHADC1DEVSTATIC-DATA-1ACMEBETADC2PRODTable 1. Service order of precedenceTo test the solution, switch on and off services in the AWS Management Console and repeat Trading queries to look at where the traffic is being redirected.Cleaning upTo avoid incurring future charges, delete the solution from~/deployment/in the terminal:terraform destroy --var-file=$VAR_FILEConclusionIn this post, we outlined the prevalent challenge of complex globally distributed microservice architectures. We demonstrated how customers can build a hierarchical service discovery mechanism to support such an environment using a combination of Amazon EC2 service and third-party software such as SpringBoot and Consul. Use this to test this solution into your sandbox environment and to see if it could bring the answer to your current challenge.Additional resources:Instance metadata and user dataAmazon VPC-to-Amazon VPC connectivity optionsAWS Transit GatewayHashicorp ConsulConsul Enterprise LicenseMarine HaddadMarine is a Global Solutions Architect in the financial services industry (FSI) Prototyping and Cloud Engineering (PACE) team, focusing on building prototypes for customers and helping accelerate migrations to the cloud. She focuses mainly on Terraform scripting and infrastructure related topics.Tom PalmerTom Palmer is Global Head of E-Trading Strategy foreign exchange (FX) at Nomura, and leads a group of quants and technologists in London and Singapore responsible for all electronic FX flow at the firm. Tom has over 20 years of experience in technology roles across media and financial services companies, and has a degree in computing from the University of London.Tony PhanTony Phan is a Sr. Solutions Architecture Specialist at HashiCorp. He works closely with the product and core field teams of strategic customers across Asia-Pacific and Japan. He focuses on driving the adoption of emerging products, and discussing Consul and service meshes. Tony lives in Sydney, Australia, and in his free time, enjoys spending time with friends and family.George AgiasoglouGeorge is a Sr. Solutions Architect for the Prototyping and Cloud Engineering (PACE) team, based out of Singapore. He helps Global Financial Services (GFS) customers get unblocked on their cloud journey, through building prototypes and proof of concepts (POCs), workload modernization, and migration to the cloud.
These days, large organizations typically have microservices environments that span across cloud platforms, on-premises data centers, and colocation facilities. The reasons for this vary but frequently include latency, local support structures, and historic architectural decisions. However, due to the complex nature of these environments, efficient mechanisms for service discovery and configuration management must be implemented […]
Marine Haddad
Amazon EC2 Amazon VPC Architecture AWS Systems Manager AWS Transit Gateway