Scaling DLT to 1M TPS on AWS: Optimizing a Regulated Liabilities Network
AWS Architecture BlogScaling DLT to 1M TPS on AWS: Optimizing a Regulated Liabilities Networkby Erica Salinas and Jack Iu | on31 JAN 2022| inAmazon EC2,Amazon Elastic Kubernetes Service,Amazon Managed Streaming for Apache Kafka (Amazon MSK),Architecture|Permalink|ShareSETLis an open source, distributed ledger technology (DLT) company that enables tokenisation, digital custody, and DLT for securities markets and payments. In mid-2021, they developed a blueprint for a Regulated Liabilities Network (RLN) that enables holding and managing a variety of tokenized value irrespective of its form. In a December 2021 collaboration withAmazon Web Services (AWS), SETL demonstrated that their RLN platform could support one million transactions per second. These findings were published in their whitepaper,The Regulated Liability Network: Whitepaper on scalability and performance.This AWS-hosted architecture scaled each processing component and the complete transaction flow. It was built usingAmazon Elastic Kubernetes Service (EKS),Amazon Elastic Cloud Compute (EC2), andAmazon Managed Streaming for Apache Kafka (MSK).This post discusses the technical implementation of the simulated network. We show how scaling characteristics were achieved, while maintaining the business requirements of atomicity and finality. We also discuss how each RLN component was optimized for high performance.Background: What is an RLN?InThe Regulated Internet of Value, Tony McLaughlin ofCitiproposed a single network to record ownership of multiple token types, each which represents a liability of a regulated entity. The RLN would give commercial banks, e-money providers, and central banks, the ability to issue liabilities and immutably record all balances and owners. Because it is designed to maintain a globally sequenced set of state changes, an unambiguous ledger of token balances can be computed and updated. Transactions, which result in two or more ledger updates, are proposed to the network. They are authorized by all impacted network participants, ordered for distribution to participant ledgers, and then persisted by multiple participant ledgers. This process is completed with five processing components.Figure 1. Core components and queues of RLNGiven the existing performance limitations faced by many DLT platforms, a main concern with the network design was its ability to meet throughput requirements. SETL collaborated with AWS to define an architecture that integrated decoupled processing components, as shown in Figure 2. The target throughput was 1,000,000 TPS for each component through the simulated network.Figure 2. Simulated RLN architecture in the AWS CloudA queue – process – queue architectural modelThe basic architectural model applied was “queue – process – queue.” Each component consumed transactions from one or more Kafka topics, performed its requisite activities, and produced output to a second Kafka topic. Components of the message flow used Amazon EKS, Amazon EC2, and Amazon MSK.Specific techniques were applied to achieve the scaling characteristics of components in the main message flow.A distinct EKS-managed node group was used for each RLN component. Each node within the group had its own EC2 instance that housed at most two Kubernetes Pods. This technique decreased potential network throttling and enabled the monitoring of pod resource utilization usingAmazon CloudWatchat the EC2 level. This aided in identifying bottlenecks, which can be challenging if large EC2 nodes house many Kubernetes Pods.Each RLN queue component had its own dedicated Kafka cluster. By isolating the Kafka topics to their own cluster, we were able to scale and monitor the cluster individually. This decreased potential chokepoints.A combination of eksctl, kubectl, andEC2 Auto Scaling groupswas used to deploy and horizontally scale each RLN component. The tooling enabled rapid results by controlling pod configuration, automating deployments, and supporting multiple performance testing iterations.Fine tuning RLN system componentsThe key metric tested was message throughput in transactions per second (TPS) consumed, processed, and produced for each component. An end-to-end test was performed, which measured throughput of the complete simulated network. Each RLN component was tuned to meet this metric as follows.TheTransaction Generatoracted as the customer entering transactions into the system (see Figure 3.) The Kafka producer property buffer.memory was changed to 536870912 to simulate an increase in producer TPS for each component.Figure 3. Simulated RLN Transaction GeneratorTheScheduler’s scaling technique was a dedicated Compute (one Pod/EC2 node) that listened to one Kafka partition in theTransaction Queueas seen in Figure 4. Additionally, turning on gzip compression (compression.type) on the producer resolved a network bottleneck for the downstream Kafka cluster.Figure 4. Simulated RLN SchedulerTheApprover’sscaling technique was like the Scheduler, however, theProposal Queuehad 10 topics as opposed to one. The approver is stateless, so it randomly selects a Kafka topic and partition to listen to. When scaling to 100 Kubernetes Pods, there would be roughly one Kafka partition per pod. As shown in Figure 5, each pod has its own EC2 instance. TheAssembler’sscaling techniques were the same as for the Scheduler and Approver components.Figure 5. Simulated RLN ApproverTheSequencer, as designed, cannot scale horizontally due to the nature of the sequencing algorithm. However, without any special tuning of the Kafka consumer configuration or any significant increase in EC2 sizing, it was able to achieve one million TPS. Its architecture is shown in Figure 6.Figure 6. Simulated RLN SequencerTheState Updaterscaled with each Kubernetes Pod listening to an assigned Approved Proposal topic (one-to-one mapping) and receiving all sequenced hashes from theHash Queue. Achieving 1 million TPS throughput required 10 nodes configured with c5.4xlarge, as seen in Figure 7.Figure 7. Simulated RLN State UpdaterSuccessful throughput testingTest results demonstrate that each component can sustain over 1 million TPS. While horizontal scaling was applied, even a single instance achieved over 10,000 TPS. Individual component performance test results can be seen in Table 1.ComponentSingle Instance (TPS)# of InstancesMultiple Instances (TPS)Generator~ 629,21921,258,438Scheduler~11,0191001,101,928Approver~10,3481001,034,868Assembler~10,6911001,069,100Sequencer~1,052,8451N/AState Updater~100,256101,002,569Table 1. Test results per individual componentNetwork tests were performed with all components integrated to create a simulated RLN network. The tests were executed with transaction generation of ~1,000,000 TPS. Throughput was measured at the output of the final component’s (State Updater) instances (see Figure 8). It was also measured in aggregate at over 1.2 million TPS (see Figure 9).Figure 8. Individual TPS for State Update component nodesFigure 9. Aggregate TPS for State Update component nodesWhile this study was designed to demonstrate capacity and throughput, the tests did replicate Kafka partitions across three Availability Zones in the selected AWS Region. Thus, testing demonstrated that the proposed architecture supports geographic resilience while meeting the throughput benchmark. Resiliency and security are areas of focus for testing in the next phases of architecting a production-ready version of RLN.Looking forwardDuring the month of joint work between the SETL and AWS technical teams, we stood up and tuned basic RLN functionality. We successfully demonstrated the scalability of the environment to at least 1M TPS. By using Kafka queues and the horizontal and vertical scalability of AWS services, we addressed the primary concern of reaching production-level throughput.The industry group collaborating on the RLN concept now has a solid technical foundation on which to build a production-ready RLN architecture. Future experimentation will include integration with financial institutions and technology partners that will provide the capabilities needed to build a successful RLN ecosystem.Further explorations include enabling digital signing, verification at scale, and expanding the network to include financial institution environments integrated with their own RLN partitions.Erica SalinasErica is a Principle Technical Program Manager and leads our digital asset initiatives within the worldwide public sector. She is part of a global team that supports central banks and international financial institutions as they seek to achieve their mandates in the midst of rapidly changing technology. Erica finds joy in applying emerging tech to gently bend history in the right direction.Jack IuJack is a Global Solutions Architect at AWS Financial Services. Jack is based in New York City, where he works with Financial Services customers to help them design, deploy, and scale applications to achieve their business goals. In his spare time, he enjoys badminton and loves to spend time with his wife and Shiba Inu.
SETL is an open source, distributed ledger technology (DLT) company that enables tokenisation, digital custody, and DLT for securities markets and payments. In mid-2021, they developed a blueprint for a Regulated Liabilities Network (RLN) that enables holding and managing a variety of tokenized value irrespective of its form. In a December 2021 collaboration with Amazon […]
Erica Salinas
Amazon EC2 Amazon Elastic Kubernetes Service Amazon Managed Streaming for Apache Kafka (Amazon MSK) Architecture