Field Notes: Building Multi-Region and Multi-Account Tools with AWS Organizations
AWS Architecture BlogField Notes: Building Multi-Region and Multi-Account Tools with AWS Organizationsby Cody Penta | on11 OCT 2021| inArchitecture,AWS CloudFormation,AWS Organizations,Technical How-to|Permalink|ShareThis blog post was updated November 19, 2021.It’s common to start with a single AWS account when you are beginning your cloud journey with AWS. Running operations such as creating, reading, updating, and deleting resources in a single AWS account can be straightforward with AWS application program interfaces (APIs). Because an organization grows, so does their account strategy, often splitting workloads across multiple accounts. Fortunately, AWS customers can useAWS Organizationsto group these accounts into logical units, also known as organizational units (OUs), to apply common policies and deploy standard infrastructure. However, this will result in an increased difficulty to run an API against all accounts, moreover, every Region that account could use. How does an organization answer these questions:What is everyAmazon FSxbackup I own?How can I do an on-demand batch job that will apply to my entire organization?What is every internet access point across my organization?This blog post shows us how we can use Organizations,AWS Single Sign-On(AWS SSO),AWS CloudFormation StackSets, and various AWS APIs to effectively build multi-account and multi-region tools that can address use cases like the ones above.Running an AWS API sequentially across hundreds of accounts—potentially, many Regions—could take hours, depending on the API you call. An important aspect we will cover throughout this solution is the importance of concurrency for these types of tools.Overview of solutionFor this solution, we have created a fictional organization called Tavern that is set up with multiple organizational units (OUs), accounts, and Regions, to reflect a real-world scenario.Figure 1. Organization configuration exampleWe will set up a user with multi-factor authentication (MFA) enabled so we can sign-in and access an admin user in the management account. Using this admin user, we will deploy a stack set across the organization that enables this user to assume limited permissions into each child account.Next, we will use theGoprogramming language because of its native concurrency capabilities. More specifically, we will implement thepipeline concurrency patternto build a multi-account and multi-region tool that will run APIs across our entire AWS footprint.Additionally, we will add two common edge cases:We block mass API actions to an account in a suspended OU (not pictured) and the management account.We block API actions in disabled regions.This will show us how to implement robust error handling in equally powerful tooling.WalkthroughLet us separate the solution into distinct steps:Create an automation user through AWS SSO.This user can optionally be an IAM user or role assumed into by a third-party identity provider (such as, Azure Active Directory). Note the ARN of this identity because that is the key piece of information we will use for crafting a policy document.Deploy a CloudFormation stack set across the organization that enables this user to assume limited access into each account.For this blog post, we will deploy an organization-wide role with `ec2:DescribeRouteTables` permissions. Feel free to expand or change the permission set based on the type of tool you build.Using Go, AWS Command Line Interface (CLI) v2, and AWS SDK for Go v2:Authenticate using AWS SSO.List every account in the organization.Assume permissions into that account.Run an API across every Region in that account.Aggregate results for every Region.Aggregate results for every account.Report back the result.For additional context, review thisGitHub repositorythat contains all code and assets for this blog post.PrerequisitesFor this walkthrough, you should have the following prerequisites:MultipleAWS accountsAWS OrganizationsAWS SSO (optional)AWS SDK for Go v2AWS CLI v2Go programming knowledge (preferred), especially Go’s concurrency modelGeneral programming knowledgeCreate an automation user in AWS SSOThe first thing we need to do is create an identity to sign into. This can either be an AWS Identity and Access Management (IAM) user, an IAM role integrated with a third-party identity provider, or—in this case—an AWS SSO user.Log into theAWS SSO user console.PressAdd userbutton.Fill in the appropriate information.Figure 2. AWS SSO create userAssign the user to the appropriate group. In this case, we will assign this user to AWSControlTowerAdmins.Figure 3. Assigning SSO user to a groupVerify the user was created. (Optionally: enable MFA).Figure 4. Verifying User Creation and MFADeploy a stack set across your organizationTo effectively run any API across the organization, we need to deploy a common role that our AWS SSO user can assume across every account. We can use AWS CloudFormation StackSets to deploy this role at scale.Write the IAM role and associated policy document. The following is an example AWS Cloud Development Kit (AWS CDK) code for such a role. Note thatorgAccount,roleName, andssoUserin the below code will have to be replaced with your own values.const role = new iam.Role(this, 'TavernAutomationRole', {
      roleName: 'TavernAutomationRole',
      assumedBy: new iam.ArnPrincipal(`arn:aws:sts::${orgAccount}:assumed-role/${roleName}/${ssoUser}`),
    })
    role.addToPolicy(new PolicyStatement({
      actions: ['ec2:DescribeRouteTables'],
      resources: ['*']
    }))Log into theCloudFormation StackSets console.PressCreate StackSetbutton.Upload the CloudFormation template containing the common role to be deployed to the organization by the preferred method.Specify name and optional description.Add any standard organization tags, and chooseService-managed permissionsoption.ChooseDeploy to organization, and decide whether to disable or enable automatic deployment and appropriate account removal behavior. For this blog post, we choose to enable automatic deployment and accounts should remove the stack with removed from the target OU.ForSpecify regions, chooseUS East (N.Virginia). Note, because this stack contains only an IAM role, and IAM is a global service, region choice has no effect.ForMaximum concurrent accounts, choosePercent, and enter100(this stack is not dependent on order).ForFailure tolerance, chooseNumber, and enter5, account deployment failures before a total rollback happens.ForRegion Concurrency, chooseSequential.Review your choices, note the deployment target (should be r-*), and acknowledge that CloudFormation might create IAM resources with custom names.Press theSubmitbutton to deploy the stack.Configure AWS SSO for the AWS CLITo use our organization tools, we must first configure AWS SSO locally. With the AWS CLI v2, we can run:aws configure ssoTo configure credentials:Run the preceding command in your terminal.Follow the prompted steps.Specify your AWS SSO Start URL:AWS SSO Region:Authenticate through the pop-up browser window.Navigate back to the CLI, and choose the management account (this is where our principle for IAM originates).Specify the default client region.Specify the default output format.Note the CLI profile name. Regardless if you choose to go with the autogenerated one or the custom one, we need this profile name for our upcoming code.Start coding to utilize the AWS SSO shared profileAfter AWS SSO is configured, we can start coding the beginning part of our multi-account tool. Our first step is to list every account belonging to our organization.var (
    stsc    *sts.Client
    orgc    *organizations.Client
    ec2c    *ec2.Client
    regions []string
)

// init initializes common AWS SDK clients and pulls in all enabled regions
func init() {
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithSharedConfigProfile("tavern-automation"))
    if err != nil {
        log.Fatal("ERROR: Unable to resolve credentials for tavern-automation: ", err)
    }

    stsc = sts.NewFromConfig(cfg)
    orgc = organizations.NewFromConfig(cfg)
    ec2c = ec2.NewFromConfig(cfg)

    // NOTE: By default, only describes regions that are enabled in the management account, not all Regions
    resp, err := ec2c.DescribeRegions(context.TODO(), &ec2.DescribeRegionsInput{})
    if err != nil {
        log.Fatal("ERROR: Unable to describe regions", err)
    }

    for _, region := range resp.Regions {
        regions = append(regions, *region.RegionName)
    }
    fmt.Println("INFO: Listing all enabled regions:")
    fmt.Println(regions)
}

// main constructs a concurrent pipeline that pushes every account ID down
// the pipeline, where an action is concurrently run on each account and
// results are aggregated into a single json file
func main() {
    var accounts []string

    paginator := organizations.NewListAccountsPaginator(orgc, &organizations.ListAccountsInput{})
    for paginator.HasMorePages() {
        resp, err := paginator.NextPage(context.TODO())
        if err != nil {
            log.Fatal("ERROR: Unable to list accounts in this organization: ", err)
        }

        for _, account := range resp.Accounts {
            accounts = append(accounts, *account.Id)
        }
    }
    fmt.Println(accounts)Implement concurrency into our codeWith a slice of every AWS account, it’s time to concurrently run an API across all accounts. We will use some familiarGo concurrency patterns, as well as fan-out and fan-in.// ... continued in main

    // Begin pipeline by calling gen with a list of every account
    in := gen(accounts...)

    // Fan out and create individual goroutines handling the requested action (getRoute)
    var out []<-chan models.InternetRoute
    for range accounts {
        c := getRoute(in)
        out = append(out, c)
    }

    // Fans in and collect the routing information from all go routines
    var allRoutes []models.InternetRoute
    for n := range merge(out...) {
        allRoutes = append(allRoutes, n)
    }In the preceding code, we called a gen() function that started construction of our pipeline. Let’s take a deeper look into this function.// gen primes the pipeline, creating a single separate goroutine
// that will sequentially put a single account id down the channel
// gen returns the channel so that we can plug it in into the next
// stage
func gen(accounts ...string) <-chan string {
    out := make(chan string)
    go func() {
        for _, account := range accounts {
            out <- account
        }
        close(out)
    }()
    return out
}We see that gen just initializes the pipeline, and then starts pushing account ID’s down the pipeline one by one.The next two functions are where all the heavy lifting is done. First, let’s investigate `getRoute()`.// getRoute queries every route table in an account, including every enabled region, for a
// 0.0.0.0/0 (i.e. default route) to an internet gateway
func getRoute(in <-chan string) <-chan models.InternetRoute {
    out := make(chan models.InternetRoute)
    go func() {
        for account := range in {
            role := fmt.Sprintf("arn:aws:iam::%s:role/TavernAutomationRole", account)
            creds := stscreds.NewAssumeRoleProvider(stsc, role)

            for _, region := range regions {
                localCfg := aws.Config{
                    Region:      region,
                    Credentials: aws.NewCredentialsCache(creds),
                }

                localEc2Client := ec2.NewFromConfig(localCfg)

                paginator := ec2.NewDescribeRouteTablesPaginator(localEc2Client, &ec2.DescribeRouteTablesInput{})
                for paginator.HasMorePages() {
                    resp, err := paginator.NextPage(context.TODO())
                    if err != nil {
                        fmt.Println("WARNING: Unable to retrieve route tables from account: ", account, err)
                        out <- models.InternetRoute{Account: account}
                        close(out)
                        return
                    }

                    for _, routeTable := range resp.RouteTables {
                        for _, r := range routeTable.Routes {
                            if r.GatewayId != nil && strings.Contains(*r.GatewayId, "igw-") {
                                fmt.Println(
                                    "Account: ", account,
                                    " Region: ", region,
                                    " DestinationCIDR: ", *r.DestinationCidrBlock,
                                    " GatewayId: ", *r.GatewayId,
                                )
    
                                out <- models.InternetRoute{
                                    Account:         account,
                                    Region:          region,
                                    Vpc:             routeTable.VpcId,
                                    RouteTable:      routeTable.RouteTableId,
                                    DestinationCidr: r.DestinationCidrBlock,
                                    InternetGateway: r.GatewayId,
                                }
                            }
                        }
                    }
                }
            }

        }
        close(out)
    }()
    return out
}A couple of key points to highlight are as follows:for account := range inWhen iterating over a channel, the current goroutine blocks, meaning we wait here until we get an account ID passed to us before continuing. We’ll keep doing this until our upstream closes the channel. In our case, our upstream closes the channel once it pushes every account ID down the channel.role := fmt.Sprintf("arn:aws:iam::%s:role/TavernAutomationRole", account)
creds := stscreds.NewAssumeRoleProvider(stsc, role)Here, we can reference our existing role that we deployed to every account and assume into that role with AWS Security Token Service (STS).for _, region := range regions {Lastly, when we have credentials into that account, we need to iterate over every region in that account to ensure we are capturing the entire global presence.These three key areas are how we build organization-level tools. The remaining code is calling the desired API and delivering the result down to the next stage in our pipeline, where we merge all of the results.// merge takes every go routine and "plugs" it into a common out channel
// then blocks until every input channel closes, signally that all goroutines
// are done in the previous stage
func merge(cs ...<-chan models.InternetRoute) <-chan models.InternetRoute {
    var wg sync.WaitGroup
    out := make(chan models.InternetRoute)

    output := func(c <-chan models.InternetRoute) {
        for n := range c {
            out <- n
        }
        wg.Done()
    }

    wg.Add(len(cs))
    for _, c := range cs {
        go output(c)
    }

    go func() {
        wg.Wait()
        close(out)
    }()
    return out
}At the end of the main function, we take our in-memory data structures representing our internet entry points and marshal it into a JSON file.// ... continued in main

    savedRoutes, err := json.MarshalIndent(allRoutes, "", "\t")
    if err != nil {
        fmt.Println("ERROR: Unable to marshal internet routes to JSON: ", err)
    }
    ioutil.WriteFile("routes.json", savedRoutes, 0644)With the code in place, we can run the code with `go run main.go` inside of your preferred terminal. The command will generate results like the following:// ... routes.json
    {
        "Account": "REDACTED",
        "Region": "eu-north-1",
        "Vpc": "vpc-1efd6c77",
        "RouteTable": "rtb-1038a979",
        "DestinationCidr": "0.0.0.0/0",
        "InternetGateway": "igw-c1b125a8"
    },
    {
        "Account": " REDACTED ",
        "Region": "eu-north-1",
        "Vpc": "vpc-de109db7",
        "RouteTable": "rtb-e042ce89",
        "DestinationCidr": "0.0.0.0/0",
        "InternetGateway": "igw-cbd457a2"
    },
    // ...Cleaning upTo avoid incurring future charges, delete the following resources:Stack set through the CloudFormation consoleAWS SSO user (if you created one)ConclusionCreating organization tools that answer difficult questions such as, “show me every internet entry point in our organization,” are possible using Organizations APIs and CloudFormation StackSets. We also learned how to use Go’s native concurrency features to build these tools that scale across hundreds of accounts.Further steps you might explore include:Visiting theGithub Repoto capture the full picture.Taking our sequential solution for iterating over Regions and making it concurrent.Exploring the possibility of accepting functions and interfaces in stages to generalize specific pipeline features.Thanks for taking the time to read, and feel free to leave comments.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesCody PentaCody Penta is a Solutions Architect at Amazon Web Services and is based out of Charlotte, NC. He has a focus in security and CDK and enjoys solving the really difficult problems in the technology world. Off the clock, he loves relaxing in the mountains, coding personal projects, and gaming.
This blog post was updated November 19, 2021.  It’s common to start with a single AWS account when you are beginning your cloud journey with AWS. Running operations such as creating, reading, updating, and deleting resources in a single AWS account can be straightforward with AWS application program interfaces (APIs). Because an organization grows, so […]
Cody Penta
Architecture AWS CloudFormation AWS Organizations Technical How-to