Deploying IBM Cloud Pak for integration on Red Hat OpenShift Service on AWS
AWS Architecture BlogDeploying IBM Cloud Pak for integration on Red Hat OpenShift Service on AWSby Eduardo Monich Fronza, Brian Olmeda, and Sankar Cherukuri | on07 NOV 2022| inAmazon Elastic Block Store (Amazon EBS),Amazon Elastic File System (EFS),Architecture|Permalink|ShareCustomers across many industries use IBM integration software, such asIBM MQ,DataPower,API Connect, andApp Connect, as the backbone that integrates and orchestrates their business-critical workloads.These customers often tell Amazon Web Services (AWS), they want to migrate their applications to AWS Cloud, as part of their business strategy: to lower costs, gain agility, and innovate faster.In this blog, we will explore how customers, who are looking at ways to run IBM software on AWS, can useRed Hat OpenShift Service on AWS (ROSA)to deployIBM Cloud Pak for Integration (CP4I)with modernized versions of IBM integration products.As ROSA is a fully managed OpenShift service that is jointly supported by AWS and Red Hat, plus managed by Red Hat site reliability engineers, customers benefit from not having to manage the lifecycle ofRed Hat OpenShift Container Platform (OCP)clusters.This post explains the steps to:Create a ROSA clusterConfigure persistent storageInstall CP4I and theIBM MQ 9.3operatorCloud Pak for integration architectureIn this blog, we are implementing a highly available ROSA cluster with three Availability Zones (AZ), three master nodes, three infrastructure nodes, and three worker nodes.Review the AWS documentation forRegions and AZsand theregions where ROSA is availableto choose the best region for your deployment.Figure 1 demonstrates the solution’s architecture.Figure 1. IBM Cloud Pak for Integration on ROSA architectureIn our scenario, we are building a public ROSA cluster, with an internet-facing Classic Load Balancer providing access to Ports 80 and 443. Consider using aROSA private clusterwhen you are deploying CP4I in your AWS account.We are usingAmazon Elastic File System (Amazon EFS)andAmazon Elastic Block Store (Amazon EBS)for our cluster’s persistent storage. Review theIBM CP4I documentationfor information about supported AWS storage options.ReviewAWS prerequisites for ROSAandAWS Security best practices in IAMdocumentation, before deploying CP4I for production workloads, to protect your AWS account and resources.CostYou are responsible for the cost of the AWS services used when deploying CP4I in your AWS account. For cost estimates, see the pricing pages for each AWS service you use.PrerequisitesBefore getting started, review the following prerequisites:This blog assumes familiarity with: CP4I, ROSA,Amazon Elastic Compute Cloud (Amazon EC2), Amazon EBS, Amazon EFS,Amazon Virtual Private Cloud,AWS Cloud9, andAWS Identity and Access Management (IAM)Access to an AWS account, with permissions to create the resources described in the installation steps sectionVerification of therequired AWS service quotasto deploy ROSA. If needed, you can request service quota increases from theAWS consoleAccess to anIBM entitlement API key: either a60-day trialor anexisting entitlementAccess to aRed Hat ROSA token; you can register on theRed Hat websiteto obtain oneAbastion hostto run the CP4I installation, we have used an AWS Cloud 9 workspace. You can use another device, provided it supports the required software packages:AWS Command Line Interface (aws cli)Red Hat OpenShift Service on AWS command-line interface (rosa)OpenShift command-line interface (oc)Kubernetes command-line tool (kubectl)The Linux utilities (jq,wget,gettext)Installation stepsTo deploy CP4I on ROSA, complete the following steps:From theAWS ROSA console, clickEnable ROSAto active the service on your AWS account (Figure 2).Figure 2. Enable ROSA on your AWS accountCreate anAWS Cloud9environment to run your CP4I installation. We used at3.smallinstance type.When it comes up, close theWelcome taband open a newTerminal tabto install the required packages:curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
wget https://mirror.openshift.com/pub/openshift-v4/clients/rosa/latest/rosa-linux.tar.gz
sudo tar -xvzf rosa-linux.tar.gz -C /usr/local/bin/

rosa download oc
sudo tar -xvzf openshift-client-linux.tar.gz -C /usr/local/bin/

sudo yum -y install jq gettextEnsure theELB service-linked roleexists in your AWS account:aws iam get-role --role-name 
"AWSServiceRoleForElasticLoadBalancing" || aws iam create-service-linked-role --aws-service-name 
"elasticloadbalancing.amazonaws.com"Create an IAM policynamed cp4i-installer-permissions with the following permissions:{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:*",
                "cloudformation:*",
                "cloudwatch:*",
                "ec2:*",
                "elasticfilesystem:*",
                "elasticloadbalancing:*",
                "events:*",
                "iam:*",
                "kms:*",
                "logs:*",
                "route53:*",
                "s3:*",
                "servicequotas:GetRequestedServiceQuotaChange",
                "servicequotas:GetServiceQuota",
                "servicequotas:ListServices",
                "servicequotas:ListServiceQuotas",
                "servicequotas:RequestServiceQuotaIncrease",
                "sts:*",
                "support:*",
                "tag:*"
            ],
            "Resource": "*"
        }
    ]
}Create an IAM role:SelectAWS serviceandEC2, then clickNext: Permissions.Select thecp4i-installer-permissionspolicy, and clickNext.Name itcp4i-installer, and clickCreate role.From your AWS Cloud9 IDE, click the grey circle button on the top right, and select Manage EC2 Instance (Figure 3).Figure 3. Manage the AWS Cloud9 EC2 instanceOn theAmazon EC2 console, select the AWS Cloud9 instance, then chooseActions / Security / Modify IAM Role.Choosecp4i-installerfrom theIAM Roledrop down, and clickUpdate IAM role(Figure 4).Figure 4. Attach the IAM role to your workspaceUpdate the IAM settings for your AWS Cloud9 workspace:aws cloud9 update-environment --environment-id $C9_PID --managed-credentials-action DISABLE
rm -vf ${HOME}/.aws/credentialsConfigure the following environment variables:export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')
export ROSA_CLUSTER_NAME=cp4iblog01Configure theaws clidefault region:aws configure set default.region ${AWS_REGION}Navigate to theRed Hat Hybrid Cloud Console, and copy yourOpenShift Cluster Manager API Token.Use the token and log in to your Red Hat account:rosa login --token=<your_openshift_api_token>Verify that your AWS account satisfies the quotas to deploy your cluster:rosa verify quotaWhen deploying ROSA for the first time, create the account-wide roles:rosa create account-roles --mode auto --yesCreate your ROSA cluster:rosa create cluster --cluster-name $ROSA_CLUSTER_NAME --sts \
  --multi-az \
  --region $AWS_REGION \
  --version 4.10.35 \
  --compute-machine-type m5.4xlarge \
  --compute-nodes 3 \
  --operator-roles-prefix cp4irosa \
  --mode auto --yes \
  --watchOnce your cluster is ready, create acluster-adminuser (it takes approximately 5 minutes):rosa create admin --cluster=$ROSA_CLUSTER_NAMELog in to your cluster using thecluster-admincredentials. You can copy the command from the output of the previous step. For example:oc login https://<your_cluster_api_address>:6443 \
  --username cluster-admin \
  --password<your_cluster-admin_password>Create an IAM policy allowing ROSA to use Amazon EFS:cat <<EOF > $PWD/efs-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
 {
   "Effect": "Allow",
   "Action": [
     "elasticfilesystem:DescribeAccessPoints",
     "elasticfilesystem:DescribeFileSystems"
   ],
   "Resource": "*"
 },
 {
   "Effect": "Allow",
   "Action": [
     "elasticfilesystem:CreateAccessPoint"
   ],
   "Resource": "*",
   "Condition": {
     "StringLike": {
       "aws:RequestTag/efs.csi.aws.com/cluster": "true"
     }
   }
 },
 {
   "Effect": "Allow",
   "Action": "elasticfilesystem:DeleteAccessPoint",
   "Resource": "*",
   "Condition": {
     "StringEquals": {
       "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
     }
   }
 }
  ]
}
EOF
POLICY=$(aws iam create-policy --policy-name "${ROSA_CLUSTER_NAME}-cp4i-efs-csi" --policy-document file://$PWD/efs-policy.json --query 'Policy.Arn' --output text) || POLICY=$(aws iam list-policies --query 'Policies[?PolicyName==`cp4i-efs-csi`].Arn' --output text)Create an IAM trust policy:export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
cat <<EOF > $PWD/TrustPolicy.json
{
  "Version": "2012-10-17",
  "Statement": [
 {
   "Effect": "Allow",
   "Principal": {
     "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
   },
   "Action": "sts:AssumeRoleWithWebIdentity",
   "Condition": {
     "StringEquals": {
       "${OIDC_PROVIDER}:sub": [
         "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-operator",
         "system:serviceaccount:openshift-cluster-csi-drivers:aws-efs-csi-driver-controller-sa"
       ]
     }
   }
 }
  ]
}
EOFCreate an IAM role with the previously created policies:ROLE=$(aws iam create-role \
  --role-name "${ROSA_CLUSTER_NAME}-aws-efs-csi-operator" \
  --assume-role-policy-document file://$PWD/TrustPolicy.json \
  --query "Role.Arn" --output text)
aws iam attach-role-policy \
  --role-name "${ROSA_CLUSTER_NAME}-aws-efs-csi-operator" \
  --policy-arn $POLICYCreate an OpenShift secret to store the AWS access keys:cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: aws-efs-cloud-credentials
  namespace: openshift-cluster-csi-drivers
stringData:
  credentials: |-
    [default]
    role_arn = $ROLE
    web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
EOFInstall theAmazon EFS CSI driveroperator:cat <<EOF | oc create -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  generateName: openshift-cluster-csi-drivers-
  namespace: openshift-cluster-csi-drivers
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/aws-efs-csi-driver-operator.openshift-cluster-csi-drivers: ""
  name: aws-efs-csi-driver-operator
  namespace: openshift-cluster-csi-drivers
spec:
  channel: stable
  installPlanApproval: Automatic
  name: aws-efs-csi-driver-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOFTrack the operator installation:watch oc get deployment aws-efs-csi-driver-operator \
 -n openshift-cluster-csi-driversInstall the AWS EFS CSI driver:cat <<EOF | oc apply -f -
apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
  name: efs.csi.aws.com
spec:
  managementState: Managed
EOFWait until the CSI driver is running:watch oc get daemonset aws-efs-csi-driver-node \
 -n openshift-cluster-csi-driversCreate a rule allowing inbound NFS traffic from your cluster’s VPC Classless Inter-Domain Routing (CIDR):NODE=$(oc get nodes --selector=node-role.kubernetes.io/worker -o jsonpath='{.items[0].metadata.name}')
VPC_ID=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].{VpcId:VpcId}' | jq -r '.[0][0].VpcId')
CIDR=$(aws ec2 describe-vpcs --filters "Name=vpc-id,Values=$VPC_ID" --query 'Vpcs[*].CidrBlock' | jq -r '.[0]')
SG=$(aws ec2 describe-instances --filters "Name=private-dns-name,Values=$NODE" --query 'Reservations[*].Instances[*].{SecurityGroups:SecurityGroups}' | jq -r '.[0][0].SecurityGroups[0].GroupId')
aws ec2 authorize-security-group-ingress \
  --group-id $SG \
  --protocol tcp \
  --port 2049 \
  --cidr $CIDR | jq .Create an Amazon EFS file system:EFS_FS_ID=$(aws efs create-file-system --performance-mode generalPurpose --encrypted --region ${AWS_REGION} --tags Key=Name,Value=ibm_cp4i_fs | jq -r '.FileSystemId')
SUBNETS=($(aws ec2 describe-subnets --filters "Name=vpc-id,Values=${VPC_ID}" "Name=tag:Name,Values=*${ROSA_CLUSTER_NAME}*private*" | jq --raw-output '.Subnets[].SubnetId'))
for subnet in ${SUBNETS[@]}; do
  aws efs create-mount-target \
    --file-system-id $EFS_FS_ID \
    --subnet-id $subnet \
    --security-groups $SG
doneCreate an Amazon EFS storage class:cat <<EOF | oc apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: $EFS_FS_ID
  directoryPerms: "750"
  gidRangeStart: "1000"
  gidRangeEnd: "2000"
  basePath: "/ibm_cp4i_rosa_fs"
EOFAdd the IBM catalog sources to OpenShift:cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM Operator Catalog
  image: 'icr.io/cpopen/ibm-operator-catalog:latest'
  publisher: IBM
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 45m
EOFGet the console URL of your ROSA cluster:rosa describe cluster --cluster=$ROSA_CLUSTER_NAME | grep ConsoleCopy your entitlement key from theIBM container software library.Log in to your ROSA web console, navigate toWorkloads > Secrets.Set the project toopenshift-config; locate and clickpull-secret(Figure 5).Figure 5. Edit thepull-secretentryExpandActionsand clickEdit Secret.Scroll to the end of the page, and clickAdd credentials(Figure 6):Registry server address: cp.icr.ioUsername field: cpPassword:your_ibm_entitlement_keyFigure 6. Configure your IBM entitlement key secretNext, navigate toOperators > OperatorHub. On theOperatorHubpage, use the search filter to locate the tile for the operators you plan to install:IBM Cloud Pak for IntegrationandIBM MQ. Keep all values as default for both installations (Figure 7). For example, IBM Cloud Pak for Integration:Figure 7. Install CP4I operatorsCreate a namespace for each CP4I workload that will be deployed. In this blog, we’ve created for the platform UI and IBM MQ:oc new-project integration
oc new-project ibm-mqReview theIBM documentationto select the appropriate license for your deployment.Deploy the platform UI:cat <<EOF | oc apply -f -
apiVersion: integration.ibm.com/v1beta1
kind: PlatformNavigator
metadata:
  name: integration-quickstart
  namespace: integration
spec:
  license:
    accept: true
    license: L-RJON-CD3JKX
  mqDashboard: true
  replicas: 3  # Number of replica pods, 1 by default, 3 for HA
  storage:
    class: efs-sc
  version: 2022.2.1
EOFTrack the deployment status, which takes approximately 40 minutes:watch oc get platformnavigator -n integrationCreate an IBM MQ queue manager instance:cat <<EOF | oc apply -f -
apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
  name: qmgr-inst01
  namespace: ibm-mq
spec:
  license:
    accept: true
    license: L-RJON-CD3JKX
    use: NonProduction
  web:
    enabled: true
  template:
    pod:
      containers:
        - env:
            - name: MQSNOAUT
              value: 'yes'
          name: qmgr
  queueManager:
    resources:
      limits:
        cpu: 500m
      requests:
        cpu: 500m
    availability:
      type: SingleInstance
    storage:
      queueManager:
        type: persistent-claim
        class: gp3
        deleteClaim: true
        size: 2Gi
      defaultClass: gp3
    name: CP4IQMGR
  version: 9.3.0.1-r1
EOFCheck the status of the queue manager:oc describe queuemanager qmgr-inst01 -n ibm-mqValidation stepsLet’s verify our installation!Run the commands to retrieve the CP4I URL and administrator password:oc describe platformnavigator integration-quickstart \
  -n integration | grep "^.*UI Endpoint" | xargs | cut -d ' ' -f3
oc get secret platform-auth-idp-credentials \
  -n ibm-common-services -o jsonpath='{.data.admin_password}' \
  | base64 -d && echoUsing the information from the previous step, access your CP4I web console.Select the option to authenticate with theIBM provided credentials (admin only)to login with your admin password.From the CP4I console, you can manage users and groups allowed to access the platform, install new operators, and view the components that are installed.Clickqmgr-inst01in theMessagingwidget to bring up your IBM MQ setup (Figure 8).Figure 8. CP4I console featuresIn theWelcome to IBM MQpanel, click theCP4IQMGRqueue manager. This shows the state, resources, and allows you to configure your instances (Figure 9).Figure 9. Queue manager detailsCongratulations! You have successfully deployed IBM CP4I on Red Hat OpenShift on AWS.Post installationReview the following topics, when you are installing CP4I on production environments:Configuring identity providerson ROSAConfigureidentity and access managementon CP4IDeploying instances of capabilities, like API Connect, App Connect, and DataPowerEnable auto scalingfor your ROSA clusterConfigure loggingandenable monitoringfor your ROSA clusterConsiderations for Amazon EFSto setup IBM MQ usingAmazon EFS storage classes.CleanupConnect to your Cloud9 workspace, and run the following steps to delete the CP4I installation, including ROSA. This avoids incurring future charges on your AWS account:EFS_EF_ID=$(aws efs describe-file-systems \
  --query 'FileSystems[?Name==`ibm_cp4i_fs`].FileSystemId' \
  --output text)
MOUNT_TARGETS=$(aws efs describe-mount-targets --file-system-id $EFS_EF_ID --query 'MountTargets[*].MountTargetId' --output text)
for mt in ${MOUNT_TARGETS[@]}; do
  aws efs delete-mount-target --mount-target-id $mt
done
aws efs delete-file-system --file-system-id $EFS_EF_ID

rosa delete cluster -c $ROSA_CLUSTER_NAME --yes --region $AWS_REGIONMonitor your cluster uninstallation logs, run:rosa logs uninstall -c $ROSA_CLUSTER_NAME --watchOnce the cluster is uninstalled, remove the operator-roles and oidc-provider, as informed in the output of the rosa delete command. For example:rosa delete operator-roles -c 1vepskr2ms88ki76k870uflun2tjpvfs --mode auto –yes
rosa delete oidc-provider -c 1vepskr2ms88ki76k870uflun2tjpvfs --mode auto --yesConclusionThis post explored how to deploy CP4I on AWS ROSA. We also demonstrated how customers can take full advantage of managed OpenShift service, focusing on further modernizing application stacks by using AWS managed services (like ROSA) for their application deployments.If you are interested in learning more about ROSA, take part in theAWS ROSA Immersion Workshop.Check out the blog onRunning IBM MQ on AWS using High-performance Amazon FSx for NetApp ONTAPto learn how to useAmazon FSx for NetApp ONTAPfor distributed storage and high availability with IBM MQ.For more information and getting started withIBM Cloud Pakdeployments, visit theAWS Marketplacefor new offerings.Further readingIBM on AWS Partner PageRed Hat OpenShift Service on AWS: architecture and networkingRed Hat OpenShift Service on AWS: private clusters with AWS PrivateLinkEduardo Monich FronzaEduardo Monich Fronza is a Partner Solutions Architect at AWS. His experience includes Cloud, solutions architecture, application platforms, containers, workload modernization, and hybrid solutions. In his current role, Eduardo helps AWS partners and customers in their cloud adoption journey.Brian OlmedaBrian Olmeda is a Senior Customer Success Manager Architect at IBM. He’s experienced in architecting, designing, and implementing via automation hybrid Cloud platforms. In his current role, he helps IBM customers as a technical expert ensuring their business goals are met with IBM software and services.Sankar CherukuriSankar Cherukuri is a AWS Partner Solutions Architect based in Singapore. His experience includes solution architecture, cloud migration, and modernization. In his current role, he helps AWS partners with technical strategy, enablement and architectural guidance that bring value to the end customers.
Customers across many industries use IBM integration software, such as IBM MQ, DataPower, API Connect, and App Connect, as the backbone that integrates and orchestrates their business-critical workloads. These customers often tell Amazon Web Services (AWS), they want to migrate their applications to AWS Cloud, as part of their business strategy: to lower costs, gain […]
Eduardo Monich Fronza
Amazon Elastic Block Store (Amazon EBS) Amazon Elastic File System (EFS) Architecture