<p>Multi-trust deployment model from <a href="https://istio.io/latest/docs/ops/deployment/deployment-models/#trust-between-meshes" rel="nofollow noreferrer">istio documentation</a></p>
<p><a href="https://i.stack.imgur.com/0BX41.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0BX41.jpg" alt="multi trust diagram" /></a></p>
<p>I want to connect multiple meshes together. I currently manage 3 different AKS clusters</p>
<ul>
<li>Operations (aks-ops-euwest-1)</li>
<li>Staging (aks-stg-euwest-1)</li>
<li>Production (aks-prod-euwest-1)</li>
</ul>
<p>I have Hashicorp Vault running on Operations, I’d like to be able to reach eg. Postgres that’s running in Staging and Production using istio mTLS (for automatic secret rotation).</p>
<p>Each of the clusters are running istio (Multi-Primary) in different networks. Each cluster has a different ClusterName, MeshID, TrustDomain and NetworkID. Though, the <code>cacerts</code> secrets are configured with a <strong>common Root CA</strong></p>
<p><a href="https://i.stack.imgur.com/GKBAL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GKBAL.png" alt="diagram" /></a></p>
<p>From the <a href="https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/" rel="nofollow noreferrer">istio documentation</a>, to enable cross-cluster communication, a special <code>eastwestgateway</code> has to be deployed. The tlsMode is <code>AUTO_PASSTHROUGH</code>.</p>
<p>These are the environment variables for the eastwestgateway</p>
<pre><code># sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
- name: ISTIO_META_ROUTER_MODE
  value: &quot;sni-dnat&quot;
# traffic through this gateway should be routed inside the network
- name: ISTIO_META_REQUESTED_NETWORK_VIEW
  value: aks-ops-euwest-1
</code></pre>
<p><strong>I don’t want to enable automatic service discovery by sharing secrets between clusters.</strong> Why? Because I want a fine-grained control over which services are going to be exposed between the meshes. I want to be able to specify <code>AuthorizationPolicies</code> that point to serviceaccounts from the remote clusters (because different trust domains)</p>
<p>Eg:</p>
<pre><code># production cluster
kind: AuthorizationPolicy
spec:
  selector:
    matchLabels:
      app: postgres
  rules:
  - from:
      source:
        - principal: spiffe://operations-cluster/ns/vault/sa/vault
</code></pre>
<p>This is from the <a href="https://istio.io/latest/docs/ops/deployment/deployment-models/?_ga=2.106416323.1124150883.1618461826-1888041227.1611307002#endpoint-discovery-with-multiple-control-planes" rel="nofollow noreferrer">istio documentation</a></p>
<blockquote>
<p>In some advanced scenarios, load balancing across clusters may not be
desired. For example, in a blue/green deployment, you may deploy
different versions of the system to different clusters. In this case,
each cluster is effectively operating as an independent mesh. This
behavior can be achieved in a couple of ways:</p>
<ul>
<li><strong>Do not exchange remote secrets between the clusters. This offers the strongest isolation between the clusters.</strong></li>
<li>Use VirtualService and DestinationRule to disallow routing between two versions of the services.</li>
</ul>
</blockquote>
<p>What the istio documentation doesn't specify, is how to enable cross-cluster communication in the case where secrets are not shared. When sharing secrets, istiod will create additional envoy configuration that will allow pods to transparently communicate through the eastwestgateway. What it doesn't specify, is how to create those configurations manually when not sharing secrets.</p>
<p>The tlsMode is <code>AUTO_PASSTHROUGH</code>. Looking at the istio repository</p>
<pre><code>    // Similar to the passthrough mode, except servers with this TLS
    // mode do not require an associated VirtualService to map from
    // the SNI value to service in the registry. The destination
    // details such as the service/subset/port are encoded in the
    // SNI value. The proxy will forward to the upstream (Envoy)
    // cluster (a group of endpoints) specified by the SNI
    // value. This server is typically used to provide connectivity
    // between services in disparate L3 networks that otherwise do
    // not have direct connectivity between their respective
    // endpoints. Use of this mode assumes that both the source and
    // the destination are using Istio mTLS to secure traffic.
    // In order for this mode to be enabled, the gateway deployment
    // must be configured with the `ISTIO_META_ROUTER_MODE=sni-dnat`
    // environment variable.
</code></pre>
<p>The interesting part is <code>The destination details such as the service/subset/port are encoded in the SNI value</code>.</p>
<p>It seems that when sharing secrets between clusters, istio will add envoy configurations that will effectively encode these service/subset/port into the SNI value of an envoy cluster. Though, when secrets are not shared, how can we acheive the same result?</p>
<p>I've looked at <a href="https://github.com/istio-ecosystem/multi-mesh-examples" rel="nofollow noreferrer">this repository</a>, but it is outdated and does not make use of the <code>eastwestgateway</code>.</p>
<p>I've also posted questions on the istio forum <a href="https://discuss.istio.io/t/mesh-federation-without-automatic-service-discovery/10276" rel="nofollow noreferrer">here</a> and <a href="https://discuss.istio.io/t/demistify-mesh-federation-multi-cluster-communication-without-automatic-endpoint-discovery/10179" rel="nofollow noreferrer">here</a>, but it's difficult to get help from there.</p>

Ludovic C
<p>In my company we're working on a concept of a service mesh federation, based on the next points:</p>
<ul>
<li>Use a private CA to issue certificates for the &quot;client&quot; k8s cluster and a &quot;provider&quot; k8s cluster. Those certificates does not have to be used during the istio install. So it's possible to use any suitable certificates to install istio, and then generate the new certificates from a common private CA to be used specifically with the mesh federation (to be installed on ingress and egress gateways).</li>
<li>Setup mtls ingress for the &quot;provider&quot; k8s cluster, as described here: <a href="https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/#configure-a-mutual-tls-ingress-gateway" rel="nofollow noreferrer">https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/#configure-a-mutual-tls-ingress-gateway</a>, use a certificate from the common private CA to setup the gateway.</li>
<li>Setup mtls egress origination for the &quot;client&quot; k8s cluster, as described here: <a href="https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/#deploy-a-mutual-tls-server" rel="nofollow noreferrer">https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/#deploy-a-mutual-tls-server</a>, use a certificate from the common private CA to setup the gateway.</li>
<li>Use an external database/registry to store the service-to-service &quot;allowed&quot; connection entries.</li>
<li>Use an external UI and API to let &quot;clients&quot; to request a new service-to-service connection and let &quot;providers&quot; to approve such a request. After the approval the service-to-service record in the database marked as &quot;approved&quot;. (Assuming &quot;client&quot; and &quot;provider&quot; clusters owned and operated by different teams.)</li>
<li>Use some external automation at the &quot;client&quot; K8s cluster to retrieve the &quot;approved&quot; record(s) from the database and setup respectively the istio objects in that cluster (Service entry, egress gateway, virtual services, destination rules).</li>
<li>Use some external automation at the &quot;provider&quot; K8s cluster to retrieve the &quot;approved&quot; record(s) from the database and setup respectively the istio objects in that cluster (ingress gateway, virtual service).</li>
<li>Use Authorization Policy to explicitly allow only the specific clients/certificates to connect to ingress at &quot;provider&quot; k8s cluster, and disallow any other certificates (particularly, the other certificates issued by the same private CA) - described here <a href="https://my.f5.com/manage/s/article/K21084547" rel="nofollow noreferrer">https://my.f5.com/manage/s/article/K21084547</a></li>
</ul>
<p>The configuration might need to be adjusted when we add more &quot;providers&quot; and &quot;clients&quot; to the federation.</p>
<p>As a result, we get totally separate clusters and separate meshes, that can be operated by different teams, and we let the services (deployed to the clusters) to connect to services in other clusters using the certificates from the common private CA, and we call it &quot;mesh federation&quot;.
Each cluster and each component can be operated separately, updated independently. We don't need to cross-create any secrets in the clusters, as suggested in the istio multi-cluster tutorial. Only a limited number of services exposed or consumed in each cluster, so there is almost no impact on performance because of the mesh federation setup.
&quot;Client&quot; services in the clusters can connect to the &quot;provider&quot; services using http, then the request is intercepted by istio and sent to the respective httpS ingress host (i.e. the &quot;provider&quot; cluster), and mtls certificated is included to the request automatically. No need to distribute certificates to client services as they are added on egress gateways. Considering we can keep the &quot;permissive&quot; istio authentication configuration, the &quot;client&quot; services can continue communicating with the rest of the services in the cluster just as before, so adding the new services to the mesh federation should be easy to do, as we would not need to change any interconnection configuration on client or provider side.</p>
