<p>I am running <code>kubernetes 1.9.4</code> on my <code>gke</code> cluster</p>

<p>I have two pods , <code>gate</code> which is trying to connect to <code>coolapp</code>, both written in <code>elixir</code></p>

<p>I am using <a href="https://github.com/bitwalker/libcluster/issues/54" rel="nofollow noreferrer">libcluster</a> to connect my nodes
I get the following error:</p>

<p><code>[libcluster:app_name] cannot query kubernetes (unauthorized): endpoints is forbidden: User "system:serviceaccount:staging:default" cannot list endpoints in the namespace "staging": Unknown user "system:serviceaccount:staging:default"</code></p>

<p>here is my config in <strong>gate</strong> under <code>config/prod</code>:</p>

<pre><code> config :libcluster,
 topologies: [
   app_name: [
     strategy: Cluster.Strategy.Kubernetes,
     config: [
       kubernetes_selector: "tier=backend",
       kubernetes_node_basename: System.get_env("MY_POD_NAMESPACE") || "${MY_POD_NAMESPACE}"]]]
</code></pre>

<p>here is my configuration:</p>

<p><strong>vm-args</strong></p>

<pre><code>## Name of the node
-name ${MY_POD_NAMESPACE}@${MY_POD_IP}
## Cookie for distributed erlang
-setcookie ${ERLANG_COOKIE}
# Enable SMP automatically based on availability
-smp auto
</code></pre>

<p><strong>creating the secrets:</strong></p>

<pre><code>kubectl create secret generic erlang-config --namespace staging --from-literal=erlang-cookie=xxxxxx
kubectl create configmap vm-config --namespace staging --from-file=vm.args
</code></pre>

<p><strong>gate/deployment.yaml</strong></p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gate
  namespace: staging
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: gate
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
      containers:
      - name: gate
        image: gcr.io/development/gate:0.1.7
        args:
          - foreground
        ports:
        - containerPort: 80
        volumeMounts:
        - name: config-volume
          mountPath: /beamconfig
        env:
        - name: MY_POD_NAMESPACE
          value: staging
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: RELEASE_CONFIG_DIR
          value: /beamconfig
        - name: ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: erlang-config
              key: erlang-cookie
      volumes:
      - name: config-volume
        configMap:
          name: vm-config
</code></pre>

<p><strong>coolapp/deployment.yaml:</strong></p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: coolapp
  namespace: staging
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: coolapp
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
     # volumes
      volumes:
      - name: config-volume
        configMap:
          name: vm-config
      containers:
      - name: coolapp
        image: gcr.io/development/coolapp:1.0.3
        volumeMounts:
        - name: secrets-volume
          mountPath: /secrets
          readOnly: true
        - name: config-volume
          mountPath: /beamconfig
        ports:
        - containerPort: 80
        args:
          - "foreground"
        env:
        - name: MY_POD_NAMESPACE
          value: staging
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: REPLACE_OS_VARS
          value: "true"
        - name: RELEASE_CONFIG_DIR
          value: /beamconfig
        - name: ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: erlang-config
              key: erlang-cookie
        # proxy_container
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
            "-instances=staging:us-central1:com-staging=tcp:5432",
            "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: cloudsql
            mountPath: /cloudsql
</code></pre>

<p><strong>The default <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/" rel="noreferrer">service account</a> for the <code>staging</code> namespace (in which apparently your Pods using libcluster are running) lacks <a href="https://kubernetes.io/docs/admin/authorization/rbac/" rel="noreferrer">RBAC</a> permissions to <em>get endpoints</em> in that namespace.</strong></p>

<p>Likely your application requires a number of other permissions (that are not mentioned in the above error message) to work correctly; identifying all such permissions is out of scope for SO.</p>

<p>A way to resolve this issue is to grant <strong>superuser permissions</strong> that service account. This is not a secure solution but a stop gap fix.</p>

<pre><code>$ kubectl create clusterrolebinding make-staging-sa-cluster-admin \
    --serviceaccount=staging:default \
    --clusterrole=cluster-admin

clusterrolebinding "make-staging-sa-cluster-admin" created
</code></pre>

<p>To grant the <strong>specific permission only</strong> (get endpoints in the staging namespace) you would need to create a <a href="https://kubernetes.io/docs/admin/authorization/rbac/#role-and-clusterrole" rel="noreferrer">Role</a> first:</p>

<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: some-permissions
  namespace: staging
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch"]
</code></pre>

<p>And create a <a href="https://kubernetes.io/docs/admin/authorization/rbac/#rolebinding-and-clusterrolebinding" rel="noreferrer">RoleBinding</a> for the default service account in the staging namespace:</p>

<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: give-default-sa-some-permissions
  namespace: staging
subjects:
- kind: ServiceAccount
  name: default
  namespace: staging
roleRef:
  kind: Role
  name: some-permissions
  apiGroup: rbac.authorization.k8s.io
</code></pre>
