title: Apache Zeppelin on K8S Interpreters crash due to UnknownHostException
body: <h1>Hi, help needed!</h1>
  <p>I've deployed apache/zeppelin:0.9.0 on my k8s cluster, under <em>zeppelin</em> namespace, following the <a href="https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html" rel="nofollow noreferrer">documentation</a>, here is my zeppeling-server.yaml file:</p>
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: zeppelin-notebook
    namespace: zeppelin
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  status: {}
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: zeppelin-server-conf-map
  data:
    # 'serviceDomain' is a Domain name to use for accessing Zeppelin UI.
    # Should point IP address of 'zeppelin-server' service.
    #
    # Wildcard subdomain need to be point the same IP address to access service inside of Pod (such as SparkUI).
    # i.e. if service domain is 'local.zeppelin-project.org', DNS configuration should make 'local.zeppelin-project.org' and '*.local.zeppelin-project.org' point the same address.
    #
    # Default value is 'local.zeppelin-project.org' while it points 127.0.0.1 and `kubectl port-forward zeppelin-server` will give localhost to connects.
    # If you have your ingress controller configured to connect to `zeppelin-server` service and have a domain name for it (with wildcard subdomain point the same address), you can replace serviceDomain field with your own domain.
    #SERVICE_DOMAIN: local.zeppelin-project.org:8080
    SERVICE_DOMAIN: https://zeppelin.karstecsa.com.ar/
    ZEPPELIN_K8S_SPARK_CONTAINER_IMAGE: spark:2.4.5
    ZEPPELIN_K8S_CONTAINER_IMAGE: apache/zeppelin:0.9.0-SNAPSHOT
    ZEPPELIN_HOME: /zeppelin
    ZEPPELIN_SERVER_RPC_PORTRANGE: 12320:12322
    # default value of 'master' property for spark interpreter.
    SPARK_MASTER: k8s://https://kubernetes.zeppelin.svc
    # default value of 'SPARK_HOME' property for spark interpreter.
    SPARK_HOME: /spark
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: zeppelin-server-conf
  data:
    nginx.conf: |
      daemon off;
      worker_processes auto;
      events {
        worker_connections 1024;
      }
      http {
        map $http_upgrade $connection_upgrade {
          default upgrade;
          '' close;
        }
  
        # first server block will be default. Proxy zeppelin server.
        server {
          listen 80;
          location / {
            proxy_pass http://localhost:8080;
            proxy_set_header Host $host;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;
          }
        }
  
        # match request domain [port]-[service].[serviceDomain]
        # proxy extra service such as spark-ui
        server {
          listen 80;
          server_name &quot;~(?&lt;svc_port&gt;[0-9]+)-(?&lt;svc_name&gt;[^.]*)\.(.*)&quot;;
          location / {
            resolver 127.0.0.1:53 ipv6=off;
            proxy_pass http://$svc_name.NAMESPACE.svc:$svc_port;
            proxy_set_header Host $host;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;
  
            # redirect rule for spark ui. 302 redirect response misses port number of service domain
            proxy_redirect ~(http:[/]+[0-9]+[-][^-]+[-][^.]+)[^/]+(\/jobs.*) $1.SERVICE_DOMAIN$2;
          }
        }
      }
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: zeppelin-server
    labels:
      app.kubernetes.io/name: zeppelin-server
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: zeppelin-server
    strategy:
      type: RollingUpdate
    template:
      metadata:
        labels:
          app.kubernetes.io/name: zeppelin-server
      spec:
        serviceAccountName: zeppelin-server
        volumes:
        - name: nginx-conf
          configMap:
            name: zeppelin-server-conf
            items:
            - key: nginx.conf
              path: nginx.conf
        - name: zeppelin-server-notebook-volume
          persistentVolumeClaim:
            claimName: zeppelin-notebook
  #      - name: zeppelin-server-conf
  #        persistentVolumeClaim:
  #          claimName: zeppelin-conf
  #      - name: zeppelin-server-custom-k8s
  #        persistentVolumeClaim:
  #          claimName: zeppelin-k8s
        nodeSelector:
          kubernetes.io/hostname: worker01convergente
        containers:
        - name: zeppelin-server
          image: apache/zeppelin:0.9.0-SNAPSHOT
          command: [&quot;sh&quot;, &quot;-c&quot;, &quot;$(ZEPPELIN_HOME)/bin/zeppelin.sh&quot;]
          lifecycle:
            preStop:
              exec:
                # SIGTERM triggers a quick exit; gracefully terminate instead
                command: [&quot;sh&quot;, &quot;-c&quot;, &quot;ps -ef | grep org.apache.zeppelin.server.ZeppelinServer | grep -v grep | awk '{print $2}' | xargs kill&quot;]
          ports:
          - name: http
            containerPort: 8080
          - name: https
            containerPort: 8443
          - name: rpc
            containerPort: 12320
          env:
          - name: POD_UID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          envFrom:
          - configMapRef:
              name: zeppelin-server-conf-map
          volumeMounts:
          - name: zeppelin-server-notebook-volume     # configure this to persist notebook
            mountPath: /zeppelin/notebook
  #        - name: zeppelin-server-conf                # configure this to persist Zeppelin configuration
  #          mountPath: /zeppelin/conf
  #        - name: zeppelin-server-custom-k8s          # configure this to mount customized Kubernetes spec for interpreter
  #          mountPath: /zeppelin/k8s
        - name: zeppelin-server-gateway
          image: nginx:1.14.0
          command: [&quot;/bin/sh&quot;, &quot;-c&quot;]
          env:
          - name: SERVICE_DOMAIN
            valueFrom:
              configMapKeyRef:
                name: zeppelin-server-conf-map
                key: SERVICE_DOMAIN
          args:
            - cp -f /tmp/conf/nginx.conf /etc/nginx/nginx.conf;
              sed -i -e &quot;s/SERVICE_DOMAIN/$SERVICE_DOMAIN/g&quot; /etc/nginx/nginx.conf;
              sed -i -e &quot;s/NAMESPACE/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/g&quot; /etc/nginx/nginx.conf;
              cat /etc/nginx/nginx.conf;
              /usr/sbin/nginx
          volumeMounts:
            - name: nginx-conf
              mountPath: /tmp/conf
          lifecycle:
            preStop:
              exec:
                # SIGTERM triggers a quick exit; gracefully terminate instead
                command: [&quot;/usr/sbin/nginx&quot;, &quot;-s&quot;, &quot;quit&quot;]
        - name: dnsmasq  # nginx requires dns resolver for dynamic dns resolution
          image: &quot;janeczku/go-dnsmasq:release-1.0.5&quot;
          args:
            - --listen
            - &quot;127.0.0.1:53&quot;
            - --default-resolver
            - --append-search-domains
            - --hostsfile=/etc/hosts
            - --verbose
  ---
  kind: Service
  apiVersion: v1
  metadata:
    name: zeppelin-server
  spec:
    ports:
      - name: http
        port: 80
      - name: rpc            # port name is referenced in the code. So it shouldn't be changed.
        port: 12320
    selector:
      app.kubernetes.io/name: zeppelin-server
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: zeppelin-server
  ---
  kind: Role
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: zeppelin-server-role
  rules:
  - apiGroups: [&quot;&quot;]
    resources: [&quot;pods&quot;, &quot;services&quot;]
    verbs: [&quot;create&quot;, &quot;get&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;list&quot;, &quot;delete&quot;, &quot;watch&quot;]
  - apiGroups: [&quot;rbac.authorization.k8s.io&quot;]
    resources: [&quot;roles&quot;, &quot;rolebindings&quot;]
    verbs: [&quot;bind&quot;, &quot;create&quot;, &quot;get&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;list&quot;, &quot;delete&quot;, &quot;watch&quot;]
  ---
  kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: zeppelin-server-role-binding
  subjects:
  - kind: ServiceAccount
    name: zeppelin-server
  roleRef:
    kind: Role
    name: zeppelin-server-role
    apiGroup: rbac.authorization.k8s.io
  </code></pre>
  <h1>What I have tried</h1>
  <ul>
  <li>At first, trying to run a &quot;helloworld&quot; with python interpreter was impossible, due to there was no kubectl installed on zeppelin:0.9.0 docker image, so I built the 0.9.0-SNAPSHOT image adding kubectl installation.</li>
  <li>Now python interpreter pod and service appears on my dashboard, but container crashes with the following error:</li>
  </ul>
  <pre><code> INFO [2020-08-29 22:04:39,544] ({main} RemoteInterpreterServer.java[&lt;init&gt;]:161) - Starting remote interpreter server on port 0, intpEventServerAddress: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc:12320
  Exception in thread &quot;main&quot; org.apache.zeppelin.shaded.org.apache.thrift.transport.TTransportException: 
  java.net.UnknownHostException: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc
      at org.apache.zeppelin.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:226)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.&lt;init&gt;(RemoteInterpreterServer.java:167)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.&lt;init&gt;(RemoteInterpreterServer.java:152)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:321)
  Caused by: java.net.UnknownHostException: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:607)
      at org.apache.zeppelin.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:221)
      ... 3 more
  </code></pre>
  <ul>
  <li>Ok! So my zeppelin-server pod's name its the one that appears in python's interpreter log, <strong>zeppelin-server-6d7f46d8d8-m9jlv</strong>, <a href="https://i.stack.imgur.com/zzPgS.png" rel="nofollow noreferrer">as shown here!</a>.</li>
  </ul>
  <p>I don't know what else to do to make it work!
  Any help, hint or suggestion will be much appreciated! Also, if further information is needed, tell me and of course I'll provide it.</p>