<p>I'm using <a href="https://github.com/kubernetes/kubernetes/blob/7ef585be224dae4ec5deae0f135653116e21a6e0/cluster/addons/registry/README.md#expose-the-registry-on-each-node" rel="nofollow noreferrer">private Docker registry addon</a> in my kubernetes cluster, and I would like to expose port 5000 on each node to pull image from localhost:5000 easily. So I placed a pod manifest file <code>/etc/kubernetes/manifests/kube-registry-proxy.manifest</code> on every node to start a local proxy for port 5000. It works when I manually deployed kubernetes on bare metal ubuntu few months ago, but failed when I try kargo, the port 5000 not listening. </p>

<p>I'm using kargo with calico network plugin, the docker registry's configurations are:</p>

<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-system-kube-registry-pv
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /registry

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kube-registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2.5.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: kube-registry-pvc

apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    protocol: TCP
</code></pre>

<p>I have created a pod manifest file <code>/etc/kubernetes/manifests/kube-registry-proxy.manifest</code> before run kargo:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: kube-registry-proxy
  namespace: kube-system
spec:
  containers:
  - name: kube-registry-proxy
    image: gcr.io/google_containers/kube-registry-proxy:0.3
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
    env:
    - name: REGISTRY_HOST
      value: kube-registry.kube-system.svc.cluster.local
    - name: REGISTRY_PORT
      value: "5000"
    - name: FORWARD_PORT
      value: "5000"
    ports:
    - name: registry
      containerPort: 5000
      hostPort: 5000
</code></pre>

<p>kube-registry-proxy is running on all nodes, but nothing listen on port 5000. Some output:</p>

<pre><code>ubuntu@k8s15m1:~$ kubectl get all --all-namespaces | grep registry-proxy
kube-system   po/kube-registry-proxy-k8s15m1             1/1       Running             1          1h
kube-system   po/kube-registry-proxy-k8s15m2             1/1       Running             0          1h
kube-system   po/kube-registry-proxy-k8s15s1             1/1       Running             0          1h

ubuntu@k8s15m1:~$ docker ps | grep registry
756fcf674288        gcr.io/google_containers/kube-registry-proxy:0.3      "/usr/bin/run_proxy"     19 minutes ago      Up 19 minutes                           k8s_kube-registry-proxy.bebf6da1_kube-registry-proxy-k8s15m1_kube-system_a818b22dc7210ecd31414e328ae28e43_7221833c

ubuntu@k8s15m1:~$ docker logs 756fcf674288 | tail
waiting for kube-registry.kube-system.svc.cluster.local to come online
starting proxy

ubuntu@k8s15m1:~$ netstat -ltnp | grep 5000

ubuntu@k8s15m1:~$ curl -v localhost:5000/v1/
*   Trying 127.0.0.1...
* connect to 127.0.0.1 port 5000 failed: Connection refused
* Failed to connect to localhost port 5000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 5000: Connection refused

ubuntu@k8s15m1:~$ kubectl get po kube-registry-proxy-k8s15m1 --namespace=kube-system -o wide
NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
kube-registry-proxy-k8s15m1   1/1       Running   3          1h        10.233.69.64   k8s15m1

ubuntu@k8s15m1:~$ curl -v 10.233.69.64:5000/v1/
*   Trying 10.233.69.64...
* Connected to 10.233.69.64 (10.233.69.64) port 5000 (#0)
&gt; GET /v1/ HTTP/1.1
&gt; Host: 10.233.69.64:5000
&gt; User-Agent: curl/7.47.0
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 404 Not Found
&lt; Content-Type: text/plain; charset=utf-8
&lt; Docker-Distribution-Api-Version: registry/2.0
&lt; X-Content-Type-Options: nosniff
&lt; Date: Tue, 14 Mar 2017 16:41:56 GMT
&lt; Content-Length: 19
&lt;
404 page not found
* Connection #0 to host 10.233.69.64 left intact
</code></pre>

<p>I think there are a couple of things going on here.</p>

<p>Foremost, be aware that Kubernetes <code>Services</code> come in 3 flavors: <code>ClusterIP</code> (which is the default), <code>NodePort</code> (which sounds very much like what you were expecting to happen), and <code>LoadBalancer</code> (which I won't mention further, but the docs do).</p>

<p>I would expect that if you updated your <code>Service</code> to explicitly request <code>type: NodePort</code>, you'll get closer to what you had in mind (but be aware that unless you changed it, <a href="https://kubernetes.io/docs/user-guide/services/#type-nodeport" rel="nofollow noreferrer">NodePort ports are limited to 30000-32767</a>.</p>

<p>Thus:</p>

<p><code>
apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeRegistry"
spec:
  type: NodePort  # &lt;--- updated line
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    nodePort: 30500   # &lt;-- you can specify, or omit this
    protocol: TCP
</code></p>

<p>If you have an opinion about the port you want the Service to listen on, feel free to specify it, or just leave it off and Kubernetes will pick one from the available space.</p>

<p>I'm going to mention this next thing for completeness, but what I'm about to say is a bad practice, so please don't do it. You can also have the <code>Pods</code> listen directly on the actual TCP/IP stack of the <code>Node</code>, by <a href="https://kubernetes.io/docs/api-reference/v1.5/#containerport-v1" rel="nofollow noreferrer">specifying <code>hostPort</code></a>; so in your case, it would be <code>hostPort: 5000</code> right below <code>containerPort: 5000</code>, causing the <code>Pod</code> to behave like a normal <code>docker -p 5000:5000</code> command would. But doing that makes scheduling <code>Pods</code> a nightmare, so please don't.</p>

<p>Secondly, about your 404 from <code>curl</code>:</p>

<p>I'm going to assume, based on the output of your curl command that <code>10.233.69.x</code> is your Service CIDR, which explains why port 5000 responded with anything. The request was in the right spirit, but <code>/v1/</code> was an incorrect URI to attempt. The <a href="https://docs.docker.com/registry/spec/api/" rel="nofollow noreferrer">Docker Registry API docs</a> contains a section about <a href="https://docs.docker.com/registry/spec/api/#api-version-check" rel="nofollow noreferrer">checking it is a V2 API instance</a>. My favorite <code>curl</code> of a registry is <code>https://registry.example.com/v2/_catalog</code> because it will return the name of every image therein, ensuring that my credentials are correct, that the registry server is operating correctly, and so on.</p>

<p>I know it's a lot to take in, so if you feel I glossed over something, let me know and I'll try to address it. Good luck!</p>
