<p>I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.</p>
<p>My setup:</p>
<ul>
<li>ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: <code>my-etcd-cluster.my-etcd-namespace.svc.cluster.local</code>)</li>
<li>I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh</li>
<li>I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD</li>
</ul>
<p>Here's my affected Go code:</p>
<pre class="lang-golang prettyprint-override"><code>cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{&quot;http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379&quot;},
    DialTimeout: 5 * time.Second,
    Username:    username,
    Password:    password,
})

if err != nil {
    return nil, fmt.Errorf(&quot;opening ETCD client failed: %v&quot;, err)
}
</code></pre>
<p>And here's an error I'm getting when <code>clientv3.New(...)</code> gets executed:</p>
<pre><code>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:&quot;2022-03-16T23:37:42.174Z&quot;,&quot;logger&quot;:&quot;etcd-client&quot;,&quot;caller&quot;:&quot;v3@v3.5.0/retry_interceptor.go:62&quot;,&quot;msg&quot;:&quot;retrying of unary invoker failed&quot;,
&quot;target&quot;:&quot;etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]&quot;,&quot;attempt&quot;:0,
&quot;error&quot;:&quot;rpc error: code = Unavailable desc = error reading from server: EOF&quot;}
...
1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {&quot;reconciler group&quot;: &quot;my-controller.something.io&quot;, &quot;reconciler kind&quot;: &quot;ETCDClient&quot;, &quot;name&quot;: &quot;test-svc-client&quot;, &quot;namespace&quot;: &quot;some-service&quot;, &quot;reconcile-etcd-client&quot;: &quot;some-service/test-svc-client&quot;}
1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {&quot;reconciler group&quot;: &quot;my-controller.something.io&quot;, &quot;reconciler kind&quot;: &quot;ETCDClient&quot;, &quot;name&quot;: &quot;test-svc-client&quot;, &quot;namespace&quot;: &quot;some-service&quot;, &quot;error&quot;: &quot;opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF&quot;}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
</code></pre>
<p>The same error happens when I'm passing some dummy, invalid credentials.</p>
<p>However, when I tried to access the database in a HTTP API manner:</p>
<pre class="lang-golang prettyprint-override"><code>postBody, _ := json.Marshal(map[string]string{
    &quot;name&quot;:     username,
    &quot;password&quot;: password,
})
responseBody := bytes.NewBuffer(postBody)

resp, err := http.Post(&quot;http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate&quot;, &quot;application/json&quot;, responseBody)
if err != nil {
    return ctrl.Result{}, fmt.Errorf(&quot;an error occured %w&quot;, err)
}
l.Info(fmt.Sprintf(&quot;code: %d&quot;, resp.StatusCode))
defer resp.Body.Close()
</code></pre>
<p>...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.</p>
<p>When I'm using port-forwarding of the ETCD service and accessing it locally, <code>clientv3.New()</code> and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.</p>
<p>EDIT:
I've also added a simple pod to try accessing my etcd db via etcdctl:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: my-controller-namespace
spec:
  containers:
  - name: etcdctl
    image: bitnami/etcd
    command:
    - sleep
    - infinity
</code></pre>
<p>When logged into the container via <code>kubectl exec</code>, I was able to access my db:</p>
<pre><code>$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user=&quot;user&quot; --password=&quot;password&quot; put foo bob
OK
</code></pre>
<p>I guess the problem is somewhere in the SDK?</p>

<p>Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (<a href="https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md" rel="nofollow noreferrer">https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md</a>):</p>
<p><a href="https://i.stack.imgur.com/W2W2W.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/W2W2W.jpg" alt="enter image description here" /></a></p>
