<p>I'm new to Kubeflow and k8s. I have setup a single node k8s cluster and installed Kubeflow on this. I'm now trying the 'conditional pipeline' simple example from &quot;Kubeflow for Machine Learning&quot; book but I am getting &quot;cannot post /apis/v1beta1/experiments&quot; error ...</p>
<pre><code>Reason: Not Found
HTTP response headers: HTTPHeaderDict({'x-powered-by': 'Express', 'content-security-policy': &quot;default-src 'none'&quot;, 'x-content-type-options': 'nosniff', 'content-type': 'text/html; charset=utf-8', 'content-length': '164', 'date': 'Fri, 11 Jun 2021 20:47:13 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy'})
HTTP response body: &lt;!DOCTYPE html&gt;
&lt;html lang=&quot;en&quot;&gt;
&lt;head&gt;
&lt;meta charset=&quot;utf-8&quot;&gt;
&lt;title&gt;Error&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;pre&gt;Cannot POST /apis/v1beta1/experiments&lt;/pre&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>
<p>Any pointer what could be going wrong?</p>
<p>If I do &quot;kubectl get -A svc | grep dashboard&quot;, I only see Kubeflow central dashboard. Could this error be related to k8s dashboard not running?</p>
<p>This is the example I am trying:</p>
<p><a href="https://github.com/intro-to-ml-with-kubeflow/intro-to-ml-with-kubeflow-examples/blob/master/pipelines/ControlStructures.ipynb" rel="nofollow noreferrer">https://github.com/intro-to-ml-with-kubeflow/intro-to-ml-with-kubeflow-examples/blob/master/pipelines/ControlStructures.ipynb</a></p>
<p>Before this I've tried below MNIST example also and I faced exact same issue -
<a href="https://github.com/anjuls/fashion-mnist-kfp-lab/blob/master/KF_Fashion_MNIST.ipynb" rel="nofollow noreferrer">https://github.com/anjuls/fashion-mnist-kfp-lab/blob/master/KF_Fashion_MNIST.ipynb</a></p>
<p>Finally, I tried to modify the kfp.Client() line to following:
kfp.Client(host='http://127.0.0.1:8001').create_run_from_pipeline_func(conditional_pipeline, arguments={})</p>
<p>After this I'm getting error related to 'healtz' -</p>
<pre><code>MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))
</code></pre>
<p>I did following:
kubectl logs ml-pipeline-ui-7ddcd74489-xrss8 -c ml-pipeline-ui -n kubeflow</p>
<p>It seems ml-pipeline is running on http://localhost:3000. So I modified the client call to following:
client = kfp.Client(host='http://localhost:3000')</p>
<p>I still get an error - this time &quot;connection refused&quot;.</p>
<pre><code>MaxRetryError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7fffd1ca0f28&gt;: Failed to establish a new connection: [Errno 111] Connection refused',))
</code></pre>

<p>I've made some progress. However, issues are not fully resolved but I can at least proceed with client creation, do health check, list existing pipelines.</p>
<p>I could find ml-pipeline service is running on following internal IP:</p>
<p>kubeflow               service/ml-pipeline         ClusterIP      172.19.31.229</p>
<p>I then used this IP in kfp.Client() API - this resulted in RBAC access issue. I then patched my k8s with following with some hint from another issue -</p>
<pre><code>apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: &quot;OFF&quot;
</code></pre>
<p>This resolved issues I was facing with kfp.Client(). But now, I'm facing below error when I try to create_experiment():</p>
<pre><code>ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Wed, 16 Jun 2021 22:40:54 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {&quot;error&quot;:&quot;Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.&quot;,&quot;message&quot;:&quot;Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.&quot;,&quot;code&quot;:3,&quot;details&quot;:[{&quot;@type&quot;:&quot;type.googleapis.com/api.Error&quot;,&quot;error_message&quot;:&quot;Invalid resource references for experiment. ListExperiment requires filtering by namespace.&quot;,&quot;error_details&quot;:&quot;Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.&quot;}]}
</code></pre>
