<p>So I am having an issue with prefect-server UI connecting to my API within my kubernetes environment. I am using a nginx as my ingress.</p>
<p>I have my ingress configuration for the prefect-server service as so:</p>
<pre><code>https://myhost.com/prefect/ -&gt; http://localhost:3000/api
</code></pre>
<p>I am able to access the UI, but it saying it cannot connect to the api:</p>
<p><a href="https://i.stack.imgur.com/Ew42r.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ew42r.png" alt="enter image description here" /></a></p>
<p>deployment.yml for prefect-server:</p>
<pre><code>...
      - image: &lt;my registry&gt;
        imagePullPolicy: Always
        name: prefect-server
        command: [&quot;prefect&quot;, &quot;server&quot;, &quot;start&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--log-level&quot;, &quot;WARNING&quot;, &quot;--port&quot;, &quot;3000&quot;]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_UI_API_URL 
            value:  &quot;http://localhost:3000/api&quot;
</code></pre>
<p>deployment.yml for agent:</p>
<pre><code>...
      - image: &lt;my registry&gt;
        imagePullPolicy: Always
        name: prefect-agent
        command: [&quot;prefect&quot;, &quot;agent&quot;, &quot;start&quot;, &quot;--work-queue&quot;, &quot;dev&quot;]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_API_URL 
            value:  &quot;http://prefect-server.prefect.svc.cluster.local:3000/api&quot; #WORKS
</code></pre>
<p>Here is the output of <code>kubectl describe service -n prefect</code></p>
<pre><code>Name:              prefect-server
Namespace:         prefect
Labels:            app.kubernetes.io/managed-by=Helm
                   run=prefect-server
Annotations:       meta.helm.sh/release-name: prefect
                   meta.helm.sh/release-namespace: prefect
Selector:          run=prefect-server
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                11.11.11.11 &lt;redacted&gt;
IPs:               11.11.11.11 &lt;redacted&gt;
Port:              &lt;unset&gt;  80/TCP
TargetPort:        3000/TCP
Endpoints:         11.22.22.22:3000 &lt;redacted&gt;
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<p>The agent is able to connect to the service container with no issues. For the prefect-server PREFECT_UI_API_URL, I have tried:</p>
<pre><code>&quot;http://prefect-server.prefect.svc.cluster.local:3000/api&quot;
&quot;https://myhost.com/prefect/api&quot;
&quot;http://127.0.0.1:3000/api&quot;
</code></pre>
<p><strong>UPDATE:</strong>
Thank you VonC for your help, I think I am heading in the right direction. I made the changes to the PREFECT_UI_API_URL and ingress. Also had to change the agent url to <code>http://prefect-server.prefect.svc.cluster.local:3000/prefect/api</code>. However, the &quot;cannot to to Server API..&quot; error goes away but the page is still 404 when I visit <code>htts://myhost.com/prefect/flow-runs</code> endpoint.</p>
<p>Output for <code>kubectl describe ingress -n prefect</code></p>
<pre><code>Name:             prefect
Namespace:        prefect
Address:          X.X.X.50
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
TLS:
  /mysecret terminates 
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  myhost.com  
                                  /prefect       prefect-server:80 (X.X.X.131:3000)
                                  /prefect/api   prefect-server:3000 (X.X.X.131:3000)
Annotations:                      field.cattle.io/publicEndpoints:
                                    [{&quot;addresses&quot;:[&quot;X.X.X.50&quot;],&quot;port&quot;:80,&quot;protocol&quot;:&quot;HTTP&quot;,&quot;serviceName&quot;:&quot;prefect:prefect-server&quot;,&quot;ingressName&quot;:&quot;prefect:prefect&quot;,&quot;hostname&quot;...
                                  ingress.kubernetes.io/secure-backends: true
                                  ingress.kubernetes.io/ssl-redirect: true
                                  kubernetes.io/ingress.class: f5
                                  meta.helm.sh/release-name: prefect
                                  meta.helm.sh/release-namespace: prefect
                                  virtual-server.f5.com/balance: round-robin
                                  
                                  virtual-server.f5.com/http-port: 443
                                  virtual-server.f5.com/ip: X.X.X.50
                                  virtual-server.f5.com/partition: kubernetes
</code></pre>
<p>My ingress.yml:</p>
<pre><code>{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prefect
  annotations:
{{- if eq .Values.ingress.class &quot;f5&quot; }}
    {{- if .Values.ingress.tls }}
    ingress.kubernetes.io/secure-backends: &quot;true&quot;
    {{- end }}
    {{- $f5healthpath := printf &quot;%s%s&quot; .Values.ingress.hostname .Values.ingress.path }}
    {{- $f5path := trimPrefix &quot;/&quot; .Values.ingress.path }}
    {{- $f5partition := .Values.ingress.f5partition }}
    {{- $f5httpport := .Values.ingress.f5httpport}}
    {{- $f5ip := .Values.ingress.f5ip }}
    {{- $f5ssl := .Values.ingress.f5ssl }}
    {{- $f5class:= .Values.ingress.class}}
    {{- range $key, $value := .Values.ingress.annotations }}
    {{ $key }}: {{ $value | replace &quot;F5HEALTHPATH&quot; $f5healthpath | replace &quot;F5PATITION&quot; $f5partition | replace &quot;F5HTTPPORT&quot; $f5httpport | replace &quot;F5IP&quot; $f5ip | replace &quot;F5SSL&quot; $f5ssl | replace &quot;F5CLASS&quot; $f5class | replace &quot;F5PATH&quot; $f5path | quote }}
    {{- end }}
{{- else }}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$2
{{- end }}
spec:
  tls:
  - secretName: /mysecret
  rules:
  - host: &quot;myhost.com&quot;
    http:
      paths:
      - path: &quot;/prefect&quot;
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 80
      - path: &quot;/prefect/api&quot;
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 3000       
{{- end }}
</code></pre>
<p>I believe the issue may be within my ingress, I would like all the paths to the prefect-server have the prefix path <code>/prefect/</code></p>

cozzymotto
<p>Double-check the <a href="https://docs.prefect.io/2.10.20/concepts/settings/#prefect_api_url" rel="nofollow noreferrer"><code>PREFECT_UI_API_URL</code></a> in your <code>prefect-server</code> deployment.<br />
This variable is used by the Prefect UI to understand where to find the API.</p>
<p>However, using &quot;<code>localhost</code>&quot; or &quot;<code>127.0.0.1</code>&quot; or &quot;<code>prefect-server.prefect.svc.cluster.local</code>&quot; would only be applicable if the UI (browser) is running on the same machine where the server is running, which is generally not the case.</p>
<ul>
<li>When using &quot;<code>localhost</code>&quot; or &quot;<code>127.0.0.1</code>&quot;, the browser will try to connect to the local machine, not the server.<br />
In the context of Kubernetes, &quot;<code>localhost</code>&quot; or &quot;<code>127.0.0.1</code>&quot; will refer to the client's machine (the machine where the browser is running), not the machine where your Kubernetes cluster is running.</li>
<li>Similarly, &quot;<code>prefect-server.prefect.svc.cluster.local</code>&quot; is only reachable within the Kubernetes cluster network. This internal network is not exposed to the internet and therefore not accessible from a client machine outside of that network.</li>
</ul>
<p>So, when you are setting the <code>PREFECT_UI_API_URL</code> environment variable, you need to provide an external, publicly accessible URL that the browser can reach over the internet. This URL should point to the external endpoint that's been created for your Prefect API service, typically using an Ingress controller like nginx.<br />
Assuming you have set up ingress to route <code>https://myhost.com/prefect/api</code> to your Prefect API, you should use this URL as the value of <code>PREFECT_UI_API_URL</code>.</p>
<p>Try updating your <code>prefect-server</code> deployment like so:</p>
<pre class="lang-yaml prettyprint-override"><code>...
      - image: &lt;my registry&gt;
        imagePullPolicy: Always
        name: prefect-server
        command: [&quot;prefect&quot;, &quot;server&quot;, &quot;start&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--log-level&quot;, &quot;WARNING&quot;, &quot;--port&quot;, &quot;3000&quot;]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_UI_API_URL 
            value:  &quot;https://myhost.com/prefect/api&quot;
</code></pre>
<p>And you need to ensure that your ingress routes <code>https://myhost.com/prefect/api</code> to the Prefect API service in your Kubernetes cluster properly. Double-check your nginx ingress configuration, it should be something like:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prefect-ingress
spec:
  rules:
  - host: &quot;myhost.com&quot;
    http:
      paths:
      - pathType: Prefix
        path: &quot;/prefect/api&quot;
        backend:
          service:
            name: prefect-server
            port:
              number: 3000
</code></pre>
<p>Do replace <code>myhost.com</code> with your actual domain name and check if the service name and port number match your cluster configuration.</p>
<hr />
<p>From the updated information you have provided, it seems you have made good progress. However, you have correctly identified that there may still be an issue with your Ingress configuration. It seems like the nginx rewrite target annotation is missing, which is important when using the ingress controller to rewrite the requested URLs.</p>
<p>The annotation <a href="https://kubernetes.github.io/ingress-nginx/examples/rewrite/#deployment" rel="nofollow noreferrer"><code>nginx.ingress.kubernetes.io/rewrite-target: /$2</code></a> will replace the URL path from the incoming request to match the target service in the Kubernetes cluster. That annotation is important for a scenario where you want to add a prefix path to your service (in your case, &quot;<code>/prefect/</code>&quot;).</p>
<p>Consider this part of your <code>ingress.yaml</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>{{- else }}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$2
{{- end }}
</code></pre>
<p>That block seems to be intended for an nginx ingress, and the rewrite target is set correctly here for an nginx ingress. However, if you are using an F5 ingress as indicated by your configuration (<code>kubernetes.io/ingress.class: f5</code>), this block will not be applied, and the nginx-specific rewrite rules will not be effective.</p>
<p>If you are using the F5 ingress controller, it may not recognize the <code>nginx.ingress.kubernetes.io/rewrite-target: /$2</code> annotation, since it's specific to the nginx ingress controller. Therefore, the F5 ingress controller may not have the ability to rewrite the URL path as expected.</p>
<p>You may need to consult the <a href="https://clouddocs.f5.com/" rel="nofollow noreferrer">F5 documentation</a> or support for guidance on how to perform URL rewrites or equivalent configurations to achieve your goal. I would check if F5 provides any URL rewriting capabilities similar to nginx's <code>rewrite-target</code>: F5 may have a different mechanism or annotation for doing this, which would need to be configured accordingly to match your desired <code>/prefect/</code> path prefix.</p>
<p>If the F5 controller doesn't provide this functionality or if you find it easier to configure with nginx, then switching to the nginx ingress controller might be an option to consider.</p>
