<p>A new version of MLFlow (1.23) provided a <code>--serve-artifacts</code> option (via <a href="https://github.com/mlflow/mlflow/pull/5045" rel="nofollow noreferrer">this</a> pull request) along with some example code. This <em>should</em> allow me to simplify the rollout of a server for data scientists by only needing to give them one URL for the tracking server, rather than a URI for the tracking server, URI for the artifacts server, and a username/password for the artifacts server. At least, that's how I understand it.</p>
<p>A complication that I have is that I need to use <code>podman</code> instead of <code>docker</code> for my containers (and without relying on <code>podman-compose</code>). I ask that you keep those requirements in mind; I'm aware that this is an odd situation.</p>
<p>What I did before this update (for MLFlow 1.22) was to create a kubernetes play yaml config, and I was successfully able to issue a <code>podman play kube ...</code> command to start a pod and from a different machine successfully run an experiment and save artifacts after setting the appropriate four env variables. I've been struggling with getting things working with the newest version.</p>
<p>I am following the <code>docker-compose</code> example provided <a href="https://github.com/mlflow/mlflow/tree/master/examples/mlflow_artifacts" rel="nofollow noreferrer">here</a>. I am trying a (hopefully) simpler approach. The following is my kubernetes play file defining a pod.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: &quot;2022-01-14T19:07:15Z&quot;
  labels:
    app: mlflowpod
  name: mlflowpod
spec:
  containers:
  - name: minio
    image: quay.io/minio/minio:latest
    ports:
    - containerPort: 9001
      hostPort: 9001
    - containerPort: 9000
      hostPort: 9000
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: /data
      name: minio-data
    args:
    - server
    - /data
    - --console-address
    - :9001

  - name: mlflow-tracking
    image: localhost/mlflow:latest
    ports:
    - containerPort: 80
      hostPort: 8090
    resources: {}
    tty: true
    env:
      - name: MLFLOW_S3_ENDPOINT_URL
        value: http://127.0.0.1:9000
      - name: AWS_ACCESS_KEY_ID
        value: minioadmin
      - name: AWS_SECRET_ACCESS_KEY
        value: minioadmin
    command: [&quot;mlflow&quot;]
    args:
      - server
      - -p 
      - 80
      - --host 
      - 0.0.0.0
      - --backend-store-uri 
      - sqlite:///root/store.db
      - --serve-artifacts
      - --artifacts-destination 
      - s3://mlflow
      - --default-artifact-root 
      - mlflow-artifacts:/
#      - http://127.0.0.1:80/api/2.0/mlflow-artifacts/artifacts/experiments
      - --gunicorn-opts 
      - &quot;--log-level debug&quot;
    volumeMounts:
    - mountPath: /root
      name: mlflow-data  

  volumes:
  - hostPath:
      path: ./minio
      type: Directory
    name: minio-data
  - hostPath:
      path: ./mlflow
      type: Directory
    name: mlflow-data
status: {}
</code></pre>
<p>I start this with <code>podman play kube mlflowpod.yaml</code>. On the same machine (or a different one, it doesn't matter), I have cloned and installed <code>mlflow</code> into a virtual environment. From that virtual environment, I set an environmental variable <code>MLFLOW_TRACKING_URI</code> to <code>&lt;name-of-server&gt;:8090</code>. I then run the <code>example.py</code> file in the <a href="https://github.com/mlflow/mlflow/tree/master/examples/mlflow_artifacts" rel="nofollow noreferrer"><code>mlflow_artifacts</code></a> example directory. I get the following response:</p>
<pre><code>....
botocore.exceptions.NoCredentialsError: Unable to locate credentials
</code></pre>
<p>Which seems like the client needs the server credentials to minIO, which I thought the proxy was supposed to take care of.</p>
<p>If I also provide the env variables</p>
<pre><code>$env:MLFLOW_S3_ENDPOINT_URL=&quot;http://&lt;name-of-server&gt;:9000/&quot; 
$env:AWS_ACCESS_KEY_ID=&quot;minioadmin&quot;
$env:AWS_SECRET_ACCESS_KEY=&quot;minioadmin&quot;
</code></pre>
<p>Then things work. But that kind of defeats the purpose of the proxy...</p>
<p>What is it about the proxy setup via kubernates play yaml and podman that is going wrong?</p>

<p>Just in case anyone stumbles upon this, I had same issue based on your description. However the problem on my side was that I was that I tried to test this with a preexisting experiment (default), and I did not create new one, so the old setting carried over, thus resulting in MLFlow trying to use s3 trough credentials and not https.</p>
<p>Hope this helps at least some of you out there.</p>
