<p>I'm using KUBEFLOW pipelines for training KERAS models with TF and I'm starting from a very simple one. </p>

<p>Model is training fine and the pipeline works properly, but I'm not able to use the output viewer for TENSORBOARD properly.
Reading from the <a href="https://www.kubeflow.org/docs/pipelines/metrics/output-viewer/#tensorboard" rel="nofollow noreferrer">documentation</a> it seems that by just adding a proper json file in the root path of the training container (<code>/mlpipeline-ui-metadata.json</code>) should be enough but even when I do so, nothing appears in the artifact section of my experiment run (while KERAS logs can be seen correctly). </p>

<p>Here's how I configured it:</p>

<p><strong>mlpipeline-ui-metadata.json</strong>  (added from the DOCKERFILE directly)</p>

<pre class="lang-sh prettyprint-override"><code>{
    "version": 1,
    "outputs": [
    {
        "type": "tensorboard",
        "source": "/tf-logs"  #Just a placeholder at the moment
    }
    ]
}
</code></pre>

<p><strong>pipeline</strong></p>

<pre class="lang-py prettyprint-override"><code>import kfp
from kfp import dsl

from kubernetes.client.models import V1EnvVar

def train_op(epochs,batch_size,dropout,first_layer_size,second_layer_size):
    dsl.ContainerOp(
        image='MY-IMAGE',
        name='my-train',
        container_kwargs={"image_pull_policy": "Always", 'env': [
            V1EnvVar('TRAIN_EPOCHS', epochs),
            V1EnvVar('TRAIN_BATCH_SIZE', batch_size),
            V1EnvVar('TRAIN_DROPOUT', dropout),
            V1EnvVar('TRAIN_FIRST_LAYER_SIZE', first_layer_size),
            V1EnvVar('TRAIN_SECOND_LAYER_SIZE', second_layer_size),
            ]},
        command=['sh', '-c', '/src/init_script.sh'],
    ).set_memory_request('2G').set_cpu_request('2')

@dsl.pipeline(
    name='My model pipeline',
    description='Pipeline for model training'
)
def my_model_pipeline(epochs,batch_size,dropout,first_layer_size,second_layer_size):

    train_task = train_op(epochs,batch_size,dropout,first_layer_size,second_layer_size)

if __name__ == '__main__':
    kfp.compiler.Compiler().compile(my_model_pipeline, 'my_model.zip')
</code></pre>

<p>I've already tried to access to the running POD (<code>kubectl exec ..</code>) and I verified that the file is actually in the right spot.</p>

<p>By the way I'm using KUBEFLOW v0.5</p>

<p><strong>TL;DR</strong>: The <code>source</code> section should point to a location on a shared storage, not the pod's local file system path</p>

<p>The <code>source</code> section in <code>mlpipeline-ui-metadata.json</code> should point to a location where the pipelines-ui pod can later reference it, i.e. it should be on a shared storage, s3 (if on AWS), mounted Kubernetes volume (if on-prem). </p>

<p>The way Kubeflow works is, at the end of the run it just zips <code>mlpipeline-ui-metadata.json</code> and stores it in a minio storage. When you click on the Artifacts section, the UI looks for this <code>source</code> section in the zipped json and tries to read the tf events files. If the tf events file are not moved to shared storage from the pod they won't be read since it's on the ephemeral pods file system alone. </p>
