<p>My team and I, we are trying to deploy a BERT NLP model to production using Kubernets and Kubeflow.</p>
<p>We almost got it. Everything was processed correctly and we've got our desired output as we can see in our log:</p>
<p><a href="https://i.stack.imgur.com/7GWgb.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7GWgb.jpg" alt="enter image description here" /></a></p>
<p><strong>But the process simply doesn't finish</strong> and the last line in our DAG keeps with the 'loading' icon, as you can see here:</p>
<p><a href="https://i.stack.imgur.com/Gv4S7.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Gv4S7.jpg" alt="enter image description here" /></a></p>
<p>Eventually, checking on Kubernetes Engine, we realized an 'attention' icon to our job with the description <strong>'containers with unready status: [main]'</strong>.</p>
<p><a href="https://i.stack.imgur.com/MWMWT.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MWMWT.jpg" alt="enter image description here" /></a></p>
<p>We can't find out what might be happening and we've being trying to find out for a whole week. I don't want to put everything we tried to not be more verbose on my post.</p>
<p>If you want to check more about our deployment code:</p>
<pre><code>import kfp
from kfp.components import OutputPath, InputPath

perform_query_op = kfp.components.create_component_from_func(
    perform_dataset_query,
    output_component_file='components/clf_perform_query.component.yaml',
    packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;google-cloud-bigquery&quot;])
treat_dataset_op = kfp.components.create_component_from_func(
    treatment_data,
    output_component_file=&quot;components/clf_treat_dataset.component.yaml&quot;,
    packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;gensim&quot;])
balance_categories_clf_op = kfp.components.create_component_from_func(
    balance_categories_clf,
    output_component_file=&quot;components/clf_balance_categories.component.yaml&quot;,
    packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;])
create_train_test_split_op = kfp.components.create_component_from_func(
    create_train_test_split,
    output_component_file=&quot;components/clf_create_train_test_split.component.yaml&quot;,
    packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;sklearn&quot;])
train_model_op = kfp.components.create_component_from_func(
    train_transformers_model,
    output_component_file=&quot;components/clf_train_transformers_model.component.yaml&quot;,
    packages_to_install=[&quot;pandas&quot;, &quot;pyarrow&quot;, &quot;gcsfs&quot;, &quot;sklearn&quot;, &quot;torch&quot;, &quot;torchvision&quot;, &quot;tqdm&quot;, &quot;transformers&quot;, &quot;google-cloud-storage&quot;],
    base_image='pytorch/pytorch:1.9.0-cuda10.2-cudnn7-runtime')

@kfp.dsl.pipeline()
def train_classifier_pipeline():
    # Step 1: get data
    queried = perform_query_op()

    # Step 2: data treatment
    treated = treat_dataset_op(queried.output)

    # Step 3: Filtering and balancing categories
    balanced = balance_categories_clf_op(treated.output)

    # Step 4: Split datasets
    splits_A = create_train_test_split_op(balanced.outputs[&quot;brand_A&quot;], &quot;brand_A&quot;)
    splits_B = create_train_test_split_op(balanced.outputs[&quot;brand_B&quot;], &quot;brand_B&quot;)
    splits_C = create_train_test_split_op(balanced.outputs[&quot;brand_C&quot;], &quot;brand_C&quot;)
    splits_D = create_train_test_split_op(balanced.outputs[&quot;brand_D&quot;], &quot;brand_D&quot;)

    # Step 5: Train transformers model
    train_model_op(splits_acom.outputs[&quot;train&quot;], splits_A.outputs[&quot;test&quot;], &quot;brand_A&quot;)\
        .set_memory_request('20G')\
        .set_memory_limit('23G')\
        .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
    train_model_op(splits_shop.outputs[&quot;train&quot;], splits_B.outputs[&quot;test&quot;], &quot;brand_B&quot;)\
        .set_memory_request('20G')\
        .set_memory_limit('23G')\
        .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
    train_model_op(splits_suba.outputs[&quot;train&quot;], splits_C.outputs[&quot;test&quot;], &quot;brand_C&quot;)\
        .set_memory_request('20G')\
        .set_memory_limit('23G')\
        .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')
    train_model_op(splits_soub.outputs[&quot;train&quot;], splits_D.outputs[&quot;test&quot;], &quot;brand_D&quot;)\
        .set_memory_request('20G')\
        .set_memory_limit('23G')\
        .set_gpu_limit('1').add_node_selector_constraint('cloud.google.com/gke-accelerator', 'nvidia-tesla-t4')

if __name__ == '__main__':
    from kfp.compiler.compiler import Compiler
    Compiler().compile(train_classifier_pipeline, &quot;components/train_classifier.pipeline.yaml&quot;)
    # Execute single run
    kfp_client = kfp.Client(host='my_path')  # PRD
    kfp_client.create_run_from_pipeline_func(train_classifier_pipeline, arguments={})
</code></pre>
<p>Any insights I will be grateful!</p>

Guilherme Giuliano Nicolau
<p>Without the logs it will be hard to determine the exact root cause. Seeing the piece of code where you indicate using T4 GPU nodes, the cause can be one of the described below:</p>
<ol>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus?_ga=2.132041740.-2000871821.1632153301&amp;_gac=1.190167385.1634247741.Cj0KCQjwqp-LBhDQARIsAO0a6aKfHj4LVumCjt_9pHZu4NIpjcDC9pznenO9oixNxro6jG_Ou7BcAc4aAsHrEALw_wcB#availability" rel="nofollow noreferrer">Availability</a></li>
</ol>
<p>There’s not enough resources in your zone, which is common when using GPUs. GPUs are available in specific regions and zones. When you request GPU quota, consider the regions in which you intend to run your clusters.</p>
<p>For a complete list of applicable regions and zones, refer to GPUs on Compute Engine.
You can also see GPUs available in your zone using the gcloud command-line tool. To see a list of all GPU accelerator types supported in each zone, run the following command:</p>
<p><code>gcloud compute accelerator-types list</code></p>
<ol start="2">
<li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus?_ga=2.132041740.-2000871821.1632153301&amp;_gac=1.190167385.1634247741.Cj0KCQjwqp-LBhDQARIsAO0a6aKfHj4LVumCjt_9pHZu4NIpjcDC9pznenO9oixNxro6jG_Ou7BcAc4aAsHrEALw_wcB#gpu_quota" rel="nofollow noreferrer">GPU quota</a></li>
</ol>
<p>Your project quota is not enough to satisfy your request (not enough resources allocated for your project to provision the requested GPU instances). Your GPU quota is the total number of GPUs that can run in your Google Cloud project. To create clusters with GPUs, your project must have sufficient GPU quota.</p>
<p>Your GPU quota should be at least equivalent to the total number of GPUs you intend to run in your cluster. If you enable cluster autoscaling, you should request GPU quota at least equivalent to your cluster's maximum number of nodes multiplied by the number of GPUs per node.</p>
<p>For example, if you create a cluster with three nodes that runs two GPUs per node, your project requires at least six GPU quota.</p>
<ol start="3">
<li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus?_ga=2.132041740.-2000871821.1632153301&amp;_gac=1.190167385.1634247741.Cj0KCQjwqp-LBhDQARIsAO0a6aKfHj4LVumCjt_9pHZu4NIpjcDC9pznenO9oixNxro6jG_Ou7BcAc4aAsHrEALw_wcB#limitations" rel="nofollow noreferrer">Limitations</a></li>
</ol>
<p>Before using GPUs on GKE, keep in mind the following limitations:</p>
<ul>
<li>You cannot add GPUs to existing node pools.</li>
<li>GPU nodes cannot be live migrated during maintenance events.</li>
<li>GPUs are only supported with general-purpose N1 machine types.</li>
<li>GPUs are not supported in Windows Server node pools.</li>
</ul>
<p>You can find the full list of aspects to have in count when using GPU on kubernetes cluster in this guide: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus?_ga=2.132041740.-2000871821.1632153301&amp;_gac=1.190167385.1634247741.Cj0KCQjwqp-LBhDQARIsAO0a6aKfHj4LVumCjt_9pHZu4NIpjcDC9pznenO9oixNxro6jG_Ou7BcAc4aAsHrEALw_wcB" rel="nofollow noreferrer">Running GPUs</a> which includes several other details including the ones described above</p>
