title: Tensorflow Object Detection on Google Cloud is very slow
body: <p>I'm using Tensorflow 2.0 Object Detection API to train a model following this guide (<a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_training_and_evaluation.md" rel="nofollow noreferrer">https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_training_and_evaluation.md</a>)</p>
  <p>The checkpoint I'm using is SSD ResNet50 V1 FPN 640x640 (RetinaNet50), which has the following config details:</p>
  <pre><code>model {
    ssd {
      num_classes: 7
      image_resizer {
        fixed_shape_resizer {
          height: 640
          width: 640
        }
      }
      feature_extractor {
        type: &quot;ssd_resnet50_v1_fpn_keras&quot;
        depth_multiplier: 1.0
        min_depth: 16
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 0.00039999998989515007
            }
          }
          initializer {
            truncated_normal_initializer {
              mean: 0.0
              stddev: 0.029999999329447746
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.996999979019165
            scale: true
            epsilon: 0.0010000000474974513
          }
        }
        override_base_feature_extractor_hyperparams: true
        fpn {
          min_level: 3
          max_level: 7
        }
      }
      box_coder {
        faster_rcnn_box_coder {
          y_scale: 10.0
          x_scale: 10.0
          height_scale: 5.0
          width_scale: 5.0
        }
      }
      matcher {
        argmax_matcher {
          matched_threshold: 0.5
          unmatched_threshold: 0.5
          ignore_thresholds: false
          negatives_lower_than_unmatched: true
          force_match_for_each_row: true
          use_matmul_gather: true
        }
      }
      similarity_calculator {
        iou_similarity {
        }
      }
      box_predictor {
        weight_shared_convolutional_box_predictor {
          conv_hyperparams {
            regularizer {
              l2_regularizer {
                weight: 0.00039999998989515007
              }
            }
            initializer {
              random_normal_initializer {
                mean: 0.0
                stddev: 0.009999999776482582
              }
            }
            activation: RELU_6
            batch_norm {
              decay: 0.996999979019165
              scale: true
              epsilon: 0.0010000000474974513
            }
          }
          depth: 256
          num_layers_before_predictor: 4
          kernel_size: 3
          class_prediction_bias_init: -4.599999904632568
        }
      }
      anchor_generator {
        multiscale_anchor_generator {
          min_level: 3
          max_level: 7
          anchor_scale: 4.0
          aspect_ratios: 1.0
          aspect_ratios: 2.0
          aspect_ratios: 0.5
          scales_per_octave: 2
        }
      }
      post_processing {
        batch_non_max_suppression {
          score_threshold: 9.99999993922529e-09
          iou_threshold: 0.6000000238418579
          max_detections_per_class: 100
          max_total_detections: 100
          use_static_shapes: false
        }
        score_converter: SIGMOID
      }
      normalize_loss_by_num_matches: true
      loss {
        localization_loss {
          weighted_smooth_l1 {
          }
        }
        classification_loss {
          weighted_sigmoid_focal {
            gamma: 2.0
            alpha: 0.25
          }
        }
        classification_weight: 1.0
        localization_weight: 1.0
      }
      encode_background_as_zeros: true
      normalize_loc_loss_by_codesize: true
      inplace_batchnorm_update: true
      freeze_batchnorm: false
    }
  }
  train_config {
    batch_size: 8
    data_augmentation_options {
      random_horizontal_flip {
      }
    }
    data_augmentation_options {
      random_crop_image {
        min_object_covered: 0.0
        min_aspect_ratio: 0.75
        max_aspect_ratio: 3.0
        min_area: 0.75
        max_area: 1.0
        overlap_thresh: 0.0
      }
    }
    sync_replicas: true
    optimizer {
      momentum_optimizer {
        learning_rate {
          cosine_decay_learning_rate {
            learning_rate_base: 0.03999999910593033
            total_steps: 25000
            warmup_learning_rate: 0.013333000242710114
            warmup_steps: 2000
          }
        }
        momentum_optimizer_value: 0.8999999761581421
      }
      use_moving_average: false
    }
    fine_tune_checkpoint: &quot;{MY_BUCKET_PATH}/data/ckpt-0&quot;
    num_steps: 25000
    startup_delay_steps: 0.0
    replicas_to_aggregate: 8
    max_number_of_boxes: 100
    unpad_groundtruth_tensors: false
    fine_tune_checkpoint_type: &quot;detection&quot;
    use_bfloat16: false
    fine_tune_checkpoint_version: V2
  }
  train_input_reader {
    label_map_path: &quot;{MY_BUCKET_PATH}/data/label_map.pbtxt&quot;
    tf_record_input_reader {
      input_path: &quot;{MY_BUCKET_PATH}/data/train.record&quot;
    }
  }
  eval_config {
    metrics_set: &quot;coco_detection_metrics&quot;
    use_moving_averages: false
  }
  eval_input_reader {
    label_map_path: &quot;{MY_BUCKET_PATH}/data/label_map.pbtxt&quot;
    shuffle: false
    num_epochs: 1
    tf_record_input_reader {
      input_path: &quot;{MY_BUCKET_PATH}/data/test.record&quot;
    }
  }
  
  </code></pre>
  <p>I'm trying to submit a job to ggcloud's ai platform with the following command:</p>
  <pre><code>gcloud ai-platform jobs submit training {my_job_id_here} \
      --runtime-version 2.1 \
      --python-version 3.5 \
      --job-dir={MY_BUCKET_PATH}/train \
      --package-path ./object_detection \
      --module-name object_detection.model_main_tf2 \
      --region us-east4 \
      -- \
      --model_dir={MY_BUCKET_PATH}/train \
      --pipeline_config_path={MY_BUCKET_PATH}/data/pipeline.config
  </code></pre>
  <p>However, the job runs very slowly (takes it 1 hr to run 100 steps), but I wonder if this is because I don't use GPU/TPU (since I'm learning on the free tier in which they give you free $300 credits), or is it due to any detail in my config file? Is there any way/tuning I can make to make it run faster?</p>