<h1>Motive</h1>

<p>I want to fully automate the deployment of many <strong>services</strong> with the help of <a href="https://cloud.google.com/cloud-build/" rel="noreferrer">Google Cloud Build</a> and <a href="https://cloud.google.com/kubernetes-engine/" rel="noreferrer">Google Kubernetes Engine</a>. Those services are located inside a <strong>monorepo</strong>, which has a folder called <code>services</code>.</p>

<p>So I created a <code>cloudbuild.yaml</code> for every service and created a build trigger. The <code>cloudbuild.yaml</code> does:</p>

<ol>
<li>run tests</li>
<li>build new version of Docker image</li>
<li>push new Docker image</li>
<li>apply changes to Kubernetes cluster</li>
</ol>

<h1>Issue</h1>

<p>As the number of services increases, the number of build triggers increases, too. There are also more and more services that are built even though they haven't changed.</p>

<p>Thus I want a mechanism, which has only <strong>one</strong> build trigger and automatically determines which services need to be rebuild.</p>

<h1>Example</h1>

<p>Suppose I have a monorepo with this file structure:</p>

<pre><code>├── packages
│   ├── enums
│   ├── components
└── services
    ├── backend
    ├── frontend
    ├── admin-dashboard
</code></pre>

<p>Then I make some changes in the <code>frontend</code> service. Since the <code>frontend</code> and the <code>admin-dashboard</code> service depend on the <code>components</code> package multiple services need to be rebuild:</p>

<ul>
<li>frontend</li>
<li>admin-dashboard</li>
</ul>

<p>But <strong>not</strong> backend!</p>

<h1>What I've Tried</h1>

<h3>(1) Multiple build triggers</h3>

<p>Setting up multiple build triggers for <strong>every</strong> service. But 80% of those builds are redundant, since most changes in the code are only related to individuals services. It's also increasingly complex to manage many build triggers, which look almost identical. A single <code>cloudbuild.yaml</code> file looks like this:</p>

<pre><code>steps:
  - name: "gcr.io/cloud-builders/docker"
    args:
      [
        "build",
        "-f",
        "./services/frontend/prod.Dockerfile",
        "-t",
        "gcr.io/$PROJECT_ID/frontend:$REVISION_ID",
        "-t",
        "gcr.io/$PROJECT_ID/frontend:latest",
        ".",
      ]
  - name: "gcr.io/cloud-builders/docker"
    args: ["push", "gcr.io/$PROJECT_ID/frontend"]

  - name: "gcr.io/cloud-builders/kubectl"
    args: ["apply", "-f", "kubernetes/gcp/frontend.yaml"]
    env:
      - "CLOUDSDK_COMPUTE_ZONE=europe-west3-a"
      - "CLOUDSDK_CONTAINER_CLUSTER=cents-ideas"
</code></pre>

<h3>(2) Looping through cloudbuild files</h3>

<p><a href="https://stackoverflow.com/questions/51861870">This</a> question is about a very similar issue. So I've tried to set up one "entry-point" <code>cloudbuild.yaml</code> file in the root of the project and looped through all services:</p>

<pre><code>steps:
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    for d in ./services/*/; do
      config="${d}cloudbuild.yaml"
      if [[ ! -f "${config}" ]]; then
        continue
      fi

      echo "Building $d ... "
      (
        gcloud builds submit $d --config=${config}
      ) &amp;
    done
    wait
</code></pre>

<p>This would eliminate the need for having multiple build triggers. But I also ran into issues with this method:</p>

<p>Every service is sent into it's own build process with the file scope of this particular service. This means, that I can only access files inside <code>/services/specific-service</code> during the build. Which is a total bummer for me (I need access to files in parent directories like <code>packages</code> and config files in the root).</p>

<h3>(3) Build only changed services</h3>

<p>Since I want a mechanism to only build changed services, I've tried to determine the services that need to be rebuilt. It seems quite easy to do this with the help of <a href="https://github.com/lerna/lerna" rel="noreferrer">lerna</a>. Running</p>

<pre><code>lerna changed --all --parseable
</code></pre>

<p>will return a list file paths to the changed packages like this:</p>

<pre><code>/home/username/Desktop/project/packages/components
/home/username/Desktop/project/services/frontend
/home/username/Desktop/project/services/admin-dashboard
</code></pre>

<p>However, the list also includes <code>packages</code> and I have no idea how I would be able to use this list in a script to loop through affected services. Also: when I trigger a build (e.g. through tagging a commit), lerna wouldn't be able to recognize changed packages during the build process as the changes have already been committed.</p>

<hr>

<p>I know this is a long one. But I think it's an important topic, so I really appreciate any help! </p>

<p>P.S.: <a href="https://github.com/flolude/cents-ideas/tree/develop" rel="noreferrer">This</a> is how my actual project looks like, if you want to take a close look at the specific use-case.</p>

Florian Ludewig
<p>To build from a <strong>monorepo</strong> you really want to build incrementally (what is changed and parts that depends on changed parts). To achieve this, your build tool need to handle a dependency graph in some way.</p>

<p>Lerna that you describe is designed for monorepos. But so is also <a href="https://bazel.build/" rel="nofollow noreferrer">Bazel</a> and it is available as an option in Google Cloud Builder, <a href="https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/bazel" rel="nofollow noreferrer">cloud-builders/bazel</a> with documentation for using in combination with a docker builder.</p>

<p>However, build tools designed for monorepos are usually more complex to setup.</p>
