title: Firebase Function timeout when calling set many times on realtime db
body: <p>In short: I have a scheduled firebase function that downloads a file, converts this into chunks of JSON and imports that into a realtime database. I've given the function 2GB of Memory and 540 Seconds (9 minutes, the maximum) to get its work done, but it still ends in a timeout about 50% of the time. There must be a leak, but it seems to be in my blind spot.</p>
  
  <p>In details:
  I have the following scheduled function, that runs once every hour. The function calls a method <code>updateDatabase</code>, which in turn, downloads a file about 35MB in size from an external server (using the method <code>getNewData</code>. This file is a whitespace-separated (kid you not) file containing 2500 rows and 2500 columns of data. I need each of these data points in a database to read them out quickly later. </p>
  
  <p>At first I simply converted this into json and tried to import that into the database, since 35MB didn't seem that big of a deal. However, this made the function run out of memory every single time. So I decided to split it up into smaller portions.</p>
  
  <p>So when the file is downloaded I first split it into rows. Then I loop over the rows, split that into columns and do a set on a realtime database on reference <code>/grid/[currentrow]</code>. This means I call set 2500 times each time. I've tried using an <code>await</code> for each row and using <code>Promise.all()</code> (the current version, but both sometimes just seem to hang at some point. I don't get any errors in the log, just a timeout when the 540 seconds have passed.</p>
  
  <p>The scheduled function:</p>
  
  <pre class="lang-js prettyprint-override"><code>exports.scheduledDataUpdate = functions
    .region("europe-west1")
    .runWith({
      timeoutSeconds: 540,
      memory: '2GB'
    })
    .pubsub.schedule("0 * * * *")
    .onRun(async () =&gt; {
      try {
        await updateDatabase();
        console.log('Database updated');
        return true;
      } catch(error) {
        console.error(error);
        return false;
      }
    });
  </code></pre>
  
  <p>which call the method <code>updateDatabase</code>:</p>
  
  <pre class="lang-js prettyprint-override"><code>async function updateDatabase() {
    let data;
    try {
      data = await getNewData().catch(err =&gt; console.error(err));
    } catch(error) {
      console.error(error);
      return null;
    }
    console.log('Data download complete');
  
    const lines = data.split("\n"); // split the data into rows (2500)
  
    lines.forEach((line, r) =&gt; {
      const cols = line.split(/\s+/); // split the row into columns (2500)
      dbUpdates.push(admin.database().ref(`/grid/${r}`).set(cols).then(() =&gt; {
        if(r &gt; 1 &amp;&amp; r % 500 === 0) {
          console.log(`updated row ${r}`); // just to get some info in logs whether some rows were processed
        }
        return true;
      }).catch((error) =&gt; {
        console.error(`Error updating row ${r} -- ${error}`);
      }));
      return true;
    });
  
    return await Promise.all(dbUpdates);
  </code></pre>
  
  <p>and, in case you are curious, the method <code>getNewData</code>:</p>
  
  <pre class="lang-js prettyprint-override"><code>const dataUrl = 'https://some.server/somefile'; // I guess you guess this is different in my code (it is)
  async function getNewData() {
    console.log('Start download of data');
    return await new Promise((resolve, reject) =&gt; {
      https.get(dataUrl, response =&gt; {
        if (response.statusCode === 200) {
          let data = "";
          return response
            .on("data", chunk =&gt; {
              data += chunk;
            })
            .on("end", () =&gt; {
              resolve(data);
            })
            .on("error", error =&gt; {
              reject(error);
              console.error(`error while downloading data: ${data}`);
            });
        } else {
          switch (response.statusCode) {
            case 301:
            case 302:
            case 303:
            case 307:
              console.warning(`Redirected to ${response.headers.location} [${response.statusCode}]`)
              return getNewData(response.headers.location);
            default:
              return reject(new Error(`Could not download new data (error ${response.statusCode})`));
          }
        }
      })
    });
  }
  </code></pre>
  
  <p>When I look at the quotas that firebase offers (I'm running on the Blaze, pay as you go subscription) this really shouldn't be that much of a problem to run. Clearly I'm missing something or making some stupid mistake, but for the life of me, I don't see it.</p>
  
  <p><strong>Update 1</strong>
  An example of log with timeout (asked by @FrankvanPuffelen)</p>
  
  <pre><code>7:15:03.224 p.m.     scheduledDataUpdate      Function execution started
  7:15:03.546 p.m.     scheduledDataUpdate      Start download of data
  7:15:08.035 p.m.     scheduledDataUpdate      Data download complete
  7:22:28.390 p.m.     scheduledDataUpdate      updated row 500
  7:24:03.244 p.m.     scheduledDataUpdate      Function execution took 540021 ms, finished with status: 'timeout'
  </code></pre>