title: How to handle large amounts of data memory-efficiently in JS
body: <p>While trying to handle offline StarDict dictionaries in JavaScript, I came across strange memory consumption behaviour of JavaScript (tested in Gecko, i.e. Firefox 27).</p>
  
  <p>Actually, I have a StarDict file *.syn of about 6.2 MB containing 313,608 uncompressed UTF-8 encoded strings plus some extra data. Then I use the code below to load this into my JavaScript as an array of 313,608 strings.</p>
  
  <p>I end up with appx. 48 MB of memory consumption. (Checked with about:memory; remember we started with 6.2 MB!) This value goes down to 37 MB when switching tabs and waiting for a few minutes. JavaScript works with UTF-16 internally, so the worst I expected was appx. 15 MB. But it's more than twice that much! I have no idea, what might have caused this and I don't know what to do in order to handle this amount of data in JavaScript more efficiently. Please give advice!</p>
  
  <p><strong>The code</strong> or <strong>How to reproduce this</strong></p>
  
  <p>Put this somewhere in your HTML's body:</p>
  
  <pre><code>&lt;input type="file" /&gt;
  </code></pre>
  
  <p>Then use this JavaScript onload code:</p>
  
  <pre><code>function getUintAt(arr, offs) {
      out = 0;
      for (var j = offs; j &lt; offs+4; j++) {
              out &lt;&lt;= 8;
              out |= arr[j] &amp; 0xff;
      }
      return out;
  }
  
  var arr = [],
      upload = document.getElementsByTagName('input')[0];
  
  upload.onchange = function (evt) {
      evt.preventDefault();
      var file = evt.target.files[0],
          reader = new FileReader();
      reader.readAsArrayBuffer(file);
      reader.onload = function (evt) {
          var buffer = evt.target.result,
              view = new Uint8Array(buffer),
              decoder = new TextDecoder("utf-8");
          for(var i = 0, j = 0; i &lt; view.length; i++) {
              if(view[i] == 0) {
                  var term = decoder.decode(view.subarray(j,i)),
                      extra = getUintAt(view, i+1);
                  arr.push({ term: term, extra: extra });
                  i += 5; j = i;
              }
          }
          console.log(arr.length);
      };
  };
  </code></pre>
  
  <p>and chose the *.syn file from <a href="http://tovotu.de/data/pape_gr-de.zip" rel="nofollow">http://tovotu.de/data/pape_gr-de.zip</a> for the file input element. You may test with similar StarDict dictionaries (i.e. that have big syn files!) and will get similar results: Processing a syn-file of size X? JavaScript will need about 7 or 8 times as much.</p>