title: Node.JS WebSocket High Memory Usage
body: <p>We currently have a production node.js application that has been underperforming for a while. Now the application is a live bidding platform, and also runs timed auctions. The actual system running live sales is perfect and works as required. We have noticed that while running our timed sales (where items in a sale have timers and they incrementally finish, and if someone bids within the last set time, it will increment the time up X amount of seconds).</p>
  
  <p>Now the issue I have found is that during the period of a timed sale finishing (which can go on for hours) if items have 60 seconds between each lots and have extensions if users bid in the last 10 seconds. So we were able to connect via the devtools and I have done heap memory exports to see what is going on, but all I can see is that all indications point to stream writeable and the buffers. So my question is what am I doing wrong. See below a screenshot of a heap memory export:</p>
  
  <p><a href="https://i.stack.imgur.com/7Bkz6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7Bkz6.png" alt="heap-memory-snapshot"></a></p>
  
  <p>As you can see from the above, there is a lot of memory being used specifically for this it was using 1473MB of physical RAM. We saw this rise very quickly (within 30 mins) and each increment seemed to be more than the last. So when it hit 3.5GB it was incrementing at around 120MB each second, and then as it got higher around 5GB it was incrementing at 500MB per second and got to around 6GB and then the worker crashed (has a max heap size of 8GB), and then we were a process down.</p>
  
  <p>So let me tell you about the platform. It is of course a bidding platform as I said earlier, the platform uses Node (v11.3.0) and is clustered using the built in cluster library. It spawns 4 workers, and has the main process (so 5 altogether). The system accepts bids, checks other bids, calculates who is winning and essentially pushes updates to the connected clients via Redis PUB/SUB then that is broadcasted to that workers connected users.</p>
  
  <p>All data is stored within redis and mysql is used to refresh data into redis as redis has performed 10x faster than mysql was able to.</p>
  
  <p>Now the way this works is on connection a small session is created against the connection, this is then used to authenticate the user (which is a message sent from the client) all message events are sent to a handler which pushes it to the correct command these commands are then all set as async functions and run async.</p>
  
  <p>Now this has no issue on small scale, but we had over 250 connections and was seeing the above behaviour and are unsure where to find a fix. We noticed when opening the top obejct, it was connected to buffer.js and stream_writable.js as well. I can also see all references are connected to system / JSArrayBufferData and all refer back to these, there are lots of objects, and we are unable to fix this issue.</p>
  
  <p>We think one of the following:</p>
  
  <ol>
  <li><p>We log to file using append mode, which logs lots of information to the console and to a file using fs.writeFile and append mode. We did some research and saw that writing to console can be a cause of this kind of behaviour.</p></li>
  <li><p>It is the get lots function which outputs all the lots for that page (currently set to 50) every time an item finishes, so if the timer ends it will ask for a full page load for all the items on that page, instead of adding new lots in.</p></li>
  <li><p>There is something else happening here that we are unaware of, maybe the external library we are using that may not be removing a reference.</p></li>
  </ol>
  
  <p>I have listed the libraries of interest that we require here:</p>
  
  <ul>
  <li>"bluebird": "^3.5.1", (For promisifying the redis library)</li>
  <li>"colors": "^1.2.5", (Used on every console.log (we call logs for everything that happens this can be around 50 every few seconds.)</li>
  <li>"nodejs-websocket": "^1.7.1", (Our websocket library)</li>
  <li>"redis": "^2.8.0", (Our redis client)</li>
  </ul>
  
  <p>Anyway, if there is anything painstakingly obvious I would love to hear, as everything I have followed online and other stack overflow questions does not relate close enough to the issue we are facing.</p>