title: Very Large Mnesia Tables in Production
body: <p>We are using Mnesia as a primary Database for a very large system. Mnesia Fragmented Tables have behaved so well over the testing period. System has got about 15 tables, each replicated across 2 sites (nodes), and each table is highly fragmented. During the testing phase, (which focused on availability, efficiency and load tests), we accepted the Mnesia with its many advantages of complex structures will do for us, given that all our applications running on top of the service are Erlang/OTP apps. We are running Yaws 1.91 as the main WebServer.<br><br>
  For efficiently configuring Fragmented Tables, we used a number of references who have used mnesia in large systems:
  <br>
  These are: <b><a href="http://igorrs.blogspot.com/2010/05/mnesia-one-year-later.html" rel="noreferrer">Mnesia One Year Later Blog</a></b>, <b><a href="http://igorrs.blogspot.com/2010/05/mnesia-one-year-later-part-2.html" rel="noreferrer">Part 2 of the Blog</a></b>, <b><a href="http://igorrs.blogspot.com/2010/05/mnesia-one-year-later-part-3.html" rel="noreferrer">Followed it even here</a></b>, <b><a href="http://igorrs.blogspot.com/2009/11/consistent-hashing-for-mnesia-fragments.html" rel="noreferrer">About Hashing</a></b>. These blog posts have helped us fine tune here and there to a better performance.</p>
  
  <p>Now, the problem. Mnesia has table size limits, yes we agree. However, limits on number of fragments have not been mentioned anywhere. For performance reasons, and to cater for large data, about how many fragments would keep mnesia "okay" ?.</p>
  
  <p>In some of our tables, we have 64 fragments. with <code>n_disc_only_copies</code> set to the number of nodes in the cluster so that each node has a copy per fragment. This has helped us solve issues of mnesia write failure if a given node is out of reach at an instant. Also in the blog above, he suggests that <code>the number of fragments should be a power of 2</code>, this statement (he says) was investigated from the way mnesia does its hashing of records. We however need more explanation on this, and which power of two are being talked about here: 2,4,16,32,64,128,...?</p>
  
  <p>The system is intended to run on HP Proliant G6, containing Intel processors (2 processors, each 4 cores, 2.4 GHz speed each core, 8 MB Cache size), 20 GB RAM size, 1.5 Terabytes disk space. Now, 2 of these high power machines are in our disposal. System Database should be replicated across the two. Each server runs Solaris 10, 64 bit.</p>
  
  <p>At what number of fragments may mnesia's performance start to de-grade? Is it okay if we increase the number of fragments from 64 to 128 for a given table? how about 65536 fragments (2 ^ 16) ? How do we scale out our mnesia to make use of the Terabyte space by using fragmentation? </p>
  
  <p>Please do provide the answers to the questions and you may provide advice on any other parameters that may enhance the System.</p>
  
  <p>NOTE: All tables that are to hold millions of records are created in <code>disc_only_copies</code> type, so no RAM problems. The RAM will be enough for the few RAM Tables we run. Other DBMS like MySQL Cluster and CouchDB will also contain data and are using the same hardware with our Mnesia DBMS. MySQL Cluster is replicated across the two servers (each holding two NDB Nodes, a MySQL server), the Management Node being on a different HOST.</p>