title: 32-Bit Application runs out of Memory
body: <p>I have a memory problem. Since 12 Years we use in our software (C++, 32Bit) own made tables to store data.
  The Tables are stored on disk. When we want to use data of it, they are loaded in Memory and stay there. 
  Some tables are very big, they have more than 2 Million rows. When we load them into memory
  they up to 400MB. Due to 32bit and memory fragmentation we can actually load maximum 2 such tables int
  o memory before other operations did't get enough of memory.</p>
  
  <p>The software is installed on more the 3000 clients. The OS on the clients is win7-win10 (32Bit and 64Bit) and some insignificant XP and Vista Systems</p>
  
  <p>So we discussed a good (fast, propper ) way to get out of this problem. Here are some ideas:</p>
  
  <ul>
  <li>switching to 64Bit</li>
  <li>switching from our own tables to sqlite or ejdb</li>
  <li>opening every table in an own process and comunicate with the process to get data of the table</li>
  <li>extend our own tables that thay can read directly from disk</li>
  </ul>
  
  <p>All Ideas are more or less propper, practicable and fast (implementing speed and execution speed). The
  advantages and disadvantages of every idea ist very complicated and will go beyond the scope of this.</p>
  
  <p>Has someone another good idea to solve this problem?</p>
  
  <hr>
  
  <p>[update]</p>
  
  <p>I will try to explain this from a different angle. First the software is 
  installed on a wide base of different windows OS. From XP to W10 on all
  kind of computers. The software can be used on single desktops as on terminal
  servers with a central LAN data pool (only a folder on a file-server).
  It collects articles in a special way. So there are a lot of informations about
  all kind of article data and also price informations from different vendors. 
  So there is a big need for hiding/crypt this information to outsiders.</p>
  
  <p>The current database is like an in-memory table of strings, doubles or long data values. 
  Each row can contain a different set of columns. But most of the tables are like
  a structured database table. The whole table data is crypted and zipped in one block.
  Once loaded the whole data is expanded in memory where we can access this data very fast.
  If an index is needed, we do this with a std::map inside the software.
  We tried to compare our current table data against SQLite and EJDB. A file which contains
  about half a million simple article data takes 3.5 MB in our data, 28MB in SQLite and 100 MB (in
  several files) on EJDB. SQLite and EJDB shows the data in plain strings ore simple binary parts
  of as example "double". So with a good editor you can match an article number with a price very
  easy.</p>
  
  <p>The software uses about 40 DLLs with several dependencys of third-party libs. So switching from
  32 to 64 BIT is a challenge. Also does it not solve our problems with 32Bit terminal servers by our
  client installations.</p>
  
  <p>Going to a real database (like MySQL, MongoDB etc.) is a big challenge too as we freqently update our
  data every month on the wide base of computers. There is not allway a internet connection to use a
  real server client modell.</p>
  
  <p>So what can we do?</p>
  
  <p>Use SQLite or EJDB or something else and crypt our data in each field ?</p>
  
  <p>Reprogramm our database so it uses smaller chunks of data which leafs on this and loaded the 
  chunks on demand as they were neaded ?
  Only the indexes are in memory. Manage the disk-data maybe with a B-Tree strategie.</p>
  
  <p>Time is short. So reinventing the wheel does not help. What would you do or use in such
  a situation ?</p>