title: Neo4J huge performance degradation after records added to spatial layer
body: <p>So I have around 70 million spatial records that i want to add to the spatial layer (I've tested with a small set and everything is smoothly, queries returning the same results as postgis and the layer operation seems fine)
  But when i try to add all the spatial records to the database, the performance degrades rapidly to the point that it gets really slow at around 5 million (around 2h running time) records and hangs at ~7.7 million (8 hours lapsed).</p>
  
  <p>Since the spatial index is an Rtree that uses the graph structure to construct itself, i am wondering why is it degrading when the number os records increase.
  Rtree insertions are O(n) if im not mistaken and thats why im concerned it might be something between the rearranging of bounding boxes, nodes that are not tree leaves that are causing the addToLayer process to get slower over time.</p>
  
  <p>Currently im adding nodes to the layer like that (lots of hardcoded stuff since im trying to figure out the problem before patterns and code style):</p>
  
  <pre><code>Transaction tx = database.beginTx();
      try {
  
          ResourceIterable&lt;Node&gt; layerNodes = GlobalGraphOperations.at(database).getAllNodesWithLabel(label);
          long i = 0L;
          for (Node node : layerNodes) {
              Transaction tx2 = database.beginTx();
              try {
                  layer.add(node);
                  i++;
                  if (i % commitInterval == 0) {
                      log("indexing (" + i + " nodes added) ... time in seconds: "
                              + (1.0 * (System.currentTimeMillis() - startTime) / 1000));
                  }
                  tx2.success();
              } finally {
                  tx2.close();
              }
          }
          tx.success();
      } finally {
          tx.close();
      }
  </code></pre>
  
  <p>Any thoughts ? Any ideas of how performance could be increased ?</p>
  
  <p>ps.: using java API
  Neo4j 2.1.2, Spatial 0.13
  Core i5 3570k @4.5Ghz, 32GB ram
  dedicated 2TB 7200 hard drive to the database (no OS, no virtual memory files, only the data itself)</p>
  
  <p>ps2.: All geometries are LineStrings (if thats important :P) they represent streets, roads, etc..</p>
  
  <p>ps3.: the nodes are already in the database, i only need to add them to the Layer so that i can perform spatial queries, bbox and wkb attributes are OK, tested and working for a small set.</p>
  
  <p>Thank you in advance</p>
  
  <p>After altering and running the code again (which takes 5hours only to insert the points into the database, no layer involved) this happened, will try to increase the jvm heap and the embeddedgraph memory parameters.</p>
  
  <pre><code>indexing (4020000 nodes added) ... time in seconds: 8557.361
  Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
      at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:140)
      at gis.CataImporter.addDataToLayer(CataImporter.java:263)
      at Neo4JLoadData.addDataToLayer(Neo4JLoadData.java:138)
      at Neo4JLoadData.main(Neo4JLoadData.java:86)
  Caused by: javax.transaction.SystemException: Kernel has encountered some problem, please perform neccesary action (tx recovery/restart)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
      at org.neo4j.kernel.impl.transaction.KernelHealth.assertHealthy(KernelHealth.java:61)
      at org.neo4j.kernel.impl.transaction.TxManager.assertTmOk(TxManager.java:339)
      at org.neo4j.kernel.impl.transaction.TxManager.getTransaction(TxManager.java:725)
      at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:119)
      ... 3 more
  Caused by: javax.transaction.xa.XAException
      at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560)
      at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448)
      at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385)
      at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123)
      at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)
      at gis.CataImporter.addDataToLayer(CataImporter.java:256)
      ... 2 more
  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      at org.neo4j.kernel.impl.nioneo.store.DynamicRecord.clone(DynamicRecord.java:179)
      at org.neo4j.kernel.impl.nioneo.store.PropertyBlock.clone(PropertyBlock.java:215)
      at org.neo4j.kernel.impl.nioneo.store.PropertyRecord.clone(PropertyRecord.java:221)
      at org.neo4j.kernel.impl.nioneo.xa.Loaders$2.clone(Loaders.java:118)
      at org.neo4j.kernel.impl.nioneo.xa.Loaders$2.clone(Loaders.java:81)
      at org.neo4j.kernel.impl.nioneo.xa.RecordChanges$RecordChange.ensureHasBeforeRecordImage(RecordChanges.java:217)
      at org.neo4j.kernel.impl.nioneo.xa.RecordChanges$RecordChange.prepareForChange(RecordChanges.java:162)
      at org.neo4j.kernel.impl.nioneo.xa.RecordChanges$RecordChange.forChangingData(RecordChanges.java:157)
      at org.neo4j.kernel.impl.nioneo.xa.PropertyCreator.primitiveChangeProperty(PropertyCreator.java:64)
      at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransactionContext.primitiveChangeProperty(NeoStoreTransactionContext.java:125)
      at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.nodeChangeProperty(NeoStoreTransaction.java:1244)
      at org.neo4j.kernel.impl.persistence.PersistenceManager.nodeChangeProperty(PersistenceManager.java:119)
      at org.neo4j.kernel.impl.api.KernelTransactionImplementation$1.visitNodePropertyChanges(KernelTransactionImplementation.java:344)
      at org.neo4j.kernel.impl.api.state.TxStateImpl$6.visitPropertyChanges(TxStateImpl.java:238)
      at org.neo4j.kernel.impl.api.state.PropertyContainerState.accept(PropertyContainerState.java:187)
      at org.neo4j.kernel.impl.api.state.NodeState.accept(NodeState.java:148)
      at org.neo4j.kernel.impl.api.state.TxStateImpl.accept(TxStateImpl.java:160)
      at org.neo4j.kernel.impl.api.KernelTransactionImplementation.createTransactionCommands(KernelTransactionImplementation.java:332)
      at org.neo4j.kernel.impl.api.KernelTransactionImplementation.prepare(KernelTransactionImplementation.java:123)
      at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.prepareKernelTx(XaResourceManager.java:900)
      at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:510)
      at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
      at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548)
      ... 7 more
  </code></pre>
  
  <p>28/07 -> Increasing memory did not help, now im testing some modifications in the RTreeIndex and LayerRTreeIndex (what exactly does the field <code>maxNodeReferences</code> does ?</p>
  
  <pre><code>// Constructor
  
  public LayerRTreeIndex(GraphDatabaseService database, Layer layer) {
      this(database, layer, 100);     
  }
  
  public LayerRTreeIndex(GraphDatabaseService database, Layer layer, int maxNodeReferences) {
      super(database, layer.getLayerNode(), layer.getGeometryEncoder(), maxNodeReferences);
      this.layer = layer;
  }
  </code></pre>
  
  <p>It is hardcoded to 100, and changing its value changes when (number of nodes added wise) my <code>addToLayer</code> method crashes into OutOfMemory error, If im not mistaken, changing that field's value increases or decreases the tree's width and depth (being 100 wider than 50, and 50 being deeper than 100).</p>
  
  <p><strong>To summarize the progress so far:</strong></p>
  
  <ul>
  <li>Incorrect usage of transactions corrected by @Jim</li>
  <li>Memory Heap increased to 27GB following @Peter 's advice</li>
  <li>3 spatial layers to go, but now the problem gets real because they're the big ones.</li>
  <li>Did some memory profiling while adding nodes to the spatial layer and i found interesting points.</li>
  </ul>
  
  <p>Memory and GC profiling: <a href="http://postimg.org/gallery/biffn9zq/" rel="nofollow noreferrer">http://postimg.org/gallery/biffn9zq/</a></p>
  
  <p>The type that uses the most memory througout the entire process is the byte[], which i can only assume belongs to the geometries wkb properties (either the geometry itself or the rtree's bbox).
  Having that in mind, I also noticed (you can check on the new profiling images) that the ammount of heap space used never goes below the 18GB mark.</p>
  
  <p>According to this question <a href="https://stackoverflow.com/questions/18746801/are-java-primitives-garbage-collected">are java primitives garbage collected</a>
  primitive types in java are raw data, therefore not being subjected to garbage collection, and are only freed from the method's stack when the method returns (so maybe when i create a new spatial layer, all those wkb byte arrays will remain in memory until I manually close the layer object).</p>
  
  <p>Does that make any sense ? isnt there a better way to manage memory resources so that the layer doesnt keep unused, old data loaded ?</p>