title: Apache Spark GraphX java.lang.ArrayIndexOutOfBoundsException
body: <p>I am trying to understand how to work with Spark-GraphX but always have some problems, so maybe somebody could advise me what to read etc. I tried to read Spark documentation and Learning Spark - O'Reilly Media book, but could not find any explanation how much memory we need for dealing with differently sized networks etc. </p>
  
  <p>For my tests I use several example datasets. I run them on 1 master node (~16Gb RAM) from Spark shell:</p>
  
  <pre><code>./bin/spark-shell --master spark://192.168.0.12:7077 --executor-memory 2900m --driver-memory 10g
  </code></pre>
  
  <p>And 3-5 workers (1 worker per 1 separate machine, which has 4Gb RAM):</p>
  
  <pre><code>./bin/spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.12:7077
  </code></pre>
  
  <p>Then from Spark Shell I run my scala scripts (not compiled):</p>
  
  <pre><code>:load /home/ubuntu/spark-1.2.1/bin/script.scala
  </code></pre>
  
  <p>I do not use HDFS yet, just copied the dataset files to each machine (with the same path names of course). On small networks like zachary club or even bigger ~256 Mb networks (after increasing <em>driver-memory</em> parameter) I am able to count triangles, wedges, etc. </p>
  
  <p>Now try to deal with 750+ Mb networks and have errors. For example, I have Wikipedia links dataset in format of 2 columns (link_from link_to), 750Mb. Try to load it:</p>
  
  <pre><code>val graph = GraphLoader.edgeListFile(sc, "graphx/data/dbpidia")
  </code></pre>
  
  <p>And get an error:</p>
  
  <pre><code>[Stage 0:==============================================&gt;     (22 + 1) / 23]
  15/04/30 22:52:46 WARN TaskSetManager: Lost task 22.0 in stage 0.0 (TID 22, host-192-168-0-18.openstacklocal): java.lang.ArrayIndexOutOfBoundsException: 1
  at org.apache.spark.graphx.GraphLoader$$anonfun$1$$anonfun$apply$1.apply(GraphLoader.scala:83)
  at org.apache.spark.graphx.GraphLoader$$anonfun$1$$anonfun$apply$1.apply(GraphLoader.scala:76)
  at scala.collection.Iterator$class.foreach(Iterator.scala:727)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
  at org.apache.spark.graphx.GraphLoader$$anonfun$1.apply(GraphLoader.scala:76)
  at org.apache.spark.graphx.GraphLoader$$anonfun$1.apply(GraphLoader.scala:74)
  at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:631)
  at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:631)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
  at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
  at org.apache.spark.scheduler.Task.run(Task.scala:56)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  15/04/30 22:52:47 WARN TaskSetManager: Lost task 22.2 in stage 0.0 (TID 24, host-192-168-0-18.openstacklocal): java.lang.ArrayIndexOutOfBoundsException
  </code></pre>
  
  <p>Actually I need to work with datasets of size >>1Tb, but get errors even on smaller one. What am I doing wrong? What are the memory limits? What strategy could you propose for >>1Tb files, how to store them better?
  Thanks.</p>