title: Arrow + Java: Populate VectorSchemaRoot (from stream / file) | Memory-Ownership | Usage patterns
body: <p>I'm doing very basic experiments with <a href="https://arrow.apache.org/" rel="noreferrer">Apache Arrow</a>, mostly in regards to passing some data between Java, C++, Python using Arrow's IPC format (to file), Parquet format (to file) and IPC format (stream through JNI).</p>
  <p>C++ and Python looks somewhat usable, but the Java-part is really troubling me.</p>
  <p>Sadly, the Java-documentation is kind of limited, but despite ackknowledging the warnings of <a href="https://arrow.apache.org/docs/java/vector_schema_root.html" rel="noreferrer">these hidden docs (not part of TOC)</a>, i'm just trying to populate some <code>VectorSchemaRoot</code> from a previously written file.</p>
  <p>Ignoring 99% of my experiment-code, the following minimal example shows my problem where i'm just creating some data, write it out (can be nicely imported in Python) and try to read it back in Java.</p>
  <p>But data-ownership seems to be in the way in regards to what i'm trying to achieve.</p>
  <p><strong>Maybe</strong> the idea of keeping <code>VectorSchemaRoot</code> as core-entry (like: <em>all my data is here</em>) is some kind of wrong usage, but i'm not sure what alternative is there. Why would i manually keep <code>IntVectors</code> and co. when this Class would do the same and serving some API to work with it.</p>
  <pre><code>package arrow.test;
  
  import java.io.FileInputStream;
  import java.io.FileOutputStream;
  import java.io.IOException;
  
  import org.apache.arrow.memory.RootAllocator;
  import org.apache.arrow.vector.BitVector;
  import org.apache.arrow.vector.FieldVector;
  import org.apache.arrow.vector.IntVector;
  import org.apache.arrow.vector.VectorSchemaRoot;
  import org.apache.arrow.vector.ipc.ArrowFileReader;
  import org.apache.arrow.vector.ipc.ArrowFileWriter;
  import org.apache.arrow.vector.types.pojo.ArrowType;
  import org.apache.arrow.vector.types.pojo.Field;
  import org.apache.arrow.vector.types.pojo.FieldType;
  import org.apache.arrow.vector.types.pojo.Schema;
  
  import com.google.common.collect.ImmutableList;
  
  public class Run {
  
    public static void main(String[] args) {
  
      ImmutableList.Builder&lt;Field&gt; builder = ImmutableList.builder();
      
      Field intField = new Field(&quot;i&quot;, FieldType.nullable(new ArrowType.Int(32, true)), null);
      builder.add(intField);      
      
      Field boolField = new Field(&quot;b&quot;, FieldType.nullable(new ArrowType.Bool()), null);
      builder.add(boolField);     
      
      RootAllocator sourceRootAlloc = new RootAllocator(Integer.MAX_VALUE);
      Schema sourceSchema = new Schema(builder.build(), null);
      VectorSchemaRoot sourceRoot  = VectorSchemaRoot.create(sourceSchema, sourceRootAlloc);
      
      FieldVector vector= sourceRoot.getVector(&quot;i&quot;);
      IntVector intVector = (IntVector) vector;
      intVector.allocateNew(5);
      intVector.set(0, 0);
      intVector.set(1, 1);
      intVector.set(2, 2);
      intVector.set(3, 3);
      intVector.set(4, 4);
      intVector.setValueCount(5);
      
      vector = sourceRoot.getVector(&quot;b&quot;);
      BitVector bitVector = (BitVector) vector;
      bitVector.allocateNew(5);
      bitVector.set(0, 1);
      bitVector.set(1, 1);
      bitVector.set(2, 0);
      bitVector.set(3, 0);
      bitVector.set(4, 1);
      bitVector.setValueCount(5);
      
      sourceRoot.setRowCount(5);
  
      System.out.println(&quot;before writing&quot;);
      System.out.println(sourceRoot.contentToTSVString());        
      
      // WRITE
      // -----
      
      try {
        FileOutputStream fileOut = new FileOutputStream(&quot;out&quot;, /*!overwrite=append*/false);
          ArrowFileWriter writer = new ArrowFileWriter(sourceRoot, null, fileOut.getChannel());
        writer.start();
        writer.writeBatch();
            writer.end();
            writer.close();
            fileOut.close();  
      }
      catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }
      
      // READ
      // ----
      
      FileInputStream fileInputStream;
        
      RootAllocator targetRootAlloc = new RootAllocator(Integer.MAX_VALUE);
      VectorSchemaRoot targetRoot = null;
      
      try {
        fileInputStream = new FileInputStream(&quot;out&quot;);
  
        ArrowFileReader reader = new ArrowFileReader(fileInputStream.getChannel(), targetRootAlloc);
        
        targetRoot = reader.getVectorSchemaRoot();    
        reader.loadNextBatch();       
        
        System.out.println(&quot;before closing stream&quot;);
        System.out.println(targetRoot.contentToTSVString());
  
        reader.close();
        fileInputStream.close();          
  
        System.out.println(&quot;after closing stream&quot;);
        System.out.println(targetRoot.contentToTSVString());      
      }
      catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }           
    }
  }
  </code></pre>
  <p>Running this (with Java 11 and <code>-Dio.netty.tryReflectionSetAccessible=true</code> as documented) leads to:</p>
  <pre><code>... irrelevant warning
  ...
  before writing
  i   b
  0   true
  1   true
  2   false
  3   false
  4   true
  
  before closing stream
  i   b
  0   true
  1   true
  2   false
  3   false
  4   true
  
  after closing stream
  Exception in thread &quot;main&quot; java.lang.IndexOutOfBoundsException: index: 0, length: 1 (expected: range(0, 0))
    at io.netty.buffer.ArrowBuf.checkIndexD(ArrowBuf.java:337)
    at io.netty.buffer.ArrowBuf.chk(ArrowBuf.java:324)
    at io.netty.buffer.ArrowBuf.getByte(ArrowBuf.java:526)
    at org.apache.arrow.vector.BaseFixedWidthVector.isSet(BaseFixedWidthVector.java:776)
    at org.apache.arrow.vector.IntVector.getObject(IntVector.java:143)
    at org.apache.arrow.vector.IntVector.getObject(IntVector.java:39)
    at org.apache.arrow.vector.VectorSchemaRoot.contentToTSVString(VectorSchemaRoot.java:268)
    at arrow.test.Run.main(Run.java:102)
  </code></pre>
  <p>while Python can do it easily:</p>
  <pre><code>import pyarrow as pa
  print(pa.__version__)
  
  buf = pa.ipc.open_file(&quot;out&quot;)
  print(buf.schema)
  
  df = buf.read_pandas()
  print(df)
  </code></pre>
  <p>outputs:</p>
  <pre><code>0.17.1
  i: int32
  b: bool
     i      b
  0  0   True
  1  1   True
  2  2  False
  3  3  False
  4  4   True
  </code></pre>
  <p><strong>Now it seems</strong>, that <code>ArrowFileReader</code> <strong>feels responsible to clean up the data despite the fact, that the allocator is defined out of it's scope</strong>. <code>targetRoot.getRowCount()</code> is correct, but each <em>VectorField</em> is of size <code>0</code>.</p>
  <p>I tried a lot of alternatives (not shown) in regards to using <code>VectorUnloader</code> and <code>VectorLoader</code> to load from some <em>reader VectorSchemaRoot</em> (local scope), transferring ownership using <code>batch.cloneWithTransfer(targetAlloc)</code>(global scope) and loading into <em>target VectorSchemaRoot</em> (global scope), but that did not work, usually due to <code>A buffer can only be associated between two allocators that share the same root</code></p>
  <p>Either i must have misunderstood a lot about the usage patterns, or (probably not, but it feels like) the Java part is pretty broken.</p>
  <p>Any ideas?</p>