title: What could be the explanation of this 'pyarrow.lib.ArrowIOError'?
body: <p>I'm working on an HDP cluster and I'm trying to read a <strong>.csv</strong> file from HDFS using <strong>pyarrow</strong>. I am able to connect to hdfs and print information about the file using the <strong>info()</strong> function. But when it comes to reading the content of the file, I get a <strong>pyarrow.lib.ArrowIOError</strong>. What could be the source of the problem?</p>
  
  <p>Here is the code I am executing</p>
  
  <pre class="lang-py prettyprint-override"><code># IMPORTS
  import pyarrow as pa
  from pyarrow import csv
  import os
  import subprocess
  
  # GET HDFS CLASSPATH
  classpath = subprocess.Popen(["/usr/hdp/current/hadoop-client/bin/hdfs", "classpath", "--glob"], stdout=subprocess.PIPE).communicate()[0]
  
  # CONFIGURE ENVIRONMENT VARIABLES
  os.environ["HADOOP_HOME"] = "/usr/hdp/current/hadoop-client"
  os.environ["JAVA_HOME"] = "/home/G60070/installs/jdk1.8.0_201/"
  os.environ["CLASSPATH"] = classpath.decode("utf-8")
  os.environ["ARROW_LIBHDFS_DIR"] = "/usr/hdp/2.6.5.0-292/usr/lib/"
  
  # USING PYARROW
  ## connect to hdfs
  fs = pa.hdfs.connect("xxxxxxx.xxx.xxx.fr", 8020)
  file = 'hdfs://xxxxxxx.xxx.xxx.fr:8020/user/F43479/trip_data_v2.csv'
  print(str(fs.info(file))) # this instruction works well
  
  ## read csv file
  csv_file = csv.read_csv(file) # this one doesn't work as expected
  csv_file
  </code></pre>
  
  <p>According to the <a href="https://arrow.apache.org/docs/python/csv.html#reading-csv-files" rel="nofollow noreferrer">pyarrow documentation</a>, I'm supposed to get the <strong>list of the csv's columns</strong> as a result.</p>
  
  <p>But I'm getting this error:
  <strong>pyarrow.lib.ArrowIOError: Failed to open local file: hdfs://xxxxxxx.xxx.xxx.fr:8020/user/F43479/trip_data_v2.csv, error: file not found</strong></p>
  
  <p>First, I thought I miswrote the filepath. I checked hdfs and the file is there.</p>
  
  <pre><code>[F43479@xxxxx dask_tests]$ hdfs dfs -ls /user/F43479/
  Found 9 items
  -rw-r-----   3 F43479 hdfs            0 2019-03-07 16:42 /user/F43479/-
  drwx------   - F43479 hdfs            0 2019-04-03 02:00 /user/F43479/.Trash
  drwxr-x---   - F43479 hdfs            0 2019-03-13 16:53 /user/F43479/.hiveJars
  drwxr-x---   - F43479 hdfs            0 2019-03-13 16:52 /user/F43479/hive
  drwxr-x---   - F43479 hdfs            0 2019-03-15 13:23 /user/F43479/nyctaxi_trip_data
  -rw-r-----   3 F43479 hdfs           36 2019-04-15 11:13 /user/F43479/test.csv
  -rw-r-----   3 F43479 hdfs  50486731416 2019-03-26 17:37 /user/F43479/trip_data.csv
  -rw-r-----   3 F43479 hdfs   5097056230 2019-04-15 13:57 /user/F43479/trip_data_v2.csv
  -rw-r-----   3 F43479 hdfs 504867312828 2019-04-02 11:15 /user/F43479/trip_data_x10.csv
  </code></pre>
  
  <p>What could be the source of the problem?</p>
  
  <p>Thanks for your potential help.</p>