title: What could be the explaination of this "pyarrow.lib.ArrowIOError: HDFS file does not exist" error when trying to read files in hdfs using Dask?
body: <p>I'm using <strong>Dask Distributed</strong> and I'm trying to create a dataframe from a CSV stored in HDFS.
  I suppose the connection to HDFS is successful as I'm able to print the dataframe columns' names. 
  However, I get the following error when I'm trying to use the <strong>len</strong> function or any other function on the dataframe: </p>
  
  <pre class="lang-py prettyprint-override"><code>pyarrow.lib.ArrowIOError: HDFS file does not exist: /user/F43479/trip_data_v2.csv
  </code></pre>
  
  <p>I don't understand why I have this error. I would like to have your opinion.</p>
  
  <p>Here is my code:</p>
  
  <pre class="lang-py prettyprint-override"><code># IMPORTS
  import dask.dataframe as dd
  from dask.distributed import Client
  import pyarrow as pa
  from pyarrow import csv
  from dask import compute,config
  import os
  import subprocess
  
  # GET HDFS CLASSPATH
  classpath = subprocess.Popen(["/usr/hdp/current/hadoop-client/bin/hdfs", "classpath", "--glob"], stdout=subprocess.PIPE).communicate()[0]
  
  # CONFIGURE ENVIRONMENT VARIABLES
  os.environ["HADOOP_HOME"] = "/usr/hdp/current/hadoop-client"
  os.environ["JAVA_HOME"] = "/home/G60070/installs/jdk1.8.0_201/"
  os.environ["CLASSPATH"] = classpath.decode("utf-8")
  os.environ["ARROW_LIBHDFS_DIR"] = "/usr/hdp/2.6.5.0-292/usr/lib/"
  
  # LAUNCH DASK DISTRIBUTED
  client = Client('10.22.104.37:8786')
  
  # SET HDFS CONNEXION
  config.set(hdfs_driver='pyarrow', host='xxxxx.xxx.xx.fr', port=8020)
  
  # READ FILE ON HDFS
  folder = 'hdfs://xxxxx.xxx.xx.fr:8020/user/F43479/'
  filepath = folder+'trip_data_v2.csv'
  df = dd.read_csv(filepath)
  
  # TREATMENTS ON FILE
  print(df.columns)# this works
  print(len(df))# produces an error
  </code></pre>
  
  <p>Here is the content of my HDFS repository:</p>
  
  <pre><code>[F43479@xxxxx dask_tests]$ hdfs dfs -ls /user/F43479/
  Found 9 items
  -rw-r-----   3 F43479 hdfs            0 2019-03-07 16:42 /user/F43479/-
  drwx------   - F43479 hdfs            0 2019-04-03 02:00 /user/F43479/.Trash
  drwxr-x---   - F43479 hdfs            0 2019-03-13 16:53 /user/F43479/.hiveJars
  drwxr-x---   - F43479 hdfs            0 2019-03-13 16:52 /user/F43479/hive
  drwxr-x---   - F43479 hdfs            0 2019-03-15 13:23 /user/F43479/nyctaxi_trip_data
  -rw-r-----   3 F43479 hdfs           36 2019-04-15 11:13 /user/F43479/test.csv
  -rw-r-----   3 F43479 hdfs  50486731416 2019-03-26 17:37 /user/F43479/trip_data.csv
  -rw-r-----   3 F43479 hdfs   5097056230 2019-04-15 13:57 /user/F43479/trip_data_v2.csv
  -rw-r-----   3 F43479 hdfs 504867312828 2019-04-02 11:15 /user/F43479/trip_data_x10.csv
  </code></pre>
  
  <p>And finally, the full result of the code execution:</p>
  
  <pre class="lang-py prettyprint-override"><code>Index(['vendor_id', 'passenger_count', 'trip_time_in_secs', 'trip_distance'], dtype='object')
  Traceback (most recent call last):
    File "dask_pa_hdfs.py", line 32, in &lt;module&gt;
      print(len(df))
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/dataframe/core.py", line 438, in __len__
      split_every=False).compute()
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/base.py", line 156, in compute
      (result,) = compute(self, traverse=False, **kwargs)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/base.py", line 397, in compute
      results = schedule(dsk, keys, **kwargs)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/client.py", line 2321, in get
      direct=direct)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/client.py", line 1655, in gather
      asynchronous=asynchronous)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/client.py", line 673, in sync
      return sync(self.loop, func, *args, **kwargs)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/utils.py", line 277, in sync
      six.reraise(*error[0])
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/six.py", line 693, in reraise
      raise value
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/utils.py", line 262, in f
      result[0] = yield future
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
      value = future.result()
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
      yielded = self.gen.throw(*exc_info)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/distributed/client.py", line 1500, in _gather
      traceback)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/six.py", line 692, in reraise
      raise value.with_traceback(tb)
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/bytes/core.py", line 133, in read_block_from_file
      with copy.copy(lazy_file) as f:
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/bytes/core.py", line 177, in __enter__
      f = SeekableFile(self.fs.open(self.path, mode=mode))
    File "/opt/anaconda3/envs/python3-dask/lib/python3.7/site-packages/dask/bytes/pyarrow.py", line 37, in open
      return self.fs.open(path, mode=mode, **kwargs)
    File "pyarrow/io-hdfs.pxi", line 431, in pyarrow.lib.HadoopFileSystem.open
    File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
  pyarrow.lib.ArrowIOError: HDFS file does not exist: /user/F43479/trip_data_v2.csv
  </code></pre>