title: Columns to Struct parquet by pyarrow and (or) pandas in Python
body: <p>I hope some of you will find a little bit time to help a beginner like me. I am working on this task for the full week and cannot find a solution. I understand and completely fine with that I have to learn each package I use and their combinations to find the right solution.</p>
  <p>The full task is to combine 5 columns (1000 rows) into 1 struct column and have it stored/transformed in a row (1000 columns) in parquet. But I stuck with the problem to combine 5 columns into 1 struct column.</p>
  <p>Originally, I receive the following columns: columns=['date', 'bidopen', 'bidclose', 'bidhigh', 'bidlow', 'tickqty']. I do not need 'date' to be part of the struct.</p>
  <p>What I tried:</p>
  <pre><code>import pyarrow as pa
  import pyarrow.parquet as pq
  import pandas as pd
  </code></pre>
  <p><strong>Option 1 - dictionary with pandas</strong></p>
  <pre><code>df = pd.read_csv('original.csv')
  df2 = df.drop(columns=['date'])
  df3 = df2.to_dict('records')
  </code></pre>
  <p>I cannot save dict to csv or parquet by pandas - the following 2 commands lead to backward transformation to pandas dataframe and saving columns separately.</p>
  <pre><code>pd.DataFrame(df3).to_csv('test_parquet.csv', index=False)
  pd.DataFrame(df3).to_parquet('test2.parquet')
  </code></pre>
  <p>If I could use dictionary as a dataframe, next I would use pandas.DataFrame.pivot to turn rows into columns. Next, I tried to convert dict to the pyarrow table (seems like potentially I could also save entries in columns (1 row)).</p>
  <pre><code>table = pa.Table.from_pydict({'data', pa.array(df3)})
  </code></pre>
  <p>After the line above I have an error and I could not find a solution (TypeError: unhashable type: 'pyarrow.lib.StructArray'). The next step would be to save the table into parquet by pyarrow.</p>
  <p><strong>Option 2 - to struct by pyarrow</strong></p>
  <p>Here I tried to work inside parquet to change the schema (or write into new schema)</p>
  <pre><code>df = pd.read_csv('original.csv')
  df = df.drop(columns=['date'])
  df.to_parquet('test.parquet')
  table = pq.read_table('test.parquet', columns=['bidopen', 'bidclose', 'bidhigh', 'bidlow', 'tickqty'])
  </code></pre>
  <p>Here I read the schema of the parquet to see the DataType of each column. Below I set the new schema:</p>
  <pre><code>struct = pa.struct([
      pa.field('bidopen', pa.float64()),
      pa.field('bidclose', pa.float64()),
      pa.field('bidhigh', pa.float64()),
      pa.field('bidlow', pa.float64()),
      pa.field('tickqty', pa.int64())
  ])
  fields = ([pa.field('data', pa.list_(struct))])
  schema = pa.schema(fields)
  writer = pq.ParquetWriter('test2.parquet', schema)
  writer.write_table(table)
  writer.close()
  </code></pre>
  <p>I got an error for which I also could not find a solution (ValueError: Table schema does not match schema used to create file:...) as I thought it would save into the new provided schema.</p>
  <p><strong>Option 3 - pyarrow cast</strong></p>
  <pre><code>#(the upper part is from the Option 2)
  ...
  schema = pa.schema(fields)
  table2 = table.cast(schema)
  writer = pq.ParquetWriter('test2.parquet', schema)
  writer.write_table(table2)
  writer.close()
  </code></pre>
  <p>I got another error (ValueError: Target schema's field names are not matching the table's field names:). Here I said - c'mon, I am doing cast exactly because the schemas are not the same... That didn't help.</p>
  <p><strong>Option 4 - another attempt to change the schema when loading from pandas to pyarrow to save it later to parquet</strong></p>
  <pre><code>arrays = [['data','data','data','data','data'],['bidopen', 'bidclose','bidhigh','bidlow','tickqty']]
  tuples = list(zip(*arrays))
  index = pd.MultiIndex.from_tuples(tuples)
  df2 = pd.DataFrame(df.values[:, 1:], columns=index)
  pa.Schema.from_pandas(df2)
  </code></pre>
  <p>Here I got an error (AttributeError: 'list' object has no attribute 'columns') which I also could not find a solution to.</p>
  <p><strong>Option 5 - pyspark</strong></p>
  <p>Here was the biggest oops for me as I spend about 3 days to &quot;learn&quot; it as it should be able to do the conversion to struct and pivoting. BUT, I found out later that I cannot save data into parquet by pyspark on my Win10 without additional packages: Hadoop and Java SDK (it is not free to use). Therefore, I stopped developing it further.</p>