title: Where to store metadata associated with files?
body: <p>This a question on storing and loading data, particularly in Python. I'm not entirely sure this is the appropriate forum, so redirect me if not.</p>
  
  <p>I'm handling about 50 1000-row CSV files, and each has 10 parameters of associated metadata. What is the best method to store this in regards to:</p>
  
  <p>(A) All the information is human-readable plain text and it's easy for a non-programming human to associate data and metadata.
  (B) It's convenient to load the metadata and each column of the csv to a python dictionary.</p>
  
  <p>I've considered four possible solutions:</p>
  
  <p>(0) Previously, I've stored smaller amounts of metadata in the filename. This is bad for obvious reasons.</p>
  
  <p>(1) Assign each CSV file a ID number, name each "ID.csv" and then produce a "metadata.csv" which maps each CSV ID number to its metadata. The shortcomings here are that using ID numbers reduces human readability. (To learn the contents of a file a non-programming human reader must manually check the "metadata.csv")</p>
  
  <p>(2) Leave the metadata at the top of CSV file. This has shortcomings in that my program would need to perform two steps: (a) get the metadata from some arbitrary number of lines at the top of the file and (b) tell the CSV reader (pandas.read_csv) to ignore first few lines.</p>
  
  <p>(3) Convert to CSV to some data serialization format like YAML, where I could then easily include the metadata. This has shortcomings of easily loading the columns of the CSV to my dictionary, and not everyone knows YAML.</p>
  
  <p>Are there any clever solutions to this problem? Thanks!</p>