title: pandas data mining from Eurostat
body: <p>I'm starting a work to analyse data from Stats Institutions like Eurostat using python, and so pandas. I found out there are two methods to get data from Eurostat.</p>
  
  <ul>
  <li><em>pandas_datareader</em>: it seems very easy to use but I found some problems to get some specific data</li>
  <li><em>pandasdmx</em>: I've found it a bit complicated but it seems a promising solution, but documentation is poor</li>
  </ul>
  
  <p>I use a free Azure notebook, <a href="https://notebooks.azure.com/#" rel="noreferrer">online service</a>, but I don't think it will complicate more my situation. </p>
  
  <p>Let me explain the problems for <strong><em>pandas_datareader</em></strong>. According to the pandas documentation, <a href="https://pandas.pydata.org/pandas-docs/version/0.19.1/ecosystem.html" rel="noreferrer">in the section API</a>, there is <a href="https://pandas-datareader.readthedocs.io/en/latest/remote_data.html#eurostat" rel="noreferrer">this short documented package</a> and it works. Apart from the shown example, that nicely works, a problem arises about other tables. For example, I can get data about <a href="http://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=prc_hpi_a&amp;lang=en" rel="noreferrer">European house price</a>, which ID table is <em>prc_hpi_a</em> with this simple code: </p>
  
  <pre><code>import pandas_datareader.data as web
  import datetime
  df = web.DataReader('prc_hpi_a', 'eurostat')
  </code></pre>
  
  <p>But the table has three types of data about dwellings: TOTAL, EXISTING and NEW. I got only <em>Existing dwellings</em> and I don't know how to get the other ones. Do you have a solution for these types of filtering.</p>
  
  <p>Secondly there is the path using <strong><em>pandasdmx</em></strong>. Here it is more complicated. My idea is to upload all data to a pandas DataFrame, and then I can analyse as I want. Easy to say, but I've not find many tutorials that explain this passage: upload data to pandas structures. For example, I found <a href="https://pypi.org/project/pandaSDMX/0.1.2/#step-1-instantiate-a-client-for-eurostat" rel="noreferrer">this tutorial</a>, but I'm stuck to the first step, that is instantiate a client:</p>
  
  <pre><code>import pandasdmx
  from pandasdmx import client
  #estat=client('Eurostat', 'milk.db')
  </code></pre>
  
  <p>and it returns:</p>
  
  <blockquote>
    <blockquote>
      <p>--------------------------------------------------------------------------- ImportError                               Traceback (most recent call
      last)  in ()
            1 import pandasdmx
      ----> 2 from pandasdmx import client
            3 estat=client('Eurostat', 'milk.db')</p>
    </blockquote>
    
    <p>ImportError: cannot import name 'client'</p>
  </blockquote>
  
  <p>What's the problem here? I've looked around but no answer to this problem</p>
  
  <p>I also followed <a href="http://pandasdmx.readthedocs.io/en/latest/#example" rel="noreferrer">this tutorial</a>: </p>
  
  <pre><code>from pandasdmx import Request
  estat = Request('ESTAT')
  metadata = estat.datastructure('DSD_une_rt_a').write()
  metadata.codelist.iloc[8:18]
  resp = estat.data('une_rt_a', key={'GEO': 'EL+ES+IE'}, params={'startPeriod': '2007'})
  data = resp.write(s for s in resp.data.series if s.key.AGE == 'TOTAL')
  data.columns.names
  data.columns.levels
  data.loc[:, ('PC_ACT', 'TOTAL', 'T')]
  </code></pre>
  
  <p>I got the data, but my purpose is to upload them to a pandas structure (Series, DataFrame, etc..), so I can handle easily according to my work.  How to do that?
  Actually I did with this working line (below the previous ones):</p>
  
  <pre><code>s=pd.DataFrame(data)
  </code></pre>
  
  <p>But it doesn't work if I try to get other data tables. Let me explain with another example about the <a href="http://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=prc_hpi_q&amp;lang=en" rel="noreferrer">Harmonized Index Current Price</a> table:</p>
  
  <pre><code>estat = Request('ESTAT')
  metadata = estat.datastructure('DSD_prc_hicp_midx').write()
  resp = estat.data('prc_hicp_midx')
  data = resp.write(s for s in resp.data.series if s.key.COICOP == 'CP00')
  </code></pre>
  
  <p>It returns an error here, that is:</p>
  
  <blockquote>
    <p>--------------------------------------------------------------------------- AttributeError                            Traceback (most recent call
    last)  in ()
          2 metadata = estat.datastructure('DSD_prc_hicp_midx').write()
          3 resp = estat.data('prc_hicp_midx')
    ----> 4 data = resp.write(s for s in resp.data.series if s.key.COICOP == 'CP00')
          5 #metadata.codelist
          6 #data.loc[:, ('TOTAL', 'INX_Q','EA', 'Q')]</p>
    
    <p>~/anaconda3_501/lib/python3.6/site-packages/pandasdmx/api.py in
    <strong>getattr</strong>(self, name)
        622         Make Message attributes directly readable from Response instance
        623         '''
    --> 624         return getattr(self.msg, name)
        625 
        626     def _init_writer(self, writer):</p>
    
    <p>AttributeError: 'DataMessage' object has no attribute 'data'</p>
  </blockquote>
  
  <p>Why does it do not get data now? What's wrong now?</p>
  
  <p>I lost almost a day looking around for some clear examples and explanations. Do you have some to propose? Is there a full and clear documentation? I found also <a href="http://pandasdmx.readthedocs.io/en/latest/usage.html#basic-usage" rel="noreferrer">this page</a> with other examples, explaining the use of categorical schemes, but it is not for Eurostat (as explained at some point)</p>
  
  <p>Both methods could work, apart from some explained issues, but I need also a suggestion to have a definitely method to use, to query Eurostat but also many other institutions like OECD, World Bank, etc...
  Could you guide me to a definitive and working solution, even if it is different for each institution?</p>