original_docstring: Combine datasets and process to extract required fields
      Parameters
      ----------
      dataframes: list of Pandas dataframes, required
          List of dataframes with raw air quality data
      Returns
      -------
      parameter_readings: Pandas dataframe
          Processed dataframe of air quality ratings
      
docstring: Combine datasets and process to extract required fields
  Parameters
  
  list of Pandas dataframes, required
  List of dataframes with raw air quality data
  Returns
  
  Pandas dataframe
  Processed dataframe of air quality ratings
docstring_tokens: Combine datasets and process to extract required fields Parameters list of Pandas dataframes required List of dataframes with raw air quality data Returns Pandas dataframe Processed dataframe of air quality ratings
code: def process_data(dataframes):
      """Combine datasets and process to extract required fields
      Parameters
      ----------
      dataframes: list of Pandas dataframes, required
          List of dataframes with raw air quality data
      Returns
      -------
      parameter_readings: Pandas dataframe
          Processed dataframe of air quality ratings
      """
  
      xray_recorder.begin_subsegment('## map')
      subsegment = xray_recorder.current_subsegment()
      try:
          # combine into single dataframe
          data = pd.concat(dataframes, sort=False)
          subsegment.put_metadata('rows', len(data))
  
          # keep only coulumns we need
          columns_to_keep = ['country','city','location','parameter','value', 'unit','date.utc']
          data.drop(set(data.columns.values) - set(columns_to_keep), axis=1, inplace=True)
          log.info(f"Total rows to process: {len(data)}")
  
          # pivot to convert air quality parameters to columns
          parameter_readings = data.pivot_table(
              index=[
                  'country',
                  'city',
                  'location',
                  'date.utc'],
              columns='parameter',
              values='value').reset_index()
          subsegment.put_annotation('MAP', 'SUCCESS')
      except Exception as e:
          subsegment.put_annotation('MAP', 'FAILURE')
          log.error("Error processing data")
          log.debug(e)
  
      xray_recorder.end_subsegment()
      return parameter_readings
code_tokens: def process_data ( dataframes ) : xray_recorder . begin_subsegment ( '## map' ) subsegment = xray_recorder . current_subsegment ( ) try : data = pd . concat ( dataframes , sort = False ) subsegment . put_metadata ( 'rows' , len ( data ) ) columns_to_keep = [ 'country' , 'city' , 'location' , 'parameter' , 'value' , 'unit' , 'date.utc' ] data . drop ( set ( data . columns . values ) - set ( columns_to_keep ) , axis = 1 , inplace = True ) log . info ( f"Total rows to process: {len(data)}" ) parameter_readings = data . pivot_table ( index = [ 'country' , 'city' , 'location' , 'date.utc' ] , columns = 'parameter' , values = 'value' ) . reset_index ( ) subsegment . put_annotation ( 'MAP' , 'SUCCESS' ) except Exception as e : subsegment . put_annotation ( 'MAP' , 'FAILURE' ) log . error ( "Error processing data" ) log . debug ( e ) xray_recorder . end_subsegment ( ) return parameter_readings
short_docstring: Combine datasets and process to extract required fields
  Parameters
short_docstring_tokens: Combine datasets and process to extract required fields Parameters
comment: """Combine datasets and process to extract required fields
      Parameters
      ----------
      dataframes: list of Pandas dataframes, required
          List of dataframes with raw air quality data
      Returns
      -------
      parameter_readings: Pandas dataframe
          Processed dataframe of air quality ratings
      """$SEP$# combine into single dataframe$SEP$# keep only coulumns we need$SEP$# pivot to convert air quality parameters to columns