original_docstring: Converts the 'schema_map' into a more flatten version which is
      compatible with BigQuery schema.
  
      If 'keep_nulls' is True then the resulting schema contains entries for
      nulls, empty arrays or empty records in the data.
  
      If 'sorted_schema' is True, the schema is sorted using the name of the
      columns. This seems to be the behavior for `bq load` using JSON data. For
      CSV files, sorting must not happen because the position of schema column
      must match the position of the column value in the CSV file.
  
      If 'infer_mode' is True, set the schema 'mode' to be 'REQUIRED' instead of
      'NULLABLE' if the field contains a value for all data records.
      
docstring: Converts the 'schema_map' into a more flatten version which is
  compatible with BigQuery schema.
  
  If 'keep_nulls' is True then the resulting schema contains entries for
  nulls, empty arrays or empty records in the data.
  
  If 'sorted_schema' is True, the schema is sorted using the name of the
  columns. This seems to be the behavior for `bq load` using JSON data. For
  CSV files, sorting must not happen because the position of schema column
  must match the position of the column value in the CSV file.
  
  
docstring_tokens: Converts the ' schema_map ' into a more flatten version which is compatible with BigQuery schema . If ' keep_nulls ' is True then the resulting schema contains entries for nulls empty arrays or empty records in the data . If ' sorted_schema ' is True the schema is sorted using the name of the columns . This seems to be the behavior for ` bq load ` using JSON data . For CSV files sorting must not happen because the position of schema column must match the position of the column value in the CSV file .
code: def flatten_schema_map(
      schema_map,
      keep_nulls=False,
      sorted_schema=True,
      infer_mode=False,
      input_format='json',
  ):
      """Converts the 'schema_map' into a more flatten version which is
      compatible with BigQuery schema.
  
      If 'keep_nulls' is True then the resulting schema contains entries for
      nulls, empty arrays or empty records in the data.
  
      If 'sorted_schema' is True, the schema is sorted using the name of the
      columns. This seems to be the behavior for `bq load` using JSON data. For
      CSV files, sorting must not happen because the position of schema column
      must match the position of the column value in the CSV file.
  
      If 'infer_mode' is True, set the schema 'mode' to be 'REQUIRED' instead of
      'NULLABLE' if the field contains a value for all data records.
      """
      if not isinstance(schema_map, dict):
          raise Exception(
              f"Unexpected type '{type(schema_map)}' for schema_map"
          )
  
      # Build the BigQuery schema from the internal 'schema_map'.
      schema = []
      map_items = sorted(schema_map.items()) if sorted_schema \
          else schema_map.items()
      for name, meta in map_items:
          # Skip over fields which have been explicitly removed
          if not meta:
              continue
  
          status = meta['status']
          filled = meta['filled']
          info = meta['info']
  
          # Schema entries with a status of 'soft' are caused by 'null' or
          # empty fields. Don't print those out if the 'keep_nulls' flag is
          # False.
          if status == 'soft' and not keep_nulls:
              continue
  
          # Copy the 'info' dictionary into the schema dict, preserving the
          # ordering of the 'field', 'mode', 'name', 'type' elements. 'bq load'
          # keeps these sorted, so we created them in sorted order using an
          # OrderedDict, so they should preserve order here too.
          new_info = OrderedDict()
          for key, value in info.items():
              if key == 'fields':
                  if not value:
                      # Create a dummy attribute for an empty RECORD to make
                      # the BigQuery importer happy.
                      new_value = [
                          OrderedDict([
                              ('mode', 'NULLABLE'),
                              ('name', '__unknown__'),
                              ('type', 'STRING'),
                          ])
                      ]
                  else:
                      # Recursively flatten the sub-fields of a RECORD entry.
                      new_value = flatten_schema_map(
                          schema_map=value,
                          keep_nulls=keep_nulls,
                          sorted_schema=sorted_schema,
                          infer_mode=infer_mode,
                          input_format=input_format
                      )
              elif key == 'type' and value in ['QINTEGER', 'QFLOAT', 'QBOOLEAN']:
                  # Convert QINTEGER -> INTEGER, similarly for QFLOAT and QBOOLEAN
                  new_value = value[1:]
              elif key == 'mode':
                  # 'infer_mode' to set a field as REQUIRED is supported for only
                  # input_format = 'csv' because the header line gives us the
                  # complete list of fields to be expected in the CSV file. In
                  # JSON data files, certain fields will often be completely
                  # missing instead of being set to 'null' or "". If the field is
                  # not even present, then it becomes incredibly difficult (not
                  # impossible, but more effort than I want to expend right now)
                  # to figure out which fields are missing so that we can mark the
                  # appropriate schema entries with 'filled=False'.
                  #
                  # The --infer_mode option is activated only for
                  # input_format == 'csv' in this function, which allows us to
                  # overload the --infer_mode flag to mean that a REQUIRED mode of
                  # an existing schema can transition to a NULLABLE mode.
                  if (infer_mode and value == 'NULLABLE' and filled
                          and input_format == 'csv'):
                      new_value = 'REQUIRED'
                  else:
                      new_value = value
              else:
                  new_value = value
              new_info[key] = new_value
          schema.append(new_info)
      return schema
code_tokens: def flatten_schema_map ( schema_map , keep_nulls = False , sorted_schema = True , infer_mode = False , input_format = 'json' , ) : if not isinstance ( schema_map , dict ) : raise Exception ( f"Unexpected type '{type(schema_map)}' for schema_map" ) schema = [ ] map_items = sorted ( schema_map . items ( ) ) if sorted_schema else schema_map . items ( ) for name , meta in map_items : if not meta : continue status = meta [ 'status' ] filled = meta [ 'filled' ] info = meta [ 'info' ] if status == 'soft' and not keep_nulls : continue new_info = OrderedDict ( ) for key , value in info . items ( ) : if key == 'fields' : if not value : new_value = [ OrderedDict ( [ ( 'mode' , 'NULLABLE' ) , ( 'name' , '__unknown__' ) , ( 'type' , 'STRING' ) , ] ) ] else : new_value = flatten_schema_map ( schema_map = value , keep_nulls = keep_nulls , sorted_schema = sorted_schema , infer_mode = infer_mode , input_format = input_format ) elif key == 'type' and value in [ 'QINTEGER' , 'QFLOAT' , 'QBOOLEAN' ] : new_value = value [ 1 : ] elif key == 'mode' : if ( infer_mode and value == 'NULLABLE' and filled and input_format == 'csv' ) : new_value = 'REQUIRED' else : new_value = value else : new_value = value new_info [ key ] = new_value schema . append ( new_info ) return schema
short_docstring: Converts the 'schema_map' into a more flatten version which is
  compatible with BigQuery schema.
short_docstring_tokens: Converts the ' schema_map ' into a more flatten version which is compatible with BigQuery schema .
comment: """Converts the 'schema_map' into a more flatten version which is
      compatible with BigQuery schema.
  
      If 'keep_nulls' is True then the resulting schema contains entries for
      nulls, empty arrays or empty records in the data.
  
      If 'sorted_schema' is True, the schema is sorted using the name of the
      columns. This seems to be the behavior for `bq load` using JSON data. For
      CSV files, sorting must not happen because the position of schema column
      must match the position of the column value in the CSV file.
  
      If 'infer_mode' is True, set the schema 'mode' to be 'REQUIRED' instead of
      'NULLABLE' if the field contains a value for all data records.
      """$SEP$# Build the BigQuery schema from the internal 'schema_map'.$SEP$# Skip over fields which have been explicitly removed$SEP$# Schema entries with a status of 'soft' are caused by 'null' or$SEP$# empty fields. Don't print those out if the 'keep_nulls' flag is$SEP$# False.$SEP$# Copy the 'info' dictionary into the schema dict, preserving the$SEP$# ordering of the 'field', 'mode', 'name', 'type' elements. 'bq load'$SEP$# keeps these sorted, so we created them in sorted order using an$SEP$# OrderedDict, so they should preserve order here too.$SEP$# Create a dummy attribute for an empty RECORD to make$SEP$# the BigQuery importer happy.$SEP$# Recursively flatten the sub-fields of a RECORD entry.$SEP$# Convert QINTEGER -> INTEGER, similarly for QFLOAT and QBOOLEAN$SEP$# 'infer_mode' to set a field as REQUIRED is supported for only$SEP$# input_format = 'csv' because the header line gives us the$SEP$# complete list of fields to be expected in the CSV file. In$SEP$# JSON data files, certain fields will often be completely$SEP$# missing instead of being set to 'null' or "". If the field is$SEP$# not even present, then it becomes incredibly difficult (not$SEP$# impossible, but more effort than I want to expend right now)$SEP$# to figure out which fields are missing so that we can mark the$SEP$# appropriate schema entries with 'filled=False'.$SEP$#$SEP$# The --infer_mode option is activated only for$SEP$# input_format == 'csv' in this function, which allows us to$SEP$# overload the --infer_mode flag to mean that a REQUIRED mode of$SEP$# an existing schema can transition to a NULLABLE mode.