app/importers/data_transformers/streaming_csv_transformer.rb
# Stream each line of the CSV and parse it.
# This includes three different types of transformations to replicate
# the behavior of CsvTransformer for now.  These could be pushed out to
# callers in the future.  These are: nil_converter for each field, discarding rows with
# dirty_data? and parsing particular field names into Ruby dates.
Stream each line of the CSV and parse it.
This includes three different types of transformations to replicate
the behavior of CsvTransformer for now.  These could be pushed out to
callers in the future.  These are: nil_converter for each field, discarding rows with
dirty_data. and parsing particular field names into Ruby dates.
Stream each line of the CSV and parse it . This includes three different types of transformations to replicate the behavior of CsvTransformer for now . These could be pushed out to callers in the future . These are : nil_converter for each field discarding rows with dirty_data . and parsing particular field names into Ruby dates .
def each_with_index(&block)
    reset_counters!
    csv_options = {
      headers: @headers,
      header_converters: :symbol,
      encoding: 'binary:UTF-8',
      converters: lambda { |h| nil_converter(h) }
    }.merge(@csv_options)
    CSV.new(@csv_string, csv_options).each.with_index do |row, index|
      cleaner = CsvRowCleaner.new(row)
      if cleaner.dirty_data?
        @skipped_dirty_rows_count = @skipped_dirty_rows_count + 1
        next
      end
      cleaner.transform_row
      block.call(row, index)
      @processed_rows_count = @processed_rows_count + 1
    end
    @log.puts("StreamingCsvTransformer#stats: #{stats}")
    nil
  end
def each_with_index ( & block ) reset_counters! csv_options = { headers : @headers , header_converters : :symbol , encoding : 'binary:UTF-8' , converters : lambda { | h | nil_converter ( h ) } } . merge ( @csv_options ) CSV . new ( @csv_string , csv_options ) . each . with_index do | row , index | cleaner = CsvRowCleaner . new ( row ) if cleaner . dirty_data? @skipped_dirty_rows_count = @skipped_dirty_rows_count + 1 next end cleaner . transform_row block . call ( row , index ) @processed_rows_count = @processed_rows_count + 1 end @log . puts ( "StreamingCsvTransformer#stats: #{stats}" ) nil end
Stream each line of the CSV and parse it.
Stream each line of the CSV and parse it .
block
<not_specific>