title: How to parse CSV file in the most performant way?
body: <p>I would like to parse big CSV files in ABAP in the most performant way under the following conditions:</p>
  <ol>
  <li>We do not know the structure of the CSV-&gt;the parse result should be table of string_table or something simular</li>
  <li>The parsing should happen in accordance to <a href="https://www.rfc-editor.org/rfc/rfc4180" rel="nofollow noreferrer">https://www.rfc-editor.org/rfc/rfc4180</a></li>
  <li>No solution specific calls</li>
  </ol>
  <p>I found a very nice blog <a href="https://blogs.sap.com/2014/09/09/understanding-csv-files-and-their-handling-in-abap/" rel="nofollow noreferrer">https://blogs.sap.com/2014/09/09/understanding-csv-files-and-their-handling-in-abap/</a> but it has its shortcoming:</p>
  <ol>
  <li>Write your own code - <em>The code example is not sufficient</em></li>
  <li>Read the file using KCD_CSV_FILE_TO_INTERN_CONVERT - <em>solution specific (not available everywhere) and will dump on fields that are big enough</em></li>
  <li>Use RTTI and dynamic programming along with FM RSDS_CONVERT_CSV  - <em>we do not know the structure in advance</em></li>
  <li>Use class CL_RSDA_CSV_CONVERTER -  <em>we do not know the structure in advance</em></li>
  </ol>
  <p>I also checked the first available solution on github - <a href="https://github.com/thedoginthewok/ZwdCSV" rel="nofollow noreferrer">https://github.com/thedoginthewok/ZwdCSV</a> . Unfortunately, it has macros in the code (absolutely unacceptable) and also requires you to know the structure in advance.</p>
  <p>I also tried to use the regex to do the job, but on big files this is too slow.</p>
  <p>Even though I am extremely annoyed by this fact, I had to create a solution myself (I cannot believe that I actually did it - it should be in the standard...)
  My first solution was a direct copy paste of Java code into ABAP (<a href="https://mkyong.com/java/how-to-read-and-parse-csv-file-in-java/" rel="nofollow noreferrer">https://mkyong.com/java/how-to-read-and-parse-csv-file-in-java/</a>). Unfortunately, as my other question <a href="https://stackoverflow.com/questions/68654605/how-to-iterate-over-string-characters-in-abap-in-performant-way">How to iterate over string characters in ABAP in performant way?</a> shown, it is not that easy to iterate over string in abap as it is in Java.
  I then tried a split/count approach and so far it has the best performance. Does anyone knows the better way achieve this?</p>
  <pre><code>REPORT z_csv_test.
  
  
  CLASS lcl_csv_parser DEFINITION CREATE PRIVATE.
  
    PUBLIC SECTION.
      TYPES:
               tt_string_matrix TYPE STANDARD TABLE OF string_table WITH EMPTY KEY.
      CLASS-METHODS:
        create
          IMPORTING
            !iv_delimiter      TYPE string DEFAULT  '&quot;'
            !iv_separator      TYPE string DEFAULT  ','
            !iv_line_separator TYPE abap_cr_lf DEFAULT cl_abap_char_utilities=&gt;cr_lf
          RETURNING
            VALUE(r_result)    TYPE REF TO lcl_csv_parser.
      METHODS:
        parse
          IMPORTING
            iv_string       TYPE string
          RETURNING
            VALUE(r_result) TYPE tt_string_matrix,
        constructor
          IMPORTING
            !iv_delimiter      TYPE string
            !iv_separator      TYPE string
            !iv_line_separator TYPE string.
    PROTECTED SECTION.
    PRIVATE SECTION.
      DATA:
        gv_delimiter         TYPE string,
        gv_separator         TYPE string,
        gv_line_separator    TYPE string,
        gv_escaped_delimiter TYPE string.
      METHODS parse_line_to_string_table
        IMPORTING
          iv_line         TYPE string
        RETURNING
          VALUE(r_result) TYPE string_table.
  ENDCLASS.
  
  CLASS lcl_csv_parser IMPLEMENTATION.
  
    METHOD create.
      r_result = NEW #(
        iv_delimiter      = iv_delimiter
        iv_line_separator = CONV #( iv_line_separator )
        iv_separator      = iv_separator  ).
    ENDMETHOD.
  
    METHOD constructor.
      me-&gt;gv_delimiter = iv_delimiter.
      me-&gt;gv_separator = iv_separator.
      me-&gt;gv_line_separator = iv_line_separator.
      me-&gt;gv_escaped_delimiter = |{ iv_delimiter }{ iv_delimiter }|.
    ENDMETHOD.
  
    METHOD parse.
      &quot;get the lines
      SPLIT iv_string AT me-&gt;gv_line_separator INTO TABLE DATA(lt_lines).
      DATA lx_open_line TYPE abap_bool VALUE abap_false.
      DATA lv_current_line TYPE string.
  
      LOOP AT lt_lines ASSIGNING FIELD-SYMBOL(&lt;ls_line&gt;).
  
        FIND ALL OCCURRENCES OF me-&gt;gv_delimiter IN &lt;ls_line&gt; IN CHARACTER MODE MATCH COUNT DATA(lv_count).
        IF ( lv_count MOD 2 )  = 1.
          IF lx_open_line = abap_true.
            lv_current_line = |{ lv_current_line }{ me-&gt;gv_line_separator }{ &lt;ls_line&gt; }|.
            lx_open_line = abap_false.
            APPEND parse_line_to_string_table( lv_current_line ) TO r_result.
          ELSE.
            lv_current_line = &lt;ls_line&gt;.
            lx_open_line = abap_true.
          ENDIF.
        ELSE.
          IF lx_open_line = abap_true.
            lv_current_line = |{ lv_current_line }{ me-&gt;gv_line_separator }{ &lt;ls_line&gt; }|.
          ELSE.
            APPEND parse_line_to_string_table( &lt;ls_line&gt; ) TO r_result.
          ENDIF.
  
        ENDIF.
      ENDLOOP.
  
    ENDMETHOD.
  
  
    METHOD parse_line_to_string_table.
      SPLIT iv_line AT me-&gt;gv_separator INTO TABLE DATA(lt_line).
      DATA lx_open_field TYPE abap_bool VALUE abap_false.
      DATA lv_current_field TYPE string.
      LOOP AT lt_line ASSIGNING FIELD-SYMBOL(&lt;ls_field&gt;).
        FIND ALL OCCURRENCES OF me-&gt;gv_delimiter IN &lt;ls_field&gt; IN CHARACTER MODE MATCH COUNT DATA(lv_count).
        IF ( lv_count MOD 2 ) = 1.
          IF lx_open_field = abap_true.
            lv_current_field = |{ lv_current_field }{ me-&gt;gv_separator }{ &lt;ls_field&gt; }|.
            lx_open_field = abap_false.
            APPEND lv_current_field TO r_result.
          ELSE.
            lv_current_field = &lt;ls_field&gt;.
            lx_open_field = abap_true.
          ENDIF.
        ELSE.
          IF lx_open_field = abap_true.
            lv_current_field = |{ lv_current_field }{ me-&gt;gv_separator }{ &lt;ls_field&gt; }|.
          ELSE.
            APPEND &lt;ls_field&gt; TO r_result.
          ENDIF.
        ENDIF.
  
      ENDLOOP.
  
      REPLACE ALL OCCURRENCES OF me-&gt;gv_escaped_delimiter IN TABLE r_result WITH me-&gt;gv_delimiter.
  
    ENDMETHOD.
  
  ENDCLASS.
  CLASS lcl_test_csv_parser DEFINITION
    FINAL
    CREATE PUBLIC .
  
    PUBLIC SECTION.
      CLASS-METHODS run.
      CLASS-METHODS get_file
        RETURNING VALUE(r_result) TYPE string.
  
  
  
    PROTECTED SECTION.
    PRIVATE SECTION.
  
  ENDCLASS.
  
  
  
  CLASS lcl_test_csv_parser IMPLEMENTATION.
  
    METHOD get_file.
      DATA lv_file_line TYPE string.
      DO 10 TIMES.
        lv_file_line = |&quot;1234,{ cl_abap_char_utilities=&gt;cr_lf }567890&quot;,{ lv_file_line }|.
      ENDDO.
      lv_file_line = lv_file_line &amp;&amp; cl_abap_char_utilities=&gt;cr_lf.
  
      DATA(lt_file_as_table) = VALUE string_table(
          FOR i = 1 THEN  i + 1 UNTIL  i = 1000000
              ( lv_file_line ) ).
  
      CONCATENATE LINES OF lt_file_as_table INTO r_result.
    ENDMETHOD.
  
  
  
  
  
    METHOD run.
      DATA lv_prepare_start TYPE timestampl.
      GET TIME STAMP FIELD lv_prepare_start.
  
      DATA(lv_file) = get_file(  ).
  
      DATA lv_prepare_end TYPE timestampl.
      GET TIME STAMP FIELD lv_prepare_end.
  
      WRITE |Preparation took { cl_abap_tstmp=&gt;subtract( tstmp1 = lv_prepare_end tstmp2 = lv_prepare_start ) }|.
  
      DATA lv_parse_start TYPE timestampl.
      GET TIME STAMP FIELD lv_parse_start.
      DATA(lo_parser) = lcl_csv_parser=&gt;create(  ).
      DATA(lt_file) = lo_parser-&gt;parse( lv_file  ).
      DATA lv_parse_end TYPE timestampl.
      GET TIME STAMP FIELD lv_parse_end.
  
      WRITE |Parse took { cl_abap_tstmp=&gt;subtract( tstmp1 = lv_parse_end tstmp2 = lv_parse_start ) }|.
  
  
    ENDMETHOD.
  
  
  
  ENDCLASS.
  
  START-OF-SELECTION.
    lcl_test_csv_parser=&gt;run( ).
  </code></pre>