parser_csv: Improve the performance for typical cases

[repeatedly]

Add parser_type parameter to switch the parser.

Signed-off-by: Masahiro Nakagawa repeatedly@gmail.com

Which issue(s) this PR fixes:
None

What this PR does / why we need it:
Parser version try of #2529.
I implemented original method because CSV module doesn't provide the way to parse single line without recreate CSV object.
This fast implementation supports typical patterns so it is useful. See test case for supported patterns:

fluentd/test/plugin/test_parser_csv.rb

Line 107 in 8b1cb4a

def test_compatibility_between_normal_and_fast_parser(param)

Here is the benchmark result:

non quote: value1,value1,value1,value1,value1,value1,value1,value1
quoted   : "fo,o","fo,o","fo,o","fo,o","fo,o","fo,o","fo,o","fo,o"
escaped  : "aa","b""b""b","c"," ",e,f,"""""",
Warming up --------------------------------------
  now with non quote     1.388k i/100ms
  new with non quote    30.281k i/100ms
     now with quoted     1.175k i/100ms
     new with quoted    10.304k i/100ms
     now with escape     1.142k i/100ms
     new with escape    11.327k i/100ms
Calculating -------------------------------------
  now with non quote     14.631k (± 2.2%) i/s -     73.564k in   5.030352s
  new with non quote    329.169k (± 3.2%) i/s -      1.665M in   5.065852s
     now with quoted     11.950k (± 2.8%) i/s -     59.925k in   5.019060s
     new with quoted    103.813k (± 7.2%) i/s -    525.504k in   5.096997s
     now with escape     11.920k (± 2.5%) i/s -     60.526k in   5.080920s
     new with escape    117.268k (± 1.9%) i/s -    589.004k in   5.024571s

• benchmark code

require 'benchmark/ips'
require 'csv'

class CP
  def initialize
    @keys = $keys.dup
    @delimiter = ','
    @quote_char = '"'
    @escape_pattern = Regexp.compile(@quote_char * 2)
  end

  def parse1(text, &block)
    values = CSV.parse_line(text, col_sep: @delimiter)
    r = Hash[@keys.zip(values)]
    yield r
  end

  def parse2(text, &block)
    r = fast_parse(text)
    yield r
  end

  def fast_parse(text)
    record = {}
    text.chomp!

    return record if text.empty?

    # use while because while is now faster than each_with_index
    columns = text.split(@delimiter, -1)
    num_columns = columns.size
    i = 0
    j = 0
    while j < num_columns
      column = columns[j]

      case column.count(@quote_char)
      when 0
        if column.empty?
          column = nil
        end
      when 1
        if column.start_with?(@quote_char)
          to_merge = [column]
          j += 1
          while j < num_columns
            merged_col = columns[j]
            to_merge << merged_col
            break if merged_col.end_with?(@quote_char)
            j += 1
          end
          column = to_merge.join(@delimiter)[1..-2]
        end
      when 2
        if column.start_with?(@quote_char) && column.end_with?(@quote_char)
          column = column[1..-2]
        end
      else
        if column.start_with?(@quote_char) && column.end_with?(@quote_char)
          column = column[1..-2]
        end
        column.gsub!(@escape_pattern, @quote_char)
      end

      record[@keys[i]] = column
      j += 1
      i += 1
    end
    record
  end
end

$keys = ["key1","key2","ke,y3","ke y4","key5","key6","key7","key8"]
keys = $keys.dup
text1 = keys.size.times.map { "value1" }.join(",")
text2 = keys.size.times.map { '"fo,o"' }.join(",")
text3 = '"aa","b""b""b","c"," ",e,f,"""""",'

puts "Ruby version: #{RUBY_VERSION}"

cp = CP.new

puts "non quote: #{text1}"
puts "quoted   : #{text2}"
puts "escaped  : #{text3}"

Benchmark.ips do |x|
  x.report('now with non quote') do
    cp.parse1(text1) { |r| }
  end

  x.report('new with non quote') do
    cp.parse2(text1) { |r| }
  end

  x.report('now with quoted') do
    cp.parse1(text2) { |r| }
  end

  x.report('new with quoted') do
    cp.parse2(text2) { |r| }
  end

  x.report('now with escape') do
    cp.parse1(text3) { |r| }
  end

  x.report('new with escape') do
    cp.parse2(text3) { |r| }
  end
end

Docs Changes:
Add parser_type to parser_csv article

Release Note:
Same as title

[ganmacs]

Is executing chomp! okay here?
If row value is a,b, , (User expects that the last column is space), should we remain the value as it is?

p CSV.parse_line('a,b, ,c, ')  #=> ["a", "b", " ", "c", " "]

[repeatedly]

chomp! don't remove space character.

[ganmacs]

Probably, it needs to add the test which fast CSV parser receives the value which it can't parse but normal one can do.
I don't understand now the difference between fast and normal parser.

[repeatedly]

I added simple test for different point check but hard to write detailed test because I don't know how CSV module parse the CSV line precisely.

[ganmacs]

How do users detect invalid records?

[repeatedly]

No way. This fast parser doesn't consider it. Users need to check their format meets fast parser before.

[ganmacs]

I think we need to add the detail document about the difference between the fast parser and the normal one and how to handing if invalid records contain.

[ganmacs]

Should This line and L37 be moved into L39's if expression?

[repeatedly]

Apply reviews. I will send a patch to fluentd-docs-gitbook.