Support for streamed parsing of very large excelx sheets

[rlburkes]

Changes should address the following issues:

Add interface to support streamed reading of an arbitrarily large xlsx document:

my_doc = Roo::Excelx.new('foo/bar/mydoc.xlsx')
my_doc.each_row_streaming(max_rows: 10) do |row|
  # each yielded row is an array of Roo::Excelx::Cell objects. 
  # max_rows allows you to limit how many rows you take which makes 
  # previewing a very large file fairly performant. 
  ...
end

Roo::Utils has a generic interface for yielding elements (streamed) of a given name to a caller.
Roo::Excelx::Cell has a coordinate that can provide row, column information to the consumer.

Added a bunch of basic test coverage for the Roo::Excelx and Roo::Utils classes to make sure I didnt break anything while doing minimal refactoring.

Hopefully you don't mind, I cleaned up a couple inconsistencies with parens, formatting, etc.

[christopherchiu]

@rlburkes I tried using your fork but I got an exception:

undefined method `present?' for #<Nokogiri::XML::Element:0x007fe29b04e590> (NoMethodError)

[rlburkes]

@christopherchiu I updated with a few changes, if you continue to see issue can you provide more details and open an issue?

[ephekt]

@christopherchiu I fixed this in my latest merge referenced above.

@rlburkes thanks for the important work here.

@Empact We'd greatly appreciate this fork merging back into master. I'll contribute to maintaining it with @rlburkes if needed. Our company ( @christopherchiu is on my team ) will also contribute.

Long live roo :)

[rlburkes]

@fareesh thanks, fixed. Shouldn't happened since options is defaulted to and empty hash, but it doesn't hurt to be consistent and err on side of caution.

[fareesh]

I have some code like so:

spreadsheet = Roo::Excelx.new(file,{},:ignore)
(3..spreadsheet.last_row).each do |row_index,index|
  # make [[:col1,:col2,:col3],["a","b","c"]] become {:col1 => "a", :col2 => "b", :col3 => "c"}
  row = Hash[[spreadsheet.row(1),spreadsheet.row(row_index)].transpose] 
  ....
end

I want to do the same thing using the each_row_streaming enumerator, which I have managed partially, but unfortunately if I have a row where only the first M out of N columns are filled, then the remaining N-M cells for that row aren't padded at the end, since each_row_streaming doesn't pad cells past the last filled cell.

How would I go about adding a parameter that makes each_row_streaming add padding everywhere, including at the end? Or is there a better way to do this?

[rlburkes]

@fareesh The current behavior of padding will pad from column 1(A) up to the last detected column. If you wish to pad after the last detected column (to a specified width) you can do so by padding the yielded row within your application.

(desired_size-row.size).times { row << nil } # or whatever you prefer to pad end of array

[Empact]

This should not descend from Exception, which is for low-level exceptions. Better to descend from StandardError or below. http://blog.nicksieger.com/articles/2006/09/06/rubys-exception-hierarchy/

[ausangshukla]

Pl can someone shed light on how to use the new each_row_streaming call to parse large XL files.
Thanks

[ephekt]

You should be able to see sample code in the readme.

On Thursday, November 28, 2013, Mohith Thimmaiah wrote:

Pl can someone shed light on how to use the new each_row_streaming call to
parse large XL files.
Thanks

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/69#issuecomment-29468888
.

[Empact]

Thanks again @rlburkes, I'd love to get this in, but it needs some cleanup to start. @ephekt, could you open a PR with your changes?

[ephekt]

Done. Opened.

On Thu, Dec 5, 2013 at 5:37 AM, Ben Woosley notifications@gh.neting.ccwrote:

Thanks again @rlburkes https://github.com/rlburkes, I'd love to get
this in, but it needs some cleanup to start. @ephekthttps://github.com/ephekt,
could you open a PR with your changes?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/69#issuecomment-29897716
.

[lappi-lynx]

@ephekt, I'm using 1.13.2 version of 'roo' but I can't find any minimal_load or each_row_streaming in the source. Can anyone explain e.g. how I can get only header from large xlsx file?

[rlburkes]

@extazystas You will need to use the fork this pull request originated from. @Empact any update on getting this merged?

[lappi-lynx]

@rlburkes, thanks - now I can use this methods. But could you please provide an example how to get only document header?

With origin gem I did it that way:

@headers = Hash.new
@document.row(1).each_with_index do |header,i|
  @headers[header] = i
end

[rlburkes]

@extazystas, make sure and check out the readme, @ephekt provided some nice documentation. This should help get you started.

Roo::Excelx.new('xlsx_file.xlsx', minimal_load: true).each_row_streaming(max_rows: 1) do |row_cells|
    # row_cells is an array of the first row (because we are limiting to 1 row) with data in your workbook.
    # If you want the excel row index, row_cells.first.coordinate.y should == excel workbook row.  
end

[lappi-lynx]

@rlburkes great, thank you!

[rlburkes]

@Empact any chance we can work to get this merged? We would love to get off our fork of Roo.

[Empact]

@rlburkes Could you rebase the PR to start? Or point out the relevant PR if it's not this one?

[Empact]

Hey @rlburkes I just pushed a major refactor of Roo::Excelx that makes all the file access lazy-loaded, e.g. sheets, comments, relationships files are only read as they are accessed.

There's still a good amount in your PR I would like to get in (e.g. the each_row_streaming could still be useful). Would you mind porting your changes over to the current codebase? The minimal_load option should no longer be necessary.

[rlburkes]

@Empact I went ahead and merged master, thanks for all your work refactoring it is definitely headed in a good direction. No longer need minimal_load flag as you noticed. Added some test coverage, and made some very minor code cleanups.

[coveralls]

Coverage increased (+0.69%) when pulling ab1b32e on rlburkes:master into a3c0724 on roo-rb:master.

[simonoff]

@rlburkes Thank you!