Allow zero-row files, and handle stream exceptions #106
Conversation
This change allows exceptions to be caught and dealt with upstream when dealing with streams
|
The first change (allow zero-row files) looks good to me. Can't say much about the second change, since I'm not an expert on JavaScript streams, but I took a quick look at the nodejs docs. Based on that, the proposed change looks correct to me and what we currently have seems to be incorrect. Maybe @kessler can also comment on the streams issue. |
|
Hello @asmuth! This PR looks exactly that I need [https://github.com//issues/116]. When do you plan to merge it? |
| } else { | ||
| callback(); | ||
| callback() |
There was a problem hiding this comment.
This repo seems to be using semicolons, might want to go along with that.
| @@ -264,14 +260,14 @@ class ParquetTransformer extends stream.Transform { | |||
|
|
|||
| _transform(row, encoding, callback) { | |||
| if (row) { | |||
| this.writer.appendRow(row).then(callback); | |||
| this.writer.appendRow(row).then(d => callback(null, d), callback) | |||
There was a problem hiding this comment.
since d is always undefined, you can do:
| this.writer.appendRow(row).then(d => callback(null, d), callback) | |
| this.writer.appendRow(row).then(() => callback(), callback) |
| } | ||
| } | ||
|
|
||
| _flush(callback) { | ||
| this.writer.close(callback); | ||
| this.writer.close(callback).then(d => callback(null, d), callback) |
There was a problem hiding this comment.
| this.writer.close(callback).then(d => callback(null, d), callback) | |
| this.writer.close(callback).then(() => callback(), callback) |
|
PR looks good in general. Unfortunately I cannot merge it and the maintainers of this project don't seem to be merging anything for more than a year (or two?) |
|
@dobesv very sorry my friend. I am well aware that this project has been neglected and I really really hope I get to do something about it some time. I went over the PR and more importantly Paul did, so it's merged and I'll publish to npm asap. |
|
@kessler It's OK, it's the nature of volunteer work - sometimes we just have better things to do! |
Creating zero-row files is uncommon, but sometimes useful for handling edge cases in a pipeline, so there's no need to arbitrarily prevent it.
The ParquetTransformer change allows exceptions to be caught and dealt with upstream when dealing with streams, previously they were being swallowed.