feat: Include CSV headers #30

bbernays · 2023-01-13T14:58:25Z

Summary

Adds ability to conditionally include headers for CSV files

bbernays · 2023-01-13T15:09:56Z

csv/write.go

@@ -7,8 +7,17 @@ import (
 	"github.com/cloudquery/plugin-sdk/schema"
 )

-func WriteTableBatch(w io.Writer, _ *schema.Table, resources [][]any) error {
+func WriteTableBatch(w io.Writer, table *schema.Table, resources [][]any, headers bool) error {


I can't come up with any other way of injecting this conditional headers argument... Suggestions would be appreciated, or I can just add a nolint to it...

I think they basically want you to split it into different functions, e.g. WriteTableBatch and WriteTableBatchNoHeaders. It might be better to force users to think about whether they want headers or not though, so I'm fine with adding nolint for this case

So I will just add the nolint

I don't see how that can be possible, unless you still had a private function with the header boolean? In that case you could rather have WriteTableBatch write the headers first, then call WriteTableBatchNoHeaders to do the rest

(Or something to that effect - probably won't work just like that) In any case, nolint is fine.

Thinking about this some more... the function WriteTableBatch(w io.Writer, table *schema.Table, resources [][]any) is a standard interface used throughout many destinations, and the plugin sdk... Maybe we need a more generic way of passing in arguments...

So maybe it is better to add an entirely new function than it is to break that function signature...

Yeah, though I think the filetypes don't need to conform to that interface. (CC @yevgenypats because he'll probably know better)

A generic way of passing arguments sounds good though, as we'll hopefully be adding support for more formats (like parquet) in the not-too-distant future as well.

I feel the current approach in this PR is the right one as there is no generic interface needed here and the caller need to know wheater they want to use headers or not.

Un related to that if we already add options to that table I would make this just a bit nicer with ...Option so we can extend this later without breaking the interface (For example, if we would want an option of changing a delimiter ).

csv/write.go

yevgenypats

looks good but seems the interface needs a bit of change because now we want to add options to write and read which means we actually want a NewClient(...Option) *Client which will have WriteTableBatch and Read methods.

and one option for nowOptionWithHeaders

yevgenypats · 2023-01-14T10:51:40Z

csv/write.go

@@ -7,8 +7,17 @@ import (
 	"github.com/cloudquery/plugin-sdk/schema"
 )

-func WriteTableBatch(w io.Writer, _ *schema.Table, resources [][]any) error {
+func WriteTableBatch(w io.Writer, table *schema.Table, resources [][]any, headers bool) error {


I feel the current approach in this PR is the right one as there is no generic interface needed here and the caller need to know wheater they want to use headers or not.

Un related to that if we already add options to that table I would make this just a bit nicer with ...Option so we can extend this later without breaking the interface (For example, if we would want an option of changing a delimiter ).

yevgenypats · 2023-01-14T10:52:57Z

csv/read.go

@@ -23,9 +23,6 @@ func Read(f io.Reader, table *schema.Table, sourceName string, res chan<- []any)
 			}
 			return err
 		}
-		if record[sourceNameIndex] != sourceName {


why this is removed? also, shouldn't the no header propogate here?

This was removed because when there are headers the row does not get read...

yevgenypats

Looks good. few comments.

csv/client.go

client.go

csv/read.go

…iletypes into include-csv-headers

csv/client.go

csv/write.go

🤖 I have created a release *beep* *boop* --- ## [1.1.0](v1.0.6...v1.1.0) (2023-01-18) ### Features * Include CSV headers ([#30](#30)) ([9ab6df8](9ab6df8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

bbernays added 3 commits January 13, 2023 09:56

Update write.go

9f632da

Update read.go

15b5ea1

Update write_read_test.go

533b327

bbernays requested review from yevgenypats and hermanschaaf January 13, 2023 14:58

github-actions bot added feat and removed feat labels Jan 13, 2023

bbernays linked an issue Jan 13, 2023 that may be closed by this pull request

Missing CSV Headers cloudquery/cloudquery#6783

Closed

4 tasks

bbernays commented Jan 13, 2023

View reviewed changes

csv/write.go Outdated Show resolved Hide resolved

Update write.go

d2b18bb

bbernays linked an issue Jan 13, 2023 that may be closed by this pull request

Update filetype library to support Headers/No headers cloudquery/cloudquery#6794

Closed

bbernays removed a link to an issue Jan 13, 2023

Missing CSV Headers cloudquery/cloudquery#6783

Closed

4 tasks

yevgenypats requested changes Jan 14, 2023

View reviewed changes

Use NewClient And Options...

0284463

yevgenypats requested changes Jan 17, 2023

View reviewed changes

csv/client.go Outdated Show resolved Hide resolved

csv/client.go Outdated Show resolved Hide resolved

client.go Outdated Show resolved Hide resolved

client.go Outdated Show resolved Hide resolved

bbernays added 6 commits January 17, 2023 16:18

leave function signature unchanged

82322a3

fixes

11cb42f

fix tests + linting

3503a42

Respond to comments

93675be

Merge branch 'main' into include-csv-headers

ec85ab2

Add test for custom delimiter

c4df064

yevgenypats reviewed Jan 17, 2023

View reviewed changes

csv/read.go Outdated Show resolved Hide resolved

bbernays added 3 commits January 17, 2023 16:40

Merge branch 'include-csv-headers' of https://github.com/cloudquery/f…

69c0630

…iletypes into include-csv-headers

comments

2fde97d

Update read.go

7c4ca6b

yevgenypats reviewed Jan 18, 2023

View reviewed changes

csv/client.go Outdated Show resolved Hide resolved

bbernays added 3 commits January 18, 2023 07:55

Update client.go

069a83f

Update read.go

d1520e2

Update read.go

f47970a

yevgenypats reviewed Jan 18, 2023

View reviewed changes

csv/write.go Show resolved Hide resolved

bbernays and others added 2 commits January 18, 2023 10:04

Update write.go

bae7a26

fix minor issues

2f57a7b

yevgenypats approved these changes Jan 18, 2023

View reviewed changes

yevgenypats added the automerge Add to automerge PRs once requirements are met label Jan 18, 2023

kodiakhq bot merged commit 9ab6df8 into main Jan 18, 2023

kodiakhq bot deleted the include-csv-headers branch January 18, 2023 16:56

cq-bot mentioned this pull request Jan 18, 2023

chore(main): Release v1.1.0 #34

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Include CSV headers #30

feat: Include CSV headers #30

bbernays commented Jan 13, 2023 •

edited

Loading

bbernays Jan 13, 2023

hermanschaaf Jan 13, 2023

bbernays Jan 13, 2023

hermanschaaf Jan 13, 2023

hermanschaaf Jan 13, 2023

bbernays Jan 13, 2023

bbernays Jan 13, 2023

hermanschaaf Jan 13, 2023

hermanschaaf Jan 13, 2023

yevgenypats Jan 14, 2023

yevgenypats left a comment

yevgenypats Jan 14, 2023

yevgenypats Jan 14, 2023

bbernays Jan 14, 2023

yevgenypats left a comment

feat: Include CSV headers #30

feat: Include CSV headers #30

Conversation

bbernays commented Jan 13, 2023 • edited Loading

Summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevgenypats left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevgenypats left a comment

Choose a reason for hiding this comment

bbernays commented Jan 13, 2023 •

edited

Loading