Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new config for csv column explicit type conversion #4781

Merged
merged 5 commits into from
Oct 4, 2018
Merged

Add new config for csv column explicit type conversion #4781

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
aae2f1f
Add new config for csv column explict type conversion
rudbast Oct 1, 2018
c7998e6
Add missing declarations for csv_column_types everywhere
rudbast Oct 2, 2018
d11d470
Throw error on first init on invalid csv column counts
rudbast Oct 2, 2018
36ce0e9
Re-add old logic & improve column check before err throw
rudbast Oct 2, 2018
3759eba
Add test case for csvexplicit type conversion
rudbast Oct 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions plugins/parsers/csv/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ values.
## If `csv_header_row_count` is set to 0, this config must be used
csv_column_names = []

## For assigning explicit data types to columns.
## Supported types: "int", "float", "bool", "string".
## If this is not specified, type conversion will be done on the types above.
csv_column_types = []

## Indicates the number of rows to skip before looking for header information.
csv_skip_rows = 0

Expand Down
30 changes: 30 additions & 0 deletions plugins/parsers/csv/parser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ type Parser struct {
Comment string
TrimSpace bool
ColumnNames []string
ColumnTypes []string
TagColumns []string
MeasurementColumn string
TimestampColumn string
Expand Down Expand Up @@ -148,6 +149,35 @@ outer:
}
}

// In case if defined column names & types count don't match.
if i < len(p.ColumnTypes) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like throwing an error if they don't match before here would be a good idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've added the throw error statement on newCSVParser, seems like it fits there better ?

var val interface{}
var err error

switch p.ColumnTypes[i] {
case "int":
val, err = strconv.ParseInt(value, 10, 64)
if err != nil {
return nil, fmt.Errorf("column type: parse int error %s", err)
}
case "float":
val, err = strconv.ParseFloat(value, 64)
if err != nil {
return nil, fmt.Errorf("column type: parse float error %s", err)
}
case "bool":
val, err = strconv.ParseBool(value)
if err != nil {
return nil, fmt.Errorf("column type: parse bool error %s", err)
}
default:
val = value
}

recordFields[fieldName] = val
continue
}

// attempt type conversions
if iValue, err := strconv.ParseInt(value, 10, 64); err == nil {
recordFields[fieldName] = iValue
Expand Down