CSV upload indexes FALSE values are truthy

[Bargs]

If a CSV contains boolean values in all caps (TRUE/FALSE), Elasticsearch will index "FALSE" as a truthy value. This is because the sever side CSV parser does not recognize FALSE as a boolean value, and sends it as a JSON string. To make matters worse, the client side parser DOES detect it as a boolean value, so the user will think everything is working fine until they realize every FALSE value has been indexed as true.

I can think of a couple ways to fix this:

• Turn off auto type detection on the server side and use the client side selections. We'd just have to normalize things like geo_point to their JSON values. This is probably the correct solution.
• Somehow normalize capitalization of boolean values.

[epixa]

What's the value of the server-side transformation over simply doing it on the client?

[Bargs]

Right now we rely on the browser to upload the raw file contents and parse it on the server in a streaming manner. It allows us to completely avoid loading the file contents into memory on the client side, except for the first few lines to show the preview. If we do the transformation to JSON on the client we'll need to stream those parse results to the backend to avoid breaking the browser on large files. I'm not sure how to do that without sending many small http requests (I tried this but it made the browser really laggy and increased ingest time) or websockets (complicated).

After giving this some thought though, I'm not sure either of my previous suggestions are the right solution. I'm not so sure we want to modify the user's data at all in the CSV upload wizard. If we detect TRUE and FALSE strings as boolean, but elasticsearch does not, that might cause even more confusion. Strictness would make things more predictable. But the downside is that Excel exports bools as TRUE/FALSE, so this is going to be a common issue.

[ck-lee]

What about showing a warning message to the user on preview, similar to the issue with white space on header fields? That way, users can decide what they want to do with it.

[Bargs]

I think the downside to creating a warning like that is, where do you stop? This is one weird case, but there are lots of others too. For instance, is null falsey? Some users might expect it to be, but elasticsearch says no. Whereas the string values "no" and "off" are considered falsey. Providing warnings will require Kibana to have an intimate knowledge of what values are considered true/false by ES.

Some day I'd like to enhance the pattern configuration step to index the test data and show the user how it's actually going to be seen by ES. But that's a longer term solution, I'm not sure what a good short term solution is.

[ck-lee]

I see what you mean. An actual ES import preview is a good idea! And I think it is worth the effort here. I personally find that CSV import feature has great potential for Kibana. Good luck with it. I am happy to help out too.

[epixa]

I'm going to close this since CSV upload was pulled. We can always refer back to this issue if/when we revisit the feature.