Dealing with trailing whitespace

[kadrlica]

This is an issue created for our fairly extensive discussion of how CFITSIO, fitsio, and pyfits treat trailing whitespace in binary tables.

I think the working suggestion is to replace leading/trailing whitespace characters with null characters when writing FITS files. This is something that will need to be done in pandas/numpy before passing to fitsio for writing.

CFITSIO Manuel:
https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node24.html

[mgckind]

Thanks, I was about to create this. And also the other way around in case when reading fits files to uploading them.

[kadrlica]

Is there any time we want to respect trailing whitespace?

[mgckind]

I want to believe there isn't. we can enforce that as well.

[kadrlica]

We could provide a toggle switch (i.e., easyaccess -w) that respects whitespace.

[mgckind]

and the default being not to respect, I'd use a flag like --enable-white-space to create consciousness while typing

[kadrlica]

I think the Oracle TRIM function is what we are looking for here. I've done some timing tests, and it looks like prepending TRIM when inserting string values has the desired effect and does not adversely impact the performance. Specifically, the implementation would be to apply

TRIM(TRAILING FROM :value)

in easy_or.insert_data for string columns.

http://www.techonthenet.com/oracle/functions/trim.php

[mgckind]

Was this incorporated in #56 ?

[mgckind]

Yes, I saw it. Closing

[kadrlica]

Yep, should be in there.