Check CF-compliance of test files

[mortenwh]

We should make sure our netcdf test files are cf-compliant... Use, e.g., http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl for checking all of them

[aleksandervines]

@mortenwh
Do you just mean the two files here https://github.com/nansencenter/nansat/tree/master/nansat/tests/data
Or these as well ftp://ftp.nersc.no/nansat/test_data/
And are there other places we got netcdf files?

[aleksandervines]

I've identified the following netcdf files in the above locations, and divided them into two groups. Do you agree on this?

Files that should be cf-compliant:
ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset.nc
ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset_time.nc
ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/ARC-METNO-ARC-TOPAZ4_2_PHYS-FOR-TDS-x10-X30-y55-Y73-20170518-20170526.nc
ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS-x10-X30-y55-Y73-201705181200-201705271200.nc
nansat/tests/data/arctic.nc
nansat/tests/data/complex.nc

netcdf files that should be as original, and not modified:
ftp://ftp.nersc.no/nansat/test_data/globcolour_l3b/L3b_20150930-20151007__GLOB_4_GSM-MODVIR_CHL1_8D_00.nc
ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/A2015223043500.L2_LAC.NorthNorwegianSeas.nc
ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/V2016020125011.L2_NPP_OC.nc

[mortenwh]

I think the files on ftp are original files not created by us. At least, that is the case for ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/ARC-METNO-ARC-TOPAZ4_2_PHYS-FOR-TDS-x10-X30-y55-Y73-20170518-20170526.nc ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS-x10-X30-y55-Y73-201705181200-201705271200.nc ftp://ftp.nersc.no/nansat/test_data/globcolour_l3b/L3b_20150930-20151007__GLOB_4_GSM-MODVIR_CHL1_8D_00.nc ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/A2015223043500.L2_LAC.NorthNorwegianSeas.nc ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/V2016020125011.L2_NPP_OC.nc In principle, those files are cf-compliant (as long as the producers have done it correctly). We need to check that all the files we have created for fast testing in nansat are cf-compliant, and I think then that should be the following: ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset.nc ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset_time.nc nansat/tests/data/arctic.nc nansat/tests/data/complex.nc Note also that we have test files at ftp://ftp.nersc.no/openwind/test_data/ but thay are all in their original formats and doesn't need compliancy tests..

…

On 10 January 2018 at 15:38, Aleksander Vines ***@***.***> wrote: I've identified the following netcdf files in the above locations, and divided them into two groups. Do you agree on this? Files that should be cf-compliant ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset.nc ftp://ftp.nersc.no/nansat/test_data/generic/generic_dataset_time.nc ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/ARC-METNO- ARC-TOPAZ4_2_PHYS-FOR-TDS-x10-X30-y55-Y73-20170518-20170526.nc ftp://ftp.nersc.no/nansat/test_data/netcdf_cf/GLOBAL_ ANALYSIS_FORECAST_PHY_001_024-TDS-x10-X30-y55-Y73- 201705181200-201705271200.nc nansat/tests/data/arctic.nc nansat/tests/data/complex.nc netcdf files that should be as original, and not modified: ftp://ftp.nersc.no/nansat/test_data/globcolour_l3b/L3b_ 20150930-20151007__GLOB_4_GSM-MODVIR_CHL1_8D_00.nc ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/A2015223043500.L2_LAC. NorthNorwegianSeas.nc ftp://ftp.nersc.no/nansat/test_data/obpg_l2_nc/V2016020125011.L2_NPP_OC.nc — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGqBWbuHJ5GOBpgvGoltzjIsETp0mMPks5tJMt_gaJpZM4MbXlw> .

-- T.: (+47) 915 47 844

[aleksandervines]

I would think files that are under the netcdf_cf folder should comply 100% with cf, which it seems they don't. But I also agree that files not created by us should stay as original.

Anyway, for the 4 files you mentioned we should check, only 1 is not comliant, generic_dataset.nc
The issue here is that ban_000 contains an attribute that starts with underscore (_Unsigned), which per cf_definition is not allowed (the only exception for us is _FillValue), I've had a mail correspondence with the person developing the cf-checker previously, regarding similar issue, which resultet in this ticket on the CF convention http://cf-trac.llnl.gov/trac/ticket/157 and this issue on a Unidata repository Unidata/rosetta#68

The use of the _Unsigned seems like a bit of a mess:

• the NUG refers to using _Unsigned as a "new proposed convention" https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
• CF mentions the use of specifying the range to indicate that its unsigned http://cfconventions.org/cf-conventions/cf-conventions.html#_data_types
• Unsigned byte according to CF conventions Unidata/netcdf4-python#493
• Best practice when the _Unsigned attribute is present in NetCDF files pydata/xarray#1444
• Interpretation of reserved _Unsigned attribute written by netCDF-Java Unidata/netcdf4-python#656
• https://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg13836.html

As a conclusion to all these discussions and documentation, I suggest that we chose to follow Unidata recommendations in the NUG over the intepretation of CF convention from the CF-convention creators. I.e. We keep _Unsigned as an attribute. We are also using the valid_range attribute, so we are following both recommendations on how to specify that it is unsigned, I think that should suffice.

@akorosov @mortenwh do you have an objection to my conclusion?

[mortenwh]

agree :)

…

On 11 January 2018 at 12:13, Aleksander Vines ***@***.***> wrote: I would think files that are under the netcdf_cf folder should comply 100% with cf, which it seems they don't. But I also agree that files not created by us should stay as original. Anyway, for the 4 files you mentioned we should check, only 1 is not comliant, generic_dataset.nc The issue here is that ban_000 contains an attribute that starts with underscore (_Unsigned), which per cf_definition is not allowed (the only exception for us is _FillValue), I've had a mail correspondence with the person developing the cf-checker previously, regarding similar issue, which resultet in this ticket on the CF convention http://cf-trac.llnl.gov/trac/ ticket/157 and this issue on a Unidata repository Unidata/rosetta#68 <Unidata/rosetta#68> The use of the _Unsigned seems like a bit of a mess: - the NUG refers to using _Unsigned as a "new proposed convention" https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html <https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html> - CF mentions the use of specifying the range to indicate that its unsigned http://cfconventions.org/cf-conventions/cf-conventions. html#_data_types - Unidata/netcdf4-python#493 <Unidata/netcdf4-python#493> - pydata/xarray#1444 <pydata/xarray#1444> - Unidata/netcdf4-python#656 <Unidata/netcdf4-python#656> - https://www.unidata.ucar.edu/support/help/MailArchives/ netcdf/msg13836.html As a conclusion to all these discussions and documentation, I suggest that we chose to follow Unidata recommendations in the NUG over the intepretation of CF convention from the CF-convention creators. I.e. We keep _Unsigned as an attribute. We are also using the valid_range attribute, so we are following both recommendations on how to specify that it is unsigned, I think that should suffice. @akorosov <https://github.com/akorosov> @mortenwh <https://github.com/mortenwh> do you have an objection to my conclusion? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGqBRO1sqahYmATSKOgpy_AtbCK0WEnks5tJezhgaJpZM4MbXlw> .

-- T.: (+47) 915 47 844