Projection MCD43C4

[jvanpassel]

Hello,

I have used MODIStsp version 1.3.9 successfully before to download MCD43C4 images. Right now I am working on a new computer and I downloaded the newest MODIStsp version 1.4.0, but I am unable to download these same images.

The difference seems to be with the native output projection in the GUI. In the new version this just says '4008', while in the older version it said '+init=epsg:4008 +proj=longlat +ellps=clrk66 +no_defs'. When I try to load the extent from the same spatial file, I now get the following error message:
'Error in st_crs.character(out_proj_crs) : invalid crs: 4008'

Whether I manually change the projection or not and then start to download the images, I always get the following error message:
[Wed Jun 03 14:01:22 2020] MODIStsp --> Starting processing
[Wed Jun 03 14:01:22 2020] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOTA/MCD43C4.006/
[Wed Jun 03 14:01:26 2020] Retrieving list of available Combined Files for Year 2012
[Wed Jun 03 14:01:27 2020] HDF File: MCD43C4.A2012001.006.2016194224247.hdf already exists on your system. Skipping download!
[Wed Jun 03 14:01:27 2020] [Wed Jun 03 14:01:27 2020] 1 files for date: 2012.01.01 were successfully downloaded!
[Wed Jun 03 14:01:27 2020] Processing Terra Nad_Refl_b1_Red files for date: 2012_01_01
Error in gdal_utils("translate", src_dataset, dst_dataset, opts) :
gdal_utils translate: an error occured
In addition: Warning message:
In CPL_gdaltranslate(source, destination, options, oo) :
GDAL Error 3: Free disk space available is 52379979776 bytes, whereas 1027057032002 are at least necessary. You can disable this check by defining the CHECK_DISK_FREE_SPACE configuration option to FALSE.

I already tried to download the older version of MODIStsp on this computer, but I'm not able to for some reason. Would you know how to solve my problem?

[lbusett]

Hi,

this is reasonably related to recent updates to rgdal / and raster packages that I will have to deal with....

Could you please share your session info ?

Lorenzo

[jvanpassel]

Hi,

Thanks for the quick reply, this is my session info:

R version 3.6.0 (2019-04-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=Dutch_Belgium.1252 LC_CTYPE=Dutch_Belgium.1252 LC_MONETARY=Dutch_Belgium.1252
[4] LC_NUMERIC=C LC_TIME=Dutch_Belgium.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] gWidgetsRGtk2_0.0-86.1 cairoDevice_2.28 gWidgets_0.0-54.2 RGtk2_2.20.36 MODIStsp_1.4.0

loaded via a namespace (and not attached):
[1] gdalUtilities_1.1.0 zoo_1.8-8 sf_0.9-3 lattice_0.20-38 mapview_2.7.8
[6] colorspace_1.4-1 htmltools_0.4.0 stats4_3.6.0 viridisLite_0.3.0 base64enc_0.1-3
[11] rlang_0.4.6 e1071_1.7-3 later_1.0.0 DBI_1.1.0 sp_1.4-2
[16] lifecycle_0.2.0 stringr_1.4.0 munsell_0.5.0 mapedit_0.6.0 raster_3.1-5
[21] htmlwidgets_1.5.1 codetools_0.2-16 fastmap_1.0.1 httpuv_1.5.3.1 crosstalk_1.1.0.1
[26] curl_4.3 parallel_3.6.0 class_7.3-15 leafem_0.1.1 xts_0.12-0
[31] Rcpp_1.0.4.6 KernSmooth_2.23-15 xtable_1.8-4 promises_1.1.0 scales_1.1.1
[36] classInt_0.4-3 satellite_1.0.2 leaflet_2.0.3 webshot_0.5.2 jsonlite_1.6.1
[41] mime_0.9 png_0.1-7 digest_0.6.25 stringi_1.4.6 shiny_1.4.0.2
[46] grid_3.6.0 rgdal_1.5-8 tools_3.6.0 bitops_1.0-6 magrittr_1.5
[51] data.table_1.12.8 xml2_1.3.2 httr_1.4.1 R6_2.4.1 units_0.6-6
[56] compiler_3.6.0

[lbusett]

Ok..,, I may have an idea about what's happening.
Would you be able to share the spatial file you are using and your processing options (you can the options to a JSON file from the GUI)

Lorenzo

[jvanpassel]

How am I supposed to share these files? When I try to add them, I get a notification that this file type is not supported here.

[jvanpassel]

I converted the JSON file to a txt file so I could share them, is this okay?
The modistspv140_options file is the one that doesn't work, while the modistspv139_options file did work on another computer.

modistspv140_options.txt

modistspv139_options.txt

I don't know how to share a shapefile here, so here is some information on it:

testarea<-readOGR(dsn = 'I:/PhD',layer = 'testarea') #small testarea within Brazil and Bolivia for faster running of script
OGR data source with driver: ESRI Shapefile
Source: "I:\PhD", layer: "testarea"
with 1 features
It has 1 fields
Integer64 fields read as strings: id
crs(testarea)
CRS arguments: +proj=longlat +datum=WGS84 +no_defs
Warning message:
In proj4string(x) : CRS object has comment, which is lost in output

[lbusett]

thanks. To share the shapefile, you could try compressing it to zip (all 3/5 files) and attach it.

[jvanpassel]

Aha thank you, here is my shapefile:

testarea.zip

[jvanpassel]

Hi,

Would you have any idea how to solve my problem?

Johanna

[lbusett]

@jvanpassel

sorry - I had very little time, these days. I'll try to have a look as soon as i can.

Lorenzo

[lbusett]

Ok, I can replicate this: looking into it now.

[lbusett]

It appears that gdalbuildvrt does not recognize correctly anymore the "corners" of the HDF files in 4008 projection.
For example, this:

gdalinfo HDF4_EOS:EOS_GRID:"D:/RTemp/RtmpaKR6yR/MODIStsp/HDFs/MCD43C4.A2016001.006.2016196180021.hdf":MCD_CMG_BRDF_0.05Deg:Perc
ent_Snow

gives:

Corner Coordinates:
Upper Left  (  -0.0500000,   0.0250000) (  0d 3' 0.00"W,  0d 1'30.00"N)
Lower Left  (  -0.0500000,  -0.0250000) (  0d 3' 0.00"W,  0d 1'30.00"S)
Upper Right (   0.0500000,   0.0250000) (  0d 3' 0.00"E,  0d 1'30.00"N)
Lower Right (   0.0500000,  -0.0250000) (  0d 3' 0.00"E,  0d 1'30.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)

, that is completely wrong.

Would you be able to test this also on your machine? If that is the case, I will have to find a way to reassign the corner coordinates before processing.

Lorenzo

[jvanpassel]

Hi,

I used this code in R instead as it didn't work in the command prompt:

gdalinfo('I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf')

And this is what I got back (I deleted most lines to make it more clear):
[1] "Driver: HDF4/Hierarchical Data Format Release 4"
[2] "Files: I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf"
[3] "Size is 512, 512"
[4] "Coordinate System is `'"
[84] "Corner Coordinates:"
[85] "Upper Left ( 0.0, 0.0)"
[86] "Lower Left ( 0.0, 512.0)"
[87] "Upper Right ( 512.0, 0.0)"
[88] "Lower Right ( 512.0, 512.0)"
[89] "Center ( 256.0, 256.0)"

[lbusett]

@jvanpassel

to see what I am needing, you should run the command on an SDS. Something like.

gdalinfo HDF4_EOS:EOS_GRID:'I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf':MCD_CMG_BRDF_0.05Deg:Percent_Snow

should work.

[jvanpassel]

Hi,
What exactly is an SDS? I tried running this command on different platforms, but I always got error messages.

[lbusett]

an SDS is one of the subdatasets of an HDF file. can you share your error message?

Lorenzo

[jvanpassel]

Thanks, I was able to specify a SDS in R and then I got the following message:

[81] "Corner Coordinates:"
[82] "Upper Left ( -0.0500000, 0.0250000) ( 0d 3' 0.00"W, 0d 1'30.00"N)"
[83] "Lower Left ( -0.0500000, -0.0250000) ( 0d 3' 0.00"W, 0d 1'30.00"S)"
[84] "Upper Right ( 0.0500000, 0.0250000) ( 0d 3' 0.00"E, 0d 1'30.00"N)"
[85] "Lower Right ( 0.0500000, -0.0250000) ( 0d 3' 0.00"E, 0d 1'30.00"S)"
[86] "Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)"

[lbusett]

Yes, same problem, apparently....

Can you tell me which GDAL version you have installed?

Lorenzo

[jvanpassel]

My R rgdal version is 1.5-10 and it uses GDAL version 3.0.4.

[lbusett]

OK. same here... It's probably a regressione related to changes in GDAL.

I'll see if I can manage a quick fix.

Lorenzo

[przell]

Hi Lorenzo,
we are also having trouble with the Projections. On Ubuntu 18.04, gdal 3.0.4, proj 7.0.0 and MODIStsp 1.4.0. We are using the non-gui processing.
Currently the parameter output_proj4 = "32632" and output_proj4 = "+proj=utm +zone=32 +datum=WGS84 +units=m +no_defs +type=crs" in the json file fail with the error:
"Internal error in out_res_sel, outproj_str or full_ext. Aborting!"

Having a quick glimpse at your functions I have noted the following:

• MODIStsp_process.R: uncommented this in line 164 # outproj_str <- sf::st_as_text(sf::st_crs(check_projection(outproj_str))). Is this intended?
• MODIStsp_process_bands.R: you are refering to outproj$wkt in line 357 -> how can this work if the input is a string and not converted since it is uncommented
• proj_bbox.R: You are handling the projection differently here. Might be worth to align this.

I can't provide the used json now since I don't have acces to the VM of the colleague who has encountered this problem. But could try to get hold of it, if it helps you.

Sorry this was just a quick review from my side. No extensive testing. Forgive me if I have missed something or am completely wrong.
Hopefully this is of help.
Best,
Peter

[lbusett]

Hi @przell

thanks for signaling this. I do not think that what iyoiu reported is the culprit, though, because "output_proj4" is automatically converted to a crs object already in MODIStps() main function (line 396).

To understand what's happening, I'd need to have a look at the processing parameters. In particular MODIS product, output resolution, output bounding box.

Could you provide me that info?

[przell]

Hi Lorenzo,
ok, perfect. Unfortunately, I hadn't checked the main MODIStsp() function 🤦 .
I will provide you with the json files and exact errors soon!
Thanks for your quick response!

[lbusett]

@jvanpassel

I probably found a solution for you problem - I'll provide a fix soon for you to test.

[lbusett]

@jvanpassel

I pushed a patch to github - ca you try installing from github and see if it solves your issue?

remotes::install_github("ropensci/MODIStsp", ref = "master")

[jvanpassel]

Yes it worked perfectly, thank you so much!

[przell]

Thanks for the quick fix!
We have reinstalled and tested for our use case but we are still receiving the same error. Here our json file. Originally we are using local MODIS files. So I also attached one using the online archive.
MODIStsp_GQ_offline_linux.json.txt
MODIStsp_GQ_online_linux.json.txt
and also the error messages:
MODIStsp_GQ_offline_linux.json:

GDAL version in use: 3.0.4
[Fri Jun 19 09:29:30 2020] MODIStsp --> Starting processing
[Fri Jun 19 09:29:30 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Fri Jun 19 09:29:35 2020] Retrieving list of available ` Terra ` Files for Year 2008
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Fri Jun 19 09:29:39 2020] HDF File: MOD09GQ.A2008001.h18v04.006.2015169042051.hdf already exists on your system. Skipping download!
[Fri Jun 19 09:29:39 2020] [Fri Jun 19 09:29:39 2020] 1 files for date: 2008.01.01 were successfully downloaded!
[Fri Jun 19 09:29:41 2020] Processing and Reprojecting Terra b1_Red files for date: 2008_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!

MODIStsp_GQ_online_linux.json:

GDAL version in use: 3.0.4
[Fri Jun 19 09:39:53 2020] MODIStsp --> Starting processing
[Fri Jun 19 09:39:53 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Fri Jun 19 09:40:07 2020] Retrieving list of available ` Terra ` Files for Year 2011
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Fri Jun 19 09:40:09 2020] Downloading Terra Files for date: 2011_01_01 : 1 of:  1
[Fri Jun 19 09:40:52 2020] [Fri Jun 19 09:40:52 2020] 1 files for date: 2011.01.01 were successfully downloaded!
[Fri Jun 19 09:40:52 2020] Processing and Reprojecting Terra b1_Red files for date: 2011_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!

Best,
Peter

[lbusett]

@przell

your problem is a different one that I will have to investigate. Could you please open a new issue? Closing this now.

Lorenzo

[lbusett]

Hi Lorenzo,
we are also having trouble with the Projections. On Ubuntu 18.04, gdal 3.0.4, proj 7.0.0 and MODIStsp 1.4.0. We are using the non-gui processing.
Currently the parameter output_proj4 = "32632" and output_proj4 = "+proj=utm +zone=32 +datum=WGS84 +units=m +no_defs +type=crs" in the json file fail with the error:
"Internal error in out_res_sel, outproj_str or full_ext. Aborting!"

Having a quick glimpse at your functions I have noted the following:

• MODIStsp_process.R: uncommented this in line 164 # outproj_str <- sf::st_as_text(sf::st_crs(check_projection(outproj_str))). Is this intended?
• MODIStsp_process_bands.R: you are refering to outproj$wkt in line 357 -> how can this work if the input is a string and not converted since it is uncommented
• proj_bbox.R: You are handling the projection differently here. Might be worth to align this.

I can't provide the used json now since I don't have acces to the VM of the colleague who has encountered this problem. But could try to get hold of it, if it helps you.

Sorry this was just a quick review from my side. No extensive testing. Forgive me if I have missed something or am completely wrong.
Hopefully this is of help.
Best,
Peter

@przell In the meantime, I tested on my machine with your options file, and everything seems to work fine.
Can I ask you to share your / your colleague's session info?

[przell]

Here are the spatial dependencies:

> library(sf)
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 7.0.0

And here is the session Info:

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_1.5-10   sp_1.3-2       MODIStsp_1.4.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4.6        lattice_0.20-38     prettyunits_1.1.1   png_0.1-7           class_7.3-15        ps_1.3.3            zoo_1.8-8          
 [8] rprojroot_1.3-2     assertthat_0.2.1    digest_0.6.25       mime_0.9            R6_2.4.1            backports_1.1.7     stats4_3.6.0       
[15] e1071_1.7-3         httr_1.4.1          pillar_1.4.1        rlang_0.4.6         curl_4.3            rstudioapi_0.11     data.table_1.12.8  
[22] callr_3.4.3         raster_3.0-12       webshot_0.5.2       stringr_1.4.0       htmlwidgets_1.5.1   munsell_0.5.0       shiny_1.4.0.2      
[29] compiler_3.6.0      httpuv_1.5.4        pkgconfig_2.0.3     base64enc_0.1-3     pkgbuild_1.0.8      htmltools_0.4.0     gdalUtilities_1.1.0
[36] mapedit_0.6.0       tidyselect_0.2.5    tibble_2.1.3        codetools_0.2-16    gWidgets_0.0-54     mapview_2.7.8       fansi_0.4.1        
[43] viridisLite_0.3.0   withr_2.2.0         crayon_1.3.4        dplyr_0.8.1         later_1.1.0.1       sf_0.9-4            bitops_1.0-6       
[50] grid_3.6.0          jsonlite_1.6.1      satellite_1.0.2     xtable_1.8-4        DBI_1.1.0           magrittr_1.5        units_0.6-6        
[57] scales_1.0.0        KernSmooth_2.23-15  cli_2.0.2           stringi_1.4.6       promises_1.1.1      remotes_2.1.1       leaflet_2.0.3      
[64] xml2_1.2.0          xts_0.12-0          tools_3.6.0         leafem_0.1.1        glue_1.4.1          purrr_0.3.4         crosstalk_1.1.0.1  
[71] processx_3.4.2      parallel_3.6.0      fastmap_1.0.1       colorspace_1.4-1    classInt_0.4-3     
> MODIStsp(
+   gui = FALSE,
+   options_file = "/mnt/CEPH_PROJECTS/ECOPOTENTIAL/IT-GRANPARADISO/Stephanie/Data/NDVI/MOD09GQ/MODIStsp_GQ_online_linux.json",
+  # options_file = "//usr/local/lib/R/site-library/MODIStsp/testdata/test_MOD13A2.json",
+  # spatial_file_path = NULL,
+ #  scroll_window = FALSE,
+  # test = NULL,
+ #  n_retries = 20,
+   verbose = TRUE
+ )
MODIStsp would like to save a "MODIStsp_Previous.json" file 
containing information about its last successfull run in the folder 
`.../your-r-library/MODIStsp/ExtData/.` 

 Do you authorize this? 

1: Yes - "MODIStsp_Previous.json" will be saved permanently and updated after
after each successfull run of the tool. You will not see this message anymore.

2: No - Previous options will be written to `tempdir` and will be lost when you
exit `R`. You will see this message at each `MODIStsp` execution.

Choice (1/2): 2
GDAL version in use: 3.0.4
[Tue Jun 23 07:20:37 2020] MODIStsp --> Starting processing
[Tue Jun 23 07:20:37 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Tue Jun 23 07:20:48 2020] Retrieving list of available ` Terra ` Files for Year 2011
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Tue Jun 23 07:20:52 2020] HDF File: MOD09GQ.A2011001.h18v04.006.2015216011131.hdf already exists on your system. Skipping download!
[Tue Jun 23 07:20:52 2020] [Tue Jun 23 07:20:52 2020] 1 files for date: 2011.01.01 were successfully downloaded!
[Tue Jun 23 07:20:52 2020] Processing and Reprojecting Terra b1_Red files for date: 2011_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!


**************************************************************************************************************************************************

I hope this helps. Thanks for the support!

[lbusett]

@przell

You seem to have a quite old version of the "raster" package.

Could you try to update it and see if it solves the issue?

Lorenzo