Debug the No File Found error

One of the most common errors comes from a bad dataset definition in datasets_setup.py.

It typically triggers this error:

		error    : "No file found for ds('IGCM_OUT%CM607-LR-pdCtrl-01xx%tas%2000-2009%global%/ccc/store/cont003/thredds%p86caub%IPSLCM6%*%pdControl%*%Analyse%*%monthly%2000_2009%*')"
Traceback (most recent call last):
  File "set_available_period_ts_clim.py", line 78, in <module>
    print cfile(ds(**dataset_dict))
  File "/home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.0.3_CESMEP/climaf/driver.py", line 741, in cfile
    result=ceval(object,format='file',deep=deep)
  File "/home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.0.3_CESMEP/climaf/driver.py", line 212, in ceval
    rep=ceval(extract,userflags=userflags,format=format)
  File "/home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.0.3_CESMEP/climaf/driver.py", line 296, in ceval
    file=ceval_script(cobject,deep,recurse_list=recurse_list) # Does return a filename, or list of filenames
  File "/home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.0.3_CESMEP/climaf/driver.py", line 361, in ceval_script
    recurse_list=recurse_list)
  File "/home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.0.3_CESMEP/climaf/driver.py", line 195, in ceval
    if not rep : raise Climaf_Driver_Error("No file found for %s"%`ds`)
climaf.driver.Climaf_Driver_Error: "No file found for ds('IGCM_OUT%CM607-LR-pdCtrl-01xx%tas%2000-2009%global%[thredds_to_store]%p86caub%IPSLCM6%*%pdControl%*%Analyse%*%monthly%2000_2009%*')"

Note: [thredds_to_store] is the root path to store via thredds

Use the cesmep_simu_finder.py to check that your request actually points at existing files

The cesmep_simu_finder is a way to:

• browse the existing files with a couple of keywords to see what is available
• quickly check that your datasets in datasets_setup file are specified correctly (i.e. points at existing files)

See this page for detailed explanations

Use set_available_period_ts_clim.py to debug

A convenient way to debug this kind of situation is with set_available_period_ts_clim.py . If you work on you comparison standard_comparison, run:

python set_available_period_ts_clim.py standard_comparison/ tas

If it returns the upper error, have a look at the log above the error message that shows the patterns CliMAF is actually using to locate your files:

		debug    : Globbing 0 files for varname on [thredds_to_store]/p86caub/IGCM_OUT/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_t2m.nc :
		debug    : Globbing 0 files for filenamevar on [thredds_to_store]/p86caub/IGCM_OUT/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_histmth.nc:
		debug    : Globbing 0 files for varname on [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_t2m.nc :
		debug    : Globbing 0 files for filenamevar on [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_histmth.nc:
		debug    : Globbing 0 files for varname on [thredds_to_store]/p86caub/IGCM_OUT/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/1M*/CM607-LR-pdCtrl-01xx_1M_2000_2009_1M_t2m.nc :
		debug    : Globbing 0 files for filenamevar on [thredds_to_store]/p86caub/IGCM_OUT/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/1M*/CM607-LR-pdCtrl-01xx_1M_2000_2009_1M_histmth.nc:
		debug    : Globbing 0 files for varname on [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/1M*/CM607-LR-pdCtrl-01xx_1M_2000_2009_1M_t2m.nc :
		debug    : Globbing 0 files for filenamevar on [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/1M*/CM607-LR-pdCtrl-01xx_1M_2000_2009_1M_histmth.nc:
		warning  : no file found for {'status': '*', 'filenameVar': 'histmth', 'domain': 'global', 'ave_length': '*', 'experiment': 'pdControl', 'clim_period_length': '*', 'period': 2000-2009, 'simulation': 'CM607-LR-pdCtrl-01xx', 'project': 'IGCM_OUT', 'frequency': 'monthly', 'DIR': '*', 'variable': 't2m', 'model': 'IPSLCM6', 'root': '[thredds_to_store]', 'login': 'p86caub', 'clim_period': '2000_2009', 'OUT': 'Analyse'}, at these data locations ['${root}/${login}/IGCM_OUT/${model}/${status}/${experiment}/${simulation}/${DIR}/${OUT}/${ave_length}/${simulation}_YYYYMMDD_YYYYMMDD_${frequency}_${variable}.nc', '${root}/${login}/${model}/${status}/${experiment}/${simulation}/${DIR}/${OUT}/${ave_length}/${simulation}_YYYYMMDD_YYYYMMDD_${frequency}_${variable}.nc', '${root}/${login}/IGCM_OUT/${model}/${status}/${experiment}/${simulation}/${DIR}/${OUT}/${frequency}${clim_period_length}/${simulation}_${frequency}_${clim_period}_1M_${variable}.nc', '${root}/${login}/${model}/${status}/${experiment}/${simulation}/${DIR}/${OUT}/${frequency}${clim_period_length}/${simulation}_${frequency}_${clim_period}_1M_${variable}.nc']

Select the pattern that is the closest to the one you are supposed to target.

1. Pay attention to the presence of 'IGCM_OUT' in the path: CliMAF checks for patterns with and without it.
2. do you want to work on 'Analyse' or 'Output' files?
3. Copy the pattern that 'should work' and try to a ls on it:

ls [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_t2m.nc

ls: impossible d'accéder à [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*/Analyse/*/CM607-LR-pdCtrl-01xx_????????_????????_1M_t2m.nc: Aucun fichier ou dossier de ce type

Simple checks:

• delete part of the path/filename to track the error, like:

ls [thredds_to_store]/p86caub/IPSLCM6/*/pdControl/CM607-LR-pdCtrl-01xx/*

• on Ciclad, check that the connection to the thredds is ok:

ls [thredds_to_store]

Keep digging, you will find the error :)!