Templates not being resolved in exts_list

[mboisson]

I am trying to use this :

exts_defaultclass = 'ConfigureMakePythonPackage'
exts_list = [
        ('sip', '4.19.17', {
                'source_urls': 'https://www.riverbankcomputing.com/static/Downloads/sip/4.19.17/',
                'source_tmpl': '%(name)s-%(version)s.tar.gz',
                'checksums': ['12bcd8f4d5feefc105bc075d12c5090ee783f7380728563c91b8b95d0ec45df3'],
                'prefix_opt': '--sysroot=',
                'configopts': 'configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm'
        }),
        ('PyQt5', '5.12.3', {
                'source_urls': 'https://www.riverbankcomputing.com/static/Downloads/PyQt5/5.12.3/',
                'source_tmpl': '%(name)s_gpl-%(version)s.tar.gz',
                'checksums': ['0db0fa37debab147450f9e052286f7a530404e2aaddc438e97a7dcdf56292110'],
                'prefix_opt': '--sysroot=',
                'configopts': 'configure.py --confirm --qmake=%(installdir)s/bin/qmake --verbose --sip=%(installdir)s/bin/sip --sip-incdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)m'
        })
]

However, the templates do not get resolved, and the build fails with

ERROR: Build of /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/Qt5/Qt5-5.12.8-GCCcore-9.3.0.eb failed (err: 'build failed (first 300 chars): cmd " /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm:$EBROOTPYTHON/include/python%(pyshortver)s" exited with exit code 1 and output:\n/bin/bash: -c: l')

I don't know if this is related to #3237 or if there's something I don't understand...

@akesandgren, @boegel ?

[boegel]

@mboisson Can you share (a snippet of) a debug log where it produces into about (failing of) resolving of templates and which template values are supported at that time?

[mboisson]

== 2020-04-30 20:13:01,277 toolchain.py:733 INFO Commands symlinked to /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/ccache via /tmp/eb-qi3pu342/tmpswl0xgsm: gcc, g++, gfortran, gfortran, gfortran
== 2020-04-30 20:13:01,277 filetools.py:434 INFO Command python found at /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python
== 2020-04-30 20:13:01,277 pythonpackage.py:380 INFO Python command being used: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python
== 2020-04-30 20:13:01,277 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=False, prefix="/tmp/"))'
== 2020-04-30 20:13:01,346 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=True, prefix="/tmp/"))'
== 2020-04-30 20:13:01,435 filetools.py:1460 INFO Creating directory /tmp/mboisson/Qt5/5.12.8/GCCcore-9.3.0/sip (parents: True, set_gid: False, sticky: False)
== 2020-04-30 20:13:01,437 run.py:222 INFO running cmd: tar xzf /home/mboisson/.local/easybuild/sources/q/Qt5/sip-4.19.17.tar.gz
== 2020-04-30 20:13:01,533 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print("%s.%s.%s" % sys.version_info[:3])'
== 2020-04-30 20:13:01,576 pythonpackage.py:512 INFO Not auto-enabling checking of $LDSHARED, Python version 3.6.9 is not recent enough
== 2020-04-30 20:13:01,577 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -V
== 2020-04-30 20:13:01,590 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print(sys.executable)'
== 2020-04-30 20:13:01,631 environment.py:97 INFO Environment variable PYTHONNOUSERSITE set to 1 (previously undefined)
== 2020-04-30 20:13:01,632 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print(sys.path)'
== 2020-04-30 20:13:01,682 easyconfig.py:1891 WARNING Unable to resolve template value configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm:         $EBROOTPYTHON/include/python%(pyshortver)s with dict {'arch': 'x86_64', 'nameletter': 's', 'toolchain_name': 'GCCcore', 'toolchain_version': '9.3.0', 'version_major': '4', 'version_minor': '19', 'version_major_minor': '4.19',             'bitbucket_account': 'qt5', 'github_account': 'qt5', 'name': 'sip', 'version': '4.19.17', 'versionsuffix': '', 'versionprefix': '', 'namelower': 'sip', 'nameletterlower': 's', 'installdir': '/home/mboisson/.local/easybuild/software/2019/ Core/qt/5.12.8', 'builddir': '/tmp/mboisson/Qt5/5.12.8/GCCcore-9.3.0'}

[mboisson]

@boegel this bit ?

[Micket]

Without digging through the code, my thoughts are that, since, obviously extensions must have their own scope (else, 'source_tmpl': '%(name)s_gpl-%(version)s.tar.gz' would break, as these must refer to the extension version and name), they also don't get master-config dependencies (and thus javaver, pyver etc.) exposed into the extensions scope... somehow

[mboisson]

It appears indeed that they don't have the master-config dependencies.

[mboisson]

Maybe it needs something like this ? #3237

[mboisson]

Or maybe it is because I'm using multi_deps and not dependencies over Python ? @bartoldeman

[boegel]

I think @Micket is right, this is basically because dependency-related templates like %(pyver)s are not defined. But the 'dependencies' easyconfig parameter does get copied down to the extension though (the whole parent easyconfig basically is, and we overwrite things like name/version afterwards to be correct for the extension, see

easybuild-framework/easybuild/framework/extension.py

Line 98 in 33d0073

self.cfg = self.master.cfg.copy(validate=False)

).

So we need to figure out why %(pyver)s & co are not included in the dict with template values.

@mboisson Can you share the full easyconfig file you're using here, so we can reproduce the issue, or a full log file to stare at?

[mboisson]

Mmm, the full log is 87MB big...

The recipe is https://gist.github.com/mboisson/9032258b0c6649686daa424620f538a7

I think it must have something to do with multi_deps

[Flamefire]

@boegel I ran into that already and investigated while developing #3285. See my comment: #3285 (comment)

TLDR: The extension handling (especially templating) needs a rework. Currently the extensions are used with templates replaced in one place, instantiated later with another round of templating which need to be kept in sync but (I guess) nobody knows what template values are available at which point. My proposal: cfg['ext_list']['options'] needs disabled templating, extensions need to be instantiated before anything is accessed (no direct access to the above cfg['ext_list']['options']) and most problems are solved.

On the pyver issue: This happens because extension replacement and such is done before multi-deps kick in. So at that point the Python dependency is still unknown and especially the python version is unknown. Hence the template resolving fails leading to hard to track errors later on. Hence my PR #3285 but it is held up on the above issue.

[mboisson]

I only understand all of the above partly. However, I am available to do some testing on a set of changes if needs be.

[mboisson]

Note that this is a regression from 4.2.0. It used to work in version 4.1.x (not this specific recipe, but other recipes are broken).

[boegel]

After discussing this with @mboisson and @bartoldeman: two problems were being mixed up.

The issue where %(pyshortver)s is not being resolved when using for an extension is a genuine bug, it seems like that never actually worked.

I was testing this with a modified plotly.py-4.4.1-intel-2019b.eb:

@@ -17,9 +17,11 @@ exts_default_options = {'source_urls': [PYPI_SOURCE]}
 exts_list = [
     ('retrying', '1.3.3', {
         'checksums': ['08c039560a6da2fe4f2c426d0766e284d3b736e355f8dd24b37367b0bb41973b'],
+        'preinstallopts': 'echo "pyshortver: %(pyshortver)s" && ',
     }),
     ('plotly', version, {
         'checksums': ['acc94f17452471ca3446c2ce491c4d1affb99b9ddd9eac4e05614ac4318f8780'],
+        'preinstallopts': 'echo "pyshortver: %(pyshortver)s" && ',
     }),
 ]

The regression that @mboisson referred to turned out to be another (self-inflicted) issue.

[boegel]

I spent some more time looking at this, and the problem boils down to this: for every extension a copy is made of the EasyConfig instance when an Extension instance is created.

This is done using the "copy constructor" method EasyConfig.copy, which effectively creates an entirely new EasyConfig instance, and copies over a couple of things (like self._config, which is the actual dictionary with all easyconfig parameters).

What does not get copied is the list of template values, and this explains the problem here: when multi_deps is involved, dependencies like Python are handled by iterating over a list of lists of build dependencies. And only when actually iterating are build dependencies taken into account for generating template values (see also the recent fix in #3346), which is not the case when creating the new EasyConfig instance.

So, the fix for this could be to basically also copy the template values over in EasyConfig.copy.

I'll get a PR opened for this, would be nice to somehow cover this in the tests too...