fix: make sure to catch jinja2 vars used in single-line for and set statements

[beckermr]

Description

This PR adds two regexes to catch variables used in single-line jinja2 for and set statements. These have been missed for a long time and appear to be the cause of at least one bug. I have not gone through the bug tracker, but I have a hunch this PR might fix other bugs related to similar situations I've seen in the past.

closes #5445

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[codspeed-hq]

CodSpeed Performance Report

Merging #5447 will not alter performance

_{Comparing beckermr:debug-skips-jinja2 (70d396c) with main (977e609)}

Summary

✅ 3 untouched benchmarks

[beckermr]

@kenodegard @isuruf or anyone else, can you review this PR?

It is an important bug fix that I'd like to see in 24.7.2 if we can.

[jaimergp]

Looking at this code for the first time I wonder why are regexes needed when apparently Jinja will happily give you a list of variables in use with three lines. Are we doing this before the selectors are parsed? Would it make sense to preprocess the selectors and then defer to the Jinja AST instead of regexes? Mentioning this because of the single-line limitations and exotic syntax not covered right now. Documenting these limitations in the function docstring might be enough.

[beckermr]

I looked at that stackoverflow for a while, but didn't do any investigating.

Here is how it behaves:

import jinja2
import jinja2.meta

env = jinja2.Environment()
parsed_content = env.parse(
"""
{% set CL_VERSION = "19.29" %}
{% if VCVER is not defined %}
{% set BLAH = "19.29" %}
{% set FOO = "19.29" %}
{% set VCVER = "" %}
{% else %}
{% set BLAH = "19.29" %}
{% set BAZ = "19.29" %}
{% endif %}
{% set clang_major = (CLANG_VERSION|default("19.29")).split(".")[0] %}
{% set cl_minor = CL_VERSION.split(".")[1] %}
{% set vc_major = VCVER.split(".")[0] %}

package:
  name: clang-win-activation
  version: {{ CLANG_VERSION }}

build:
  number: 0
  {% if clang_major|int == 16 and cl_minor|int >= 40 %}
  skip: true
  {% endif %}

outputs:
  - name: clang_win-64
    build:
      run_exports:
        strong:
          - vc >={{ VCVER }}
    requirements:
      run:
        - clang {{ CLANG_VERSION }}.*
"""
)
tokens = jinja2.meta.find_undeclared_variables(parsed_content)

print(tokens)

This gives

% python jinja.py
{'BAZ', 'VCVER', 'FOO', 'BLAH', 'CLANG_VERSION'}

So the behavior is basically that anything that touches a conditional with a default value or uses a default filter is marked as undefined. I'd argue that BLAH should not be returned since it is in both branches.

As for when this happens, searches for jinja2 variables happens after selectors:

conda-build/conda_build/metadata.py

Lines 2856 to 2892 in d908fe1

	def _get_used_vars_meta_yaml(self, force_top_level=False, force_global=False):
	# make variant dict hashable so that memoization works
	variant_keys = tuple(sorted(self.config.variant.keys()))
	reqs_text, recipe_text = self._get_used_vars_meta_yaml_helper(
	force_top_level=force_top_level,
	force_global=force_global,
	apply_selectors=False,
	)
	all_used_selectors = find_used_variables_in_text(
	variant_keys, recipe_text, selectors_only=True
	)
	reqs_text, recipe_text = self._get_used_vars_meta_yaml_helper(
	force_top_level=force_top_level,
	force_global=force_global,
	apply_selectors=True,
	)
	all_used_reqs = find_used_variables_in_text(
	variant_keys, recipe_text, selectors_only=False
	)
	all_used = all_used_reqs.union(all_used_selectors)
	# things that are only used in requirements need further consideration,
	# for omitting things that are only used in run
	if force_global:
	used = all_used
	else:
	requirements_used = find_used_variables_in_text(variant_keys, reqs_text)
	outside_reqs_used = all_used - requirements_used
	requirements_used = trim_build_only_deps(self, requirements_used)
	used = outside_reqs_used | requirements_used
	return used

and so using the jinja2 functions won't work unless we change the order (edit see below for more details, jinja2 functions won't ever work in all cases).

[beckermr]

Other details that are fun.

1. conda-build, depending on the options, will move text blocks around and concatenate them when searching for variables which can break jinja2 as well.
2. The conda-build docs specify that selectors are processed after jinja2 which is the opposite of the search in the sense that jinja2 vars are searched for after selectors have been applied. :/

[beckermr]

@jaimergp @wolfv @h-vetinari I have added a comment explaining why we have to use a regex here. Let's merge this one instead of letting the perfect be the enemy of the good.

[wolfv]

yeah, I think at this point it's impossible to fix this properly, sadly. LGTM!

[beckermr]

@kenodegard @jaimergp @jezdez It appears a dev release of conda on the canary channel is buggy and that is breaking the tests here. What do we do in these cases?

[isuruf]

This regex is not very good.
The following yaml matches glib and glibc both.

{% set a = glibc %}

[beckermr]

any suggestions to fix?

[isuruf]

set_regex = r"(?:^|[^\{])\{%\s*set\s*.*\s*=\s*.*" + v_regex + r"(?![a-zA-Z_0-9])(?:[^%]*?)?%\}"

[beckermr]

See #5514