bool(LazySelectSequence) can return None

[johncmerfeld]

Apache Airflow version

2.10.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The downstream effect is that

• when doing an Xcom pull for a group of task IDs
• when the returned list is empty
• an exception is raised because Jinja fails to render the template

[2024-11-13, 18:05:16 UTC] {abstractoperator.py:780} ERROR - Exception rendering Jinja template for task '...'
Traceback (most recent call last):
  File "airflow/models/abstractoperator.py", line 772, in _do_render_template_fields
    rendered_content = self.render_template(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "airflow/template/templater.py", line 171, in render_template
    return self._render(template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "airflow/models/abstractoperator.py", line 727, in _render
    return super()._render(template, context, dag=dag)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "airflow/template/templater.py", line 126, in _render
    return render_template_as_native(template, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "airflow/utils/helpers.py", line 306, in render_template_as_native
    return render_template(template, cast(MutableMapping[str, Any], context), native=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "airflow/utils/helpers.py", line 295, in render_template
    return jinja2.nativetypes.native_concat(nodes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "jinja2/nativetypes.py", line 25, in native_concat
    head = list(islice(values, 2))
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 24, in root
  File "jinja2/async_utils.py", line 45, in wrapper
    return normal_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "jinja2/filters.py", line 1352, in sync_do_list
    return list(value)
           ^^^^^^^^^^^
  File "jinja2/filters.py", line 1492, in sync_do_map
    if value:
       ^^^^^
TypeError: __bool__ should return bool, returned NoneType

What you think should happen instead?

In this case an empty list should be returned

The root cause is that LazySelectSequence's __bool__ method is not airtight. I don't understand the internals well enough to know why this is so, but this is clearly an internal error, given a valid template:

"{{  ti.xcom_pull(task_ids=['a', 'b', 'c'], key='return_value')  | map(attribute=0) | list }}"

How to reproduce

• Use a template like this to pull the return values from a list of tasks: "{{ ti.xcom_pull(task_ids=['a', 'b', 'c'], key='return_value') | map(attribute=0) | list }}"
• Ensure that all of those tasks Skip or otherwise do not return a value

Operating System

Ubuntu 24.04.1

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

• Root cause is in Unify lazy db sequence implementations #39426
• Seems related to issue XCom_pull of multiple task_ids not returning the list values #41983
• Can confirm we do not have this problem on 2.9.x

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.