include in build_flags expands path when it should not expand

[CapnBry]

What kind of issue is this?

• PlatformIO Core.
  If you’ve found a bug, please provide an information below.

Configuration

Operating system:

Windows 10 Pro 22H2 19045.3086

PlatformIO Version (platformio --version):

PlatformIO Core, version 6.1.8
PlatformIO Core, version 6.1.9a1

Description of problem

A continuation of #4682, when an environment's build_flags contains an -include item with a relative path and the file is in gcc's include path, the build fails as the builder has converted the relative path into an invalid absolute one. The difference between this and #4682 is that issue had a variable in it that needed to be expanded, and this is strictly a relative path to an item in the gcc include path. The documentation states that -include:

Process file as if #include "file" appeared as the first line of the primary source file.

However, in 6.1.8 this has changed as the build process now replaces any -include with an absolute path, even when the path is not valid. In 6.1.7 and previous versions the following flags work build_flags = -Iinclude -include XXX.h if XXX.h is in ${PROJECT_DIR}/include. The path to XXX.h gets expanded in 6.1.8 to the builder directory:

C:\Users\bmayland\.platformio\platforms\ststm32\builder\XXX.h

Setting the build_flags = -include "${PROJECT_DIR}/include/XXX.h does not work in 6.1.8 as the variable is not expanded, and in 6.1.9a1 the variable is expanded (thanks to 01ab1fa), but it is not backward compatible with the previous versions, or the documentation. Also working is build_flags = -include include/XXX.h

Our project supports building older versions as well, meaning we'd have to cut new releases for all our old versions just due to this incompatibility.

Suggested Fix

Perhaps the include can just not expand the path if the generated absolute path does not exist?

builder.py

    # fix relative path for "-include"
    for i, f in enumerate(result.get("CCFLAGS", [])):
        if isinstance(f, tuple) and f[0] == "-include":
            p = env.subst(f[1].get_path())
            if os.path.exists(p):
                result["CCFLAGS"][i] = (f[0], os.path.abspath(p))
            else:
                result["CCFLAGS"][i] = (f[0], p)

Additional info

I'm sorry I do not have a simple project to reproduce this issue, as our project is somewhat large and I'd have to start something from scratch. This is the project:
https://github.com/ExpressLRS/ExpressLRS/

[Jason2866]

@ivankravets This is a major bug. It breaks project Tasmota completely and every other project that does something like this

build_flags                 = ${esp_defaults.build_flags}
                              -include "sdkconfig.h"

How can we force to use PIO Core v6.1.7 as fast fix?

fyi @barbudor @arendst

[TD-er]

Yep, also broke ESPEasy builds....

How can we force to use PIO Core v6.1.7 as fast fix?

You could perhaps enforce a specific version of Platformio in the requirements.txt file?

[ivankravets]

Sorry, the hot fix 6.1.9 is coming.

[ivankravets]

Please re-test the latest development version (pio upgrade --dev). If it works, we will release 6.1.9. Thanks!

[hvrauhal]

Build succeeds for me in the ELRS Configurator with 6.1.9rc1, and they were broken with the current release.

[TD-er]

Just wondering, how to define such an unreleased version in requirements.txt?
As that's how I handle Python lib versions in my build env.

[ivankravets]

Just pass

platformio==https://github.com/platformio/platformio-core/archive/refs/heads/develop.zip

[TD-er]

Apparently not, see: https://github.com/letscontrolit/ESPEasy/actions/runs/5474537142/jobs/9969453955#step:7:176

[ivankravets]

Sorry, my typo. See syntax https://pip.pypa.io/en/stable/reference/requirements-file-format/

This should work:

platformio @ https://github.com/platformio/platformio-core/archive/refs/heads/develop.zip

[TD-er]

As the first env completed already, I think it did indeed fix the builds again.

[Jason2866]

Can confirm. Latest development version does work.

[ivankravets]

Big THANKS for the confirmation. 6.1.9 is coming!

[ivankravets]

6.1.9 is out!

https://github.com/platformio/platformio-core/releases/tag/v6.1.9

[CapnBry]

Yup I woke up and installed the dev version (which ended up being 6.1.10) which worked, rolled back to the release 6.1.9 (which was released 8 minutes earlier) and it also works again. Thank you!