MSVC build fails owing to overly long build directory names

[tp-m]

So I'm building gst-build on Windows, with glib as a subproject. The build fails with the following error message:

[1/1857] Compiling C object subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj. FAILED: subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj cl @subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj.rsp subprojects/glib/gio/tests/gdbus-object-manager-example/gdbus-example-objectmanager-generated.c: fatal error C1041: cannot open program database 'subprojects\\glib\\gio\\tests\\gdbus-object-manager-example\\subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha\\meson-generated_.._gdbus-example-objectmanager-generated.c.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS

Turns out the problem is the overly long build directory paths here, exceeding the default 260 path limit.

The build works fine if I use c:\build as build directory instead.

I understand why these paths might be needed in the flat layout case, but Meson knows that we don't use a flat layout here, so I think it shouldn't be using these long cluttery paths in this case.

[tp-m]

cc @sarum9in

[dcbaker]

We've run into the in mesa as well.

[jpakkane]

I thought we were already using response files for these but apparently we only do that for linker invocations.

[tp-m]

I think response files are already being used (I remember being irritated by ninja -v not showing any details), but won't help. They help bypass the max command line length aiui.

It looks to me like the args make it fine to cl.exe but something somewhere then constructs an absolute path and that blows the default 260 char path limit. And Meson makes the paths much longer than they need to be in this case.

[sarum9in]

So the problem is #3246.
This is the code that generates the target id, which is the major part of the name.

        name_part = self.name.replace('/', '@').replace('\\', '@')
        assert not has_path_sep(self.type_suffix())
        myid = name_part + self.type_suffix()
        if self.subdir:
            subdir_part = self.subdir.replace('/', '@').replace('\\', '@')
            myid = subdir_part + '@@' + myid

It was shorter in the past:

        base = self.name + self.type_suffix()
        if self.subproject == '':
            return base
        return self.subproject + '@@' + base

The problem with previous approach is that it did not allow same ids in different subdirectories. New approach takes this into account, but unfortunately it effectively doubles the name size.

I have a possible workaround in mind: let's hash subdir_part of id with some short stable hashing function (e.g. sha). So the code would effectively be

        name_part = self.name.replace('/', '@').replace('\\', '@')
        assert not has_path_sep(self.type_suffix())
        myid = name_part + self.type_suffix()
        if self.subdir:
            subdir_part = self.subdir.replace('/', '@').replace('\\', '@')
            myid = SOME_SHA_FUNC(subdir_part) + '@@' + myid

This will not look as nice unfortunately but will solve the problem.
Another option is to check the size of the subdir_part and hash it only if it exceeds N (e.g. 100) symbols. Or similar but for the whole id, which will provide more consistent experience.

Thoughts?

[tp-m]

I don't mind the hash - and for me the paths are quite unwieldy anyway, so I don't think there's too much lost from an aesthetic point of view - perhaps an abbreviated/short hash would be enough here? (8 chars)

[xclaesse]

On a related note, using a hash could be needed to uniquify target ID per install directory. See #4019 (comment).

[sarum9in]

@tp-m I will prepare a patch today/tomorrow and create a PR. Let's give it a try if it works for your builds and see if we get any other feedback regarding more obfuscated IDs.

[sarum9in]

@tp-m could you please try #4263 and comment if it works for you?

[jon-turney]

Does setting the registry key mentioned in https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file#maximum-path-length-limitation help?

[nirbheek]

Does setting the registry key mentioned in https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file#maximum-path-length-limitation help?

That does help, and it is also an option in the Python installer, but it only works on Windows 10 so we need to fix this either way.

[lazka]

I've opened https://gitlab.gnome.org/GNOME/glib/merge_requests/370 in glib to work this around temporarily.

[ghost]

This issue still affects files generated by the Windows module for compiling Windows rc files. The build fails with the error:

C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(5040,39): error MSB4023: Cannot evaluate the item metadata "%(FullPath)". The item metadata "%(FullPath)" cannot be applied to the path "fedb06e@@Windows resource for file 'src_maintenance-utility_maintenance-utility.rc'@cus\Windows resource for file 'src_maintenance-utility_maintenance-utility.rc'.exe". C:\Users\omninull\projects\driver\subprojects\utilities\vs-projects\vs2015-x86_64-debug\src\maintenance-utility\fedb06e@@Windows resource for file 'src_maintenance-utility_maintenance-utility.rc'@cus\Windows resource for file 'src_maintenance-utility_maintenance-utility.rc'.exe

It seems that the Windows module wasn't updated when this change was made.

Additionally, I started attempting a simple fix but still ran into the path limit issue due to prepending "Windows resource for file " to the beginning of the name, causing a failure with a log file:

C:\Users\omninull\projects\driver\subprojects\utilities\vs-projects\vs2015-x86_64-debug\src\maintenance-utility\fedb06e@@Windows resource for file 'maintenance-utility.rc'@cus\Windows .E9BEDD92.tlog\Windows resource for file 'maintenance-utility.rc'.lastbuildstate

Granted, this could be fixed with some directory layout changes to my project, but it would be nice to not have to do that.

Note, this only occurs when building with a Visual Studio solution file. Building via ninja is fine.

[braindevices]

still affected, I think meson need to watch the filename/path length and use hash for the ones exceed limit.

[romeoxbm]

Still affected as well