Do not fail to build if 'zig build-lib' etc. arguments exceed OS limits

[slimsag]

These patches fix a major issue which prevents using Zig's build system from being viable in larger projects with lots of source files on Windows (and macOS + Linux, to a lesser degree.)

In Mach engine we're seeing command line arguments to zig build-lib exceed the 32 KiB limit that Windows imposes in the CreateProcess API, due to the number of source files and compiler flags we need in order to build WebGPU/Dawn.

I've fixed this issue in three steps (best reviewed as individual commits):

1. Refactored zig build-lib [args] args handling to use an iterator, in anticipation of response file support.
2. Added response file support to zig build-lib @args.rsp (and build-exe, etc. friends) so you can pass them a text file of arguments instead.
3. Changed the Builder to check if the arguments it would pass to any zig build command are >30 KiB and, if so, use e.g. a zig build-lib @zig-cache/args/<SHA2 of args> response file instead.

Fixes #10693
Fixes hexops/mach#167

[kubkon]

CreateProcess is a Win32 function, correct? Perhaps you'd be willing to investigate if using NT directly would not impose such a limit in Windows?

[slimsag]

@kubkon

Perhaps you'd be willing to investigate if using NT directly would not impose such a limit in Windows?

I was not able to find such an API, maybe it'd be something undocumented?

In any case, I should note that macOS and Linux also have limitations here (256 KiB and 256 MiB, respectively): I think it would be advisable to have response file support like this in place, as clang and others use the same approach to solve this issue.

[slimsag]

Addressed all the feedback :)

[slimsag]

friendly ping :) I have some quite cool tech demos that will be blocked on this fix landing

[andrewrk]

Thanks for the enhancement!

Looks like this introduce a regression in zig run:

+ /Users/runner/work/1/s/build.host/release/bin/zig run ../doc/docgen.zig -- /Users/runner/work/1/s/build.host/release/bin/zig ../doc/langref.html.in release/docs/langref.html
thread 135918 panic: expected output arg
/Users/runner/work/1/s/doc/docgen.zig:36:49: 0x100c16d2f in main (docgen)
    const out_file_name = args_it.next() orelse @panic("expected output arg");
                                                ^
/Users/runner/work/1/s/build.host/release/lib/zig/std/start.zig:561:37: 0x100c88098 in std.start.callMain (docgen)
            const result = root.main() catch |err| {
                                    ^
/Users/runner/work/1/s/build.host/release/lib/zig/std/start.zig:495:12: 0x100c27577 in std.start.callMainWithArgs (docgen)
    return @call(.{ .modifier = .always_inline }, callMain, .{});
           ^
/Users/runner/work/1/s/build.host/release/lib/zig/std/start.zig:460:12: 0x100c274b5 in std.start.main (docgen)
    return @call(.{ .modifier = .always_inline }, callMainWithArgs, .{ @intCast(usize, c_argc), c_argv, envp });
           ^

[andrewrk]

Reverted in 5ab5e2e