Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not fail to build if 'zig build-lib' etc. arguments exceed OS limits #10950

Merged
merged 8 commits into from
Feb 23, 2022
Merged

Do not fail to build if 'zig build-lib' etc. arguments exceed OS limits #10950

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0778f64
use process.argsWithAllocator iterator for command line handling
slimsag Feb 20, 2022
e41d4df
enable passing build-[lib|exe|obj] params via @args.rsp file
slimsag Feb 20, 2022
2bd10f4
std: have Builder use response files if zig build args exceed OS limits
slimsag Feb 20, 2022
fb61db1
zig build: use orelse to fetch next argument
slimsag Feb 21, 2022
9c1c474
std: Builder: account for null term considering response files
slimsag Feb 21, 2022
efd473b
std: Builder: use response files for zig test invocations too
slimsag Feb 21, 2022
b341967
zig build: use a custom iterator for merging args and response file
slimsag Feb 21, 2022
c577926
zig build: style: remove unnecessary constant bindings
slimsag Feb 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions lib/std/build.zig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ const fmt_lib = std.fmt;
const File = std.fs.File;
const CrossTarget = std.zig.CrossTarget;
const NativeTargetInfo = std.zig.system.NativeTargetInfo;
const Sha256 = std.crypto.hash.sha2.Sha256;

pub const FmtStep = @import("build/FmtStep.zig");
pub const TranslateCStep = @import("build/TranslateCStep.zig");
Expand Down Expand Up @@ -2888,6 +2889,41 @@ pub const LibExeObjStep = struct {

try zig_args.append("--enable-cache");

// Windows has an argument length limit of 32,766 characters, macOS 262,144 and Linux
// 2,097,152. If our args exceed 30 KiB, we instead write them to a "response file" and
// pass that to zig, e.g. via 'zig build-lib @args.rsp'
var args_length: usize = 0;
for (zig_args.items) |arg| {
args_length += arg.len + 1; // +1 to account for null terminator
}
if (args_length >= 30 * 1024) {
const args_dir = try fs.path.join(
builder.allocator,
&[_][]const u8{ builder.pathFromRoot("zig-cache"), "args" },
);
try std.fs.cwd().makePath(args_dir);

// Write the args to zig-cache/args/<SHA256 hash of args> to avoid conflicts with
// other zig build commands running in parallel.
const partially_quoted = try std.mem.join(builder.allocator, "\" \"", zig_args.items[2..]);
const args = try std.mem.concat(builder.allocator, u8, &[_][]const u8{ "\"", partially_quoted, "\"" });

var args_hash: [Sha256.digest_length]u8 = undefined;
Sha256.hash(args, &args_hash, .{});
var args_hex_hash: [Sha256.digest_length * 2]u8 = undefined;
_ = try std.fmt.bufPrint(
&args_hex_hash,
"{s}",
.{std.fmt.fmtSliceHexLower(&args_hash)},
);

const args_file = try fs.path.join(builder.allocator, &[_][]const u8{ args_dir, args_hex_hash[0..] });
try std.fs.cwd().writeFile(args_file, args);

zig_args.shrinkRetainingCapacity(2);
try zig_args.append(try std.mem.concat(builder.allocator, u8, &[_][]const u8{ "@", args_file }));
}

const output_dir_nl = try builder.execFromStep(zig_args.items, &self.step);
const build_output_dir = mem.trimRight(u8, output_dir_nl, "\r\n");

Expand Down
Loading