Genrule producing a directory cannot be provided to pkg_zip

[nickbreen]

Trying to add the result of a directory producing genrule fails.

# A trivial genrule that generates a directory
genrule(
    name = "dir",
    outs = ["dir"],
    cmd = """mkdir $@; touch $@/{file.{a,b}}""",
)

pkg_zip(
    name = "zip",
    srcs = [":dir"],
)

Using release 0.6.0.

Result:

Traceback (most recent call last):
  File "[elided]/bin/external/rules_pkg/pkg/private/zip/build_zip.runfiles/rules_pkg/pkg/private/zip/build_zip.py", line 132, in <module>
    main(arg_parser.parse_args())
  File "[elided]/bin/external/rules_pkg/pkg/private/zip/build_zip.runfiles/rules_pkg/pkg/private/zip/build_zip.py", line 127, in main
    _add_manifest_entry(args, zip_file, entry, default_mode, ts)
  File "[elided]/bin/external/rules_pkg/pkg/private/zip/build_zip.runfiles/rules_pkg/pkg/private/zip/build_zip.py", line 104, in _add_manifest_entry
    with open(src, 'rb') as src:
IsADirectoryError: [Errno 21] Is a directory: 'bazel-out/k8-fastbuild/bin/dir'

Expected a zip file unzip -l :

  Length      Date    Time    Name
---------  ---------- -----   ----
        0  01-01-1980 00:00   dir/file.a
        0  01-01-1980 00:00   dir/file.b
---------                     -------
        0                     2 files

[nickbreen]

Note that the files contained in the dir are not known until the genrule is executed.

I.e. the genrule above would be more accurately described thus:

genrule(
    name = "dir",
    outs = [
        "dir",
    ],
    cmd = """mkdir $@; mktemp --tmpdir=$@ file.XXXXXX; mktemp --tmpdir=$@ file.XXXXXX""",
)

It's actually a webpack output in my case, so there are files with hashes in the file name... therefore unpredictable. I.e.

dist/index.html
dist/js/app.76068f9e.js  
dist/js/app-legacy.2f88e26e.js  
dist/js/chunk-vendors.9efbec0f.js  
dist/js/chunk-vendors-legacy.ea35b136.js
...etc...

[aiuto]

You can't do this with a genrule. Bazel does not preserve unknown outputs of one action and pass them on to dependent actions.

If you need to emit an arbitrary tree of files, then you'll need to create a rule that declares a tree artifact. pkg_zip should be able to consume those just fine.

[nickbreen]

I've refactored to use a rule an have had two slightly different experiences:

Both using a BUILD file similar to this:

generate_a_directory(
    name = "the-directory",
    out = "dist",
)

pkg_zip(
    name = "the-pkg",
    srcs = [":the-directory"],
)

First attempt

Define the pre-declared output with attrs = { "out": attr.output() }

Fails to execute with:

IsADirectoryError: [Errno 21] Is a directory: 'bazel-out/k8-fastbuild/bin/dist'

Which is the same experience with a genrule

Second attempt

Define the name of the output with attrs = { "out": attr.string() }

Declare a 'tree-artifact' directory output with ctx.actions.declare_directory(ctx.attr.out).

This executes, the output of :the-directory has file and sub-directory contents, but the zip file is empty.

[aiuto]

You have to used ctx.actions.declare_directory https://docs.bazel.build/versions/main/skylark/lib/actions.html#declare_directory

You can see a sort of example in our rule for generating sample tree artifacts for use in tests.
https://github.com/bazelbuild/rules_pkg/blob/main/tests/util/defs.bzl

[aiuto]

Oh, wait. I didn't see the second attempt. That might be roughly right, but it is hard to tell just from your one line snippet.
Do you have a more complete reproduction?

[aiuto]

Never mind. It is broken. I updated #309 to remind me.
It probably is not a hard problem, just yank something like the code from biuld_tar.add_tree and put it into build_zip

[nickbreen]

TY, I'll take a look at #309 and read-up. If I can I'll see if I can contribute some tests and a PR.

[janpfeifer]

I reached this thread after having a similar problem with pkg_tar.

My genrule generates a directory, I want pkg_tar to generate a .tar.gz.

This used to work in Bazel 5 (I'm compiling a derivative of tensorflow). But recently I upgraded to Bazel 6, using rules_pkg, and it doesn't seem to work anymore. I assume the same issue as here with pkg_zip.

Any suggestions of work around that would work with pkg_tar ?

thx!!

[aiuto]

If it uses mkdir to create the directory, that is unsupported. Since this is a genrule, that is probably the case.
If you create the directory from a true rule, using declare_directory, that should be fine.

[janpfeifer]

Thanks @aiuto, that makes sense. Indeed I'm creating a directory manually and copying over the files that I need to package (they are the srcs).

In the end I did the less complicated thing, which is just to create another genrule() that invoked tar directly the way I needed -- I code in bash much better than I code in bazel :)