Add build.files and negative globs to rattler-build (#819)

This PR adds more globbing options, e.g.:

```yaml
files:
  include:
    - foo/
  exclude:
    - foo/*.txt
```

docs/automatic_linting.md

@@ -10,4 +10,4 @@ To enable automatic linting with the YAML language server, you need to add the f
 # yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json
 ```
 
-**Alternatively**, if you prefer not to add this line to your file, you can install the [JSON Schema Store Catalog extension](https://marketplace.visualstudio.com/items?itemName=remcohaszing.schemastore). This extension will also enable automatic linting for your recipe files.
+**Alternatively**, if you prefer not to add this line to your file, you can install the [JSON Schema Store Catalog extension](https://marketplace.visualstudio.com/items?itemName=remcohaszing.schemastore). This extension will also enable automatic linting for your recipe files.

docs/build_options.md

@@ -8,25 +8,80 @@ There are some specialized build options to control various features:
 
 These are all found under the `build` key in the `recipe.yaml`.
 
-## Always include and always copy files
+## Include only certain files in the package
 
-There are some options that control the inclusion of files in the final package.
+Sometimes you may want to include only a subset of the files installed by the
+build process in your package. For this, the `files` key can be used. Only _new_
+files are considered for inclusion (ie. files that were not in the host
+environment beforehand).
 
-The `always_include_files` option can be used to include files even if they are
-already in the environment as part of some other host dependency. This is normally
-"clobbering" and should be used with caution (since packages should not have any overlapping files).
+```yaml title="recipe.yaml"
+build:
+  # select files to be included in the package
+  # this can be used to remove files from the package, even if they are installed in the
+  # environment
+  files: list of globs
+```
+
+For example, to only include the header files in a package, you could use:
+
+```yaml title="recipe.yaml"
+build:
+  files:
+    - include/**/*.h
+```
+
+Glob patterns throughout the recipe file can also use a flexible `include` /
+`exclude` pair, such as:
+
+```yaml title="recipe.yaml"
+build:
+  files:
+    include:
+      - include/**/*.h
+    exclude:
+      - include/**/private.h
+```
+
+### Glob evaluation
+
+Glob patterns are used throughout the build options to specify files. The
+patterns are matched against the relative path of the file in the build
+directory. Patterns can contain `*` to match any number of characters, `?` to
+match a single character, and `**` to match any number of directories.
+
+For example:
+
+- `*.txt` matches all files ending in `.txt`
+- `**/*.txt` matches all files ending in `.txt` in any directory
+- `**/test_*.txt` matches all files starting with `test_` and ending in `.txt`
+  in any directory
+- `foo/` matches all files under the `foo` directory
+
+The globs are always evaluted relative to the prefix directory. If you have no
+`include` globs, but an `exclude` glob, then all files are included except those
+that match the `exclude` glob. This is equivalent to `include: ['**']`.
 
-The `always_copy_files` option can be used to copy files instead of linking them.
-This is useful for files that might be modified inside the environment (e.g. configuration files).
-Normally, files are linked from a central cache into the environment to save space – that means
-that files modified in one environment will be modified in all environments. This is not always
-desirable, and in that case you can use the `always_copy_files` option.
+## Always include and always copy files
 
-??? note "How `always_copy_files` works"
-    The `always_copy_files` option works by setting the `no_link` option in the
-    `info/paths.json` to `true` for the files in question. This means that the
-    files are copied instead of linked when the package is installed.
+There are some options that control the inclusion of files in the final package.
 
+The `always_include_files` option can be used to include files even if they are
+already in the environment as part of some other host dependency. This is
+normally "clobbering" and should be used with caution (since packages should not
+have any overlapping files).
+
+The `always_copy_files` option can be used to copy files instead of linking
+them. This is useful for files that might be modified inside the environment
+(e.g. configuration files). Normally, files are linked from a central cache into
+the environment to save space – that means that files modified in one
+environment will be modified in all environments. This is not always desirable,
+and in that case you can use the `always_copy_files` option.
+
+??? note "How `always_copy_files` works" The `always_copy_files` option works by
+setting the `no_link` option in the `info/paths.json` to `true` for the
+files in question. This means that the files are copied instead of linked
+when the package is installed.
 
 ```yaml title="recipe.yaml"
 build:
@@ -38,19 +93,6 @@ build:
   always_copy_files: list of globs
 ```
 
-!!! note "Glob patterns"
-    Glob patterns are used througout the build options to specify files. The
-    patterns are matched against the relative path of the file in the build
-    directory.
-    Patterns can contain `*` to match any number of characters, `?` to match a
-    single character, and `**` to match any number of directories.
-
-    For example:
-
-    - `*.txt` matches all files ending in `.txt`
-    - `**/*.txt` matches all files ending in `.txt` in any directory
-    - `**/test_*.txt` matches all files starting with `test_` and ending in `.txt` in any directory
-
 ## Merge build and host environments
 
 In very rare cases you might want to merge the build and host environments to
@@ -135,7 +177,6 @@ build:
 
     # used to prefer this variant less
     down_prioritize_variant: integer (defaults to 0, higher is less preferred)
-
 ```
 
 ## Dynamic linking configuration
@@ -160,21 +201,21 @@ If you want to stop `rattler-build` from relocating the binaries, you can set
 `binary_relocation` to `false`. If you want to only relocate some binaries, you
 can select the relevant ones with a glob pattern.
 
-To read more about `rpath`s and how rattler-build creates relocatable binary packages,
-see the [internals](internals.md) docs.
+To read more about `rpath`s and how rattler-build creates relocatable binary
+packages, see the [internals](internals.md) docs.
 
 If you link against some libraries (possibly even outside of the prefix, in a
 system location), then you can use the `missing_dso_allowlist` to allow linking
 against these and suppress any warnings. This list is pre-populated with a list
 of known system libraries on the different operating systems.
 
 As part of the post-processing, `rattler-build` checks for overlinking and
-overdepending. "Overlinking" is when a binary links against a library that is not
-specified in the run requirements. This is usually a mistake because the library
-would not be present in the environment when the package is installed.
+overdepending. "Overlinking" is when a binary links against a library that is
+not specified in the run requirements. This is usually a mistake because the
+library would not be present in the environment when the package is installed.
 
-Conversely, "overdepending" is when a library is part of the run requirements, but
-is not actually used by any of the binaries/libraries in the package.
+Conversely, "overdepending" is when a library is part of the run requirements,
+but is not actually used by any of the binaries/libraries in the package.
 
 ```yaml title="recipe.yaml"
 build:

docs/highlevel.md

@@ -19,7 +19,7 @@ dependencies are. From the recipe file, `rattler-build` executes several steps:
 
 2. **Fetch source**:
 
-   Retrieve specified source files, such as `.tar.gz` files, `git` repositories, local paths. 
+   Retrieve specified source files, such as `.tar.gz` files, `git` repositories, local paths.
    Additionally, this step will apply patches that can be specified alongside the source file.
 
 3. **Install build environments**:

docs/tutorials/python.md

@@ -1,10 +1,10 @@
 # Writing a Python package
 
-Writing a Python package is fairly straightforward, especially for "Python-only" packages. 
+Writing a Python package is fairly straightforward, especially for "Python-only" packages.
 In the second example we will build a package for `numpy` which contains compiled code.
 
 ## A Python-only package
-The following recipe uses the `noarch: python` setting to build a `noarch` package that can be installed on any platform without modification. 
+The following recipe uses the `noarch: python` setting to build a `noarch` package that can be installed on any platform without modification.
 This is very handy for packages that are pure Python and do not contain any compiled extensions.
 
 Additionally, `noarch: python` packages work with a range of Python versions (contrary to packages with compiled extensions that are tied to a specific Python version).
@@ -72,7 +72,7 @@ rattler-build build --recipe ./ipywidgets
 ## A Python package with compiled extensions
 
 We will build a package for `numpy` – which contains compiled code.
-Since compiled code is `python` version-specific, we will need to specify the `python` version explicitly. 
+Since compiled code is `python` version-specific, we will need to specify the `python` version explicitly.
 The best way to do this is with a "variant_config.yaml" file:
 
 ```yaml title="variant_config.yaml"

src/linux/link.rs

@@ -1,6 +1,5 @@
 //! Relink shared objects to use an relative path prefix
 
-use globset::GlobSet;
 use goblin::elf::{Dyn, Elf};
 use goblin::elf64::header::ELFMAG;
 use goblin::strtab::Strtab;
@@ -14,6 +13,7 @@ use std::io::Read;
 use std::path::{Path, PathBuf};
 
 use crate::post_process::relink::{RelinkError, Relinker};
+use crate::recipe::parser::GlobVec;
 use crate::system_tools::{SystemTools, Tool};
 use crate::utils::to_lexical_absolute;
 
@@ -135,7 +135,7 @@ impl Relinker for SharedObject {
         prefix: &Path,
         encoded_prefix: &Path,
         custom_rpaths: &[String],
-        rpath_allowlist: Option<&GlobSet>,
+        rpath_allowlist: &GlobVec,
         system_tools: &SystemTools,
     ) -> Result<(), RelinkError> {
         if !self.has_dynamic {
@@ -172,7 +172,7 @@ impl Relinker for SharedObject {
                 let resolved = self.resolve_rpath(rpath, prefix, encoded_prefix);
                 if resolved.starts_with(encoded_prefix) {
                     final_rpaths.push(rpath.clone());
-                } else if rpath_allowlist.map(|g| g.is_match(rpath)).unwrap_or(false) {
+                } else if rpath_allowlist.is_match(rpath) {
                     tracing::info!("Rpath in allow list: {}", rpath.display());
                     final_rpaths.push(rpath.clone());
                 }
@@ -197,10 +197,7 @@ impl Relinker for SharedObject {
                     "$ORIGIN/{}",
                     relative_path.to_string_lossy()
                 )));
-            } else if rpath_allowlist
-                .map(|glob| glob.is_match(rpath))
-                .unwrap_or(false)
-            {
+            } else if rpath_allowlist.is_match(rpath) {
                 tracing::info!("rpath ({:?}) for {:?} found in allowlist", rpath, self.path);
                 final_rpaths.push(rpath.clone());
             } else {
@@ -402,7 +399,6 @@ fn builtin_relink(elf_path: &Path, new_rpath: &[PathBuf]) -> Result<(), RelinkEr
 #[cfg(test)]
 mod test {
     use super::*;
-    use globset::{Glob, GlobSetBuilder};
     use std::{fs, path::Path};
     use tempfile::tempdir_in;
 
@@ -426,10 +422,7 @@ mod test {
         let binary_path = tmp_dir.join("zlink");
         fs::copy(prefix.join("zlink"), &binary_path)?;
 
-        let globset = GlobSetBuilder::new()
-            .add(Glob::new("/usr/lib/custom**").unwrap())
-            .build()
-            .unwrap();
+        let globvec = GlobVec::from_vec(vec!["/usr/lib/custom**"], None);
 
         // default rpaths of the test binary are:
         // - /rattler-build_zlink/host_env_placehold/lib
@@ -441,7 +434,7 @@ mod test {
             &prefix,
             encoded_prefix,
             &[],
-            Some(&globset),
+            &globvec,
             &SystemTools::default(),
         )?;
         let object = SharedObject::new(&binary_path)?;
@@ -487,7 +480,7 @@ mod test {
             &prefix,
             encoded_prefix,
             &[String::from("lib/")],
-            None,
+            &GlobVec::default(),
             &SystemTools::default(),
         )?;
         let object = SharedObject::new(&binary_path)?;

src/macos/link.rs

@@ -1,5 +1,4 @@
 //! Relink a dylib to use relative paths for rpaths
-use globset::GlobSet;
 use goblin::mach::Mach;
 use memmap2::MmapMut;
 use scroll::Pread;
@@ -10,6 +9,7 @@ use std::io::Read;
 use std::path::{Path, PathBuf};
 
 use crate::post_process::relink::{RelinkError, Relinker};
+use crate::recipe::parser::GlobVec;
 use crate::system_tools::{SystemTools, Tool};
 use crate::utils::to_lexical_absolute;
 
@@ -149,7 +149,7 @@ impl Relinker for Dylib {
         prefix: &Path,
         encoded_prefix: &Path,
         custom_rpaths: &[String],
-        rpath_allowlist: Option<&GlobSet>,
+        rpath_allowlist: &GlobVec,
         system_tools: &SystemTools,
     ) -> Result<(), RelinkError> {
         let mut changes = DylibChanges::default();
@@ -177,7 +177,7 @@ impl Relinker for Dylib {
                 let resolved = self.resolve_rpath(rpath, prefix, encoded_prefix);
                 if resolved.starts_with(encoded_prefix) {
                     final_rpaths.push(rpath.clone());
-                } else if rpath_allowlist.map(|g| g.is_match(rpath)).unwrap_or(false) {
+                } else if rpath_allowlist.is_match(rpath) {
                     tracing::info!("Rpath in allow list: {}", rpath.display());
                     final_rpaths.push(rpath.clone());
                 }
@@ -203,7 +203,7 @@ impl Relinker for Dylib {
                 final_rpaths.push(new_rpath.clone());
                 // changes.change_rpath.insert(rpath.clone(), new_rpath);
                 // modified = true;
-            } else if rpath_allowlist.map(|g| g.is_match(rpath)).unwrap_or(false) {
+            } else if rpath_allowlist.is_match(rpath) {
                 tracing::info!("Allowlisted rpath: {}", rpath.display());
                 final_rpaths.push(rpath.clone());
             } else {
@@ -533,11 +533,11 @@ mod tests {
     use tempfile::tempdir_in;
 
     use super::{install_name_tool, RelinkError};
-    use crate::post_process::relink::Relinker;
     use crate::{
         macos::link::{Dylib, DylibChanges},
         system_tools::SystemTools,
     };
+    use crate::{post_process::relink::Relinker, recipe::parser::GlobVec};
 
     #[test]
     fn test_relink_builtin() -> Result<(), RelinkError> {
@@ -663,7 +663,7 @@ mod tests {
                 tmp_prefix,
                 &encoded_prefix,
                 &[],
-                None,
+                &GlobVec::default(),
                 &SystemTools::default(),
             )
             .unwrap();