New list-backends built in goal.
#1
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
EDIT: Not sure what it was, but it seems to be fixed now at least. |
kaos
pushed a commit
that referenced
this pull request
Dec 27, 2021
…and Protobuf (pantsbuild#13981) Because we know that a `SingleSourceField` has exactly one file—and we can ban `*` so can compute its file path without using the engine—we can simplify dependency inference when creating the global module mapping. There's no need to actually look up the file system. A downside of this PR is that we have lazier validation that the `source` field matches files on disk, i.e. that the file is there. But there are still plenty of places this gets validated, such as when computing the dependencies of a target because we have to inspect its source code. (Compared to creating a global module mapping for dep inference in general.) There's a marginal speed up, too: Before: ``` ❯ hyperfine -w 1 -r 25 './pants --no-pantsd dependencies src/python/pants/util/strutil.py:util' Benchmark #1: ./pants --no-pantsd dependencies src/python/pants/util/strutil.py:util Time (mean ± σ): 4.576 s ± 0.139 s [User: 3.180 s, System: 0.657 s] Range (min … max): 4.373 s … 5.003 s 25 runs ``` After: ``` ❯ hyperfine -w 1 -r 25 './pants --no-pantsd dependencies src/python/pants/util/strutil.py:util' Benchmark #1: ./pants --no-pantsd dependencies src/python/pants/util/strutil.py:util Time (mean ± σ): 4.487 s ± 0.172 s [User: 3.019 s, System: 0.628 s] Range (min … max): 4.224 s … 5.094 s 25 runs ``` [ci skip-rust] [ci skip-build-wheels]
kaos
pushed a commit
that referenced
this pull request
Dec 31, 2021
…uild#13379) (pantsbuild#13388) Before: ``` ❯ ./pants check testprojects/src/go:: 14:51:39.15 [INFO] Completed: pants.backend.go.goals.check.check_go - go succeeded. Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: ✓ go succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:52:54.54 [ERROR] Completed: pants.backend.go.goals.check.check_go - go failed (exit code 1). Partition #1 - github.com/pantsbuild/pants/testprojects/src/go/pants_test: ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body Partition #2 - github.com/pantsbuild/pants/testprojects/src/go/pants_test/bar: 𐄂 go failed. ``` After: ``` ❯ ./pants check testprojects/src/go:: 14:20:40.79 [INFO] Completed: Check Go compilation - go succeeded. ✓ go compile succeeded. ``` ``` ❯ ./pants check testprojects/src/go:: 14:23:16.18 [ERROR] Completed: Compile with Go - github.com/pantsbuild/pants/testprojects/src/go/pants_test failed (exit code 1). ./testprojects/src/go/pants_test/foo.go:3:1: syntax error: non-declaration statement outside function body 14:23:16.18 [ERROR] Completed: Check Go compilation - go failed (exit code 1). 𐄂 go compile failed. ``` That is, we now: - Stream results for each package, rather than waiting to batch at the very end. - Don't log the `Partition #1` part, which is verbose and confusing. - Log success at DEBUG level, rather than INFO level, for less noise. (Reminder: these workunits will show up in the dynamic UI still) [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
Upgrade Tokio to v1.21.1 plus upgrade related Tokio ecosystem crates. Besides bug fixes, the main motivation to upgrade is to gain access to `tokio::task::JoinSet` for use in pantsbuild#16952.
…17018) Fix various bugs remaining in the cgo support. Add tests demonstrating that C++, Objective-C, and Fortran support works (when the system supports the applicable compiler/libraries). Fixes pantsbuild#16836 Fixes pantsbuild#16830
Initial implementation for a codegen feature in the OpenAPI backend
…hs (pantsbuild#17030) Previously, `PythonBootstrap` had a method that would calculate the expanded interpreter paths when they are first consumed. This change moves the calculation of those expanded interpreter paths to the initial creation of `PythonBootstrap`. Relatedly, a bunch of static methods defined on `PythonBootstrap` are now private functions at module scope. This is prework that will significantly improve the clarity of my next piece of work on `PythonBootstrap`, namely converting all of the various filesystem-dependent functions to `@rule`s. **None of the helper methods have been changed** Addresses pantsbuild#16800.
…d#17036) Remove use of the `[jvm].reproducible_jars` option for resources jars, since they are now deterministic by default. [ci skip-rust] [ci skip-build-wheels]
…valid settings (pantsbuild#17041) Per discussion in pantsbuild#16800, we're OK with using direct filesystem access to support pyenv support (and separately, asdf support) for local runtime environments only. This adds a preprocessing function that supports: * no-opping for local/undefined environments * handling the case where the search paths are unaltered from the default, by stripping out unsupported search paths * raising an exception with fixing instructions if unsupported search paths are requested for remote/docker environments n.b. `PythonBootstrap` is still incorrectly cached for pyenv users, and that will be addressed in one (hopefully) final PR.
There's no need for us to set up local environments, now that local environments are only to _override_ values from the subsystems. (Before, we were going to deprecate the subsystems.) We may in the future want to add a macos_arm64 environment. That is safe for us to do without adding other local environments thanks to pantsbuild#16983. See pantsbuild#17043 for an alternative way we could be leveraging environments locally. [ci skip-rust]
It's very little code for us to support the `filter` goal, which avoids our users needing to rewrite a bunch of scripts and CI config. The important thing we wanted is for `./pants help` to render properly, that this is no longer a goal and is now a subsystem. Turns out, `help` was broken before! We special case this goal to handle this properly. [ci skip-rust] [ci skip-build-wheels]
This gives more flexibility to likely user stories, such as wanting to use Docker in CI but not locally, or vice versa. Before, the sole way to turn off Docker was on Linux that you used a different platform for the Docker image than the local host. That condition is not possible on macOS, so there was no way to disable Docker there. [ci skip-rust] [ci skip-build-wheels]
See pantsbuild#17043 for a demo of this, and an explanation why we take this approach. [ci skip-rust] [ci skip-build-wheels]
…uild#16952) Add the concept of "session end tasks" to Pants sessions so that Rust code can schedule async tasks for execution where the tasks should complete before a particular Pants run is considered complete. (It is called "tail tasks" internally.) First use case: Schedule remote cache writes as session end tasks.
[ci skip-rust]
[ci skip-rust]
This moves the path-specific options for `GolangSubsystem` into an `EnvironmentAware` block, and factors `cgo_tool_search_paths` into a property based on the new `depends_on_env_vars` functionality. Further, the `GoBootstrap` code is refactored to pre-compute the go search paths, saving a request to the uncacheable `asdf` rules if they are not necessary. Still todo -- erroring if `asdf` is used in a non-local environment. Unblocks work on pantsbuild#16800
…r class unions (pantsbuild#17040) The diff should show the difference in syntax here. The new way not only appeases `mypy` through a one-and-only type definition, but continues to allow for the per-subclass distinct-but-equal semantics. Note I had to combine the technique with `unions` directly because `@union` stores an attribute pointing to the type itself. When copied this strategy failed. That's OK though because this technique is only useful for unions 😄 [ci skip-rust] [ci skip-build-wheels]
…ction. (pantsbuild#17069) As reported in pantsbuild#17047, a non-deterministic `KeyError` can be triggered during the construction of `CoarsenedTargets`. I haven't been able to reproduce it in isolation, so this change adds debug output which will hopefully help point to the source of the issue. [ci skip-rust] [ci skip-build-wheels]
Following discussion with @stuhood, it emerged that none of the filesystem-sensitive code in `python_bootstrap` is cache-correct, in that the functions that hit the filesystem were not marked `uncacheable`. This PR refactors `_expand_interpreter_search_paths` and `_get_pyenv_paths` into rules, marking `_get_pyenv_paths` as uncacheable. This reduces the footprint of the sandbox-unsafe code to just `_get_pyenv_paths`. It also reduces the scope of `AsdfToolPathsRequest`, and moves the point where it is requested to inside the path expansion rule, and only if `asdf` is actually requested. Closes pantsbuild#16800.
Closes pantsbuild#16523 [ci skip-rust]
…ild#16909) Support running `go generate` on Go packages via new `go-generate` goal. Closes pantsbuild#13881. [ci skip-build-wheels]
Use the `chevron` templating library which is already in the Pants lockfile to process Go coverage setup code as a template instead of the bespoke string substitution currently performed. [ci skip-build-wheels]
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0. - [Release notes](https://github.com/servo/rust-url/releases) - [Commits](servo/rust-url@v2.2.2...v2.3.0) --- updated-dependencies: - dependency-name: url dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
golangci-lint is a "meta" linter for go source code, that aggregates dozens of other linters into a single command-line tool. [ci skip-build-wheels] [ci skip-rust]
Turned out this option is necessary to run targets with packages depending on Qt, matplotlib etc. Closes pantsbuild#14559
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.20 to 0.4.21. - [Release notes](https://github.com/chronotope/chrono/releases) - [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md) - [Commits](chronotope/chrono@v0.4.20...v0.4.21) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [ci skip-build-wheels]
As explained in pantsbuild#17077, `Subsystem.EnvironmentAware` defines a field called env_vars which prevents defining an option called `--env-vars`. Work around the issue for now by renaming the option. [ci skip-build-wheels]
…uild#17538) This fixes a bug/regression in 2.13 -> 2.14 where `CreateArchive` requires an output directory. In 2.13 this'd pass `""` as a create-directory request, and that'd... do something. In 2.14 the Rust code uses `DigestTries` to improve performance, but an empty trie component generates an error: ``` 15:00:12.12 [ERROR] 1 Exception encountered: Engine traceback: in select in pants.core.goals.package.package_asset in pants_backend_oci.goals.package.package_oci_image (examples:oci) in pants_backend_oci.util_rules.build_image_bundle.build_oci_bundle_package in pants.core.util_rules.archive.create_archive Traceback (no traceback): <pants native internals> Exception: Path `` was unexpectedly empty ``` I believe this change in behavior comes from this optimization by @Eric-Arellano: https://github.com/pantsbuild/pants/pull/16648/files#diff-78b61c9a9b42b3895a2c71b90e31aceee5a5f38780edddbd4daec17ee4386d98R475. I believe backporting this to the 2.14 series makes sense to fix the regression. If the output directory should be required, a proper breaking change might be in order for 2.15 with a renaming of the parameter (<code>output_file**name** -> output_file**path**</code>) and proper diagnostic.
When loading `Directory` protos from the `Store`, we were not loading their `SymlinkNode`s, which caused a mismatching digest when loading `Snapshot`s which contain symlinks. Fixes pantsbuild#17542.
This has the docker target output a `$name.docker-info.json` file when packaged, to end up in `dist/` or for use as a dependency. The file contains the full specification of the packaged image, including each of the `registry/repository:tag` names it was tagged with. This stops users from having to parse the pants output (or query docker itself) if they need this information, such as to place into a cloud deploy template, or to run the image locally.
Plumbs --pip-version through to pex commands that support it. Newer pips have perf improvements that may benefit users, in particular for very long-running resolves. Does not change the default. We can do that via deprecation in a followup.
Previously we only did so for non-interactive processes. This is helpful when debugging issues with `./pants run`.
direnv can only cache env variables, not ulimit settings see direnv/direnv#777
Also fix a function arg default that was mutable to be immutable.
The dependency rules feature is implemented in layers, with the core feature always being enabled for parsing the BUILD file level construct for capturing the __dependents_rules__ and __dependencies_rules__ instructions. Without a backend implementing the rule checks, these instructions turns into no-ops.
See pantsbuild#17039. Given a testbed of <details> <summary>input</summary> ```python # Copyright 2022 Pants project contributors (see CONTRIBUTORS.md). # Licensed under the Apache License, Version 2.0 (see LICENSE). import json # unownable, root level import os.path # unownable, not root level import watchdog # dependency not included import yaml # dependency included import yamlpath # owned by other resolve try: import weakimport # weakimport missing except ImportError: ... open("src/python/configs/prod.json") # asset open("testprojects/pants-plugins/src/python/test_pants_plugin/__init__.py") ``` </details> we get <details> <summary>output</summary> ``` { "src/python/pants/backend/python/dependency_inference/t.py": { "imports": [ { "name": "weakimport", "reference": { "lineno": 12, "weak": true }, "resolved": { "status": "ImportOwnerStatus.weak_ignore", "address": [] }, "possible_resolve": null }, { "name": "json", "reference": { "lineno": 4, "weak": false }, "resolved": { "status": "ImportOwnerStatus.unownable", "address": [] }, "possible_resolve": null }, { "name": "os.path", "reference": { "lineno": 5, "weak": false }, "resolved": { "status": "ImportOwnerStatus.unownable", "address": [] }, "possible_resolve": null }, { "name": "watchdog", "reference": { "lineno": 7, "weak": false }, "resolved": { "status": "ImportOwnerStatus.unowned", "address": [] }, "possible_resolve": null }, { "name": "yaml", "reference": { "lineno": 8, "weak": false }, "resolved": { "status": "ImportOwnerStatus.unambiguous", "address": [ "3rdparty/python#PyYAML", "3rdparty/python#types-PyYAML" ] }, "possible_resolve": null }, { "name": "yamlpath", "reference": { "lineno": 9, "weak": false }, "resolved": { "status": "ImportOwnerStatus.unowned", "address": [] }, "possible_resolve": [ [ "src/python/pants/backend/helm/subsystems:yamlpath", "helm-post-renderer" ] ] } ], "assets": [ { "name": "src/python/configs/prod.json", "reference": "src/python/configs/prod.json", "resolved": { "status": "ImportOwnerStatus.unowned", "address": [] }, "possible_resolve": null }, { "name": "testprojects/pants-plugins/src/python/test_pants_plugin/__init__.py", "reference": "testprojects/pants-plugins/src/python/test_pants_plugin/__init__.py", "resolved": { "status": "ImportOwnerStatus.unambiguous", "address": [ "testprojects/pants-plugins/src/python/test_pants_plugin/__init__.py:../../../../pants_plugins_directory" ] }, "possible_resolve": null } ] } } ``` </details> Telling you, for each file, for each import, what dependencies pants thought it could have, and what it decided to do with them. This uses almost all the same code as the main dependency inference code, with the exception of the top-level orchestration of it. I think that's pretty close, there's about 100 lines of semi-duplicate code. There's also a more advanced mode that dumps information about each stage of the process. I think this might be useful for people digging through the dependency inference process but not really for end-users. we get it for free, though. Fixes pantsbuild#17039. --- this is fairly critical for performance, so here are benchmarks (with comparison-of-means t-test) | | main | this | difference | P-score | | --- | --- | --- | --- | --- | | `hyperfine --runs=10 './pants --no-pantsd dependencies --transitive ::'` | 21.839 s ± 0.326 s | 22.142 s ± 0.283 s | 1.38% | 0.0395 | | `hyperfine --warmup=1 --runs=10 './pants dependencies --transitive ::'` | 1.798 s ± 0.074 s | 1.811 s ± 0.076 s | 0.72% | 0.7029 | | `hyperfine --runs=10 './pants --no-pantsd dependencies ::'` | 21.547 s ± 0.640 s | 21.863 s ± 1.072 s | 1.47% | 0.4339 | | `hyperfine --warmup=1 --runs=10 './pants dependencies ::'` | 1.828 s ± 0.091 s | 1.844 s ± 0.105 s | 0.88% | 0.7200 | So it looks like this MR might impact performance, by about 1%, although those p-values are mighty unconvincing. LMK if we want to increase runs and get more statistics, I've run the stats a few times throughout and this looks about right, so I think we can proceed with the review under the assumption that there is currently a 1% performance overhead. I'm open to suggestions on improving performance.
…ls (pantsbuild#17567) Added support for a version option to Prettier and Pyright via an NpxToolBase.
…unning in a sandbox (pantsbuild#17566) Fixes pantsbuild#17540 The key is that the DAP client is sending `pathMappings`, which normally sets `remoteRoot` to `"."` which for `run` isn't true. There's no easy way to have the server set it's own mapping so we monkeypatch.
Changes between v1.49.0 and v1.50.1: golangci/golangci-lint@v1.49.0...v1.50.1
Resurrection of pantsbuild#16738 --- This PR explains how to override/complement any list values that may have been provided in lower-precedence sources. Current documentation page is here. For example, having ``` [coverage-py] report = ["xml"] ``` and running ``` ./pants test --use-coverage --coverage-py-report=html :: ... Wrote xml coverage report to `dist/coverage_results` Wrote html coverage report to `dist/coverage_results` ``` I found this behaviour not obvious (thinking that only the HTML report will be produced), hence the PR. [ci skip-rust] [ci skip-build-wheels]
[ci skip-rust] [ci skip-build-wheels]
kaos
force-pushed
the
list_backends_goal
branch
from
df4dbd9 to
79f6b1c
Compare
kaos
pushed a commit
that referenced
this pull request
May 16, 2023
The Pants native client which was introduced in pantsbuild#11922 has so far only been usable in the Pants repo when the `USE_NATIVE_PANTS` environment variable was set. To make the native client available to end users, this change begins distributing the native-client binary in Pants wheels. A corresponding change in the `scie-pants` repo (pantsbuild/scie-pants#172) uses the native client to run `pants`. To reduce the surface area of `scie-pants` (rather than having it be responsible for handling the special `75` exit code similar to the `pants` script integration), this PR also moves to having the native-client execute its own fallback (via `execv`) to the Pants entrypoint. In future, the `pantsbuild/pants` `pants` script could also use that facility (e.g. by specifying a separate `pants_server` bash script as the entrypoint to use as the `_PANTS_SERVER_EXE`). ---- As originally demonstrated on pantsbuild#11831, the native client is still much faster than the legacy client. Using pantsbuild/scie-pants#172, the timings look like: ``` Benchmark #1: PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 1.161 s ± 0.067 s [User: 830.6 ms, System: 79.2 ms] Range (min … max): 1.054 s … 1.309 s 10 runs Benchmark #2: PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 271.0 ms ± 30.6 ms [User: 8.9 ms, System: 6.9 ms] Range (min … max): 241.5 ms … 360.6 ms 12 runs Summary 'PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ran 4.29 ± 0.54 times faster than 'PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ``` Fixes pantsbuild#11831.
kaos
pushed a commit
that referenced
this pull request
Jun 7, 2023
pantsbuild#19243) …9047)" This reverts commit 0aedb6b. contains breaking change: ``` Error: 3.20 [ERROR] Failed to load the pants.backend.experimental.javascript.register backend: ImportError("cannot import name 'UserChosenNodeJSResolveAliases' from 'pants.backend.javascript.subsystems.nodejs' (/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/subsystems/nodejs.py)") Traceback (most recent call last): File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/init/extension_loader.py", line 141, in load_backend module = importlib.import_module(backend_module) File "/home/gha/.pyenv/versions/3.9.13/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/experimental/javascript/register.py", line 9, in <module> from pants.backend.javascript.goals import lockfile, tailor, test File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/goals/lockfile.py", line 9, in <module> from pants.backend.javascript import nodejs_project_environment File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/nodejs_project_environment.py", line 9, in <module> from pants.backend.javascript import package_json, resolve File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/resolve.py", line 9, in <module> from pants.backend.javascript import nodejs_project File "/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/nodejs_project.py", line 19, in <module> from pants.backend.javascript.subsystems.nodejs import NodeJS, UserChosenNodeJSResolveAliases ImportError: cannot import name 'UserChosenNodeJSResolveAliases' from 'pants.backend.javascript.subsystems.nodejs' (/home/gha/actions-runner1/_work/pants/pants/src/python/pants/backend/javascript/subsystems/nodejs.py) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Showcasing the new
BuiltinGoalwith alist-backendsgoal. It also shows how it integrates using subsystem options, just like any other goal. Neat ;)