Zinc 1.0.0 upgrade: Python portion #4729
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(nb: not expecting this to pass CI yet, because I'm waiting on sbt/zinc#325 to land to get an actual release with this logging API) |
kwlzn
approved these changes
Jul 18, 2017
|
|
||
|
|
||
| class SyntheticTargetNotFound(Exception): | ||
| pass |
There was a problem hiding this comment.
prefer a simple docstring over pass
| options_scope = 'jvm-dependency-context' | ||
|
|
||
| _target_closure_kwargs = dict(include_scopes=Scopes.JVM_COMPILE_SCOPES, respect_intransitive=True) | ||
| _compiler_plugin_types = (AnnotationProcessor, JavacPlugin, ScalacPlugin) |
There was a problem hiding this comment.
prefer uppercase for constants where possible
| for export in getattr(target, 'exports', []): | ||
| if not isinstance(export, Target): | ||
| addr = Address.parse(export, relative_to=target.address.spec_path) | ||
| export = target._build_graph.get_target(addr) |
There was a problem hiding this comment.
any way to avoid access of this private _build_graph attribute on target?
There was a problem hiding this comment.
This is existing code migrated from elsewhere. Will avoid making substantive changes if possible.
| return hash(type(self)) | ||
|
|
||
| def __eq__(self, other): | ||
| return type(self) == type(other) |
There was a problem hiding this comment.
a note explaining why hash and eq here are keyed on type only might be helpful.
| self.dist.real_home: '/dev/null/java_home/', | ||
| get_buildroot(): '/dev/null/buildroot/', | ||
| self.get_options().pants_workdir: '/dev/null/workdir/', | ||
| } |
There was a problem hiding this comment.
might be better to use a clearer, not-likely-to-exist path like e.g. /remapped_by_pants/... here vs /dev/null?
There was a problem hiding this comment.
Maybe, but I also want attempts to actually read/write to these paths to fail in noticeable way. Will include pants in the name.
| def _compute_unnecessary_deps(self, target, actual_deps): | ||
| """Computes unused deps for the given Target. | ||
|
|
||
| :returns: A set of directly declared but unused targets, and a set of suggested replacements. |
stuhood
force-pushed
the
stuhood/zinc-upgrade-2017-06-python
branch
from
3d0391a
to
0d2bbcf
Compare
|
This should now be mostly-passing in CI, and is ready for review. |
mateor
approved these changes
Jul 21, 2017
There was a problem hiding this comment.
Thanks Stu!
I am glad to see the divorce from zinc analysis, that certainly felt like it was creaking. And surfacing some of the classpath handling to subsystems is very welcome.
You all have done a lot of work to get here, and I don't know all the details. But supportive of the direction and willing to ShipIt as such
| @@ -10,7 +10,7 @@ Licensed under the Apache License, Version 2.0 (see LICENSE). | |||
| <property name="m2.repo.relpath" value="[organisation]/[module]/[revision]"/> | |||
| <property name="m2.repo.pom" value="${m2.repo.relpath}/[module]-[revision].pom"/> | |||
| <property name="m2.repo.artifact" | |||
| value="${m2.repo.relpath}/[artifact](-[classifier])-[revision].[ext]"/> | |||
| value="${m2.repo.relpath}/[artifact]-[revision](-[classifier]).[ext]"/> | |||
There was a problem hiding this comment.
Ha! I have this bug internally, it turns out. I was just bitten by it the other day, but did not catch this was the cause. When properly configured, our CIs aren't using the {m2.x} definitions, so I just fixed that and didn't keep digging. I will have to land the equivalent change :)
I recently rewrote all our ivyprofile and settings files and have found that one line changes often reflect hours of frustration from the author 😱
| self.ZINC_EXTRACTOR_TOOL_NAME, | ||
| scope=self.options_scope) | ||
|
|
||
| def compile_classpath(self, products, classpath_product_key, target, extra_cp_entries=None): |
There was a problem hiding this comment.
This is a bit of a break with the existing Classpath-* API, where methods that operate over targets usually take the target as the first argument.
| extra_cp_entries=extra_cp_entries) | ||
|
|
||
| @classmethod | ||
| def compile_classpath_for(cls, |
There was a problem hiding this comment.
I had to read this a couple times to figure out how the arguments passed by the caller map to these params. It feels unusual to have a classmethod take an instance of the class as an argument. It is not using it as a bootstrap in a factory so I am a bit in the dark as to what is motivating that. Just seems like it would be a better fit as an instance method or something.
I am just scrolling through in a web browser so forgive dumb questions. I also see some "merge with X when 1.6.0", so I assume there is some sort of bug/deprecation that is forcing your hand.
stuhood
pushed a commit
that referenced
this pull request
Jul 28, 2017
### Problem The current version of zinc used in pants suffers from sbt/zinc#355, and thus hasn't been pulled in and released via #4729 ### Solution Bump to zinc 1.0.0-RC3, which pulls in the fix for sbt/zinc#355.
stuhood
pushed a commit
that referenced
this pull request
Aug 11, 2017
### Problem While working through #4477, it became obvious that: 1. the performance achieved when the analysis cache is missed is pretty abysmal 2. a `log4j` JMX error was being logged during startup 3. the `forkJava` and `javaHome` settings are redundant: the only reason to pass `javaHome` would be to trigger forking Java 4. a large amount of unnecessary classloading was happening due to #4744 ### Solution 1. Set the default `zinc.analysis.cache.limit` value to `Int.MaxValue`, which will allow users who want to cap the size to set it lower, but otherwise not lead to performance cliffs when a target has more dependencies than the current limit. 2. Implicitly disable log4j JMX usage by setting the `log4j2.disable.jmx` property if it is not already set. 3. Remove the `forkJava` setting, to prepare to use the `javaHome` setting explicitly in #4729 4. Enable usage of `ClassLoaderCache` ### Result Fixes #4744, and removes a few more blockers for #4729.
Merged
stuhood
pushed a commit
that referenced
this pull request
Nov 3, 2017
### Problem #4729 is currently blocked on the fix for sbt/zinc#389, which is now released in zinc `1.0.3`. ### Solution Bump to zinc 1.0.3 to pick up that fix.
stuhood
pushed a commit
that referenced
this pull request
Nov 4, 2017
### Problem The color flag is currently being respected for console logging, but not for compiler/reporter logging. ### Solution Pass `--color` through to the `ReporterConfig` using the builder API. ### Result #4729 is unblocked, and: ``` ./pants compile --no-colors testprojects/src/java/org/pantsbuild/testproject/dummies:compilation_failure_target ``` results in no color, as expected.
stuhood
force-pushed
the
stuhood/zinc-upgrade-2017-06-python
branch
from
eabe279
to
6838be1
Compare
stuhood
added a commit
to twitter/pants
that referenced
this pull request
May 8, 2018
stuhood
changed the title
This was referenced May 8, 2018
Merged
stuhood
pushed a commit
that referenced
this pull request
May 9, 2018
### Problem We're a few months behind on zinc releases... although they have not yet been incorporated anyway (pending #4729). ### Solution Bump to the latest, and fix a deprecation.
stuhood
force-pushed
the
stuhood/zinc-upgrade-2017-06-python
branch
from
4e63fe2
to
6f5acda
Compare
stuhood
added a commit
to twitter/pants
that referenced
this pull request
May 9, 2018
|
This is reviewable again. I'll do some benchmarking tomorrow, and then maybe be trying to land it on Thursday (will wait until after the |
benjyw
approved these changes
May 9, 2018
| @@ -180,8 +180,6 @@ exclude_patterns: [ | |||
| [compile.zinc] | |||
| jvm_options: [ | |||
| '-Xmx4g', '-XX:+UseConcMarkSweepGC', '-XX:ParallelGCThreads=4', | |||
| # bigger cache size for our big projects (default is just 5) | |||
| '-Dzinc.analysis.cache.limit=1000', | |||
There was a problem hiding this comment.
How come we don't need this any more?
There was a problem hiding this comment.
I changed the default in the JVM code: because it's a weak-reference cache, making it huge is safe.
| @@ -0,0 +1,118 @@ | |||
| # coding=utf-8 | |||
| # Copyright 2014 Pants project contributors (see CONTRIBUTORS.md). | |||
| @@ -39,7 +39,10 @@ class ScalaPlatform(JvmToolMixin, ZincLanguageMixin, InjectablesMixin, Subsystem | |||
|
|
|||
| :API: public | |||
| """ | |||
| options_scope = 'scala-platform' | |||
| options_scope = 'scala' | |||
| main='no.such.main.Main', | ||
| custom_rules=shader_rules) | ||
|
|
||
| cls.register_jvm_tool(register, | ||
| Zinc.ZINC_EXTRACTOR_TOOL_NAME, |
There was a problem hiding this comment.
Seems arbitrary that this is the only tool we register here whose name is in a symbolic constant.
| @@ -1,60 +0,0 @@ | |||
| # coding=utf-8 | |||
There was a problem hiding this comment.
OMG this makes me so, so happy.
|
Performance looks about even with the previous release, although memory usage appears to be higher. It's not clear yet whether the memory usage will be a blocker from our perspective, but I will spend some time investigating it tomorrow morning. |
benjyw
approved these changes
May 10, 2018
|
Memory usage does not look like a blocker, although we'll follow up upstream if need be. |
…he compiler repeatedly.
…distribution test.
stuhood
force-pushed
the
stuhood/zinc-upgrade-2017-06-python
branch
from
e8f85ad
to
877184e
Compare
stuhood
force-pushed
the
stuhood/zinc-upgrade-2017-06-python
branch
from
e54b76b
to
bdad62a
Compare
stuhood
pushed a commit
that referenced
this pull request
May 11, 2018
### Problem (this change depends on #4728) As described on #4728, we currently need internal knowledge of zinc analysis in order to make it portable, and to extract products that are used by other tasks. Since the zinc analysis format is changing significantly (moving to protobuf, gaining/losing fields, etc), we need to stop parsing it. ### Solution Incorporates the version of the zinc wrapper published in #4728, and uses it in a `analysis.zinc` task to generate the `product_deps_by_src` and `classes_by_source` products from a `zinc_analysis` product published by zinc. Expands the extracted `Zinc` subsystem from #4720 to encapsulate compile dependency calculation, and expose the `FingerprintStrategy` and `DependencyContext` that `compile.zinc` was using. ### Result Zinc analysis is now a black box from the perspective of the pants python code. Also, the `compile.jvm-dep-check` task moved to the `lint` goal, and gained the unused dependency checks that used to be inlined in `compile.zinc` (and which would have prevented separation of the analysis extraction task). Fixes #4477, fixes #4513, fixes #4185, fixes #5444, and hopefully fixes a few other odd incremental compilation bugs.
stuhood
pushed a commit
that referenced
this pull request
May 11, 2018
### Problem (this change depends on #4728) As described on #4728, we currently need internal knowledge of zinc analysis in order to make it portable, and to extract products that are used by other tasks. Since the zinc analysis format is changing significantly (moving to protobuf, gaining/losing fields, etc), we need to stop parsing it. ### Solution Incorporates the version of the zinc wrapper published in #4728, and uses it in a `analysis.zinc` task to generate the `product_deps_by_src` and `classes_by_source` products from a `zinc_analysis` product published by zinc. Expands the extracted `Zinc` subsystem from #4720 to encapsulate compile dependency calculation, and expose the `FingerprintStrategy` and `DependencyContext` that `compile.zinc` was using. ### Result Zinc analysis is now a black box from the perspective of the pants python code. Also, the `compile.jvm-dep-check` task moved to the `lint` goal, and gained the unused dependency checks that used to be inlined in `compile.zinc` (and which would have prevented separation of the analysis extraction task). Fixes #4477, fixes #4513, fixes #4185, fixes #5444, and hopefully fixes a few other odd incremental compilation bugs.
stuhood
pushed a commit
that referenced
this pull request
Jun 12, 2018
…s. (#5953) ### Problem Zinc used to be shaded before the `1.x.y` upgrade (#4729), but shading was removed due to an overabundance of optimism. When testing the zinc upgrade internally, we experienced a classpath collision between an annotation processor and zinc (in guava, although zinc has many other dependencies that could cause issues). ### Solution Shade zinc, and ensure that our annotation processor uses a very old guava in order to attempt to force collisions in future.
CMLivingston
pushed a commit
to CMLivingston/pants
that referenced
this pull request
Jun 22, 2018
Closes pantsbuild#5831 Prep for release 1.8.0dev3 (pantsbuild#5937) Ban bad `readonly` shell pattern (pantsbuild#5924) This subverts `set -e` and means that failed commands don't fail the script, which is responsible for late failures on CI such as https://travis-ci.org/pantsbuild/pants/jobs/389174049 which failed to download protoc, but only failed when something tried to use it. Fixup macosx platform version. (pantsbuild#5938) This needs to match the Travis CI osx platform we pre-build wheels on. Work towards pantsbuild#4896. Allow pants to select targets by file(s) (pantsbuild#5930) Fixes pantsbuild#5912 Problem There should be an option to accept literal files, and then select the targets that own those files. Similar to how the --changed-parent triggers a diff and the targets are selected based on the result. The proposed syntax is something like: $ ./pants \ --owner-of=this/is/a/file/name.java \ # flag triggers owner lookup --owner-of=this/is/a/file/name/too.py \ # flag triggers owner lookup compile # goal Solution I've created a global option --owner-of= that takes a list of files as a parameter, and created a OwnerCalculator class to handle the logic similar to how ChangeCalculator works with the --changed-* subsystem. Result Now users will be able to run goals on files without needing to know which target owns those files. Also, the list-owners goal can be deprecated in favor of --owner-of=some/file.py list It is important to note that multiple target selection methods are not allowed, so it fails when more than one of --changed-*, --owner-of, or target specs are supplied. e.g. this fails: $ ./pants \ --owner-of=this/is/a/file/name.java \ # flag triggers owner lookup --owner-of=this/is/a/file/name/too.py \ # flag triggers owner lookup compile <another target> Integration test for daemon environment scrubbing (pantsbuild#5893) This shows that pantsbuild#5898 works, which itself fixed pantsbuild#5854 Add the --owner-of= usage on Target Address documentation (pantsbuild#5931) Problem The documentation of the feature proposed in PR pantsbuild#5930 Solution I decided to put it inside Target Addresses because that is where a user would look if they needed a feature like this, I think. Result More docs, and that's always good. Add script to get a list of failing pants from travis (pantsbuild#5946) [jvm-compile] template-methodify JvmCompile further; add compiler choices (pantsbuild#5923) Introduce `JvmPlatform.add_compiler_choice(name)`, which allows plugins to register compilers that can be configured. This patch pulls out some template methods in JvmCompile to make it easier to extend. It also pushes some of the implementations of those methods down into ZincCompile, where appropriate. These changes should be covered by existing tests, but it could make sense to add tests around the interfaces of the new template methods. I don't anticipate there being a large number of implementations at this time though, so I didn't think it'd be worth it. Add the following template methods * `create_empty_extra_products` Allows subclasses to create extra products that other subclasses might not need, that ought to be constructed even if no compile targets are necessary. * `register_extra_products_from_contexts` rename of `_register_vts`. This allows subclasses to register their extra products for particular targets. * `select_runtime_context` Not 100% happy with this, but I'm working on something that needs to have different types of compile contexts. It allows subclasses to specify a context that provides paths for the runtime classpath if the default context isn't quite right for the usages in the base class. * `create_compile_jobs` Pulled this out into a separate method so that subclasses can create multiple graph jobs per target. * Pushed down behavior from JvmCompile that should live in zinc via the template methods extracted above. There's probably more that could be done here, but this was the first cut of it. * Moved the execute definition from BaseZincCompile to ZincCompile so that it's possible to subclass BaseZincCompile with a different compiler name. release notes for 1.7.0.rc1 (pantsbuild#5942) Use target not make_target in some tests (pantsbuild#5939) This pushes parsing of the targets through the engine, rather than bypassing it. This is important because I'm about to make these targets require an EagerFilesetWithSpec as their source/sources arg, rather than being happy with a list of strings. Add new remote execution options (pantsbuild#5932) As described in pantsbuild#5904, a few configuration values that are important to testing of remote execution are currently hardcoded. Extract existing options to a `ExecutionOptions` collection (which should become a `Subsystem` whenever we add support for consuming `Subsystems` during bootstrap), and add the new options. Fixes pantsbuild#5904. Separate the resolution cache and repository cache in Ivy (pantsbuild#5844) move glob matching into its own file (pantsbuild#5945) See pantsbuild#5871, where we describe an encapsulation leak created by implementing all of the glob expansion logic in the body of `VFS`. - Create `glob_matching.rs`, exporting the `GlobMatching` trait, which exports the two methods `canonicalize` and `expand`, which call into methods in a private trait `GlobMatchingImplementation`. **Note:** `canonicalize` calls `expand`, and vice versa, which is why both methods were moved to `glob_matching.rs`. Orthogonal glob matching logic is made into a trait that is implemented for all types implementing `VFS`, removing the encapsulation leak. The `VFS` trait is now just four method signature declarations, making the trait much easier to read and understand. Enable fromfile support for --owner-of and increase test coverage (pantsbuild#5948) The new `--owner-of` option was broken in the context of `pantsd`, but didn't have test coverage due to the `--changed` and `--owner-of` tests not running under `pantsd`. Additionally, `fromfile` support is useful for this option, but was not enabled. Mark some integration tests as needing to run under the daemon, and enable `fromfile` support for `--owner-of`. [pantsd] Robustify client connection logic. (pantsbuild#5952) Fixes pantsbuild#5812 under full-on-assault stress testing via: $ watch -n.1 'pkill -f "pantsd \[" pantsd-runner' this will mostly behave like: WARN] pantsd was unresponsive on port 55620, retrying (1/3) WARN] pantsd was unresponsive on port 55620, retrying (2/3) WARN] pantsd was unresponsive on port 55626, retrying (3/3) WARN] caught client exception: Fallback(NailgunExecutionError(u'Problem executing command on nailgun server (address: 127.0.0.1:55630): TruncatedHeaderError(u"Failed to read nailgun chunk header (TruncatedRead(u\'Expected 5 bytes before socket shutdown, instead received 0\',)).",)',),), falling back to non-daemon mode 23:30:24 00:00 [main] (To run a reporting server: ./pants server) 23:30:38 00:14 [setup] 23:30:39 00:15 [parse] ... mid-flight terminations (simulated via single-shot pkill calls) also result in a more descriptive error with traceback proxying: 23:40:51 00:04 [zinc] 23:40:51 00:04 [javac] 23:40:51 00:04 [cpp] 23:40:51 00:04 [errorprone] 23:40:51 00:04 [findbugs]CRITICAL] CRITICAL] lost active connection to pantsd! Exception caught: (<class 'pants.bin.remote_pants_runner.Terminated'>) File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 73, in <module> main() File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 69, in main PantsLoader.run() File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 65, in run cls.load_and_execute(entrypoint) File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_loader.py", line 58, in load_and_execute entrypoint_main() File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_exe.py", line 39, in main PantsRunner(exiter, start_time=start_time).run() File "/Users/kwilson/dev/pants/src/python/pants/bin/pants_runner.py", line 39, in run return RemotePantsRunner(self._exiter, self._args, self._env, bootstrap_options).run() File "/Users/kwilson/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 162, in run self._run_pants_with_retry(port) File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 221, in execute return self._session.execute(cwd, main_class, *args, **environment) File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 94, in execute return self._process_session() File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_client.py", line 69, in _process_session for chunk_type, payload in self.iter_chunks(self._sock, return_bytes=True): File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_protocol.py", line 206, in iter_chunks chunk_type, payload = cls.read_chunk(sock, return_bytes) File "/Users/kwilson/dev/pants/src/python/pants/java/nailgun_protocol.py", line 182, in read_chunk raise cls.TruncatedHeaderError('Failed to read nailgun chunk header ({!r}).'.format(e)) Exception message: abruptly lost active connection to pantsd runner: NailgunError(u'Problem talking to nailgun server (address: 127.0.0.1:55707, remote_pid: -28972): TruncatedHeaderError(u"Failed to read nailgun chunk header (TruncatedRead(u\'Expected 5 bytes before socket shutdown, instead received 0\',)).",)',) Re-shade zinc to avoid classpath collisions with annotation processors. (pantsbuild#5953) Zinc used to be shaded before the `1.x.y` upgrade (pantsbuild#4729), but shading was removed due to an overabundance of optimism. When testing the zinc upgrade internally, we experienced a classpath collision between an annotation processor and zinc (in guava, although zinc has many other dependencies that could cause issues). Shade zinc, and ensure that our annotation processor uses a very old guava in order to attempt to force collisions in future. Improve PythonInterpreterCache logging (pantsbuild#5954) When users have issues building their Python interpreter cache, they are often very confused because does not currently log much about the process to help users debug. Here we add log lines describing what/where Pants looks to build the interpreter cache, and the results of what it found. This should help users better understand/debug the process. use liblzma.dylib for xz on osx and add platform-specific testing to the rust osx shard (pantsbuild#5936) See pantsbuild#5928. The `xz` archiver wasn't tested on osx at all, and failed to find `liblzma.so` on osx (it should have been `liblzma.dylib`). There were additional errors with library search paths reported in that PR which I was not immediately able to repro. This PR hopefully fixes all of those errors, and ensures they won't happen again with the addition of platform-specific testing (see previous issue at pantsbuild#5920). - Switch to a statically linked `xz`. - Fix the incorrect key `'darwin'` in the platform dictionary in the `LLVM` subsystem (to `'mac'`). - Add the tag `platform_specific_behavior` to the new python target `tests/python/pants_test/backend/python/tasks:python_native_code_testing`, which covers the production of `python_dist()`s with native code. - Add the `-z` argument to `build-support/bin/ci.sh` to run all tests with the `platform_specific_behavior` tag. Also clean up old unused options in the getopts call, and convert echo statements to a simpler heredoc. - Change the name of the "Rust Tests OSX" shard to "Rust + Platform-specific Tests OSX", and add the `-z` switch to the `ci.sh` invocation. **Note:** the tests in `tests/python/pants_test/backend/native/subsystems` are going to be removed in pantsbuild#5815, otherwise they would be tagged similarly. `./pants test tests/python/pants_test/backend/python/tasks:python_native_code_testing` now passes on osx, and this fact is now being tested in an osx shard in travis. Support output directory saving for local process execution. (pantsbuild#5944) Closes pantsbuild#5860 reimplement a previous PR -- ignore this This commit is a reimplementation of registering @rules for backends, because this PR began before that one was split off. add some simple examples to demonstrate how to use backend rules ...actually make the changes to consume backend rules in register.py revert accidental change to a test target remove extraneous log statement fix lint errors add native backend to release.sh isolate native toolchain path and hope for the best add config subdir to native backend really just some more attempts start trying to dispatch based on platform extend EngineInitializer to add more rules from a backend refactor Platform to use new methods in osutil.py refactor the native backend to be a real backend and expose rules register a rule in the python backend to get a setup.py environment make python_dist() tests pass make lint pass create tasks and targets for c/c++ sources - refactors the "native toolchain" and introduces the "binaries" subdirectory of subsystems start by adding a new ctypes testproject add c/c++ sources add example BUILD file add some native targets add tasks dir remove gcc try to start adding tasks clean some leftover notes in BuildLocalPythonDistributions update NativeLibrary with headers move DependencyContext to target.py add native compile tasks houston we have compilation run: ./pants -ldebug --print-exception-stacktrace compile testprojects/src/python/python_distribution/ctypes: for an idea use the forbidden product request change target names to avoid conflict with cpp contrib and flesh out cpp_compile now we are compiling code we can link things now now we know how to infer headers vs sources fix the test case and fix include dir collection (un)?suprisingly, everything works, but bdist_wheel doesn't read MANIFEST.in houston we have c++ bring back gcc so we can compile halfway done with osx support now things work on osx???????? ok, now it works on linux again too first round of review - NB: refactors the organization of the `results_dir` for python_dist()! - move ctypes integration testing into python backend tests revert some unnecessary changes refactor native_artifact to be a datatype fix some copyright years add ctypes integration test add assert_single_element method in collections.py decouple the native tools for setup.py from the execution environment streamline the selection of the native tools for setup.py invocation make gcc depend on binutils on linux for the 'as' assembler fix logging visibility by moving it back into the task make the check for the external llvm url more clear refactor local dist building a bit - use SetupPyRunner.DIST_DIR as the source of truth - add a separate subdir of the python_dist target's results_dir for the python_dist sources - move shraed libs into the new subdir fix imports second round of review - fixed bugs - expanded error messages and docstrings make a couple docstring changes fix dist platform selection ('current' is not a real platform) lint fixes fix broken regex which modifies the `.dylib` extension for python_dist() fix the ctypes integration test respond to some review comments clear the error message if we can't find xcode cli tools refactor the compilation and linking pipeline to use subsystems - also add `PythonNativeCode` subsystem to bridge the native and python backends refactor the compilation and linking pipeline to use subsystems add some notes fix rebase issues add link to pantsbuild#5788 -- maybe use variants for args for static libs move `native_source_extensions` to a new `PythonNativeCode` subsystem update native toolchain docs and remove bad old tests move tgt_closure_platforms into the new `PythonNativeCode` subsystem remove unnecessary logging remove compile_settings_class in favor of another abstractmethod refactor `NativeCompile` and add documentation improve debug logging in NativeCompile document NativeCompileSettings refactor and add docstrings convert provides= to ctypes_dylib= and add many more docstrings remove or improve TODOs improve or remove FIXMEs improve some docstrings, demote a FIXME, and add a TODO link FIXMEs to a ticket add notes to the ctypes testproject update mock object for strict deps -- test passes fix failing integration test on osx add hack to let travis pass fix the system_id key in llvm and add a shameful hack to pass travis swap the order of alias_types remove unused EmptyDepContext class remove -settings suffix from compile subsystem options scopes add AbstractClass to NativeLibrary bump implementation_version for python_dist() build - we have changed the layout of the results_dir in this PR add ticket link and fix bug revert indentation changes to execute() method refactor `assert_single_element()` revert addition of `narrow_relative_paths()` add link to pantsbuild#5950 move common process invocation logic into NativeCompile revert an unnecessary change turn an assert into a full exception revert unnecessary change use get_local_platform() wherever possible delete duplicate llvm subsystem fix xcode cli tools resolution change `ctypes_dylib=` to `ctypes_native_library=` add a newline move UnsupportedPlatformError to be a class field remove unused codegen_types field fix zinc-compiler options to be valid ones Construct rule_graph recursively (pantsbuild#5955) The `RuleGraph` is currently constructed iteratively, but can be more-easily constructed recursively. Switch to constructing the `RuleGraph` recursively, and unify a few disparate diagnostic messages. Helps to prepare for further refactoring in pantsbuild#5788. Allow manylinux wheels when resolving plugins. (pantsbuild#5959) Also plumb manylinux resolution support for the python backend, on by default, but configurable via `python_setup.resolver_use_manylinux`. Fixes pantsbuild#5958 `exclude-patterns` and `tag` should apply only to roots (pantsbuild#5786) The `--exclude-patterns` flag currently applies to inner nodes, which causes odd errors. Moreover, tags should also apply only to roots. See pantsbuild#5189. - added `tag` & `exclude_patterns` as params to `Specs` - add tests for both - modify changed tests to pass for inner node filtering Fixes pantsbuild#5189. Remove value wrapper on the python side of ffi. (pantsbuild#5961) As explained in the comment in this change, the overhead of wrapping our CFFI "handle"/`void*` instances in a type that is shaped like the `Value` struct was significant enough to care about. Since the struct has zero overhead on the rust side, whether we represent it as typedef or a struct on the python side doesn't make a difference (except in terms of syntax). 6% faster `./pants list ::` in Twitter's repo. return an actual field use temporary native-compile goal Cobertura coverage: Include the full target closure's classpath entries for instrumentation (pantsbuild#5879) Sometimes Cobertura needs access to the dependencies of class files being instrumented in order to rewrite them (pantsbuild#5878). This patch adds an option that creates a manifest jar and adds an argument to the Cobertura call so that it can take advantage of it. class files that need to determine a least upper bound in order to be rewritten can now be instrumented. Fixes pantsbuild#5878 add ticket link fix xcode install locations and reduce it to a single dir_option return the correct amount of path entries Record start times per graph node and expose a method to summarize them. (pantsbuild#5964) In order to display "significant" work while it is running on the engine, we need to compute interesting, long-running leaves. Record start times per entry in the `Graph`, and add a method to compute a graph-aware top-k longest running leaves. We traverse the graph "longest running first", and emit the first `k` leaves we encounter. While this will almost certainly need further edits to maximize it's usefulness, visualization can begun to be built atop of it. Prepare the 1.8.0.dev4 release (pantsbuild#5969) Mark a few options that should not show up in `./pants help`. (pantsbuild#5968) `./pants help` contains core options that are useful to every pants command, and the vast majority of global options are hidden in order to keep it concise. A few non-essential options ended up there recently. Hide them. adding more documentation for python_app (pantsbuild#5965) The python_app target doesn't have the documentation specific for it and has a documentation that is specific to jvm_app. Added a few lines of documentation. There is no system-wide change, only a documentation change. Remove DeprecatedPythonTaskTestBase (pantsbuild#5973) Use PythonTaskTestBase instead. Fixes pantsbuild#5870 Chris first commit on fresh rebase Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party unrevert reverted fix (NEEDS FOLLOWUP ISSUE!) put in a better fix for the strict_deps error until the followup issue is made add ticket link Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party Shorten safe filenames further, and combine codepaths to make them readable. (pantsbuild#5971) Lower the `safe_filename` path component length limit to 100 characters, since the previous 255 value did not account for the fact that many filesystems also have a limit on total path length. This "fixes" the issue described in pantsbuild#5587, which was caused by using this method via `build_invalidator.py`. Additionally, merge the codepath from `Target.compute_target_id` which computes a readable shortened filename into `safe_filename`, and expand tests. This removes some duplication, and ensure that we don't run into a similar issue with target ids. The specific error from pantsbuild#5587 should be prevented, and consumers of `safe_filename` should have safe and readable filenames. Fixes pantsbuild#5587. Whitelist the --owner-of option to not restart the daemon. (pantsbuild#5979) Because the `--owner-of` option was not whitelisted as `daemon=False`, changing its value triggered unnecessary `pantsd` restarts. Whitelist it. Prepare the 1.8.0rc0 release. (pantsbuild#5980) Robustify test_namespace_effective PYTHONPATH. (pantsbuild#5976) The real problem is noted, but this quick fix should bolster against interpreter cache interpreters pointing off to python binaries that have no setuptools in their associated site-packages. Fixes pantsbuild#5972 make_target upgrades sources to EagerFilesetWithSpec (pantsbuild#5974) This better simulates how the engine parses BUILD files, giving a more faithful experience in tests. I'm about to make it a warning/error to pass a list of strings as the sources arg, so this will make tests which use make_target continue to work after that. Also, make cloc use base class scheduler instead of configuring its own. Lib and include as a dep-specifc location source attribute is automatically promoted to sources (pantsbuild#5908) This means that either the `source` or `sources` attribute can be used for any rule which expects sources. Places that `source` was expected still verify that the correct number of sources are actually present. Places that `sources` was expected will automatically promote `source` to `sources`. This is a step towards all `sources` attributes being `EagerFilesetWithSpec`s, which will make them cached in the daemon, and make them easier to work with with both v2 remote execution and in the v2 engine in general. It also provides common hooks for input file validation, rather than relying on them being done ad-hoc in each `Target` constructor. For backwards compatibility, both attributes will be populated on `Target`s, but in the future only the sources value will be provided. `sources` is guaranteed to be an `EagerFilesetWithSpec` whichever of these mechanisms is used. A hook is provided for rules to perform validation on `sources` at build file parsing time. Hooks are put in place for non-contrib rule types which currently take a `source` attribute to verify that the correct number of sources are provided. I imagine at some point we may want to add a "file type" hook too, so that rules can error if files of the wrong type were added as sources. This is a breaking change for rules which use both the `source` and `sources` attributes (and where the latter is not equivalent to the former), or where the `source` attribute is used to refer to something other than a file. `source` is now becoming a reserved attribute name, as `sources` and `dependencies` already are. This is also a breaking change for rules which use the `source` attribute, but never set `sources` in a Payload. These will now fail to parse. This is also a slightly breaking change for the `page` rule - before, omitting the `source` attribute would parse, but fail at runtime. Now, it will fail to parse. This is also a breaking change in that in means that the source attribute is now treated like a glob, and so if a file is specified which isn't present, it will be ignored instead of error. This feels a little sketchy, but it turns out we did the exact same thing by making all sources lists be treated like globs... Override get_sources for pants plugins (pantsbuild#5984) 1.7.0 release notes (pantsbuild#5983) No additional changes, so it's a very short release note. Fixups for native third party work hardcode in c/c++ language levels for now remove all the unnecessary code relating to file extensions Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party Caching tests are parsed through the engine (pantsbuild#5985) reimplement a previous PR -- ignore this This commit is a reimplementation of registering @rules for backends, because this PR began before that one was split off. add some simple examples to demonstrate how to use backend rules ...actually make the changes to consume backend rules in register.py revert accidental change to a test target remove extraneous log statement fix lint errors add native backend to release.sh isolate native toolchain path and hope for the best add config subdir to native backend really just some more attempts start trying to dispatch based on platform extend EngineInitializer to add more rules from a backend refactor Platform to use new methods in osutil.py refactor the native backend to be a real backend and expose rules register a rule in the python backend to get a setup.py environment make python_dist() tests pass make lint pass create tasks and targets for c/c++ sources - refactors the "native toolchain" and introduces the "binaries" subdirectory of subsystems start by adding a new ctypes testproject add c/c++ sources add example BUILD file add some native targets add tasks dir remove gcc try to start adding tasks clean some leftover notes in BuildLocalPythonDistributions update NativeLibrary with headers move DependencyContext to target.py add native compile tasks houston we have compilation run: ./pants -ldebug --print-exception-stacktrace compile testprojects/src/python/python_distribution/ctypes: for an idea use the forbidden product request change target names to avoid conflict with cpp contrib and flesh out cpp_compile now we are compiling code we can link things now now we know how to infer headers vs sources fix the test case and fix include dir collection (un)?suprisingly, everything works, but bdist_wheel doesn't read MANIFEST.in houston we have c++ bring back gcc so we can compile halfway done with osx support now things work on osx???????? ok, now it works on linux again too first round of review - NB: refactors the organization of the `results_dir` for python_dist()! - move ctypes integration testing into python backend tests revert some unnecessary changes refactor native_artifact to be a datatype fix some copyright years add ctypes integration test add assert_single_element method in collections.py decouple the native tools for setup.py from the execution environment streamline the selection of the native tools for setup.py invocation make gcc depend on binutils on linux for the 'as' assembler fix logging visibility by moving it back into the task make the check for the external llvm url more clear refactor local dist building a bit - use SetupPyRunner.DIST_DIR as the source of truth - add a separate subdir of the python_dist target's results_dir for the python_dist sources - move shraed libs into the new subdir fix imports second round of review - fixed bugs - expanded error messages and docstrings make a couple docstring changes fix dist platform selection ('current' is not a real platform) lint fixes fix broken regex which modifies the `.dylib` extension for python_dist() fix the ctypes integration test respond to some review comments clear the error message if we can't find xcode cli tools refactor the compilation and linking pipeline to use subsystems - also add `PythonNativeCode` subsystem to bridge the native and python backends refactor the compilation and linking pipeline to use subsystems add some notes fix rebase issues add link to pantsbuild#5788 -- maybe use variants for args for static libs move `native_source_extensions` to a new `PythonNativeCode` subsystem update native toolchain docs and remove bad old tests move tgt_closure_platforms into the new `PythonNativeCode` subsystem remove unnecessary logging remove compile_settings_class in favor of another abstractmethod refactor `NativeCompile` and add documentation improve debug logging in NativeCompile document NativeCompileSettings refactor and add docstrings convert provides= to ctypes_dylib= and add many more docstrings remove or improve TODOs improve or remove FIXMEs improve some docstrings, demote a FIXME, and add a TODO link FIXMEs to a ticket add notes to the ctypes testproject update mock object for strict deps -- test passes fix failing integration test on osx add hack to let travis pass fix the system_id key in llvm and add a shameful hack to pass travis swap the order of alias_types remove unused EmptyDepContext class remove -settings suffix from compile subsystem options scopes add AbstractClass to NativeLibrary bump implementation_version for python_dist() build - we have changed the layout of the results_dir in this PR add ticket link and fix bug revert indentation changes to execute() method refactor `assert_single_element()` revert addition of `narrow_relative_paths()` add link to pantsbuild#5950 move common process invocation logic into NativeCompile revert an unnecessary change turn an assert into a full exception revert unnecessary change use get_local_platform() wherever possible delete duplicate llvm subsystem fix xcode cli tools resolution change `ctypes_dylib=` to `ctypes_native_library=` add a newline move UnsupportedPlatformError to be a class field remove unused codegen_types field fix zinc-compiler options to be valid ones return an actual field use temporary native-compile goal add ticket link fix xcode install locations and reduce it to a single dir_option return the correct amount of path entries unrevert reverted fix (NEEDS FOLLOWUP ISSUE!) put in a better fix for the strict_deps error until the followup issue is made add ticket link hardcode in c/c++ language levels for now remove all the unnecessary code relating to file extensions fix osx failures and leave a ticket link Add rang header-only lib for integration testing Merge branch 'ctypes-test-project' of github.com:cosmicexplorer/pants into clivingston/ctypes-test-project-third-party Fix TestSetupPyInterpreter.test_setuptools_version (pantsbuild#5988) Previously the test failed to populate the `PythonInterpreter` data product leading to a fallback to the current non-bare interpreter which allowed `setuptools` from `site-packages` to leak in. Fixes pantsbuild#5467 Refactor conan grab into subsystem Engine looks up default sources when parsing (pantsbuild#5989) Rather than re-implementing default source look-up. This pushes sources parsing of default sources through the engine, in parallel, rather than being later synchronous python calls. This also works for plugin types, and doesn't change any existing APIs. It updates the Go patterns to match those that the engine currently performs, rather than ones which aren't actually used by any code. Add unit tests, refactor C/C++ targets which can be compiled/linked and used in python_dist() with ctypes (pantsbuild#5815) It is currently possible to expose native code to Python by compiling it in a `python_dist()` target, specifying C or C++ source files as a `distutils.core.Extension` in the `setup.py` file, as well as in the target's sources. `python_dist()` was introduced in pantsbuild#5141. We introduced a "native toolchain" to compile native sources for this use case in pantsbuild#5490. Exposing Python code this way requires using the Python native API and `#include <Python.h>` in your source files. However, python can also interact with native code that does not use the Python native API, using the provided `ctypes` library. For this to work, the `python_dist()` module using `ctypes` needs to have a platform-specific shared library provided within the package. This PR introduces the targets, tasks, and subsystems to compile and link a shared library from native code, then inserts it into the `python_dist()` where it is easily accessible. - Introduce the `ctypes_compatible_c_library()` target which covers C sources (`ctypes_compatible_cpp_library()` for C++), and can specify what to name the shared library created from linking the object files compiled from its sources and dependencies. - Introduce `CCompile`, `CppCompile`, and `LinkSharedLibraries` to produce the shared libraries from the native sources. The compile tasks use options `CCompileSettings` or `CppCompileSettings` to define file extensions for "header" and "source" files. - Introduce the `CCompileSettings` and `CppCompileSettings` subsystems to control compile settings for those languages. - Convert `BuildLocalPythonDistributions` to proxy to the native backend through the new `PythonNativeCode` subsystem. - Move all the `BinaryTool` subsystems to a `subsystems/binaries` subdirectory, and expose them to the v2 engine through `@rule`s defined in the subsystem's file. - Move some of the logic in `pex_build_util.py` to `setup_py.py`, and expose datatypes composing the setup.py environment through `@rule`s in `setup_py.py`. `SetupPyRunner.for_bdist_wheel()` was created to set the wheel's platform, if the `python_dist()` target contains any native sources of its own, or depends on any `ctypes_compatible_*_library`s. **Note:** the new targets are specifically prefixed with `ctypes_compatible_` because we don't yet eclipse the functionality of `contrib/cpp`. When the targets become usable for more than this one use case, the name should be changed. To see how to link up native and Python code with `ctypes`, here's (most of) the contents of `testprojects/src/python/python_distribution/ctypes`: *BUILD*: ```python ctypes_compatible_c_library( name='c_library', sources=['some_math.h', 'some_math.c', 'src-subdir/add_three.h', 'src-subdir/add_three.c'], ctypes_dylib=native_artifact(lib_name='asdf-c'), ) ctypes_compatible_cpp_library( name='cpp_library', sources=['some_more_math.hpp', 'some_more_math.cpp'], ctypes_dylib=native_artifact(lib_name='asdf-cpp'), ) python_dist( sources=[ 'setup.py', 'ctypes_python_pkg/__init__.py', 'ctypes_python_pkg/ctypes_wrapper.py', ], dependencies=[ ':c_library', ':cpp_library', ], ) ``` *setup.py*: ```python setup( name='ctypes_test', version='0.0.1', packages=find_packages(), # Declare two files at the top-level directory (denoted by ''). data_files=[('', ['libasdf-c.so', 'libasdf-cpp.so'])], ) ``` *ctypes_python_pkg/ctypes_wrapper.py*: ```python import ctypes import os def get_generated_shared_lib(lib_name): # These are the same filenames as in setup.py. filename = 'lib{}.so'.format(lib_name) # The data files are in the root directory, but we are in ctypes_python_pkg/. rel_path = os.path.join(os.path.dirname(__file__), '..', filename) return os.path.normpath(rel_path) asdf_c_lib_path = get_generated_shared_lib('asdf-c') asdf_cpp_lib_path = get_generated_shared_lib('asdf-cpp') asdf_c_lib = ctypes.CDLL(asdf_c_lib_path) asdf_cpp_lib = ctypes.CDLL(asdf_cpp_lib_path) def f(x): added = asdf_c_lib.add_three(x) multiplied = asdf_cpp_lib.multiply_by_three(added) return multiplied ``` Now, the target `testprojects/src/python/python_distribution/ctypes` can be depended on in a BUILD file, and other python code can freely use `from ctypes_python_pkg.ctypes_wrapper import f` to start jumping into native code. 1. pantsbuild#5933 2. pantsbuild#5934 3. pantsbuild#5949 4. pantsbuild#5950 5. pantsbuild#5951 6. pantsbuild#5962 7. pantsbuild#5967 8. pantsbuild#5977 Add simple integration test Pull in master Minor cleanup Fix CI errors Debug log stdout Fix integration test Fix integration test Fix lint error on third party
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
(this change depends on #4728)
As described on #4728, we currently need internal knowledge of zinc analysis in order to make it portable, and to extract products that are used by other tasks. Since the zinc analysis format is changing significantly (moving to protobuf, gaining/losing fields, etc), we need to stop parsing it.
Solution
Incorporates the version of the zinc wrapper published in #4728, and uses it in a
analysis.zinctask to generate theproduct_deps_by_srcandclasses_by_sourceproducts from azinc_analysisproduct published by zinc.Expands the extracted
Zincsubsystem from #4720 to encapsulate compile dependency calculation, and expose theFingerprintStrategyandDependencyContextthatcompile.zincwas using.Result
Zinc analysis is now a black box from the perspective of the pants python code. Also, the
compile.jvm-dep-checktask moved to thelintgoal, and gained the unused dependency checks that used to be inlined incompile.zinc(and which would have prevented separation of the analysis extraction task).Fixes #4477, fixes #4513, fixes #4185, fixes #5444, and hopefully fixes a few other odd incremental compilation bugs.