Adding Contributing file #2
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
There was a problem hiding this comment.
@zeroxoneb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@zeroxoneb has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@facebook-github-bot has updated the pull request. Re-import the pull request |
|
@zeroxoneb merged this pull request in fe1fd51. |
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 14, 2022
Summary:
Having a default of 3 saves an _enormous_ amount of time when building btrfs loopback images. Most btrfs loopback targets are for tests and so don't need high compression, we should optimize for speed of builds instead. For the targets that ship into prod we can set this compression level appropriately on a case by case basis.
A single data point about build speed improvements can be seen from my devserver when building the base VM image: `//metalos/os/vm:base.btrfs`:
With the old default of max compression (turns out 15 is the highest that btrfs supports) it takes ~2 minutes to build just the loopback target:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 1.0 sec
Creating action graph: finished in 17.9 sec
Downloading... 0/12 artifacts, 0.00 bytes, 100.0% cache miss (for updated rules)
Building... 05:18.1 min (99%) 8880/8881 jobs, 17/8881 updated
- //metalos/os/vm:base.btrfs... 02:22.8 min (running computing_output_hashes[9.4 sec])
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Run #1 with compression at `3` clocks in at 32 seconds:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 1.1 sec
Creating action graph: finished in 18.4 sec
Building... 48.2 sec (99%) 146/147 jobs, 0/147 updated
- //metalos/os/vm:base.btrfs... 32.1 sec (running computing_output_hashes[8.8 sec])
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Run #2 with compression at `15` clocks in at ~2:05 minutes:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 2.3 sec
Creating action graph: finished in 21.1 sec
Building... 02:18.5 min (99%) 142/143 jobs, 0/143 updated
- //metalos/os/vm:base.btrfs... 02:05.3 min (running computing_output_hashes[8.2 sec])
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Run #2 with compression at `3` clocks in at ~23.8 seconds:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 0.9 sec
Creating action graph: finished in 14.5 sec
Building... 35.0 sec (99%) 142/143 jobs, 0/143 updated
- //metalos/os/vm:base.btrfs... 23.8 sec (preparing)
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Run #3 with compression at `15` clocks in at ~2:04 minutes:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 1.8 sec
Creating action graph: finished in 17.0 sec
Downloading... 7/7 artifacts, 19.93 Kbytes, 0.0% cache miss (for updated rules)
Building... 04:32.5 min (99%) 8932/8933 jobs, 16/8933 updated
- //metalos/os/vm:base.btrfs... 02:04.8 min (running computing_output_hashes[8.1 sec])
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Run #3 with compression at `3` clocks in at ~28 seconds:
```
]# buck build --show-output //metalos/os/vm:base.btrfs
Parsing buck files: finished in 1.7 sec
Creating action graph: finished in 16.3 sec
Downloading... 7/7 artifacts, 19.95 Kbytes, 0.0% cache miss (for updated rules)
Building... 02:59.1 min (99%) 8932/8933 jobs, 16/8933 updated
- //metalos/os/vm:base.btrfs... 28.3 sec (running computing_output_hashes[7.9 sec])
//metalos/os/vm:base.btrfs buck-out/gen/aab7ed39/metalos/os/vm/base.btrfs/layer.btrfs
```
Additionally, this does a drive by cleanup of the `LoopbackVolume` and `BtrfsLoopbackVolume` classes to simply accept `label` and `compression_level` explicitly instead of via the `loopback_opts_t` type. This makes the interface clearer IMO since the `loobpack_opts_t` type is really just for package image to consume.
Reviewed By: snarkmaster
Differential Revision: D35440279
fbshipit-source-id: 759cda6dea2adca9de01a322472273894be999ef
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 27, 2023
Summary: RPMs in an image come in two classes that we care about: 1) explicitly user-installed (with `feature.rpms_install`) 2) dependencies of the above RPMS in class #1 should be protected at all costs. The image API is designed to be very explicit, we don't want things to unexpectedly change out from under the author. To accomplish this we refuse to complete any dnf transaction that _implicitly_ removes any of these packages. To remove a user-installed package, one must use `feature.rpms_remove(_if_exists)`. Note that this is slightly different from antlir1 (that accomplishes _similar_ but not quite _identical_ functionality via `dnf-plugin-flunk_dependent_remove`) in that transitively installed rpms (class #2 above) _can_ be removed implicitly in a transaction that removes the rpm that originally pulled the transitively-removed package in as a dependency, but this is desirable since the user never asked for those RPMs in the first place. As a consequence of this, RPMs that are required for image operation must always be explicitly installed - if they are only brought in as a dependency, it may be transitively removed later if no longer required by any user-installed rpms. Test Plan: Before updating the remove-test ``` ❯ buck2 test fbcode//antlir/antlir2/test_images/rpms:remove-test File changed: fbcode//antlir/antlir2/antlir2_compile/dnf-driver.py Action failed: fbcode//antlir/antlir2/test_images/rpms:remove (antlir2_map plan) Local command returned non-zero exit code 1 Reproduce locally: `env -- "RUST_LOG=antlir2=trace" sudo buck-out/v2/gen/fbcode/bb71128f9fe99274/antlir/antlir2/antlir2/ ...<omitted>... re" "--plan=buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/plan" (run `buck2 log what-failed` to get the full command)` stdout: D0427 07:38:43.264026 2690061 antlir/antlir2/antlir2/src/cmd/map.rs:137] [map, create_new_subvol{working_volume: WorkingVolume("antlir2-out")}] produced r/w subvol 'Subvolume { id: 22544, path: "/data/users/vmagro/fbsource/antlir2-out/1682606323-13dc4a48b2234ea29d4211eafd516523-fbcode--antlir-antlir2-test_images-rpms:remove-(ovr_config--platform-linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)-plan", info: SubvolumeInfo { id: 22544, parent_id: Some(5), dir_id: Some(137696645), flags: (empty), uuid: be621f55-cd8c-9a42-aee3-5e5da213192c, parent_uuid: Some(affa5a57-7b83-4145-98b8-ff61884576ba), received_uuid: None, generation: 810965, ctransid: 810965, otransid: 810965, stransid: None, rtransid: None } }' I0427 07:38:43.264103 2690061 antlir/antlir2/antlir2/src/cmd/map.rs:101] [map, create_new_subvol{working_volume: WorkingVolume("antlir2-out")}] return: Subvolume { id: 22544, path: "/data/users/vmagro/fbsource/antlir2-out/1682606323-13dc4a48b2234ea29d4211eafd516523-fbcode--antlir-antlir2-test_images-rpms:remove-(ovr_config--platform-linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)-plan", info: SubvolumeInfo { id: 22544, parent_id: Some(5), dir_id: Some(137696645), flags: (empty), uuid: be621f55-cd8c-9a42-aee3-5e5da213192c, parent_uuid: Some(affa5a57-7b83-4145-98b8-ff61884576ba), received_uuid: None, generation: 810965, ctransid: 810965, otransid: 810965, stransid: None, rtransid: None } } D0427 07:38:43.264981 2690061 antlir/antlir2/antlir2/src/cmd/map.rs:220] [map] isolated command: "sudo" "systemd-nspawn" "--quiet" "--directory" "buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/facebook/images/build_appliance/__build-appliance.c9__/subvol_symlink" "--ephemeral" "--register=no" "--private-network" "--as-pid2" "--chdir" "/data/users/vmagro/fbsource" "--setenv" "RUST_LOG=antlir2=trace" "--bind-ro" "/data/users/vmagro/fbsource" "--bind-ro" "/mnt/gvfs" "--bind-ro" "/usr/local/fbcode" "--bind-ro" "/data/users/vmagro/fbsource" "--bind-ro" "/data/users/vmagro/fbsource/antlir2-out" "--bind-ro" "/data/users/vmagro/fbsource/buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/dnf_repodatas" "--bind" "/data/users/vmagro/fbsource/antlir2-out" "--bind" "/data/users/vmagro/fbsource/buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/plan" "--" "/data/users/vmagro/fbsource/buck-out/v2/gen/fbcode/bb71128f9fe99274/antlir/antlir2/antlir2/__antlir2__/shared/antlir2" "plan" "--label" "fbcode//antlir/antlir2/test_images/rpms:remove (ovr_config//platform/linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)" "--target-arch" "x86_64" "--depgraph-json" "buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/depgraph.json.pre" "--root" "/data/users/vmagro/fbsource/antlir2-out/1682606323-13dc4a48b2234ea29d4211eafd516523-fbcode--antlir-antlir2-test_images-rpms:remove-(ovr_config--platform-linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)-plan" "--dnf-repos" "buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/dnf_repodatas" "--dnf-versionlock" "buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/facebook/flavor/__centos9--versionlock__/versionlock.json" "--plan" "buck-out/v2/gen/fbcode/e97a0788aa35bdc8/antlir/antlir2/test_images/rpms/__remove__/plan" Traceback (most recent call last): File "/antlir/dnf-driver.py", line 298, in <module> main() File "/antlir/dnf-driver.py", line 283, in main T0427 07:38:49.915650 2 antlir/antlir2/antlir2_compile/src/rpms.rs:188] [plan] dnf-driver: TransactionResolved { install: {}, remove: {Package { name: "foo", epoch: 0, version: "3", release: "1", arch: "noarch" }, Package { name: "foobar", epoch: 0, version: "3", release: "1", arch: "noarch" }, Package { name: "foobarbaz", epoch: 0, version: "3", release: "1", arch: "noarch" }} } raise RuntimeError( RuntimeError: This transaction would remove some explicitly installed packages. Modify the image features to explicitly remove these packages: foobarbaz dnf-driver.py failed stderr: "isolated command failed: exit status: 1" Buck UI: https://www.internalfb.com/buck2/6f1c7b30-262c-4419-8aa7-7de6383077d6 Test UI: https://www.internalfb.com/intern/testinfra/testrun/5066549746485369 RE: reSessionID-35a63c03-bfee-4be3-a1c5-e1c5d2144584 Up: 16 MiB Down: 0 B Jobs completed: 89. Time elapsed: 46.6s. Cache hits: 0%. Commands: 68 (cached: 0, remote: 0, local: 68) for action key `(target: `fbcode//antlir/antlir2/test_images/rpms:remove (ovr_config//platform/linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)`, id: `6`)`: Failed to build 'fbcode//antlir/antlir2/test_images/rpms:remove (ovr_config//platform/linux:x86_64-fbcode-platform010-clang#e97a0788aa35bdc8)' 1 BUILDS FAILED Tests finished: Pass 0. Fail 0. Fatal 0. Skip 0. 1 builds failed ``` ``` ❯ buck2 test fbcode//antlir/antlir2/test_images/rpms: ``` https://www.internalfb.com/intern/testinfra/testrun/5629499701321388 Reviewed By: sergeyfd Differential Revision: D45356544 fbshipit-source-id: 26c8e2b51bae703bb26313adbb4bc6cb6f40a8a2
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 30, 2024
Summary: Move `soloctl` run to `unshare`. Chef builds can now work without root. Attempt #2 (previous was D56523503) but there are more tests now to prevent the same issue (D56670042 and D56710477) Test Plan: ``` ❯ buck2 test fbcode//antlir/antlir2/features/facebook/chef_solo/... Buck UI: https://www.internalfb.com/buck2/55b451c7-2a78-4fed-99b4-fcd4649185ab Test UI: https://www.internalfb.com/intern/testinfra/testrun/3377699949391821 Network: Up: 52MiB Down: 1.4KiB (reSessionID-1959a1bd-bdbf-484f-b951-87cb59fdae2f) Jobs completed: 1402. Time elapsed: 4:48.1s. Cache hits: 0%. Commands: 364 (cached: 0, remote: 0, local: 364) Tests finished: Pass 104. Fail 0. Fatal 0. Skip 0. Build failure 0 ``` ```name="Rootless tests" ❯ buck2 test -c antlir2.rootless=1 fbcode//antlir/antlir2/features/facebook/chef_solo/... Buck UI: https://www.internalfb.com/buck2/4004fcb8-bc92-40b8-8c82-d850c8c4832f Test UI: https://www.internalfb.com/intern/testinfra/testrun/10133099192912544 Note: Using experimental modern dice Network: Up: 55MiB Down: 1.1GiB (reSessionID-e71b5ff4-3eac-4ffb-bdef-c04d01e30b29) Jobs completed: 480607. Time elapsed: 6:52.9s. Cache hits: 20%. Commands: 1570 (cached: 314, remote: 40, local: 1216). Fallback: 8/1256 Tests finished: Pass 103. Fail 1. Fatal 0. Skip 0. Build failure 0 1 TESTS FAILED ✗ antlir/antlir2/features/facebook/chef_solo/tests/run_chef_again:run-chef-again - main ``` Only the `run-chef-test` test fails because of something being different about the test execution environment, the builds all work fine ``` ❯ buck2 test fbcode//antlir/antlir2/features/facebook/chef_solo/tests:test-stubs-preinstalled fbcode//antlir/antlir2/features/facebook/chef_solo/tests:test-stubs-installed-during-chef Buck UI: https://www.internalfb.com/buck2/529a92e9-00c7-4542-93d5-ae8f8cde74e5 Test UI: https://www.internalfb.com/intern/testinfra/testrun/2251800045779287 Network: Up: 911KiB Down: 15MiB (reSessionID-ee85565f-b979-48f2-9b17-7646d726b718) Jobs completed: 157. Time elapsed: 2:06.7s. Cache hits: 1%. Commands: 89 (cached: 1, remote: 0, local: 88) Tests finished: Pass 4. Fail 0. Fatal 0. Skip 0. Build failure 0 ``` Reviewed By: epilatow Differential Revision: D56670371 fbshipit-source-id: 5f8dc123ea2f348705e2ecb89ba57d281a511748
facebook-github-bot
pushed a commit
that referenced
this pull request
May 2, 2024
Summary: Wherever possible, make rootless image builds the default. Now, the three things that still require root are: * sendstream packages * btrfs loopback packages (which internally uses sendstreams). * `boot`ed `image_test`s All of these holdouts now do a few things configuration-wise: 1) add a `uses_sudo` label for CI scheduling purposes 2) set the `rootless=False` (which impacts the configuration transition) 3) sets `(target_)?compatible_with` so that buck2 loudly complains if any are attempting to build with the configuration `//antlir/antlir2/antlir2_rootless:rootless` #2 means that the entire dependency chain for these packages are built in the full `sudo` mode. In practice, since there are so many sendstreams, this does not make much of a difference today, but paves the way for sendstreams to not require root, and gives us good CI coverage of the rest of the unprivileged build process. Test Plan: waitforsandcastle Reviewed By: epilatow Differential Revision: D56524632 fbshipit-source-id: 634f95c9a5b11a0b27ec9c5e9ea82736df17c199
facebook-github-bot
pushed a commit
that referenced
this pull request
Oct 1, 2024
Summary: When run in TSAN, we get errors like ``` OMP: Error #178: Function Can't open SHM2 failed: OMP: System error #2: No such file or directory Error: VM run failed: ExitStatus(unix_wait_status(65024)) ``` Luckily, all we need to do is to add a tmpfs on `/dev/shm`. Test Plan: ```name="TSAN passes" ❯ buck2 test fbcode//mode/dev-tsan fbcode//metalos/vm/tests/antlir:rust-test -- is_root Buck UI: https://www.internalfb.com/buck2/fb6095a1-2060-4c59-92b8-75296909495a Test UI: https://www.internalfb.com/intern/testinfra/testrun/16607023677718079 Tests finished: Pass 1. Fail 0. Fatal 0. Skip 0. Build failure 0 ``` ```name="Default mode still passes" ❯ buck2 test fbcode//metalos/vm/tests/antlir:rust-test -- is_root Buck UI: https://www.internalfb.com/buck2/2a1e95f7-a30c-41e1-95be-1a18b11cd3e4 Test UI: https://www.internalfb.com/intern/testinfra/testrun/5066549828682805 Tests finished: Pass 1. Fail 0. Fatal 0. Skip 0. Build failure 0 ``` Reviewed By: justintrudell Differential Revision: D63710699 fbshipit-source-id: 3daf979e8fa71956a954346e70c775844b733ea7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is pull request was created automatically because we noticed your project was missing a Contributing file.
CONTRIBUTING files explain how a developer can contribute to the project - which you should actively encourage.
This PR was crafted with love by Facebook's Open Source Team.