Tracking Issue for autodiff #124509

traviscross · 2024-04-29T06:59:26Z

This is a tracking issue for the automatic differentiation experiment ("autodiff").

The feature gate for the issue will be #![feature(autodiff)].

About tracking issues

Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label. Discussion comments will get marked as off-topic or deleted. Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Steps

Get compiler MCP approved.
- Integrate Enzyme into nightly rustc compiler-team#611
Get lang experiment approved.
- We approved this in the lang triage meeting on 2024-05-01.
Land the experimental implementation in nightly.
Accept an RFC.
Add documentation to the dev guide.
- See the instructions.
Add documentation to the reference.
- See the instructions.
Add formatting for new syntax to the style guide.
- See the nightly style procedure.

Unresolved Questions

TODO.

cc @ZuseZ4 @oli-obk

The text was updated successfully, but these errors were encountered:

traviscross · 2024-05-01T20:45:08Z

@rustbot labels -I-lang-nominated

We discussed this on the lang triage call today and were happy to see this move forward as an experiment. I'll be continuing as the liaison for this, so please let me know whenever anything is needed to keep this moving along.

…san68 Autodiff Upstreaming - enzyme backend Tracking issue: rust-lang#124509 Part of rust-lang#129175 This PR should allow building Enzyme from source on Tier 1 targets (when also building LLVM), except MSVC. It's only a small fraction (~200 lines) of the whole upstream PR, but due to bootstrapping and the number of configurations in which rustc can be build I assume that this will be the hardest to merge, so I'm starting with it. Happy to hear what changes are required to be able to upstream this code. **Content:** It contains a new configure flag `--enable-llvm-enzyme`, and will build the new Enzyme submodule when it is set. **Discussion:** Apparently Rust CI isn't able to clone repositories outside the rust-lang org? At least I'm seeing this error in CI: ``` git@github.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` Does that mean we would need to mirror github.com/EnzymeAD/Enzyme in rust-lang, until LLVM upgrades Enzyme from an Incubator project to something that ships as part of the monorepo? Tracking: - rust-lang#124509

Autodiff Upstreaming - enzyme backend Tracking issue: rust-lang/rust#124509 Part of rust-lang/rust#129175 This PR should allow building Enzyme from source on Tier 1 targets (when also building LLVM), except MSVC. It's only a small fraction (~200 lines) of the whole upstream PR, but due to bootstrapping and the number of configurations in which rustc can be build I assume that this will be the hardest to merge, so I'm starting with it. Happy to hear what changes are required to be able to upstream this code. **Content:** It contains a new configure flag `--enable-llvm-enzyme`, and will build the new Enzyme submodule when it is set. **Discussion:** Apparently Rust CI isn't able to clone repositories outside the rust-lang org? At least I'm seeing this error in CI: ``` git@github.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` Does that mean we would need to mirror github.com/EnzymeAD/Enzyme in rust-lang, until LLVM upgrades Enzyme from an Incubator project to something that ships as part of the monorepo? Tracking: - rust-lang/rust#124509

add has_enzyme/needs-enzyme to the test infra This unblocks merging the Enzyme / Autodiff frontend. For the full implementation, see: rust-lang#129175 We don't want to run tests that require Enzyme / Autodiff support when we build rustc without the required features. It correctly filtered out a test which started with `//@ needs-enzyme`. ``` running 80 tests i............................................................................... test result: ok. 79 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 380.41ms ``` Tracking: - rust-lang#124509 r? jieyouxu

add has_enzyme/needs-enzyme to the test infra This unblocks merging the Enzyme / Autodiff frontend. For the full implementation, see: rust-lang/rust#129175 We don't want to run tests that require Enzyme / Autodiff support when we build rustc without the required features. It correctly filtered out a test which started with `//@ needs-enzyme`. ``` running 80 tests i............................................................................... test result: ok. 79 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 380.41ms ``` Tracking: - rust-lang/rust#124509 r? jieyouxu

…ouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//`@needs-enzyme`` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //`@[has_support]` needs-enzyme //`@[no_support]` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? `@jieyouxu` Tracking: - rust-lang#124509

…ouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//``@needs-enzyme``` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //``@[has_support]`` needs-enzyme //``@[no_support]`` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? ``@jieyouxu`` Tracking: - rust-lang#124509

…ouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//```@needs-enzyme```` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //```@[has_support]``` needs-enzyme //```@[no_support]``` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? ```@jieyouxu``` Tracking: - rust-lang#124509

…ouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//````@needs-enzyme````` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //````@[has_support]```` needs-enzyme //````@[no_support]```` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? ````@jieyouxu```` Tracking: - rust-lang#124509

…ouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//`````@needs-enzyme`````` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //`````@[has_support]````` needs-enzyme //`````@[no_support]````` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? `````@jieyouxu````` Tracking: - rust-lang#124509

Rollup merge of rust-lang#131470 - EnzymeAD:enzyme-testinfra2, r=jieyouxu add test infra to explicitely test rustc with autodiff/enzyme disabled I assume this is not what you want for now, but I'll update the PR once I understand how the ignore- directives work. To summarize the situation, we want a feature gate test where we don't enable the autodiff feature using `#![feature(autodiff)]`. There are two situations. 1) We have a rustc which was build without autodiff support (current default): It gives one error about the feature being needed and one error about this rustc version being build without autodiff support. 2) We have a rustc which was build with autodiff support (i.e. for now a custom build): It gives one error about the feature being needed. We have a `//`````@needs-enzyme`````` directive which we can use in revisions for the second case. However, we have no way to specify that needs-enzyme implies that the second error should not be seen. This ads a way of passing the following test: ``` //@ revisions: has_support no_support //`````@[has_support]````` needs-enzyme //`````@[no_support]````` needs-enzyme-disabled #![crate_type = "lib"] #[autodiff(dfoo, Reverse)] //[has_support]~^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~^^ ERROR use of unstable library feature 'autodiff' [E0658] //[no_support]~| ERROR this rustc version does not support autodiff fn foo() {} ``` Cherry picking this PR to my frontend pr makes the test above pass in both configurations (enzyme=true/false in config.toml). I'm open to other changes that make this testcase pass. r? `````@jieyouxu````` Tracking: - rust-lang#124509

Autodiff Upstreaming - enzyme frontend This is an upstream PR for the `autodiff` rustc_builtin_macro that is part of the autodiff feature. For the full implementation, see: rust-lang#129175 **Content:** It contains a new `#[autodiff(<args>)]` rustc_builtin_macro, as well as a `#[rustc_autodiff]` builtin attribute. The autodiff macro is applied on function `f` and will expand to a second function `df` (name given by user). It will add a dummy body to `df` to make sure it type-checks. The body will later be replaced by enzyme on llvm-ir level, we therefore don't really care about the content. Most of the changes (700 from 1.2k) are in `compiler/rustc_builtin_macros/src/autodiff.rs`, which expand the macro. Nothing except expansion is implemented for now. I have a fallback implementation for relevant functions in case that rustc should be build without autodiff support. The default for now will be off, although we want to flip it later (once everything landed) to on for nightly. For the sake of CI, I have flipped the defaults, I'll revert this before merging. **Dummy function Body:** The first line is an `inline_asm` nop to make inlining less likely (I have additional checks to prevent this in the middle end of rustc. If `f` gets inlined too early, we can't pass it to enzyme and thus can't differentiate it. If `df` gets inlined too early, the call site will just compute this dummy code instead of the derivatives, a correctness issue. The following black_box lines make sure that none of the input arguments is getting optimized away before we replace the body. **Motivation:** The user facing autodiff macro can verify the user input. Then I write it as args to the rustc_attribute, so from here on I can know that these values should be sensible. A rustc_attribute also turned out to be quite nice to attach this information to the corresponding function and carry it till the backend. This is also just an experiment, I expect to adjust the user facing autodiff macro based on user feedback, to improve usability. As a simple example of what this will do, we can see this expansion: From: ``` #[autodiff(df, Reverse, Duplicated, Const, Active)] pub fn f1(x: &[f64], y: f64) -> f64 { unimplemented!() } ``` to ``` #[rustc_autodiff] #[inline(never)] pub fn f1(x: &[f64], y: f64) -> f64 { ::core::panicking::panic("not implemented") } #[rustc_autodiff(Reverse, Duplicated, Const, Active,)] #[inline(never)] pub fn df(x: &[f64], dx: &mut [f64], y: f64, dret: f64) -> f64 { unsafe { asm!("NOP"); }; ::core::hint::black_box(f1(x, y)); ::core::hint::black_box((dx, dret)); ::core::hint::black_box(f1(x, y)) } ``` I will add a few more tests once I figured out why rustc rebuilds every time I touch a test. Tracking: - rust-lang#124509

Autodiff Upstreaming - enzyme frontend This is an upstream PR for the `autodiff` rustc_builtin_macro that is part of the autodiff feature. For the full implementation, see: rust-lang#129175 **Content:** It contains a new `#[autodiff(<args>)]` rustc_builtin_macro, as well as a `#[rustc_autodiff]` builtin attribute. The autodiff macro is applied on function `f` and will expand to a second function `df` (name given by user). It will add a dummy body to `df` to make sure it type-checks. The body will later be replaced by enzyme on llvm-ir level, we therefore don't really care about the content. Most of the changes (700 from 1.2k) are in `compiler/rustc_builtin_macros/src/autodiff.rs`, which expand the macro. Nothing except expansion is implemented for now. I have a fallback implementation for relevant functions in case that rustc should be build without autodiff support. The default for now will be off, although we want to flip it later (once everything landed) to on for nightly. For the sake of CI, I have flipped the defaults, I'll revert this before merging. **Dummy function Body:** The first line is an `inline_asm` nop to make inlining less likely (I have additional checks to prevent this in the middle end of rustc. If `f` gets inlined too early, we can't pass it to enzyme and thus can't differentiate it. If `df` gets inlined too early, the call site will just compute this dummy code instead of the derivatives, a correctness issue. The following black_box lines make sure that none of the input arguments is getting optimized away before we replace the body. **Motivation:** The user facing autodiff macro can verify the user input. Then I write it as args to the rustc_attribute, so from here on I can know that these values should be sensible. A rustc_attribute also turned out to be quite nice to attach this information to the corresponding function and carry it till the backend. This is also just an experiment, I expect to adjust the user facing autodiff macro based on user feedback, to improve usability. As a simple example of what this will do, we can see this expansion: From: ``` #[autodiff(df, Reverse, Duplicated, Const, Active)] pub fn f1(x: &[f64], y: f64) -> f64 { unimplemented!() } ``` to ``` #[rustc_autodiff] #[inline(never)] pub fn f1(x: &[f64], y: f64) -> f64 { ::core::panicking::panic("not implemented") } #[rustc_autodiff(Reverse, Duplicated, Const, Active,)] #[inline(never)] pub fn df(x: &[f64], dx: &mut [f64], y: f64, dret: f64) -> f64 { unsafe { asm!("NOP"); }; ::core::hint::black_box(f1(x, y)); ::core::hint::black_box((dx, dret)); ::core::hint::black_box(f1(x, y)) } ``` I will add a few more tests once I figured out why rustc rebuilds every time I touch a test. Tracking: - rust-lang#124509 try-job: dist-x86_64-msvc

Autodiff Upstreaming - enzyme frontend This is an upstream PR for the `autodiff` rustc_builtin_macro that is part of the autodiff feature. For the full implementation, see: rust-lang/rust#129175 **Content:** It contains a new `#[autodiff(<args>)]` rustc_builtin_macro, as well as a `#[rustc_autodiff]` builtin attribute. The autodiff macro is applied on function `f` and will expand to a second function `df` (name given by user). It will add a dummy body to `df` to make sure it type-checks. The body will later be replaced by enzyme on llvm-ir level, we therefore don't really care about the content. Most of the changes (700 from 1.2k) are in `compiler/rustc_builtin_macros/src/autodiff.rs`, which expand the macro. Nothing except expansion is implemented for now. I have a fallback implementation for relevant functions in case that rustc should be build without autodiff support. The default for now will be off, although we want to flip it later (once everything landed) to on for nightly. For the sake of CI, I have flipped the defaults, I'll revert this before merging. **Dummy function Body:** The first line is an `inline_asm` nop to make inlining less likely (I have additional checks to prevent this in the middle end of rustc. If `f` gets inlined too early, we can't pass it to enzyme and thus can't differentiate it. If `df` gets inlined too early, the call site will just compute this dummy code instead of the derivatives, a correctness issue. The following black_box lines make sure that none of the input arguments is getting optimized away before we replace the body. **Motivation:** The user facing autodiff macro can verify the user input. Then I write it as args to the rustc_attribute, so from here on I can know that these values should be sensible. A rustc_attribute also turned out to be quite nice to attach this information to the corresponding function and carry it till the backend. This is also just an experiment, I expect to adjust the user facing autodiff macro based on user feedback, to improve usability. As a simple example of what this will do, we can see this expansion: From: ``` #[autodiff(df, Reverse, Duplicated, Const, Active)] pub fn f1(x: &[f64], y: f64) -> f64 { unimplemented!() } ``` to ``` #[rustc_autodiff] #[inline(never)] pub fn f1(x: &[f64], y: f64) -> f64 { ::core::panicking::panic("not implemented") } #[rustc_autodiff(Reverse, Duplicated, Const, Active,)] #[inline(never)] pub fn df(x: &[f64], dx: &mut [f64], y: f64, dret: f64) -> f64 { unsafe { asm!("NOP"); }; ::core::hint::black_box(f1(x, y)); ::core::hint::black_box((dx, dret)); ::core::hint::black_box(f1(x, y)) } ``` I will add a few more tests once I figured out why rustc rebuilds every time I touch a test. Tracking: - rust-lang/rust#124509 try-job: dist-x86_64-msvc

rustbot removed the I-lang-nominated Nominated for discussion during a lang team meeting. label May 1, 2024

ZuseZ4 mentioned this issue May 1, 2024

Timeline for integrating into Rust upstream? EnzymeAD/rust#67

Open

ZuseZ4 mentioned this issue Aug 17, 2024

Autodiff Upstreaming - single commit #129175

Draft

traviscross mentioned this issue Aug 17, 2024

Autodiff Upstreaming - enzyme backend #129176

Merged

jieyouxu added the F-autodiff `#![feature(autodiff)]` label Aug 18, 2024

ZuseZ4 mentioned this issue Aug 23, 2024

Autodiff Upstreaming - enzyme frontend #129458

Merged

This was referenced Sep 7, 2024

Autodiff Upstreaming - rustc_codegen_llvm changes EnzymeAD/rust#178

Open

Autodiff Upstreaming - rustc_codegen_llvm changes #130060

Open

This was referenced Sep 22, 2024

tool builds with Enzyme/autodiff fails #130637

Closed

move enzyme flags from general cargo to rustc-specific cargo #130648

Merged

ZuseZ4 mentioned this issue Sep 29, 2024

add has_enzyme/needs-enzyme to the test infra #131044

Merged

ZuseZ4 mentioned this issue Oct 10, 2024

add test infra to explicitely test rustc with autodiff/enzyme disabled #131470

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking Issue for autodiff #124509

Tracking Issue for autodiff #124509

traviscross commented Apr 29, 2024 •

edited by jieyouxu

Loading

traviscross commented May 1, 2024 •

edited

Loading

Tracking Issue for autodiff #124509

Tracking Issue for autodiff #124509

Comments

traviscross commented Apr 29, 2024 • edited by jieyouxu Loading

About tracking issues

Steps

Related

Unresolved Questions

traviscross commented May 1, 2024 • edited Loading

traviscross commented Apr 29, 2024 •

edited by jieyouxu

Loading

traviscross commented May 1, 2024 •

edited

Loading