feat: support compiling to webassembly #2254

eduardomourar · 2023-01-26T15:04:23Z

Motivation and Context

#2087

In order to compile the AWS SDK to WebAssembly, it requires to patch the ring dependency and disable most default features.

Description

This will introduce some additional improvements and allow the AWS SDK to be compiled to WebAssembly without any workaround.

Testing

A basic integration test has been added that targets wasm32-wasi. Keep in mind that we are mocking the response until outbound HTTP request is added to the WASI standard.

Checklist

I have updated CHANGELOG.next.toml if I made changes to the smithy-rs codegen or runtime crates
I have updated CHANGELOG.next.toml if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

jdisanti

This is awesome! Thank you for the contribution ❤️

My biggest comment is around disabling SSO. I think it will be better to swap out ring instead. Let me know what you think.

aws/rust-runtime/aws-config/Cargo.toml

aws/rust-runtime/aws-endpoint/Cargo.toml

...egen/src/main/kotlin/software/amazon/smithy/rustsdk/endpoints/OperationInputTestGenerator.kt

jdisanti · 2023-01-26T17:53:09Z

aws/sdk/integration-tests/Cargo.toml

@@ -15,4 +15,5 @@ members = [
    "s3control",
    "sts",
    "transcribestreaming",
+    "webassembly",


We have some prerequisite work to do before this will run in CI. I'll see if I can get around to that today.

Our best option is to install cargo-wasi then run cargo wasi test -- --nocapture

That aspect of it didn't even cross my mind yet 😅

I did the prerequisite work to actually include the tests in the first place in #2255.

in the tokio crate, they have a nice a setup for testing against the different webassembly targets: https://github.com/tokio-rs/tokio/blob/06f1a601bb05b1aba9f95020a7fa7572899c588f/.github/workflows/ci.yml#L529-L590

aws/sdk/integration-tests/webassembly/tests/test.rs

rust-runtime/aws-smithy-client/Cargo.toml

rust-runtime/aws-smithy-http/Cargo.toml

Co-authored-by: John DiSanti <johndisanti@gmail.com>

This reverts commit d8fcf49.

jdisanti

Some minor things.

aws/rust-runtime/aws-config/src/lib.rs

aws/rust-runtime/aws-config/src/profile/credentials/exec.rs

aws/rust-runtime/aws-inlineable/Cargo.toml

aws/rust-runtime/aws-types/Cargo.toml

jdisanti · 2023-01-27T18:36:23Z

For CI, there are a few different problems.

The Kotlin unit tests aren't compiling:

e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (101, 79): Unresolved reference: runtimeConfig
e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (103, 57): Function invocation 'awsCredentialTypesTestUtil(...)' expected
e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (103, 57): No value passed for parameter 'runtimeConfig'
e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (125, 79): Unresolved reference: runtimeConfig
e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (127, 57): Function invocation 'awsCredentialTypesTestUtil(...)' expected
e: /home/build/workspace/smithy-rs/aws/sdk-codegen/src/test/kotlin/software/amazon/smithy/rustsdk/EndpointsCredentialsTest.kt: (127, 57): No value passed for parameter 'runtimeConfig'

The aws-config crate's tests fail when doing cargo test --all-features.
Looks like aws/sdk/integration-tests/webassembly/tests/test.rs needs rustfmt run on it.
Something weird going on with the unused dependencies check on aws-config:

unused dependencies:
`aws-config v0.54.1 (/home/build/workspace/aws-sdk-smoketest/sdk/aws-config)`
└─── dependencies
     └─── "aws-sdk-sso"
Note: These dependencies might be used by other targets.
      To find dependencies that are not used by any target, enable `--all-targets`.
Note: They might be false-positive.
      For example, `cargo-udeps` cannot detect usage of crates that are only used in doc-tests.
      To ignore some dependencies, write `package.metadata.cargo-udeps.ignore` in Cargo.toml.

jdisanti · 2023-01-27T18:37:13Z

Note to work on aws-config locally, you may need to run ./gradlew aws:sdk:assemble so that the STS/SSO crates it depends on actually exist.

Co-authored-by: John DiSanti <johndisanti@gmail.com>

eduardomourar · 2023-03-24T09:26:56Z

@jdisanti and @rcoh , could we get this approved and merged, please?

jdisanti

This looks great! I'm going to work on merging it today and wiring it up to CI.

aws/sdk/integration-tests/webassembly/.vscode/settings.json

jdisanti · 2023-03-24T18:51:00Z

I added a compilation check for wasm32-unknown-unknown and wasm32-wasi for aws-config to CI. I'm not confident we won't break things until we can run tests, but the tests don't currently compile. I've filed #2499 to track fixing that.

I'm willing to merge this once the CI portion passes to at least keep your progress, but there is a chance we may break things in future releases without this visibility in CI.

Thank you for the work here!

tools/ci-scripts/check-aws-config

jdisanti · 2023-03-24T23:29:18Z

Looks like it passed CI. Adding it to the merge queue.

* feat(aws-config): define default feature for sso * chore(aws): disable unused features * chore(rust-runtime): disable unused features * chore: add missing tokio fs feature * chore: disable spawn blocking on wasm * chore: fix sso feature * chore: allow unused variables when not sso * chore: remove unnecessary dependency filter for wasm * chore: add integration test for wasm * chore: ensure test-util feature is applied to dev dependencies * chore: disable retry config Co-authored-by: John DiSanti <johndisanti@gmail.com> * chore: simplify features as suggested * chore: no default features for aws types * chore: rename credentials-sso feature * Revert "chore: simplify features as suggested" This reverts commit d8fcf49. * chore: set right name in default features * chore: create smithy client test util in runtime types * Update aws/rust-runtime/aws-types/Cargo.toml Co-authored-by: John DiSanti <johndisanti@gmail.com> * Update aws/rust-runtime/aws-inlineable/Cargo.toml Co-authored-by: John DiSanti <johndisanti@gmail.com> * Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs Co-authored-by: John DiSanti <johndisanti@gmail.com> * Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs Co-authored-by: John DiSanti <johndisanti@gmail.com> * Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs Co-authored-by: John DiSanti <johndisanti@gmail.com> * Update aws/rust-runtime/aws-config/src/lib.rs Co-authored-by: John DiSanti <johndisanti@gmail.com> * chore: use hardcoded credentials feature * chore: make wasm example for s3 instead * chore: fix formatting * chore: fix kotlin formatting * chore: fix kotlin unit tests * chore: use timeout config from smithy types * chore: move tests into main file * chore: add vscode setttings to target wasi by default * chore: fix test-util feature for smithy client * chore: separate adapter into own module * chore: fix test with no default features * Fix typo * Update changelog * Check compilation of `aws-config` against `wasm32` in CI * Fix Dockerfile issue and use correct cargo-wasi command * Small Dockerfile fix * Add missing `tar` binary to Docker image --------- Co-authored-by: Eduardo Rodrigues <eduardomourar@users.noreply.github.com> Co-authored-by: John DiSanti <johndisanti@gmail.com> Co-authored-by: John DiSanti <jdisanti@amazon.com> Co-authored-by: Russell Cohen <rcoh@amazon.com>

eduardomourar added 10 commits January 26, 2023 01:23

feat(aws-config): define default feature for sso

ffa9923

chore(aws): disable unused features

91a8dcc

chore(rust-runtime): disable unused features

1fbd97b

chore: add missing tokio fs feature

06a62b7

chore: disable spawn blocking on wasm

a725e0c

chore: fix sso feature

cae60ef

chore: allow unused variables when not sso

ef00e01

chore: remove unnecessary dependency filter for wasm

497c4b9

chore: add integration test for wasm

59791df

chore: ensure test-util feature is applied to dev dependencies

7478507

eduardomourar requested review from a team as code owners January 26, 2023 15:04

eduardomourar requested review from LukeMathWalker and 82marbag January 26, 2023 15:04

jdisanti reviewed Jan 26, 2023

View reviewed changes

eduardomourar and others added 7 commits January 26, 2023 20:54

chore: disable retry config

56b7b33

Co-authored-by: John DiSanti <johndisanti@gmail.com>

chore: simplify features as suggested

d8fcf49

chore: no default features for aws types

86a5689

chore: rename credentials-sso feature

c0d61ed

Revert "chore: simplify features as suggested"

a5b6316

This reverts commit d8fcf49.

chore: set right name in default features

f2d47b5

chore: create smithy client test util in runtime types

df4ab9e

jdisanti reviewed Jan 27, 2023

View reviewed changes

eduardomourar and others added 5 commits January 27, 2023 21:09

Update aws/rust-runtime/aws-types/Cargo.toml

154342f

Co-authored-by: John DiSanti <johndisanti@gmail.com>

Update aws/rust-runtime/aws-inlineable/Cargo.toml

937022c

Co-authored-by: John DiSanti <johndisanti@gmail.com>

Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs

08df0df

Co-authored-by: John DiSanti <johndisanti@gmail.com>

Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs

98d336e

Co-authored-by: John DiSanti <johndisanti@gmail.com>

Update aws/rust-runtime/aws-config/src/profile/credentials/exec.rs

45cae40

Co-authored-by: John DiSanti <johndisanti@gmail.com>

Merge branch 'main' into feat/support-wasm

d353ac3

eduardomourar force-pushed the feat/support-wasm branch from d41ca4d to 4e1487b Compare March 23, 2023 09:30

chore: fix test with no default features

0beea87

eduardomourar force-pushed the feat/support-wasm branch from 4e1487b to 0beea87 Compare March 24, 2023 01:41

Merge branch 'main' into feat/support-wasm

0d52a87

jdisanti approved these changes Mar 24, 2023

View reviewed changes

aws/sdk/integration-tests/webassembly/.vscode/settings.json Outdated Show resolved Hide resolved

jdisanti and others added 3 commits March 24, 2023 10:43

Fix typo

ffe83ea

Update changelog

0ec8283

Check compilation of aws-config against wasm32 in CI

01f106c

Merge remote-tracking branch 'origin/main' into support-wasm

bc2fe6c

eduardomourar commented Mar 24, 2023

View reviewed changes

tools/ci-scripts/check-aws-config Outdated Show resolved Hide resolved

jdisanti added 4 commits March 24, 2023 12:16

Fix Dockerfile issue and use correct cargo-wasi command

ba00b09

Small Dockerfile fix

04c7094

Add missing tar binary to Docker image

7e90fb3

Merge branch 'main' into feat/support-wasm

4bfd4e4

jdisanti enabled auto-merge March 24, 2023 23:29

jdisanti added this pull request to the merge queue Mar 25, 2023

github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Mar 25, 2023

Merge branch 'main' into feat/support-wasm

5e63eec

rcoh enabled auto-merge March 29, 2023 18:56

rcoh added this pull request to the merge queue Mar 29, 2023

Merged via the queue into smithy-lang:main with commit 65058d9 Mar 29, 2023

eduardomourar deleted the feat/support-wasm branch March 29, 2023 20:35

eduardomourar mentioned this pull request Mar 31, 2023

feat: make outgoing request based on wasi-http proposal #2520

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support compiling to webassembly #2254

feat: support compiling to webassembly #2254

eduardomourar commented Jan 26, 2023 •

edited by jdisanti

Loading

jdisanti left a comment

jdisanti Jan 26, 2023

eduardomourar Jan 26, 2023 •

edited

Loading

jdisanti Jan 26, 2023

eduardomourar Jan 28, 2023

jdisanti left a comment

jdisanti commented Jan 27, 2023

jdisanti commented Jan 27, 2023

eduardomourar commented Mar 24, 2023

jdisanti left a comment

jdisanti commented Mar 24, 2023

jdisanti commented Mar 24, 2023

feat: support compiling to webassembly #2254

feat: support compiling to webassembly #2254

Conversation

eduardomourar commented Jan 26, 2023 • edited by jdisanti Loading

Motivation and Context

Description

Testing

Checklist

jdisanti left a comment

Choose a reason for hiding this comment

jdisanti Jan 26, 2023

Choose a reason for hiding this comment

eduardomourar Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

jdisanti Jan 26, 2023

Choose a reason for hiding this comment

eduardomourar Jan 28, 2023

Choose a reason for hiding this comment

jdisanti left a comment

Choose a reason for hiding this comment

jdisanti commented Jan 27, 2023

jdisanti commented Jan 27, 2023

eduardomourar commented Mar 24, 2023

jdisanti left a comment

Choose a reason for hiding this comment

jdisanti commented Mar 24, 2023

jdisanti commented Mar 24, 2023

eduardomourar commented Jan 26, 2023 •

edited by jdisanti

Loading

eduardomourar Jan 26, 2023 •

edited

Loading