Add test support for emscripten targets

[malbarbo]

Also fix some definitions to make the tests pass for asmjs-unknown-emscripten. The build fails for wasm-unknown-emscripten with an assert error.

[malbarbo]

I'm waiting the build fails for wasm-unknown-emscripten so I can open a issue and link the error.

[malbarbo]

This failure is expected, I will disable wasm test an report the assert faillure to emscripten.

Edit: issue report.

[malbarbo]

Sadly, this changes will limit the file size that can be addressed to 2GB, but this is emscripten issue... It seems that all *64_t types and *64 functions related with file system are alias to they "normal" version...

[alexcrichton]

This looks awesome, thanks! Would it be possible to avoid the large whitelist of functions in libc-test/build.rs? could those functions be pushed down a layer perhaps?

[malbarbo]

could those functions be pushed down a layer perhaps?

I think this will make libstd (and others crates) more difficult to maintain. Take for example the fork function, it is not available on emscripten (it is in the whitelist) but it is used in libstd. An user will only get an error if he tries to use a function that calls fork, but he can use all the others functions. This is the same approach used in emscripten, that also defines the prototype of some functions but does not provides an implementation.

I don't know how many of the whitelisted functions are used in libstd, but I think removing them from libc would make it impossible to make libstd compile for emscripten.

[alexcrichton]

Oh ok in that case mind clarifying something? So does fork exist in emscripten? If it does then how come tests are failing? If not, how does libstd link correctly?

[malbarbo]

So, now I'm confused. It toke me some time to create the whitelist, and now I think I need to review it. fork and others functions can be removed and tests still pass, but the majority of the functions are still there.

Some functions, for example aio_*, are compiled ok when used in all.c, but fails when linking libc-test. I think that all.c is only compiled and not linked, and aio_* prototypes are defined in emscripten headers, so it works. But when libc-test are linked, the symbols are not found and the linking fails.

Now I don't know what is the best approach here. If we remove these definitions, a library that uses them cannot be compiled, and must cfg_out functions that use these definitions. On the other hand, if we leave it, the library can be compiled but will fail to link (in case an unimplemented function is necessary, but I think it will link ok if no such a function is used).

[lilianmoraru]

emsdk install sdk-1.37.14-64bit doesn't work yet(of course, it was tagged ~30 minutes ago).
I'll be back here to continue spamming the issue when it will be available 😄

[lilianmoraru]

The SDKs are built and stored on https://s3.amazonaws.com/mozilla-games/emscripten/.
Searched for mozilla-games but didn't find much. No idea who does the builds...

[malbarbo]

Update the emscripten sdk, let's see how travis goes. I didn't work on the whitelist issue.

[lilianmoraru]

The wasm build passed.
@malbarbo, this is fantastic!!
Thanks for moving this forward.

[alexcrichton]

Awesome, thanks @malbarbo!

My only hesitation is the large whitelist of functions to ignore in libc-test/build.rs. Does libstd use these functions? If so, how come we can't check them? If not, can they be omitted on emscripten? (presumably they don't exist)

[bors]

☔ The latest upstream changes (presumably #649) made this pull request unmergeable. Please resolve the merge conflicts.

[lilianmoraru]

@alexcrichton What needs to be done(finished-up) here to push this forward?
Proper emscripten and wasm support in Rust is basically blocked by this and it would be great to have this in.

[alexcrichton]

@lilianmoraru I'd like to understand more about the whitelist of functions here that aren't tested, but other than that it should just be a rebase away from landing

[lilianmoraru]

Note to self for follow-up(I've lost track of what needs to follow, I try to remember some of the points):

• Add emscripten support danburkert/fs2-rs#19 -> retry tantivy compilation
• no release yet for Fix asm2wasm wasm-only i64 switches with a large (>32 bit) offset #1109 WebAssembly/binaryen#1111 ?( float tests fails on wasm32-unknown-emscripten rust#42630 )
• no release yet for fix f32 frem in asm2wasm #1105 WebAssembly/binaryen#1106 ?( f32::rem fails on wasm32-unknown-emscripten rust#42629 )
• emscripten 1.37.14+ update in Rust(+CI updates?) -> libc update in Rust -> wasm needs to be enabled back Enable testing wasm32-unknown-emscripten on travis rust#42646 -> rustup, rust-buildbot changes? -> update/retest wasm-validator errors when trying to compile rust-brotli to wasm32 rust#42518

[pepyakin]

@lilianmoraru

• no release yet for Fix asm2wasm wasm-only i64 switches with a large (>32 bit) offset #1109 WebAssembly/binaryen#1111 ?( float tests fails on wasm32-unknown-emscripten rust#42630 )
• no release yet for fix f32 frem in asm2wasm #1105 WebAssembly/binaryen#1106 ?( f32::rem fails on wasm32-unknown-emscripten rust#42629 )

I might be terribly wrong, but isn't binaryen builds from the master?

https://github.com/rust-lang/rust/blob/master/src/ci/docker/scripts/emscripten-wasm.sh#L31

curl "https://storage.googleapis.com/wasm-llvm/builds/linux/lkgr.json" \
  | jq '.repositories.binaryen.hash'

at time of this writing, points on 21d06ae witch seems to be master, meaning that mentioned fixes there already.

had to force-push..

It doesn't mean that you did something wrong.

[alexcrichton]

Thanks for taking this on @lilianmoraru! Other than the .travis.yml weirdness, was there anything I could help out with?

[lilianmoraru]

@alexcrichton Nah. I pushed my changes to @malbarbo's branch( malbarbo#1 ), now I'll just wait.
I do worry that a rebase is not enough because I saw some new code that should probably be under the cfg(target_os = "emscripten"), cfg(not(target_os = "emscripten")).

[alexcrichton]

I've rebased this in #742 with what I had in mind, basically just copying around a lot more things to avoid many changes to libc-test and ensure everything is validated on emscripten.

[alexcrichton]

Thanks again though for the patch @malbarbo!

[lilianmoraru]

Personally, I would of liked to preserve the history of @malbarbo's work :(

[alexcrichton]

Egad sorry! I've merged it all into the other PR

[malbarbo]

Thanks @alexcrichton for finishing this! I was worried that removing the functions white list would be too disturbing. Moving emscripten up in the hierarchy was the right choice.

@lilianmoraru sorry for not replaying to you messages..., I've been very busy lately. And thank you for helping moving this forward.