Where should espup install the different toolchains?

[SergioGasquez]

The goal of this discussion is to reevaluate the way we distribute the different components required for both approaches (std and no_std).

At the moment, here is what and where we install it:

• espup install will deploy:
  • Xtensa Rust toolchain: $RUSTUP_HOME/toolchain/esp
  • LLVM (if using --profile-minimal only libs) : $HOME/.espressif/tools/
  • GCC for all the targets: $HOME/.espressif/tools/
• espup install -e/--esp-idf-version <VERSION> will deploy:
  • Xtensa Rust toolchain: $RUSTUP_HOME/toolchain/esp
  • LLVM (if using --profile-minimal only libs): $HOME/.espressif/tools/
  • ESP-IDF: $HOME/.espressif/esp-idf-<hash>
    • It also install GCC under: $HOME/.espressif/tools/
  • ldproxy: $CARGO_HOME/bin/

Here are a few suggestions that have been proposed during the 2023-01-19 Community Meeting:

• @ivmarkov: Install the required toolchains under $RUSTUP_HOME/toolchain/esp
  • This would avoid confusion and it will separate "esp-rs" toolchains from "esp-idf" toolchains
  • This would be a breaking change for espup/rust-build scripts
• @ivmarkov: Include the required toolchains (or some of those, like esp-clang) in the rust-build artefacts
• @MabezDev : Try to install esp-clang via ESP-IDF (as another tool)
  • We would need a way to set LIBCLANG_PATH

Also, it's important to mention, that in the future, we would like espup to install standalone products like ESP-IDF

Any input or suggestion is more than welcome! @georgik @MabezDev @jessebraham @bjoernQ @JurajSadel @ivmarkov @igrr

[bjoernQ]

Maybe another option: $HOME/.espressif/rust-tools - this way it's clearer that it's for Rust not for esp-idf but it also shows that it's from some Espresssif tool?

If not option 2 from above sounds good to me

[ivmarkov]

Also, it's important to mention, that in the future, we would like espup to install standalone products like ESP-IDF

I'm a bit struggling with the notion of "installing" ESP IDF which is just a GitHub repo, and hence - what is the need to use a dedicated tool like espup to install ESP IDF.
My understanding how ESP IDF works today is as follows:

• User does git clone https://github.com/espressif/esp-idf (or a variation, if a different branch or tag is necessary)
  • My understanding is that espup currently automates this step, hence my questions about the usefulness of this "installation"
• User runs the install.sh/ps1/bat script to install tooling (or just uses python to install tooling, as esp-idf-sys - using embuild - does automatically under the hood)
  • Or is this what you actually want to replace with espup? If yes, would that replace the usage of python for installation, or just the install.* script? If the latter, perhaps you should just look into using embuild in espup, as embuild already directly uses python for installation, and does not call the install.* scripts at all

W.r.t. how Clang should be installed for STD builds... given that the current STD build philosophy is that esp-idf-sys drives the installation of ALL tooling except Rust itself and clang (clang for xtensa targets only), I would prefer to minimize this even further:

• Clang for xtensa targets can be viewed as something that is delivered together with Rust for xtensa in a hidden directory. As in $RUSTUP_HOME/toolchain/esp. Note also that we only use and need the libclang DLL/so. Nothing else from "LLVM"
• Clang for xtensa targets can be installed by esp-idf-sys itself. Clang for riscv can be assumed to be present on the developer machine, just like Rust is assumed to be present.

My bottom line is that having espup creating an ~/.espressif folder is bringing extra confusion, because this is NOT the default folder that esp-idf-sys uses to download ESP IDF itself, and to install the necessary tooling. However folks see the ~/.espressif folder (potentially with the ESP IDF inside) and assume that this is stuff that esp-idf-sys needs and uses, while this is just not the case. The only single thing esp-idf-sys needs and uses is the libclang DLL/so.

Perhaps most important is (and sorry for finishing with that rather than starting with that!) is where we would like to position espup in the future. As in, what is the roadmap for espup itself. Some possibilities off the top of my head:

• (Minimal) espup is the tool to (currently) download Rust (and potentially clang-the-dll-only) xtensa toolchain. Might go away in future. If that's our thinking, I'm strongly against espup generating any artefacts outside the $RUSTUP_HOME/toolchain/esp dir.
• (Semi-minimal) espup is the tool to download Rust (and potentially clang-the-dll-only) xtensa toolchain as well as riscv toolchain which is as stable as it gets, but with nightly features enabled. While the need to use nightly features is diminishing over time and will likely get to 0 soon for no_std, STD will continue to need one nightly feature for the foreseeable future - cargo -Zbuild-std which nobody wants to stabilize at the moment. If that's our thinking, I'm again strongly against espup generating any artefacts outside the $RUSTUP_HOME/toolchain/esp dir.
• (Maximum) espup has aspirations to retire shell scripts from inside ESP IDF and to have a story/use case, which is applicable not just to Rust, but also to C development with ESP IDF. If that's the case - fine - but can you then elaborate on that use case?

[ivmarkov]

One more thing I've missed to comment: OK, espup also installs ldproxy. Perhaps the argument w.r.t. "installing" ESP IDF is applicable here as well: how useful is that, when compared to user issuing cargo install ldproxy?

... and, if we go that route, shall espup also automate the installation of espflash as well? And the wokwi server? and the cargo template? You know, other examples of tooling we currently bring up by just using cargo install.

[MabezDev]

I think the main reason espup currently installs ldproxy is that it's used in the xtensa-toolchain action, and using cargo install on github runners is very very slow, it's better to download the latest artefact. Maybe this should be installed via the action instead though.

Clang for xtensa targets can be viewed as something that is delivered together with Rust for xtensa in a hidden directory. As in $RUSTUP_HOME/toolchain/esp. Note also that we only use and need the libclang DLL/so. Nothing else from "LLVM"

There is already some precedence for distributing DLL/so with rustup toolchains, running ls .rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib yields a list of LLVM-related shared objects. I don't see a reason why we can't throw in our so here too.

[SergioGasquez]

I think the main reason espup currently installs ldproxy is that it's used in the xtensa-toolchain action, and using cargo install on github runners is very very slow, it's better to download the latest artefact. Maybe this should be installed via the action instead though.

Not really, espup does not use gh artifacts to install any crate (at the moment), rust-build script did this. This feature was requested by @georgik to be able to install several crates while installing the environment. In the xtensa-toolchain action, is the action itself that takes care of installing ldproxy

[SergioGasquez]

I'm a bit struggling with the notion of "installing" ESP IDF which is just a GitHub repo, and hence - what is the need to use a dedicated tool like espup to install ESP IDF. My understanding how ESP IDF works today is as follows:

At the moment, espup uses embuild to "install" ESP-IDF.

[SergioGasquez]

And yeah, I have not covered the "no_std-using-Espressif-GCC-as-linker" use case, but Scott mentioned this is anyway going away, so perhaps we should not even consider it in the discussion?

@MabezDev do you think we could have a rough estimation of when we could drop GCC as a linker? I'm just interested in the order of magnitude of the answer

[MabezDev]

I have a question instead of an answer (sorry :D), and it somewhat ties in with what I think @ivmarkov is getting at. What do we think the future of espup is? In the context of, LLVM upstreaming is a success and Xtensa enabled Clang/LLVM/Rust lands in peoples package managers. Do we still want to use espup for the installation of other things? ESP-IDF sources are mentioned, but I don't think that is particularly valuable at the moment from my usage, I either use cloned sources locally, or let embuild get the sources for me.

Regarding #95, this sounds like something ESP-IDF tooling should be doing, IMO espup is Rust tooling. There is no reason why we can't write something that does that in Rust, but does it belong in espup?

[SergioGasquez]

I have a question instead of an answer (sorry :D), and it somewhat ties in with what I think @ivmarkov is getting at. What do we think the future of espup is? In the context of, LLVM upstreaming is a success and Xtensa enabled Clang/LLVM/Rust lands in peoples package managers. Do we still want to use espup for the installation of other things? ESP-IDF sources are mentioned, but I don't think that is particularly valuable at the moment from my usage, I either use cloned sources locally, or let embuild get the sources for me.

Regarding #95, this sounds like something ESP-IDF tooling should be doing, IMO espup is Rust tooling. There is no reason why we can't write something that does that in Rust, but does it belong in espup?

Completely agree with both of you. We should start by defining where we would like to see espup in the future, I don't think I can answer this, tbh. As a personal preference, I would go for something like what @ivmarkov defined as minimal or semi-minimal. I.e: espup should install the minimum set of toolchains that would get your environment running for both approaches (it would rely on esp-idf-sys to take care of esp-idf in std apps). And, once/if LLVM and Rust changes are upstreamed, we could probably drop espup. But, again that is just my opinion (and is not a hard opinion) and I don't think I have nor the knowledge or the experience to decide what we should do with espup!

[ivmarkov]

I.e: espup should install the minimum set of toolchains that would get your environment running for both approaches (it would rely on esp-idf-sys to take care of esp-idf in std apps). And, once/if LLVM and Rust changes are upstreamed, we could probably drop espup. But, again that is just my opinion (and is not a hard opinion) and I don't think I have nor the knowledge or the experience to decide what we should do with espup!

Nobody has the knowledge so we have to collectively arrive at it! :)

Re: toolchains: important to clarify that - putting aside Rust and Clang - all other things (namely, GCC, cmake and maybe others) are in fact not static. This means to say that the toolchains are actually dependent on the particular ESP IDF version that user has cloned. As in - the tools.json file that is inside <esp-idf>/tools directory - is specifying what toolchains (as in toolchain versions and names) that particular ESP IDF version needs. This is another reason why the esp-idf-sys is downloading the toolchains during the build (if they are not downloaded already).
This also means that if the user switches to another ESP IDF branch, whatever espup has downloaded for the previous ESP IDF branch in terms of toolchains might not be applicable, and new toolchain versions might have to be downloaded. I guess I'm just putting another argument why it is not worth it for espup to extend its responsibilities.

[jessebraham]

I personally think that once LLVM and Rust have been upstreamed that espup no longer serves any purpose. There are existing tools for installing Rust and users can cargo install anything else they need.

Especially from the perspective of a bare-metal user, there's not really anything else I need.

[georgik]

There is also another reason for developing espup. Let me explain different vectors and a different set of user problems.

There is a Windows Installer for ESP-IDF: https://github.com/espressif/idf-installer

The installer's primary goal is to provide a simple tool to deploy the development stack on the user's machine. Windows is a slightly more complex challenge than one would expect. The installer prepares the environment and mutates it to the state where all essential requirements are fulfilled.

The problem with the current installer is technological. It's based on ancient Inno Setup using Pascal for scripting, making it very hard to extend or add complex logic. It's impossible to achieve a similar feature set as we can find in Visual Studio in the smaller, which is pretty versatile.

To resolve this problem, we created an idf-env project in Rust to test whether we can develop a binary that can run across multiple OSes and provides the user with a working ESP-IDF environment. We were replacing step-by-step functions in idf-installer by calling idf-env CLI. This proved to be a working solution. There was also an initial deployment of Rust tooling.

Based on this positive experience, we decided to create a new tool known as espup, which would focus on Rust in the first place and will be designed better than the original idf-env. Ideally, the tool should phase out more parts of the idf-installer or even replace it entirely.

During this research, we found out that some functions are more crucial to Rust users to get the environment running. There are different solutions to the mutation of the user's environment, so some functions are implemented in another way (often better) in espup.

Since espup can also deploy ESP-IDF using embuild, it makes the tool a candidate for replacing this functionality in idf-installer or replacing it entirely if GUI is provided.

The good question is whether we should put the GUI part into espup since it introduces licensing issues. The original prototype of GUI shows that it's possible to create small binary with multi-platform GUI. The question is whether we should keep it there or move it to a separate project as initially intended.

Basically, what is needed from the community/industry is to have some installer with GUI. Right now, it's idf-installer. It would be best to have Rust based installer. The installer does not need to be espup, yet it should reuse maximum functionality from espup by calling CLI or functions in the crate. Having a good visible interface of espup, either CLI/functions is critical because people are often developing their deployment tweaks which we need to be able to foresee in the beginning.

One vital part of idf-installer is the ability to operate entirely in Offline mode, where all dependencies are bundled into a single binary. This is extremely important for users behind strict firewalls or without an internet connection. Until now, we could not find a solution in Rust that allows so simple and fast bundling of all binaries and files into a single binary like Inno Setup is doing in idf-installer. Until this point is solved, we will still rely on the idf-installer part to do the bunding part, but most system mutation functions should move to Rust.

[SergioGasquez]

Maybe we could go for the following approach (its basically a mix of the last comments):

• espup should be a minimal installer that "dies" when LLVM and Xtensa Rust get upstreamed.
  • It will only install Xtensa Rust, LLVM and GCC (until LLD is ready).
  • It would require rustup, git and python (only for std) installed.
• idf-evn or idf-installer can use espup and add other fancy stuff to the installation (install dependencies, extra crates, esp-idf...)

[Nukesor]

I think in the end it boils down to two separate use-cases which might or might not be implemented with espup:
(1) The "extended" idf-installer scope, that I tried to define above. This has - probably - the audience of users who just start with ESP IDF and are struggling with basic stuff
(2) The "just-install-Espressif-Rust-and-possibly-Clang-dll-and-keep-out-of-the-way" scope, we are discussing so far.

Sorry for dropping in late, but I just wanted to add another use-case.
There are a few teams out there, one of mine included, that consist of developers for different technologies. Some do work on a mobile app, some are web developers, some do database stuff, some develop firmware.
But all of them need to easily build all parts of the project at all times, at best without any knowledge about any other part of the project. This is important when integration testing all parts of a project.

Hence, I don't really see a problem with a project that's about preparing a development environment, actually taking care of the full required setup of a project. I personally don't want to write my own scripts that clone and keep the esp-idf repository on the correct tag. I was actually quite fond that espup did that for us.

[ivmarkov]

If you are using / depending on esp-idf-sys, it is the esp-idf-sys build itself which clones the ESP IDF repo with a specific tag/branch/commit hash. It is NOT the espup tool. espup installs ESP IDF "just in case".

In fact, I'm requesting espup no longer to fiddle with ESP IDF precisely because it confuses folks like you as to what ESP IDF directory is used, and in general which tool is responsible for what.

[ivmarkov]

Now, if you have a complex C/C++ codebase with Rust intermingled, then sure - espup taking ownership of some of the stuff which is currently done by a bunch of shell/Python scripts living inside the ESP IDF repo itself might have some benefits. Plus the scope of the existing Windows ESP IDF installer, which is for "newbies in ESP IDF C".

But the above is not really anymore the small scope of a "tool helping Rust folks install the xtensa-specific toolchains" but something much bigger, which should involve the C ESP IDF community itself. I'm neutral to that scope, personally.

[SergioGasquez]

Could you elaborate on your use case @Nukesor? As @ivmarkov mentioned, unless you have a complex codebase, esp-idf-sys (a crate that all our std projects require) will take care of downloading and installing esp-idf, so you won't have to write any script that does that.

[SergioGasquez]

Please, vote on this comment which approach you think we should follow with espup: for minimal approach use the 👎 , for extended approach use 👍 .

Look at the discussion comments for more insight into both approaches.

I will consider the voting closed on February 6th at 10 AM (CET).

[SergioGasquez]

For clarification, in summary, we can define the two approaches as:

• minimal: espup dies when Xtensa changes are upstreamed and only installs the minimal required stuff
• extended: espup is able to set up Rust environments, ESP-IDF, other crates ...

Once the voting is concluded we can define the details of the selected approach.

[SergioGasquez]

The voting has finished! Results:

• 7 votes for minimal (:-1:)
• 0 votes for extended (:+1:)

We can now proceed to discuss minor details of the minimal implementation!

[SergioGasquez]

Now that we've agreed on the future of espup we can go back to the initial question of where toolchains should be installed, and detail what and what not should be installed with espup. Here are some of my thoughts/questions:

• rustup
  • I think the user should already have Rust installed and we should not carry the burden of "duplicating what the Rust team is doing (even though in Windows, it requires installing some SDKs and dependencies that are not automatically handled by rustup)
  • I would add Rust as dependency and link to the Rust installation page
  • We already had some discussion on this topic, see Windows: Install MIGW dependency for x86_64-pc-windows-gnu #10
• python and `git
  • They shouldn't be requirements of espup, they are only required on std.
• Shall we allow installing multiple Xtensa Rust toolchains?
  • This would allow having multiple versions installed at the same time.
  • This would require an argument to specify the toolchain name.
• LLVM/Clang:
  • We can install it under $RUSTUP_HOME/toolchain/esp
    • Since we want to be minimalistic, I would only install libs. But, shall we allow the user to install the whole LLVM with an arg?
• GCC (temporary):
  • Until LLD is ready we should install GCC to use it as a linker in no_std applications
  • We can install it under $RUSTUP_HOME/toolchain/esp
    • If a user wants to build an std app it will end with two versions of GCC toolchains in the PATH: one provided by espup and another one provided by ESP-IDF (installed by esp-idf-sys), they might be different or the same, but in any case, ESP-IDF versions would always be correct but espup versions may not be.
• [Windows]: Shall we inject the environment variables into the system and not require users to source the export-file every time?
  • I wouldn't do things in the name of the user, and I would only generate the export file so they have the chance to decide what to do with it.
  • I think injecting variables is somewhat common in Windows (?) that's why I'm raising the question.

[SergioGasquez]

If a user wants to build an std app it will end with two versions of GCC toolchains in the PATH: one provided by espup and another one provided by ESP-IDF (installed by esp-idf-sys), they might be different or the same, but in any case, ESP-IDF versions would always be correct but espup versions may not be.

One possible solution would be to have a --std flag that does not install GCC

[bjoernQ]

Sounds good to me

Shall we allow installing multiple Xtensa Rust toolchains?

I guess that would be a nice feature and shouldn't make things too complex? I have a few Xtensa Rust toolchains installed on my system but on the other hand it's probably not too common for most users (and also, I don't use espup myself - so no need to add this feature just for me)

[Windows]: Shall we inject the environment variables into the system and not require users to source the export-file every time?

I'd say yes since it's a common thing on Windows - but maybe tell the user what espup did to the system path (most users won't read that probably but it's better to do it anyways IMHO)

[jessebraham]

rustup

I agree, let's defer this to the existing tool. As you said, the user should likely already have this installed, if they don't since we're using miette for errors we can even print a message and a link telling people to install it if it's missing, easy peasy.

python and git

Yup, don't want these being touched on my system please. Python versions can be a can of worms depending on the platform.

Shall we allow installing multiple Xtensa Rust toolchains?

Probably a good idea.

LLVM/Clang

As @ivmarkov has pointed out (and I agree) we should probably avoid the ~/.espressif/tools directory; we can put them in ~/.rustup/toolchains/esp/{bin,lib}/ or something.

GCC

See above.

[MabezDev]

Other bits have been answered, but GCC, given that users will need to add it to their path from $RUSTUP_HOME/toolchain/esp, what if we just make them install it manually anyways? How does GCC from espup currently get into $PATH?

[SergioGasquez]

How does GCC from espup currently get into $PATH?

The generated export file adds them to the $PATH, so sourcing the export file updates it.

what if we just make them install it manually anyways?

GCC manual installation is a bit tedious so I don't really like this approach. It requires downloading, uncompressing the file and then updating the $PATH pointing to wherever they uncompressed it.

[SergioGasquez]

espup has now been refactored, version 0.3.0 installs only the required toolchains and does not allow installing any extra components (crates, esp-idf...) . Here is the resulting behavior:

• espup install installs under $RUSTUP_HOME/toolchain/<toolchain_name>:
  • Xtensa Rust (if any Xtensa target is provided)
  • Nightly Rust (if any RISC-V target is provided)
  • Xtensa LLVM clang
  • GCC for the provided targets (to be served as linker in no_std apps)
    • This would be removed once LLD can be used
    • There is a --std flag that skips this step
• espup update will update the Xtensa rust under $RUSTUP_HOME/toolchain/<toolchain_name>:
  • It will use latest if no -v/toolchain_version is provided
  • If the semver of the version trying to install is the same as the version installed it won't install it but it will print a warning message (as we use an extended version of semver and they might be differences)
• espup uninstall will delete $RUSTUP_HOME/toolchain/<toolchain_name>

Detailed list of changes in: #182
Any feedback or suggestion is more than welcomed!