cpu/esp32: Upgrade to ESP-IDF v4.4

[gschorcht]

Contribution description

This PR provides an upgrade of the ESP32 port to ESP-IDF v4.4 with following changes:

• package esp_sdk is upgraded to newest ESP-IDF release v4.4
• ESP32 vendor code is completely removed from RIOT repository
• required ESP32 code is compiled directly from ESP-IDF sources provided by package esp32_sdk
• makefiles are almost ready to be used with other ESP32 variants (ESP32S, ESP32S2, ESP32S3, ESP32C3)

To ensure that the RIOT port for ESP32 is still working and to keep the changes as small as possible, the implementations of the peripherals are not changed for the moment. They still use the hardware directly and therefore can only be used for the ESP32.

The next steps will be to change step by step the peripheral implementations to use the HAL provided by the ESP-IDF (PR #17841) for all ESP32 variants. This will allow to support these variants (PR #17842, PR #17843) without having to re-implement the peripheral implementations for each ESP32 variant again and again.

Testing procedure

Can't be compiled nor tested in CI since the PR requires a new toolchain version. This new toolchain version includes

• xtensa-esp32-gcc 8.4.0 compiled for RIOT with newlib-nano 3.3.0
• openocd_esp32
• qemu_esp32

I will provide it on GitHub in next days.

Issues/PRs references

[asterix11]

Regarding the ESP32C3 I would be interested in how this is working internally, as RIOT is structured in a way that we have a esp32 "cpu"-folder even though in most esp32 variants the actual CPU is xtensa except for the ESP32C3, which uses a RISC-V processor.

What I am most interested in here is that the ESP32C3 is an official supported Rust target called riscv32imc-esp-espidf even supporting std lib. See: https://doc.rust-lang.org/nightly/rustc/platform-support.html

But what is confusing me is the fact, that in RIOT CPUs say, that they support Rust by setting the rust_target feature in Makefile.features flag. But this obviously can not be enabled for all ESP32 variants, but only the riscv version right now. How would that work?

[gschorcht]

Regarding the ESP32C3 I would be interested in how this is working internally, as RIOT is structured in a way that we have a esp32 "cpu"-folder even though in most esp32 variants the actual CPU is xtensa except for the ESP32C3, which uses a RISC-V processor.

What I am most interested in here is that the ESP32C3 is an official supported Rust target called riscv32imc-esp-espidf even supporting std lib. See: https://doc.rust-lang.org/nightly/rustc/platform-support.html

But what is confusing me is the fact, that in RIOT CPUs say, that they support Rust by setting the rust_target feature in Makefile.features flag. But this obviously can not be enabled for all ESP32 variants, but only the riscv version right now. How would that work?

I am currently working on the EPS32C3 support, also to see how best to structure the cpu/esp32 folder. But what I can say now is that we probably need to split the CPU structure a bit, even if the makefiles in this PR already handle the different CPUs with different architectures. It is not yet clear how, but probably something like:

folder	includes	contains
cpu/esp_common	-	shared code for all ESPs including ESP8266
cpu/esp32_common	esp_common	peripheral drivers for all ESP32 variants
cpu/esp32_riscv	cpu/esp32_common	RISC-V specific code like thread_arch and irq_arch
cpu/esp32_xtensa	cpu/esp32_common	Xtensa specific code like thread_arch and irq_arch
cpu/esp32	cpu/esp32_xtensa	CPU/peripheral configuration for ESP32
cpu/esp32s2	cpu/esp32_xtensa	CPU/peripheral configuration for ESP32S2
cpu/esp32s3	cpu/esp32_xtensa	CPU/peripheral configuration ESP32S3
cpu/esp32c3	cpu/esp32_riscv	CPU/peripheral configuration ESP32C3
cpu/esp32h2	cpu/esp32_riscv	CPU/peripheral configuration ESP32H2

In this way it should be possible to enable the rust_target in cpu/esp32_riscv for RISC-V variants. Another option would be to enable Rust conditionally in a shared Makefile.features:

ifneq (,$filter riscv%,$(TARGET_ARCH))
  FEATURES_PROVIDED += rust_target
endif

But to be honest, I can't say if and how it will work with Rust at all. Neither do I have experience with Rust, nor do I know how it uses the ESP-IDF. The RIOT port uses only a small part and a slightly patched version of the ESP-IDF. At the moment I don't have time to learn anything about Rust, so this is not my focus.

[benpicco]

Do you have a PR for riotdocker too?

[gschorcht]

Do you have a PR for riotdocker too?

Not yet. At the moment I am working intensively on the port for the ESP32C3 (RISC-V based) to see what is the best way to restructure the cpu/esp32 and the toolchain to handle the different ESP32x variants.

[gschorcht]

Do you have a PR for riotdocker too?

Not yet. At the moment I am working intensively on the port for the ESP32C3 (RISC-V based) to see what is the best way to restructure the cpu/esp32 and the toolchain to handle the different ESP32x variants.

WIth PR #17734 and some additional small changes of this PR (I will provide them soon) it will become possible to use the precompiled Espressif toolchains as they are.

[gschorcht]

@leandrolanzieri In addition to this ESP32 port for ESP-IDF 4.4, I have already a working and complete port for the RISC-V based ESP32C3 which can be adapted quite easy to other ESP32x variants.
So now I am at the point where I wonder how we should structure the architectures/CPUs also with regard to Kconfig and the CPU, CPU_MODEL, CPU_FAM variables.

Could you please have a look at https://www.espressif.com/en/products/socs where you can get a complete overview of all ESP32x SoCs, their variants and modules that are used by most boards. Do you have any idea how we can structure them in a way that makes sense without defining each possible combination.

For example, the RISC-V based ESP32-C3 doesn't have flash on chip, but the ESP32­-C3-­WROOM­-02x module which is used by ESP32-C3-DevKitC-02 has 4 MB external flash on module. On the other hand, ESP32-C3-FN4 and ESP32-C3-FH4 have 4 MB embedded flash that are used by module ESP32-C3-MINI-1x which in turn is used by the ESP32-C3-DevKitM-1 board. In fact all boards that use these modules have the same size of RAM/ROM/SRAM and Flash.

The situation becomes much more difficult for ESP32-S3 where there are 5 chip variants with different sizes of Flash and PSRAM on chip. It becomes even more difficult because there are 20 versions of the ESP32-S3-WROOM-1x module with different sizes of Flash and PSRAM on module. That is, the initial idea to identify the size of RAM and Flash by the CPU model doesn't work here.

Any suggestion is welcome.

[benpicco]

I ran some tests on my esp32-wroom-32, all looking good:

• examples/gnrc_border_router (esp_wifi & esp_now)
• tests/mtd_raw
• tests/periph_gpio
• tests/periph_i2c
• tests/periph_rtc
• tests/periph_rtt
• tests/periph_hwrng
• tests/periph_adc

I don't have a board with esp_eth interface, maybe @yarrick could test that.
But if you already checked that there is no regression, I would trust your testing, you can just merge in that case.

[gschorcht]

I don't have a board with esp_eth interface, maybe @yarrick could test that.
But if you already checked that there is no regression, I would trust your testing, you can just merge in that case.

Test output of examples/gnrc_networking on esp32-olimex-evb:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DOUT, clock div:2
load:0x3fff0030,len:1584
load:0x40078000,len:11692
load:0x40080400,len:2780
entry 0x400805ac
32 kHz XTAL not found, switching to internal 150 kHz oscillator

initializing ESP32 Ethernet MAC (EMAC) device
NETOPT_TX_END_IRQ not implemented by driver
main(): This is RIOT! (Version: 2022.07-devel-498-gaa9dd-cpu/esp32/upgrade_esp-idf_v4.4)
RIOT network stack example application
All up, running the shell now
> ifconfig
Iface  8  HWaddr: 32:AE:A4:47:4C:CD  Link: up 
          L2-PDU:1500  MTU:1500  HL:64  RTR  
          RTR_ADV  
          Source address length: 6
          Link type: wired
          inet6 addr: fe80::30ae:a4ff:fe47:4ccd  scope: link  VAL
          inet6 group: ff02::2
          inet6 group: ff02::1
          inet6 group: ff02::1:ff47:4ccd
          inet6 group: ff02::1a
          
          Statistics for Layer 2
            RX packets 28  bytes 3004
            TX packets 1 (Multicast: 1)  bytes 0
            TX succeeded 1 errors 0
          Statistics for IPv6
            RX packets 1  bytes 72
            TX packets 1 (Multicast: 1)  bytes 64
            TX succeeded 1 errors 0

> 
> ping6 fe80::c5b4:388a:60e5:d4ea
12 bytes from fe80::c5b4:388a:60e5:d4ea%8: icmp_seq=0 ttl=255 time=2.178 ms
12 bytes from fe80::c5b4:388a:60e5:d4ea%8: icmp_seq=1 ttl=255 time=0.728 ms
12 bytes from fe80::c5b4:388a:60e5:d4ea%8: icmp_seq=2 ttl=255 time=0.641 ms

--- fe80::c5b4:388a:60e5:d4ea PING statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.641/1.182/2.178 ms

[gschorcht]

Thanks to everyone who reviewed, supported, and finally merged this large PR, especially @benpicco. As next step, I will rebase PR #17841 which replaces peripheral implentations for ESP32 by an implementation that uses the ESP-IDF HAL for ESP32x peripherals. This is the prerequesite for the port to ESP32-C3 PR #17844 and ESP32-S3 that is already completely working.