Save build properties in profiles

[alranel]

Describe the request

Extend the build profiles feature to also include build properties in the profiles.

Suppose you're compiling a sketch for ESP8266. In the IDE, you'll select multiple options from the Tools menu (like Flash Size, Erase Flash etc.) which affect the compilation of the core. In the CLI, unfortunately, the menu structure is not exposed so instead of using those shortcuts you need to specify all the single properties. For instance:

arduino-cli compile --fqbn esp8266:esp8266:generic -e --build-property build.flash_size=2M --build-property build.flash_ld=eagle.flash.2m.ld --build-property build.spiffs_pagesize=256 --build-property build.rfcal_addr=0x1FC000

Now, when adding --dump-profile I would expect such build properties to be included in the resulting file in order to get a "reproducibile build". However, they are not included.

TL;DR: build properties should be added to the sketch file specification and then implemented in --dump-profile and --profile.

Describe the current behavior

As of now, the --dump-profile option implemented as part of the "reproducible builds" project (see RFC) includes information about board, core and libraries but does not include any additional information such as build properties.

Arduino CLI version

0.27.1

Operating system

macOS

Operating system version

10.15.7

Additional context

No response

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the nightly build
• My request contains all necessary details

[per1234]

Hi @alranel. Thanks for your suggestion.

the menu structure is not exposed so instead of using those shortcuts you need to specify all the single properties

Even though the requested capability of being able to define properties in profiles does have valid advanced use cases (e.g., arduino/tooling-rfcs#9), in this specific case it is not necessary and I would even strongly discourage this approach when using Arduino CLI from the command line since these properties are not intended to be part of the public interface of the platform and might be changed at any time without the platform developers considering it a breaking change.

The correct approach to setting "custom board options" is via the FQBN.

The format of the FQBN is like this:

<vendor ID>:<architecture>:<board ID>[:<menu ID>=<option ID>[,<menu ID>=<option ID>]...]

You can learn all the available menu IDs and option IDs for a given board by running the following arduino-cli command:

arduino-cli board details --fqbn <FQBN>

(where <FQBN> is replaced by the fully qualified board name of the board you are using)

For example:

$ arduino-cli board details -b esp8266:esp8266:generic
Board name:                Generic ESP8266 Module
FQBN:                      esp8266:esp8266:generic
Board version:             3.0.2

Identification properties: board=generic

Package name:              esp8266
Package maintainer:        ESP8266 Community
Package URL:               https://arduino.esp8266.com/stable/package_esp8266com_index.json
Package website:           https://github.com/esp8266/Arduino
Package online help:       https://esp8266.com/arduino

Platform name:             esp8266
Platform category:         ESP8266
Platform architecture:     esp8266
Platform URL:              https://github.com/esp8266/Arduino/releases/download/3.0.2/esp8266-3.0.2.zip
Platform file name:        esp8266-3.0.2.zip
Platform size (bytes):     38204730
Platform checksum:         SHA-256:653add97df0a850ebc26f105355be749fb83c2db7b1ddf3e77950e694059eb9c

Required tool:             esp8266:xtensa-lx106-elf-gcc                                       3.0.4-gcc10.3-1757bed

Required tool:             esp8266:mkspiffs                                                   3.0.4-gcc10.3-1757bed

Required tool:             esp8266:mklittlefs                                                 3.0.4-gcc10.3-1757bed

Required tool:             esp8266:python3                                                    3.7.2-post1

Option:                    CPU Frequency                                                                            xtal

                           80 MHz                                                             ✔                     xtal=80

                           160 MHz                                                                                  xtal=160

Option:                    VTables                                                                                  vt

                           Flash                                                              ✔                     vt=flash

                           Heap                                                                                     vt=heap

                           IRAM                                                                                     vt=iram

Option:                    C++ Exceptions                                                                           exception

                           Disabled (new aborts on oom)                                       ✔                     exception=disabled

                           Enabled                                                                                  exception=enabled

Option:                    Stack Protection                                                                         stacksmash

                           Disabled                                                           ✔                     stacksmash=disabled

                           Enabled                                                                                  stacksmash=enabled

Option:                    SSL Support                                                                              ssl

                           All SSL ciphers (most compatible)                                  ✔                     ssl=all

                           Basic SSL ciphers (lower ROM use)                                                        ssl=basic

Option:                    MMU                                                                                      mmu

                           32KB cache + 32KB IRAM (balanced)                                  ✔                     mmu=3232

                           16KB cache + 48KB IRAM (IRAM)                                                            mmu=4816

                           16KB cache + 48KB IRAM and 2nd Heap (shared)                                             mmu=4816H

                           16KB cache + 32KB IRAM + 16KB 2nd Heap (not shared)                                      mmu=3216

                           128K External 23LC1024                                                                   mmu=ext128k

                           1M External 64 MBit PSRAM                                                                mmu=ext1024k

Option:                    Non-32-Bit Access                                                                        non32xfer

                           Use pgm_read macros for IRAM/PROGMEM                               ✔                     non32xfer=fast

                           Byte/Word access to IRAM/PROGMEM (very slow)                                             non32xfer=safe

Option:                    Reset Method                                                                             ResetMethod

                           dtr (aka nodemcu)                                                  ✔                     ResetMethod=nodemcu

                           no dtr (aka ck)                                                                          ResetMethod=ck

                           no dtr, no_sync                                                                          ResetMethod=nodtr_nosync

Option:                    Crystal Frequency                                                                        CrystalFreq

                           26 MHz                                                             ✔                     CrystalFreq=26

                           40 MHz                                                                                   CrystalFreq=40

Option:                    Flash Frequency                                                                          FlashFreq

                           40MHz                                                              ✔                     FlashFreq=40

                           80MHz                                                                                    FlashFreq=80

                           20MHz                                                                                    FlashFreq=20

                           26MHz                                                                                    FlashFreq=26

Option:                    Flash Mode                                                                               FlashMode

                           DOUT (compatible)                                                  ✔                     FlashMode=dout

                           DIO                                                                                      FlashMode=dio

                           QOUT                                                                                     FlashMode=qout

                           QIO (fast)                                                                               FlashMode=qio

Option:                    Flash Size                                                                               eesz

                           1MB (FS:64KB OTA:~470KB)                                           ✔                     eesz=1M64

                           1MB (FS:128KB OTA:~438KB)                                                                eesz=1M128

                           1MB (FS:144KB OTA:~430KB)                                                                eesz=1M144

                           1MB (FS:160KB OTA:~422KB)                                                                eesz=1M160

                           1MB (FS:192KB OTA:~406KB)                                                                eesz=1M192

                           1MB (FS:256KB OTA:~374KB)                                                                eesz=1M256

                           1MB (FS:512KB OTA:~246KB)                                                                eesz=1M512

                           1MB (FS:none OTA:~502KB)                                                                 eesz=1M

                           2MB (FS:64KB OTA:~992KB)                                                                 eesz=2M64

                           2MB (FS:128KB OTA:~960KB)                                                                eesz=2M128

                           2MB (FS:256KB OTA:~896KB)                                                                eesz=2M256

                           2MB (FS:512KB OTA:~768KB)                                                                eesz=2M512

                           2MB (FS:1MB OTA:~512KB)                                                                  eesz=2M1M

                           2MB (FS:none OTA:~1019KB)                                                                eesz=2M

                           4MB (FS:2MB OTA:~1019KB)                                                                 eesz=4M2M

                           4MB (FS:3MB OTA:~512KB)                                                                  eesz=4M3M

                           4MB (FS:1MB OTA:~1019KB)                                                                 eesz=4M1M

                           4MB (FS:none OTA:~1019KB)                                                                eesz=4M

                           8MB (FS:6MB OTA:~1019KB)                                                                 eesz=8M6M

                           8MB (FS:7MB OTA:~512KB)                                                                  eesz=8M7M

                           16MB (FS:14MB OTA:~1019KB)                                                               eesz=16M14M

                           16MB (FS:15MB OTA:~512KB)                                                                eesz=16M15M

                           512KB (FS:32KB OTA:~230KB)                                                               eesz=512K32

                           512KB (FS:64KB OTA:~214KB)                                                               eesz=512K64

                           512KB (FS:128KB OTA:~182KB)                                                              eesz=512K128

                           512KB (FS:none OTA:~246KB)                                                               eesz=512K

Option:                    Builtin Led                                                                              led

                           2                                                                  ✔                     led=2

                           0                                                                                        led=0

                           1                                                                                        led=1

                           3                                                                                        led=3

                           4                                                                                        led=4

                           5                                                                                        led=5

                           6                                                                                        led=6

                           7                                                                                        led=7

                           8                                                                                        led=8

                           9                                                                                        led=9

                           10                                                                                       led=10

                           11                                                                                       led=11

                           12                                                                                       led=12

                           13                                                                                       led=13

                           14                                                                                       led=14

                           15                                                                                       led=15

                           16                                                                                       led=16

Option:                    Espressif FW                                                                             sdk

                           nonos-sdk 2.2.1+100 (190703)                                       ✔                     sdk=nonosdk_190703

                           nonos-sdk 2.2.1+119 (191122)                                                             sdk=nonosdk_191122

                           nonos-sdk 2.2.1+113 (191105)                                                             sdk=nonosdk_191105

                           nonos-sdk 2.2.1+111 (191024)                                                             sdk=nonosdk_191024

                           nonos-sdk 2.2.1 (legacy)                                                                 sdk=nonosdk221

                           nonos-sdk pre-3 (180626 known issues)                                                    sdk=nonosdk3v0

Option:                    lwIP Variant                                                                             ip

                           v2 Lower Memory                                                    ✔                     ip=lm2f

                           v2 Higher Bandwidth                                                                      ip=hb2f

                           v2 Lower Memory (no features)                                                            ip=lm2n

                           v2 Higher Bandwidth (no features)                                                        ip=hb2n

                           v2 IPv6 Lower Memory                                                                     ip=lm6f

                           v2 IPv6 Higher Bandwidth                                                                 ip=hb6f

Option:                    Debug port                                                                               dbg

                           Disabled                                                           ✔                     dbg=Disabled

                           Serial                                                                                   dbg=Serial

                           Serial1                                                                                  dbg=Serial1

Option:                    Debug Level                                                                              lvl

                           None                                                               ✔                     lvl=None____

                           SSL                                                                                      lvl=SSL

                           TLS_MEM                                                                                  lvl=TLS_MEM

                           HTTP_CLIENT                                                                              lvl=HTTP_CLIENT

                           HTTP_SERVER                                                                              lvl=HTTP_SERVER

                           SSL+TLS_MEM                                                                              lvl=SSLTLS_MEM

                           SSL+HTTP_CLIENT                                                                          lvl=SSLHTTP_CLIENT

                           SSL+HTTP_SERVER                                                                          lvl=SSLHTTP_SERVER

                           TLS_MEM+HTTP_CLIENT                                                                      lvl=TLS_MEMHTTP_CLIENT

                           TLS_MEM+HTTP_SERVER                                                                      lvl=TLS_MEMHTTP_SERVER

                           HTTP_CLIENT+HTTP_SERVER                                                                  lvl=HTTP_CLIENTHTTP_SERVER

                           SSL+TLS_MEM+HTTP_CLIENT                                                                  lvl=SSLTLS_MEMHTTP_CLIENT

                           SSL+TLS_MEM+HTTP_SERVER                                                                  lvl=SSLTLS_MEMHTTP_SERVER

                           SSL+HTTP_CLIENT+HTTP_SERVER                                                              lvl=SSLHTTP_CLIENTHTTP_SERVER

                           TLS_MEM+HTTP_CLIENT+HTTP_SERVER                                                          lvl=TLS_MEMHTTP_CLIENTHTTP_SERVER

                           SSL+TLS_MEM+HTTP_CLIENT+HTTP_SERVER                                                      lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVER

                           CORE                                                                                     lvl=CORE

                           WIFI                                                                                     lvl=WIFI

                           HTTP_UPDATE                                                                              lvl=HTTP_UPDATE

                           UPDATER                                                                                  lvl=UPDATER

                           OTA                                                                                      lvl=OTA

                           OOM                                                                                      lvl=OOM

                           MDNS                                                                                     lvl=MDNS

                           HWDT                                                                                     lvl=HWDT

                           HWDT_NOEXTRA4K                                                                           lvl=HWDT_NOEXTRA4K

                           CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS                                               lvl=COREWIFIHTTP_UPDATEUPDATEROTAOOMMDNS

                           CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS+HWDT                                          lvl=COREWIFIHTTP_UPDATEUPDATEROTAOOMMDNSHWDT

                           CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS+HWDT_NOEXTRA4K                                lvl=COREWIFIHTTP_UPDATEUPDATEROTAOOMMDNSHWDT_NOEXTRA4K

                           SSL+TLS_MEM+HTTP_CLIENT+HTTP_SERVER+CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS                       lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVERCOREWIFIHTTP_UPDATEUPDATEROTAOOMMDNS
                           SSL+TLS_MEM+HTTP_CLIENT+HTTP_SERVER+CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS+HWDT                       lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVERCOREWIFIHTTP_UPDATEUPDATEROTAOOMMDNSHWDT
                           SSL+TLS_MEM+HTTP_CLIENT+HTTP_SERVER+CORE+WIFI+HTTP_UPDATE+UPDATER+OTA+OOM+MDNS+HWDT_NOEXTRA4K                       lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVERCOREWIFIHTTP_UPDATEUPDATEROTAOOMMDNSHWDT_NOEXTRA4K
                           NoAssert-NDEBUG                                                                          lvl=NoAssert-NDEBUG

Option:                    Erase Flash                                                                              wipe

                           Only Sketch                                                        ✔                     wipe=none

                           Sketch + WiFi Settings                                                                   wipe=sdk

                           All Flash Contents                                                                       wipe=all

Option:                    Upload Speed                                                                             baud

                           115200                                                             ✔                     baud=115200
                           57600                                                                                    baud=57600
                           256000                                                                                   baud=256000
                           512000                                                                                   baud=512000
                           921600                                                                                   baud=921600
                           3000000                                                                                  baud=3000000
Programmers:               Id                                                                 Name

Here we can see that the ID for the "Flash Size" menu is eesz and the ID for the "2MB (FS:none OTA:~1019KB)" option is 2M.

So the full FQBN is:

esp8266:esp8266:generic:eesz=2M

Much easier than digging up all those build properties!

Setting custom build options via the FQBN is covered in the platform specification (though not in a very user friendly manner) and to some extent in the FAQ. I do think the documentation of this needs to be expanded though as it is a common source of confusion. We are using this public issue to track that: #355

implemented in --dump-profile

--dump-profile will configure the profile according to the FQBN passed to the command:

$ ./arduino-cli compile --dump-profile --fqbn esp8266:esp8266:generic:eesz=2M

[...]

profiles:
  generic:
    fqbn: esp8266:esp8266:generic:eesz=2M
    platforms:
      - platform: esp8266:esp8266 (3.0.2)
        platform_index_url: https://arduino.esp8266.com/stable/package_esp8266com_index.json

[alranel]

Thank you for the answer Per! That was not easy to understand from documentation indeed, and might also be a good idea to mention this in the examples related to --dump-profile to avoid headaches to the ESP* users that rely very much on these flags. For now, this issue and your reply will hopefully help people looking for this.
Feel free to close this issue then!

[umbynos]

Closing as this is already tracked by #355