From 320e26430a5b6f3302f7a0d490e050e75cf43d18 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 25 Jun 2021 09:16:21 +0200 Subject: [PATCH 1/4] First draft of Reproducible builds RFC --- RFCs/0003-build-profiles.md | 216 ++++++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 RFCs/0003-build-profiles.md diff --git a/RFCs/0003-build-profiles.md b/RFCs/0003-build-profiles.md new file mode 100644 index 0000000..87c0a9c --- /dev/null +++ b/RFCs/0003-build-profiles.md @@ -0,0 +1,216 @@ +# Sketch build profiles and reproducible builds + +## Overview + +The Arduino build system and Sketch specifications lack a way to share and archive sketches and guarantee reproducibility of the built firmware at any time in the future as it was last successfully built. + +These incompatibilities might arise due to newer versions of a Platform, Tool, or Library. + +## Problem + +An Arduino project is known as Sketch. It comprises its main .ino file plus optional local source code (in the same folder) and an optional extra src folder containing additional sources if required. + +The sketch compiles against a set of globally installed libraries and platforms. When said library and/or part of the hardware platform is updated (sometimes introducing some breaking changes) that sketch might not compile anymore or behave differently. + +Currently, Arduino guarantees no portability and reproducibility of a build at a later time unless exact instructions are provided by the developer to the final user such as the required platform (including version and possibly a 3rd party platform URL to download and install the platform) or all the required libraries including the exact version of each one. + +## Goals + +This RFC aims to propose new augmentations to the current library management and a build system that can be entirely transparent to the user and retains backward compatibility yet allowing more flexibility for the future. + +The following should be true: + +- Implement ways for any Arduino Sketch to behave as a reproducible build with an option for versioning + +- Allow sharing of such sketch and “environment settings” from + + - IDE 1.8 and 2.0 using Archive Sketch + - Arduino Cloud using Share/Download Sketch + +## Constraints + +- Must be compatible across build systems + + - Arduino CLI + - Arduino IDE 2.0 + - Arduino IDE 1.8 + - Arduino Cloud + +Must not break pre-existing setups or be completely transparent to / ignored by previous versions of the tools + +## Proposed Solution + +The proposal is to add a project file called `sketch.yaml`. This file will be in YAML format and it will contain a description of all the resources required to compile a sketch, in particular: + +- the board FQBN +- the target core platform version (and the 3rd party platform index URL if needed) +- a possible core platform that is a dependency of the target core platform (again with version and index URL) +- the libraries used (including their version) + +We will call the set of the information above a **profile**. Each profile will be saved in the file `sketch.yaml` and this file may have one or more profiles. + +The format of the file is the following: + +``` +profiles: + : + notes: + fqbn: + platforms: + - platform: () + platform_index_url: <3RD_PARTY_PLATFORM_URL> + - platform: () + platform_index_url: <3RD_PARTY_PLATFORM_DEPENDENCY_URL> + libraries: + - () + - () + - () + +...more profiles here... + +default_profile: +``` + +We have a `profiles:` section containing all the profiles. Each field is self-explanatory, in particular: + +- `` is the profile identifier, it’s a user-defined field, and the allowed characters are alphanumerics, underscore `_`, dot `.`, and dash `-` +- `` is the target core platform identifier, for example, `arduino:avr` or `adafruit:samd` +- `` is the target core platform version required +- `<3RD_PARTY_PLATFORM_URL>` is the index URL to download the target core platform (also known as “Additional Boards Manager URLs” in the Arduino IDE). This field can be omitted for the official `arduino:*` platforms. +- ``, ``, and `<3RD_PARTY_PLATFORM_DEPENDENCY_URL>` contains the same information as ``, ``, and `<3RD_PARTY_PLATFORM_URL>` but for the core platform dependency of the main core platform. These fields are optional. +- `` is the version required for the library, for example, `1.0.0` +- `` is a free text string available to the developer to add comments +- `` is the profile used by default (more on that later) + +A complete example of a `sketch.yaml` may be the following: + +``` +profiles: + nanorp: + fqbn: arduino:mbed_nano:nanorp2040connect + platforms: + - platform: arduino:mbed_nano (2.1.0) + libraries: + - ArduinoIoTCloud (1.0.2) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) + + another_profile_name: + notes: testing the limit of the AVR platform, may be unstable + fqbn: arduino:avr:uno + platforms: + - platform: arduino:avr (1.8.4) + libraries: + - VitconMQTT (1.0.1) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) + + tiny: + notes: testing the very limit of the AVR platform, it will be very unstable + fqbn: attiny:avr:ATtinyX5:cpu=attiny85,clock=internal16 + platforms: + - platform: attiny:avr@1.0.2 + platform_index_url: https://raw.githubusercontent.com/damellis/attiny/ide-1.6.x-boards-manager/package_damellis_attiny_index.json + - platform: arduino:avr@1.8.3 + libraries: + - ArduinoIoTCloud (1.0.2) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) + + feather: + fqbn: adafruit:samd:adafruit_feather_m0 + platforms: + - platform: adafruit:samd (1.6.0) + platform_index_url: https://adafruit.github.io/arduino-board-index/package_adafruit_index.json + libraries: + - ArduinoIoTCloud (1.0.2) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) + +default_profile: nanorp +``` + +## Building a sketch + +When a `sketch.yaml` file exists in the sketch, it can be leveraged to compile the sketch with the `--profile` flag in the compile command: + +``` +arduino-cli compile [sketch] --profile nanorp +``` + +In this case, the sketch will be compiled using the core platform and libraries specified in the `nanorp` profile. If a core platform or a library is missing it will be automatically downloaded and installed on the fly in a dedicated directory inside the data folder. The dedicated storage is not accessible to the user and is meant as a “cache” of the resources used to build the sketch. + +When using the “profile” based build, the globally installed platforms and libraries are excluded from the compile and **may not be used in any way**, in other words, the build is isolated from the system and will rely only on the resources specified in the profile: this will ensure that the build is portable and reproducible independently from the platforms and libraries installed in the system. + +## Using a default profile + +If a `default_profile` is specified in the `sketch.yaml` then the “classic” compile command: + +``` +arduino-cli compile [sketch] +``` + +will trigger a profile-based build using the default profile indicated in the `sketch.yaml`. + +## Vendoring libraries with custom modifications + +We will add the possibility to add custom libraries directly in the sketch using the `libraries` subdirectory in the sketch root folder. + +A typical usage scenario is when the sketch needs a library that is not part of the library manager, or if the sketch needs a library with some customizations that are not available in the upstream release. + +To accomplish this the idea is to put the custom libraries' source code directly in the sketch `libraries` subdirectory (exactly as we do for the globally installed libraries in the `libraries` subdirectory of the sketchbook root folder). For example: + +``` +~/Arduino/HumiditySensor$ tree +HumiditySensor +├── HumiditySensor.ino +├── sketch.yaml +└── libraries + ├── DHTSensorCustom + │ ├── library.properties + │ ├── README + │ └── src + │ ├── DHT.cpp + │ ├── DHT.h + │ └── DHTSensorCustom.h + └── NTPClientEth + ├── NTPClient.h + └── NTPClient.cpp +``` + +If we run a “classic” build, without profiles, then the libraries `DHTSensorCustom` and `NTPClientEth` will be included at compile time as they were installed globally. + +If we want to run a reproducible build with profiles, we need to explicitly specify the bundled libraries we intend to use in the `sketch.yaml` in this way: + +``` +profiles: + nanorp: + fqbn: arduino:mbed_nano:nanorp2040connect + platforms: + - platform: arduino:mbed_nano (2.1.0) + libraries: + - ArduinoIoTCloud (1.0.2) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) + vendored_libraries: + - NTPClientEth + + uno: + notes: testing the limit of the AVR platform, may be unstable + fqbn: arduino:avr:uno + platforms: + - platform: arduino:avr (1.8.4) + libraries: + - VitconMQTT (1.0.1) + - Arduino_ConnectionHandler (0.6.4) + vendored_libraries: + - DHTSensorCustom + - NTPClientEth +``` + +In this case: + +- running a build using the profile `nanorp` will use the bundled `NTPClientEth` library, but not `DHTSensorCustom` +- running a build with the profile `uno` will use both `NTPClientEth` and `DHTSensorCustom` + +This behavior is required to allow different profiles to use a different set of vendored libraries. From 715791f21ef7690abeff7bee5b20091e9ce3b505 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 1 Jul 2021 10:26:36 +0200 Subject: [PATCH 2/4] Fix indentation --- RFCs/0003-build-profiles.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/RFCs/0003-build-profiles.md b/RFCs/0003-build-profiles.md index 87c0a9c..a1daa1e 100644 --- a/RFCs/0003-build-profiles.md +++ b/RFCs/0003-build-profiles.md @@ -87,13 +87,13 @@ A complete example of a `sketch.yaml` may be the following: ``` profiles: nanorp: - fqbn: arduino:mbed_nano:nanorp2040connect - platforms: - - platform: arduino:mbed_nano (2.1.0) - libraries: - - ArduinoIoTCloud (1.0.2) - - Arduino_ConnectionHandler (0.6.4) - - TinyDHT sensor library (1.1.0) + fqbn: arduino:mbed_nano:nanorp2040connect + platforms: + - platform: arduino:mbed_nano (2.1.0) + libraries: + - ArduinoIoTCloud (1.0.2) + - Arduino_ConnectionHandler (0.6.4) + - TinyDHT sensor library (1.1.0) another_profile_name: notes: testing the limit of the AVR platform, may be unstable From 1ebfb6f82cf3ac2eb9d6dd266664b7e245856e47 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 1 Jul 2021 10:35:42 +0200 Subject: [PATCH 3/4] Some other small typos and changes --- RFCs/0003-build-profiles.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/RFCs/0003-build-profiles.md b/RFCs/0003-build-profiles.md index a1daa1e..65c534a 100644 --- a/RFCs/0003-build-profiles.md +++ b/RFCs/0003-build-profiles.md @@ -10,9 +10,9 @@ These incompatibilities might arise due to newer versions of a Platform, Tool, o An Arduino project is known as Sketch. It comprises its main .ino file plus optional local source code (in the same folder) and an optional extra src folder containing additional sources if required. -The sketch compiles against a set of globally installed libraries and platforms. When said library and/or part of the hardware platform is updated (sometimes introducing some breaking changes) that sketch might not compile anymore or behave differently. +The sketch compiles against a set of globally installed libraries and platforms. When said library and/or part of the hardware platform is updated, sometimes introducing some breaking changes, that sketch might not compile anymore or behave differently. -Currently, Arduino guarantees no portability and reproducibility of a build at a later time unless exact instructions are provided by the developer to the final user such as the required platform (including version and possibly a 3rd party platform URL to download and install the platform) or all the required libraries including the exact version of each one. +Currently, Arduino guarantees no portability and reproducibility of a build at a later time unless exact instructions are provided by the developer to the final user, such as the required platform, including version and possibly a 3rd party platform URL to download and install the platform, or all the required libraries including the exact version of each one. ## Goals @@ -111,7 +111,7 @@ profiles: platforms: - platform: attiny:avr@1.0.2 platform_index_url: https://raw.githubusercontent.com/damellis/attiny/ide-1.6.x-boards-manager/package_damellis_attiny_index.json - - platform: arduino:avr@1.8.3 + - platform: arduino:avr@1.8.3 libraries: - ArduinoIoTCloud (1.0.2) - Arduino_ConnectionHandler (0.6.4) @@ -140,7 +140,7 @@ arduino-cli compile [sketch] --profile nanorp In this case, the sketch will be compiled using the core platform and libraries specified in the `nanorp` profile. If a core platform or a library is missing it will be automatically downloaded and installed on the fly in a dedicated directory inside the data folder. The dedicated storage is not accessible to the user and is meant as a “cache” of the resources used to build the sketch. -When using the “profile” based build, the globally installed platforms and libraries are excluded from the compile and **may not be used in any way**, in other words, the build is isolated from the system and will rely only on the resources specified in the profile: this will ensure that the build is portable and reproducible independently from the platforms and libraries installed in the system. +When using the profile-based build, the globally installed platforms and libraries are excluded from the compile and **can not be used in any way**, in other words, the build is isolated from the system and will rely only on the resources specified in the profile: this will ensure that the build is portable and reproducible independently from the platforms and libraries installed in the system. ## Using a default profile @@ -150,7 +150,7 @@ If a `default_profile` is specified in the `sketch.yaml` then the “classic” arduino-cli compile [sketch] ``` -will trigger a profile-based build using the default profile indicated in the `sketch.yaml`. +will, instead, trigger a profile-based build using the default profile indicated in the `sketch.yaml`. ## Vendoring libraries with custom modifications From 14f7b2fb7df61b35746cd89aa6069adafe1bb89c Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 12 Jul 2021 15:41:14 +0200 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: per1234 --- RFCs/0003-build-profiles.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/RFCs/0003-build-profiles.md b/RFCs/0003-build-profiles.md index 65c534a..780c4ae 100644 --- a/RFCs/0003-build-profiles.md +++ b/RFCs/0003-build-profiles.md @@ -8,11 +8,11 @@ These incompatibilities might arise due to newer versions of a Platform, Tool, o ## Problem -An Arduino project is known as Sketch. It comprises its main .ino file plus optional local source code (in the same folder) and an optional extra src folder containing additional sources if required. +An Arduino project is known as Sketch. It comprises its main `.ino` file plus optional local source code (in the same folder) and an optional extra `src` folder containing additional sources if required. The sketch compiles against a set of globally installed libraries and platforms. When said library and/or part of the hardware platform is updated, sometimes introducing some breaking changes, that sketch might not compile anymore or behave differently. -Currently, Arduino guarantees no portability and reproducibility of a build at a later time unless exact instructions are provided by the developer to the final user, such as the required platform, including version and possibly a 3rd party platform URL to download and install the platform, or all the required libraries including the exact version of each one. +Currently, the only way to guarantee portability and reproducibility of a sketch build at a later time is for the developer to provide instructions to the final user for the installation of the sketch dependencies, including the exact versions of the boards platform and libraries. ## Goals @@ -73,7 +73,7 @@ default_profile: We have a `profiles:` section containing all the profiles. Each field is self-explanatory, in particular: -- `` is the profile identifier, it’s a user-defined field, and the allowed characters are alphanumerics, underscore `_`, dot `.`, and dash `-` +- `` is the profile identifier, it’s a user-defined field, and the allowed characters are alphanumerics, underscore `_`, dot `.`, and dash `-` - `` is the target core platform identifier, for example, `arduino:avr` or `adafruit:samd` - `` is the target core platform version required - `<3RD_PARTY_PLATFORM_URL>` is the index URL to download the target core platform (also known as “Additional Boards Manager URLs” in the Arduino IDE). This field can be omitted for the official `arduino:*` platforms. @@ -156,9 +156,9 @@ will, instead, trigger a profile-based build using the default profile indicated We will add the possibility to add custom libraries directly in the sketch using the `libraries` subdirectory in the sketch root folder. -A typical usage scenario is when the sketch needs a library that is not part of the library manager, or if the sketch needs a library with some customizations that are not available in the upstream release. +A typical usage scenario is when the sketch needs a library that is not offered for installation from Library Manager, or if the sketch needs a library with some customizations that are not available in the upstream release. -To accomplish this the idea is to put the custom libraries' source code directly in the sketch `libraries` subdirectory (exactly as we do for the globally installed libraries in the `libraries` subdirectory of the sketchbook root folder). For example: +To accomplish this the idea is to put the custom libraries' source code directly in the `libraries` subdirectory of the sketch (exactly as we do for the globally installed libraries in the `libraries` subdirectory of the sketchbook root folder). For example: ``` ~/Arduino/HumiditySensor$ tree