Optimize build. Only used libraries downloaded & compiled. #18699

rhapsodyv · 2020-07-18T21:03:06Z

Description

With more time, and digging deep in the PlatformIO code, I could create a script that enable/install the dependencies according with the features enabled in Configuration.h.

To do it, the script change the current configuration (platformio.ini contents) loaded by the PlatformIO. It isn't a "standard" way, but works.

The script:

Uses the current compiler to check the enabled features in Configuration.h
Must keep a list of the features and your dependencies
Must run in every env, first, with a "pre:". (well, at least in env that work with features that have dependencies)
Works with private libs in Marlin/lib dir too.
Export a method that all other scripts can use to check if a Marlin feature is enabled: env.MarlinFeatureIsEnabled(feature)
It may work with any feature that have any dependency or extra_script
It may work to add/remove src_filter too. Important: its doesn't support parenting. If a parent folder was removed from compile, a child folder cannot be added.
It respects env lib_ignore, so it dont add a lib if its ignored
If the env already have a lib with the same name, the script dont append it. So, the env can have their own version of the lib.
Its even better: it may be able to change almost any platformio.ini config according with features enabled in Configuration.h, at runtime.

Output while compiling:

Feature enabled: TFT_LVGL_UI
Adding lib_deps for TFT_LVGL_UI...

Benefits

We won't need anymore create an env for each feature in a board
Dependencies only will be installed when needed
Dependencies only will be compiled when needed
Save times and downloads

Related Issues

#18500

Tasks

Move the features configs to platformio.ini
Load the build flags for each platform automatically (created buildroot/share/PlatformIO/scripts/common-features-dependencies.h to get features without any HAL)
Make a dependency overwritable by the env lib_deps (check if the env already have a dep with the same name, so
ignore it)
Respect lib_ignore configs
Do we need that all lib_ignore, now that deps are add only when used?
src_filter change support
comma support
remove env for specific features

…lly all its dependencies

…eatures!! Like HAS_TRINAMIC, to add TMCStepper lib only when needed!

…in into features-and-dependencies

rhapsodyv · 2020-07-19T02:08:13Z

@thinkyhead Look the last commit. Now it can check even for more advanced features, like HAS_TRINAMIC, to install tmc lib only when need.

But I don’t know if we need go too deep.

Anyway, it’s a good prof of concept.

rhapsodyv · 2020-07-19T03:30:07Z

CI is falling, because Marlin/src/inc/MarlinConfigPre.h includes Marlin/src/HAL/platforms.h, that needs to know in advance which platform we are compiling.
To discover that, I did the script process all build flags and pass to g++ all the defines I found in the current build process.

But, in some boards, the definition of the platform isn't in a build flag (like SAMD51_grandcentral_m4). To solve the failing build, we need just to add the platform definition for each "parent" env. It's easy.

Another approach, is to have all features definition in more "simple" file, that dont depends on HAL to generate the marlin features definitions.

rhapsodyv · 2020-07-19T03:50:59Z

Well, it was just one board missing the platform definition in the build flags ☺️ 🤣

thinkyhead · 2020-07-19T08:41:13Z

None of the "board" defines are present when the compiler is invoked from the python script, and this is why the platforms.h file needs extra help. I wonder if there's some way to get the flags from buildroot/share/PlatformIO/boards/MyBoard.json and/or the MyBoard.json files included in the platform and add them to the command-line.

thinkyhead · 2020-07-19T08:57:22Z

Here's a typical command-line for an AVR build, with our missing flags supplied by PlatformIO…

avr-g++ -o .pio/build/mega2560/src/src/lcd/extui/lib/mks_ui/draw_printing.cpp.o -c -Wno-register -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -fmax-errors=5 -g -fmerge-all-constants -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega2560 -DPLATFORMIO=40304 -DARDUINO_AVR_MEGA2560 -D__MARLIN_FIRMWARE__ -D__AVR__ -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -IMarlin -I$HOME/.platformio/packages/framework-arduino-avr/libraries/EEPROM/src -I.pio/libdeps/mega2560/U8glib-HAL_ID1932/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/Wire/src -I.pio/libdeps/mega2560/TMCStepper/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/SoftwareSerial/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/SPI/src -I$HOME/.platformio/packages/framework-arduino-avr/cores/arduino -I$HOME/.platformio/packages/framework-arduino-avr/variants/mega Marlin/src/lcd/extui/lib/mks_ui/draw_printing.cpp

rhapsodyv · 2020-07-19T09:32:14Z

We can move the features deps to platformio.ini in a features section. What do you think?

#
# Features and Dependecies
#
[features]
TFT_LVGL_UI=MKS-LittlevGL=https://github.com/makerbase-mks/MKS-LittlevGL/archive/master.zip
HAS_TRINAMIC=TMCStepper@~0.7.1
SR_LCD_2W_NL=SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
SR_LCD_3W_NL=SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
DIGIPOT_MCP4018=SlowSoftI2CMaster
DIGIPOT_MCP4451=SlowSoftI2CMaster
HAS_TMC26X=TMC26XStepper=https://github.com/trinamic/TMC26XStepper/archive/master.zip
HAS_L64XX=Arduino-L6470@0.8.0
NEOPIXEL_LED=Adafruit NeoPixel@1.5.0
MAX6675_IS_MAX31865=Adafruit MAX31865 library@~1.1.0
HAS_GRAPHICAL_LCD=U8glib-HAL@0.4.1
HAS_CHARACTER_LCD=LiquidCrystal@1.5.0
  LiquidTWI2@1.2.7

thinkyhead · 2020-07-19T09:36:25Z

That could work. Note that under some environments, older/different versions of libraries are used. So we need to be able to specify those differences by overriding the values in the [features] section within individual environments.

rhapsodyv · 2020-07-19T09:39:36Z

That could work.

I will move to [features].

Note that under some environments, older/different versions of libraries are used. So we need to be able to specify those differences by overriding the values in the [features] section within individual environments.

Right now I just prepend the lib_deps to the current env lib_deps. But we can check: if the env already have a lib_dep with the same name, we just ignore.

rhapsodyv · 2020-07-19T10:21:22Z

None of the "board" defines are present when the compiler is invoked from the python script, and this is why the platforms.h file needs extra help. I wonder if there's some way to get the flags from buildroot/share/PlatformIO/boards/MyBoard.json and/or the MyBoard.json files included in the platform and add them to the command-line.

I loaded the boards.json flags. But, some platform, the arch definition is in a py script, like:
~/.platformio/packages/framework-arduinoespressif32/tools/platformio-build.py
It's that py that insert the ARDUINO_ARCH_ESP32 definition.

I will check if we can run our script after the platform platformio-build.py

rhapsodyv · 2020-07-19T10:37:51Z

pre: script runs before the current platform script. post: runs after, but after the library dependency finder too. So, if we use 'pre', we dont have all platform flags. If you run 'post', is late to add dependencies to the build.

PlatformIO code:

# HERE our script run
for item in env.GetExtraScripts("pre"):
    env.SConscript(item, exports="env")

# HERE the build platform script run, but it run all the lib deps too!
env.SConscript("$BUILD_SCRIPT")

...

# here is the post script
for item in env.GetExtraScripts("post"):
    env.SConscript(item, exports="env")

Do we really need process everything, including HAL, to know the enabled features?

If we create a simple ifdef to ignore HAL in the MarlinConfigPre, it may work... Or, we can create a header just with the propose of discover the enabled features, that dont include any HAL.
Or we will need to have the arch definition in the build_flags of each env. It’s not a big deal, considering that it will never change.

ellensp · 2020-07-23T04:41:23Z

That is wrong version of Adafruit NeoPixel library for STM32F103RET6
change
NEOPIXEL_LED = Adafruit NeoPixel@1.5.0
to
NEOPIXEL_LED = Adafruit NeoPixel@1.2.4

but I dont know how to do this automatically in the new system.
@rhapsodyv can this be done?

rhapsodyv · 2020-07-23T04:50:17Z

That is wrong version of Adafruit NeoPixel library for STM32F103RET6
change
NEOPIXEL_LED = Adafruit NeoPixel@1.5.0
to
NEOPIXEL_LED = Adafruit NeoPixel@1.2.4

but I dont know how to do this automatically in the new system.
@rhapsodyv can this be done?

The env can overwrite any lib it needs to a specific version.

[env:my_env]
lib_deps = FOO@1.2.3

But his problem is other. There's no NEOPIXEL_LED enabled in that config. I'm almost sure the script ins't running in his env... If the script isn't running, and they have forgotten unused libs in his .pio/lib_deps, the compiler will try use it...
If our script run, it will lib_ignore everything not used, so it will not break the compilation when there're unused libs in .pio/lib_deps.

@mindspawn could you confirm you env isn't (or it is...) running the commons scripts?

rhapsodyv · 2020-07-23T04:56:24Z

@ivankravets Another important thing: it would be very useful if we can have a script that run always, regardless of what is in the current env. Right now, to run our script, we had to edit all envs to call it.

mindspawn · 2020-07-23T05:00:07Z

@rhapsodyv your right deleting my workspace, cloning the repo and building again seemed to have fixed the build issue. Thanks a lot :)

pfeerick · 2020-07-23T08:56:44Z

it would be very useful if we can have a script that run always, regardless of what is in the current env. Right now, to run our script, we had to edit all envs to call it.

You should be able to use the common env for that? i.e. https://docs.platformio.org/en/latest/projectconf/section_env.html#common-env

Or are you referring to the common env extra_scripts being overridden by the local env extra_script setting (as IIRC that is what happens, rather than it being appended to, meaning you need to add something like${common.extra_scripts} to the extra_scripts of each env)?

ivankravets · 2020-07-23T13:09:13Z

Oh, very long thread :) Thanks to everyone who joined this thread.

If there is a bug linked with PlatformIO Core, please create a simple project to reproduce it and file an issue here https://github.com/platformio/platformio-core/issues
We have deployed the new registry (currently, only backend). The new registry has strict rules and requires a library to be compliant with Semantic Versioning.
Developers will be able in PlatformIO Core 4.4 to declare dependencies based on their owners (avoid IDs, conflicts, etc).

We highly recommend using SemVer for dependencies and avoid using external archives, tarballs, VCS repositories. If you need a library and it has not been published in the registry, you can publish it into the own scope https://docs.platformio.org/en/latest/librarymanager/creating.html

The only PIO Core 4.4 is released, we show instruction on how to use libraries by some owner. The syntax will be the next:

lib_deps = 
   ownername/packagename @ SemVer-requirements

For example,

lib_deps = 
   armmbed/mbedtls@^2

Means, use the latest version of mbedtls in range 2.0.0-2.Y.Z.

rhapsodyv · 2020-07-23T14:12:36Z

Or are you referring to the common env extra_scripts being overridden by the local env extra_script setting (as IIRC that is what happens, rather than it being appended to, meaning you need to add something like${common.extra_scripts} to the extra_scripts of each env)?

Yes, the common env extra_scripts are being overridden by the local env extra_script. We add ${common.extra_scripts} to all envs... But, some users had their platformio.ini edited and getting it messed up while merging.

It would be better if we can have a way to define a global pre script, the run always.

rhapsodyv · 2020-07-23T14:17:23Z

Oh, very long thread :) Thanks to everyone who joined this thread.

If there is a bug linked with PlatformIO Core, please create a simple project to reproduce it and file an issue here https://github.com/platformio/platformio-core/issues

We have deployed the new registry (currently, only backend). The new registry has strict rules and requires a library to be compliant with Semantic Versioning.

Developers will be able in PlatformIO Core 4.4 to declare dependencies based on their owners (avoid IDs, conflicts, etc).

It would be great if we have a 4 item:

Developers being able to declare dependencies based on a header file with all defined marcros/configurations the user enabled :-)

Thanks @ivankravets . I will create an issue to discuss about it.

thinkyhead · 2020-07-24T02:15:47Z

Thanks for moving discussion to the PIO forums. There's no bug to speak of; we're just trying to do fancy things with "Advanced Scripting" to optimize our build times. We'll be using the power of src_filter plus the added [features] block to get even more granular.

I will see about adding Adafruit Neopixel (and SPI) back to the lib_ignore of platforms that the library is incompatible with, but which include another library that declares it as a "dependency."

rhapsodyv · 2020-07-24T02:23:28Z

I will see about adding Adafruit Neopixel (and SPI) back to the lib_ignore of platforms that the library is incompatible with, but which include another library that declares it as a "dependency."

We only got this kind of compiling error when user had a merge issue in platformio.ini... Adding a lib_ignore, may not solve it too.... But anyway... adding it won't do any bad either.

Right now, every known lib that isn't used by the env is automatically and explicity ignored, adding it to lib_ignore.

thinkyhead · 2020-07-24T02:59:28Z

Right now, every known lib that isn't used by the env is automatically and explicity ignored, adding it to lib_ignore.

Interesting. When building for ESP32 with WIFISUPPORT you'll notice that ESPAsyncTCP is not named as a direct lib_deps entry. However, one of the other items in the lib_deps list does list it as a dependency, so it gets downloaded and compiled. For this reason I did need to add it to the lib_ignore list for WIFISUPPORT.

If the script is supposed to be stripping out all the lib_deps that we have not explicitly named, it is somehow missing ESPAsyncTCP?

rhapsodyv · 2020-07-24T03:06:33Z

If the script is supposed to be stripping out all the lib_deps that we have not explicitly named, it is somehow missing ESPAsyncTCP?

Take a look at the output... it must have outputting something like this: Ignoring libs: ...

Adding src_filter for TFT_LVGL_UI... 
Ignoring libs:  ['TMCStepper', 'Adafruit MAX31865 library', 'U8glib-HAL', 'LiquidCrystal', 'TMC26XStepper', 'SlowSoftI2CMaster', 'SailfishLCD', 'arduinoWebSockets', 'ESP32SSDP', 'ESP Async WebServer', 'AsyncTCP', 'Adafruit NeoPixel', 'Arduino-L6470', 'LiquidTWI2', 'ESP3DLib']

rhapsodyv · 2020-07-24T03:39:02Z

ESPAsyncTCP

Now I saw what is happening. The script dont know how to handle it: lib_ignore=ESPAsyncTCP...

HAS_LCD_MENU        = src_filter=+<src/lcd/menu>
(ESP32_)?WIFISUPPORT = AsyncTCP, ESP Async WebServer
  ESP3DLib=https://github.com/luc-github/ESP3DLib.git
  arduinoWebSockets=https://github.com/Links2004/arduinoWebSockets.git
  ESP32SSDP=https://github.com/luc-github/ESP32SSDP.git
  lib_ignore=ESPAsyncTCP

You added the parsing in the load_config function:


			elif name == 'lib_ignore':
				FEATURE_DEPENDENCIES[ukey]['lib_ignore'] = rest

But, it's missing the magic code in the install_features_dependencies() function :-)
so, right now, this lib_ignore=ESPAsyncTCP in features have no effect :-)

rhapsodyv · 2020-07-24T03:41:28Z

But, it's missing the magic code in the install_features_dependencies() function :-)
so, right now, this lib_ignore=ESPAsyncTCP in features have no effect :-)

Here is the missing code in install_features_dependencies():

		if 'lib_ignore' in FEATURE_DEPENDENCIES[feature]:
			print("Ignoring libs for %s... " % feature)
			lib_ignore = env.GetProjectOption("lib_ignore") + [FEATURE_DEPENDENCIES[feature]['lib_ignore']]
			proj = env.GetProjectConfig()
			proj.set("env:" + env["PIOENV"], "lib_ignore", lib_ignore)

thinkyhead · 2020-07-24T03:45:17Z

Added some extra logging, and it does seem to process the line…

Log Output

Feature Dep Line:  MKS-LittlevGL=https://github.com/makerbase-mks/MKS-LittlevGL/archive/master.zip
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/mks_ui>
Feature Filter:  +<src/lcd/extui/lib/mks_ui>
Feature Dep Line:  TMCStepper@~0.7.1
Feature Dep Line:  SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
Feature Dep Line:  SlowSoftI2CMaster
Feature Dep Line:  TMC26XStepper=https://github.com/trinamic/TMC26XStepper/archive/master.zip
Feature Dep Line:  Arduino-L6470@0.8.0
Feature Dep Line:  Adafruit NeoPixel@1.5.0
Feature Dep Line:  Adafruit MAX31865 library@~1.1.0
Feature Dep Line:  U8glib-HAL@0.4.1
Feature Dep Line:  src_filter=+<src/lcd/dogm>
Feature Filter:  +<src/lcd/dogm>
Feature Dep Line:  LiquidCrystal@1.5.0
Feature Dep Line:  LiquidTWI2@1.2.7
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/ftdi_eve_touch_ui>
Feature Filter:  +<src/lcd/extui/lib/ftdi_eve_touch_ui>
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/dgus>
Feature Filter:  +<src/lcd/extui/lib/dgus>
Feature Dep Line:  src_filter=+<src/lcd/dwin>
Feature Filter:  +<src/lcd/dwin>
Feature Dep Line:  src_filter=+<src/lcd/menu>
Feature Filter:  +<src/lcd/menu>
Feature Dep Line:  AsyncTCP
Feature Dep Line:  ESP Async WebServer
Feature Dep Line:  ESP3DLib=https://github.com/luc-github/ESP3DLib.git
Feature Dep Line:  arduinoWebSockets=https://github.com/Links2004/arduinoWebSockets.git
Feature Dep Line:  ESP32SSDP=https://github.com/luc-github/ESP32SSDP.git
Feature Dep Line:  lib_ignore=ESPAsyncTCP
Feature Ignore:  ESPAsyncTCP
CC -D__MARLIN_FIRMWARE__ -DCORE_DEBUG_LEVEL=0 -w -dM -E -x c++ buildroot/share/PlatformIO/scripts/common-features-dependencies.h
Adding lib_deps for (ESP32_)?WIFISUPPORT... 
Ignoring libs:  ['Adafruit MAX31865 library', 'LiquidTWI2', 'LiquidCrystal', 'SlowSoftI2CMaster', 'TMCStepper', 'U8glib-HAL', 'Arduino-L6470', 'TMC26XStepper', 'MKS-LittlevGL', 'SailfishLCD', 'Adafruit NeoPixel']

Dependency Graph

|-- <AsyncTCP> 1.1.1
|-- <ESP Async WebServer> 1.2.3
|   |-- <AsyncTCP> 1.1.1
|   |-- <FS> 1.0
|   |-- <WiFi> 1.0
|-- <ESP3DLib> 1.0.0 #fd0f7b6
|   |-- <DNSServer> 1.1.0
|   |   |-- <WiFi> 1.0
|   |-- <ESP32SSPD> 1.1.0 #36420ca
|   |   |-- <WiFi> 1.0
|   |-- <ESPmDNS> 1.0
|   |   |-- <WiFi> 1.0
|   |-- <FS> 1.0
|   |-- <Preferences> 1.0
|   |-- <SPIFFS> 1.0
|   |   |-- <FS> 1.0
|   |-- <Update> 1.0
|   |-- <WebServer> 1.0
|   |   |-- <WiFi> 1.0
|   |   |-- <FS> 1.0
|   |-- <WebSockets> 2.2.1 #19c39d5
|   |   |-- <WiFi> 1.0
|   |   |-- <SPI> 1.0
|   |   |-- <WiFiClientSecure> 1.0
|   |   |   |-- <WiFi> 1.0
|   |-- <WiFi> 1.0
|   |-- <ArduinoOTA> 1.0
|   |   |-- <Update> 1.0
|   |   |-- <WiFi> 1.0
|   |   |-- <ESPmDNS> 1.0
|   |   |   |-- <WiFi> 1.0
|-- <WebSockets> 2.2.1 #19c39d5
|   |-- <WiFi> 1.0
|   |-- <SPI> 1.0
|   |-- <WiFiClientSecure> 1.0
|   |   |-- <WiFi> 1.0
|-- <ESP32SSPD> 1.1.0 #36420ca
|   |-- <WiFi> 1.0
|-- <SPI> 1.0
|-- <EEPROM> 1.0.3
|-- <ArduinoOTA> 1.0
|   |-- <Update> 1.0
|   |-- <WiFi> 1.0
|   |-- <ESPmDNS> 1.0
|   |   |-- <WiFi> 1.0
|-- <ESPmDNS> 1.0
|   |-- <WiFi> 1.0
|-- <WiFi> 1.0
|-- <FS> 1.0
|-- <SPIFFS> 1.0
|   |-- <FS> 1.0
|-- <Wire> 1.0.1
|-- <ESP32 BLE Arduino> 1.0.1

rhapsodyv · 2020-07-24T03:53:30Z

Added some extra logging, and it does seem to process the line…

It process, put that in FEATURE_DEPENDENCIES[feature]['lib_ignore'], but it isn't processed after by install_features_dependencies(). I just sent the missing code.

If there's a folder named ESPAsyncTCP in your .pio/lib_deps, it could enter in your dependency graph and get compiled, if the LDF "thinks" you referenced it.... So its the reason for the explicit ignore for all known libs.

thinkyhead · 2020-07-24T04:09:40Z

What are the different "lib_ldf_mode" values in PIO, and is one of them more suitable for our usage?

thinkyhead · 2020-07-24T04:11:41Z

Looks like off may be better for our purposes. It considers the manifests but doesn't try to be clever and follow #includes.

thinkyhead · 2020-07-24T04:47:47Z

Yeah, so lib_ldf_mode=off is probably great for tiny projects.

…aries (MarlinFirmware#18699)"""" This reverts commit 702ca04.

* Fix material preset type * Fix material preset editing * NO_LCD_REINIT for FSMC (or, no SD_DETECT_PIN) * Clean up preheat edit items * [cron] Bump distribution date (2020-06-26) * Minor LCD cleanup, improvement * Fix typo in DWIN, preheat array * Update Italian language (MarlinFirmware#18414) * [samd51] Manifest assigned timers priority (MarlinFirmware#18402) * Update MKS Robin Nano auto-build env (MarlinFirmware#18417) * SAMD51 cleanup (MarlinFirmware#18419) * BTT SKR Pro runout pins (MarlinFirmware#18416) * Update Brazilian Portuguese language (MarlinFirmware#18411) * Fix Fan Speed menu items (MarlinFirmware#18400) * Patch some DGUS warnings * Allow larger ADC debounce Reference MarlinFirmware#17205 * [cron] Bump distribution date (2020-06-27) * [cron] Bump distribution date (2020-06-28) * Update language fonts * Tool Change Migration fixes and debugging (MarlinFirmware#18448) * ExtUI additions (MarlinFirmware#18447) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Add Romanian language (MarlinFirmware#18455) Co-authored-by: cristyanul <47246451+cristyanul@users.noreply.github.com> * Fix "probing failed" false positives (MarlinFirmware#18435) * Adjustable delta_diagonal_rod_trim (MarlinFirmware#18423) * Russian, Ukranian for wide LCD (MarlinFirmware#18433) * Tweak pins formatting * Suppress unused var warning * ClosedLoop as singleton * [cron] Bump distribution date (2020-06-29) * MKS Robin Nano flash-based EEPROM (MarlinFirmware#18466) * Use "reset reason" defines (MarlinFirmware#18462) * [cron] Bump distribution date (2020-06-30) * [cron] Bump distribution date (2020-07-01) * Fix missing parenthesis (MarlinFirmware#18473) * Fix FYSETC CHEETAH V1.2 SD re-insert (MarlinFirmware#18474) Include this board with other Cheetah stepper reset. * Hide menu item with no fan (MarlinFirmware#18470) * Permit independent X2,Y2,Z2,Z3,Z4 endstop inverting (MarlinFirmware#18481) * Add TFT_LVGL_UI support (MarlinFirmware#18438) * Per-Hotend Default PIDs (MarlinFirmware#18483) * Enforce sensor range for temperature target (MarlinFirmware#18465) * Mitigate stepper timeout * Add CHAMBER PWM code * Structured thermistor tables * Fix reversed sensor ranges * Prevent temps outside sensor range * [cron] Bump distribution date (2020-07-02) * Tweak stepper shutdown test * Extend thermistor 1047 to 500°C * Updated lock / unlock actions * [cron] Bump distribution date (2020-07-03) * Keep filament runout pin for Creality Melzi (MarlinFirmware#18504) * Add FYSETC 2.42 inch OLED support (MarlinFirmware#18485) * Option to set manual Babystepping distance in mm (MarlinFirmware#18503) Co-authored-by: Scott Lahteine <github@thinkyhead.com> * Clean up LCD Manual Move / UBL Mesh Edit (MarlinFirmware#18373) * FYSETC OLED followup (MarlinFirmware#18519) * Ensure Git applies Unix line-endings in tests (MarlinFirmware#18495) * Fix axis name in serial output (MarlinFirmware#18522) * Fix env:mks_robin_nano35 (MarlinFirmware#18516) * FIx Sanguino/1284p board_upload.maximum_size (MarlinFirmware#18502) * M150 I to set Neopixel by index (MarlinFirmware#18490) * [cron] Bump distribution date (2020-07-04) * PID Accelerated Edit (MarlinFirmware#18532) * Patch Bad PR action * Two lock-closed per day * [cron] Bump distribution date (2020-07-05) * Fix "'ubl' not declared" error (MarlinFirmware#18541) * Fix thermistors exist-for-reading tests (MarlinFirmware#18533) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Fix "Probing mesh point" message Closes MarlinFirmware#17770 Co-Authored-By: Jan-Gerard van der Toorn <jan-gerard@users.noreply.github.com> * Limit actions to main * Probing points followup (MarlinFirmware#18552) * Update actions on bugfix branch * Add CI for pushed commits Co-Authored-By: thisiskeithb <13375512+thisiskeithb@users.noreply.github.com> * Add CI for pushed commits (MarlinFirmware#18549) * Fix Neopixel set_color (MarlinFirmware#18544) * Fix stall sensitivity adjustment for FTDI screens (MarlinFirmware#18554) * Minor pins cleanup * Add comment to M412 * Reduce string duplication * STM32: No compile-time check for PWM_PIN (MarlinFirmware#18539) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Filament Runout Inverting => State (MarlinFirmware#18537) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Simplify home_z_safely, respect HOME_AFTER_DEACTIVATE * [cron] Bump distribution date (2020-07-06) * Temporarily use patched lock-threads * Filament state followup * Get SAMD51 CXX flags from script * Use Material Preset 1 for PID autotune (MarlinFirmware#18446) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Fix broken POWER_LOSS_RECOVERY prompt (MarlinFirmware#18557) Co-authored-by: Scott Lahteine <github@thinkyhead.com> * Add Lerdge S,X,K (MarlinFirmware#18302) * Fix TMC homing phase coils alignment (MarlinFirmware#18528) Co-authored-by: Fabio Santos <fabiosan@live.com> * Scale runout distance setting for editable range (MarlinFirmware#18567) * 0.7.1 <= TMCStepper <= 0.7.9 (MarlinFirmware#18564) * Reduce 'first loop' temperature residency time (MarlinFirmware#18421) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Update and fix POWER_MONITOR (MarlinFirmware#18561) * [cron] Bump distribution date (2020-07-07) * Temp Residency followup * Whitespace cleanup * Combine command macros * Fix typo in Configuration_adv.h (MarlinFirmware#18570) * [cron] Bump distribution date (2020-07-08) * Tweak PLR debug * Fix warning with PIO * SKR 1.4 alert for unsupported LCD * Update Chinese (zh_CN) language (MarlinFirmware#18580) * Lock threads updated * Show fixed V in Power Display with no V sensor (MarlinFirmware#18579) * Fix User Temp Sensor (1000), reversed Pt100 / Pt1000 (MarlinFirmware#18590) * Fix some comments * Add `Cap:RUNOUT` Co-Authored-By: Julius ter Pelkwijk <mrseeker@users.noreply.github.com> * [cron] Bump distribution date (2020-07-09) * Fix missing controller fan include * 2.0.5.4 : Use ststm32 < 6.2 * Consolidate probe clearance, add section debug (MarlinFirmware#18576) * Better section / function log * Add do_z_clearance motion function * Remove outdated comment (MarlinFirmware#18597) * Clean up some MMU comments * MarlinUI support for up to 5 Material Presets (MarlinFirmware#18488) - Add `I` preset parameter to `G26`, `M106`, `M140`, and `M190`. - Extend menu items to permit a string interpolation. - Keep material names in a list and interpolate in menu items. - Extend material presets to support up to 5 predefined materials. Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * "M122 I" to reinitialize TMC (MarlinFirmware#18571) * [cron] Bump distribution date (2020-07-10) * Add current offset for POWER_MONITOR sensor (MarlinFirmware#18600) * Fix CMSIS / USB-FD regression (MarlinFirmware#18602) * Fix Chitu V6 with LVGL UI (MarlinFirmware#18608) * Fix CHITU_F103 pio.board name (MarlinFirmware#18486) * Capitalize "G-code" * [cron] Bump distribution date (2020-07-11) * Option to assist thermocouple debug (MarlinFirmware#18621) * Fix pid_debug_flag * Specify supported library versions * [cron] Bump distribution date (2020-07-12) * Clarify MICROSTEP_MODES * Option for extra endstop check (MarlinFirmware#18424) * Fix G35 output formatting (MarlinFirmware#18631) * [cron] Bump distribution date (2020-07-13) * LiquidCrystal@1.0.0 for LPC * Creality V4 cleanup * [cron] Bump distribution date (2020-07-14) * Fix / improve menu items (MarlinFirmware#18644) * LPC fix NeoPixel fork * Fix leveling "Point n of 3" message (MarlinFirmware#18639) * Increase STM32F1 Servo Timer Interrupt Priority (MarlinFirmware#18637) * [cron] Bump distribution date (2020-07-15) * Specify ststm32 @ 6.1.x * Tool-change debug option * Add REPORT_FAN_CHANGE as an option * Clean up DWIN code * General cleanup * Fix some Power Loss Recovery behaviors (MarlinFirmware#18558) * PLR followup * Fix Hotend Idle Timeout trigger (MarlinFirmware#18657) * [cron] Bump distribution date (2020-07-16) * Fix SKR 1.4 LCD / ADC Keypad sanity-check (MarlinFirmware#18670) * Broadcast host actions Fixes MarlinFirmware#18565 * Sanity-check BABYSTEP_MULTIPLICATOR_* * [cron] Bump distribution date (2020-07-17) * Fix material_preset reference issue * Chopper Timing Preset for 24V 0.9° (MarlinFirmware#18662) * Fix passive Chamber Temp reading (MarlinFirmware#18674) * Fix check_hotends with > 1 hotend * Tweak and use SD_ORDER * ExtUI for Anycubic I3 Mega (MarlinFirmware#18655) * Support FT81050 with FYSETC F6 (MarlinFirmware#18678) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Smaller SD EEPROM file on Ender 3 V2 * Use safe_delay for PSU_POWERUP_DELAY (MarlinFirmware#18680) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Fix E3V2 M0/M1 and wait_for_user * [cron] Bump distribution date (2020-07-18) * Chitu V6 - Stepper Z2 pins (MarlinFirmware#18683) * Fix DGUS write variable endianness (MarlinFirmware#18689) * Trigorilla Pro board (MarlinFirmware#18692) * Prettier BLTouch menu items (MarlinFirmware#18682) * Return to Status on PID Autotune (MarlinFirmware#18695) * Misc formatting, cleanup * [cron] Bump distribution date (2020-07-19) * More customizable DGUSDisplay (MarlinFirmware#18700) * Fix encoder reverse, partial steps handling (MarlinFirmware#18694) * Update Chinese (cn) language (MarlinFirmware#18705) * Fix endian DGUS WriteVariable (MarlinFirmware#18703) * Fix BLTouch PWM reliability in HAL/STM32 (MarlinFirmware#18702) * [cron] Bump distribution date (2020-07-20) * SAMD51: More reliable Servo/BLTouch PWM (MarlinFirmware#18710) * Only download & compile required libraries (MarlinFirmware#18699) * Reduce Step Smoothing ceiling to 50% CPU usage (MarlinFirmware#18719) * Fix some LVGL warnings * Patch auto-deps for Windows CXX (MarlinFirmware#18721) * [cron] Bump distribution date (2020-07-21) * Matching mat.bed_temp condition * Ignore unused (but downloaded) libraries (MarlinFirmware#18728) * Use development version of PlatformIO (MarlinFirmware#18724) * Fix build for Windows path edge case * Filter some unused Marlin/src subfolders (MarlinFirmware#18729) * New DGUS UI var / definition syntax (MarlinFirmware#18718) * Optimize LCD, Wifi, etc. libraries (MarlinFirmware#18730) * Fix inline AnycubicSerial method * Anycubic followup * [cron] Bump distribution date (2020-07-22) * Fix Spanish preheat strings (MarlinFirmware#18743) * Explicit SdFat * Custom Nozzle Wipe (MarlinFirmware#18736) * [cron] Bump distribution date (2020-07-23) * Linear Advance: Only change E DIR when needed (MarlinFirmware#18744) * Fix links to secure sites (MarlinFirmware#18745) * Fast Filament Change temperature check (MarlinFirmware#18738) * General cleanup * [cron] Bump distribution date (2020-07-24) * Update FYSETC AIO II TMC UART pins (MarlinFirmware#18749) * Update links, README, contributing, etc. * Update links, README, contributing, etc. * ibid. * Add CONTROLLER_FAN_IGNORE_Z (MarlinFirmware#18735) * Simplify encoder handling (MarlinFirmware#18754) * Fix ESP3D_WIFISUPPORT ini typo * Apply lib_ignore from [features] (MarlinFirmware#18762) * Ender 3 V2 BL24C16 EEPROM support (MarlinFirmware#18758) * Filament Runout pin for Creality v4.2.2 board (MarlinFirmware#18763) * [cron] Bump distribution date (2020-07-25) * Ender 3 V2: Inject 'G28 O' before G29 (MarlinFirmware#18767) * Fix Lerdge variant script (MarlinFirmware#18771) * G425/G35 menu items; G425 pre- and post- scripts (MarlinFirmware#18737) * Robin nano V2, TFT LVGL UI parameters, and more (MarlinFirmware#18500) * Clean up after changes * Update (c) * Fix ui.external_control compile error (MarlinFirmware#18778) * [cron] Bump distribution date (2020-07-26) * Fix search for best compiler (MarlinFirmware#18779) * Close host's Filament Change dialog (MarlinFirmware#18785) * More folders only compiled when their feature is Enabled (MarlinFirmware#18780) Co-authored-by: Scott Lahteine <thinkyhead@users.noreply.github.com> * Randomize firmware.bin, fix unflag * Fix MKS Robin Nano V2 build / upload (MarlinFirmware#18784) * Fix compile-time PWM_PIN (MarlinFirmware#18793) Co-authored-by: Scott Lahteine <github@thinkyhead.com> * Prevent ESP3D_WIFISUPPORT with non-ESP32 board (MarlinFirmware#18792) * Add 0 to EXTRUDERS :[json] (MarlinFirmware#18786) * [cron] Bump distribution date (2020-07-27) * Config for Anycubic Mega TFT * Marlin Release 2.0.6 * [cron] Bump distribution date (2020-07-28) * Fix WiFi / ESP32 sanity check (MarlinFirmware#18808) * Fix garbled print_xyz output (MarlinFirmware#18810) * Fix DELTA + TMC sensorless homing + SPI endstops (MarlinFirmware#18811) * DISABLE_DEBUG required to free Robin nano Z_DIR_PIN (MarlinFirmware#18832) * Fix NEOPIXEL_LED compile w/out PRINTER_EVENT_LEDS (MarlinFirmware#18824) * Add TEMP_PROBE_PIN for Einsy Rambo (MarlinFirmware#18823) * Fix Cancel Objects index display (zero-based) (MarlinFirmware#18841) * Fix UNKNOWN_Z_NO_RAISE in G28 Bug introduced in 73fc077 * In G28 Z is sort-of known * Update include_tree * Add monitor_flags * Fix Duplicator i3 Plus pin diagram (MarlinFirmware#18862) * Fix FLYF407ZG pins (MarlinFirmware#18865) * Fix Spindle/Laser PWM DC (MarlinFirmware#18871) * No mks_robin extra_scripts in Trigorilla build (MarlinFirmware#18872) * Fix Z height after tool change (MarlinFirmware#18951) * Move Cancel Object menu, fix canceled item index (MarlinFirmware#18930) * Update Italian language (MarlinFirmware#18886) * Update Slovak language (MarlinFirmware#18884) * HW PWM sanity checks for SPINDLE_LASER_FREQUENCY (MarlinFirmware#18947) * Fix IS_PROBE_PIN macro (MarlinFirmware#19024) Co-authored-by: Scott Lahteine <github@thinkyhead.com> Co-authored-by: thinkyhead <thinkyhead@users.noreply.github.com> Co-authored-by: Giuliano Zaro <3684609+GMagician@users.noreply.github.com> Co-authored-by: Keith Bennett <13375512+thisiskeithb@users.noreply.github.com> Co-authored-by: Johnny Eshak <info@johnnytheone.com> Co-authored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com> Co-authored-by: Jason Smith <jason.inet@gmail.com> Co-authored-by: InsanityAutomation <38436470+InsanityAutomation@users.noreply.github.com> Co-authored-by: cristyanul <47246451+cristyanul@users.noreply.github.com> Co-authored-by: JP Flouret <jflouret@microsoft.com> Co-authored-by: Fabio Santos <fabiosan@live.com> Co-authored-by: Serhiy-K <52166448+Serhiy-K@users.noreply.github.com> Co-authored-by: ellensp <ellensp@hotmail.com> Co-authored-by: George Fu <nailao_5918@163.com> Co-authored-by: Robby Candra <robbycandra.mail@gmail.com> Co-authored-by: Victor Oliveira <rhapsodyv@gmail.com> Co-authored-by: cbteeple <cbteeple@g.harvard.edu> Co-authored-by: notabucketofspam <43456540+notabucketofspam@users.noreply.github.com> Co-authored-by: Jan-Gerard van der Toorn <jan-gerard@users.noreply.github.com> Co-authored-by: Oliver Jean Eifler <olli.eifler@gmail.com> Co-authored-by: ManuelMcLure <manuel@mclure.org> Co-authored-by: GhostlyCrowd <jeffjiggens@gmail.com> Co-authored-by: Luc Hoang Long <i.me.mine@luchoanglong.com> Co-authored-by: J.C. Nelson <32139633+xC0000005@users.noreply.github.com> Co-authored-by: shuttercat <67816426+shuttercat@users.noreply.github.com> Co-authored-by: espr14 <espr14@gmail.com> Co-authored-by: cccc <cuiwei_cv@163.com> Co-authored-by: Julius ter Pelkwijk <mrseeker@users.noreply.github.com> Co-authored-by: Axel <ansepulveda@uc.cl> Co-authored-by: Speaka <48431623+Speaka@users.noreply.github.com> Co-authored-by: Chris Pepper <p3p@p3psoft.co.uk> Co-authored-by: Leo <leo@nutz.de> Co-authored-by: Evgeny Z <Evg33@users.noreply.github.com> Co-authored-by: Sergey1560 <53866542+Sergey1560@users.noreply.github.com> Co-authored-by: JBA <44487003+hub-jba@users.noreply.github.com> Co-authored-by: Markus Towara <mtowara@gmail.com> Co-authored-by: yufanyufan <yufanyufan@gmail.com> Co-authored-by: Claus Näveke <github@naeveke.de> Co-authored-by: cr20-123 <66994235+cr20-123@users.noreply.github.com> Co-authored-by: Pascal de Bruijn <pmjdebruijn@pcode.nl> Co-authored-by: Mark Langezaal <mark.langezaal@gmail.com> Co-authored-by: Ivan Kravets <me@ikravets.com> Co-authored-by: Diego von Deschwanden <68632259+Diegovd@users.noreply.github.com> Co-authored-by: fleek <flee7100@gmail.com> Co-authored-by: MKS-Sean <56996910+MKS-Sean@users.noreply.github.com> Co-authored-by: giryan <giryan@users.noreply.github.com> Co-authored-by: Victor Tseng <palatis@gmail.com> Co-authored-by: MoellerDi <MoellerDi@users.noreply.github.com> Co-authored-by: natemason <natemason@live.com.au> Co-authored-by: swissnorp <67485708+swissnorp@users.noreply.github.com> Co-authored-by: Sebastien Andrivet <sebastien@andrivet.com> Co-authored-by: FLYmaker <49380822+FLYmaker@users.noreply.github.com> Co-authored-by: Claus Näveke <nitek@blickt.es> Co-authored-by: cbaugher <kb9ydn@gmail.com> Co-authored-by: Roman Moravčík <roman.moravcik@gmail.com> Co-authored-by: Julius Mumme <jufimu12@gmail.com>

rhapsodyv and others added 6 commits July 18, 2020 17:50

Script that checks for enabled marlin features and install automatica…

f26453a

…lly all its dependencies

extraline

865b400

all chitu using dependency script

44e7044

small tweak

8ac171f

now it process the MarlinConfigPre.h, so we can check for even more f…

a0eee44

…eatures!! Like HAS_TRINAMIC, to add TMCStepper lib only when needed!

Merge branch 'features-and-dependencies' of github.com:rhapsodyv/Marl…

2561210

…in into features-and-dependencies

platform definition in the build flags

56bfd2e

thinkyhead force-pushed the features-and-dependencies branch from 187da5e to a9a1a39 Compare July 19, 2020 09:20

thinkyhead added 2 commits July 19, 2020 04:27

Remaining common lib_deps

ee4c9a2

Sanity, cleanup

033a170

thinkyhead force-pushed the features-and-dependencies branch from a9a1a39 to 033a170 Compare July 19, 2020 09:27

squish

9cf3290

rhapsodyv added 3 commits July 19, 2020 06:43

a new [features] section in the platformio.ini

afa9c9b

handle build_flags with values (tuples)

f492882

use the build flags of board.json too

216d115

rhapsodyv added 5 commits July 19, 2020 13:07

some builds dont have boards defined

c89f4a7

correct define

f34bb49

header file to get all marlin enabled features

c4d6b28

common.extra_scripts for every env

6f4c360

dont need build.extra_flags

ccda77e

rhapsodyv mentioned this pull request Jul 24, 2020

handle lib_ignore for features #18762

Merged

zvoniimiir added a commit to zvoniimiir/Marlin that referenced this pull request Jul 24, 2020

Revert "Revert "Revert "Revert "Only download & compile required libr…

b14e53b

…aries (MarlinFirmware#18699)"""" This reverts commit 702ca04.

This was referenced Jul 24, 2020

Dependency Ordering Fix #18770

Closed

Dependency Ordering Fix #18772

Closed

Build fail on Docker without GCC #18779

Merged

More folders only compiled when their feature is Enabled #18780

Merged

Marlor mentioned this pull request Aug 2, 2020

[BUG] Inconsistencies in autobuild/dependencies script #18900

Closed

rhapsodyv mentioned this pull request Aug 15, 2020

Fix build script for upcoming PIO 4.4 #19034

Merged

albertogg pushed a commit to albertogg/Marlin that referenced this pull request Aug 31, 2020

Only download & compile required libraries (MarlinFirmware#18699)

9ac6929

vgadreau pushed a commit to vgadreau/Marlin that referenced this pull request Dec 9, 2020

Only download & compile required libraries (MarlinFirmware#18699)

42ce28c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize build. Only used libraries downloaded & compiled. #18699

Optimize build. Only used libraries downloaded & compiled. #18699

rhapsodyv commented Jul 18, 2020 •

edited

Loading

rhapsodyv commented Jul 19, 2020 •

edited

Loading

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

thinkyhead commented Jul 19, 2020

thinkyhead commented Jul 19, 2020 •

edited

Loading

rhapsodyv commented Jul 19, 2020

thinkyhead commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020 •

edited

Loading

ellensp commented Jul 23, 2020

rhapsodyv commented Jul 23, 2020 •

edited

Loading

rhapsodyv commented Jul 23, 2020

mindspawn commented Jul 23, 2020

pfeerick commented Jul 23, 2020 •

edited

Loading

ivankravets commented Jul 23, 2020 •

edited

Loading

rhapsodyv commented Jul 23, 2020

rhapsodyv commented Jul 23, 2020

thinkyhead commented Jul 24, 2020 •

edited

Loading

rhapsodyv commented Jul 24, 2020

thinkyhead commented Jul 24, 2020 •

edited

Loading

rhapsodyv commented Jul 24, 2020 •

edited

Loading

rhapsodyv commented Jul 24, 2020

rhapsodyv commented Jul 24, 2020 •

edited

Loading

thinkyhead commented Jul 24, 2020

rhapsodyv commented Jul 24, 2020

thinkyhead commented Jul 24, 2020 •

edited

Loading

thinkyhead commented Jul 24, 2020

thinkyhead commented Jul 24, 2020

Optimize build. Only used libraries downloaded & compiled. #18699

Optimize build. Only used libraries downloaded & compiled. #18699

Conversation

rhapsodyv commented Jul 18, 2020 • edited Loading

Description

Benefits

Related Issues

Tasks

rhapsodyv commented Jul 19, 2020 • edited Loading

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

thinkyhead commented Jul 19, 2020

thinkyhead commented Jul 19, 2020 • edited Loading

rhapsodyv commented Jul 19, 2020

thinkyhead commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020

rhapsodyv commented Jul 19, 2020 • edited Loading

ellensp commented Jul 23, 2020

rhapsodyv commented Jul 23, 2020 • edited Loading

rhapsodyv commented Jul 23, 2020

mindspawn commented Jul 23, 2020

pfeerick commented Jul 23, 2020 • edited Loading

ivankravets commented Jul 23, 2020 • edited Loading

rhapsodyv commented Jul 23, 2020

rhapsodyv commented Jul 23, 2020

thinkyhead commented Jul 24, 2020 • edited Loading

rhapsodyv commented Jul 24, 2020

thinkyhead commented Jul 24, 2020 • edited Loading

rhapsodyv commented Jul 24, 2020 • edited Loading

rhapsodyv commented Jul 24, 2020

rhapsodyv commented Jul 24, 2020 • edited Loading

thinkyhead commented Jul 24, 2020

rhapsodyv commented Jul 24, 2020

thinkyhead commented Jul 24, 2020 • edited Loading

thinkyhead commented Jul 24, 2020

thinkyhead commented Jul 24, 2020

rhapsodyv commented Jul 18, 2020 •

edited

Loading

rhapsodyv commented Jul 19, 2020 •

edited

Loading

thinkyhead commented Jul 19, 2020 •

edited

Loading

rhapsodyv commented Jul 19, 2020 •

edited

Loading

rhapsodyv commented Jul 23, 2020 •

edited

Loading

pfeerick commented Jul 23, 2020 •

edited

Loading

ivankravets commented Jul 23, 2020 •

edited

Loading

thinkyhead commented Jul 24, 2020 •

edited

Loading

thinkyhead commented Jul 24, 2020 •

edited

Loading

rhapsodyv commented Jul 24, 2020 •

edited

Loading

rhapsodyv commented Jul 24, 2020 •

edited

Loading

thinkyhead commented Jul 24, 2020 •

edited

Loading