Arch setup script #13111

MaEtUgR · 2019-10-07T13:35:34Z

Describe problem solved by the proposed pull request
A lot of new hardware doesn't work properly on Ubuntu out of the box. We should support Arch and Arch based distributions like Manjaro with a setup script.

Describe your preferred solution
I followed the clean ubuntu script in this repository and adopted it to the requirements on Arch.

Test data / coverage
I tested the script on a Vanialla Manjaro 18.0.1 (latest). Everything but gazebo works and is super quick to set up compared to Ubuntu. The step that takes longest is to load the downgraded arm-gcc which we are hopefully able to get rid of at some point.

Test on pure Arch is still missing, if someone has a quick way to check it I would appreciate.

Describe possible alternatives
I saw https://github.com/PX4/containers/blob/master/docker/px4-dev/Dockerfile_base-archlinux but e.g. it downloads arm-gcc using pacman which is super nice but PX4 doesn't compile on GCC 9 yet.

dagar · 2019-10-07T13:57:17Z

If we're going to carry scripts like this I'd like to use it all the way through for container setup, actual CI builds, and documentation. Otherwise these things tend to slowly rot off on the side.

jlecoeur · 2019-10-07T14:34:28Z

it downloads arm-gcc using pacman which is super nice but PX4 doesn't compile on GCC 9 yet.

Yeah things are updated quickly on ArchLinux. It would be best to install the same compiler on all linux platforms.

If we're going to carry scripts like this I'd like to use it all the way through for container setup, actual CI builds, and documentation.

I agree.
Can we create a repository "PX4/install_scripts" and add it as a submodule of the firmware, container, and docs repos?

dagar · 2019-10-07T14:41:22Z

Can we create a repository "PX4/install_scripts" and add it as a submodule of the firmware, container, and docs repos?

We can do that if it will help version and keep things in sync. Although now that I think about it again the Dockerfiles (from PX4/containers) could/should probably live in PX4/Firmware as well. That would leave the documentation, but it already has mechanisms to stay synced with PX4/Firmware for param and airframe metadata.

jlecoeur · 2019-10-07T14:55:48Z

OK, fewer submodules 👍

MaEtUgR · 2019-10-07T15:17:30Z

Although now that I think about it again the Dockerfiles (from PX4/containers) could/should probably live in PX4/Firmware as well.

While having them in the same spot, what about using the setup scripts also in the Dockerfiles? That would be less duplication and double verification of using the same in CI.

dagar · 2019-10-07T15:22:15Z

While having them in the same spot, what about using the setup scripts also in the Dockerfiles?

Yes that's exactly what I'd like to do.

hamishwillee · 2019-10-07T22:46:36Z

While having them in the same spot, what about using the setup scripts also in the Dockerfiles?

Yes that's exactly what I'd like to do.

We've been talking about that for over a year. How do we make that happen?

Docs currently point to correct version of ubuntu.sh script - ie v1.9.0 docs point to ubuntu.sh at that Firmware tag. Something to consider if we move the files.

julianoes

Nice!

Tools/setup/arch.sh

Tools/setup/ubuntu.sh

julianoes · 2019-10-08T07:28:07Z

Tools/setup/arch.sh

+	sudo usermod -aG uucp $USER
+
+	# Remove modem manager (interferes with PX4 serial port/USB serial usage).
+	sudo pacman -R modemmanager --noconfirm


This is evil. I vote we don't do that.

This is needed (and done) on ubuntu. So it may be evil, but consistently evil. Of course if the ubuntu and arch modem managers are different this could be pointless.

I am also curious to know why this is evil.

Because let's say you use modemmanager because you use LTE in your laptop and then the PX4 installer comes along and decides to just remove it.

The right thing to do is to make pull requests against modemmanager and add all IDs of all Pixhawks so that it ignores them.

The 3DR USB ID is ignored since 2014:
https://github.com/freedesktop/ModemManager/commit/9447527c357a91bbd3660274c41fd1b802e4cf84

We should only display it if and only if there is a port available but it's already occupied by someone else.

I put this to a test with a Pixhawk 4 and a CUAV V5 nano (both fmu-v5) and having the modemmanager (version 1.10.4 latest arch) installed clearly breaks a lot of flashing attempts. Especially every attmpt where you start the script and then power the board every single time. Here's the output of me repeating this case with both board alternately, removing modemmmanager and all subsequent tries are successful. I don't consider to leave it if the chance of successful flashing is that low.

[maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 003d00233138510638393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 3.9% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 003d00233138510638393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 0.0% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 003d00233138510638393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 0.0% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 002000223138510138393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 0.0% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 003d00233138510638393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 0.0% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 ========================================================================================================== WARNING: You should uninstall ModemManager as it conflicts with any non-modem serial device (like Pixhawk) ========================================================================================================== Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 002000223138510138393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [ ] 0.0% ERROR: timeout waiting for data (1 bytes) FAILED: platforms/nuttx/CMakeFiles/upload cd /home/maetugr/Firmware/build/px4_fmu-v5_default && /usr/bin/python /home/maetugr/Firmware/Tools/px_uploader.py --port "/dev/serial/by-id/*-3D_Robotics_PX4_FMU_v5.x*,/dev/serial/by-id/*PX4_BL*,/dev/serial/by-id/*BL_FMU*,/dev/serial/by-id/usb-The_Autopilot*,/dev/serial/by-id/usb-Bitcraze*,/dev/serial/by-id/pci-Bitcraze*,/dev/serial/by-id/usb-Gumstix*,/dev/serial/by-id/usb-Hex_ProfiCNC*,/dev/serial/by-id/usb-UVify_FMU_BL*" /home/maetugr/Firmware/build/px4_fmu-v5_default/px4_fmu-v5_default.px4 ninja: build stopped: subcommand failed. make: *** [Makefile:202: px4_fmu-v5] Error 1 [maetugr@ManjaroVM Firmware]$ sudo -R pacman modemmanager sudo: invalid option -- 'R' usage: sudo -h | -K | -k | -V usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user] usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user] [command] usage: sudo [-AbEHknPS] [-C num] [-g group] [-h host] [-p prompt] [-T timeout] [-u user] [VAR=value] [-i|-s] [<command>] usage: sudo -e [-AknS] [-C num] [-g group] [-h host] [-p prompt] [-T timeout] [-u user] file ... [maetugr@ManjaroVM Firmware]$ sudo pacman -R modemmanager [sudo] password for maetugr: checking dependencies... :: networkmanager optionally requires modemmanager: cellular network support Packages (1) modemmanager-1.10.4-1 Total Removed Size: 9.55 MiB :: Do you want to remove these packages? [Y/n] :: Processing package changes... (1/1) removing modemmanager [###########################################################################################] 100% :: Running post-transaction hooks... (1/5) Reloading system bus configuration... (2/5) Updating icon theme caches... (3/5) Reloading system manager configuration... (4/5) Reloading device manager configuration... (5/5) Arming ConditionNeedsUpdate... [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 002000223138510138393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [====================] 100.0% Verify : [====================] 100.0% Rebooting. Elapsed Time 28.415 [maetugr@ManjaroVM Firmware]$ make px4_fmu-v5 upload [0/4] Performing build step for 'px4io_firmware' ninja: no work to do. [3/4] uploading px4 Loaded firmware for board id: 50,0 size: 1670412 bytes (80.92%), waiting for the bootloader... Found board id: 50,0 bootloader version: 5 on /dev/serial/by-id/usb-3D_Robotics_PX4_BL_FMU_v5.x_0-if00 sn: 003d00233138510638393738 chip: 10016451 family: b'STM32F7[6|7]x' revision: b'Z' flash: 2064384 bytes Windowed mode: False Erase : [====================] 100.0% Program: [====================] 100.0% Verify : [====================] 100.0% Rebooting. Elapsed Time 28.433

Fair enough, just leave it in for now then.

When/if you remove this let me know? The ubuntu.sh removes it too, and should stay/go for same reasons.

As I understand this, it could cause a firmware update to fail. What would that look like to the user? [I'm thinking of adding this as a docs note]

Let's leave it in for now and check on it another day.

@hamishwillee it's already covered here: https://dev.px4.io/master/en/debug/faq.html#usb-errors

MaEtUgR · 2019-10-08T08:47:20Z

I quickly tried and the script in this pr is definitely docker suitable.
I ran this minimal docker file:

FROM archlinux/base:latest

# install sudo to run setup script which works outside docker
RUN pacman -Sy --noconfirm sudo

# Copy setup scripts into container
COPY . /setup

RUN source /setup/arch.sh

The only problem I ran into was the PATH for arm-gcc getting set in ~/.profile which apparently doesn't work in docker but I can fix that. Then it builds posix and nuttx just fine.

mrpollo · 2019-10-08T19:13:58Z

I can support this PR with a few dev hours to get the docker images and CI setup, can we make a todo list for tracking what is missing?

julianoes · 2019-10-09T13:49:52Z

What's wrong with the SITL mixer test?

MaEtUgR · 2019-10-09T14:09:57Z

What's wrong with the SITL mixer test?

It can't possibly be related. Let me rebase and fix the reboot messages, then CI runs again.

EDIT: everything passes...

MaEtUgR requested review from jlecoeur and julianoes October 7, 2019 13:35

MaEtUgR self-assigned this Oct 7, 2019

julianoes reviewed Oct 8, 2019

View reviewed changes

MaEtUgR added 4 commits October 9, 2019 15:20

ubuntu.sh: fix space and capitalization

274d819

Add setup script for Arch Linux

365b16c

arch.sh: remove numerous confirmations, added some base packages

3ee717a

setup scripts: adjust messages according to review

760a204

MaEtUgR force-pushed the arch-setup-script branch from 1ca08b5 to 760a204 Compare October 9, 2019 14:11

julianoes approved these changes Oct 10, 2019

View reviewed changes

julianoes merged commit 7f63ed8 into master Oct 10, 2019

julianoes deleted the arch-setup-script branch October 10, 2019 06:59

hamishwillee mentioned this pull request Oct 11, 2019

ubuntu.sh: No automatic removal of modemmanager #13156

Merged

This was referenced Oct 11, 2019

Arch Linux Setup: optionally install gazebo from AUR and fix pip Call #13160

Merged

Arch setup: Update how to use new setup script PX4/PX4-Devguide#878

Merged

weekly-digest bot mentioned this pull request Oct 13, 2019

Weekly Digest (6 October, 2019 - 13 October, 2019) #13171

Closed

MaEtUgR added Admin: Enhancement (improvement) 💡 Tools Sub-tools used within PX4 ecosystem (scripts, etc) labels Oct 13, 2019

MaEtUgR mentioned this pull request Oct 14, 2019

arch.sh: switch permissions of pip install (again) #13179

Merged

MaEtUgR mentioned this pull request Sep 9, 2021

Make Arch Linux setup script executable #18215

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arch setup script #13111

Arch setup script #13111

MaEtUgR commented Oct 7, 2019 •

edited

Loading

dagar commented Oct 7, 2019

jlecoeur commented Oct 7, 2019

dagar commented Oct 7, 2019 •

edited

Loading

jlecoeur commented Oct 7, 2019

MaEtUgR commented Oct 7, 2019 •

edited

Loading

dagar commented Oct 7, 2019

hamishwillee commented Oct 7, 2019

julianoes left a comment

julianoes Oct 8, 2019

hamishwillee Oct 8, 2019

jlecoeur Oct 8, 2019

julianoes Oct 8, 2019

julianoes Oct 8, 2019

julianoes Oct 9, 2019

MaEtUgR Oct 9, 2019 •

edited

Loading

julianoes Oct 9, 2019

hamishwillee Oct 10, 2019

julianoes Oct 10, 2019

MaEtUgR commented Oct 8, 2019 •

edited

Loading

mrpollo commented Oct 8, 2019

julianoes commented Oct 9, 2019

MaEtUgR commented Oct 9, 2019 •

edited

Loading

Arch setup script #13111

Arch setup script #13111

Conversation

MaEtUgR commented Oct 7, 2019 • edited Loading

dagar commented Oct 7, 2019

jlecoeur commented Oct 7, 2019

dagar commented Oct 7, 2019 • edited Loading

jlecoeur commented Oct 7, 2019

MaEtUgR commented Oct 7, 2019 • edited Loading

dagar commented Oct 7, 2019

hamishwillee commented Oct 7, 2019

julianoes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaEtUgR Oct 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaEtUgR commented Oct 8, 2019 • edited Loading

mrpollo commented Oct 8, 2019

julianoes commented Oct 9, 2019

MaEtUgR commented Oct 9, 2019 • edited Loading

MaEtUgR commented Oct 7, 2019 •

edited

Loading

dagar commented Oct 7, 2019 •

edited

Loading

MaEtUgR commented Oct 7, 2019 •

edited

Loading

MaEtUgR Oct 9, 2019 •

edited

Loading

MaEtUgR commented Oct 8, 2019 •

edited

Loading

MaEtUgR commented Oct 9, 2019 •

edited

Loading