Use CMake

[protobits]

I'm opening this issue to discuss whether there would be interest in eventually migrating the build system to CMake. The motivation is in part a recent message by @dagar where he asked about this. But furthermore, the build system is sometimes quite a pain to maintain, and is extremely complex to fully understand.

The benefits of migrating towards CMake could be the following:

• Improved cross platform developer experience (Windows native build likely feasible)
• Significantly improved build times, both incremental and clean (supports not only Makefile generation, but ninja which is highly efficient for large parallel builds)
• Solves dependency handling on its own
• Better integration with 3rd party projects (consuming NuttX) and libs (eg LLVM libc++ cmake). Can also interact with pure Make based projects easily.
• Proper IDE support (VSCode, CLion, etc), nice things like Intellisense will "just work"
• Potential for tighter integration between configuration and build

Furthermore, it can help alleviate issues that we face:

• As CMake is not just a language but includes higher-level functionality (functions handling build targets, etc.) which structures the build in known ways, it can be more easily understood and maintained. Makefiles use hand-written rules which require explicit documentation (which does not really exist right now) for "the right way" to do things (how to structure a directory, which variables to define, in what order to add includes, etc) and is thus easier to break.
• Our dependency handling is brittle: it has been broken and fixed many times and requires much manual work (it still is not right, particularly for external projects)
• Multiplatform support requires separate code paths and approaches which need to be kept in sync
• Our build requires slow traversal of all sub-directories for various operations
• Object files dispersed over source directories. CMake builds in separate directory and opens the potential to having multiple builds of the same code base simultaneously.
• We have to deal with collision between object file names in different directories (currently handled by embedding path to file as part of object file name)

From the technical standpoint, migration would be indeed a lot of work but not necessarily hard. @dagar has already done prior work on this (on older version of NuttX). See here:

• https://github.com/PX4/NuttX/commits/pr-cmake
• https://github.com/PX4/NuttX-apps/tree/pr-cmake

In general, the approach is more or less like this:

1. Define a set of CMake utility functions, to define libraries and applications
2. Replace every Make.defs with a CMakeLists.txt, which defines the source files to be added from that directory
3. Define custom targets for things like menuconfig, etc.
4. Define custom commands for things like creating symlinks, etc.
5. Provide a top-level Makefile that simplifies the "mkdir build; cd build; cmake ..; make" procedure

The idea would be to do this work in a separate branch and initially reach a minimal POC (support one board for example, and most of the complexity such as different platforms, kernel build, etc.). This can then be iterated until the user experience is right and then finally extended to all boards/arch.

@dagar has offered to help and I would also like to do so. At the moment, we would like to assess whether the community at least feels this is worthy of an attempt so we can slowly start heading this direction (we could aim for a major release of NuttX, so this is not rushed either).

[xiaoxiang781216]

@davids5 do you have some performance number from PX4 CMake compared with NuttX Makefile?

[davids5]

@xiaoxiang781216 I have the slides from the 2019 Trip to meet with you. Shall I post them here?

[davids5]

From CI standpoint here are 73 build of NuttX (1.5M SLOC )including PX4 (6M SLOC) in 12m 13s done as work flows
https://github.com/PX4/PX4-Autopilot/actions/runs/722727878

[protobits]

Those are impressive times!

[dagar]

From CI standpoint here are 73 build of NuttX (1.5M SLOC )including PX4 (6M SLOC) in 12m 13s done as work flows
https://github.com/PX4/PX4-Autopilot/actions/runs/722727878

Current PX4/NuttX is a bit more complex because the underlying NuttX build is still with Make, but PX4 parallelizes it per NuttX library (via cmake).

The build history from my previous attempt (PX4/NuttX#46) has been lost, but it was drastically faster per build (multiple times faster).

EDIT: I will verify the numbers for current comparison, but recursive Makefile usage seems to be a huge contributor to the difference.

[xiaoxiang781216]

The number is great. The out of tree is also very useful.

[xiaoxiang781216]

@dagar I saw you already done a huge work((PX4/NuttX#46), I am postive to integrate your work into mainline to simplify the daily work in PX4 and NuttX community. @v01d should we come up a detailed propose and send to dev@ for discussion?

[protobits]

If you want you can send email to mailing list linking here, so that we do not split the discussion.
As for the detailed proposal, I briefly discussed it with @dagar and the result is in the description of this issue above. I wouldn't know what else to add. I think the preliminary work by @dagar already gives a good starting point (I would create a branch and start porting the work and adapting/improving as necessary).

[protobits]

@xiaoxiang781216 do you want to send an email linking to this issue?
I see some positive feedback already but it would be great to know others got the chance to read this.

[xiaoxiang781216]

Ok, I will send an email in this weekend.

[antmerlino]

I'm onboard with this. All I've wanted for years now was to be able to consume NuttX and NuttX-apps in a larger project directly from cmake. I mean, I do it now - but it's not clean by any means - mostly because of the in-source building. Out of tree building is probably the #1 benefit for doing this, IMO. But I think the benefits list is actually pretty long.

[davids5]

+1 It is a good move. Build speed will increase and we should be able to get CI times down to allow a review-merge cycle that does not span hours or days. We may also benefit from reducing the circular dependencies to a minimum. *From:* Anthony Merlino ***@***.*** *Sent:* Saturday, April 10, 2021 7:06 PM *To:* apache/incubator-nuttx *Cc:* David Sidrane; Mention *Subject:* Re: [apache/incubator-nuttx] Use CMake (#3455) I'm onboard with this. All I've wanted for years now was to be able to consume NuttX and NuttX-apps in a larger project directly from cmake. I mean, I do it now - but it's not clean by any means - mostly because of the in-source building. Out of tree building is probably the #1 <#1> benefit for doing this, IMO. But I think the benefits list is actually pretty long. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3455 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO3BXIK4INH3PCCOCBWGRLTID7X7ANCNFSM42O4HLAA> .

[protobits]

Given the positive feedback so far I decided to start building the POC in the hopes that this helps move the decision forward.
I'm basing my work on @dagar 's pr-cmake_master branch, which still is somewhat old w.r.t. current master. I already managed to succesfully build for a specific board (stm32f103-minimum) but lots of things still need to be done. It works quite well so far though.

I'll try to continue a bit more and maybe open a draft PR soon, mostly for visbility, as it will take quite some time to ensure this is mergeable (and even so, I would pospone merging until other big changes are done). I mostly added new files and not touched existing makefiles, which will allow for easy rebasing as time goes on. I also would like some input in some points I haven't yet addressed correctly.

[xiaoxiang781216]

Nice! I expect that many known issues will be resolved by migrating to CMake:

1. Support more host
2. Out of tree build
3. Fast build speed
4. Remove the duplicated setting in board's Make.defs
5. Various hack in Makefile(ar lock, long command line, file name mangle...)

[niurenyige]

really looking forward to that.

[easonxiang]

Any progress of this topic?
It is not easy to understand and maintain the current build system :(

[xiaoxiang781216]

The work is inactive for a while. From the feedback, the community want to support both method. Welcome to contrib.

[xiaoxiang781216]

cmake support merge to the master, let's close this issue now.