building STM32 with CMake

[massonal]

Hello,

As I said in my introduction message, I am currently working on the CMake support for STM32 boards. I have been told this was a feature some of you had requested, to overcome limitations of Arduino IDE (e.g., long build time...).
As this project is coming into beta, I would really like to get some feedback on it, in terms of features, performance, usability...

Here are the various entry points to get you started:

• project's README
• project's wiki
• tutorial-examples

Although being comfortable with CMake would be useful when using this project, it is not really a requirement: I have done my best to abstract any CMake-related complexity away.

The setup instructions are listed on a dedicated wiki page; briefly, you'll need CMake, Ninja, Python, plus a few Python modules to get running.

What is currently implemented:

• Building a sketch for any board the STM32 core supports, _quickly_ (about 10s on my machine);
• All the feature switches present in the Arduino IDE (and more!);
• Reading board definitions from boards.txt;
• Third-party Arduino libraries (typically in ~/Arduino/sketchbook);

The following is _not_ supported (yet?):

• Uploading;
• Third-party bootloaders (Maple DFU Bootloader, HID Bootloader 2.2);
• Library management (download, update...);
• Automatic library dependency detection (you have to state explicitly that you use, e.g., the Wire library);

[ag123]

cool, would be trying it out
3rd party bootloaders and utilities (e.g. flash utilities) can possibly be added along the way.
CMake adds more flexibility than do the classical 'arduino ide'.
The thing is the CMake files are configurations and macros which makes this more feasible.
And it allows 'swapping' of libraries, e.g. by the different series. and possibly boards and variants

[massonal]

cool, would be trying it out
3rd party bootloaders and utilities (e.g. flash utilities) can possibly be added along the way.
CMake adds more flexibility than do the classical 'arduino ide'.
The thing is the CMake files are configurations and macros which makes this more feasible.
And it allows 'swapping' of libraries, e.g. by the different series. and possibly boards and variants

Thanks for giving it a try
I see we have the same view on the benefits of CMake vs Arduino IDE. I would add that CMake is much faster at building than Arduino IDE, at least under Windows. (I just ran a benchmark, got ca. -60% build time using CMake vs arduino-cli on my machine !)

Regarding the bootloader support: I shall consider integrating this feature, but some of the changes it carries may not be CMake-friendly (e.g., selecting the Serial port the board is on...).
I am also working on the integration with IDEs, maybe this is something that can be delegated to them? Considering for example ST's Cube IDE, or Cube2. This would also make sense regarding debuggers.
Anyway this topic will definitely get discussed on my end; I'll let you know what comes out of it.

[ag123]

oops, I'm kind of procrastinating a little as I've not updated/refreshed my core for a while.
I very much prefer a CMake build, in fact currently I'm using a Makefile build which is worse.
CMake would help a lot to standardize/organize things on how to link the different series, variants, boards, libraries, etc.
much neater and the community would benefit a lot from that

I'd try to make time perhaps this week, do an update for a built.
I'm working in Linux.

Going for CMake has important implications, it would make it easier for projects like Marlin
https://marlinfw.org/
to integrate the standard core and you can imagine all the '3d printers' would likely base off this.
It would be good to organize this well so that this could become a 'build framework' sort of.

[ag123]

I just tried it out (in linux), really cool

it seemed the boards db file is generated? it is huge
https://github.com/massonal/Arduino_Cor ... s_db.cmake
would be good to document the generation steps say after adding a custom board.

oh, and the wiki is nice
https://github.com/massonal/Arduino_Core_STM32/wiki
I'd guess it would be merged in the main wiki in its own section when the commits are merged.

it is also rather interesting that I noted in each variant directory, e.g. for F401xx
https://github.com/massonal/Arduino_Cor ... -D-E)(U-Y)
it needs its own CMakeList.txt file
https://github.com/massonal/Arduino_Cor ... eLists.txt
that's 'ok' as after all a variant is a 'library'

a thing though would cmake/scripts/cmake_easy_setup.py
https://github.com/massonal/Arduino_Cor ... E_CMAKE.md
be able to provide a board list?
that would help with the -b board parameter in case of a mismatch.

it seemed the '2 level' (or in fact 3 level) series - board namespace is simplified to a single list?
I'd guess it is sufficiently unique for now.

the speed up is *considerable*
i think it is partly because cmake narrows the build to a particular variant/board and possibly even a particular series.
the readme is a good starting point
https://github.com/massonal/Arduino_Cor ... E_CMAKE.md
it skipped over huge swatches of ifdefs and just build what is needed.

I'm thinking a "GUI" would help, that could perhaps be done say using a 'web based' setup
e.g. using Flask
https://flask.palletsprojects.com/en/2.2.x/
there are better python frameworks
e.g. Django
https://www.djangoproject.com/
that's probably not a priority for now, kind of a 'nice to have'. it makes generating CMakeList.txt for sketches "less intimidating"

I'm still exploring the features, and logging off for now. Just managed to piece things together, literally build Python3.10 from source.
That in my linux distribution is too old. For Windows users, python offers binaries, so that is easier.
https://www.python.org/downloads/
i used the unix makefile instead of ninja, it works just well.
cmake provides binaries for various os
https://cmake.org/download/
the readme is a good starting point
https://github.com/massonal/Arduino_Cor ... E_CMAKE.md

[ag123]

a little note for those less familiar with these 'unix' stuff.
it is kind of a "2 step" build process.

1 - cmake/scripts/cmake_easy_setup.py -b <board> -s <sketch folder>
did a good job of generating the CMakeList.txt file. - this is a "configuration" file as input to CMake.
Normally, one would need to code this CMakeList.txt file manually.
2. CMake generates another (set of) configuration (makefile) file(s) - this is processed by either ninja or (unix) make
This 2nd step is the actual thing that drive the build

the 2 step process makes it possible to generate lean configuration (makefile) files, I'd guess is a factor that provides the speedup.
and the 1st step adds the flexibility to 'swap configuration' e.g. change variants, add libraries etc.

the other thing I prefer about this is that the build configuration CMakeList.txt and/or makefiles is stored with the project.
More often than not a sketch/app is associated with a particular board and/or set of libraries.
Arduino kept that separate, and hence one often need to track the settings separately.

much earlier, I build the sketches in Eclipse CDT, no Arduino add ons, the individual variables, paths and flags needs to be set manually in each project.
it turns out there are more than a hundred of the disparate variables, paths, flags for each project. it is simply too tedious and I end up resorting to Makefile builds, which is the 2nd step coded manually.

[massonal]

Wow, this had you talk a lot!
I'm glad you enjoyed it overall.

Just before anything else, you built Python from source for this?
I should point people to https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa in the wiki, that might have saved the trouble. ("This PPA contains more recent Python versions packaged for Ubuntu." -- down to 18.04)

it seemed the boards db file is generated?

Yes, indeed. It is updated automatically from boards.txt when calling `set_board()`, and also manually during the update process.
I will reference this in the wiki.



I already had some feedback on the UI (board list, GUI). This seems to be a popular request!
I'm not sure about a dedicated GUI -- there are already generic GUIs for CMake, plus the IDEs... (I am working on IDE integration rn.) If it would be done, I agree on the solution: web-based with Python seems the most ergonomic and portable way to do it.
On the other hand, a prompt for the -b flag of cmake_easy_setup would be a nice fit.
For the moment, the point of this script was to fill placeholders only, without any verification. CMake would later complain if the board did not exist in the database (after updating it, as needed). Adding some early checks in the script is definitely possible, too, though. I shall implement that in the coming days.

I just wanted to stress that the CMakeLists.txt that cmake/scripts/cmake_easy_setup.py generates should always be proofread -- to fill in the dependencies, at least. As I explained in the wiki, this can hardly be automated with CMake, given the way Arduino does it.



Anyway, thanks for all the feedback! Be sure to keep me updated if you have any more

[ag123]

I'm trying to get up to speed (I'm learning) with CMake as well.
a thing about CMake is, "it isn't very well documented".
A lot of things are documented as single line examples (which show its relative effectiveness to reduce verbose makefile configuration codes).
But it leaves a lot of guesswork about the implication of those codes, i.e. what does it generate?

I've been looking at examples, and it seemed every *single source file* dependency need to be declared in a

Code: Select all

add_library( ....);
add_executable(....);
target_link_libraries(....);

statements.

but it turns out that the recent CMake versions is able to do wildcard glob patterns and directory recursion
https://stackoverflow.com/questions/320 ... sing-cmake
https://cmake.org/cmake/help/latest/command/file.html
In my Makefile build
viewtopic.php?f=41&t=183
I used all these wildcard pattern matching tricks and recursion and condensed the build into a single unix Makefile.
While some may think it isn't too structured, the idea of all these recursion and wildcard pattern matching globs to include whole subdirectory hierarchy is a necessary 'convenience' as otherwise there would be makefiles littered in all the subdirectories.
For CMake, i.e. CMakeList.txt I think we can organize them into modules or 'libraries' where each independent module can consist its own directory sub-trees. This is very much the case in our world of rather complex apps, and libraries e.g. SD Card libraries with FSFat drivers etc.

oh as I searched the web for tutorials, docs and materials, I came across some rather useful sites/pages/articles
i found this nice online ebook : modern cmake.
https://cliutils.gitlab.io/modern-cmake/

building with CMake - this is a real STM32 based example, I'd think HAL etc
https://blog.feabhas.com/2021/07/cmake- ... dark-arts/
https://blog.feabhas.com/2021/07/cmake- ... ug-builds/
https://blog.feabhas.com/2021/08/cmake- ... anisation/

more links from google searches
https://matgomes.com/header-include-directories-cmake/
https://cmake.org/cmake/help/latest/gui ... index.html
https://courses.grainger.illinois.edu/c ... tes/cmake/

lets have a dedicated page in a wiki where we link tutorials, docs, references, articles etc that are relevant to CMake for those wanting to learn about it.
CMake is a pretty complex build system as over the years, with new versions, the features actually changes.
The recent versions has features where older versions lack, e.g. that directory recursion and wildcard pattern globing, I think it is pretty important.

[ag123]

Wow, this had you talk a lot!
I'm glad you enjoyed it overall.

Just before anything else, you built Python from source for this?
I should point people to https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa in the wiki, that might have saved the trouble. ("This PPA contains more recent Python versions packaged for Ubuntu." -- down to 18.04)

oops, I ran an old OpenSuse distribution that has not been updated for several years, changing things is tedious as there are lots of apps installed.
The bulk majority of 'Windows' users I'd guess may have an easier time as python.org and cmake.org distributes binaries for the latest releases.

it seemed the boards db file is generated?

Yes, indeed. It is updated automatically from boards.txt when calling `set_board()`, and also manually during the update process.
I will reference this in the wiki.

I already had some feedback on the UI (board list, GUI). This seems to be a popular request!
I'm not sure about a dedicated GUI -- there are already generic GUIs for CMake, plus the IDEs... (I am working on IDE integration rn.) If it would be done, I agree on the solution: web-based with Python seems the most ergonomic and portable way to do it.
On the other hand, a prompt for the -b flag of cmake_easy_setup would be a nice fit.
For the moment, the point of this script was to fill placeholders only, without any verification. CMake would later complain if the board did not exist in the database (after updating it, as needed). Adding some early checks in the script is definitely possible, too, though. I shall implement that in the coming days.

I just wanted to stress that the CMakeLists.txt that cmake/scripts/cmake_easy_setup.py generates should always be proofread -- to fill in the dependencies, at least. As I explained in the wiki, this can hardly be automated with CMake, given the way Arduino does it.

I think it may be good to emit a message e.g. just text to say that the generated CMakeLists.txt normally address simple "hello world" or "blink" style template apps and should be proofread manually.
in addition, I still think having an option that just print a list of valid boards from the db helps, at least users can select the board from the list and specify it in the parameter while running cmake_easy_setup.py


oh, I think there should be a "install" page in the wiki
https://github.com/massonal/Arduino_Core_STM32/wiki
where it is basically the contents of
https://github.com/massonal/Arduino_Cor ... E_CMAKE.md
the prerequisites and the places to get them
e.g.

• https://www.python.org/downloads/
• https://cmake.org/download/
• https://ninja-build.org/
  https://github.com/ninja-build/ninja

etc

[massonal]

oh, I think there should be a "install" page in the wiki
https://github.com/massonal/Arduino_Core_STM32/wiki
where it is basically the contents of
https://github.com/massonal/Arduino_Cor ... E_CMAKE.md
the prerequisites and the places to get them

How about https://github.com/massonal/Arduino_Cor ... wiki/Setup ? or https://github.com/massonal/Arduino_Cor ... tart-guide ? Are you suggesting these get merged? (given that the latter links to the former)

in addition, I still think having an option that just print a list of valid boards from the db helps, at least users can select the board from the list and specify it in the parameter while running cmake_easy_setup.py

Maybe it wasn't clear earlier, I actually agree with that. I just implemented it, by the way: if the board name is passed incorrect, the script runs a fuzzy finder and prompts the user for likely matches. This works with partial names also, e.g., "NUCLEO", "BLUEPILL"...
https://github.com/massonal/Arduino_Cor ... mit/175fe4

lets have a dedicated page in a wiki where we link tutorials, docs, references, articles etc that are relevant to CMake for those wanting to learn about it.
CMake is a pretty complex build system as over the years, with new versions, the features actually changes.
The recent versions has features where older versions lack, e.g. that directory recursion and wildcard pattern globing, I think it is pretty important.

Agreed! I put it here, with the links you shared and some of my own: https://github.com/massonal/Arduino_Cor ... r-readings
It's not a full page, just a section, but I think that's enough for now.