Stability issue of nightly tests

[hohwille]

As PO of devonfw-ide, I want automated integration tests so that we can deliver quality and get feedback if some PR merge breaks our product by causing a bug.
Therefore we have implemented #691 and related stories.
The problem now is that we have added all available commands and that takes too long.
Github gives a lot of resources for free to support open-source projects.
Obviously there are limits and Microsoft can not give every project unlimited computing time (otherwise people would also mine bitcoins with GHA, etc.).
As it turns out, our tests run slow and after 6:00 hours github simply cancels the action what seems fair.

Questions:

1. Why does it take more than 6:00 h to run our tests? Are also the resources (CPU/bandwith/...) throttled so hard? I did not notice this so far. Maybe there is a problem on our side or one special commandlet is taking too long and test should be removed.
2. Should we focus on the most important commandlets if we have limits?
3. Could we find a way to run selected commandlets (e.g. by day of the week) so every commandlet is covered but not all in the same nightly-build?
4. Can we improve our logging so we can see easier how long the test took per command and what the status is? My initial idea was already that we have a final summary with all the tests we have run including their status (OK/NOK, duration). If we get cancelled by github should we print out this summary after each test so we spam the log more but also can see intermediate status?

[maybeec]

I would strongly recommend to run every install in a separate job using the dynamic matrix feature of github actions

https://github.com/devonfw-tutorials/tutorials/blob/main/.github/workflows/buildMain.yml#L24

[alfeilex]

I would strongly recommend to run every install in a separate job using the dynamic matrix feature of github actions

https://github.com/devonfw-tutorials/tutorials/blob/main/.github/workflows/buildMain.yml#L24

That's great suggestion. I see some difficulties with parallel execution of some commands. Currently the installation of each command runs in the ide that is build in test-setup. Some commands depend on other command... so there might be a conflict if e.g. azure (depends on python) and python command itself are run in parallel.

If we decide to use the dynamic matrix function, we should probably change our complete test structure.

[maybeec]

There are two questions here for me:

1. The question for me is here, whether the devonfw IDE would automatically install python in case you install azure client.
   If so, you can do both in parallel as every execution would run in a newly setup machine.
2. If the dependencies are not automatically installed, do you really need the dependencies for testing proper installation? Or is the installation crashing otherwise? It's about the question of installation verification which might look a little different depending on the capabilities of devonfw ide. But both is possible

[hohwille]

@maybeec 1. applies.
Besides waste (carbon footprint) there is nothing speaking against your suggestion to use dynamic matrix feature.

For the moment we have resolved the fundamental problems with PR #937.
Now some real bugs are revealed that we also fix.
Then we hope to see green nightly tests :)

To improve the stability the dynamic matrix feature is also promising to me.