[New Feature]: Execution of available SBG worflows

[LucaCinquini]

Start executing on Airflow whatever SBG workflows are available by this time

Acceptance Criteria:
o Demonstrated execution of available SBG workflows triggered from the Airflow UI and/or Airflow API
o CI/CD pipeline for nightly execution of available SBG workflows

Dependencies:
From SE: provide one or more working SBG workflows

[LucaCinquini]

Starting to work on creating tasks that executes the steps as Docker containers invoked by the KubernetesPodOperator. In parallel, we will also work on executing the full workflow as a single CWL document.

[GodwinShen]

@mike-gangl I see we have this ticket planned for this sprint and @LucaCinquini listed the SBG workflow from SE team as a dependency. Given that we're 2 days from closing this current sprint, do we need to move this ticket out to 24.1.4?

[GodwinShen]

Moved this ticket to 24.1.4.

[LucaCinquini]

The issue is that those sprints need to be set separately for the U-SPS and Unity boards...

[LucaCinquini]

The SBG Preprocess workflow is now working end-to-end as a single CWL Task, tested in the mcp-venue-dev.
The remaining permission issues were solved by removing the option "--no-match-user" when invoking the cwl-runner tool.