[Master issue] Harden data pipeline infrastructure with extra goodies and cleanup like retries, unit testing, gtest junit xml support, and documentation

[tt-rkim]

Transferring cleanup list from #9319

Cleanup tasks

• More defensive programming / re-tries for missing logs / copied jobs for workflow run retries and artifact downloading since we can hang there / error out there because a job doesn't exist anymore
• Create unit tests and related test infrastructure for the data collection tools so we can have better refactoring flexibility later #12885
• gtest information support
• Fixing errors in the data:
  • Handle data collection cases where artifacts aren't carried over between attempts  #12840
  • re-runs can create new submission time stamps, but we still look for the earliest job in the earliest run, causing negative queue times
• Mypy / pydantic / strict type checking because we have so many random fields
• Investigate and fix high incidence of CI/CD data generation failure with JSON blob generation exception #12271
• Investigate using qooman/actions--context to get github_job_id instead of using the cursed uuid infrastructure #11711
• Upload benchmarking data for single-card demos to sftp server #10691
• Restrict workflow data collection to main branch for now and submit a data team ticket to have a branch field, and then use it in our collection process #11469
• Save environment data and upload benchmark data even when CI jobs fail (single-card demos, t3k demos) #11614
• Add git_branch_name field to pipeline for CI/CD once data team adds to schema #11704
• Update event data to use correct original event for a workflow_call instance rather than GITHUB_EVENT_NAME for cicd data #10533
• Add git_branch_name and github_job_run_link to environment.csv for benchmarks #10680
• De-duplicate environment CSV generation code so that we can have CI/CD information generation in one place #11134
• Align device_info fields with customer team's data #11137
• Add Blackhole as an option to infra/data_collection scripts #11238
• Cache environment for ci/cd infra workflow so it doesn't install every time #12877
• Add GitHub link to pipeline/job so it shows up in CI/CD data #12878
• Add other pipelines, such as T3K/TG/TGG, to ci/cd collection set #12879
• Extract test report files out of test container for Docker jobs so that we can upload them, as currently none are being uploaded after the Docker changes #12891
• Add handling in logs + artifacts download script for data collection for logs that don't exist #12966
• Fix no "properties" found on JUnit XML artifacts during ci/cd data collection for jobs that timed out #13419
• bug: https://github.com/tenstorrent/tt-metal/actions/runs/11111440141 w/ https://github.com/tenstorrent/tt-metal/actions/runs/11110261767/job/30868548829 (timeout in middle, so maybe no junit)
• bug: https://github.com/tenstorrent/tt-metal/actions/runs/11158826752 w/ https://github.com/tenstorrent/tt-metal/actions/runs/11159570104/job/31018206893
• bug: https://github.com/tenstorrent/tt-metal/actions/runs/11482896288
• Extend data collection unit test cases to increase coverage of corner cases #13116
• Add arch- tags to Galaxy workflows as they don't specify them and the CI/CD data relies on the labels being there #13285
• Add job-level error logging, such as for machine failures or GitHub internal errors #13363
• Change the trigger of the CI/CD data production to a webhook system, so we can properly capture all workflow runs rather than depending on the names with workflow_call #13380

[tt-rkim]

Looks like timeouts during pytests can cause no data to be loaded into the junit xml

Nahhhhh this is a different, check #12799 from now on

[tt-rkim]

Looks like re-runs don't repeat artifacts: https://github.com/tenstorrent/tt-metal/actions/runs/10908473854/

tried by: https://github.com/tenstorrent/tt-metal/actions/runs/10910636802/job/30281509160