Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add portable linux source build leg #75546

Merged
merged 15 commits into from
Oct 6, 2022
Merged

Add portable linux source build leg #75546

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
72625f7
Add portable linux source build leg
mmitche Sep 7, 2022
d8e009d
Invert conditional
mmitche Sep 13, 2022
eac193f
Set the platform name so that log artifacts don't overlap
mmitche Sep 13, 2022
3a2dca5
Try something to set asset manifest name
mmitche Sep 13, 2022
65ccd91
Merge remote-tracking branch 'upstream/main' into add-portable-linux-leg
mmitche Sep 20, 2022
6811513
Respond to feedback
mmitche Sep 20, 2022
a5bf093
Update arcade
mmitche Sep 20, 2022
cac3d1f
Merge remote-tracking branch 'upstream/main' into add-portable-linux-leg
mmitche Oct 3, 2022
1630623
Move insert to end
mmitche Oct 3, 2022
0e1fab4
Fix up non-portable parameter
mmitche Oct 3, 2022
bf026ca
Fix up target rid
mmitche Oct 3, 2022
e73ec54
Merge remote-tracking branch 'upstream/main' into add-portable-linux-leg
mmitche Oct 4, 2022
8e4039a
- Run only linux-x64 portable officially
mmitche Oct 4, 2022
84643a7
Move PR leg into dev-innnerloop
mmitche Oct 4, 2022
3b6b4d1
Remove helix group in innerloop
mmitche Oct 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions eng/pipelines/common/global-build-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ parameters:
shouldContinueOnError: false
dependOnEvaluatePaths: false
isOfficialBuild: false
buildingOnSourceBuildImage: false
isSourceBuild: false
isNonPortableSourceBuild: false
runtimeFlavor: 'coreclr'
runtimeVariant: ''
helixQueues: ''
Expand Down Expand Up @@ -113,9 +114,9 @@ jobs:
value: ''

- name: _sclEnableCommand
${{ if eq(parameters.buildingOnSourceBuildImage, true) }}:
${{ if eq(parameters.isSourceBuild, true) }}:
value: scl enable llvm-toolset-7.0 --
${{ if ne(parameters.buildingOnSourceBuildImage, true) }}:
${{ if ne(parameters.isSourceBuild, true) }}:
value: ''

- ${{ each variable in parameters.variables }}:
Expand All @@ -136,15 +137,15 @@ jobs:
artifact: Mono_Offsets_${{monoCrossAOTTargetOS}}
path: '$(Build.SourcesDirectory)/artifacts/obj/mono/offsetfiles'

- ${{ if eq(parameters.buildingOnSourceBuildImage, true) }}:
- ${{ if eq(parameters.isSourceBuild, true) }}:
- template: /eng/common/templates/steps/source-build.yml
parameters:
platform:
buildScript: $(_sclEnableCommand) $(Build.SourcesDirectory)$(dir)build$(scriptExt)
nonPortable: true
# Use a custom RID that isn't in the RID graph here to validate we don't break the usage of custom rids that aren't in the graph.
targetRID: banana.24-x64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You remove the custom RID, but we want to keep it to make sure that we can build with arbitrary RIDs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we probably need another leg, or one that only runs in PRs? Preference?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ViktorHofer @tmds @MichaelSimons I made some tweaks:

  • Build only linux-x64 portable officially (nothing else flows)
  • Add missing PR legs for linux-x64 (to avoid official breaks)
  • Add rolling CI for Banana and centos7

nonPortable: ${{ parameters.isNonPortableSourceBuild }}
targetRID: ${{ parameters.targetRid }}
runtimeOS: linux
name: ${{ parameters.platform }}

- ${{ if in(parameters.osGroup, 'OSX', 'iOS', 'tvOS', 'MacCatalyst') }}:
- script: $(Build.SourcesDirectory)/eng/install-native-dependencies.sh ${{ parameters.osGroup }} ${{ parameters.archType }} azDO
Expand All @@ -163,7 +164,7 @@ jobs:
filePath: $(Build.SourcesDirectory)/eng/pipelines/mono/update-machine-certs.ps1

# Build
- ${{ if eq(parameters.buildingOnSourceBuildImage, false) }}:
- ${{ if eq(parameters.isSourceBuild, false) }}:
- script: $(_sclEnableCommand) $(Build.SourcesDirectory)$(dir)build$(scriptExt) -ci -arch ${{ parameters.archType }} $(_osParameter) ${{ parameters.buildArgs }} $(_officialBuildParameter) $(_crossBuildPropertyArg) $(_cxx11Parameter) $(_richCodeNavigationParam) $(_buildDarwinFrameworksParameter) $(_overrideTestScriptWindowsCmdParameter)
displayName: Build product
${{ if eq(parameters.useContinueOnErrorDuringBuild, true) }}:
Expand Down
55 changes: 53 additions & 2 deletions eng/pipelines/common/platform-matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,57 @@ jobs:
disableClrTest: true
${{ insert }}: ${{ parameters.jobParameters }}

# Linux x64 Source Build
# Centos 7 x64 Source Build

- ${{ if containsValue(parameters.platforms, 'SourceBuild_Centos7_x64') }}:
- template: xplat-setup.yml
parameters:
jobTemplate: ${{ parameters.jobTemplate }}
helixQueuesTemplate: ${{ parameters.helixQueuesTemplate }}
variables: ${{ parameters.variables }}
osGroup: Linux
archType: x64
targetRid: centos.7-x64
platform: Centos7_Linux_x64
shouldContinueOnError: ${{ parameters.shouldContinueOnError }}
container:
image: centos-7-source-build-20210714125450-5d87b80
registry: mcr
jobParameters:
runtimeFlavor: ${{ parameters.runtimeFlavor }}
stagedBuild: ${{ parameters.stagedBuild }}
buildConfig: ${{ parameters.buildConfig }}
helixQueueGroup: ${{ parameters.helixQueueGroup }}
isSourceBuild: true
isNonPortableSourceBuild: true
${{ insert }}: ${{ parameters.jobParameters }}
mmitche marked this conversation as resolved.
Show resolved Hide resolved

# Non-existent RID Source Build

- ${{ if containsValue(parameters.platforms, 'SourceBuild_Banana24_x64') }}:
- template: xplat-setup.yml
parameters:
jobTemplate: ${{ parameters.jobTemplate }}
helixQueuesTemplate: ${{ parameters.helixQueuesTemplate }}
variables: ${{ parameters.variables }}
osGroup: Linux
archType: x64
targetRid: banana.24-x64
platform: Banana24_Linux_x64
shouldContinueOnError: ${{ parameters.shouldContinueOnError }}
container:
image: centos-7-source-build-20210714125450-5d87b80
registry: mcr
jobParameters:
runtimeFlavor: ${{ parameters.runtimeFlavor }}
stagedBuild: ${{ parameters.stagedBuild }}
buildConfig: ${{ parameters.buildConfig }}
helixQueueGroup: ${{ parameters.helixQueueGroup }}
isSourceBuild: true
isNonPortableSourceBuild: true
${{ insert }}: ${{ parameters.jobParameters }}

# Portable Linux x64 Source Build

- ${{ if containsValue(parameters.platforms, 'SourceBuild_Linux_x64') }}:
- template: xplat-setup.yml
Expand All @@ -300,8 +350,9 @@ jobs:
stagedBuild: ${{ parameters.stagedBuild }}
buildConfig: ${{ parameters.buildConfig }}
helixQueueGroup: ${{ parameters.helixQueueGroup }}
isSourceBuild: true
isNonPortableSourceBuild: false
${{ insert }}: ${{ parameters.jobParameters }}
buildingOnSourceBuildImage: true

# Linux s390x

Expand Down
12 changes: 5 additions & 7 deletions eng/pipelines/global-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,11 @@ jobs:
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
buildConfig: release
buildConfig: Release
platforms:
- SourceBuild_Linux_x64
jobParameters:
nameSuffix: SourceBuild
timeoutInMinutes: 120
condition:
or(
eq(dependencies.evaluate_paths.outputs['SetPathVars_non_mono_and_wasm.containsChange'], true),
eq(variables['isRollingBuild'], true))
nameSuffix: PortableSourceBuild
extraStepsParameters:
name: SourceBuildPackages
timeoutInMinutes: 95
6 changes: 1 addition & 5 deletions eng/pipelines/runtime-official.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,9 +364,6 @@ stages:
runTests: false
useHelix: false

#
# Build Sourcebuild leg
#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
Expand All @@ -375,13 +372,12 @@ stages:
platforms:
- SourceBuild_Linux_x64
jobParameters:
nameSuffix: SourceBuild
nameSuffix: PortableSourceBuild
extraStepsTemplate: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
extraStepsParameters:
name: SourceBuildPackages
timeoutInMinutes: 95


#
# Installer Build
#
Expand Down
38 changes: 38 additions & 0 deletions eng/pipelines/runtime.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1296,3 +1296,41 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_coreclr.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
eq(variables['isRollingBuild'], true))

#
# Sourcebuild legs
# We have 3 important legs for source-build:
# - Centos.7 (ensures that RH keeps working)
# - Linux-x64 portable (used for dependency flow and downstream PR verification)
# - Banana.24 - Non-existent RID to ensure we don't break RIDs we don't know about.
#
# Running all of these everywhere is wasteful. Run Banana.24 and CentOS.7 in rolling CI,
# Run Linux-x64 in PR.

- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
buildConfig: Release
helixQueueGroup: pr
platforms:
- SourceBuild_Centos7_x64
jobParameters:
nameSuffix: Centos7SourceBuild
extraStepsParameters:
name: SourceBuildPackages
timeoutInMinutes: 95
condition: eq(variables['isRollingBuild'], true)

- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
buildConfig: Debug
helixQueueGroup: pr
platforms:
- SourceBuild_Banana24_x64
jobParameters:
nameSuffix: Banana24SourceBuild
extraStepsParameters:
name: SourceBuildPackages
timeoutInMinutes: 95
condition: eq(variables['isRollingBuild'], true)