Enable Security and Compliance tasks in our Release pipeline #11849

miniksa · 2021-11-30T23:17:13Z

Enables a series of tasks run against our release pipeline that validate the security and compliance status of our code in an automated fashion. These checks include:

Component Governance - (we had this one, it was moved to here) - Inventories open-source components used in our build
PREfast - C/C++ static analysis for common code errors and exploits
Policheck - Searches source code, comments, and text for words that could be sensitive legally, culturally, or geopolitically
Credscan - Looks for credentials left behind in the code/documents and build output files
BinSkim - Searches for common vulnerabilities in binaries
CheckCFlags - Validates that compile/link flags match the policies recommended by Windows engineering for inclusion into the OS product image
CFGCheck/XFGCheck - Validates that the CFG and/or XFG settings were enabled at compile and link time to guard against control flow attacks.

We're also required to run the SBOM one, but that was done in a separate PR and we're still pending the detectors being updated.

References

Move from CFG to XFG when XFG works and is mandatory #11948 - Move from CFG to XFG once XFG task folks get back to me on it
Enable Bug Filing for SecComp tasks #11949 - Enable bug filing for SecComp tasks
Bulk process bugs filed by SecComp tasks, complete compliance work #11950 - Bulk process bugs filed by SecComp tasks
Validate SBOM once detectors are fixed #11947 - Validate SBOM when checkers come online

Checklist

- Fixes Insert Windows Engineering C++ Compliance Task into Pipelines #10735
- Fixes Feature Request: Enable BinSkim for making sure our binaries are good #908
- I work here
- If it fits, it sits.

… use recycled artifacts to speed this up

… (which I was supposed to move to anyway) uses x64 by default.

… the heap thing.

miniksa · 2021-12-15T21:03:14Z

src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp

+template<typename T>
+static bool isProfilesDefaultsOrigin(const T& profile)
+{
+    return profile && profile.Origin() != winrt::Microsoft::Terminal::Settings::Model::OriginTag::ProfilesDefaults;
+}
+
+template<typename T>
+static bool isProfilesDefaultsOriginSub(const T& sub)
+{
+    return sub && isProfilesDefaultsOrigin(sub.SourceProfile());
+}
+


This change was because of an internal compiler error. I have a thread out to the team to hopefully debug and fix the compiler. It couldn't handle the other form.

miniksa · 2021-12-15T21:03:30Z

src/common.build.pre.props

@@ -168,6 +168,7 @@
      <IntrinsicFunctions>true</IntrinsicFunctions>
      <PreprocessorDefinitions>NDEBUG;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <RuntimeTypeInfo>false</RuntimeTypeInfo>
+      <AdditionalOptions>%(AdditionalOptions)</AdditionalOptions>


Please ignore this. It will make it easier for me to move to XFG later as more flags go here.

miniksa · 2021-12-15T21:08:08Z

build/pipelines/release.yml

  cancelTimeoutInMinutes: 1
  steps:
  - checkout: self
    clean: true
-    fetchDepth: 1


PGO can't resolve from peer branches with depth 1 here, so I'll allow it for release builds.

build/config/PolicheckExclusions.xml

build/pipelines/release.yml

zadjii-msft

yea I'm fine with this. Dustin wanted it moved to another file (stages.yml or something), but like, good enough for me.

zadjii-msft · 2022-01-03T22:21:00Z

build/pipelines/release.yml

+        teamId: 7105
+        authId: $(TouchdownAppId)
+        authKey: $(TouchdownAppKey)
+        resourceFilePath: >-


I don't love that we've got these Resources.resw paths in here all hardcoded, kinda just begging for us to forget to update this list in the future, but ¯\_(ツ)_/¯

That's how they were... We can maybe improve that later.

Yep. I learned afterwards that we can wildcard them!

Honestly doing compliance probably doesn't need loc resources.. but np

miniksa · 2022-01-03T23:07:22Z

From the meeting, @DHowett wants me to pull the compliance one into a template of steps and then link it here.

miniksa · 2022-01-03T23:07:40Z

/azp run

azure-pipelines · 2022-01-03T23:07:53Z

Azure Pipelines successfully started running 1 pipeline(s).

ghost · 2022-01-05T18:44:54Z

Hello @miniksa!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

Enables a series of tasks run against our release pipeline that validate the security and compliance status of our code in an automated fashion. These checks include: - Component Governance - (we had this one, it was moved to here) - Inventories open-source components used in our build - PREfast - C/C++ static analysis for common code errors and exploits - Policheck - Searches source code, comments, and text for words that could be sensitive legally, culturally, or geopolitically - Credscan - Looks for credentials left behind in the code/documents and build output files - BinSkim - Searches for common vulnerabilities in binaries - CheckCFlags - Validates that compile/link flags match the policies recommended by Windows engineering for inclusion into the OS product image - CFGCheck/XFGCheck - Validates that the CFG and/or XFG settings were enabled at compile and link time to guard against control flow attacks. We're also required to run the SBOM one, but that was done in a separate PR and we're still pending the detectors being updated. - #11948 - Move from CFG to XFG once XFG task folks get back to me on it - #11949 - Enable bug filing for SecComp tasks - #11950 - Bulk process bugs filed by SecComp tasks - #11947 - Validate SBOM when checkers come online - [x] - Fixes #10735 - [x] - Fixes #908 - [x] - I work here - [x] - If it fits, it sits.

ghost · 2022-02-03T18:23:45Z

🎉Windows Terminal Preview v1.13.10336.0 has been released which incorporates this pull request.:tada:

Handy links:

miniksa added 30 commits November 30, 2021 15:10

Add compliance task to pipeline

1977f6b

attempt machine version test and change to artifact feed connection

a151fed

bippity boppity boo

d9ffba3

goof less

6a63f3c

undo temporary testing stuff

8b38ccc

re-sprinkle helix access tokens

75e5c95

conditionally append access token

b9dc74e

use correct method name

ebe8075

attempt to break out compliance tasks onto 2019 image and temporarily…

ffc5dc3

… use recycled artifacts to speed this up

add checkout bit and nerf dependency

1dc142b

wrong indent because yaml

75c69fd

apparently pkgesseccomp not updated to 12 for 1es

8fa0dd1

wrong destination folder for copy pasta

68c52c2

try a different place for archives?

40f77ff

make testing this slightly easier temporarily

ed7ccac

try to show output

61eea11

try to show output

96a7f25

try to show output

865a0d7

more unzip fun

2e3c8ac

messin' about

446f4b5

vmimage

b237283

add pkges setup task to compliance pipe

5c4c708

update match patterns

77eab3c

separate extract into two steps I guess

a4d6ad8

throw more things at the wall

f7b1e96

remove line

4f697bd

let's see how this blends

826f8c0

fiddle about

a9a3789

ran out of heap space on compile with x86. fortunately v3 of the task…

aa12f45

… (which I was supposed to move to anyway) uses x64 by default.

apparently npm is required by v3. go back to v2 and force x64 to pass…

c9ded19

… the heap thing.

This comment has been minimized.

Sign in to view

miniksa added 2 commits December 15, 2021 12:23

spellcheck

510b8dd

Merge branch 'main' into dev/miniksa/compliance

7bcb698

miniksa changed the title ~~[DRAFT] compliance~~ Enable Security and Compliance tasks in our Release pipeline Dec 15, 2021

This comment has been minimized.

Sign in to view

ghost added Area-Build Issues pertaining to the build system, CI, infrastructure, meta Issue-Task It's a feature request, but it doesn't really need a major design. Product-Meta The product is the management of the products. labels Dec 15, 2021

need to restore pgo step too

40381eb

miniksa commented Dec 15, 2021

View reviewed changes

miniksa marked this pull request as ready for review December 15, 2021 21:10

carlos-zamora reviewed Dec 15, 2021

View reviewed changes

build/config/PolicheckExclusions.xml Show resolved Hide resolved

build/pipelines/release.yml Outdated Show resolved Hide resolved

build/pipelines/release.yml Outdated Show resolved Hide resolved

build/pipelines/release.yml Show resolved Hide resolved

carlos-zamora approved these changes Dec 16, 2021

View reviewed changes

zadjii-msft added this to the Engineering Improvements 2021 milestone Jan 3, 2022

zadjii-msft approved these changes Jan 3, 2022

View reviewed changes

miniksa added 2 commits January 4, 2022 15:06

separate out compliance job, I think?

c641dee

Merge branch 'main' into dev/miniksa/compliance

e1e7cbd

miniksa added the AutoMerge Marked for automatic merge by the bot when requirements are met label Jan 5, 2022

ghost merged commit 805ac4c into main Jan 5, 2022

ghost deleted the dev/miniksa/compliance branch January 5, 2022 18:45

This was referenced Feb 3, 2022

Feature Request: Enable BinSkim for making sure our binaries are good #908

Closed

Insert Windows Engineering C++ Compliance Task into Pipelines #10735

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Security and Compliance tasks in our Release pipeline #11849

Enable Security and Compliance tasks in our Release pipeline #11849

miniksa commented Nov 30, 2021 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

miniksa Dec 15, 2021

miniksa Dec 15, 2021

miniksa Dec 15, 2021

zadjii-msft left a comment

zadjii-msft Jan 3, 2022

miniksa Jan 3, 2022

DHowett Jan 3, 2022

miniksa commented Jan 3, 2022

miniksa commented Jan 3, 2022

azure-pipelines bot commented Jan 3, 2022

ghost commented Jan 5, 2022

ghost commented Feb 3, 2022

Enable Security and Compliance tasks in our Release pipeline #11849

Enable Security and Compliance tasks in our Release pipeline #11849

Conversation

miniksa commented Nov 30, 2021 • edited Loading

References

Checklist

This comment has been minimized.

This comment has been minimized.

miniksa Dec 15, 2021

Choose a reason for hiding this comment

miniksa Dec 15, 2021

Choose a reason for hiding this comment

miniksa Dec 15, 2021

Choose a reason for hiding this comment

zadjii-msft left a comment

Choose a reason for hiding this comment

zadjii-msft Jan 3, 2022

Choose a reason for hiding this comment

miniksa Jan 3, 2022

Choose a reason for hiding this comment

DHowett Jan 3, 2022

Choose a reason for hiding this comment

miniksa commented Jan 3, 2022

miniksa commented Jan 3, 2022

azure-pipelines bot commented Jan 3, 2022

ghost commented Jan 5, 2022

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

ghost commented Feb 3, 2022

miniksa commented Nov 30, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.