From 8bda34ad921e6e88f4ef2e97a8181f5ac67230ff Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Fri, 7 Jan 2022 15:13:39 -0800 Subject: [PATCH 1/6] Initial landing page with link to first query --- Documentation/AzureDevOps/TestReportingQueries.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 Documentation/AzureDevOps/TestReportingQueries.md diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md new file mode 100644 index 00000000000..e69de29bb2d From a74887d949fd06d0520550d23257850fb7a33889 Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Fri, 7 Jan 2022 15:14:25 -0800 Subject: [PATCH 2/6] Initial landing page with link to first query --- .../AzureDevOps/TestReportingQueries.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md index e69de29bb2d..8dd3dc35803 100644 --- a/Documentation/AzureDevOps/TestReportingQueries.md +++ b/Documentation/AzureDevOps/TestReportingQueries.md @@ -0,0 +1,19 @@ +# Test Reporting Queries + +The following is a list of default queries to use to look up information about failed tests. Feel free to change them for your own usages. + +Caveats (Jan 7, 2022): +- Test data is only recently populated, thus, we only have about 2.5 weeks worth of data. +- There is a [known issue](https://github.com/dotnet/core-eng/issues/14708) with how we're capturing this data that is currently being worked on, thus, some of the data we have may not be complete. + +## Tests That Have Failed X% of the Time in the Recent Timespan + +This query will return a list of tests that have failed a certain percentage in the recent provided timespan. The default example in this query provides a list of tests in the dotnet/runtime repo that have failed 10% of the time in the last 7 days. + +Variables: +- `ts`: [Kusto timespan format](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/timespan). Default is `7d`. +- `repo`: Repository to filter on. Set to empty string to inclue all repositories. Default is `dotnet/runtime`. +- `failureThreshold`: Double value denoting failure rate percentage. Default is `0.1` or 10%. +- `excludeAlwaysFailing`: Set to true to filter out tests that are always failing to get a list of tests that are "flakey". Default is `true`. + +:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query \ No newline at end of file From be2c2b97ef79a6d5eeeb38ee7bc1f9c4921290f6 Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Tue, 11 Jan 2022 10:46:37 -0800 Subject: [PATCH 3/6] Changes per code review feedback --- .../AzureDevOps/TestReportingQueries.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md index 8dd3dc35803..261343e426a 100644 --- a/Documentation/AzureDevOps/TestReportingQueries.md +++ b/Documentation/AzureDevOps/TestReportingQueries.md @@ -3,9 +3,15 @@ The following is a list of default queries to use to look up information about failed tests. Feel free to change them for your own usages. Caveats (Jan 7, 2022): +- This data is only accessible by internal Microsoft employees. - Test data is only recently populated, thus, we only have about 2.5 weeks worth of data. - There is a [known issue](https://github.com/dotnet/core-eng/issues/14708) with how we're capturing this data that is currently being worked on, thus, some of the data we have may not be complete. +## Index +- [Test Reporting Queries](#test-reporting-queries) + - [Index](#index) + - [Tests That Have Failed X% of the Time in the Recent Timespan](#tests-that-have-failed-x-of-the-time-in-the-recent-timespan) + ## Tests That Have Failed X% of the Time in the Recent Timespan This query will return a list of tests that have failed a certain percentage in the recent provided timespan. The default example in this query provides a list of tests in the dotnet/runtime repo that have failed 10% of the time in the last 7 days. @@ -16,4 +22,21 @@ Variables: - `failureThreshold`: Double value denoting failure rate percentage. Default is `0.1` or 10%. - `excludeAlwaysFailing`: Set to true to filter out tests that are always failing to get a list of tests that are "flakey". Default is `true`. -:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query \ No newline at end of file +``` +let ts = 7d; // Timespan value +let repo = "dotnet/runtime"; // Optional: set to empty string if you want results from all repositories +let failureThreshold = 0.1; // Double value denoting the lowest test fail rate to return +let excludeAlwaysFailing = true; // Boolean. Set to true to exclude test results that are always failing +let subtable = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) and iff(repo == "", Repository == Repository, Repository == repo) +| summarize numerator=sum(FailCount), denom=sum(PassCount+FailCount+PassOnRetryCount), asOfDate=max(ReportDate) by BuildDefinitionName, TestName, ArgumentHash; +let argumentHashMap = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) +| summarize by ArgumentHash, Arguments; +subtable +| where denom > 0 and (todouble(numerator)/todouble(denom)) >= failureThreshold and iff(excludeAlwaysFailing, (todouble(numerator)/todouble(denom)) < 1, true) +| lookup (argumentHashMap) on ArgumentHash +| project BuildDefinitionName, TestName, Arguments, FailRate=(todouble(numerator) / todouble(denom)), FailCount=numerator, TotalRunCount=denom; +``` + +:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query editor \ No newline at end of file From d56c9bac1a057ec03ecab90aa510c6c52c847ba7 Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Thu, 13 Jan 2022 11:10:25 -0800 Subject: [PATCH 4/6] Add query for tests' expected pass rate --- .../AzureDevOps/TestReportingQueries.md | 36 ++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md index 261343e426a..a08acd88667 100644 --- a/Documentation/AzureDevOps/TestReportingQueries.md +++ b/Documentation/AzureDevOps/TestReportingQueries.md @@ -11,6 +11,8 @@ Caveats (Jan 7, 2022): - [Test Reporting Queries](#test-reporting-queries) - [Index](#index) - [Tests That Have Failed X% of the Time in the Recent Timespan](#tests-that-have-failed-x-of-the-time-in-the-recent-timespan) + - [Mean Value for the Expected Pass Rate of Tests](#mean-value-for-the-expected-pass-rate-of-tests) + - [Pass Rate Confidence Level](#pass-rate-confidence-level) ## Tests That Have Failed X% of the Time in the Recent Timespan @@ -39,4 +41,36 @@ subtable | project BuildDefinitionName, TestName, Arguments, FailRate=(todouble(numerator) / todouble(denom)), FailCount=numerator, TotalRunCount=denom; ``` -:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query editor \ No newline at end of file +:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query editor + +## Mean Value for the Expected Pass Rate of Tests + +This query will return a list of tests and the mean value for the expected pass rate based on historical data. (Comparably, an inverse to [Tests That Have Failed X% of the Time in the Recent Timespan](#tests-that-have-failed-x-of-the-time-in-the-recent-timespan)) + +Calculate the expected value (E[_Y_]) for how often a test is likely to pass on its initial run (retries not included) based on its historical data (e.g. monthly, weekly, daily aggregates). Ex: A test is known to fail once out of every seven runs. Its expected value (E[_Y_]) is determined to be 85.7%, meaning, we expect this test to succeed 85.7% of the time. + +Variables: +- `ts`: [Kusto timespan format](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/timespan). Default is `7d`. +- `repo`: Repository to filter on. Set to empty string to inclue all repositories. Default is `dotnet/runtime`. +- `excludeAlwaysPassing`: Set to true to filter out tests that are always passing. Default is `true`. + +``` +let ts = 7d; // Timespan value +let repo = "dotnet/runtime"; // Optional: set to empty string if you want results from all repositories +let excludeAlwaysPassing = true; // Boolean. Set to true to exclude test results that are always passing +let subtable = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) and iff(repo == "", Repository == Repository, Repository == repo) +| summarize numerator=sum(PassCount), denom=sum(PassCount+FailCount+PassOnRetryCount), asOfDate=max(ReportDate) by BuildDefinitionName, TestName, ArgumentHash; +let argumentHashMap = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) +| summarize by ArgumentHash, Arguments; +subtable +| where denom > 0 and iff(excludeAlwaysPassing, (todouble(numerator)/todouble(denom)) < 1, true) +| lookup (argumentHashMap) on ArgumentHash +| project BuildDefinitionName, TestName, Arguments, MeanPassRate=(todouble(numerator) / todouble(denom)), PassCount=numerator, TotalRunCount=denom +| order by MeanPassRate; +``` + +## Pass Rate Confidence Level + +Calculate the confidence level for a test’s pass rate. We want to have enough “fudge” room on either side of the expected value when new results come in so that slightly tipping the expected value in one direction or another (especially for tests that don’t run very often) doesn’t cause the test to be seen as “interesting”. Suggested calculation: mean +/- 1.96*(standardDeviation/sqrt(n)), where n is the number of known tests used to calculate the standardDeviation \ No newline at end of file From ee2256d5da5b7b5827359460f0ec1b279ae5a998 Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Fri, 14 Jan 2022 13:09:44 -0800 Subject: [PATCH 5/6] Added more reports --- .../AzureDevOps/TestReportingQueries.md | 55 ++++++++++++++++--- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md index a08acd88667..834068386e8 100644 --- a/Documentation/AzureDevOps/TestReportingQueries.md +++ b/Documentation/AzureDevOps/TestReportingQueries.md @@ -2,17 +2,17 @@ The following is a list of default queries to use to look up information about failed tests. Feel free to change them for your own usages. -Caveats (Jan 7, 2022): +Caveats (Jan 14, 2022): - This data is only accessible by internal Microsoft employees. - Test data is only recently populated, thus, we only have about 2.5 weeks worth of data. - There is a [known issue](https://github.com/dotnet/core-eng/issues/14708) with how we're capturing this data that is currently being worked on, thus, some of the data we have may not be complete. ## Index -- [Test Reporting Queries](#test-reporting-queries) - - [Index](#index) - [Tests That Have Failed X% of the Time in the Recent Timespan](#tests-that-have-failed-x-of-the-time-in-the-recent-timespan) - - [Mean Value for the Expected Pass Rate of Tests](#mean-value-for-the-expected-pass-rate-of-tests) - - [Pass Rate Confidence Level](#pass-rate-confidence-level) + - [Mean Value for the Expected Pass Rate for Tests](#mean-value-for-the-expected-pass-rate-for-tests) + - [Mean Value for the Expected Pass on Retry Rate for Tests](#mean-value-for-the-expected-pass-on-retry-rate-for-tests) + - [Build Analysis Reporting](#build-analysis-reporting) + - [Sentiment Tracker Feedback](#sentiment-tracker-feedback) ## Tests That Have Failed X% of the Time in the Recent Timespan @@ -43,7 +43,7 @@ subtable :part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51SzW7bMAy+5ymInGzUaNrTgHkukC4YdtkypHkBpqZjbbJkSFRTF3v4UTLitGkOxXwwbIr8/kRNDOyhgk91CRefxQK2qiPfo4En1IFmWmYc9Vam5rVlQ7xwwbA0zctpZt2zsgb1Z/CRwgJ1PQ/g2SmzB9XAYAMc0EQoH7RoaJztALVO2F6xdYp8ImtQ6eBo20pra3UtxDfXt+VJ4MqGnaZRHtRkLEcSbgm0PZAX/viKMOCQKcpxxMGZBE/PjzrUtNQHHPw3aYrDFbALVCb4e2s1obmGh9FKPEmWxsER/WiDW2RAR2Il4iVWAUxMPuwYo9IKli/iaEVP695vZdw/hK5DN8z+wqElmd5ICI5XUe1dBbi3Gfsc0NSSXZON8Uv+8yJ1priGWDn9nZ/EmVzwfWJSLwQmdCR5WFdJLYvOv1q5yLxIGXap+gu9T9Wr6fwq1tZmQ+yG4wD6dRPFVh0+ZyftOewGuA9K1ytqlFFxJ35iRwVE0+PX0u1Fh+Hv6NsyxYSvKj+w/9+03ngVHa+JTrS+nB2vZYJL9uEOblLgGds6LVg2BZYvplrqzfPI+25Pj9d1acOKD+J+gdsibVz0o639E3rIzhLKwZo39qS1d/Y3PfJH0/cFRGGbeIeXhMECzqWNE2kDqqlRsC2j3gQzHqTe8h/Tt/8tZwQAAA==) to query editor -## Mean Value for the Expected Pass Rate of Tests +## Mean Value for the Expected Pass Rate for Tests This query will return a list of tests and the mean value for the expected pass rate based on historical data. (Comparably, an inverse to [Tests That Have Failed X% of the Time in the Recent Timespan](#tests-that-have-failed-x-of-the-time-in-the-recent-timespan)) @@ -71,6 +71,45 @@ subtable | order by MeanPassRate; ``` -## Pass Rate Confidence Level +:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51U3WvbMBB/L/R/OAoDm5lkexgdzVJIP8b20GWkZTDGHi7xOdEiS0Y6NXXpH7+T3DhpycNYXqxI9/u4n84eDuH6oaEFUwk/UAeCyjr4jt7DDJnO4BL1ImhZAq8IaFt7n2qz618/f+cJsrIbsBWTAQQmz6A8aLUm3QJbaCKhNaDYgzKKFWpwwUDmiJ0iD8YKwix0KKnM4fhoOIQ5ehF6Bq2UZ+vUQnAlMkJGg+UAamt4pdsCNkTr+CxRiSAul46W4tnnA2nvDCa9pbWxGxMdVVIp5AsCG1icA92Ta8HLw0RrfgBfRTc5Ody1sJXE5Gpl5Ego5wQfPwxO3xRQExplltHXM1jSk/rkQip9WCxIQKk8isdsWdU0EEFNUuNhDKflCA7+xNKdFPsGTWepAzlqrMBOSsuGeChNRMqTUQ+aNqysQX0mbSYfVDcsPcsVmCWoClobYIMmUvmgxUTlbA2odeL2Kl4B+U6NHtJtTfQGWx8HJnKMgV2gUVK7sFZLDgO47cTiSRLtgF0YWyFeIQM6ErHIlwZGCDspH+aMc01CP3kMjq7oftr4O8H721DX6NrjoyfYrEjwMzHq+CoO7PlYJsFm7HNAU0p/VdZFJBmdFKkytdTGnd2/1ycRk0cBn7TUI4EJNTmUgrHsZbH5Sytp5zKAZGz9cvftZxm1bhX3pmYmM99uAeinVXQ7rvEh25nPYd7CRVC6vKIqvTHWfMOaCohtd6uJW4oPw1/Qr0ZdUri3dYPNfwf2sl2xsq+1U/Yiu72cHWPKAM7hXR/7oVkpIGNb2iDQrM8zH/Z7iSXP4RO8L9LsJFPa2nVoIHvVaB6/E/seY23j7J/46v1jjL6AG5nXaC9++saH7MEQXhssoL/pcV8o/JZRz4LpDlJt9GRdSS4Gui81+gtMm5LlhgUAAA==) to query editor -Calculate the confidence level for a test’s pass rate. We want to have enough “fudge” room on either side of the expected value when new results come in so that slightly tipping the expected value in one direction or another (especially for tests that don’t run very often) doesn’t cause the test to be seen as “interesting”. Suggested calculation: mean +/- 1.96*(standardDeviation/sqrt(n)), where n is the number of known tests used to calculate the standardDeviation \ No newline at end of file +## Mean Value for the Expected Pass on Retry Rate for Tests + +Retries are meant to unblock and prevent a build from failing due to failing test, but they are still indicative of unwanted behavior, therefore, we need to track how often a test passes when retries are introduced. Ex: A test has a 100% pass rate, but only when the test re-runs after a failure every six runs, so it’s expected value for re-runs is 83.3%. + +Variables: +- `ts`: [Kusto timespan format](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/timespan). Default is `14d`. +- `repo`: Repository to filter on. Set to empty string to inclue all repositories. Default is `dotnet/arcade`. + +``` +let ts = 14d; // Timespan value +let repo = "dotnet/arcade"; // Optional: set to empty string if you want results from all repositories +let subtable = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) and iff(repo == "", Repository == Repository, Repository == repo) and PassOnRetryCount > 0 +| summarize numerator=sum(PassOnRetryCount), denom=sum(FailCount+PassOnRetryCount), asOfDate=max(ReportDate) by BuildDefinitionName, TestName, ArgumentHash; +let argumentHashMap = AzureDevOpsTestsSummary +| where ReportDate >= ago(ts) +| summarize by ArgumentHash, Arguments; +subtable +| where denom > 0 +| lookup (argumentHashMap) on ArgumentHash +| project BuildDefinitionName, TestName, Arguments, MeanPassOnRetryRate=(todouble(numerator) / todouble(denom)), PassOnRetryCount=numerator, TotalRunCount=denom +| order by MeanPassOnRetryRate; +``` + +:part_alternation_mark: [Link](https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAA51U3WvbMBB/L/R/OAoDm5lkexgdzVJIP8b20GWkZTDGHi7xOdEiS0Y6NXXpH7+T3DhpycNYXqxI9/u4n84eDuH6oaEFUwk/UAeCyjr4jt7DDJnO4BL1ImhZAq8IaFt7n2qz618/f+cJsrIbsBWTAQQmz6A8aLUm3QJbaCKhNaDYgzKKFWpwwUDmiJ0iD8YKwix0KKnM4fhoOIQ5ehF6Bq2UZ+vUQnAlMkJGg+UAamt4pdsCNkTr+CxRiSAul46W4tnnA2nvDCa9pbWxGxMdVVIp5AsCG1icA92Ta8HLw0RrfgBfRTc5Ody1sJXE5Gpl5Ego5wQfPwxO3xRQExplltHXM1jSk/rkQip9WCxIQKk8isdsWdU0EEFNUuNhDKflCA7+xNKdFPsGTWepAzlqrMBOSsuGeChNRMqTUQ+aNqysQX0mbSYfVDcsPcsVmCWoClobYIMmUvmgxUTlbA2odeL2Kl4B+U6NHtJtTfQGWx8HJnKMgV2gUVK7sFZLDgO47cTiSRLtgF0YWyFeIQM6ErHIlwZGCDspH+aMc01CP3kMjq7oftr4O8H721DX6NrjoyfYrEjwMzHq+CoO7PlYJsFm7HNAU0p/VdZFJBmdFKkytdTGnd2/1ycRk0cBn7TUI4EJNTmUgrHsZbH5Sytp5zKAZGz9cvftZxm1bhX3pmYmM99uAeinVXQ7rvEh25nPYd7CRVC6vKIqvTHWfMOaCohtd6uJW4oPw1/Qr0ZdUri3dYPNfwf2sl2xsq+1U/Yiu72cHWPKAM7hXR/7oVkpIGNb2iDQrM8zH/Z7iSXP4RO8L9LsJFPa2nVoIHvVaB6/E/seY23j7J/46v1jjL6AG5nXaC9++saH7MEQXhssoL/pcV8o/JZRz4LpDlJt9GRdSS4Gui81+gtMm5LlhgUAAA==) to query editor + +## Build Analysis Reporting + +This Power BI page contains the following reports: +- Details of PRs outcomes when merged (e.g. when a PR was merged on red) +- Build outcomes and retry metrics +- Tests that pass on rerun + +:part_alternation_mark: [Link](https://msit.powerbi.com/groups/me/reports/8bdd4339-ea39-4e67-963a-fc44a450605b/ReportSection?ctid=72f988bf-86f1-41af-91ab-2d7cd011db47) to Power BI Report + +## Sentiment Tracker Feedback + +This report tracks the usage and trends of the feedback we receive via the sentiment tracker in the Build Analysis check on the PRs. + +:part_alternation_mark: [Link](https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/e6deb422-46fc-4f80-8892-ba7036081986/ReportSection) to Power BI Report \ No newline at end of file From a65544eed024001b2fd7e14c9151bd82ee45e591 Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Fri, 14 Jan 2022 13:17:54 -0800 Subject: [PATCH 6/6] Added link to 'how to get a hold of engineering services' for folks that want to request access to the reports --- Documentation/AzureDevOps/TestReportingQueries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/AzureDevOps/TestReportingQueries.md b/Documentation/AzureDevOps/TestReportingQueries.md index 834068386e8..fbbf7d9ef89 100644 --- a/Documentation/AzureDevOps/TestReportingQueries.md +++ b/Documentation/AzureDevOps/TestReportingQueries.md @@ -3,7 +3,7 @@ The following is a list of default queries to use to look up information about failed tests. Feel free to change them for your own usages. Caveats (Jan 14, 2022): -- This data is only accessible by internal Microsoft employees. +- This data is only accessible by internal Microsoft employees. (If you are an internal Microsoft employee, you can request access from the [.NET Engineering Services team](https://github.com/dotnet/core-eng/wiki/How-to-get-a-hold-of-Engineering-Servicing).) - Test data is only recently populated, thus, we only have about 2.5 weeks worth of data. - There is a [known issue](https://github.com/dotnet/core-eng/issues/14708) with how we're capturing this data that is currently being worked on, thus, some of the data we have may not be complete.