Write a meta-analyzer to flag analyzers that use VB/C# compiler reserved diagnostic IDs

[mavasani]

See dotnet/roslyn#25748 for more details and previous attempt to put such an diagnostic in the compiler itself

[Evangelink]

@mavasani so the idea here is to report on diagnostic ids with the following patterns: CA|CS|BC|RS and then digits, is that it?

[mavasani]

We only want to flag CS and BC prefixes followed by and integral suffix as they would clash with C# and VB compiler diagnostic IDs respectively. No other prefixes should be flagged.

[sharwell]

We would also flag the special diagnostic ID format since it's tied into the core formatter.

[mavasani]

@Evangelink based on the discussions at dotnet/roslyn#40351, I think your original suggestion of flagging even CA IDs seems reasonable. I think we should make this rule configurable so end users can tweak the banned regex to their own preference. For example, a repo might want to outright enforce that only a certain prefix and length of ID be used.

We have a very similar rule which allows users to provide a custom additional file to enforce the category and rule ID mapping, rule ID prefix and format:

roslyn-analyzers/src/Microsoft.CodeAnalysis.Analyzers/UnitTests/MetaAnalyzers/DiagnosticDescriptorCreationAnalyzerTests.cs

Lines 419 to 664 in 9c08642

	#region RS1018 (DiagnosticIdMustBeInSpecifiedFormatRuleId) and RS1020 (UseCategoriesFromSpecifiedRangeRuleId)
	[Fact]
	public void RS1018_RS1020_CSharp_VerifyDiagnostic()
	{
	var source = @"
	using System;
	using System.Collections.Immutable;
	using Microsoft.CodeAnalysis;
	using Microsoft.CodeAnalysis.Diagnostics;
	[DiagnosticAnalyzer(LanguageNames.CSharp, LanguageNames.VisualBasic)]
	class MyAnalyzer : DiagnosticAnalyzer
	{
	private static LocalizableResourceString dummyLocalizableTitle = null;
	private static readonly DiagnosticDescriptor descriptor =
	new DiagnosticDescriptor(""Id1"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""NotAllowedCategory"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor2 =
	new DiagnosticDescriptor(""DifferentPrefixId"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor3 =
	new DiagnosticDescriptor(""Prefix200"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithRange"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor4 =
	new DiagnosticDescriptor(""Prefix101"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor5 =
	new DiagnosticDescriptor(""MySecondPrefix400"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor6 =
	new DiagnosticDescriptor(""MyThirdPrefix"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics
	{
	get
	{
	return ImmutableArray.Create(descriptor, descriptor2, descriptor3, descriptor4, descriptor5, descriptor6);
	}
	}
	public override void Initialize(AnalysisContext context)
	{
	}
	}
	";
	string additionalText = @"
	# FORMAT:
	# 'Category': Comma separate list of 'StartId-EndId' or 'Id' or 'Prefix'
	CategoryWithNoIdRangeOrFormat
	CategoryWithPrefix: Prefix
	CategoryWithRange: Prefix000-Prefix099
	CategoryWithId: Prefix100
	CategoryWithPrefixRangeAndId: MyFirstPrefix, MySecondPrefix000-MySecondPrefix099, MySecondPrefix300
	";
	DiagnosticResult[] expected = new[] {
	// Test0.cs(13,87): warning RS1020: Category 'NotAllowedCategory' is not from the allowed categories specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1020ExpectedDiagnostic(13, 87, "NotAllowedCategory", AdditionalFileName),
	// Test0.cs(16,34): warning RS1018: Diagnostic Id 'DifferentPrefixId' belonging to category 'CategoryWithPrefix' is not in the required range and/or format 'PrefixXXXX' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1018ExpectedDiagnostic(16, 34, "DifferentPrefixId", "CategoryWithPrefix", "PrefixXXXX", AdditionalFileName),
	// Test0.cs(19,34): warning RS1018: Diagnostic Id 'Prefix200' belonging to category 'CategoryWithRange' is not in the required range and/or format 'Prefix0-Prefix99' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1018ExpectedDiagnostic(19, 34, "Prefix200", "CategoryWithRange", "Prefix0-Prefix99", AdditionalFileName),
	// Test0.cs(22,34): warning RS1018: Diagnostic Id 'Prefix101' belonging to category 'CategoryWithId' is not in the required range and/or format 'Prefix100-Prefix100' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1018ExpectedDiagnostic(22, 34, "Prefix101", "CategoryWithId", "Prefix100-Prefix100", AdditionalFileName),
	// Test0.cs(25,34): warning RS1018: Diagnostic Id 'MySecondPrefix400' belonging to category 'CategoryWithPrefixRangeAndId' is not in the required range and/or format 'MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1018ExpectedDiagnostic(25, 34, "MySecondPrefix400", "CategoryWithPrefixRangeAndId", "MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300", AdditionalFileName),
	// Test0.cs(28,34): warning RS1018: Diagnostic Id 'MyThirdPrefix' belonging to category 'CategoryWithPrefixRangeAndId' is not in the required range and/or format 'MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetCSharpRS1018ExpectedDiagnostic(28, 34, "MyThirdPrefix", "CategoryWithPrefixRangeAndId", "MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300", AdditionalFileName)
	};
	VerifyCSharp(source, GetAdditionalFile(additionalText), expected);
	}
	[Fact]
	public void RS1018_RS1020_VisualBasic_VerifyDiagnostic()
	{
	var source = @"
	Imports System
	Imports System.Collections.Immutable
	Imports Microsoft.CodeAnalysis
	Imports Microsoft.CodeAnalysis.Diagnostics
	<DiagnosticAnalyzer(LanguageNames.CSharp, LanguageNames.VisualBasic)>
	Class MyAnalyzer
	Inherits DiagnosticAnalyzer
	Private Shared dummyLocalizableTitle As LocalizableResourceString = Nothing
	Private Shared ReadOnly descriptor As DiagnosticDescriptor = New DiagnosticDescriptor(""Id1"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""NotAllowedCategory"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor2 As DiagnosticDescriptor = New DiagnosticDescriptor(""DifferentPrefixId"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor3 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix200"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithRange"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor4 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix101"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor5 As DiagnosticDescriptor = New DiagnosticDescriptor(""MySecondPrefix400"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor6 As DiagnosticDescriptor = New DiagnosticDescriptor(""MyThirdPrefix"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Public Overrides ReadOnly Property SupportedDiagnostics() As ImmutableArray(Of DiagnosticDescriptor)
	Get
	Return ImmutableArray.Create(descriptor, descriptor2, descriptor3, descriptor4, descriptor5, descriptor6)
	End Get
	End Property
	Public Overrides Sub Initialize(ByVal context As AnalysisContext)
	End Sub
	End Class
	";
	string additionalText = @"
	# FORMAT:
	# 'Category': Comma separate list of 'StartId-EndId' or 'Id' or 'Prefix'
	CategoryWithNoIdRangeOrFormat
	CategoryWithPrefix: Prefix
	CategoryWithRange: Prefix000-Prefix099
	CategoryWithId: Prefix100
	CategoryWithPrefixRangeAndId: MyFirstPrefix, MySecondPrefix000-MySecondPrefix099, MySecondPrefix300
	";
	DiagnosticResult[] expected = new[] {
	// Test0.vb(12,144): warning RS1020: Category 'NotAllowedCategory' is not from the allowed categories specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1020ExpectedDiagnostic(12, 144, "NotAllowedCategory", AdditionalFileName),
	// Test0.vb(13,92): warning RS1018: Diagnostic Id 'DifferentPrefixId' belonging to category 'CategoryWithPrefix' is not in the required range and/or format 'PrefixXXXX' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1018ExpectedDiagnostic(13, 92, "DifferentPrefixId", "CategoryWithPrefix", "PrefixXXXX", AdditionalFileName),
	// Test0.vb(14,92): warning RS1018: Diagnostic Id 'Prefix200' belonging to category 'CategoryWithRange' is not in the required range and/or format 'Prefix0-Prefix99' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1018ExpectedDiagnostic(14, 92, "Prefix200", "CategoryWithRange", "Prefix0-Prefix99", AdditionalFileName),
	// Test0.vb(15,92): warning RS1018: Diagnostic Id 'Prefix101' belonging to category 'CategoryWithId' is not in the required range and/or format 'Prefix100-Prefix100' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1018ExpectedDiagnostic(15, 92, "Prefix101", "CategoryWithId", "Prefix100-Prefix100", AdditionalFileName),
	// Test0.vb(16,92): warning RS1018: Diagnostic Id 'MySecondPrefix400' belonging to category 'CategoryWithPrefixRangeAndId' is not in the required range and/or format 'MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1018ExpectedDiagnostic(16, 92, "MySecondPrefix400", "CategoryWithPrefixRangeAndId", "MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300", AdditionalFileName),
	// Test0.vb(17,92): warning RS1018: Diagnostic Id 'MyThirdPrefix' belonging to category 'CategoryWithPrefixRangeAndId' is not in the required range and/or format 'MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300' specified in the file 'DiagnosticCategoryAndIdRanges.txt'.
	GetBasicRS1018ExpectedDiagnostic(17, 92, "MyThirdPrefix", "CategoryWithPrefixRangeAndId", "MyFirstPrefixXXXX, MySecondPrefix0-MySecondPrefix99, MySecondPrefix300-MySecondPrefix300", AdditionalFileName),
	};
	VerifyBasic(source, GetAdditionalFile(additionalText), expected);
	}
	[Fact]
	public void RS1018_RS1020_CSharp_NoDiagnosticCases()
	{
	var source = @"
	using System;
	using System.Collections.Immutable;
	using Microsoft.CodeAnalysis;
	using Microsoft.CodeAnalysis.Diagnostics;
	[DiagnosticAnalyzer(LanguageNames.CSharp, LanguageNames.VisualBasic)]
	class MyAnalyzer : DiagnosticAnalyzer
	{
	private static LocalizableResourceString dummyLocalizableTitle = null;
	private static readonly DiagnosticDescriptor descriptor =
	new DiagnosticDescriptor(""Id1"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithNoIdRangeOrFormat"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor2 =
	new DiagnosticDescriptor(""Prefix"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor2_2 =
	new DiagnosticDescriptor(""Prefix101"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor3 =
	new DiagnosticDescriptor(""Prefix001"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithRange"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor4 =
	new DiagnosticDescriptor(""Prefix100"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor5 =
	new DiagnosticDescriptor(""MyFirstPrefix001"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor6 =
	new DiagnosticDescriptor(""MySecondPrefix050"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	private static readonly DiagnosticDescriptor descriptor7 =
	new DiagnosticDescriptor(""MySecondPrefix300"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault: true, helpLinkUri: ""HelpLink"");
	public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics
	{
	get
	{
	return ImmutableArray.Create(descriptor, descriptor2, descriptor2_2, descriptor3, descriptor4, descriptor5, descriptor6, descriptor7);
	}
	}
	public override void Initialize(AnalysisContext context)
	{
	}
	}
	";
	string additionalText = @"
	# FORMAT:
	# 'Category': Comma separate list of 'StartId-EndId' or 'Id' or 'Prefix'
	CategoryWithNoIdRangeOrFormat
	CategoryWithPrefix: Prefix
	CategoryWithRange: Prefix000-Prefix099
	CategoryWithId: Prefix100
	CategoryWithPrefixRangeAndId: MyFirstPrefix, MySecondPrefix000-MySecondPrefix099, MySecondPrefix300
	";
	VerifyCSharp(source, GetAdditionalFile(additionalText));
	}
	[Fact]
	public void RS1018_RS1020_VisualBasic_NoDiagnosticCases()
	{
	var source = @"
	Imports System
	Imports System.Collections.Immutable
	Imports Microsoft.CodeAnalysis
	Imports Microsoft.CodeAnalysis.Diagnostics
	<DiagnosticAnalyzer(LanguageNames.CSharp, LanguageNames.VisualBasic)>
	Class MyAnalyzer
	Inherits DiagnosticAnalyzer
	Private Shared dummyLocalizableTitle As LocalizableResourceString = Nothing
	Private Shared ReadOnly descriptor As DiagnosticDescriptor = New DiagnosticDescriptor(""Id1"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithNoIdRangeOrFormat"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor2 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor2_2 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefix"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor3 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix001"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithRange"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor4 As DiagnosticDescriptor = New DiagnosticDescriptor(""Prefix100"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor5 As DiagnosticDescriptor = New DiagnosticDescriptor(""MyFirstPrefix001"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor6 As DiagnosticDescriptor = New DiagnosticDescriptor(""MySecondPrefix050"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Private Shared ReadOnly descriptor7 As DiagnosticDescriptor = New DiagnosticDescriptor(""MySecondPrefix300"", dummyLocalizableTitle, ""MyDiagnosticMessage"", ""CategoryWithPrefixRangeAndId"", DiagnosticSeverity.Warning, isEnabledByDefault:=True, helpLinkUri:=""HelpLink"")
	Public Overrides ReadOnly Property SupportedDiagnostics() As ImmutableArray(Of DiagnosticDescriptor)
	Get
	Return ImmutableArray.Create(descriptor, descriptor2, descriptor2_2, descriptor3, descriptor4, descriptor5, descriptor6, descriptor7)
	End Get
	End Property
	Public Overrides Sub Initialize(ByVal context As AnalysisContext)
	End Sub
	End Class
	";
	string additionalText = @"
	# FORMAT:
	# 'Category': Comma separate list of 'StartId-EndId' or 'Id' or 'Prefix'
	CategoryWithNoIdRangeOrFormat
	CategoryWithPrefix: Prefix
	CategoryWithRange: Prefix000-Prefix099
	CategoryWithId: Prefix100
	CategoryWithPrefixRangeAndId: MyFirstPrefix, MySecondPrefix000-MySecondPrefix099, MySecondPrefix300
	";
	VerifyBasic(source, GetAdditionalFile(additionalText));
	}
	#endregion

. We should be able to utilize or refactor a bunch of this analyzer to add the new functionality.

[Evangelink]

@mavasani So to sum it up, I need to create a new analyzer which by default will handle:

• CA + 4 digits
• CS + 4 digits
• BC + 5 digits
• format + ???

Nothing hardcoded for RS

It will also allow users to provide a regex as ID check for each category. Does it need to be in the .editorconfig or as a separate file like DiagnosticCategoryAndIdRanges.txt?

[mavasani]

@Evangelink I think we should start with a simple analyzer that just flags CA, BC and CS diagnostics. I don't think you need to check specific digit count after the prefix, but just that it is a number. Let's leave adding configurability as a separate follow-up item, if we get customer requests.

For flagging the CA diagnostics, you probably need to add a hardcoded list of assembly names that shouldn't be flagged, which should include Microsoft.CodeQuality.Analyzers, Microsoft.NetCore.Analyzers, Microsoft.NetFramework.Analyzers and also all of their C# and VB analyzer assembly names.