-
Notifications
You must be signed in to change notification settings - Fork 373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable AUTHORING UX approval tests creation #3868
Comments
@vlada-shubina @DavidKarlas a number of repos are starting to create .NET 7 templates (e.g, dotnet/winforms#6206), and having a test infra for templates could be very helpful. |
This feature will be useful for testing: #3418 to avoid the need of using custom settings location and installing the template. |
Based on brainstroming session with @vlada-shubina those are the tasks we came up with: Investigations:
Subtasks for MVP (not to be exposed to customers):
V2 (preparation for customers exposed toolset):
Next iterations (ideally part of the first customer facing version):
|
Great plan!
This point sounds strange... Perhaps I don't quite understand the intent here, could you elaborate on this please? |
We plan to use Verify.NET to build the framework. We need to investigate if it can do verification of multiple files out of the box. So we far we are using it only for single file\object validation. |
We use Verify in Windows Forms repos quite a bit, and I don't think it's possible to verify multiple files simultaneously, such verifications need to be serialized. That is, how do you present multiple failures in a diff tool? protected async Task VerifyAsync(
IVisualStudioDocument mainFile,
IVisualStudioDocument codeBehindFile,
[CallerMemberName] string testMethodName = "")
{
var (TestMethodName, AdditionalPath) = GetTestContext(testMethodName);
await VerifyAsync(mainFile.GetTextBuffer().CurrentSnapshot, $"{TestMethodName}_{MainFileSuffix}", AdditionalPath);
await VerifyAsync(codeBehindFile.GetTextBuffer().CurrentSnapshot, $"{TestMethodName}_{CodeBehindFileSuffix}", AdditionalPath);
}
private static Task VerifyAsync(ITextSnapshot textSnapshot, string methodName, string additionalPath)
=> Verifier.Verify(textSnapshot.GetText())
.UseDirectory($@"TestData\{additionalPath}")
.UseFileName(methodName); Pining @SimonCropp for sharing his thoughts on this. |
@RussKie yeah thats not ideal. where do i find that code so i can have a go a making it better? |
Just to explain our use case better: we would like to verify the template output, that can be N files with random folder structure.
The template output is pretty static (with some exceptions like guids, dates etc) so approval tests seems to be good match to test them. Ideally we do the following:
and this produces the snapshot identical to folder/files in the Imho, this use case (verifying all the files in the folder) is pretty generic and implementing it in Verify.NET would be of benefit. |
Thank you for the context, now I think I get it - executing a template will produce several files, and each template may have a different number of those files. So essentially we need to verify a folder content. I don't think it'd be difficult to create a helper method that verifies a content of a folder, though it'd make our lives easier if that helper was provided out of a box. :) In Windows Forms scenarios we generally compare one or two known files (more like entities that represent files, and those may not even exist on disk).
I can interpret this wish in a few different ways, though not sure if my interpretation aligns with yours.
|
Thank you @SimonCropp for jumping in. In essence, here we're talking about verifying the result of
Do you think it's a worthy addition to your already awesome library? The sample I mentioned earlier is in a close source, but I'm sure @vlada-shubina or @JanKrivanek may be able to explain their test procedures in more details. |
Added couple of test drafts in #5163. |
now.txt Usage stats of non-deterministic macro attached:
|
@vlada-shubina i should have a beta "verifydirectory" for you in a couple of days |
@vlada-shubina if you update to the current beta of Verify, you can try this https://github.com/VerifyTests/Verify#verifydirectory |
Awesome work! Thanks. First very basic test passed. The only obstacle I'm stuck with is here: update: if possible I would prefer different naming
Having verified and received mixed in the file name is a big confusing, see: |
i have deployed a new version with that constraint removed. I no longer makes sense anyway given the current file naming logic
hmm. let me think on that and get back to you |
@vlada-shubina how does this look https://github.com/VerifyTests/Verify/blob/main/docs/naming.md#usesplitmodeforuniquedirectory |
Thanks, it works great. |
Closing as work in context of 7.0 was done. |
Background
Today there is no good story for template authors to test their templates and ensure they work as intended after doing changes and environment around them changes(.NET Framework, Template Engine...)
We have https://github.com/dotnet/templating/tree/main/tools/ProjectTestRunner but its pretty hard to understand and navigate for novice template author, also tooling is not compiled as dotnet global tool to just use...
We run tests in our repo using Process.Start("dotnet new console) and than check output see example here.. Again, not very good way for template author to run/maintain tests.
Outcomes
This enables template development inner loop. We want to support approval tests, which means template author would do:
dotnet new console
once to create initial contentJustification
Prerequisite
What needs to be solved how to handle random values like PortNumber or GUIDs...
Subtasks
Investigations:
We will leverage global.json for this - simplified approach:
Subtasks for MVP (not to be exposed to customers):
templates authoring toolset
. Packaged as nuget - @vlada-shubinaV2 (preparation for customers exposed toolset):
Command
) fromMicrosoft.DotNet.Cli.Utils
or find another utility for wrapping CLI processes - and get rid of copied code within theMicrosoft.TemplateEngine.Authoring.TemplateVerifier
. (this task might be joined with Consider dropping dependency toMicrosoft.DotNet.Cli.Utils
. #5296)Support batch execution logic for multiple test cases (probably configured by files)Next iterations (ideally part of the first customer facing version):
The text was updated successfully, but these errors were encountered: