Test updated vscode-test API and yeoman generated extensions

[octref]

Refs: #74560, #74555 and #74436

• Mac @bpasero
• Linux @Tyriar
• Windows @aeschli

Complexity: 3

We have tested the new vscode-test API in #74436. I resolved remaining complaints of #74560 and #74555 in this milestone.

High Level Changes:

• vscode-test API change: https://github.com/microsoft/vscode-test/blob/master/CHANGELOG.md#100-next0--2019-06-24 for Run test options: Polish APIs #74560.
• run() function signature change. See runTests.d.ts spec not consistent with code #74555 (comment).

Prep:

• run npm install -g generator-code@next
  This should install generator-code@1.2.0-next.2
• Download the latest insider of your platform to desktop

Test 1

Docs

Take a look at the updated helloworld-test-sample. Make sure the test is runnable with npm run compile && npm run test. Look through the runTests.ts and suite/index.ts and make sure the API makes sense to you. Try adjusting the runTests options as well to see if it works as expected.

Then do the same thing with yo code. Ensure they generate the same code as the sample, and their test script in package.json is runnable.

Open issues for any confusing API, docs, sample code, etc.

Test 2

We made a change to how the test runner script is launched:

interface ITestRunner {
	/** Old test runner API, as exported from `vscode/lib/testrunner` */
	run(testsRoot: string, clb: (error: Error, failures?: number) => void): void;
}

interface INewTestRunner {
	/** New test runner API, as explained in the extension test doc */
	run(): Promise<void>;
}

Current insider version should allow both test runner to be run.

1. Old extension that depends on vscode uses vscode/lib/testrunner and exports a run function of ITestRunner signature.
2. Current sample has a hand-written run function of ITestRunner signature
3. When we release, I'll update the sample/generator to use INewTestRunner

Your task is to make sure INewTestRunner runs fine for our Insiders.
You have to close all Insider instance before running integration test scripts.

• Checkout the test sample under this branch
• Update this line to point to the downloaded & unzipped VS Code
• Run the integration test through npm run compile && npm run test. Make sure it runs test fine and exits normally.
• Now throw an error, or introduce a failing test. Make sure the test script reports the error and exits with exit code 1.

[bpasero]

I finished testing.