Support running extension tests through cmd line

[jpoon]

Add support for running plugin/extension tests through the command line. This is particularly useful for things like CI. I was able to setup gulp+mocha, but any tests that reference vscode doesn't work.

Paraphrasing @bpasero, "this is a consequence of having an extension host that runs your plugin. This host is really the entire VS Code app and cannot be emulated from the command line easily. "

[Tyriar]

+1 - I believe atom manage to do this using a similar mechanism to Chromium's browser tests.

[bpasero]

+1 this is on our backlog. however you will not be able to run tests without having a windowing system because in order to provide the VS Code APIs for the tests we still need to open a window.

[bpasero]

This will be supported in the next release with the following syntax: code --extensionDevelopmentPath=<path to extension> --extensionTestsPath=<path to compiled tests>

The output goes directly to console and the exit code is set properly.

Note: in order for the exit code to function properly you will need vscode@0.10.6 at least as dependency from your extension.

[bpasero]

@jpoon @Tyriar maybe I closed this too early. What you can now do is run tests from the command line but only if code is not already running. This should however be good enough for running extension tests as part of Travis CI. We are actually able to run some of our API tests on Mac and Linux through Travis even though the tests require a real code instance to spawn.

[Tyriar]

@bpasero will that constraint go away when launchNewInstance (or whatever it will be) is set to true? #48

[bpasero]

@Tyriar unfortunately not.

[jpoon]

@bpasero What release should have this? 0.10.2?

[bpasero]

@jpoon no, mid december!

[bpasero]

Documented here: https://code.visualstudio.com/docs/extensions/testing-extensions#_running-tests-automatically-on-travis-ci-build-machines

[jpoon]

Thanks @bpasero.

It looks like tests that rely on the active window fail (vscode.window.activeTextEditor => undefined). I'm not sure if this is inherent to Travis -- does it allow you to open a window?

Unfortunately, I'm running on Windows and can't debug this further (any plans for adding Windows support?). On Travis, I do see these warnings:

Xlib: extension "RANDR" missing on display ":99.0".

[bpasero]

Yes we have tests that rely on that too, if you have the tests somewhere online I can try to run them on Mac/Linux from the command line to see whats going on.

[jpoon]

Are your tests that rely on vscode.windowactiveTextEditor` working?

Guess I'm doing something wrong on my end then. Our tests are here: https://github.com/VSCodeVim/Vim/tree/master/test

[bpasero]

@jpoon check out our tests that we manage to get run: https://github.com/Microsoft/vscode/tree/master/extensions/vscode-api-tests

[bpasero]

And are those tests working when you run them via Code?

[jpoon]

Yep. Tests work when running with Code. Here's one of the failed builds: https://travis-ci.org/VSCodeVim/Vim/builds/99793291

  1) Mode Handler ctor:
     TypeError: Cannot read property 'selection' of undefined
      at new Motion (src/motion/motion.ts:22:61)
      at new ModeHandler (src/mode/modeHandler.ts:19:24)
      at Context.<anonymous> (test/mode/modeHandler.test.ts:8:27)

where line 22 of motion.ts is:

    public constructor(mode : MotionMode = null) {
        let currentPosition = vscode.window.activeTextEditor.selection.active;

[bpasero]

@jpoon in that test I am not seeing code that would open a text editor, so I wonder how the tests run green when running inside Code, maybe you can explain: https://github.com/VSCodeVim/Vim/blob/master/test/motion.test.ts

Does it make any difference if you run the tests locally using the same command you run in travis?

[jpoon]

I am not seeing code that would open a text editor

💡 -- that was it! When running in Code, it seems as though the tests automatically open a text editor, hence didn't need to do it explicitly. After refactoring my tests to open an editor....

https://travis-ci.org/VSCodeVim/Vim/builds/99801078 🎉

Thanks @bpasero .

[bpasero]

@jpoon very cool!