VS Python analysis engine integration #1144

MikhailArkhipov · 2018-03-21T18:51:05Z

Fixes #1101
Fixes #1087
Fixes #1063
Fixes #1051
Fixes #1004
Fixes #998
Fixes #408
Fixes #82

VSC part, not active for general audience yet. (Needs integration in the PTVS repo first).
Not all functionality is exactly like Jedi, see Meta for tracking Language Server issues #1006
Tests are not active on Appveyor
There is a task to run tests with the VS Analysis engine (couple of signature tests can fail b/c of Generate SignatureAnalysis without using GetExpressionRange PTVS#3171.
Jedi and VS Analysis engine are activated via python.jediEnabled setting (default true)
Tests pass on Mac except issue with import *
Activation is separated into 'classic' and 'analysis`
Not tested on Linux
.NET Core 2.x SDK or at least runtime is required for running VS Python Engine
Building the engine will be documented separately
There is nothing to add to news just yet

This pull request:

Has a title summarizes what is changing
Includes a news entry file
Has unit tests & code coverage is not adversely affected (within reason)
Works on all actively maintained versions of Python (e.g. Python 2.7 & the latest Python 3 release)
Works on Windows 10, macOS

codecov · 2018-03-21T19:06:23Z

Codecov Report

Merging #1144 into master will decrease coverage by 0.02%.
The diff coverage is 26.31%.

@@            Coverage Diff             @@
##           master    #1144      +/-   ##
==========================================
- Coverage   71.93%   71.91%   -0.03%     
==========================================
  Files         260      260              
  Lines       11933    11951      +18     
  Branches     2121     2123       +2     
==========================================
+ Hits         8584     8594      +10     
- Misses       3220     3231      +11     
+ Partials      129      126       -3

Impacted Files	Coverage Δ
src/client/common/types.ts	`100% <ø> (ø)`	⬆️
src/client/common/configuration/service.ts	`69.23% <26.31%> (-24.89%)`	⬇️
src/client/linters/baseLinter.ts	`87.5% <0%> (-1.05%)`	⬇️
src/client/debugger/Main.ts	`50.12% <0%> (-0.5%)`	⬇️
...rc/client/debugger/PythonProcessCallbackHandler.ts	`52.3% <0%> (+0.65%)`	⬆️
src/client/debugger/PythonProcess.ts	`48.75% <0%> (+2.91%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b9dd72...3e071e0. Read the comment docs.

DonJayamanne

Will need to run through the code one again to review this.

DonJayamanne · 2018-03-21T21:03:52Z

src/client/activation/analysis.ts

+            synchronize: {
+                configurationSection: PYTHON
+            },
+            outputChannel: outputChannel,


Instead of outputChannel: outputChannel, you can write outputChannel

C# habits...

DonJayamanne · 2018-03-21T21:04:08Z

src/client/activation/analysis.ts

+            outputChannel: outputChannel,
+            initializationOptions: {
+                interpreter: {
+                    properties: properties


Instead of properties: properties you can write properties

DonJayamanne · 2018-03-21T21:10:10Z

src/client/activation/analysis.ts

+
+    private async getInterpreterData(): Promise<InterpreterData> {
+        const execService = await this.executionFactory.create();
+        const result = await execService.exec(['-c', 'import sys; print(sys.version); print(sys.prefix)'], {});


Please remember, sys.version can contain strings. E.g. if using a beta version of Python or similar, the version could be 3.7.1Beta.
Please confirm this is not a problem.
My suggestion is to use sys.version_info (safest option).

PTVS uses just major part, but yes, cleaner is better.

DonJayamanne · 2018-03-21T21:14:03Z

src/client/activation/analysis.ts

+
+        // Microsoft Python code analysis engine needs full path to the interpreter
+        const interpreterService = this.services.get<IInterpreterService>(IInterpreterService);
+        const interpreter = await interpreterService.getActiveInterpreter();


How does this analysis engine work with a multi root workspace?
E.g. workspace folder A & B contain two separate versions of Python.
Will we have only one analysis engine or two, will they all use the same version of python?

If these haven't been considered yet, please create an issue to track this so it gets resolved eventually.

Yes, good point. So far VS project has same interpreter. But not per folder. However, solution can have multiple projects with different interpreters.

#1149

DonJayamanne · 2018-03-21T21:15:21Z

src/client/activation/analysis.ts

+    }
+
+    private async getSearchPaths(): Promise<string> {
+        const execService = await this.executionFactory.create();


We're assuming that we're always dealing with a single workspace here

Yep. Same #1149

Added comments

DonJayamanne · 2018-03-21T21:18:01Z

src/client/common/configuration/service.ts

@@ -32,6 +39,10 @@ export class ConfigurationService implements IConfigurationService {
        return process.env.VSC_PYTHON_CI_TEST === '1';
    }

+    public async checkDependencies(): Promise<boolean> {


I don't think checking dependencies should go into the ConfigurationService (configuration service deals with config settings)

OK. Configuration sounded like related to dependency checks. I'll separate.

DonJayamanne · 2018-03-21T21:23:30Z

src/test/autocomplete/base.test.ts

@@ -95,6 +97,10 @@ suite('Autocomplete', () => {

    // https://github.com/DonJayamanne/pythonVSCode/issues/630
    test('For "abc.decorators"', async () => {
+        // Disabled for MS Python Code Analysis, see https://github.com/Microsoft/PTVS/issues/3857
+        if (IS_ANALYSIS_ENGINE_TEST) {


Please can we add an issue in our Report to remove this condition. Else even if PTVS issue is resolved, we'll never end up removing this if condition unless someone remembers to do so.
Having a github issue will remind of this.

It is more to separate Jedi from PTVS rather than for a specific issue. Also to run PTVS tests separately on Appveyor. #1150

DonJayamanne · 2018-03-21T21:23:48Z

src/test/autocomplete/base.test.ts

        }).then(done, done);
    });

    // https://github.com/Microsoft/vscode-python/issues/110
    test('Suppress in strings/comments', async () => {
+        // Excluded from MS Python Code Analysis b/c skipping of strings and comments
+        // is not yet there. See https://github.com/Microsoft/PTVS/issues/3798
+        if (IS_ANALYSIS_ENGINE_TEST) {


Please can we add an issue in our Report to remove this condition. Else even if PTVS issue is resolved, we'll never end up removing this if condition unless someone remembers to do so.
Having a github issue will remind of this.

DonJayamanne · 2018-03-21T21:24:03Z

src/test/autocomplete/pep484.test.ts

-    suiteSetup(async () => {
+    suiteSetup(async function () {
+        // https://github.com/Microsoft/PTVS/issues/3917
+        if (IS_ANALYSIS_ENGINE_TEST) {


Please can we add an issue in our Report to remove this condition. Else even if PTVS issue is resolved, we'll never end up removing this if condition unless someone remembers to do so.
Having a github issue will remind of this.

DonJayamanne · 2018-03-21T21:24:11Z

src/test/autocomplete/pep526.test.ts

-    suiteSetup(async () => {
+    suiteSetup(async function () {
+        // https://github.com/Microsoft/PTVS/issues/3917
+        if (IS_ANALYSIS_ENGINE_TEST) {


Please can we add an issue in our Report to remove this condition. Else even if PTVS issue is resolved, we'll never end up removing this if condition unless someone remembers to do so.
Having a github issue will remind of this.

DonJayamanne · 2018-03-22T17:58:24Z

src/client/activation/analysis.ts

+            if (settings.autoComplete) {
+                const extraPaths = settings.autoComplete.extraPaths;
+                if (extraPaths && extraPaths.length > 0) {
+                    searchPaths = `${searchPaths};${extraPaths.join(';')}`;


Isn't the path delimeter OS dependent. On Linx path seprators is :
You can use path.sep

PTVS engine split paths by ;. I believe ; is universal for providing multiple paths. Alternative is to marshal array over to LS.

is universal for providing multiple paths.

No, its not. Try the following in your terminal (linux/mac)
echo $PATH

DonJayamanne · 2018-03-22T18:03:49Z

src/client/activation/analysis.ts

+            // tslint:disable-next-line:no-string-literal
+            properties['PrefixPath'] = interpreterData.prefix;
+            // tslint:disable-next-line:no-string-literal
+            properties['DatabasePath'] = path.join(context.extensionPath, analysisEngineFolder);


What is this folder path for?

Are we creating files?

Will having two instances of VS Code or multi root workspaces affect this adversely.

This folder will get deleted everytime the user updates their extension.

When storing files you're better off using context.storagePath

Yes, it creates files. They are in unique folders for different interpreters and paths.

It is OK (and good) to delete folder on upgrade since format of the intellisense cache can change. It will be rebuilt.

Spoke to @huguesv today about these directories. He mentioned that we won't need to create/delete them every time. According to him they will never change (unless of course the PTVS changes).
Possible he's not aware of some of the new changes.

Either way can we somehow try to minimize this or re-use folders rather than creating one per each interpreter.
This could result in a large number of directories (if using virtual envs or pyenv, then each workspace could potentially have a separate environment).

Or please create a separate issue for this (rather than trying to solve this in this PR).

Does PTVS have the ability to re-create/rebuild this database?
I.e. do we need to consider adding the ability (later as a separate PR) for users to re-build this database?
I know a simple solution is to uninstall/re-install the extension, but that sounds like a hack and not very user friendly. But before we do, is such a feature necessary? If yes, lets do that in a separate PR/issue.

DonJayamanne · 2018-03-22T18:07:41Z

src/client/activation/analysis.ts

+        const paths = result.stdout.split(',')
+            .filter(p => this.isValidPath(p))
+            .map(p => this.pathCleanup(p));
+        return paths.join(';');


The path separator OS dependent (if that's what this is).
You would want to use path.sep instead of hard coding ;

PTVS splits multiple paths by ; while path.sep is \ vs / I believe

sorry, I meant path.delimiter

DonJayamanne · 2018-03-22T18:09:43Z

src/client/activation/analysis.ts

+    private async isDotNetInstalled(): Promise<boolean> {
+        const ps = this.services.get<IProcessService>(IProcessService);
+        const result = await ps.exec('dotnet', ['--version']);
+        return result.stdout.trim().startsWith('2.');


You would need a trycatch in case dotnet crashes for some reason without returning errors in stderr

const result = await ps.exec('dotnet', ['--version']).catch(()= > { return {stdout:''};})

DonJayamanne · 2018-03-24T04:56:02Z

src/client/activation/downloader.ts

            await this.unpackArchive(context.extensionPath, localTempFilePath);
        } finally {
            fs.unlink(localTempFilePath, noop);
        }
    }

    private async downloadFile(): Promise<string> {
-        const platformString = this.getPlatformDesignator();
+        const platformString = this.platformData.getPlatformDesignator();


You might want to display the progress in VS Code using the VS Code API:
https://code.visualstudio.com/docs/extensionAPI/vscode-api#ProgressOptions

Check out the withProgress method.

DonJayamanne · 2018-03-24T05:00:46Z

.vscode/tasks.json

            "type": "gulp",
-            "task": "watch",
+            "task": "hygiene-watch",


Not sure this will improve performance, let me know.
The only other solution I can think of is to use tsc watch and run hygiene so it display errors ONLY in terminal window and throw errors when trying to commit (problem with this is you won't see errors in problems window, but faster).

Or the fastest approach is tsc --watch -p ./ & don't bother with hygiene just let it complain during commit

DonJayamanne · 2018-03-24T05:05:02Z

src/client/activation/downloader.ts

+        return true;
+    }
+
+    private reportDownloadSize(response: IncomingMessage): number {


Please use an existing package to download a file and report the progress (https://www.npmjs.com/package/request-progress).
This does exactly what you need.

DonJayamanne · 2018-03-24T05:06:02Z

src/client/activation/downloader.ts

+        this.output.append('done.');
+    }
+
+    private handleError(message: string) {


Please use request-progress npm

DonJayamanne · 2018-03-24T05:06:27Z

src/client/activation/downloader.ts

+        if (response.rawHeaders.length >= 2 && response.rawHeaders[0] === 'Content-Length') {
+            const size = parseInt(response.rawHeaders[1], 10);
+            if (size > 0) {
+                this.output.append(` ${Math.round(size / 1024)} KB...`);


Please report download progress using VS Code api, so users too can see this.

DonJayamanne · 2018-03-24T05:07:34Z

src/client/activation/downloader.ts

+                deferred.resolve();
+            })
+            .on('error', (err) => {
+                this.handleError(`Unable to unpack downloaded file. Error ${err}.`);


You'll need to bubble up the error, (throwing exceptions inside handleError will not work.
deferred.reject(err)

I'd throw an exception here and call handleError from catch in downloadAnalysisEngine.

DonJayamanne · 2018-03-24T05:56:12Z

src/client/activation/downloader.ts

+        });
+
+        let firstResponse = true;
+        https.get(uri, (response) => {


replace with npm package.

DonJayamanne · 2018-03-24T05:57:28Z

src/client/activation/downloader.ts

+                deferred.resolve();
+            })
+            .on('error', (err) => {
+                this.handleError(`Unable to unpack downloaded file. Error ${err}.`);


change to .on('error', deferred.reject.bind(deferred))
else it won't work

DonJayamanne · 2018-03-24T05:58:08Z

src/client/activation/downloader.ts

+        this.output.append('Verifying download... ');
+        const verifier = new HashVerifier();
+        if (!await verifier.verifyHash(filePath, this.platformData.getExpectedHash())) {
+            this.handleError('Hash of the downloaded file does not match.');


No need to call handleError, just throw the exception here throw new Error(.....), let the calling method handle and log all exceptions.

DonJayamanne · 2018-03-24T06:06:32Z

src/client/activation/platformData.ts

+        if (this.platform.isLinux && this.platform.is64bit) {
+            return 'linux-x64';
+        }
+        throw new Error('Python Analysis Engine does not support 32-bit Linux.');


I've created an issue for this, we need to document this limitation in our documentation https://github.com/Microsoft/vscode-python/issues/new

DonJayamanne · 2018-03-24T06:07:25Z

src/client/common/types.ts

-    readonly terminal: ITerminalSettings;
-    readonly sortImports: ISortImportSettings;
-    readonly workspaceSymbols: IWorkspaceSymbolSettings;
+    readonly linting: ILintingSettings | undefined;


No need to change anything (you can also use readonly linting?: ILintingSettings)

DonJayamanne · 2018-03-25T07:10:09Z

src/client/activation/analysisEngineHashes.ts

+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+export const analysis_engine_win_x86_sha512 = '6ce1f4a4a83ea7a662f2034de3f83ba8342863e9c1d97dfc23981931e69d1358b8953ae9fbec05be4d755118d695c974a39cd4fd1d6e4735bf5d60d58ab44fe9';


Will this change for each build? If yes, then we might want to put this into package.json

Yes. The file is generated and injected in VSTS. Default state is empty string which means 'dont check'

This reverts commit e240c3f.

This reverts commit af573be.

MikhailArkhipov · 2018-03-27T23:24:50Z

vscode-python.csproj

@@ -0,0 +1,20 @@
+<Project Sdk="Microsoft.NET.Sdk">


This file is for signing step only. VSTS task wants C# project.

MikhailArkhipov · 2018-03-28T05:24:04Z

Need to restore lost changes after rebase

DonJayamanne requested changes Mar 21, 2018

View reviewed changes

DonJayamanne requested changes Mar 22, 2018

View reviewed changes

DonJayamanne requested changes Mar 24, 2018

View reviewed changes

DonJayamanne reviewed Mar 25, 2018

View reviewed changes

MikhailArkhipov added 24 commits March 27, 2018 14:58

Undo changes

a764bc9

Test fixes

9d1b2cc

Increase timeout

a91291a

Remove double event listening

bf266af

Remove test

7bc6bd6

Revert "Remove test"

8ce8b48

This reverts commit e240c3f.

Revert "Remove double event listening"

e3a549e

This reverts commit af573be.

Undo changes

c879aa6

Test fixes

716968f

.NET Core check

63a1385

Better find dotnet

7ea7717

Fix pip test

75be0da

Linting tests

2701768

Undo accidental changes

a22e705

Add clone and build PTVS

5441644

Appveyor PTVS build

97175e1

Fix slashes

1e97e6a

Enable build

86ac269

Try absolute path

7b4e73c

Fix xcopy switch

bd1fd77

Activate Analysis Engine test on Appveyor

72b91f1

Temporary only run new tests

e32320b

Disable PEP hint tests

55bb349

Test fix

3826ffa

MikhailArkhipov added 16 commits March 27, 2018 15:08

Disable appveyor build and tests for PTVS for now

b62ba25

Remove analysis engine test from the set

a13770b

Remove VS image for now

ac646c8

Build/sign VSXI project

f754812

Run vsce from cmd

826dcee

Rename

8cb0c44

Abs path vsce

05d4d23

Path

7fd6a8b

Move project

bbcc65d

Ignore publishing project

8d170c0

Try csproj

869b38c

Add framework

d8c80f6

Ignore build output folder

56e6b56

Package before build

ce6b360

Try batch instead of PS

a7847d8

Fix path quotes

05c0bf0

MikhailArkhipov commented Mar 27, 2018

View reviewed changes

Changes lost on squash

3e071e0

MikhailArkhipov closed this Mar 28, 2018

MikhailArkhipov mentioned this pull request Mar 28, 2018

VS Python analysis engine integration #1231

Merged

5 tasks

lock bot locked as resolved and limited conversation to collaborators Jul 31, 2019

VS Python analysis engine integration #1144

VS Python analysis engine integration #1144

Conversation

MikhailArkhipov commented Mar 21, 2018 • edited Loading

codecov bot commented Mar 21, 2018 • edited Loading

Codecov Report

DonJayamanne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MikhailArkhipov Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MikhailArkhipov commented Mar 28, 2018

MikhailArkhipov commented Mar 21, 2018 •

edited

Loading

codecov bot commented Mar 21, 2018 •

edited

Loading

MikhailArkhipov Mar 22, 2018 •

edited

Loading