Support parsing recursive make build output messages from tasks.json

[alessandrod]

• VSCode Version: master
• OS Version: any

I've been using vscode for a couple of days and I'm loving it. I'm a GStreamer (http://gstreamer.freedesktop.org) developer and I would like to be able to build gst with vscode.

GStreamer uses recursive make to build itself. The way it works is that you run make at the root of the project and then make descends in the nested directories and builds the files contained in them (I'm simplifying but that's the gist).

Here's what the build output looks like:

alessandro@june:~/src/gst1/head/gst-plugins-bad$ make --print-directory
make: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad'
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-recursive
make[1]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad'
Making all in gst-libs
make[2]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad/gst-libs'
Making all in gst
...

And here's what a build error looks like:

make[2]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad/sys'
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C applemedia
make[3]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad/sys/applemedia'
  CC       libgstapplemedia_la-vtdec.lo
vtdec.c:38:1: error: unknown type name 'FOO'
FOO
^

So as you can see the error references a relative filename (vtdec.c) and earlier in the output there's a line specifying in which directory the file is located (/Users/alessandro/src/gst1/head/gst-plugins-bad/sys/applemedia).

vscode already supports relative filenames in problem matchers by letting users specify a static filePrefix to be prefixed to filenames. I'm proposing to add a method for problem matchers to modify the filePrefix as they scan build output.

[alessandrod]

Here's the tasks.json that I can use to parse the output https://gist.github.com/alessandrod/82f08750ee4c36860b3719765ad6140f

[dbaeumer]

I looked at the PR and your problem and this is not solvable that way in a general way. The problem is that multi line matchers can only loop on the last match right now. They are made to loop over matches for a single file where usually the first line of the match is the file to be compiles and then all following lines contain errors.

I your case a matcher with the line prefix patch will only be able to match one error in a directory. If you have something like this:

make[3]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad/sys/applemedia'
  CC       libgstapplemedia_la-vtdec.lo
vtdec.c:38:1: error: unknown type name 'FOO'
  CC       libgstapplemedia_la-other.lo
other.c:38:1: error: unknown type name 'OTHER'

then that second error will not be matched because the pattern for the file prefix will not match anymore.

The only solution I can think about right now is to have a file location matcher which people could specify of a matcher that looks backwards in the output. The backwards looking I actually want to avoid. This gets pretty complicated and requires that we always keep the whole output.

Is it ok to close the PR. If you want I can guide you to a solution for a file prefix matcher. But that is definitely more work.

[alessandrod]

I your case a matcher with the line prefix patch will only be able to match one error in a directory. If you have something like this:

make[3]: Entering directory `/Users/alessandro/src/gst1/head/gst-plugins-bad/sys/applemedia'
CC libgstapplemedia_la-vtdec.lo
vtdec.c:38:1: error: unknown type name 'FOO'
CC libgstapplemedia_la-other.lo
other.c:38:1: error: unknown type name 'OTHER'

then that second error will not be matched because the pattern for the file prefix will not match anymore.

That is why if you look at the pattern in tasks.json, I specified:

       "pattern": [
            {
                "regexp": "^((.*): Entering directory `(.*)'|(.*))$",
                "filePrefix": 3
            },

The |(.*) side essentially makes the "Entering directory..." pattern optional. When it does match, the filePrefix field is updated, else it stays to the last value set. This makes the matcher work for multiple errors in the same file and directory. Perhaps it's not the most elegant solution, but it does work and doesn't need buffering the whole output.

If you have a better approach in mind I'm more than happy to hear about it tho!

[dbaeumer]

I didn't catch this, however I still think that this is not working in a general case. Reason being that the first match must cover 3 lines and all following matches must cover 2 lines and the multi line matcher will not handle that right now.

IMO a better and more explicit approach would be something like this (see fileLocation property):

{
            "name": "make-compile",
            "owner": "cpp",
            "fileLocation": ["relative", { "regexp" : "^((.*): Entering directory `(.*)'$", "index": 3 }]
            "pattern": [
                {
                    "regexp": "CC (.*)"
                },
                {
                    "regexp": "^(.*):(\\d+):(\\d+):\\s+(warning|error):\\s+(.*)$",
                    "file": 1,
                    "line": 2,
                    "column": 3,
                    "severity": 4,
                    "message": 5
                }
            ]
        }

[alessandrod]

I didn't catch this, however I still think that this is not working in a general case. Reason being that the first match must cover 3 lines and all following matches must cover 2 lines and the multi line matcher will not handle that right now.

Oh that's true, I had missed that.

IMO a better and more explicit approach would be something like this (see fileLocation property):

I like this, I'll give it a go.

[dbaeumer]

Great. Let me know if you need help.

[evmar]

By the way, for compatibility with other tools that parse the make output, the ninja build system (http://ninja-build.org) also emits lines like

 Entering directory `foo'

(with same the funny quotes).

[xaljox]

I tried the "fileLocation": ["relative", { "regexp" : "^((.): Entering directory `(.)'$", "index": 3 }]
stuff, but it doesn't seem to work. The moment I put the regexp stuff in the Tasks.json file, then nothing will work, even all the tasks are gone. Also the tasks not depending on the problemMatcher are no longer available, the whole tasks list turns empty.
Perhaps the best thing to do for the moment is to have some sort of postprocessing on the output.
Thus to have something like this in the tasks.json:
make -w 2>&1 | fixpaths
The program fixpaths processes the output of the make step and by parsing the entering and leaving directory statements it adjusts the warning and error lines, so it makes the correct absolute paths. The rest of the lines it just passes through or perhaps filter it a bit to have less noise.
When options like -j8 is used it gets tricky Then there is a set of possible open directories and thus a search must be performed which possible absolute path is valid.

[xaljox]

Made a program fixpath, it can be found at:
https://github.com/xaljox/fixpath

It processes the Entering/Leaving directory lines and keeps track of the possible open directories.
Due to multi threaded invocations of make with the -j8 it is not always clear to which directory a relative path must be resolved.
All possible directories are tried to see were the file really exists and the full path to that file is then used in the altered output.

A possible corresponding tasks.json is:
{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "0.1.0",
    "command": "bash",
    "args": [
        "-c"
    ],
    "isShellCommand": true,
    "showOutput": "always",
    "suppressTaskName": true,
    "options": {
        "cwd": "${workspaceRoot}"
    },
    "tasks": [
        {
            "taskName": "make",
            "args": [
                "cd build; make -j8 -w 2>&1 | fixpath"
            ],
            "isBuildCommand": true,
            "problemMatcher": {
                "owner": "cpp",
                "fileLocation": "absolute",
                "pattern": {
                    "regexp": "^(.*):(\\d+):(\\d+):\\s+(warning|error):\\s+(.*)$",
                    "file": 1,
                    "line": 2,
                    "column": 3,
                    "severity": 4,
                    "message": 5
                }
            }
        }
    ]
}

[dbaeumer]

Cool.

[db4]

@dbaeumer

IMO a better and more explicit approach would be something like this (see fileLocation property):

{
            "name": "make-compile",
            "owner": "cpp",
            "fileLocation": ["relative", { "regexp" : "^((.*): Entering directory `(.*)'$", "index": 3 }]
            "pattern": [
                {
                    "regexp": "CC (.*)"
                },
                {
                    "regexp": "^(.*):(\\d+):(\\d+):\\s+(warning|error):\\s+(.*)$",
                    "file": 1,
                    "line": 2,
                    "column": 3,
                    "severity": 4,
                    "message": 5
                }
            ]
}

Is regexp in fileLocation already supported or it's just you proposal for future versions of VS Code? I see no mention in https://code.visualstudio.com/docs/editor/tasks-appendix that regexp is allowed there.

[dbaeumer]

No regexp in fileLocation is not supported.

[sergey-s-betke]

There is workaround with GNU make output postprocessing: Metrolog/marks@bc7bbc2

[DanTup]

@dbaeumer Is this issue to cover code-based/dynamic problem matchers that aren't restricted to regex? (eg. if a tool had a JSON output format)? I think so based on other links here, but the title isn't very clear.

[dbaeumer]

Yes, the solution for that problem will be to allow extensions to parse the output of the terminal themselves without the need of a problem matcher.

[buserror]

So how was that particular problem fixed? I'm just trying vscode and sure enough, all of my projects use sub-make, as it's a pretty standard pattern for linux development.
There are countless references to this issue in other issues, but after reading the whole log, I can't find an actual solution.
Is the current solution still involves a post-process of the output?

[alexr00]

You can probably solve this using "fileLocation": "autoDetect" in the problem matcher. This issue tracks adding a programmatic problem matcher.

[alexr00]

With "autoDetect" this is now a dupe of #59337

[vscodebot]

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for existing issues here. See also our issue reporting guidelines.

Happy Coding!

[yeggles]

With "autoDetect" this is now a dupe of #59337

Is it? It doesn't work for me. The only thing that seemed like it could work would be to allow regex expressions in the fileLocation setting. On my company's project the Makefile of the project calls a good 50+ other Makefiles and before doing so the scripts change directory so any errors reported by the make command will have broken links due to the fact that it will it will ignore the change in directory.

[alexr00]

In that case, this could be solved by programatic problem matching, which is tracked by #59337