files with a path that contains '@' are skipped #3463

ferkulat · 2021-10-26T20:47:04Z

Describe the bug
Given a compilation_database.json where all file paths contain a '@'.
For example:

{
  "directory": "C:/jenkins/workspace/MS-1234-5.6-use_withCredentials@2/b",
  "command": "C://msys64//mingw64//bin//c++.exe ",
  "file": "C:/jenkins/workspace/MS-1234-5.6-use_withCredentials@2/lol_common/components/Logger/Source/Logger.cpp"
},

The command is removed because of company internals. But it also had an '@' in there.

Codechecker prints:

[INFO 2021-10-26 19:24] - No analysis is required.
There were no compilation commands in the provided compilation database or all of them were skipped.

CodeChecker version
6.17.0

Desktop (please complete the following information)

OS: windows

Additional context
those paths are used on jenkins to create workspaces
Please notice:
The original compile_commands.json has \ in all paths. Those get replaced with / before passing it to Codechecker.
Otherwise the jenkins plugin 'warnings-ng' would not find and display them in the UI.
But as long as there are no @ in there, it works.

The text was updated successfully, but these errors were encountered:

csordasmarton · 2021-10-27T11:29:35Z

@ferkulat Do you see anything when you run the analysis (CodeChecker analyze) with the --verbose debug option?

ferkulat · 2021-10-27T12:02:51Z

I was able to reproduce it in a toy project:

PS C:\Users\User\devel\workspace@2\csv2xls\cmake-build-debug> cd ..
PS C:\Users\User\devel\workspace@2\csv2xls> CodeChecker.exe analyze .\cmake-build-debug\compile_commands.json --verbose debug -o .\cmake-build-debug\results
[DEBUG][2021-10-27 13:57:50] {system} [15024] <26628> - analyze.py:937 main() - args: Namespace(logfile='.\\cmake-build-debug\\compile_commands.json', jobs=24, keep_gcc_include_fixed=False, keep_gcc_intrin=False, output_format='plist', makefile=False, compile_uniqueing='none', config_file=None, analyzer_config=['clang-tidy:HeaderFilterRegex=.*'], func=<function main at 0x000001531A9F6280>, func_process_config_file=<function process_config_file at 0x000001531A9C2CA0>, verbose='debug', output_path='C:\\Users\\User\\devel\\workspace@2\\csv2xls\\cmake-build-debug\\results')
[DEBUG][2021-10-27 13:57:50] {system} [15024] <26628> - analyze.py:938 main() - Output will be stored to: 'C:\Users\User\devel\workspace@2\csv2xls\cmake-build-debug\results'
[DEBUG][2021-10-27 13:57:50] {buildlogger} [15024] <26628> - log_parser.py:1356 parse_unique_log() - Writing compiler info into:C:\Users\User\devel\workspace@2\csv2xls\cmake-build-debug\results\compiler_info.json
[DEBUG][2021-10-27 13:57:50] {buildlogger} [15024] <26628> - log_parser.py:1359 parse_unique_log() - Parsing log file done.
[INFO][2021-10-27 13:57:50] {system} [15024] <26628> - analyze.py:963 main() - No analysis is required.
There were no compilation commands in the provided compilation database or all of them were skipped.
PS C:\Users\User\devel\workspace@2\csv2xls>

compile_commands.json.gz

The file cmake-build-debug\results\compiler_info.json contains just this: {}

csordasmarton · 2021-10-27T14:22:52Z

I think we skip compilation commands in this part of the source code but I don't know why:

codechecker/analyzer/codechecker_analyzer/buildlog/log_parser.py

Lines 1313 to 1316 in d2db1b4

    
           if not action.lang: 
        
               continue 
        
           if action.action_type != BuildAction.COMPILE: 
        
               continue

On Windows CodeChecker is not really tested.

ferkulat · 2021-10-27T15:14:34Z

I could even reproduce it on linux with the same result:

[marcel@xps csv2xls@2]$ /home/marcel/.local/bin/CodeChecker analyze b/compile_commands.json -o results
[INFO 2021-10-27 17:06] - No analysis is required.
There were no compilation commands in the provided compilation database or all of them were skipped.
[marcel@xps csv2xls@2]$ /home/marcel/.local/bin/CodeChecker analyze --verbose debug b/compile_commands.json -o results
[DEBUG][2021-10-27 17:12:54] {system} [1034223] <140704287356736> - analyze.py:952 main() - args: Namespace(logfile='b/compile_commands.json', jobs=8, keep_gcc_include_fixed=False, keep_gcc_intrin=False, output_format='plist', makefile=False, compile_uniqueing='none', config_file=None, analyzer_config=['clang-tidy:HeaderFilterRegex=.*'], func=<function main at 0x7ff843e318b0>, func_process_config_file=<function process_config_file at 0x7ff843e31430>, verbose='debug', output_path='/home/marcel/ClionProjects/csv2xls@2/results')
[DEBUG][2021-10-27 17:12:54] {system} [1034223] <140704287356736> - analyze.py:953 main() - Output will be stored to: '/home/marcel/ClionProjects/csv2xls@2/results'
[DEBUG][2021-10-27 17:12:54] {buildlogger} [1034223] <140704287356736> - log_parser.py:1356 parse_unique_log() - Writing compiler info into:/home/marcel/ClionProjects/csv2xls@2/results/compiler_info.json
[DEBUG][2021-10-27 17:12:54] {buildlogger} [1034223] <140704287356736> - log_parser.py:1359 parse_unique_log() - Parsing log file done.
[INFO][2021-10-27 17:12:54] {system} [1034223] <140704287356736> - analyze.py:978 main() - No analysis is required.
There were no compilation commands in the provided compilation database or all of them were skipped.

compile_commands.zip

Before submitting this issue, I wanted to debug it myself.
But I am not a python developer and I fail to set it up with pycharm.
I find pipenv and venv confusing.

csordasmarton · 2021-10-28T07:24:23Z

The installation steps can be found here: https://github.com/Ericsson/codechecker#linux

If you want to develop CodeChecker the only difference in these steps is that you have to install our developement environment instead of production. So instead of make venv use make venv_dev and you have to active this dev env: source $PWD/venv_dev/bin/activate

ferkulat · 2021-10-30T12:06:38Z

@csordasmarton I could debug to the part where it skips files that have an @ in there.
Please have a look at the screenshot.
If the filename has an @ in there it does not add this file, but tries to add entries from source_files, which is empty at this moment.
This was added in commit 2cb9bc1
I am tempted to remove this part, but at the moment don't know what those "response files" are.
And how important the support if it is.

ferkulat · 2021-10-30T12:28:11Z

@csordasmarton If I understand it correctly, the file name of a response file starts with an @.
To fix this, I would remove the path part from the file entry, before checking for @?

csordasmarton · 2021-10-30T12:44:48Z

See: https://llvm.org/docs/CommandLine.html#response-files

Some systems, such as certain variants of Microsoft Windows and some older Unices have a relatively low limit on command-line length. It is therefore customary to use the so-called ‘response files’ to circumvent this restriction. These files are mentioned on the command-line (using the “@file”) syntax. The program reads these files and inserts the contents into argv, thereby working around the command-line length limits.

I think we can change this line to check whether a file starts with a @ instead of containing it:

codechecker/analyzer/codechecker_analyzer/buildlog/log_parser.py

Line 1181 in 4be1eca

if '@' in entry['file']:

if '@' in entry['file']: vs entry['file'].startswith('@')

@csordasmarton

Instead of checking for `@` in the file name, check if a file name starts with `@`. Fixes Ericsson#3463. Codechecker supports response files. These are files which you can pass to clang to work around limits in command line length. Those files names start with a `@`. The CI/CD solution "Jenkins" creates workspaces by putting `@` in the path name. A file entry in compile_commands.json could look like: "file": "/home/workspace/branchname@2/somefile.cpp" In this case all files in the compile data base are treated as response file. And the analyzer skips all files and does nothing. With the help of @csordasmarton this got fixed by checking if a file starts with an `@`.

csordasmarton added analyzer 📈 Related to the analyze commands (analysis driver) bug 🐛 labels Oct 27, 2021

ferkulat mentioned this issue Oct 30, 2021

[analyzer] Fix check for response files #3474

Merged

csordasmarton closed this as completed in #3474 Nov 3, 2021

whisperity added the platform-Windows 🖥 label Dec 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

files with a path that contains '@' are skipped #3463

files with a path that contains '@' are skipped #3463

ferkulat commented Oct 26, 2021 •

edited

Loading

csordasmarton commented Oct 27, 2021

ferkulat commented Oct 27, 2021 •

edited

Loading

csordasmarton commented Oct 27, 2021

ferkulat commented Oct 27, 2021

csordasmarton commented Oct 28, 2021

ferkulat commented Oct 30, 2021

ferkulat commented Oct 30, 2021

csordasmarton commented Oct 30, 2021

files with a path that contains '@' are skipped #3463

files with a path that contains '@' are skipped #3463

Comments

ferkulat commented Oct 26, 2021 • edited Loading

csordasmarton commented Oct 27, 2021

ferkulat commented Oct 27, 2021 • edited Loading

csordasmarton commented Oct 27, 2021

ferkulat commented Oct 27, 2021

csordasmarton commented Oct 28, 2021

ferkulat commented Oct 30, 2021

ferkulat commented Oct 30, 2021

csordasmarton commented Oct 30, 2021

ferkulat commented Oct 26, 2021 •

edited

Loading

ferkulat commented Oct 27, 2021 •

edited

Loading