Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

«log» sometimes does not put in "file": the subpath #3860

Closed
dilyanpalauzov opened this issue Mar 15, 2023 · 12 comments · Fixed by #4146
Closed

«log» sometimes does not put in "file": the subpath #3860

dilyanpalauzov opened this issue Mar 15, 2023 · 12 comments · Fixed by #4146

Comments

@dilyanpalauzov
Copy link
Contributor

I call

$ CodeChecker log -b "make" -o json.json &> cc-log-out

Then cc-log-out contains:

make[2]: Entering directory '/git/productname/something/src/abcxyz'
mkdir -p /git/productname/something/obj-g/abcxyzb0.dll/unix
g++  -c -DLINUX -Dlinux -fPIC -DPIC -DTATRA_OFFSET -I.  -I/git/productname/something/src/include  -I/git/productname/something/src/unix/include -m32 -DUSE_FFI  -fprofile-arcs -ftest-coverage -pg -g -DDEBUG -D_DEBUG  -I. -Dunix  -D_UNIX_ -DUNIX   -DSTRICT  -o /git/productname/something/obj-g/abcxyzb0.dll/unix/tatra.o /git/productname/something/src/abcxyz/unix/tatra.cpp

/git/productname/something/unix/custom_link_command /git/productname/something/lib-g/abcxyzb0.dll  /git/productname/something/obj-g/abcxyzb0.dll/abcxyz.def -m32 -fprofile-arcs -ftest-coverage  -ldl -lpthread  [… many .o files …] /git/productname/something/obj-g/abcxyzb0.dll/unix/tatra.o  -L/git/productname/something/lib-g -L/git/productname/something/lib-g
make[2]: Leaving directory '/git/productname/something/src/abcxyz'

custom_link_command calls c++. Despite having «dll» this is all Linux code. Then json.json includes:

{
         "directory": "/git/productname/something/src/abcxyz",
         "command": "/usr/libexec/gcc/x86_64-redhat-linux/12/cc1plus  -quiet -I . -I /git/productname/something/src/include -I 
/git/productname/something/src/unix/include -I . -imultilib 32 -D_GNU_SOURCE -D LINUX -D linux -D PIC -D TATRA_OFFSET -D USE_FFI -D DEBUG -D _DEBUG -D unix -D _UNIX_ -D UNIX -D STRICT  /git/productname/something/src/abcxyz/unix/tatra.cpp -quiet -dumpdir /git/productname/something/obj-g/abcxyzb0.dll/unix/ -dumpbase tatra.cpp  -dumpbase-ext .cpp -m32 -mtune=generic -march=i686 -g -p -fPIC -fprofile-arcs -ftest-coverage -o /tmp/ccfnhb0i.s",
         "file": "/git/productname/something/src/abcxyz/unix/tatra.cpp"
}
,
{
         "directory": "/git/productname/something/src/abcxyz",
         "command": "/usr/libexec/gcc/x86_64-redhat-linux/12/cc1plus -quiet -I . -I /git/productname/something/src/include -I 
/git/productname/something/src/unix/include -I . -imultilib 32 -D_GNU_SOURCE -D LINUX -D linux -D PIC -D TATRA_OFFSET -D USE_FFI -D DEBUG -D _DEBUG -D unix -D _UNIX_ -D UNIX -D STRICT  /git/productname/something/src/abcxyz/unix/tatra.cpp -quiet -dumpdir /git/productname/something/obj-g/abcxyzb0.dll/unix/ -dumpbase tatra.cpp -dumpbase-ext .cpp -m32 -mtune=generic -march=i686 -g -p -fPIC -fprofile-arcs -ftest-coverage -o /tmp/ccfnhb0i.s",
         "file": "tatra.cpp"
}

As next I execute «CodeChecker analyze json.json --analyzers clangsa --ctu --z3-refutation on -o x123», which runs in parallel. It
complains:

[INFO 2023-03-15 16:28] - [71/372] tatra.cpp
… text cut …
[INFO 2023-03-15 16:28] - [82/372] tatra.cpp
[INFO 2023-03-15 16:28] - [83/372] file.cpp
[ERROR 2023-03-15 16:28] - Error generating AST.

command:

/usr/local/bin/clang-15 -c -x c++ -I /git/productname/something/src/abcxyz -I  /git/productname/something/src/include -I 
/git/productname/something/src/unix/include -I /git/productname/something/src/abcxyz -D_GNU_SOURCE -D LINUX -D linux -D 
 PIC -D TATRA_OFFSET -D USE_FFI -D DEBUG -D _DEBUG -D unix -D _UNIX_ -D UNIX -D STRICT -m32 -mtune=generic -march=i686 -fPIC -fprofile-arcs -ftest-coverage -isystem /usr/include/c++/12 -isystem  /usr/include/c++/12/x86_64-redhat-linux -isystem 
/usr/include/c++/12/backward -isystem /usr/local/include -isystem /usr/include /git/productname/something/src/abcxyz/tatra.cpp -std=gnu++17 -emit-ast -D__clang_analyzer__ -w -o /git/productname/something/unix/x123/ctu-dir/ast/git/productname/something/src/abcxyz/tatra.cpp.ast

stderr:

clang-15: error: no such file or directory: 
'/git/productname/something/src/abcxyz/tatra.cpp'
clang-15: error: no input files

[ERROR 2023-03-15 16:28] - Error generating function map.

command:

/usr/local/bin/clang-extdef-mapping  /git/productname/something/src/abcxyz/tatra.cpp -- -c -x c++ -I 
/git/productname/something/src/abcxyz -I /git/productname/something/src/include -I /git/productname/something/src/unix/include -I /git/productname/something/src/abcxyz -D_GNU_SOURCE -D LINUX -D linux -D  PIC -D TATRA_OFFSET -D USE_FFI -D DEBUG -D _DEBUG -D unix -D _UNIX_ -D UNIX -D STRICT -m32 -mtune=generic -march=i686 -fPIC -fprofile-arcs  -ftest-coverage -isystem /usr/include/c++/12 -isystem /usr/include/c++/12/x86_64-redhat-linux -isystem  /usr/include/c++/12/backward -isystem /usr/local/include -isystem /usr/include -std=gnu++17

stderr:

error: no such file or directory: 
'/git/productname/something/src/abcxyz/tatra.cpp'
error: no input files
error: unable to handle compilation, expected exactly one compiler job 
in ''
Error while processing /git/productname/something/src/abcxyz/tatra.cpp.

The problem report is, that for "directory": "/git/productname/something/src/abcxyz" “log” creates two files:

"file": "/git/productname/something/src/abcxyz/unix/tatra.cpp" and
"file": "tatra.cpp"

but these are supposed to be the same file. In particular /git/productname/something/src/abcxyz/tatra.cpp does not exist.

I use the latest codechecker (git commit eac25bc ) compiled from source and I run codechecker within its virtualevn (created by «make venv»).

@dilyanpalauzov
Copy link
Contributor Author

This is the output when log -verbose debug is executed:

[INFO 2023-03-16 17:17:27][src/ldlogger-logger.c:192] - Processing command: /usr/libexec/gcc/x86_64-redhat-linux/12/cc1plus /usr/libexec/gcc/x86_64-redhat-linux/12/cc1plus -quiet -I . -I /git/productname/something/src/include -I /git/productname/something/src/unix/include -I . -imultilib 32 -D_GNU_SOURCE -D LINUX -D linux -D PIC -D TATRA_OFFSET -D USE_FFI -D unix -D _UNIX_ -D UNIX -D STRICT /git/productname/something/src/reutil/unix/tatra.cpp -quiet -dumpdir /git/productname/something/obj/abcxyz0.dll/unix/ -dumpbase tatra.cpp -dumpbase-ext .cpp -m32 -mtune=generic -march=i686 -p -fPIC -fprofile-arcs -ftest-coverage -o /tmp/cckO2bgC.s
[INFO 2023-03-16 17:17:27][src/ldlogger-logger.c:192] - Processing command: as as -I . -I /git/productname/something/src/include -I /git/productname/something/src/unix/include -I . --32 -o /git/productname/something/obj/abcxyz0.dll/unix/tatra.o /tmp/cckO2bgC.s
[INFO 2023-03-16 17:17:27][src/ldlogger-tool.c:165] - 'as' does not match any program name! Current environment variables are: CC_LOGGER_GCC_LIKE (gcc:g++:clang:clang++:cc:c++), CC_LOGGER_JAVAC_LIKE()
[INFO 2023-03-16 17:17:28][src/ldlogger-logger.c:192] - Processing command: /usr/bin/sed sed s/\.dll$//
[INFO 2023-03-16 17:17:28][src/ldlogger-tool.c:165] - '/usr/bin/sed' does not match any program name! Current environment variables are: CC_LOGGER_GCC_LIKE (gcc:g++:clang:clang++:cc:c++), CC_LOGGER_JAVAC_LIKE()
[INFO 2023-03-16 17:17:28][src/ldlogger-logger.c:192] - Processing command: /usr/bin/basename basename /git/productname/something/lib/abcxyz0
[INFO 2023-03-16 17:17:28][src/ldlogger-tool.c:165] - '/usr/bin/basename' does not match any program name! Current environment variables are: CC_LOGGER_GCC_LIKE (gcc:g++:clang:clang++:cc:c++), CC_LOGGER_JAVAC_LIKE()
[INFO 2023-03-16 17:17:28][src/ldlogger-logger.c:192] - Processing command: /usr/bin/dirname dirname /git/productname/something/lib/abcxyz0.dll
[INFO 2023-03-16 17:17:28][src/ldlogger-tool.c:165] - '/usr/bin/dirname' does not match any program name! Current environment variables are: CC_LOGGER_GCC_LIKE (gcc:g++:clang:clang++:cc:c++), CC_LOGGER_JAVAC_LIKE()
[INFO 2023-03-16 17:17:28][src/ldlogger-logger.c:192] - Processing command: /usr/bin/c++ c++ -o /git/productname/something/lib/abcxyz0.dll -shared -Wl,-Bsymbolic -Wl,--retain-symbols-file=/git/productname/something/obj/abcxyz0.dll/reutil.def -m32 -fprofile-arcs -ftest-coverage -ldl -lpthread [...many .o files...] /git/productname/something/obj/abcxyz0.dll/unix/tatra.o -L/git/productname/something/lib -L/git/productname/something/lib
[WARNING 2023-03-16 17:17:28][src/ldlogger-tool-gcc.c:557] - No source file was found.

So the build runs basename and dirname.

@dilyanpalauzov
Copy link
Contributor Author

It turned out that the parameter after cc1plus -dumpbase is considered as input file, when it has appropriate filename extension. But it is not an input file. The patch below skips the parameter after -dumpbase. With the patch the generated log.json file contains `"command": ".../cc1plus … -dumpbase -dumpbase-ext …" . That is, it skips the parameter after -dumpbase, but keeps -dumpbase as part of the command. Nevertheless this change does solve my problem.

diff --git a/analyzer/tools/build-logger/src/ldlogger-tool-gcc.c b/analyzer/tools/build-logger/src/ldlogger-tool-gcc.c
index 6a67c714..f3503d39 100644
--- a/analyzer/tools/build-logger/src/ldlogger-tool-gcc.c
+++ b/analyzer/tools/build-logger/src/ldlogger-tool-gcc.c
@@ -392,6 +392,10 @@ int loggerGccParserCollectActions(
        */
       if (startsWith(current, "-I"))
        lastIncPos = action->arguments.size + (current[2] ? 0 : 1);
+      else if (startsWith(current, "-dumpbase"))
+        /* file after -dumpbase must be ignored, it is not a source file
+         */
+        i++;
       else if (startsWith(current, "-isystem"))
         lastSysIncPos = action->arguments.size + (current[8] ? 0 : 1);

@dilyanpalauzov
Copy link
Contributor Author

Can this get some attention, please?

Here the problem is described very well and a suboptimal fix is included. Somebody with experience on the codechecker internals can easily create a perfect fix.

@bruntib
Copy link
Contributor

bruntib commented May 22, 2023

Hi,
There is a recent discussion about an issue which, I think, has the same root cause: #3906. Could you, please, check if using make version < 4.3 fixes the issue? If it does, then an improvement of the logger should be a long-term solution which we should schedule to implement sometime.

@dilyanpalauzov
Copy link
Contributor Author

With GNU make 4.2.1 this problem does not happen.

@bruntib
Copy link
Contributor

bruntib commented Jun 21, 2023

Hi,
The issue is supposed to be fixed by #3930. Could you please check if it works with your original make version?
Thank you!

@dilyanpalauzov
Copy link
Contributor Author

No, this does not help. With make 4.4 it fails, with make 4.2.1 it works.

@dilyanpalauzov
Copy link
Contributor Author

By it fails I mean that the build process terminates very early with the message “make: *** No rule to make target '/src/build/abc/def', needed by '/build/def'. Stop”.

@bruntib
Copy link
Contributor

bruntib commented Aug 10, 2023

Could you, please, send some environmental info and a minimal reproducer? What linux distribution/version do you use? How did you install "make" itself? Through a package manager, or you compiled it for yourself? If compiled, then what flags did you use? Could you also attach a sample Makefile which presents this issue?
Thank you!

@dilyanpalauzov
Copy link
Contributor Author

I create an empty file /home/d/abc/a.c, then /home/d/cctest/makefile with this content

ifndef A
  A=/home/d/abc
endif
export A
all:
        $(MAKE) -C b

and /home/d/cctest/b/makefile with this content

a.c: $(A)/a.c
        cp $< $@

With make 4.4

$ cd /home/d/cctest && make

prints

make -C b
make[1]: Entering directory '/home/d/cctest/b'
cp /home/d/abc/a.c a.c
make[1]: Leaving directory '/home/d/cctest/b'

And now

$ rm -f b/a.c
$ cd /home/d/cctest && CodeChecker log -o a -b make

[INFO 2023-08-12 12:00] - Starting build...
[INFO 2023-08-12 12:00] - Using CodeChecker ld-logger.
make -C b
make: Entering directory '/home/d/cctest/b'
make: *** No rule to make target '/a.c', needed by 'a.c'.  Stop.
make: Leaving directory '/home/d/cctest/b'
make: *** [makefile:6: all] Error 2
[INFO 2023-08-12 12:00] - Build failed.

make 4.4 is installed by Fedora 38 (WSL2).

@dilyanpalauzov
Copy link
Contributor Author

Could you reproduce this?

bruntib added a commit to bruntib/codechecker that referenced this issue Jan 4, 2024
GNU make 4.3 uses posix_spawn() for initiating recursive make calls
in subdirectories:

make -C <subdirectory>

Since we overwrote posix_spawn() through LD_PRELOAD and we didn't
forward environment variables to the subprocess, the build of the
submodule didn't get the environment variables set by a Makefile.
This commit fixes this issue so the logging phase behaves the same
way as the original build.

Fixes Ericsson#3860
@bruntib
Copy link
Contributor

bruntib commented Jan 4, 2024

Hi,
Thanks for the reproducer, it really helped to understand the problem. I created a PR to fix this issue, hopefully this will solve it: #4146

bruntib added a commit to bruntib/codechecker that referenced this issue Apr 3, 2024
GNU make 4.3 uses posix_spawn() for initiating recursive make calls
in subdirectories:

make -C <subdirectory>

Since we overwrote posix_spawn() through LD_PRELOAD and we didn't
forward environment variables to the subprocess, the build of the
submodule didn't get the environment variables set by a Makefile.
This commit fixes this issue so the logging phase behaves the same
way as the original build.

Fixes Ericsson#3860
bruntib added a commit to bruntib/codechecker that referenced this issue Apr 3, 2024
GNU make 4.3 uses posix_spawn() for initiating recursive make calls
in subdirectories:

make -C <subdirectory>

Since we overwrote posix_spawn() through LD_PRELOAD and we didn't
forward environment variables to the subprocess, the build of the
submodule didn't get the environment variables set by a Makefile.
This commit fixes this issue so the logging phase behaves the same
way as the original build.

Fixes Ericsson#3860
bruntib added a commit to bruntib/codechecker that referenced this issue Apr 3, 2024
GNU make 4.3 uses posix_spawn() for initiating recursive make calls
in subdirectories:

make -C <subdirectory>

Since we overwrote posix_spawn() through LD_PRELOAD and we didn't
forward environment variables to the subprocess, the build of the
submodule didn't get the environment variables set by a Makefile.
This commit fixes this issue so the logging phase behaves the same
way as the original build.

Fixes Ericsson#3860
bruntib added a commit to bruntib/codechecker that referenced this issue Apr 3, 2024
GNU make 4.3 uses posix_spawn() for initiating recursive make calls
in subdirectories:

make -C <subdirectory>

Since we overwrote posix_spawn() through LD_PRELOAD and we didn't
forward environment variables to the subprocess, the build of the
submodule didn't get the environment variables set by a Makefile.
This commit fixes this issue so the logging phase behaves the same
way as the original build.

Fixes Ericsson#3860
@whisperity whisperity added this to the release 6.24.0 milestone May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants