Blame info not working

[jimis]

I upgraded to v6.17.0 and I don't see the new blame info button activated. In the logs I see:

+ CodeChecker store /home/user/codechecker_results/qtbase-dev-20211008-1bac82fa99 --trim-path-prefix /home/user/src/qt/qt5/qtbase/ -n qtbase-dev-20211008-1bac82fa99 --url http://localhost:8002/qtbase-dev
[INFO 2021-10-08 02:28] - Storing analysis results for run 'qtbase-dev-20211008-1bac82fa99'
[INFO 2021-10-08 02:28] - Checking for local valid sessions.
[INFO 2021-10-08 02:28] - Processing report files done.
[INFO 2021-10-08 02:28] - Collecting review comments ...
[INFO 2021-10-08 02:28] - Collecting review comments done.
[INFO 2021-10-08 02:28] - Get missing file content hashes from the server...
[INFO 2021-10-08 02:28] - Get missing file content hashes done.
[INFO 2021-10-08 02:28] - Collecting blame information for source files...
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/kernel/qobject_p.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/gui/platform/unix/dbusmenu/qdbusmenutypes_p.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/text/qchar.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/kernel/qobject.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/gui/rhi/qrhi_p.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/global/qsimd_p.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/itemmodels/qitemselectionmodel.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/io/qfile.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
[INFO 2021-10-08 02:29] - Collecting blame information done.
[INFO 2021-10-08 02:29] - Storing results (10.9MiB) to the server...
[INFO 2021-10-08 02:29] - Storage finished successfully.

I am assuming it's related to using --trim-path-prefix.
Also related to the fact that some paths are prepended with 'root', because of

codechecker/web/client/codechecker_client/cmd/store.py

Line 688 in 0168f93

file_path = os.path.join('root', f.lstrip('/'))

I believe that every file is not even found in __get_blame_info() and that returns silently:

codechecker/web/client/codechecker_client/blame_info.py

Lines 19 to 24 in 0168f93

	def __get_blame_info(file_path: str):
	""" Get blame info for the given file. """
	try:
	repo = Repo(file_path, search_parent_directories=True)
	except InvalidGitRepositoryError:
	return

[csordasmarton]

There is a known issue in our side with the blame information storage that we do not store blame information for source files which are already stored into the database. We will try to fix it in the next release to store blame information for every source files which were stored previously to the database (with an old version of CodeChecker which doesn't support blame information storage) and do not have blame information.

Can you please try to store your results to a new product or to a new database to make sure that this will solve the problem? If not then it will be a different problem.

[jimis]

Indeed, the same results are being stored with correct blame info into a new product.
I got confused by these zipfile.py warnings with the "root" prefix. I wonder why I get those.

[csordasmarton]

@jimis I created a patch which I think will solve this duplicated warning issue. Could you please try it: #3446.

[jimis]

Thanks @csordasmarton, I'll test but I think I'll not patch our production instance so it might take a while to test on a new instance :-)

Meanwhile here are some selected lines (only referring to file string.h) from a debug run of store command. I believe it sheds some light about those duplication warnings:

store.py:686 assemble_zip() - File contents for '/home/user/src/qt/qt5/qtbase/src/corelib/text/qstring.h' needed by the server
store.py:686 assemble_zip() - File contents for '/home/user/src/qt/qt5/qtbase/build/include/QtCore/../../../src/corelib/text/qstring.h' needed by the server
/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'root/home/user/src/qt/qt5/qtbase/src/corelib/text/qstring.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
blame_info.py:72 __get_blame_info() - Collected blame info for /home/user/src/qt/qt5/qtbase/src/corelib/text/qstring.h
blame_info.py:72 __get_blame_info() - Collected blame info for /home/user/src/qt/qt5/qtbase/build/include/QtCore/../../../src/corelib/text/qstring.h

It seems paths are not compressed (or canonicalized) i.e. removing /../ parts, as a result effectively same paths are recognized as different files.

EDIT: These long uncompressed paths exist already in the plist files in the results directory. I was thinking that the paths should be canonicalized before being written there. But are plist files generated by codechecker, or directly by clang?

[csordasmarton]

@jimis plist files for Clang Static Analyzer are generated by the analyzer but for Clang Tidy these are generated by CodeChecker (Clang Tidy is not able to product plist output but only plain text).

[jimis]

@jimis I created a patch which I think will solve this duplicated warning issue. Could you please try it: #3446.

@csordasmarton I still get the warning sometimes, even after cherry-picking your fix to 6.17.0.

[jimis]

A quick search in my plist files shows that different-but-equivalent paths are present only in clangsa plist files, and not in the clang-tidy ones. Maybe this is Clang's bug.

[bruntib]

Solved by #3488.