Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speeding up store by removing nested query #4358

Merged
merged 1 commit into from
Oct 8, 2024

Conversation

dkrupp
Copy link
Member

@dkrupp dkrupp commented Oct 7, 2024

With the baseline when storing a ~156 000 for the second time in a new run the storage times out.
(The first storage finishes in in ~10 mins.)
The fixed nested query is causing the statement timeout.

hash_24557': 'e7fc0836a60a7c8eb989eb878d94ede2', 'bug_hash_24558': '50bdcad45061c035d58ca2839ceac1f7', 'bug_hash_24559': '114a6dec38b087d904920a3ad623b21f', 'bug_hash_24560': 'f5cabc79de293a2a3fbc9dd27f34b0d0', 'bug_hash_24561': 'b5f377e878c2ccdc8ada7ad31e53ecc3', 'bug_hash_24562': '4636a65fa5ecea3e4c10e35e8a336052', 'bug_hash_24563': '7872b5d1b71a95f032bba792801a7169', 'bug_hash_24564': '344a2b1746647e88ca43ffe4fd8cb2b1', 'bug_hash_24565': 'b57e197a6f17b32ce0da8f164c80377e', 'bug_hash_24566': 'ee77f8d3ebb81445133f3b16447f3c63', 'bug_hash_24567': 'ee6d84b699d692a7baa5f9c301833d02', 'bug_hash_24568': 'cd4776edbe1fefd035fc9686db41bcb6', 'bug_hash_24569': '4094014d193d026a157259c13fd5f6e0', 'bug_hash_24570': '384f8352b5c7173745b5120be0c83a98', 'bug_hash_24571': '697e85e7d23338d6138e317fe8c1d696', 'bug_hash_24572': 'bf285e6ddfae5c33022f744b8c8a93de', 'bug_hash_24573': 'a02ebeb6e3133a5c453383f24d5a5c4b', 'bug_hash_24574': 'a6b586fb50a2b17eae3d633278e680c0', 'bug_hash_24575': 'a581a707ebc4f4f4be9d56a671b96986', 'bug_hash_24576': '26316b04373897548e0ac3bf44601dd0', 'bug_hash_24577': 'fb012c562049d1e812a8b02412d12e3c', 'bug_hash_24578': '38ef09d557788384de1f91c2a2cf15fe', 'bug_hash_24579': 'f083d5f851855a88d2286cad09e8f378', 'bug_hash_24580': 'd2a3b9c0e5605fd014ac276b9dc68e1d', 'bug_hash_24581': 'bcfe185cb6a02550cdcd3ab4dba682b8', 'bug_hash_24582': '9a8147561ce667df4c1ca83ceced2a6d', 'bug_hash_24583': '3597eaa83ebfb48016fea3f5be447ab0', 'bug_hash_24584': '181fa0affbe32f8ea05582bf88805b50', 'bug_hash_24585': '643e84e7ebcae4c12315f353bb4d1bbe', 'bug_hash_24586': 'cec6f6aa854c58495ebe9a85403ac549', 'bug_hash_24587': '83e86014a973e3f7aafb5d168f827fdb', 'bug_hash_24588': 'd38c2d4e85d24230124468ca11057b79', 'bug_hash_24589': '71b93074571ac3b56c4b00738dba2566', 'bug_hash_24590': '9e426da211adb3ab6bb4b8ae518e2655', 'bug_hash_24591': '968cb5fd52f38e724eb839126bc63670', 'bug_hash_24592': '3065d4eaa13a917fb3afcdb5e603683f', 'bug_hash_24593': 'b3a58fcedba596c5e259d0acdf3c174e', 'bug_hash_24594': 'e2f07deb799e94027fab55a71d817a71', 'bug_hash_24595': '0d4dd31c329f551b7e151b25dd2253a5', 'bug_hash_24596': 'bf08cce17f40a77e904189043702b80c', 'bug_hash_24597': 'c80bb9a267c762314b3a222a48d1210f', 'bug_hash_24598': '230fdfa1bf016d9ec6273375d15d2372', 'bug_hash_24599': '9ce1a1d7ff19cecb4a685e9f06315c65', 'bug_hash_24600': 'ff0041cdfb78a31a2f63a88857ee09cf'}]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
[DEBUG][2024-10-07 17:27:39] {system} [39670] <139954203088704> - store.py:844 _timeout_watchdog() - Timeout timer of 3600 seconds (1:00:00) for PID 39670 stopped.

real	26m50.968s
user	4m49.589s
sys	0m10.952s

After the update the storage finishes in ~6 minutes:
2nd store

time CodeChecker store /workspace/test-projects/xerces-c/reports_full/ --url http://localhost:8001/test -f -n xerces_test_full_625_client_test3 --verbose debug


----======== Summary ========----
---------------------------------------------------------
Number of processed analyzer result files        | 698   
Number of analyzer reports                       | 156340
Number of source files                           | 0     
Number of source files with source code comments | 0     
Number of blame information files                | 0     
---------------------------------------------------------
----=================----
[INFO][2024-10-07 15:41:18] {system} [4174113] <140075808073536> - store.py:642 assemble_zip() - Compressing report zip file...
[INFO][2024-10-07 15:41:24] {system} [4174113] <140075808073536> - store.py:651 assemble_zip() - Compressing report zip file done (308.4MiB / 5.5MiB).
[INFO][2024-10-07 15:41:25] {system} [4174113] <140075808073536> - store.py:950 main() - Storing results to the server...
[DEBUG][2024-10-07 15:41:25] {system} [4174113] <140075808073536> - store.py:831 _timeout_watchdog() - Set up timer for 3600 seconds (1:00:00) for PID 4174113
[DEBUG][2024-10-07 15:46:26] {system} [4174113] <140075808073536> - store.py:844 _timeout_watchdog() - Timeout timer of 3600 seconds (1:00:00) for PID 4174113 stopped.
[INFO][2024-10-07 15:46:26] {system} [4174113] <140075808073536> - store.py:1007 main() - Storage finished successfully.

real	6m49.961s
user	4m33.565s
sys	0m10.421s

@dkrupp dkrupp requested review from bruntib and vodorok as code owners October 7, 2024 12:34
@dkrupp dkrupp added this to the release 6.25.0 milestone Oct 7, 2024
@bruntib bruntib merged commit a6663b3 into Ericsson:master Oct 8, 2024
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants