-
-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IOException when building from mapping pack files into memory #584
Comments
@qmfrederik Could this be because of trying to map a single, very large pack file? Or is this due to a memory leak in the object cache you build up? |
Thanks, for reporting, @ssa3512 . Could you share the size of your pack files (i.e. Do you know whether MSBuild is running as a 32-bit or 64-bit process? |
@qmfrederik here is from the CI server that failed. Working on getting from the users that also reported issues. The CI build was set to MSBuild x86 - I am assuming this controls what it runs with. As for the users, they are using Visual Studio 2019 16.8 or newer. Is there a setting somewhere that would control 32 vs 64 bit for building?
|
Here's one from a user that was getting the error in Visual Studio. Another giant directory listing04/09/2021 07:59 AM . 04/09/2021 07:59 AM .. 03/30/2021 09:14 AM 40,412 pack-01803f62f88707186aa464f19881e52d01df35a2.idx 03/30/2021 09:14 AM 2,392,047 pack-01803f62f88707186aa464f19881e52d01df35a2.pack 03/19/2021 11:22 AM 23,024 pack-05dfd19c5ae32c399376322121f6ba1a69e04756.idx 03/19/2021 12:51 PM 1,686,001 pack-05dfd19c5ae32c399376322121f6ba1a69e04756.pack 08/11/2020 06:49 AM 35,960 pack-076e7bac8432438b685027efd843bccf98c4d9d4.idx 08/11/2020 06:49 AM 3,012,101 pack-076e7bac8432438b685027efd843bccf98c4d9d4.pack 03/17/2021 08:08 AM 29,380 pack-0abf19dc303bcd17da481e5c198eb65ad0dbb41c.idx 03/17/2021 08:28 AM 2,522,447 pack-0abf19dc303bcd17da481e5c198eb65ad0dbb41c.pack 09/21/2020 01:56 PM 13,896 pack-0c1b205912662991bcde09e2300893ab8810badc.idx 09/21/2020 01:56 PM 1,257,200 pack-0c1b205912662991bcde09e2300893ab8810badc.pack 11/20/2020 01:15 PM 58,332 pack-0d90ec3556098845284b7458c998947ee4676025.idx 03/17/2021 08:28 AM 4,614,980 pack-0d90ec3556098845284b7458c998947ee4676025.pack 02/25/2021 06:53 PM 40,384 pack-0fdefdd3a326fb65a942f72658b3a08f28fdf30c.idx 02/25/2021 06:53 PM 3,148,512 pack-0fdefdd3a326fb65a942f72658b3a08f28fdf30c.pack 03/16/2021 07:58 AM 16,220 pack-109342b8d9e9d0c48e7fc6960583a7236267720b.idx 03/17/2021 08:28 AM 804,283 pack-109342b8d9e9d0c48e7fc6960583a7236267720b.pack 07/28/2020 10:35 AM 20,420 pack-1122f515b6a2dd53a6565a3a53ed1dc1cb356c96.idx 07/28/2020 10:35 AM 1,828,842 pack-1122f515b6a2dd53a6565a3a53ed1dc1cb356c96.pack 01/06/2021 02:40 PM 49,792 pack-13ec7aa5dc8a8d4155438947b418348ba87acff6.idx 01/06/2021 02:40 PM 3,674,454 pack-13ec7aa5dc8a8d4155438947b418348ba87acff6.pack 07/24/2020 08:01 AM 25,880 pack-14cec300e3548035694c1d3fa0168139624ca18a.idx 07/24/2020 04:57 PM 1,314,327 pack-14cec300e3548035694c1d3fa0168139624ca18a.pack 10/13/2020 03:50 PM 21,904 pack-164e975e1b10219758c275a2e7e8af2b798a6ae5.idx 10/13/2020 03:50 PM 1,086,486 pack-164e975e1b10219758c275a2e7e8af2b798a6ae5.pack 09/21/2020 05:03 PM 7,736 pack-1c24214daa4519574a6bf0ea3942c7e3518d50ab.idx 09/21/2020 05:03 PM 713,043 pack-1c24214daa4519574a6bf0ea3942c7e3518d50ab.pack 12/18/2020 12:02 PM 168,316 pack-1c3108ca7afaa538e9d8f1509a031b4f2a2236dc.idx 12/18/2020 12:02 PM 13,118,570 pack-1c3108ca7afaa538e9d8f1509a031b4f2a2236dc.pack 02/19/2021 03:25 PM 54,020 pack-1e3e4b2d62d48be2331610b4557def59b03f9216.idx 02/19/2021 03:38 PM 5,013,214 pack-1e3e4b2d62d48be2331610b4557def59b03f9216.pack 09/17/2020 11:55 AM 26,944 pack-1f9d771e1168344f013a87b034d49308c00b6be0.idx 09/17/2020 11:55 AM 2,493,598 pack-1f9d771e1168344f013a87b034d49308c00b6be0.pack 08/13/2020 07:28 AM 17,900 pack-218e337f36c1807768bdab850e336363b24ae7df.idx 08/13/2020 07:28 AM 1,733,500 pack-218e337f36c1807768bdab850e336363b24ae7df.pack 10/21/2020 06:50 AM 25,992 pack-21ccce6e248e7be17f21a3f0c9e3cf6b6eb2629f.idx 10/21/2020 06:50 AM 1,842,945 pack-21ccce6e248e7be17f21a3f0c9e3cf6b6eb2629f.pack 08/18/2020 05:55 PM 23,444 pack-21e2b8cf2396ee2eefa4bb841c6d1535f6a2c7d4.idx 08/18/2020 05:55 PM 1,153,609 pack-21e2b8cf2396ee2eefa4bb841c6d1535f6a2c7d4.pack 12/01/2020 08:10 AM 90,448 pack-21f6633d78dd8717c8101d17160281fb978772f5.idx 12/01/2020 10:07 AM 5,981,766 pack-21f6633d78dd8717c8101d17160281fb978772f5.pack 03/26/2021 04:29 PM 7,176 pack-24220745fb0f29005b3160f926864689308204f1.idx 03/26/2021 04:29 PM 653,001 pack-24220745fb0f29005b3160f926864689308204f1.pack 08/03/2020 04:12 PM 54,860 pack-24737f6dacfb2612980e2dd7b01d697551c98303.idx 08/03/2020 04:12 PM 1,707,387 pack-24737f6dacfb2612980e2dd7b01d697551c98303.pack 08/25/2020 10:13 AM 19,216 pack-267ec64f7309ce5b08f1ff62630ca48520605935.idx 08/25/2020 10:13 AM 1,507,367 pack-267ec64f7309ce5b08f1ff62630ca48520605935.pack 03/26/2021 12:42 PM 6,112 pack-28bc0d26f4d8867c7978c2d83eb908dcd1883f60.idx 03/26/2021 01:18 PM 268,585 pack-28bc0d26f4d8867c7978c2d83eb908dcd1883f60.pack 12/07/2020 03:53 PM 86,892 pack-2ab8a046e1d7acdb8373ff8de43ed4af8be2d7cd.idx 12/07/2020 03:53 PM 5,194,831 pack-2ab8a046e1d7acdb8373ff8de43ed4af8be2d7cd.pack 10/14/2020 08:36 AM 12,132 pack-35aebaad2b76d2562b0e3bb07172aa8c5899ea2b.idx 10/14/2020 08:36 AM 739,242 pack-35aebaad2b76d2562b0e3bb07172aa8c5899ea2b.pack 03/08/2021 06:07 PM 14,764 pack-35d8ecf49e89fc7c73550e52032301ce2c68bed9.idx 03/09/2021 01:27 PM 1,415,446 pack-35d8ecf49e89fc7c73550e52032301ce2c68bed9.pack 08/28/2020 12:02 PM 22,408 pack-36783a86394a4531edce1a78745af85369abef33.idx 08/28/2020 12:02 PM 1,571,347 pack-36783a86394a4531edce1a78745af85369abef33.pack 03/25/2021 07:02 PM 60,936 pack-36da8d8aa8e0a16056405c45467d970db26fac4e.idx 03/26/2021 01:18 PM 4,452,828 pack-36da8d8aa8e0a16056405c45467d970db26fac4e.pack 07/20/2020 04:37 PM 15,184 pack-390b3b99b4011f328c57977b074429465add87e5.idx 07/20/2020 04:38 PM 1,450,479 pack-390b3b99b4011f328c57977b074429465add87e5.pack 01/13/2021 10:56 AM 40,468 pack-3a4e2d94e54bc992526d0896deb09222b9d7b720.idx 01/13/2021 10:56 AM 2,749,804 pack-3a4e2d94e54bc992526d0896deb09222b9d7b720.pack 01/07/2021 01:31 PM 22,744 pack-3c832e155a33357d3a2ad8671e1c6964c3756596.idx 01/07/2021 01:31 PM 1,918,367 pack-3c832e155a33357d3a2ad8671e1c6964c3756596.pack 03/18/2021 03:33 PM 36,184 pack-418807ef74e23f53ae29f0c956f1ea3a122b7b1b.idx 03/23/2021 12:11 PM 3,052,910 pack-418807ef74e23f53ae29f0c956f1ea3a122b7b1b.pack 07/31/2020 05:53 PM 14,092 pack-437886dd2ef76e91bcec305a9c231f0e50587f92.idx 07/31/2020 06:00 PM 1,034,333 pack-437886dd2ef76e91bcec305a9c231f0e50587f92.pack 01/26/2021 01:09 PM 62,672 pack-45daa4a5a4cbb1369a174882651343ffab02f00a.idx 01/26/2021 01:09 PM 4,918,244 pack-45daa4a5a4cbb1369a174882651343ffab02f00a.pack 09/29/2020 07:31 AM 37,724 pack-46ce1df128cf0eeccb07f3348ddd38ab3e66981b.idx 10/01/2020 12:09 PM 3,347,366 pack-46ce1df128cf0eeccb07f3348ddd38ab3e66981b.pack 07/24/2020 04:57 PM 7,708 pack-47dcfc1e725a2c46a588f49f320ea84d02bf8443.idx 07/25/2020 08:02 AM 875,562 pack-47dcfc1e725a2c46a588f49f320ea84d02bf8443.pack 10/15/2020 04:09 PM 46,572 pack-48caba21f21774b73054046c91998f543a622083.idx 10/15/2020 04:09 PM 2,976,098 pack-48caba21f21774b73054046c91998f543a622083.pack 08/27/2020 04:57 PM 26,076 pack-4a767ebf0b4a0066a168843cd4ab14f3293ae234.idx 08/27/2020 04:57 PM 1,851,859 pack-4a767ebf0b4a0066a168843cd4ab14f3293ae234.pack 09/25/2020 07:32 AM 21,204 pack-4aa2168b1c12c168a5c44585406e4efb94cb1c74.idx 03/17/2021 08:28 AM 1,960,662 pack-4aa2168b1c12c168a5c44585406e4efb94cb1c74.pack 01/27/2021 02:51 PM 27,028 pack-4b60fd8a92adb273242d16398a526990568c6987.idx 01/27/2021 02:51 PM 2,242,478 pack-4b60fd8a92adb273242d16398a526990568c6987.pack 02/26/2021 12:19 PM 17,452 pack-4c759da68ca85c069b67e1c7610ff0c382bc72cc.idx 02/26/2021 12:19 PM 1,515,288 pack-4c759da68ca85c069b67e1c7610ff0c382bc72cc.pack 03/12/2021 12:21 PM 10,452 pack-4d07282a0a2cff2b5e3019299f1e4cb364f42203.idx 03/12/2021 12:21 PM 885,366 pack-4d07282a0a2cff2b5e3019299f1e4cb364f42203.pack 03/24/2021 07:50 AM 56,064 pack-4e568b945cc191bd5690bb0f7211e30a48cc87da.idx 03/24/2021 07:50 AM 4,036,885 pack-4e568b945cc191bd5690bb0f7211e30a48cc87da.pack 10/08/2020 02:24 PM 44,332 pack-4f079f217bbdbf45341805afb6c219c15ba04fc3.idx 10/08/2020 02:24 PM 2,945,031 pack-4f079f217bbdbf45341805afb6c219c15ba04fc3.pack 07/21/2020 01:58 PM 12,440 pack-517e3ece0b8f92a68dea7c8c5a5db3e523563c3f.idx 07/23/2020 11:16 AM 984,830 pack-517e3ece0b8f92a68dea7c8c5a5db3e523563c3f.pack 09/24/2020 11:30 AM 16,444 pack-531210c9d65b90dcdf6c48441d2723210ea736fe.idx 09/24/2020 11:30 AM 1,417,287 pack-531210c9d65b90dcdf6c48441d2723210ea736fe.pack 11/18/2020 08:54 AM 64,632 pack-551690ea206190e54f0dea63c3a5d7b2d9647b6b.idx 11/23/2020 11:56 AM 4,234,456 pack-551690ea206190e54f0dea63c3a5d7b2d9647b6b.pack 02/24/2021 12:10 PM 33,048 pack-562b62fa31283c077c960a7e874ba17d3b7d8d93.idx 03/04/2021 08:46 PM 2,498,164 pack-562b62fa31283c077c960a7e874ba17d3b7d8d93.pack 03/10/2021 12:46 PM 47,664 pack-580e0c1bcad7919135d92f28150a2620db7bc049.idx 03/11/2021 12:44 PM 3,107,270 pack-580e0c1bcad7919135d92f28150a2620db7bc049.pack 11/03/2020 02:45 PM 150,480 pack-586f0c4434105004e263ce20c0ef9a675f92248c.idx 11/03/2020 03:05 PM 10,636,384 pack-586f0c4434105004e263ce20c0ef9a675f92248c.pack 08/07/2020 11:21 AM 14,316 pack-5b21f0ead4c6581723eba3e73bd672b31b4e5c4f.idx 08/07/2020 11:21 AM 1,040,564 pack-5b21f0ead4c6581723eba3e73bd672b31b4e5c4f.pack 12/08/2020 05:55 PM 28,120 pack-5bffe1875f15dda3b0697e2af35cb18431df9ebe.idx 12/08/2020 05:55 PM 2,325,476 pack-5bffe1875f15dda3b0697e2af35cb18431df9ebe.pack 01/15/2021 02:22 PM 94,956 pack-5c6279cc90de80f404381a2ce75a7996f7a317dc.idx 01/15/2021 02:22 PM 4,867,561 pack-5c6279cc90de80f404381a2ce75a7996f7a317dc.pack 09/22/2020 06:59 AM 5,636 pack-5d2ec3c137099f689d6efe9fb918b28c725c3a15.idx 09/22/2020 06:59 AM 655,868 pack-5d2ec3c137099f689d6efe9fb918b28c725c3a15.pack 09/10/2020 12:13 PM 37,192 pack-61177358fceeaa89d911edf5e2f89b940ad1ffd7.idx 09/10/2020 12:13 PM 2,720,069 pack-61177358fceeaa89d911edf5e2f89b940ad1ffd7.pack 07/22/2020 03:20 PM 19,664 pack-641bffd8a3217ad9d4a505ff0fdb41c9ae633c5a.idx 07/24/2020 04:57 PM 1,950,678 pack-641bffd8a3217ad9d4a505ff0fdb41c9ae633c5a.pack 03/04/2021 07:20 PM 26,524 pack-64e20846183cb0e83c10962bd0f57130a575f225.idx 03/04/2021 08:46 PM 1,366,912 pack-64e20846183cb0e83c10962bd0f57130a575f225.pack 11/06/2020 12:37 PM 21,064 pack-659818b2b9c3f2e8c64c8eadb528f209972a4cce.idx 11/06/2020 12:37 PM 1,868,674 pack-659818b2b9c3f2e8c64c8eadb528f209972a4cce.pack 08/12/2020 11:11 AM 31,116 pack-6601e52cd2b879367aa40e2fef34b84e2e897ef0.idx 08/12/2020 11:11 AM 2,314,262 pack-6601e52cd2b879367aa40e2fef34b84e2e897ef0.pack 10/13/2020 11:06 AM 27,756 pack-692029b8f164eea0c3610d46da511a6bb17cfccf.idx 10/13/2020 11:06 AM 1,873,168 pack-692029b8f164eea0c3610d46da511a6bb17cfccf.pack 09/03/2020 02:01 PM 11,572 pack-6db1165d5872604a09bb65868ec22175d516b33e.idx 09/03/2020 02:01 PM 742,263 pack-6db1165d5872604a09bb65868ec22175d516b33e.pack 01/18/2021 12:12 PM 45,172 pack-6e3a0762b861db289a8a858c6490217fc3f50965.idx 01/18/2021 12:12 PM 4,055,257 pack-6e3a0762b861db289a8a858c6490217fc3f50965.pack 01/22/2021 09:12 AM 53,852 pack-6ee9be9eb1658896a45aa1dfcedc0ce04e6359eb.idx 01/22/2021 09:12 AM 5,039,255 pack-6ee9be9eb1658896a45aa1dfcedc0ce04e6359eb.pack 07/23/2020 11:07 AM 12,272 pack-6ff510f6b0d88cfb96e5cd63161f10d5e0589a39.idx 07/23/2020 11:16 AM 420,320 pack-6ff510f6b0d88cfb96e5cd63161f10d5e0589a39.pack 07/21/2020 09:22 AM 11,908 pack-74700b7b4d0ec598d32a8aa1f020ce6907f7b5a8.idx 07/23/2020 11:16 AM 931,899 pack-74700b7b4d0ec598d32a8aa1f020ce6907f7b5a8.pack 03/26/2021 09:13 AM 7,624 pack-74b77c568b3a464d7287c9bdb30fb6e0dd25029e.idx 03/26/2021 09:13 AM 738,358 pack-74b77c568b3a464d7287c9bdb30fb6e0dd25029e.pack 10/07/2020 07:27 AM 29,856 pack-77542c7411b49580eb5df85fe384f2939e87103c.idx 10/07/2020 07:27 AM 2,478,628 pack-77542c7411b49580eb5df85fe384f2939e87103c.pack 02/03/2021 06:20 PM 39,768 pack-7787890a91d4c39781a1aeb1f573cef1e0532d24.idx 02/03/2021 06:20 PM 3,021,186 pack-7787890a91d4c39781a1aeb1f573cef1e0532d24.pack 07/28/2020 01:29 PM 6,224 pack-77d2593ae87de649a9ad3ceb176e88cf24e8e666.idx 07/28/2020 01:29 PM 585,497 pack-77d2593ae87de649a9ad3ceb176e88cf24e8e666.pack 02/17/2021 08:13 AM 88,852 pack-7a78e417c1346d9b49903bf9a65d84d66a659fac.idx 03/23/2021 12:11 PM 133,721,058 pack-7a78e417c1346d9b49903bf9a65d84d66a659fac.pack 07/29/2020 07:17 AM 19,104 pack-7b6491d55f3170016dcafb64efc5f3bbc8d4d3ea.idx 07/30/2020 06:43 PM 1,721,476 pack-7b6491d55f3170016dcafb64efc5f3bbc8d4d3ea.pack 04/08/2021 10:25 AM 30,192 pack-7d3abe2ca8d4354bd45269c6da459360ba752724.idx 04/08/2021 10:25 AM 2,721,669 pack-7d3abe2ca8d4354bd45269c6da459360ba752724.pack 04/05/2021 06:21 PM 15,968 pack-7dc59c2888a89b051965c34542087ae39a7851b9.idx 04/05/2021 06:21 PM 1,507,448 pack-7dc59c2888a89b051965c34542087ae39a7851b9.pack 02/02/2021 08:29 AM 150,648 pack-7e6e62288276d60aa8c219fc536c9dbf720af914.idx 02/02/2021 08:29 AM 8,412,479 pack-7e6e62288276d60aa8c219fc536c9dbf720af914.pack 09/21/2020 07:41 AM 49,316 pack-825f5352dd97d6e0809bf1902d4b7b7bba74eb33.idx 09/21/2020 07:41 AM 3,724,871 pack-825f5352dd97d6e0809bf1902d4b7b7bba74eb33.pack 07/22/2020 05:04 PM 10,956 pack-85c1d972a5e3dbc0661cd10ef31b96d5eb23c4bb.idx 07/24/2020 04:57 PM 625,519 pack-85c1d972a5e3dbc0661cd10ef31b96d5eb23c4bb.pack 10/09/2020 06:31 PM 28,260 pack-888ea11200f695e0bbd9ef6ad6a4bf527e83a686.idx 10/09/2020 06:31 PM 3,019,069 pack-888ea11200f695e0bbd9ef6ad6a4bf527e83a686.pack 03/04/2021 11:56 AM 12,944 pack-8a3bc4ce8c28943e414a4266ec3f0e8e8dd8d46d.idx 03/04/2021 08:46 PM 1,292,359 pack-8a3bc4ce8c28943e414a4266ec3f0e8e8dd8d46d.pack 07/30/2020 02:58 PM 25,320 pack-8b3866d7edf0412a13305acd26d5599f5259318c.idx 07/31/2020 06:00 PM 1,781,236 pack-8b3866d7edf0412a13305acd26d5599f5259318c.pack 04/07/2021 09:46 AM 38,396 pack-900eb27c5f6c94f766285a567f4c182f62ed3b1f.idx 04/07/2021 09:46 AM 3,926,534 pack-900eb27c5f6c94f766285a567f4c182f62ed3b1f.pack 02/23/2021 09:05 AM 41,980 pack-912ed8a45a2b3cc2b17b0423313dde04e95ff4ac.idx 02/23/2021 09:05 AM 2,869,274 pack-912ed8a45a2b3cc2b17b0423313dde04e95ff4ac.pack 03/15/2021 02:44 PM 45,564 pack-91bb1ce2d00f464cbd3469655f4075e0810f2f18.idx 03/17/2021 08:28 AM 3,723,818 pack-91bb1ce2d00f464cbd3469655f4075e0810f2f18.pack 08/04/2020 02:46 PM 14,120 pack-933a0ff3e3e0a0eb02108ba0d8da7df949c82dae.idx 08/06/2020 09:43 AM 910,392 pack-933a0ff3e3e0a0eb02108ba0d8da7df949c82dae.pack 08/24/2020 08:27 AM 52,256 pack-9670f0b2e160e4a50c195363e7340e7c47db8bd2.idx 08/24/2020 08:27 AM 2,588,198 pack-9670f0b2e160e4a50c195363e7340e7c47db8bd2.pack 09/16/2020 10:42 AM 32,012 pack-97240f9262d0980ff06f44598d17a5913d2cfc06.idx 09/16/2020 10:42 AM 2,501,190 pack-97240f9262d0980ff06f44598d17a5913d2cfc06.pack 11/12/2020 10:26 AM 42,596 pack-985af2f2a323508abc732971a0dcd2bbad33515d.idx 11/12/2020 10:26 AM 3,305,479 pack-985af2f2a323508abc732971a0dcd2bbad33515d.pack 04/09/2021 07:59 AM 19,272 pack-9b01f79352c78cc1ad817da71c47986653f0ad78.idx 04/09/2021 07:59 AM 1,834,097 pack-9b01f79352c78cc1ad817da71c47986653f0ad78.pack 09/23/2020 02:20 PM 47,832 pack-9b9915c71fbc5048ca00ec7567bb076c22db475d.idx 09/23/2020 02:20 PM 3,634,444 pack-9b9915c71fbc5048ca00ec7567bb076c22db475d.pack 09/08/2020 08:31 AM 48,112 pack-9d22bbd40535618666243393e8ecbc87caf53451.idx 09/08/2020 08:31 AM 3,376,049 pack-9d22bbd40535618666243393e8ecbc87caf53451.pack 11/10/2020 03:10 PM 30,416 pack-9dc893a2377bbd973e41cb25dfb17be3a4ad3ba1.idx 11/10/2020 04:55 PM 2,270,019 pack-9dc893a2377bbd973e41cb25dfb17be3a4ad3ba1.pack 03/20/2021 09:09 AM 17,648 pack-9f5c1a840b8850e338de646888bc95eca8274cea.idx 03/22/2021 03:47 PM 2,249,700 pack-9f5c1a840b8850e338de646888bc95eca8274cea.pack 12/02/2020 08:17 AM 32,964 pack-a0770e99c3def71ac624a11ccddf177418b98808.idx 12/02/2020 08:17 AM 2,263,076 pack-a0770e99c3def71ac624a11ccddf177418b98808.pack 08/20/2020 07:42 AM 23,080 pack-a158cfc23069a5dff70b61028d1f687c146e9b50.idx 08/20/2020 07:42 AM 1,425,144 pack-a158cfc23069a5dff70b61028d1f687c146e9b50.pack 10/06/2020 07:16 AM 49,568 pack-a20e5472a2b233cfef31d95af853ed887e6ef26a.idx 10/06/2020 07:16 AM 5,221,047 pack-a20e5472a2b233cfef31d95af853ed887e6ef26a.pack 12/29/2020 09:37 AM 82,776 pack-a571e412546fe7d9bae11702f19fa68e85da7d18.idx 12/29/2020 09:37 AM 6,614,377 pack-a571e412546fe7d9bae11702f19fa68e85da7d18.pack 12/03/2020 01:42 PM 32,460 pack-ab85e3618aa2e107fa9c0b26ea4c1952032efa81.idx 12/03/2020 01:42 PM 1,890,888 pack-ab85e3618aa2e107fa9c0b26ea4c1952032efa81.pack 08/06/2020 09:37 AM 17,032 pack-b07a34ded940420dad43dc214f78297d8c79e103.idx 08/06/2020 09:52 AM 1,093,783 pack-b07a34ded940420dad43dc214f78297d8c79e103.pack 02/04/2021 12:52 PM 27,112 pack-b2ab9c8c33a206737dd0d2d327709551e628a043.idx 02/04/2021 12:52 PM 2,036,715 pack-b2ab9c8c33a206737dd0d2d327709551e628a043.pack 03/02/2021 07:49 AM 26,412 pack-b34328050a7094f42128ffdc79812e0d1a4782fb.idx 03/02/2021 07:49 AM 1,679,994 pack-b34328050a7094f42128ffdc79812e0d1a4782fb.pack 09/29/2020 10:10 AM 8,688 pack-b5d43c2d2b662936c84d34072e6248049be47e60.idx 09/29/2020 10:10 AM 884,969 pack-b5d43c2d2b662936c84d34072e6248049be47e60.pack 09/23/2020 03:30 PM 6,280 pack-b8bde221b1bec40696656009f50bf0239adc9ef7.idx 09/23/2020 03:30 PM 566,013 pack-b8bde221b1bec40696656009f50bf0239adc9ef7.pack 02/23/2021 11:24 AM 25,152 pack-b8f344ce951f82d568eaa59583527187fd10f999.idx 02/23/2021 11:24 AM 1,114,020 pack-b8f344ce951f82d568eaa59583527187fd10f999.pack 09/14/2020 11:40 AM 31,536 pack-b8f64a18dc81a02f797d20f3d5820e15b75bd4ad.idx 09/14/2020 11:40 AM 3,000,974 pack-b8f64a18dc81a02f797d20f3d5820e15b75bd4ad.pack 08/04/2020 08:42 AM 9,248 pack-ba3a186876fe4979a8f124c67e68afc97e07935d.idx 08/04/2020 08:42 AM 1,027,919 pack-ba3a186876fe4979a8f124c67e68afc97e07935d.pack 03/11/2021 12:12 PM 46,544 pack-bab71f246ac53517686fca7aa5001a695336de6d.idx 03/11/2021 12:44 PM 2,535,125 pack-bab71f246ac53517686fca7aa5001a695336de6d.pack 03/02/2021 10:43 AM 5,860 pack-bb54e4b947f771a64e5caebab8265af69d126bd0.idx 03/02/2021 10:43 AM 438,355 pack-bb54e4b947f771a64e5caebab8265af69d126bd0.pack 03/31/2021 10:02 AM 33,356 pack-bf8a81dd4bfd80e06ebb31c2376c8e7883e17e84.idx 03/31/2021 10:02 AM 2,302,249 pack-bf8a81dd4bfd80e06ebb31c2376c8e7883e17e84.pack 09/01/2020 06:53 AM 27,588 pack-bfb19de17acf70480f157d7238e3cc7d182f150a.idx 09/01/2020 06:53 AM 2,065,733 pack-bfb19de17acf70480f157d7238e3cc7d182f150a.pack 01/12/2021 09:10 AM 94,172 pack-c3c0d7bb4630ff422f9a36864eb53a63cc53cbd7.idx 01/12/2021 09:10 AM 4,912,214 pack-c3c0d7bb4630ff422f9a36864eb53a63cc53cbd7.pack 01/05/2021 08:47 AM 45,032 pack-c5a21a8b6d524177ee78b688650ae61e7ec4166e.idx 01/05/2021 08:47 AM 3,643,010 pack-c5a21a8b6d524177ee78b688650ae61e7ec4166e.pack 10/01/2020 11:53 AM 42,680 pack-c9406de06d0a0cc435c5e58bc8cb8cfd2549fd73.idx 10/01/2020 12:09 PM 2,661,417 pack-c9406de06d0a0cc435c5e58bc8cb8cfd2549fd73.pack 03/17/2021 01:49 PM 11,376 pack-c9cee6cf48d0d9c2a7ca1b9949ea0cec0284a266.idx 03/17/2021 01:49 PM 1,166,344 pack-c9cee6cf48d0d9c2a7ca1b9949ea0cec0284a266.pack 10/19/2020 07:39 AM 24,536 pack-cb61bfbb0abcf5e0dbdad5c76a9db4d305ef547a.idx 10/19/2020 07:39 AM 2,157,755 pack-cb61bfbb0abcf5e0dbdad5c76a9db4d305ef547a.pack 07/20/2020 10:20 AM 16,062,124 pack-ce3d47e72afdf1590f202e297e12ca5d8b68ec44.idx 03/17/2021 08:28 AM 248,384,478 pack-ce3d47e72afdf1590f202e297e12ca5d8b68ec44.pack 03/08/2021 04:00 PM 43,436 pack-ce60697498609309686c89907c7e998155b70a15.idx 03/09/2021 01:27 PM 3,849,985 pack-ce60697498609309686c89907c7e998155b70a15.pack 11/23/2020 10:39 AM 58,136 pack-d01b2869dd9a3e024d38d690999fa370b4e17693.idx 11/23/2020 11:56 AM 4,149,900 pack-d01b2869dd9a3e024d38d690999fa370b4e17693.pack 12/22/2020 09:59 AM 94,788 pack-d0bec3c16cca20b65265efe3b09143c1051feb24.idx 12/22/2020 09:59 AM 7,137,702 pack-d0bec3c16cca20b65265efe3b09143c1051feb24.pack 02/27/2021 09:45 AM 25,936 pack-d3ce40bedf9c60e7b9433f7ae06c64fdbeeadba7.idx 02/27/2021 12:57 PM 2,133,322 pack-d3ce40bedf9c60e7b9433f7ae06c64fdbeeadba7.pack 03/16/2021 12:18 PM 8,856 pack-d43f6a2eb7b53bde41c02c200c68b7e616e55f94.idx 03/16/2021 12:18 PM 730,887 pack-d43f6a2eb7b53bde41c02c200c68b7e616e55f94.pack 02/05/2021 12:17 PM 39,628 pack-d45be1f6024f503dc0c34f5a53d6d45507d5edb0.idx 02/05/2021 12:17 PM 3,518,972 pack-d45be1f6024f503dc0c34f5a53d6d45507d5edb0.pack 12/22/2020 03:24 PM 12,944 pack-d52783d990ed4fa0626cf62a0782023d181845f3.idx 12/22/2020 03:25 PM 659,511 pack-d52783d990ed4fa0626cf62a0782023d181845f3.pack 03/11/2021 07:01 PM 15,268 pack-d8d692c6daa2025d9b8078742b8067e1b355a09c.idx 03/11/2021 07:01 PM 1,511,879 pack-d8d692c6daa2025d9b8078742b8067e1b355a09c.pack 08/18/2020 08:43 AM 45,116 pack-da81ca8c46fc64b054fe571435df15502697064c.idx 08/18/2020 08:43 AM 3,712,332 pack-da81ca8c46fc64b054fe571435df15502697064c.pack 09/03/2020 08:45 AM 55,252 pack-dd23585a5d27bfec93827bfaeb2dcc48a739c99f.idx 09/03/2020 08:45 AM 3,952,032 pack-dd23585a5d27bfec93827bfaeb2dcc48a739c99f.pack 08/05/2020 12:48 PM 19,524 pack-decd70896c329b3a265cba079203cc337dca7c28.idx 08/06/2020 09:43 AM 1,766,307 pack-decd70896c329b3a265cba079203cc337dca7c28.pack 02/08/2021 12:39 PM 43,660 pack-dfdc8db6552813b23daf274338babd3b37693f2f.idx 02/08/2021 12:39 PM 3,039,969 pack-dfdc8db6552813b23daf274338babd3b37693f2f.pack 10/01/2020 01:40 PM 10,872 pack-e5e4725c7b8f3a29bcd81f0e18ff68c64ff33c62.idx 10/01/2020 01:40 PM 966,259 pack-e5e4725c7b8f3a29bcd81f0e18ff68c64ff33c62.pack 09/30/2020 07:37 AM 26,524 pack-e60e339a34ffa21487c8c593fbd500011a98b285.idx 10/01/2020 12:09 PM 1,771,974 pack-e60e339a34ffa21487c8c593fbd500011a98b285.pack 09/09/2020 08:56 AM 49,652 pack-e90c16f9ecd313235b5b4268a01cfd1806a48216.idx 09/09/2020 08:56 AM 3,213,624 pack-e90c16f9ecd313235b5b4268a01cfd1806a48216.pack 11/05/2020 10:35 AM 25,292 pack-e96e488f0be40d600cdb73ddda7ad3c38aad477c.idx 11/05/2020 10:35 AM 2,168,434 pack-e96e488f0be40d600cdb73ddda7ad3c38aad477c.pack 11/18/2020 01:27 PM 12,104 pack-eaddf47738cd4ead0e532c85a69ec99cb938c2a6.idx 11/18/2020 01:27 PM 1,360,602 pack-eaddf47738cd4ead0e532c85a69ec99cb938c2a6.pack 07/24/2020 04:49 PM 26,244 pack-eddf2fee745d93349babc0ab04545192a141c873.idx 07/24/2020 04:57 PM 1,977,277 pack-eddf2fee745d93349babc0ab04545192a141c873.pack 03/03/2021 11:58 AM 30,836 pack-ee3e3f0aee2cb605597e1a85b75b533a473d1316.idx 03/04/2021 08:46 PM 2,727,769 pack-ee3e3f0aee2cb605597e1a85b75b533a473d1316.pack 04/05/2021 08:07 AM 43,436 pack-eeaf040c76681eddac82bfafef27850dd3c913a2.idx 04/05/2021 08:07 AM 4,103,828 pack-eeaf040c76681eddac82bfafef27850dd3c913a2.pack 02/10/2021 09:43 AM 43,184 pack-f1e8072bfde37a422a70b0e58fd5990ff1900e54.idx 02/10/2021 09:43 AM 3,182,217 pack-f1e8072bfde37a422a70b0e58fd5990ff1900e54.pack 03/09/2021 12:50 PM 23,080 pack-f426c6ec21410346ae23d15a26a23a64db2cbdbd.idx 03/09/2021 01:27 PM 1,978,187 pack-f426c6ec21410346ae23d15a26a23a64db2cbdbd.pack 08/26/2020 02:56 PM 27,532 pack-f4b7d648ca91eddf2f601e631d3b831a941ecee5.idx 08/26/2020 02:56 PM 2,522,700 pack-f4b7d648ca91eddf2f601e631d3b831a941ecee5.pack 01/19/2021 09:11 AM 37,052 pack-f4f7bc0f18322f4c0196b9b806b84202c9fe7243.idx 01/19/2021 09:11 AM 3,581,798 pack-f4f7bc0f18322f4c0196b9b806b84202c9fe7243.pack 09/15/2020 10:36 AM 17,172 pack-fb748fe5a0238cff7a6c30b1dafc4580d4a19e10.idx 09/15/2020 10:36 AM 1,035,589 pack-fb748fe5a0238cff7a6c30b1dafc4580d4a19e10.pack 08/14/2020 08:56 AM 12,888 pack-fc9f855b62f2e2c5f77fffac02aecf1b4b32af03.idx 08/14/2020 10:45 AM 1,043,188 pack-fc9f855b62f2e2c5f77fffac02aecf1b4b32af03.pack 282 File(s) 756,021,106 bytes 2 Dir(s) 89,487,589,376 bytes free |
None of these files look very large, but I suppose perhaps the 129MB+ files might be hard to map into contiguous memory that large. |
Here's one more data point:
Mapping the index files is fine but mapping the whole pack files is definitely problematic. |
Here's an experiment with mapping only small chunks of the pack file: https://github.com/dotnet/Nerdbank.GitVersioning/compare/master...filipnavara:big-pack?expand=1 The approach is not perfect but it highlights few things:
In the end I have a feeling that it may be more reasonable to map only the index files and not the pack files. If it's a problem performance wise it's probably fine to do that only as a fallback when certain parameters are exceeded (file size threshold, 32-bit process, the mapping fails, etc.). .NET 6 will have better APIs to handle this kind of random reads (dotnet/runtime#53669) but unfortunately that's still some months away. |
@qmfrederik Do you have time to investigate @filipnavara's data and proposal? |
I'd be happy to do some more testing and try out other approaches or even run benchmarks. However, with the window for .NET 6 runtime changes slowly closing in I need to focus on completing some work there before moving on to other projects. |
All help is welcome, so I'd be more than happy if @filipnavara wants to invest some time in this 😄 .
I think the main reason was
Yes, that may be the case. If I remember correctly, the stream pooling and usage of memory mapped files was introduced at about the same time. It is possible that the memory mapping doesn't have much of a perf impact on pack files, and the real gain comes from stream pooling. There are a couple of perf tests which run as part of CI, so it should be fairly easy to tell whether changes regressed performance or not 😃 |
Keep in mind the workaround we have of setting the |
I'm now starting to run into this in local dev too in VS. I guess VS 2022 can't come fast enough :P Searching for a way to configure the use of 64-bit msbuild from VS but that doesn't seem to be possible. Guessing that's a limitation of a 32-bit process not being able to spawn a 64-bit one, but this certainly isn't my area of expertise. So rather than need to run msbuild from cmdline, is there a way to perhaps toggle off ngvb in local dev? Which would probably be something we'd want in any case to avoid the compile-time overhead... Is toggling it out of the csproj if not Release configuration a viable alternative? Can't say I've ever toggled off a package ref by <PackageReference Include="Nerdbank.GitVersioning" Condition=" '$(Configuration)' == 'Release' ">
<Version>3.4.190</Version>
<PrivateAssets>all</PrivateAssets>
</PackageReference> |
@benmccallum If you have the time to dig into this, could you give #626 a try and let us know if this fixes the problem for you? There may be a perf regression n macOS, but since you mention VS, I guess you're on Windows so you should be good. Alternatively, set the |
There's never a problem with 32-bit processes spawning 64-bit ones. It's just that that's not what msbuild in VS was designed to do. |
@qmfrederik, I can look to give #626 a go at some point, is it available as a nuget preview release somewhere? |
@benmccallum It used to be, but it got GC'd by Azure Pipelines. I've queued a new build here: https://dev.azure.com/andrewarnott/OSS/_build/results?buildId=4966&view=results |
This is now available in https://github.com/dotnet/Nerdbank.GitVersioning/releases/tag/v3.4.228 |
Thanks @AArnott , sorry I didn't get around to give feedback in time. Will get on the new version tomorrow and give it a whirl. |
Seems to be working well for us on all platforms, 👍🏻 Thanks again |
Several developers reporting and at least one CI build failed. Rolled back to 3.3.37 for now.
The text was updated successfully, but these errors were encountered: