Parallel benchmarking sometimes hangs in container mode

[PhilippWendler]

When executing many benchmarks in parallel, benchexec sometimes hangs if container mode is enabled. For reproducing it, for example use the following benchmark definition and benchexec benchmark-example-calculatepi-many.xml --container -N 100:

benchmark-example-calculatepi-many.xml

<?xml version="1.0"?>
<!DOCTYPE benchmark PUBLIC "+//IDN sosy-lab.org//DTD BenchExec benchmark 1.18//EN" "https://www.sosy-lab.org/benchexec/benchmark-1.18.dtd">
<benchmark displayName="Calculate Pi" tool="calculatepi" timelimit="900s" memlimit="100 MB">
  <tasks>
<withoutfile>1</withoutfile>
<withoutfile>2</withoutfile>
<withoutfile>3</withoutfile>
<withoutfile>4</withoutfile>
<withoutfile>5</withoutfile>
<withoutfile>6</withoutfile>
<withoutfile>7</withoutfile>
<withoutfile>8</withoutfile>
<withoutfile>9</withoutfile>
<withoutfile>10</withoutfile>
<withoutfile>11</withoutfile>
<withoutfile>12</withoutfile>
<withoutfile>13</withoutfile>
<withoutfile>14</withoutfile>
<withoutfile>15</withoutfile>
<withoutfile>16</withoutfile>
<withoutfile>17</withoutfile>
<withoutfile>18</withoutfile>
<withoutfile>19</withoutfile>
<withoutfile>20</withoutfile>
<withoutfile>21</withoutfile>
<withoutfile>22</withoutfile>
<withoutfile>23</withoutfile>
<withoutfile>24</withoutfile>
<withoutfile>25</withoutfile>
<withoutfile>26</withoutfile>
<withoutfile>27</withoutfile>
<withoutfile>28</withoutfile>
<withoutfile>29</withoutfile>
<withoutfile>30</withoutfile>
<withoutfile>31</withoutfile>
<withoutfile>32</withoutfile>
<withoutfile>33</withoutfile>
<withoutfile>34</withoutfile>
<withoutfile>35</withoutfile>
<withoutfile>36</withoutfile>
<withoutfile>37</withoutfile>
<withoutfile>38</withoutfile>
<withoutfile>39</withoutfile>
<withoutfile>40</withoutfile>
<withoutfile>41</withoutfile>
<withoutfile>42</withoutfile>
<withoutfile>43</withoutfile>
<withoutfile>44</withoutfile>
<withoutfile>45</withoutfile>
<withoutfile>46</withoutfile>
<withoutfile>47</withoutfile>
<withoutfile>48</withoutfile>
<withoutfile>49</withoutfile>
<withoutfile>50</withoutfile>
<withoutfile>51</withoutfile>
<withoutfile>52</withoutfile>
<withoutfile>53</withoutfile>
<withoutfile>54</withoutfile>
<withoutfile>55</withoutfile>
<withoutfile>56</withoutfile>
<withoutfile>57</withoutfile>
<withoutfile>58</withoutfile>
<withoutfile>59</withoutfile>
<withoutfile>60</withoutfile>
<withoutfile>61</withoutfile>
<withoutfile>62</withoutfile>
<withoutfile>63</withoutfile>
<withoutfile>64</withoutfile>
<withoutfile>65</withoutfile>
<withoutfile>66</withoutfile>
<withoutfile>67</withoutfile>
<withoutfile>68</withoutfile>
<withoutfile>69</withoutfile>
<withoutfile>70</withoutfile>
<withoutfile>71</withoutfile>
<withoutfile>72</withoutfile>
<withoutfile>73</withoutfile>
<withoutfile>74</withoutfile>
<withoutfile>75</withoutfile>
<withoutfile>76</withoutfile>
<withoutfile>77</withoutfile>
<withoutfile>78</withoutfile>
<withoutfile>79</withoutfile>
<withoutfile>80</withoutfile>
<withoutfile>81</withoutfile>
<withoutfile>82</withoutfile>
<withoutfile>83</withoutfile>
<withoutfile>84</withoutfile>
<withoutfile>85</withoutfile>
<withoutfile>86</withoutfile>
<withoutfile>87</withoutfile>
<withoutfile>88</withoutfile>
<withoutfile>89</withoutfile>
<withoutfile>90</withoutfile>
<withoutfile>91</withoutfile>
<withoutfile>92</withoutfile>
<withoutfile>93</withoutfile>
<withoutfile>94</withoutfile>
<withoutfile>95</withoutfile>
<withoutfile>96</withoutfile>
<withoutfile>97</withoutfile>
<withoutfile>98</withoutfile>
<withoutfile>99</withoutfile>
<withoutfile>100</withoutfile>
<withoutfile>101</withoutfile>
<withoutfile>102</withoutfile>
<withoutfile>103</withoutfile>
<withoutfile>104</withoutfile>
<withoutfile>105</withoutfile>
<withoutfile>106</withoutfile>
<withoutfile>107</withoutfile>
<withoutfile>108</withoutfile>
<withoutfile>109</withoutfile>
<withoutfile>110</withoutfile>
<withoutfile>111</withoutfile>
<withoutfile>112</withoutfile>
<withoutfile>113</withoutfile>
<withoutfile>114</withoutfile>
<withoutfile>115</withoutfile>
<withoutfile>116</withoutfile>
<withoutfile>117</withoutfile>
<withoutfile>118</withoutfile>
<withoutfile>119</withoutfile>
<withoutfile>120</withoutfile>
<withoutfile>121</withoutfile>
<withoutfile>122</withoutfile>
<withoutfile>123</withoutfile>
<withoutfile>124</withoutfile>
<withoutfile>125</withoutfile>
<withoutfile>126</withoutfile>
<withoutfile>127</withoutfile>
<withoutfile>128</withoutfile>
<withoutfile>129</withoutfile>
<withoutfile>130</withoutfile>
<withoutfile>131</withoutfile>
<withoutfile>132</withoutfile>
<withoutfile>133</withoutfile>
<withoutfile>134</withoutfile>
<withoutfile>135</withoutfile>
<withoutfile>136</withoutfile>
<withoutfile>137</withoutfile>
<withoutfile>138</withoutfile>
<withoutfile>139</withoutfile>
<withoutfile>140</withoutfile>
<withoutfile>141</withoutfile>
<withoutfile>142</withoutfile>
<withoutfile>143</withoutfile>
<withoutfile>144</withoutfile>
<withoutfile>145</withoutfile>
<withoutfile>146</withoutfile>
<withoutfile>147</withoutfile>
<withoutfile>148</withoutfile>
<withoutfile>149</withoutfile>
<withoutfile>150</withoutfile>
<withoutfile>151</withoutfile>
<withoutfile>152</withoutfile>
<withoutfile>153</withoutfile>
<withoutfile>154</withoutfile>
<withoutfile>155</withoutfile>
<withoutfile>156</withoutfile>
<withoutfile>157</withoutfile>
<withoutfile>158</withoutfile>
<withoutfile>159</withoutfile>
<withoutfile>160</withoutfile>
<withoutfile>161</withoutfile>
<withoutfile>162</withoutfile>
<withoutfile>163</withoutfile>
<withoutfile>164</withoutfile>
<withoutfile>165</withoutfile>
<withoutfile>166</withoutfile>
<withoutfile>167</withoutfile>
<withoutfile>168</withoutfile>
<withoutfile>169</withoutfile>
<withoutfile>170</withoutfile>
<withoutfile>171</withoutfile>
<withoutfile>172</withoutfile>
<withoutfile>173</withoutfile>
<withoutfile>174</withoutfile>
<withoutfile>175</withoutfile>
<withoutfile>176</withoutfile>
<withoutfile>177</withoutfile>
<withoutfile>178</withoutfile>
<withoutfile>179</withoutfile>
<withoutfile>180</withoutfile>
<withoutfile>181</withoutfile>
<withoutfile>182</withoutfile>
<withoutfile>183</withoutfile>
<withoutfile>184</withoutfile>
<withoutfile>185</withoutfile>
<withoutfile>186</withoutfile>
<withoutfile>187</withoutfile>
<withoutfile>188</withoutfile>
<withoutfile>189</withoutfile>
<withoutfile>190</withoutfile>
<withoutfile>191</withoutfile>
<withoutfile>192</withoutfile>
<withoutfile>193</withoutfile>
<withoutfile>194</withoutfile>
<withoutfile>195</withoutfile>
<withoutfile>196</withoutfile>
<withoutfile>197</withoutfile>
<withoutfile>198</withoutfile>
<withoutfile>199</withoutfile>
<withoutfile>200</withoutfile>
<withoutfile>201</withoutfile>
<withoutfile>202</withoutfile>
<withoutfile>203</withoutfile>
<withoutfile>204</withoutfile>
<withoutfile>205</withoutfile>
<withoutfile>206</withoutfile>
<withoutfile>207</withoutfile>
<withoutfile>208</withoutfile>
<withoutfile>209</withoutfile>
<withoutfile>210</withoutfile>
<withoutfile>211</withoutfile>
<withoutfile>212</withoutfile>
<withoutfile>213</withoutfile>
<withoutfile>214</withoutfile>
<withoutfile>215</withoutfile>
<withoutfile>216</withoutfile>
<withoutfile>217</withoutfile>
<withoutfile>218</withoutfile>
<withoutfile>219</withoutfile>
<withoutfile>220</withoutfile>
<withoutfile>221</withoutfile>
<withoutfile>222</withoutfile>
<withoutfile>223</withoutfile>
<withoutfile>224</withoutfile>
<withoutfile>225</withoutfile>
<withoutfile>226</withoutfile>
<withoutfile>227</withoutfile>
<withoutfile>228</withoutfile>
<withoutfile>229</withoutfile>
<withoutfile>230</withoutfile>
<withoutfile>231</withoutfile>
<withoutfile>232</withoutfile>
<withoutfile>233</withoutfile>
<withoutfile>234</withoutfile>
<withoutfile>235</withoutfile>
<withoutfile>236</withoutfile>
<withoutfile>237</withoutfile>
<withoutfile>238</withoutfile>
<withoutfile>239</withoutfile>
<withoutfile>240</withoutfile>
<withoutfile>241</withoutfile>
<withoutfile>242</withoutfile>
<withoutfile>243</withoutfile>
<withoutfile>244</withoutfile>
<withoutfile>245</withoutfile>
<withoutfile>246</withoutfile>
<withoutfile>247</withoutfile>
<withoutfile>248</withoutfile>
<withoutfile>249</withoutfile>
<withoutfile>250</withoutfile>
<withoutfile>251</withoutfile>
<withoutfile>252</withoutfile>
<withoutfile>253</withoutfile>
<withoutfile>254</withoutfile>
<withoutfile>255</withoutfile>
<withoutfile>256</withoutfile>
<withoutfile>257</withoutfile>
<withoutfile>258</withoutfile>
<withoutfile>259</withoutfile>
<withoutfile>260</withoutfile>
<withoutfile>261</withoutfile>
<withoutfile>262</withoutfile>
<withoutfile>263</withoutfile>
<withoutfile>264</withoutfile>
<withoutfile>265</withoutfile>
<withoutfile>266</withoutfile>
<withoutfile>267</withoutfile>
<withoutfile>268</withoutfile>
<withoutfile>269</withoutfile>
<withoutfile>270</withoutfile>
<withoutfile>271</withoutfile>
<withoutfile>272</withoutfile>
<withoutfile>273</withoutfile>
<withoutfile>274</withoutfile>
<withoutfile>275</withoutfile>
<withoutfile>276</withoutfile>
<withoutfile>277</withoutfile>
<withoutfile>278</withoutfile>
<withoutfile>279</withoutfile>
<withoutfile>280</withoutfile>
<withoutfile>281</withoutfile>
<withoutfile>282</withoutfile>
<withoutfile>283</withoutfile>
<withoutfile>284</withoutfile>
<withoutfile>285</withoutfile>
<withoutfile>286</withoutfile>
<withoutfile>287</withoutfile>
<withoutfile>288</withoutfile>
<withoutfile>289</withoutfile>
<withoutfile>290</withoutfile>
<withoutfile>291</withoutfile>
<withoutfile>292</withoutfile>
<withoutfile>293</withoutfile>
<withoutfile>294</withoutfile>
<withoutfile>295</withoutfile>
<withoutfile>296</withoutfile>
<withoutfile>297</withoutfile>
<withoutfile>298</withoutfile>
<withoutfile>299</withoutfile>
<withoutfile>300</withoutfile>
<withoutfile>301</withoutfile>
<withoutfile>302</withoutfile>
<withoutfile>303</withoutfile>
<withoutfile>304</withoutfile>
<withoutfile>305</withoutfile>
<withoutfile>306</withoutfile>
<withoutfile>307</withoutfile>
<withoutfile>308</withoutfile>
<withoutfile>309</withoutfile>
<withoutfile>310</withoutfile>
<withoutfile>311</withoutfile>
<withoutfile>312</withoutfile>
<withoutfile>313</withoutfile>
<withoutfile>314</withoutfile>
<withoutfile>315</withoutfile>
<withoutfile>316</withoutfile>
<withoutfile>317</withoutfile>
<withoutfile>318</withoutfile>
<withoutfile>319</withoutfile>
<withoutfile>320</withoutfile>
<withoutfile>321</withoutfile>
<withoutfile>322</withoutfile>
<withoutfile>323</withoutfile>
<withoutfile>324</withoutfile>
<withoutfile>325</withoutfile>
<withoutfile>326</withoutfile>
<withoutfile>327</withoutfile>
<withoutfile>328</withoutfile>
<withoutfile>329</withoutfile>
<withoutfile>330</withoutfile>
<withoutfile>331</withoutfile>
<withoutfile>332</withoutfile>
<withoutfile>333</withoutfile>
<withoutfile>334</withoutfile>
<withoutfile>335</withoutfile>
<withoutfile>336</withoutfile>
<withoutfile>337</withoutfile>
<withoutfile>338</withoutfile>
<withoutfile>339</withoutfile>
<withoutfile>340</withoutfile>
<withoutfile>341</withoutfile>
<withoutfile>342</withoutfile>
<withoutfile>343</withoutfile>
<withoutfile>344</withoutfile>
<withoutfile>345</withoutfile>
<withoutfile>346</withoutfile>
<withoutfile>347</withoutfile>
<withoutfile>348</withoutfile>
<withoutfile>349</withoutfile>
<withoutfile>350</withoutfile>
<withoutfile>351</withoutfile>
<withoutfile>352</withoutfile>
<withoutfile>353</withoutfile>
<withoutfile>354</withoutfile>
<withoutfile>355</withoutfile>
<withoutfile>356</withoutfile>
<withoutfile>357</withoutfile>
<withoutfile>358</withoutfile>
<withoutfile>359</withoutfile>
<withoutfile>360</withoutfile>
<withoutfile>361</withoutfile>
<withoutfile>362</withoutfile>
<withoutfile>363</withoutfile>
<withoutfile>364</withoutfile>
<withoutfile>365</withoutfile>
<withoutfile>366</withoutfile>
<withoutfile>367</withoutfile>
<withoutfile>368</withoutfile>
<withoutfile>369</withoutfile>
<withoutfile>370</withoutfile>
<withoutfile>371</withoutfile>
<withoutfile>372</withoutfile>
<withoutfile>373</withoutfile>
<withoutfile>374</withoutfile>
<withoutfile>375</withoutfile>
<withoutfile>376</withoutfile>
<withoutfile>377</withoutfile>
<withoutfile>378</withoutfile>
<withoutfile>379</withoutfile>
<withoutfile>380</withoutfile>
<withoutfile>381</withoutfile>
<withoutfile>382</withoutfile>
<withoutfile>383</withoutfile>
<withoutfile>384</withoutfile>
<withoutfile>385</withoutfile>
<withoutfile>386</withoutfile>
<withoutfile>387</withoutfile>
<withoutfile>388</withoutfile>
<withoutfile>389</withoutfile>
<withoutfile>390</withoutfile>
<withoutfile>391</withoutfile>
<withoutfile>392</withoutfile>
<withoutfile>393</withoutfile>
<withoutfile>394</withoutfile>
<withoutfile>395</withoutfile>
<withoutfile>396</withoutfile>
<withoutfile>397</withoutfile>
<withoutfile>398</withoutfile>
<withoutfile>399</withoutfile>
<withoutfile>400</withoutfile>
<withoutfile>401</withoutfile>
<withoutfile>402</withoutfile>
<withoutfile>403</withoutfile>
<withoutfile>404</withoutfile>
<withoutfile>405</withoutfile>
<withoutfile>406</withoutfile>
<withoutfile>407</withoutfile>
<withoutfile>408</withoutfile>
<withoutfile>409</withoutfile>
<withoutfile>410</withoutfile>
<withoutfile>411</withoutfile>
<withoutfile>412</withoutfile>
<withoutfile>413</withoutfile>
<withoutfile>414</withoutfile>
<withoutfile>415</withoutfile>
<withoutfile>416</withoutfile>
<withoutfile>417</withoutfile>
<withoutfile>418</withoutfile>
<withoutfile>419</withoutfile>
<withoutfile>420</withoutfile>
<withoutfile>421</withoutfile>
<withoutfile>422</withoutfile>
<withoutfile>423</withoutfile>
<withoutfile>424</withoutfile>
<withoutfile>425</withoutfile>
<withoutfile>426</withoutfile>
<withoutfile>427</withoutfile>
<withoutfile>428</withoutfile>
<withoutfile>429</withoutfile>
<withoutfile>430</withoutfile>
<withoutfile>431</withoutfile>
<withoutfile>432</withoutfile>
<withoutfile>433</withoutfile>
<withoutfile>434</withoutfile>
<withoutfile>435</withoutfile>
<withoutfile>436</withoutfile>
<withoutfile>437</withoutfile>
<withoutfile>438</withoutfile>
<withoutfile>439</withoutfile>
<withoutfile>440</withoutfile>
<withoutfile>441</withoutfile>
<withoutfile>442</withoutfile>
<withoutfile>443</withoutfile>
<withoutfile>444</withoutfile>
<withoutfile>445</withoutfile>
<withoutfile>446</withoutfile>
<withoutfile>447</withoutfile>
<withoutfile>448</withoutfile>
<withoutfile>449</withoutfile>
<withoutfile>450</withoutfile>
<withoutfile>451</withoutfile>
<withoutfile>452</withoutfile>
<withoutfile>453</withoutfile>
<withoutfile>454</withoutfile>
<withoutfile>455</withoutfile>
<withoutfile>456</withoutfile>
<withoutfile>457</withoutfile>
<withoutfile>458</withoutfile>
<withoutfile>459</withoutfile>
<withoutfile>460</withoutfile>
<withoutfile>461</withoutfile>
<withoutfile>462</withoutfile>
<withoutfile>463</withoutfile>
<withoutfile>464</withoutfile>
<withoutfile>465</withoutfile>
<withoutfile>466</withoutfile>
<withoutfile>467</withoutfile>
<withoutfile>468</withoutfile>
<withoutfile>469</withoutfile>
<withoutfile>470</withoutfile>
<withoutfile>471</withoutfile>
<withoutfile>472</withoutfile>
<withoutfile>473</withoutfile>
<withoutfile>474</withoutfile>
<withoutfile>475</withoutfile>
<withoutfile>476</withoutfile>
<withoutfile>477</withoutfile>
<withoutfile>478</withoutfile>
<withoutfile>479</withoutfile>
<withoutfile>480</withoutfile>
<withoutfile>481</withoutfile>
<withoutfile>482</withoutfile>
<withoutfile>483</withoutfile>
<withoutfile>484</withoutfile>
<withoutfile>485</withoutfile>
<withoutfile>486</withoutfile>
<withoutfile>487</withoutfile>
<withoutfile>488</withoutfile>
<withoutfile>489</withoutfile>
<withoutfile>490</withoutfile>
<withoutfile>491</withoutfile>
<withoutfile>492</withoutfile>
<withoutfile>493</withoutfile>
<withoutfile>494</withoutfile>
<withoutfile>495</withoutfile>
<withoutfile>496</withoutfile>
<withoutfile>497</withoutfile>
<withoutfile>498</withoutfile>
<withoutfile>499</withoutfile>
<withoutfile>500</withoutfile>
<withoutfile>501</withoutfile>
<withoutfile>502</withoutfile>
<withoutfile>503</withoutfile>
<withoutfile>504</withoutfile>
<withoutfile>505</withoutfile>
<withoutfile>506</withoutfile>
<withoutfile>507</withoutfile>
<withoutfile>508</withoutfile>
<withoutfile>509</withoutfile>
<withoutfile>510</withoutfile>
<withoutfile>511</withoutfile>
<withoutfile>512</withoutfile>
<withoutfile>513</withoutfile>
<withoutfile>514</withoutfile>
<withoutfile>515</withoutfile>
<withoutfile>516</withoutfile>
<withoutfile>517</withoutfile>
<withoutfile>518</withoutfile>
<withoutfile>519</withoutfile>
<withoutfile>520</withoutfile>
<withoutfile>521</withoutfile>
<withoutfile>522</withoutfile>
<withoutfile>523</withoutfile>
<withoutfile>524</withoutfile>
<withoutfile>525</withoutfile>
<withoutfile>526</withoutfile>
<withoutfile>527</withoutfile>
<withoutfile>528</withoutfile>
<withoutfile>529</withoutfile>
<withoutfile>530</withoutfile>
<withoutfile>531</withoutfile>
<withoutfile>532</withoutfile>
<withoutfile>533</withoutfile>
<withoutfile>534</withoutfile>
<withoutfile>535</withoutfile>
<withoutfile>536</withoutfile>
<withoutfile>537</withoutfile>
<withoutfile>538</withoutfile>
<withoutfile>539</withoutfile>
<withoutfile>540</withoutfile>
<withoutfile>541</withoutfile>
<withoutfile>542</withoutfile>
<withoutfile>543</withoutfile>
<withoutfile>544</withoutfile>
<withoutfile>545</withoutfile>
<withoutfile>546</withoutfile>
<withoutfile>547</withoutfile>
<withoutfile>548</withoutfile>
<withoutfile>549</withoutfile>
<withoutfile>550</withoutfile>
<withoutfile>551</withoutfile>
<withoutfile>552</withoutfile>
<withoutfile>553</withoutfile>
<withoutfile>554</withoutfile>
<withoutfile>555</withoutfile>
<withoutfile>556</withoutfile>
<withoutfile>557</withoutfile>
<withoutfile>558</withoutfile>
<withoutfile>559</withoutfile>
<withoutfile>560</withoutfile>
<withoutfile>561</withoutfile>
<withoutfile>562</withoutfile>
<withoutfile>563</withoutfile>
<withoutfile>564</withoutfile>
<withoutfile>565</withoutfile>
<withoutfile>566</withoutfile>
<withoutfile>567</withoutfile>
<withoutfile>568</withoutfile>
<withoutfile>569</withoutfile>
<withoutfile>570</withoutfile>
<withoutfile>571</withoutfile>
<withoutfile>572</withoutfile>
<withoutfile>573</withoutfile>
<withoutfile>574</withoutfile>
<withoutfile>575</withoutfile>
<withoutfile>576</withoutfile>
<withoutfile>577</withoutfile>
<withoutfile>578</withoutfile>
<withoutfile>579</withoutfile>
<withoutfile>580</withoutfile>
<withoutfile>581</withoutfile>
<withoutfile>582</withoutfile>
<withoutfile>583</withoutfile>
<withoutfile>584</withoutfile>
<withoutfile>585</withoutfile>
<withoutfile>586</withoutfile>
<withoutfile>587</withoutfile>
<withoutfile>588</withoutfile>
<withoutfile>589</withoutfile>
<withoutfile>590</withoutfile>
<withoutfile>591</withoutfile>
<withoutfile>592</withoutfile>
<withoutfile>593</withoutfile>
<withoutfile>594</withoutfile>
<withoutfile>595</withoutfile>
<withoutfile>596</withoutfile>
<withoutfile>597</withoutfile>
<withoutfile>598</withoutfile>
<withoutfile>599</withoutfile>
<withoutfile>600</withoutfile>
<withoutfile>601</withoutfile>
<withoutfile>602</withoutfile>
<withoutfile>603</withoutfile>
<withoutfile>604</withoutfile>
<withoutfile>605</withoutfile>
<withoutfile>606</withoutfile>
<withoutfile>607</withoutfile>
<withoutfile>608</withoutfile>
<withoutfile>609</withoutfile>
<withoutfile>610</withoutfile>
<withoutfile>611</withoutfile>
<withoutfile>612</withoutfile>
<withoutfile>613</withoutfile>
<withoutfile>614</withoutfile>
<withoutfile>615</withoutfile>
<withoutfile>616</withoutfile>
<withoutfile>617</withoutfile>
<withoutfile>618</withoutfile>
<withoutfile>619</withoutfile>
<withoutfile>620</withoutfile>
<withoutfile>621</withoutfile>
<withoutfile>622</withoutfile>
<withoutfile>623</withoutfile>
<withoutfile>624</withoutfile>
<withoutfile>625</withoutfile>
<withoutfile>626</withoutfile>
<withoutfile>627</withoutfile>
<withoutfile>628</withoutfile>
<withoutfile>629</withoutfile>
<withoutfile>630</withoutfile>
<withoutfile>631</withoutfile>
<withoutfile>632</withoutfile>
<withoutfile>633</withoutfile>
<withoutfile>634</withoutfile>
<withoutfile>635</withoutfile>
<withoutfile>636</withoutfile>
<withoutfile>637</withoutfile>
<withoutfile>638</withoutfile>
<withoutfile>639</withoutfile>
<withoutfile>640</withoutfile>
<withoutfile>641</withoutfile>
<withoutfile>642</withoutfile>
<withoutfile>643</withoutfile>
<withoutfile>644</withoutfile>
<withoutfile>645</withoutfile>
<withoutfile>646</withoutfile>
<withoutfile>647</withoutfile>
<withoutfile>648</withoutfile>
<withoutfile>649</withoutfile>
<withoutfile>650</withoutfile>
<withoutfile>651</withoutfile>
<withoutfile>652</withoutfile>
<withoutfile>653</withoutfile>
<withoutfile>654</withoutfile>
<withoutfile>655</withoutfile>
<withoutfile>656</withoutfile>
<withoutfile>657</withoutfile>
<withoutfile>658</withoutfile>
<withoutfile>659</withoutfile>
<withoutfile>660</withoutfile>
<withoutfile>661</withoutfile>
<withoutfile>662</withoutfile>
<withoutfile>663</withoutfile>
<withoutfile>664</withoutfile>
<withoutfile>665</withoutfile>
<withoutfile>666</withoutfile>
<withoutfile>667</withoutfile>
<withoutfile>668</withoutfile>
<withoutfile>669</withoutfile>
<withoutfile>670</withoutfile>
<withoutfile>671</withoutfile>
<withoutfile>672</withoutfile>
<withoutfile>673</withoutfile>
<withoutfile>674</withoutfile>
<withoutfile>675</withoutfile>
<withoutfile>676</withoutfile>
<withoutfile>677</withoutfile>
<withoutfile>678</withoutfile>
<withoutfile>679</withoutfile>
<withoutfile>680</withoutfile>
<withoutfile>681</withoutfile>
<withoutfile>682</withoutfile>
<withoutfile>683</withoutfile>
<withoutfile>684</withoutfile>
<withoutfile>685</withoutfile>
<withoutfile>686</withoutfile>
<withoutfile>687</withoutfile>
<withoutfile>688</withoutfile>
<withoutfile>689</withoutfile>
<withoutfile>690</withoutfile>
<withoutfile>691</withoutfile>
<withoutfile>692</withoutfile>
<withoutfile>693</withoutfile>
<withoutfile>694</withoutfile>
<withoutfile>695</withoutfile>
<withoutfile>696</withoutfile>
<withoutfile>697</withoutfile>
<withoutfile>698</withoutfile>
<withoutfile>699</withoutfile>
<withoutfile>700</withoutfile>
<withoutfile>701</withoutfile>
<withoutfile>702</withoutfile>
<withoutfile>703</withoutfile>
<withoutfile>704</withoutfile>
<withoutfile>705</withoutfile>
<withoutfile>706</withoutfile>
<withoutfile>707</withoutfile>
<withoutfile>708</withoutfile>
<withoutfile>709</withoutfile>
<withoutfile>710</withoutfile>
<withoutfile>711</withoutfile>
<withoutfile>712</withoutfile>
<withoutfile>713</withoutfile>
<withoutfile>714</withoutfile>
<withoutfile>715</withoutfile>
<withoutfile>716</withoutfile>
<withoutfile>717</withoutfile>
<withoutfile>718</withoutfile>
<withoutfile>719</withoutfile>
<withoutfile>720</withoutfile>
<withoutfile>721</withoutfile>
<withoutfile>722</withoutfile>
<withoutfile>723</withoutfile>
<withoutfile>724</withoutfile>
<withoutfile>725</withoutfile>
<withoutfile>726</withoutfile>
<withoutfile>727</withoutfile>
<withoutfile>728</withoutfile>
<withoutfile>729</withoutfile>
<withoutfile>730</withoutfile>
<withoutfile>731</withoutfile>
<withoutfile>732</withoutfile>
<withoutfile>733</withoutfile>
<withoutfile>734</withoutfile>
<withoutfile>735</withoutfile>
<withoutfile>736</withoutfile>
<withoutfile>737</withoutfile>
<withoutfile>738</withoutfile>
<withoutfile>739</withoutfile>
<withoutfile>740</withoutfile>
<withoutfile>741</withoutfile>
<withoutfile>742</withoutfile>
<withoutfile>743</withoutfile>
<withoutfile>744</withoutfile>
<withoutfile>745</withoutfile>
<withoutfile>746</withoutfile>
<withoutfile>747</withoutfile>
<withoutfile>748</withoutfile>
<withoutfile>749</withoutfile>
<withoutfile>750</withoutfile>
<withoutfile>751</withoutfile>
<withoutfile>752</withoutfile>
<withoutfile>753</withoutfile>
<withoutfile>754</withoutfile>
<withoutfile>755</withoutfile>
<withoutfile>756</withoutfile>
<withoutfile>757</withoutfile>
<withoutfile>758</withoutfile>
<withoutfile>759</withoutfile>
<withoutfile>760</withoutfile>
<withoutfile>761</withoutfile>
<withoutfile>762</withoutfile>
<withoutfile>763</withoutfile>
<withoutfile>764</withoutfile>
<withoutfile>765</withoutfile>
<withoutfile>766</withoutfile>
<withoutfile>767</withoutfile>
<withoutfile>768</withoutfile>
<withoutfile>769</withoutfile>
<withoutfile>770</withoutfile>
<withoutfile>771</withoutfile>
<withoutfile>772</withoutfile>
<withoutfile>773</withoutfile>
<withoutfile>774</withoutfile>
<withoutfile>775</withoutfile>
<withoutfile>776</withoutfile>
<withoutfile>777</withoutfile>
<withoutfile>778</withoutfile>
<withoutfile>779</withoutfile>
<withoutfile>780</withoutfile>
<withoutfile>781</withoutfile>
<withoutfile>782</withoutfile>
<withoutfile>783</withoutfile>
<withoutfile>784</withoutfile>
<withoutfile>785</withoutfile>
<withoutfile>786</withoutfile>
<withoutfile>787</withoutfile>
<withoutfile>788</withoutfile>
<withoutfile>789</withoutfile>
<withoutfile>790</withoutfile>
<withoutfile>791</withoutfile>
<withoutfile>792</withoutfile>
<withoutfile>793</withoutfile>
<withoutfile>794</withoutfile>
<withoutfile>795</withoutfile>
<withoutfile>796</withoutfile>
<withoutfile>797</withoutfile>
<withoutfile>798</withoutfile>
<withoutfile>799</withoutfile>
<withoutfile>800</withoutfile>
<withoutfile>801</withoutfile>
<withoutfile>802</withoutfile>
<withoutfile>803</withoutfile>
<withoutfile>804</withoutfile>
<withoutfile>805</withoutfile>
<withoutfile>806</withoutfile>
<withoutfile>807</withoutfile>
<withoutfile>808</withoutfile>
<withoutfile>809</withoutfile>
<withoutfile>810</withoutfile>
<withoutfile>811</withoutfile>
<withoutfile>812</withoutfile>
<withoutfile>813</withoutfile>
<withoutfile>814</withoutfile>
<withoutfile>815</withoutfile>
<withoutfile>816</withoutfile>
<withoutfile>817</withoutfile>
<withoutfile>818</withoutfile>
<withoutfile>819</withoutfile>
<withoutfile>820</withoutfile>
<withoutfile>821</withoutfile>
<withoutfile>822</withoutfile>
<withoutfile>823</withoutfile>
<withoutfile>824</withoutfile>
<withoutfile>825</withoutfile>
<withoutfile>826</withoutfile>
<withoutfile>827</withoutfile>
<withoutfile>828</withoutfile>
<withoutfile>829</withoutfile>
<withoutfile>830</withoutfile>
<withoutfile>831</withoutfile>
<withoutfile>832</withoutfile>
<withoutfile>833</withoutfile>
<withoutfile>834</withoutfile>
<withoutfile>835</withoutfile>
<withoutfile>836</withoutfile>
<withoutfile>837</withoutfile>
<withoutfile>838</withoutfile>
<withoutfile>839</withoutfile>
<withoutfile>840</withoutfile>
<withoutfile>841</withoutfile>
<withoutfile>842</withoutfile>
<withoutfile>843</withoutfile>
<withoutfile>844</withoutfile>
<withoutfile>845</withoutfile>
<withoutfile>846</withoutfile>
<withoutfile>847</withoutfile>
<withoutfile>848</withoutfile>
<withoutfile>849</withoutfile>
<withoutfile>850</withoutfile>
<withoutfile>851</withoutfile>
<withoutfile>852</withoutfile>
<withoutfile>853</withoutfile>
<withoutfile>854</withoutfile>
<withoutfile>855</withoutfile>
<withoutfile>856</withoutfile>
<withoutfile>857</withoutfile>
<withoutfile>858</withoutfile>
<withoutfile>859</withoutfile>
<withoutfile>860</withoutfile>
<withoutfile>861</withoutfile>
<withoutfile>862</withoutfile>
<withoutfile>863</withoutfile>
<withoutfile>864</withoutfile>
<withoutfile>865</withoutfile>
<withoutfile>866</withoutfile>
<withoutfile>867</withoutfile>
<withoutfile>868</withoutfile>
<withoutfile>869</withoutfile>
<withoutfile>870</withoutfile>
<withoutfile>871</withoutfile>
<withoutfile>872</withoutfile>
<withoutfile>873</withoutfile>
<withoutfile>874</withoutfile>
<withoutfile>875</withoutfile>
<withoutfile>876</withoutfile>
<withoutfile>877</withoutfile>
<withoutfile>878</withoutfile>
<withoutfile>879</withoutfile>
<withoutfile>880</withoutfile>
<withoutfile>881</withoutfile>
<withoutfile>882</withoutfile>
<withoutfile>883</withoutfile>
<withoutfile>884</withoutfile>
<withoutfile>885</withoutfile>
<withoutfile>886</withoutfile>
<withoutfile>887</withoutfile>
<withoutfile>888</withoutfile>
<withoutfile>889</withoutfile>
<withoutfile>890</withoutfile>
<withoutfile>891</withoutfile>
<withoutfile>892</withoutfile>
<withoutfile>893</withoutfile>
<withoutfile>894</withoutfile>
<withoutfile>895</withoutfile>
<withoutfile>896</withoutfile>
<withoutfile>897</withoutfile>
<withoutfile>898</withoutfile>
<withoutfile>899</withoutfile>
<withoutfile>900</withoutfile>
<withoutfile>901</withoutfile>
<withoutfile>902</withoutfile>
<withoutfile>903</withoutfile>
<withoutfile>904</withoutfile>
<withoutfile>905</withoutfile>
<withoutfile>906</withoutfile>
<withoutfile>907</withoutfile>
<withoutfile>908</withoutfile>
<withoutfile>909</withoutfile>
<withoutfile>910</withoutfile>
<withoutfile>911</withoutfile>
<withoutfile>912</withoutfile>
<withoutfile>913</withoutfile>
<withoutfile>914</withoutfile>
<withoutfile>915</withoutfile>
<withoutfile>916</withoutfile>
<withoutfile>917</withoutfile>
<withoutfile>918</withoutfile>
<withoutfile>919</withoutfile>
<withoutfile>920</withoutfile>
<withoutfile>921</withoutfile>
<withoutfile>922</withoutfile>
<withoutfile>923</withoutfile>
<withoutfile>924</withoutfile>
<withoutfile>925</withoutfile>
<withoutfile>926</withoutfile>
<withoutfile>927</withoutfile>
<withoutfile>928</withoutfile>
<withoutfile>929</withoutfile>
<withoutfile>930</withoutfile>
<withoutfile>931</withoutfile>
<withoutfile>932</withoutfile>
<withoutfile>933</withoutfile>
<withoutfile>934</withoutfile>
<withoutfile>935</withoutfile>
<withoutfile>936</withoutfile>
<withoutfile>937</withoutfile>
<withoutfile>938</withoutfile>
<withoutfile>939</withoutfile>
<withoutfile>940</withoutfile>
<withoutfile>941</withoutfile>
<withoutfile>942</withoutfile>
<withoutfile>943</withoutfile>
<withoutfile>944</withoutfile>
<withoutfile>945</withoutfile>
<withoutfile>946</withoutfile>
<withoutfile>947</withoutfile>
<withoutfile>948</withoutfile>
<withoutfile>949</withoutfile>
<withoutfile>950</withoutfile>
<withoutfile>951</withoutfile>
<withoutfile>952</withoutfile>
<withoutfile>953</withoutfile>
<withoutfile>954</withoutfile>
<withoutfile>955</withoutfile>
<withoutfile>956</withoutfile>
<withoutfile>957</withoutfile>
<withoutfile>958</withoutfile>
<withoutfile>959</withoutfile>
<withoutfile>960</withoutfile>
<withoutfile>961</withoutfile>
<withoutfile>962</withoutfile>
<withoutfile>963</withoutfile>
<withoutfile>964</withoutfile>
<withoutfile>965</withoutfile>
<withoutfile>966</withoutfile>
<withoutfile>967</withoutfile>
<withoutfile>968</withoutfile>
<withoutfile>969</withoutfile>
<withoutfile>970</withoutfile>
<withoutfile>971</withoutfile>
<withoutfile>972</withoutfile>
<withoutfile>973</withoutfile>
<withoutfile>974</withoutfile>
<withoutfile>975</withoutfile>
<withoutfile>976</withoutfile>
<withoutfile>977</withoutfile>
<withoutfile>978</withoutfile>
<withoutfile>979</withoutfile>
<withoutfile>980</withoutfile>
<withoutfile>981</withoutfile>
<withoutfile>982</withoutfile>
<withoutfile>983</withoutfile>
<withoutfile>984</withoutfile>
<withoutfile>985</withoutfile>
<withoutfile>986</withoutfile>
<withoutfile>987</withoutfile>
<withoutfile>988</withoutfile>
<withoutfile>989</withoutfile>
<withoutfile>990</withoutfile>
<withoutfile>991</withoutfile>
<withoutfile>992</withoutfile>
<withoutfile>993</withoutfile>
<withoutfile>994</withoutfile>
<withoutfile>995</withoutfile>
<withoutfile>996</withoutfile>
<withoutfile>997</withoutfile>
<withoutfile>998</withoutfile>
<withoutfile>999</withoutfile>
<withoutfile>1000</withoutfile>
  </tasks>
  <rundefinition/>
</benchmark>

This reproduces it reliably for me on Python 3.6 (Ubuntu 18.04) and Python 3.7 (from our Docker container for tests). After killing all child processes of benchexec (with sudo kill -9), the main process continues successfully and terminates (reporting a few failed runs).

Stack traces of the hanging processes

Python 3.7:

#0  0x00007ffb818d517f in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ffb81e9d26f in drop_gil (tstate=tstate@entry=0x55c1cb2a5060) at Python/ceval_gil.h:183
#2  0x00007ffb81d8bcbd in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:979
#3  0x00007ffb81e9eeac in _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffb7df03d20, globals=globals@entry=0x7ffb7deb1ca8, 
    locals=locals@entry=0x0, args=args@entry=0x7ffb8229f060, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=0x0, 
    kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7ffb58525ba8, name=0x7ffb7ee305f0, qualname=0x7ffb7de99b70)
    at Python/ceval.c:3930
#4  0x00007ffb81db5607 in _PyFunction_FastCallDict (func=0x7ffb5adeb378, args=0x7ffb8229f060, nargs=0, kwargs=0x0)
    at Objects/call.c:376
#5  0x00007ffb7c763f6d in _CallPythonObject (pArgs=<optimized out>, flags=4353, converters=0x7ffb8229f048, 
    callable=0x7ffb5adeb378, setfunc=0x7ffb7c768d10 <i_set>, restype=0x7ffb7deb9900, mem=0x7ffb5ac3cfe0)
    at /usr/src/python/Modules/_ctypes/callbacks.c:232
#6  closure_fcn (cif=<optimized out>, resp=0x7ffb5ac3cfe0, args=0x7ffb5ac3ce60, userdata=<optimized out>)
    at /usr/src/python/Modules/_ctypes/callbacks.c:292
#7  0x00007ffb7c550e2f in ffi_closure_unix64_inner () from target:/usr/lib/x86_64-linux-gnu/libffi.so.6
#8  0x00007ffb7c5511a0 in ffi_closure_unix64 () from target:/usr/lib/x86_64-linux-gnu/libffi.so.6
#9  0x00007ffb80f06d0f in clone () from target:/lib/x86_64-linux-gnu/libc.so.6

Python 3.6:

#0  0x00007fcf09d7b9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib/x86_64-linux-gnu/libpthread.so.0                                                                                                      
#1  0x000000000052a431 in drop_gil (tstate=tstate@entry=0x23b7480) at ../Python/ceval_gil.h:200                                                                                                                    
#2  0x000000000052eb1e in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:1190                                                                                        
#3  0x000000000052ce56 in PyEval_EvalFrameEx (f=f@entry=0x7fcf04351240, throwflag=throwflag@entry=0) at ../Python/ceval.c:754                                                                                      
#4  0x000000000052d905 in _PyEval_EvalCodeWithName (_co=0x7fcf076cc1c0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2,           
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x7fcf084729f8, name=0x0, qualname=0x0) at ../Python/ceval.c:4166                                                                                                    
#5  0x000000000052df0b in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fcf0a350080, argcount=<optimized out>, kws=kws@entry=0x0, kwcount=0,        
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x7fcf084729f8) at ../Python/ceval.c:4187                                                                                                                            
#6  0x0000000000479ac3 in function_call (func=0x7fcef5ae9618, arg=0x7fcf0a350058, kw=<optimized out>) at ../Objects/funcobject.c:604                                                                               
#7  0x000000000044a431 in PyObject_Call (func=func@entry=0x7fcef5ae9618, args=args@entry=0x7fcf0a350058, kwargs=kwargs@entry=0x0) at ../Objects/abstract.c:2261                                                    
#8  0x000000000052e456 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fcef5ae9618, args=args@entry=0x7fcf0a350058, kwargs=kwargs@entry=0x0) at ../Python/ceval.c:4771                                        
#9  0x000000000044a29d in PyObject_CallObject (o=o@entry=0x7fcef5ae9618, a=a@entry=0x7fcf0a350058) at ../Objects/abstract.c:2187                                                                                   
#10 0x00007fcf06a504ba in _CallPythonObject (mem=0x7fcef59fafe0, restype=0x7fcf076e36e0, setfunc=0x7fcf06a5619b <i_set>, callable=0x7fcef5ae9618, converters=0x7fcf0a350058, flags=4357, pArgs=0x7fcef59fae60)     
    at ./Modules/_ctypes/callbacks.c:234                                                                                                                                                                           
#11 0x00007fcf06a5071e in closure_fcn (cif=<optimized out>, resp=<optimized out>, args=<optimized out>, userdata=<optimized out>) at ./Modules/_ctypes/callbacks.c:296                                             
#12 0x00007fcf0683cb4f in ffi_closure_unix64_inner () from target:/usr/lib/x86_64-linux-gnu/libffi.so.6                                                                                                            
#13 0x00007fcf0683cf16 in ffi_closure_unix64 () from target:/usr/lib/x86_64-linux-gnu/libffi.so.6                                                                                                                  
#14 0x00007fcf08eaa88f in clone () from target:/lib/x86_64-linux-gnu/libc.so.6  

Possible Explanation

The entry point of the process is of course clone. There are three stack entries related to the callback into the Python world that ctypes does for us, and then stack entries for calling a Python method. The process hangs in a pthread wait in drop_gil of the Python interpreter. In this line, drop_gil has successfully dropped the global interpreter lock (GIL) and wants to wait until another thread has picked it up immediately because of gil_drop_request (explanation taken from grpc/grpc#18075 (comment) and the implementation documentation). Of course, after the fork there exists only one thread, and the condition will never be notified, leading to the infinite wait. Because the documentation says that gil_drop_request is used by threads only if they do not get the GIL for some time, this also explains why it happens only with many parallel runs. Otherwise there is less thread contention.

My current assumption is that this deadlock happens immediately in the callback from clone. The first thing we do is calling PyOS_AfterFork_Child which should (at the very least on Python 3.7, where it was improved) properly adjust the interpreter state including GIL and threads after a fork. Furthermore, whenever a process hangs in the state, it still has lots of open file descriptors (all those copied from the parent). This means that close_open_fs has not been called yet, and this is one of the next things that the child process would do.

Possible Solutions

I tried ensuring that the GIL is taken before the clone and not released until after the call to PyOS_AfterFork_Child by calling ctypes.pythonapi.PyGILState_Ensure() before the clone, but this had no effect (although it should according to my understanding). I also attempted to let ctypes avoid releasing the GIL by changing the callback type to a PYFUNCTYPE, but again no change.

Maybe there is some other way of ensuring that the GIL is not relased in this piece of code?

I guess it would certainly work if we call PyOS_BeforeFork, clone, and PyOS_AfterFork_Child from C code (in a small C library), because then the GIL would not be touched in between. However, this would mean giving up on Python being a pure-Python tool, which would make it harder for people to use (and the library would even need at least two versions, for Python < 3.7 and Python 3.7).

Maybe we can disable gil_drop_request while we are about to fork?

Setting a high time for sys.setswitchinterval would mean that the chance that gil_drop_request is set becomes practically zero. This seems to work, but the performance impact on BenchExec is unclear (though probably it should make BenchExec at most less responsive, which is not so important, and might even increase throughput).

As a workaround, we could wait for a response from the child in the parent, and if the child does nothing for a few seconds we could kill the child and try again.

[PhilippWendler]

We added the proposed workaround using sys.setswitchinterval and for Linux x86_64 we have a proper fix: We generate a function with pure machine code dynamically and use that one as callback, so no Python code gets executed before PyOS_AfterFork_Child() is called.

Summary of the current solution:

• For Linux x86_64 we use generated native code in the callback (proper fix).
• On other architectures we use sys.setswitchinterval() as workaround in benchexec.
• Direct users of ContainerExecutor/RunExecutor are fine in practice if they use few threads. This includes runexec and containerexec because both use few threads.
• Direct users of ContainerExecutor/RunExecutor that use many threads on a different architecture might want to use sys.setswitchinterval() themselves or contribute a native-code callback for their architecture.