[DEFECT] too many threads modifying variable in DistributedMemoryRunner

[joshua-cogliati-inl]

Thank you for the defect report

• I am using the latest version of RAVEN.
• I have read the Wiki.
• I have created a minimum, reproducible example
  that demonstrates the defect.

Defect Description

Basically, in JobHandler terminateJobs which modifies the run list ( __running ) on a different thread than cleanJobQueue and fillJobQueue, both of which assume that only the loop thread can modify the run list.

In DistributedMemoryRunner, isDone checks self.thread.finished, but kill calls del self.thread
So if isDone is called after del self.thread, then self.thread.finished will error out.

Traceback (most recent call last):
  File "/home/fred/miniconda3/envs/raven_libraries_heron_newer_ray_and_h5py/li
b/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/fred/miniconda3/envs/raven_libraries_heron_newer_ray_and_h5py/li
b/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/projects/raven_db/fred/raven/ravenframework/JobHandler.py", line 442,
 in startLoop
    self.cleanJobQueue()
  File "/projects/raven_db/fred/raven/ravenframework/JobHandler.py", line 922, in cleanJobQueue
    if run is not None and run.isDone():
  File "/projects/raven_db/fred/raven/ravenframework/Runners/DistributedMemoryRunner.py", line 69, in isDone
    if self.thread is None:
AttributeError: 'DistributedMemoryRunner' object has no attribute 'thread'

Steps to Reproduce

Run a long optimizer that uses terminateJobs.

Expected Behavior

Not to crash.

Screenshots and Input Files

No response

OS

Linux

OS Version

No response

Dependency Manager

CONDA

For Change Control Board: Issue Review

• Is it tagged with a type: defect or task?
• Is it tagged with a priority: critical, normal or minor?
• If it will impact requirements or requirements tests, is it tagged with requirements?
• If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
• Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure

• If the issue is a defect, is the defect fixed?
• If the issue is a defect, is the defect tested for in the regression test system? (If not explain why not.)
• If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
• If the issue is a defect, does it impact the latest release branch? If yes, is there any issue tagged with release (create if needed)?
• If the issue is being closed without a pull request, has an explanation of why it is being closed been provided?

[PaulTalbot-INL]

Approved to close via #1879 and/or #1883, noting that it's possible new issues may be discovered that will require re-addressing this issue.

[PaulTalbot-INL]

Also approved to close via #1899.