vine: check the availability of staged inputs

[JinZhou5042]

Proposed Changes

Fix #3993

So the problem was that when a worker crashes when sending back permanent outputs, all children tasks fail because of input missing.

It turned out that the return value of retrieve_output and vine_manager_get_output_files are not correctly received and handled.

In the original version, result is initialized as vine_result_code_t result = VINE_SUCCESS; which is 0 in integer:

typedef enum {
	VINE_SUCCESS = 0,
	VINE_WORKER_FAILURE,
	VINE_APP_FAILURE,
	VINE_MGR_FAILURE,
	VINE_END_OF_LIST,
} vine_result_code_t;

And result &= xxx will never change its initial value, this results in the manager assuming that the outputs have always been successfully retrieved.

I was able to reproduce the stated error and with these changes that issue disappeared.

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[dthain]

Hmm, I don't think this is the right solution.

Regular files should not be "lost" because they should exist at the manager once the original is done.

The problem is evident in the log describing the problem: an output transfer is interrupted, and the corresponding task is (incorrectly) declared as complete.

The correct solution is for that original task to go back into the queue.

[JinZhou5042]

Thanks for the clarification, this sounds to be a more feasible way.

[JinZhou5042]

@dthain Thanks for helping me find this bug!

[dthain]

Aha, that definitely looks like a bug.
Probably left over from some old code that used the &= pattern.
Good catch!

[dthain]

@btovar can you double check this key bit of logic?

[btovar]

@JinZhou5042 nice catch!