Handle ipfs can't get lock

[7yl4r]

ipfs will raise ipfs/kubo#6363 when two processes are trying to concurrently use ipfs - even when we are just using it to hash a file.

get_hash could catch this exception and wait for ipfs to be ready.

example output from this error:

[2019-08-02 03:07:39,508] {bash_operator.py:123} INFO - imars_etl.imars_etl.cli: INFO     === IMaRS Extract-Transform-Load Tool v0.12.0 ===
[2019-08-02 03:07:39,512] {bash_operator.py:123} INFO - imars_etl.imars_etl.cli: INFO     =================================================
[2019-08-02 03:07:40,079] {bash_operator.py:123} INFO - Error: cannot acquire lock: Lock FcntlFlock of /home/airflow/.ipfs/repo.lock failed: resource temporarily unavailable
[2019-08-02 03:07:40,101] {bash_operator.py:123} INFO - Traceback (most recent call last):
[2019-08-02 03:07:40,101] {bash_operator.py:123} INFO -   File "/usr/local/bin/imars-etl", line 11, in <module>
[2019-08-02 03:07:40,101] {bash_operator.py:123} INFO -     load_entry_point('imars-etl', 'console_scripts', 'imars-etl')()
[2019-08-02 03:07:40,101] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/__main__.py", line 9, in _main
[2019-08-02 03:07:40,101] {bash_operator.py:123} INFO -     main(sys.argv[1:])
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/cli.py", line 53, in main
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -     result = fn(**vars(args))
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/Load.py", line 31, in load
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -     **validate_args(kwargs, DEFAULTS=LOAD_DEFAULTS)
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/validate_args.py", line 50, in validate_args
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -     raise_cannot_constrain=False
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/metadata_constraints.py", line 105, in _ensure_constistent_metadata
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -     metad, *constraint, raise_cannot_constrain=raise_cannot_constrain
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/constrain_dict.py", line 37, in relation
[2019-08-02 03:07:40,102] {bash_operator.py:123} INFO -     val = getter(*in_vals)
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/get_hash.py", line 5, in get_hash
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -     return _gethash_ipfs(filepath)
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -   File "/opt/imars_etl/imars_etl/Load/get_hash.py", line 17, in _gethash_ipfs
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -     filepath
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -   File "/usr/lib64/python3.6/subprocess.py", line 336, in check_output
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -     **kwargs).stdout
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -   File "/usr/lib64/python3.6/subprocess.py", line 418, in run
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO -     output=stdout, stderr=stderr)
[2019-08-02 03:07:40,103] {bash_operator.py:123} INFO - subprocess.CalledProcessError: Command '['ipfs', 'add', '-Q', '-n', 'wv2_unzipped/058855092010_01_003/058855092010_01/058855092010_01_P001_MUL/17FEB04161530-M1BS-058855092010_01_P001.NTF']' returned non-zero exit status 1.
[2019-08-02 03:07:40,478] {bash_operator.py:127} INFO - Command exited with return code 123
[2019-08-02 03:07:42,822] {models.py:1788} ERROR - Bash command failed