Set Index hashtree sync_stop timeout to infinity

[systream]

When stopping a riak node, riak_kv_vnode terminates and riak_kv_index_hashtree:sync_stop/1 called.
This may end up in crash if it could not finish in 5 sec:

CRASH REPORT Process <0.1193.0> with 1 neighbours exited with reason: {timeout,{gen_server,call,[<0.1248.0>,stop]}} in gen_fsm:terminate/7 line 600


I think this could created a situation where a normal riak stop can cause this error:

CRASH REPORT Process <0.3158.0> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"Corruption: truncated record at end of file"}} in hashtree:new_segment_store/2 line 728 in gen_server:init_it/6 line 328

[martinsumner]

I think there are plenty of situations where 5s could be too short, so I agree about changing this. However, do we want to allow this to try forever, especially as it can recover from the crash?

My gut feeling is to use 60s not infinity. That is just an immediate reaction though, happy to discuss.

I will review that area of the code in more detail later this week, but I will make sure whatever we agree to change goes into the 3.0.9 release which should be out in the next 2 weeks.

[systream]

Agree, 60 sec could be enough (or at least helps a lot) and we won't run the risk what if it hangs somehow.

Thanks!