Fixes handling of stop channel and failed barrier attempts.

[slackpad]

There were two issues here. First, we needed to not exit when there was a timeout trying to write the barrier, because Raft might not step down, so we'd be left as the leader but having run all the step down actions.

Second, we didn't close over the stopCh correctly, so it was possible to nil that out and have the leaderLoop never exit. We close over it properly AND sequence the nil-ing of it AFTER the leaderLoop exits for good measure, so the code is more robust.

We also added a pre-poll before we wait in the leaderLoop, since the exit condition is mixed in with a bunch of other stuff, so if we wait a long time for the barrier we will have hit the interval and may get kind of stuck waiting for the select to pick the stopCh case (making another failed attempt at doing a barrier could cost us another 2 minutes, for example).

Fixes #3545

[magiconair]

This looks correct. I have three small nitpicks:

1. I'd call defer wg.Done out of habit in case the function becomes more complex
2. I'd name go func arg different (e.g. ch) to avoid accidental future confusion
3. I think writing this as a switch statement may help readability since it gets rid of some brackets

switch {
case isLeader:
	if stopCh != nil {
		s.logger.Printf("[ERR] consul: attempted to start the leader loop while running")
		continue
	}

	stopCh = make(chan struct{})
	wg.Add(1)
	go func(ch chan struct{}) {
		defer wg.Done()
		s.leaderLoop(ch)
	}(stopCh)
	s.logger.Printf("[INFO] consul: cluster leadership acquired")

default:
	if stopCh == nil {
		s.logger.Printf("[ERR] consul: attempted to stop the leader loop while not running")
		continue
	}

	s.logger.Printf("[DEBUG] consul: shutting down leader loop")
	close(stopCh)
	wg.Wait()
	stopCh = nil
	s.logger.Printf("[INFO] consul: cluster leadership lost")
}

[slackpad]

Thanks @magiconair these are all good suggestions. PTAL at the last push.

[preetapan]

LGTM