p.lastIndex is not sequential on event watches

[freman]

consul version for both Client and Server

Client: Consul v0.9.2
Server: Consul v0.9.2

consul info for both Client and Server

Server:

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease =
	revision = 75ca2ca
	version = 0.9.2
consul:
	bootstrap = false
	known_datacenters = 1
	leader = true
	leader_addr = 127.0.0.1:8300
	server = true
raft:
	applied_index = 196
	commit_index = 196
	fsm_pending = 0
	last_contact = 0
	last_log_index = 196
	last_log_term = 2
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
	latest_configuration_index = 1
	num_peers = 0
	protocol_version = 2
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 2
runtime:
	arch = amd64
	cpu_count = 4
	goroutines = 70
	max_procs = 4
	os = linux
	version = go1.8.3
serf_lan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 1185
	event_time = 1186
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1

Operating system and Environment details

MacOS, Ubuntu, Docker

Description of the Issue (and unexpected/desired result)

Using the go api for watches

	"github.com/hashicorp/consul/api"
	"github.com/hashicorp/consul/watch"

Watching events can result in more than one events being returned
After a bit of digging I've found that it seems to be that the index's aren't sequential

watch.eventWatch calls which calls event.List which loads a lastindex from the meta that is returned to plan.Run

		// If the index is unchanged do nothing
		if index == p.lastIndex {
			continue
		}

		// Update the index, look for change
		oldIndex := p.lastIndex
		p.lastIndex = index
		if oldIndex != 0 && reflect.DeepEqual(p.lastResult, result) {
			continue
		}
// This if is the problem, it assumes sequential indexing...
		// if p.lastIndex < oldIndex {
		// 	p.lastIndex = 0
		// }

Reproduction steps

I simply wrote a tiny go client

package main

import (
	"fmt"

	"github.com/hashicorp/consul/api"
	"github.com/hashicorp/consul/watch"
)

func main() {
	w, _ := watch.Parse(map[string]interface{}{
		"type": "event",
		"name": "deploy-router",
	})
	w.Handler = func(a uint64, b interface{}) {
		if list, ok := b.([]*api.UserEvent); ok {
			fmt.Println("got", len(list))
		}
	}
	w.Run("localhost:8500")
}

and ran consul in docker, sending events in.

When I comment out that if it behaves as expected only returning one event

[slackpad]

Closing this as a duplicate and linking it to #3742 which has a more up-to-date discussion.