Load balance traffic to N>1 server nodes

[kamran-m]

Currently the logic of tchannel-go to choose a peer is

for i := 0; i < size; i++ {
	popped := l.peerHeap.popPeer()
	if canChoosePeer(popped.HostPort()) {
		ps = popped
		break
	}
	psPopList = append(psPopList, popped)
}
...

Which means, the client node will only talk to one server node, the top chosen one, unless there is a change in scores. This means that even if we use the HRW scorer there will be a traffic imbalance, specially for cases where the number of client and server nodes are low.

I would like to introduce peerConnectionCount (default=1), so that we can pick the top peerConnectionCount nodes and do a random load balancing among those instead of talking to only one node. This should reduce the imbalance of traffic to server nodes.

In future, I think it's better if the whole logic of score calculator, peer heap and the new peer connection count, move to something like PeerSelectionStrategy, so other strategies would be easily added.

cc @prashantv @Raynos