Add the ability to handle newly subscribed peers

[aschmahmann]

Per conversations with @vyzo @raulk and @Stebalien we are refactoring the content of #171 which does 2 things.

1. Gives developers control over what messages are broadcast
2. Gives developers an "OnJoin" method that can specify an action to do when first connecting to a peer (e.g. send a specific message, utilize an external protocol, etc.)

Instead of doing this by making routers extensible, we will be leaving router extensibility for a later date and solve the issues above by:

1. Using stateful validators + application layer publishing to effectively control which messages are broadcast. This can happen outside of the go-libp2p-pubsub package.
2. Adding a mechanism for PubSub to expose when it connects to a new peer

This PR deals with the latter.

[vyzo]

This looks good. Shouldn't we also notify when peers leave the topic however?

[vyzo]

An issue here is that if we subscribe after peers have been discovered, there will be no notifications.
I think we might want to emit notifications for all known peers when we first subscribe.

[aschmahmann]

While you could already get the existing peers with the FindPeers command, I think you're right that this would likely be useful for many developers. I'll likely just make the buffered channel large enough to fill with the initial peers and fill it up on initialization.

[aschmahmann]

This looks good. Shouldn't we also notify when peers leave the topic however?

@vyzo That's fine by me. Any thoughts on how important receiving all the leave messages is? In particular, while joins might be slow enough that concurrent joins will not get dropped due to channel buffer overflow disconnecting from the internet might instead lead to mass connectivity dropping and therefore some missed NextPeerLeave events.

Also, wanted to confirm that blacklisting based disconnects should count towards Leave events also.

[vyzo]

There is a race to read join and leave notifications, as they are emitted in different channels.
I think we need a single channel for both notifications and use a struct with a field that specifies the notification type.

[vyzo]

this is looking good, just a few minor points.

[vyzo]

only thing remaining to fix is the enum, other than that it looks good.

[aschmahmann]

Latest idea for how to do this:

We effectively want an infinite length buffer so that no Joins/Leaves are lost. However, we can be slightly efficient by ignoring when the same peer Joins and Leaves before we even ask for a notification.

Strategy:
Have a channel that receives updates, if the channel is full empty it into the buffer which is a map protected by a mutex.
When asked for an update we pull from the buffer if it's not empty, otherwise we pull from the channel.

There are some side points around how to get this right and look clean that I could definitely use some review on.

[Stebalien]

Have a channel that receives updates, if the channel is full empty it into the buffer which is a map protected by a mutex.
When asked for an update we pull from the buffer if it's not empty, otherwise we pull from the channel.

Maybe? I'm concerned about races reading the channel. Someone needs to drain the channel and we can't have both the subscriber and pubsub do this (or risk out of order events).

If we want to go with this approach, we could just put everything into the map up front (and use a channel for signaling).

[Stebalien]

This is racy. We could be in the middle of draining this queue into the map and read an leave event before the join (now in the map).

[Stebalien]

(sorry, we missed a case)

[aschmahmann]

@Stebalien we thinking this works? I should only need to push a signal if there are events in the log right?

[Stebalien]

LGTM.

[Stebalien]

[vyzo]

alright, let's merge this.