WIP: Notifications subscription

[mxstbr]

Haven't managed the upgrade to the new (and unreleased) subscription-transport-ws yet, ran into some weird bugs I couldn't resolve. (probably just me being tired)

I've reset that and will take another stab at it fresh and clean first thing tomorrow to get that finished so I can look at DM notifications.

[mxstbr]

Dug into this for a few hours now, still blocked.

It turns out while the new subscriptions-transport-ws and graphql-subscriptions together work beautifully, the new API that they use requires graphql@^0.10.0 which was just released. That's fine, I thought, I'll just upgrade it.

Turns out graphql-server-express includes graphql@0.9.x in it's peerDependencies, so no matter which version I try to install our server resorts back to 0.9.x. This means we're effectively blocked until graphql-server-express releases a new version with the peerDependencies upgraded.

I reached out in the Slack channel and hope to get a response and release as soon as SF wakes up.

[superbryntendo]

how problematic do you think it would be to fork and change the dependency for now?

[brianlovin]

Thanks for digging into this man, hopefully we can get resolution soon (just a heads up tomorrow is a US holiday so those folks might be offline, not sure :/)

[mxstbr]

PR to update was merged! apollographql/apollo-server#407

Just waiting for npm publish now 👌

[mxstbr]

apollo-upload-client still depends on an old version of graphql which breaks the server, PR submitted and reached out to the author: jaydenseric/apollo-upload-client#12

[mxstbr]

Fixed that, fixed some more bugs, upgraded some more things and now message subscriptions are working again 🎉

Onwards to the actual task of this PR: notifications subscriptions

[mxstbr]

W00t you can now listen to new notifications! 🎉

This turned out to be a bit harder than originally anticipated because to listen to notifications of a user we need to, of course, know who the user is—the issue is that when connecting over websockets (of course, again) none of the HTTP express middlewares run so we have no clue if a user is authenticated or not. Essentially, I had to do all of that manually to then pass the current user (or not) into the GraphQL context but it's done now so you can listen to new notifications for yourself! (see commit cb9e608 for that part)

Now going to do some cleanup and then the frontend side of things 🔥

[brianlovin]

@mxstbr - just pushed the initial setup for usersNotifications join table; I am going to walk my pup and head to Bryn's. After that I'll handle the following logic (unless you were planning to?):

1. On thread creation, generate usersNotifications for users in that channel
2. On message sent, generate usersNotifications for participants on that thread

I'm not really sure how to do an initial population of this table through the generators / it'll be a bit of work :)

[brianlovin]

Here's my proposed changes to notifications for right now that will minimally impact our time to launch (~1 day?) but will dramatically reduce our debt in the future.

I propose we think of all notifications as the following story:

(An) Actor(s) triggers an Event which impacts an Entity which notifies Recipients.

Examples:

• bryn created a thread in channel x which notifies members of channel x
• max posted a message in thread a which notifies creator and participants of thread a
• brian changed the privacy of channel y which notifies members of channel y
• jon and jack reacted to a message which notifies message sender

We store this notification-level data on the notifications table. This will look like this:

{
  id: id
  actors: [userId]
  event: eventType_eventValue (e.g. thread_created, channel_made_private)
  entity {
    type: enum string
    id: id
  }
  attachments: [ Attachments ]
}

Attachments can be anything, which will allow us a ton of flexibility to put any arbitrary data structure in a notification that we want. E.g. imagine you get a notification that a community owner just created a new channel; what if we could have an attachment like:

{
  type: channelCreated
  data: channelId
}

and then on the client side we can iterate through the attachments to render a button in the notification to join that channel!

In addition, we create a join table usersNotifications which stores metadata related to a specific user's relationship with a notification. It will look like this:

{
  id: id
  userId: id
  notificationId: id
  isSeen: boolean
  isRead: boolean
}

What this looks like in practice

Bryn creates a new thread in a channel where I am a member.
Actor: Bryn
Event: thread_created
Entity:

• type: 'thread'
• id: threadId
  Recipients: All members of the channel where the thread entity was posted
  Attachments: Thread title + excerpt of thread body

So the resulting objects in the database are:

Notification:

{
  id: '1234567890'
  actors: [ bryns-unique-userid ]
  event: 'thread_created'
  entity {
    type: 'channel'
    id: channel-id-for-new-thread
  }
  attachments [
    {
      type: 'thread'
      data {
        title
        excerpt
      }
    }
  ]
}

UsersNotifications (generated for all eligible recipients):

{
  id: uuid
  userId: some-recipient-uid
  notificationId: '1234567890'
  isSeen: false
  isRead: false
}

Backend Pipeline

An event occurs on the clientside - let's keep the same example of a thread being created. That thread gets stored in the db which will trigger the following chain of events:

1. Thread gets stored and the resulting thread object is returned to the client and piped into our notifications pipeline with an object like:

{
  type: 'thread_created'
  entity {
    type: 'channel'
    id: channel-id-for-new-thread
  }
}

2. The first step of the notification pipeline is to figure out who should be considered for receiving a notification. In this case we would take the entity's channel Id, get all the members of that community, and construct an array of userIds. In the future we might also have channelSettings that would be checked here...not sure the exact use cases, but...future option value :)

3. From that array of user Ids we will create the records in the join table and finalize the published notification. In the future, this is where we will be able to retrieve usersSettings from a separate table and determine if this user wants to see the notification. We will also be able to check if the user has permission to see the notification (e.g. a user shouldn't get notified about new messages in a thread, where they were blocked from the channel that thread was posted in).

4. Once we've ended the pipeline, we'll just save the notification and using gql subscriptions all the necessary recipients will know about the new event. In the future, however, we will put any delivery settings here, for example if the user wanted an email, we can send that from this step.

GraphQL

The nice thing about this proposed setup is that we have much more powerful filtering options. We can get events by entity (e.g. show me all notifications for the channels I own). We can also do cool things like: get me all notifications for new threads that I saw but didn't click on (e.g. filtering by seen or unseen, as well as by notification type).

Notification bundling

We can re-use a lot of the same bundling logic we already have - basically loop through a user's notifications, group them by the entity type + id, and merge the actors from those two objects. Based on the merges that happen in here, we can modify any strings needed (e.g. multi-actor: Brian and Max posted new threads in Channel X; single-actor: Brian posted a new thread in Channel X, Thread Title)

Attachments

Attachments could be anything in the future, as noted above. In the short term we'll probably have attachments for:

• media messages
• thread previews

In the medium term we could have attachments for:

• users who requested to join a private channel i own (with actions to block/approve)
• channels created in communities i'm a member of (with actions to join channel)

In the long term if we have polls, or events, or video streaming, or AMAs, or anything like that, we could handle those here.

Some examples:

Thread preview

{
 attachmentType: 'thread'
 data: {
    title: Title
    excerpt: Excerpt
  }
}

Media message

{
  attachmentType: 'media'
  data: {
    url: fileUrl.png
  }
}

Pending User in Private Channel You Own

{
  attachmentType: 'pendingChannelUser'
  data: {
    userId: id
    channelId:  id
  }
}

Of course with gql we can resolve a lot of ids and return full channel/user objects :)

Proposed notifications for v2 launch

1. New messages on a thread you created or are a participant of

Proposed fast follow notifications

1. New thread posted in channels you are an owner or member of
2. New channel created in community you are owner or member of
3. Thread you created was deleted by an admin/moderator
4. Thread you created was frozen by an admin/moderator
5. User requested to join pending channel you own
6. My request to join a private channel was approved by channel owner
7. etc.

[brianlovin]

I think we have a few paths to shipping:

1. If we decide that we want to go with the proposal above (or some variant of it) we can do one of two things:
   a. comment out the notifications stuff in the client for now and fast-follow the notifications stuff as soon as v2 launches
   b. delay the v2 launch until we properly get a better foundation in place for notifications
2. If we decide that we do not like the above proposal and want to ship as-is with v2, we need to do the following:
   a. finish notifications subscription (this branch)
   b. refactor the logic to write/read from the new join table (latest commits here)
   c. refactor the client side queries to get a user's notifications

I personally am a fan of 1A. V1 of Spectrum already has no notifications, so shipping v2 without notifications is still a huge win anyways. I think it will take 1-2 extra days of work to rework our current notifications logic to be a bit more general purpose based on the proposal above, which we could ship shortly after launching v2.

If we go with option 2, I still think it'll take several more hours of work to build + test, whether that's on max or I we can decide tonight.

Let's discuss when you're awake @mxstbr and @uberbryn

[mxstbr]

We agreed to go with 1A above and make this happen first thing after launch.

I've disabled notifications client-side and server-side temporarily. Merging this so it doesn't go out of date.