Support Tenanting

[gkinsman]

A topic topology that is common at JustEat is to have an SQS topic per tenant. For example, suppose we had the following setup:

Topic uk-simplemessage => Queue uk-simplemessage-component
Topic it-simplemessage => Queue it-simplemessage-component

Although JE have moved away from this pattern to global queues (i.e. no tenant name in the queue), it's possible that there's room for a feature in JustSaying to handle this case a bit more nicely when it is necessary.

Problems with this pattern today:

1. It's not possible to link a message type to more than one physical topic or queue. If I want to subscribe to every topic for a tenant, I'd need to create a new message type for each tenant to link to the queue name and handle those:

builder.ForTopic<UkSimpleMessage>(x => x.WithName("uk-simplemessage-component"))
builder.ForTopic<ItSimpleMessage>(x => x.WithName("it-simplemessage-component"))

2. Conversely, it's also not possible to choose a tenanted topic to publish to, as there is a 1:1 mapping of topic to message type.

publisher.Publish(myMessage) // Only one topic is associated to this publisher at the moment.

This problem could be generalised into the following:

I want to be able to subscribe to more than one topic for a particular message, and to choose a publisher dynamically at publish time.

Proposal 1: Templated topic and queue names

The idea here is that we could accept templates as topic names, and the user would provide options for those templates at the right time.

Subscribing

For example, for the subscriber example above, the user would pass a templated topic name with an object of the properties to create topics with:

var messageTemplateProperties = new { tenant = new [] {"uk, it"} };
builder.ForTopic<SimpleMessage>(x => x.WithTopicName("{tenant}-simplemessage", messageTemplateProperties))

JustSaying would then use the properties to create multiple topic names, and create topic subscriptions for each tenanted topic. This would work similarly to how Dapper accepts SQL parameters.

Publishing

On the publish side, the user would do something similar, but the template properties would only be passed in at publish time, as that's when the tenant is known:

builder.WithTopic<SimpleMessage>(x => x.WithName("{tenant}-simplemessage"))
...

var messageTemplateProperties = new { tenant  = "uk" } }; // Could come from ambient context, or an HTTP request
messagePublisher.Publish(new SimpleMessage(), messageTemplateProperties);

It would be an error to not pass in template properties if the topic or queue name was templated, and also to pass them if the queue/topic was not templated.

Benefits of this design

• This pattern should be generalisable to naming conventions, although forcing a template parameter on every publish would likely be an edge case. In reality most apps subscribe to a mix of tenanted and non-tenanted queues.
• Easy to debug via the interrogation system
• Potentially has uses beyond just tenants

Drawbacks

• Additional complexity in managing subscriptions and publications

Proposal 2: Subscribe multiple topics to single queue

Instead of creating a queue and subscription per topic as above, perhaps we could 'fan-in' here instead, and create a single queue that would be subscribed to multiple topics.

The user could subscribe to multiple topics while using the same queue:

builder.ForTopic<SimpleMessage>(x => x.WithTopicName("uk-simplemessage").WithName("simplemessage-component"))
builder.ForTopic<SimpleMessage>(x => x.WithTopicName("it-simplemessage").WithName("simplemessage-component"))

Benefits of this design

• It's simpler to implement than templated topics as there is fewer moving parts
• Conceptually simple - it's just a 'fan-in'

Drawbacks

• It solves our specific problem only, and doesn't really provide a lot of extensibility
• Doesn't solve the publisher problem - we'd still need a way to publish to different queues based on tenant.