Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental v2/channel and consumer group #1661

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from

Commits on Mar 6, 2024

  1. Add channel groups

    Add a client side channel group that listens for active channels and
    subscribes/unsubscribes as the set of active changes.
    
    Add a consumer group based on presence set
    
    Add typing information for ChannelGroups
    
    Fix tests for v2
    
    ChannelGroups is a default realtime module
    
    types: add channel group typings
    
    consumergroup: use modulo-based hashing scheme
    
    The `hashring` package is node-only as it depends on the native `crypto`
    package. Replaced with a simple modulo hash scheme for now.
    
    Fixes the case where the channel is already attached and the channel is
    obtained with new rewind options that require a re-attach.
    
    Updates the consumer group partitioning test to more robustly assert
    that channels are partitioned across consumers.
    
    channelgroup: make get sync, subscribe async
    
    This matches the pattern used by Channels, which is sync to obtain the
    channel and async on subscribe in order to await attachment. In the
    channel group case, awaiting the subscribe awaits the joining of the
    consumer group.
    
    consumergroup: make consumerId required field
    
    consumergroup: make hashring a required field
    
    format: apply prettier formatting rules
    
    jsdoc: add consumer group docs
    
    channelgroups: add active channel name option
    
    channelgroup: add explicit join method
    
    Exposes the join method on the channel group, which is analogous to
    attach on the channel. Uses this in tests for more robust assertions.
    
    Additionally avoids re-attaching to the consumer group channel if
    already attached, and obtains presence membership synchronously in the
    join.
    
    consumergroup: use subscribe over on
    
    The `on` method was not reliable across clients, despite being
    documented in
    https://ably.com/docs/presence-occupancy/presence?lang=java#synced, so
    use subscribe instead.
    
    channelgroup: fix assigned channel processing
    
    We need to keep the total set of active channels around as updating the
    assigned channel set when the membership changes requires computing the
    new assignments from the complete channel set, not the previously set of
    assigned channels.
    
    consumergroup: include consumerId in logs
    
    consumergroups: add test for consumer group resize
    
    test: remove explicit join from test
    
    lint: apply formatting and cleanup
    
    test: replace var with let
    
    test: remove unnecessary outer try-catch
    
    test: add prefix to channels
    
    Avoid channel name collisions causing tests to fail from concurrent test
    runs in CI.
    
    channelgroup: detach from channel on un-assignment
    
    channelgroup: add unsubscribe listener method
    
    test: fix rebalance test waits for consumers
    
    channelgroup: add leave method
    
    test: remove dangling console logs
    
    test: test consumer group scale down event
    
    test: prefix consumer group channel
    
    Similar to the active channel, we need to avoid conflicts.
    
    consumergroup: fix current member tracking
    
    We store the current active set of members in the hashring.
    
    test: fix done condition w/ at-least-once delivery
    
    Messages can be delivered more than once during a consumer group
    rescaling event, so deduplicate the results when checking the end
    condition.
    
    channelgroups: use Utils.inspectError in logs
    
    channelgroups: do not share channel object
    
    The Channels object used by the ChannelGroup for internal channel
    operations no longer shares the same object exposed on the client via
    the .channels() method. This is to ensure that independent usage of an
    individual channel that happens to be included in a channel group is not
    impacted by its usage in the channel group.
    
    test: tidy up leave test
    
    Now that we can correctly handle a channel group and channel being used
    independently from the same client, this tidies up the leave test to
    remove the additional client previously needed.
    
    test: rename waitForConsumers for clarity
    
    test: rename waitForConsumers for clarity
    
    channelgroups: do not share channel object
    
    The Channels object used by the ChannelGroup for internal channel
    operations no longer shares the same object exposed on the client via
    the .channels() method. This is to ensure that independent usage of an
    individual channel that happens to be included in a channel group is not
    impacted by its usage in the channel group.
    
    channelgroups: add module integration
    
    modules: update ChannelGroups module definitions
    
    channelgroup: add temp rewind channel group option
    
    channelgroup: unsubscribe channel after timeout
    
    In order to avoid keeping the channel alive, we add a configurable
    timeout after which the channel will be unsubscribed if no messages are
    received. This is to avoid keeping the channel active. This can lead to
    missed messages if the a message is published after the client
    unsubscribes and before the channel becomes inactive. This is an
    acceptable edge case for the client-side simulation, especially with the
    default 1h timeout.
    
    deps: remove unused hashring types pkg
    
    utils: remove arrIndexOf polyfill
    
    consumergroup: rename hashring to locator
    
    test: use async style tests for channel groups
    
    Replaces the use of the `done()` callback with an async function style
    test.
    
    This allows us to await channel publish results and more easily handle
    race conditions in tests.
    
    channelgroup: use qualifier options
    
    Previously we relied on a new BaseRealtime instance with it's own
    Channels object to separate usage of channels in the ChannelGroup from
    independent external usage of those channels from the regular
    client.channels.get() method. This led to various problems with shared
    Auth state such as nonces in token requests which caused connections to
    terminate and tests to fail.
    
    A simpler solution is to avoid creating a new client instance and
    instead share the Channel pool, but force the library to treat channels
    used from the ChannelGroup independently (with their own attachment) by
    setting dummy options in the qualifier, which is used as the key in the
    channel map.
    
    This implementation does not support channels in the channel group which
    already have a qualifier. This is acceptable for the experimental
    client-side simulation of the feature.
    zknill authored and mschristensen committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    bcf965f View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Configuration menu
    Copy the full SHA
    dc3fb7e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1ccee4a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fb78dd7 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. channelgroup: deduplicate presence set by clientId

    Each client will be entered in the presence set once per connection, so
    we need to deduplicate the presence set by clientId.
    mschristensen committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    d3b7272 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Configuration menu
    Copy the full SHA
    3a5caae View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. Merge pull request #1725 from ably/experimental/channel-group-event-e…

    …mitter
    
    channelgroup: emit channel assigned/active events
    mschristensen authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    05bd66e View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Configuration menu
    Copy the full SHA
    2dab4c7 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Merge pull request #1728 from ably/experimental/channel-group-member-…

    …events
    
    consumergroup: emit membership updated event
    mschristensen authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    25bdfc1 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. channelgroup: only emit active channels on changed

    Since the set of active channels are surfaced on $ably:active is a
    superset of those specified by a particular channel group expression,
    this commit ensures we only emit an active.updated event when the set of
    channels matching the consumer group expression have actually changed.
    mschristensen committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    029f5f5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7462806 View commit details
    Browse the repository at this point in the history