Topic Operation fails with "Failed to sync topics. Won't retry sync. TOO_MANY_SUBSCRIBERS" and won't callback

[Zachary625]

https://github.com/firebase/firebase-android-sdk/blame/dc3082560d5f556560e534dc739e048e4d4dae95/firebase-messaging/src/main/java/com/google/firebase/messaging/TopicsSyncTask.java#L108

TL;DR: Subscribe/Unsubscribe won't callback with TOO_MANY_SUBSCRIBERS, seeking explanation & suggestions

Environment:
Unity 2018.4.11f1
Firebase Unity SDK 11.8.1 using Andorid BoM 32.7.4, containing messaging-23.4.1.

First I thought something was wrong with our integration since there're lots of components in Firebase Unity SDK and Androd/Unity/Cpp SDK Guides to reference, then I found out that even calling Subscribe/Unsubscirbe with Java API, this error would still yield the TopicSubscriber in a halted state, with no continuation on follow up tasks unless a new invokation was made, in which case the callback would trigger with a false success result.

Perhaps this is intende behaviour? Any suggestion on how to handle this behaviour, perhaps with a timeout on callbacks?

Any help is appreciated! Thx!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[lehcar09]

Hi @Zachary625, thank you for reaching out. For context, based on our documentation,

Topic messaging supports unlimited subscriptions for each topic. However, FCM enforces limits in these areas:

• One app instance can be subscribed to no more than 2000 topics.
• If you are using batch import to subscribe app instances, each request is limited to 1000 app instances.
• The frequency of new subscriptions is rate-limited per project. If you send too many subscription requests in a short period of time, FCM servers will respond with a 429 RESOURCE_EXHAUSTED ("quota exceeded") response. Retry with exponential backoff.

I tried reproduce the issue, however, I did not encounter the issue. Can you share a code snippet/ MCVE to help us investigate the issue? Thanks!

[Zachary625]

Hi @Zachary625, thank you for reaching out. For context, based on our documentation,

Topic messaging supports unlimited subscriptions for each topic. However, FCM enforces limits in these areas:

• One app instance can be subscribed to no more than 2000 topics.
• If you are using batch import to subscribe app instances, each request is limited to 1000 app instances.
• The frequency of new subscriptions is rate-limited per project. If you send too many subscription requests in a short period of time, FCM servers will respond with a 429 RESOURCE_EXHAUSTED ("quota exceeded") response. Retry with exponential backoff.

I tried reproduce the issue, however, I did not encounter the issue. Can you share a code snippet/ MCVE to help us investigate the issue? Thanks!

Hi Thanks for your attentions, code snippet is just one call to unsubscribe/subscribe a topic on Android, but the required condition to trigger the issue is reaching above the 3000QPS of subscribe/unsubscribe per project (in which case the LogCat would err with aforementioned message), our project so happens is encountering this situation and is working on a thorough solution, but since we operate our topics and tokens purely on client side it would be a bit tricky if we don't have the callbacks for certainty.

[lehcar09]

Thank you for the additional information @Zachary625. I have raised this issue to our engineers. I'll get back to you once I got their feedback. Hang tight!

[gsiddh]

@Zachary625, this issue is missing code snippet on how you are invoking the subscribe call.

So, just to make sure, are you adding listener to your subscribe invocation. An example is shown in our integration guide: https://firebase.google.com/docs/cloud-messaging/android/topic-messaging#subscribe_the_client_app_to_a_topic

You can also add a on failure listener. Would that help?

[Zachary625]

@Zachary625, this issue is missing code snippet on how you are invoking the subscribe call.

So, just to make sure, are you adding listener to your subscribe invocation. An example is shown in our integration guide: https://firebase.google.com/docs/cloud-messaging/android/topic-messaging#subscribe_the_client_app_to_a_topic

You can also add a on failure listener. Would that help?

This is our actual implementation in C#, confirmed issue exists in il2pp builds of Unity 2018.4.11f1, using Firebase Unity SDK 8.0.0 & 11.8.1(Managed DLL recompiled to target .NET 3.5)

Firebase.Messaging.FirebaseMessaging.SubscribeAsync(topic).ContinueWith((task) => {
    Debugger.Log(string.Format("@FirebaseManager.SubscribeTopic({0}): completed: {1}, cancelled: {2}, faulted: {3}",
            topic, task.IsCompleted, task.IsCanceled, task.IsFaulted));
    // ...other logic
});

This is attempted but failed workaround using Java with same project using Firebase Unity SDK 11.8.1

Task<Void> task = FirebaseMessaging.getInstance().subscribeToTopic(topic);
_SubscribeTasks.put(topic, task);
task.continueWith(new Continuation<Void, Object>() {
    @Override
    public Object then(@NonNull Task<Void> task) throws Exception {
        _SubscribeTasks.remove(topic);
        TopicResult tr = new TopicResult();
        tr.Topic = topic;
        tr.Completed = task.isComplete();
        tr.Cancelled = task.isCanceled();
        tr.Failed = !task.isSuccessful();
        if(task.getException() != null) {
            tr.Exception = task.getException().getMessage();
        }
        MainActivity.SendToUnity(_UnitySendMessageGameObjectName, "OnSubscribedTopic", new Gson().toJson(tr));
        return null;
    }
});

Alternatively but still failed:

FirebaseMessaging.getInstance().subscribeToTopic(topic)
        .addOnCompleteListener(new OnCompleteListener<Void>() {
            @Override
            public void onComplete(@NonNull Task<Void> task) {
                TopicResult result = new TopicResult();
                result.Topic = topic;
                result.Completed = true;
                if(task.getException() != null) {
                    result.Exception = task.getException().getMessage();
                }
                MainActivity.SendToUnity(_UnitySendMessageGameObjectName, "OnSubscribedTopic", new Gson().toJson(result));
            }
        })
        .addOnCanceledListener(new OnCanceledListener() {
            @Override
            public void onCanceled() {
                TopicResult result = new TopicResult();
                result.Topic = topic;
                result.Cancelled = true;
                MainActivity.SendToUnity(_UnitySendMessageGameObjectName, "OnSubscribedTopic", new Gson().toJson(result));
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                TopicResult result = new TopicResult();
                result.Topic = topic;
                result.Failed = true;
                if(e != null) {
                    result.Exception = e.getMessage();
                }
                MainActivity.SendToUnity(_UnitySendMessageGameObjectName, "OnSubscribedTopic", new Gson().toJson(result));
            }
        })
        .addOnSuccessListener(new OnSuccessListener<Void>() {
            @Override
            public void onSuccess(Void unused) {
                TopicResult result = new TopicResult();
                result.Topic = topic;
                result.Completed = true;
                MainActivity.SendToUnity(_UnitySendMessageGameObjectName, "OnSubscribedTopic", new Gson().toJson(result));
            }
        });

Hope these can be useful, I'm new to java programming using gms Tasks so correct me if I'm wrong. Again thank you very much for the effort!

[Zachary625]

Also, on iOS, same TOO_MANY_SUBSCRIBERS failures can callback with System.AggregateException can be detected.

[gsiddh]

Ack @Zachary625, I am raising this internally for further investigation. Will respond back once I have more.

[gsiddh]

@Zachary625 I think you might be running into the 2K topic subscriptions per app instance limit as mentioned here: https://firebase.google.com/docs/cloud-messaging/android/topic-messaging

This is different from the rate of topic subscription request limit.

Can you double check if it is the former? If you provide me the fcm token of the app instance where you hit the limit, I can take a look in our logs / databases.

[Zachary625]

This here says one app instance can at most subscribe to 2000 topics right? We actually combined 4 properties into one meta property is used as a topic, one installation can have less than a dozen of possible topics to subscribe to, and if one instance is basically one installation of Firebase on client's device, 2000 is almost impossible to reach (if duplicate subscriptions from one instance to one topic is ignored, which was naturally assumed by us).
Example token of my own dev android device being:

ew3CtzG8T5WM32GaJVg7GG:APA91bFTiWRU-I3Acc0ZThmb_u1GG1OQB11FDDhNv6xh3T5cYUA066tPEU8iBaWwR24O-YEff9N4Org4EnEiXgiXRztMI8yBepi38H-lgdtY7lP8cBxTKm_NMVeO_JQiQeEqZGk8Zn2I

This token is for my private testing purposes, generated and last verified notifiable by subscribed topic on June 20th, 2024, feel free to message me about sensitive operations or inquiries, thanks for your attention and effort!