Zendesk/support group_memberships only returning 100 records (pagination put in place by Zendesk recently)

[zbrak]

Environment

• Airbyte version: 0.32.8-alpha
• OS Version / Instance: GKE 1.22.8-gke.201
• Deployment: Google Kubernetes Engine
• Source Connector and version: Zendesk/support 0.2.10
• Destination Connector and version: BigQuery Denormalized type struct 0.1.11
• Step where error happened: Sync Job on group_memberships

Current Behavior

Only returns 100 records as pagination max is 100.

(I believe this change was made by Zendesk recently - I used to get hundreds-thousands records returned)

Expected Behavior

Full return of all group memberships.

Logs

logs-122576-0.txt

Steps to Reproduce

1. run current version Zendesk group_memberships where there are >100 group memberships

Are you willing to submit a PR?

Yes if we are able to identify how to adapt existing pagination within the zendesk/support connector to group_memberships

[marcosmarxm]

I'm able to reproduce issue using Zendesk Support 0.2.10

[marcosmarxm]

After apply same logic from other streams the sync start working with pagination:

class GroupMemberships(SourceZendeskSupportCursorPaginationStream):
    """GroupMemberships stream: https://developer.zendesk.com/api-reference/ticketing/groups/group_memberships/"""

    def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        next_page = self._parse_next_page_number(response)
        return next_page if next_page else None

    def request_params(
        self, stream_state: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None, **kwargs
    ) -> MutableMapping[str, Any]:
        params = {"page": 1, "per_page": self.page_size, "sort_by": "asc"}
        start_time = self.str2unixtime((stream_state or {}).get(self.cursor_field))
        params["start_time"] = start_time if start_time else self.str2unixtime(self._start_date)
        if next_page_token:
            params["page"] = next_page_token
        return params

[marcosmarxm]

After implementing the code: