Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CbpStrategy accepting OBP params #517

Merged
merged 4 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,13 +132,31 @@ There are two ways to do pagination in the Zendesk API, **CBP (Cursor Based Pagi
The use of the correct pagination is encapsulated using the iterator pattern, which allows you to retrieve all resources in all pages, without having to deal with pagination at all:

```php
$ticketsIterator = $client->tickets()->iterator();
$iterator = $client->tickets()->iterator();

foreach ($ticketsIterator as $ticket) {
process($ticket); // Your implementation
foreach ($iterator as $ticket) {
echo($ticket->id . " ");
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now prefer code that is fully testable, without having to define process

}
```

If you want a specific sort order, please refer to the sorting section in the documentation ([Tickets, for example](https://developer.zendesk.com/api-reference/ticketing/tickets/tickets/#sorting)). If you are upgrading from OBP, please refer to [the upgrade guide](./UPGRADE_GUIDE.md).

##### Iterator with params example

```php
$params = ['my' => 'param1', 'extra' => 'param2'];
$iterator = $client->tickets()->iterator($params);

foreach ($iterator as $ticket) {
echo($ticket->id . " ");
}
```

* Change page size with: `$params = ['page[size]' => 5];`
* Change sorting with: `$params = ['sort' => '-updated_at'];`
* Refer to the docs for details, including allowed sort fields
* Combine everything: `$params = ['page[size]' => 2, 'sort' => 'updated_at', 'extra' => 'param'];`

#### Find All using CBP (fine)

If you still want use `findAll()`, until CBP becomes the default API response, you must explicitly request CBP responses by using the param `page[size]`.
Expand Down
90 changes: 90 additions & 0 deletions UPGRADE_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Upgrade guide
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a little duplication in the info below, but I think it's worth giving a TLDR summary.


## Useful links

* [Pagination](https://developer.zendesk.com/api-reference/introduction/pagination)
* [Ticketing sorting](https://developer.zendesk.com/api-reference/ticketing/tickets/tickets/#sorting)

## CBP Basics

**OBP (Offset Based Pagination)** is quite inefficient, and increasingly so the higher the page you fetch. Switching to **CBP (Cursor Based Pagination)** will improve your application performance. OBP will eventually be subject to limits.
Zendesk-NathanBellette marked this conversation as resolved.
Show resolved Hide resolved

When in OBP you request page 100, the DB will need to index all 100 pages of records before it can load the rows for the page you requested.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rephrase this? Maybe something like:

When using OBP, if you request page 100, the DB will need to index all records that proceed it before it can load the rows for the page you requested.


CBP works based on a cursors, so if the order is `id`, 10 elements per page and the last item on your page is 111, the response includes a link to the next page "next 10 elements after id 111", and only your page is indexed before the rows are fetched.

## API calls

URL example:

* OBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort_order=desc&sort_by=updated_at&per_page=2
* CBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort=-updated_at&page[size]=2

### CBP ordering

When moving from OBP to CBP sorting params change as well:

* From: `?sort_name=updated_at&sort_order=desc`
* To: `sort=-updated_at`

Ticket example:

* OBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort_order=desc&sort_by=updated_at&per_page=2
* CBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort=-updated_at&page[size]=2

However, the list of **attributes you can sort by might also change** with the pagination type:

https://developer.zendesk.com/api-reference/ticketing/tickets/tickets/#sorting

* OBP: `"assignee", "assignee.name", "created_at", "group", "id", "locale", "requester"`
* CBP: `"updated_at", "id", "status"`

Example:

* OBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort_order=desc&sort_by=assignee.name&per_page=2 `HTTP 200`, works
* CBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort=assignee.name&page[size]=2 `HTTP 400`

```json
{
"error": "InvalidPaginationParameter",
"description": "sort is not valid"
}
```

If this is your situation, **you need to change sorting order** to a supported one.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly change this to:

If this is your situation, you will need to change the sorting order to a supported one.


## The new iterator

Your best solution to implement CBP is to use the newly provided iterator on your resources, for example:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpickiest nitpick. I

The most elegant way to implement CBP....


```php
$params = ['my' => 'param1', 'extra' => 'param2'];
$iterator = $client->tickets()->iterator($params);

foreach ($iterator as $ticket) {
echo($ticket->id . " ");
}
```

This will choose the right type of pagination and adapt your parameters for pagination and ordering to work with CBP.

##### Iterator with params example

```php
$params = ['my' => 'param1', 'extra' => 'param2'];
$iterator = $client->tickets()->iterator($params);

foreach ($iterator as $ticket) {
echo($ticket->id . " ");
}
```

* Change page size with: `$params = ['page[size]' => 5];`
* Change sorting with: `$params = ['sort' => '-updated_at'];`
* Refer to the docs for details, including allowed sort fields
* Combine everything: `$params = ['page[size]' => 2, 'sort' => 'updated_at', 'extra' => 'param'];`


## Parallel requests

If you are fetching multiple pages in parallel using OBP, you need to refactor to a serial execution, and fetch one page at a time.
1 change: 0 additions & 1 deletion src/Zendesk/API/Traits/Resource/FindAll.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
namespace Zendesk\API\Traits\Resource;

use Zendesk\API\Exceptions\RouteException;
use Zendesk\API\Traits\Utility\Pagination\CbpStrategy;

trait FindAll
{
Expand Down
10 changes: 5 additions & 5 deletions src/Zendesk/API/Traits/Resource/Pagination.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,32 @@

namespace Zendesk\API\Traits\Resource;

use Zendesk\API\Traits\Utility\Pagination\AbstractStrategy;
use Zendesk\API\Traits\Utility\Pagination\CbpStrategy;
use Zendesk\API\Traits\Utility\Pagination\PaginationIterator;

trait Pagination {
/**
* Usage:
* $ticketsIterator = $client->tickets()->iterator();
* foreach ($ticketsIterator as $ticket) {
* process($ticket)
* process($ticket);
* }
*
* @return PaginationIterator to fetch all pages.
*/
public function iterator($params = [])
{
$strategyClass = $this->paginationStrategyClass();
$strategy = new $strategyClass($this->resourcesKey(), AbstractStrategy::DEFAULT_PAGE_SIZE);
return new PaginationIterator($this, $strategy, $params);
$strategy = new $strategyClass($this->resourcesKey(), $params);

return new PaginationIterator($this, $strategy);
}

/**
* Override this method in your resources
*
* @return string subclass of AbstractStrategy used for fetching pages
*/

protected function paginationStrategyClass() {
return CbpStrategy::class;
}
Expand Down
29 changes: 24 additions & 5 deletions src/Zendesk/API/Traits/Utility/Pagination/AbstractStrategy.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,39 @@

abstract class AbstractStrategy
{
public const DEFAULT_PAGE_SIZE = 100;

/**
* @var string The response key where the data is returned
*/
protected $resourcesKey;
protected $params;
protected $pageSize;

public function __construct($resourcesKey, $pageSize = self::DEFAULT_PAGE_SIZE)
public function __construct($resourcesKey, $params)
{
$this->resourcesKey = $resourcesKey;
$this->pageSize = $pageSize;
$this->params = $params;
}

public function params()
{
return $this->params;
}

protected function pageSize()
{
if (isset($this->pageSize)) {
return $this->pageSize;
} else if (isset($this->params['page[size]'])) {
$this->pageSize = $this->params['page[size]'];
} else if (isset($this->params['per_page'])) {
$this->pageSize = $this->params['per_page'];
} else {
$this->pageSize = DEFAULT_PAGE_SIZE;
}

return $this->pageSize;
}

abstract public function getPage($pageFn);
abstract public function page($getPageFn);
abstract public function shouldGetPage($position);
}
59 changes: 52 additions & 7 deletions src/Zendesk/API/Traits/Utility/Pagination/CbpStrategy.php
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,65 @@ class CbpStrategy extends AbstractStrategy
private $afterCursor = null;
private $started = false;

public function getPage($pageFn)
public function page($getPageFn)
{
$this->started = true;
$params = ['page[size]' => $this->pageSize];
if ($this->afterCursor) {
$params['page[after]'] = $this->afterCursor;
}
$response = $pageFn($params);

$response = $getPageFn();
$this->afterCursor = $response->meta->has_more ? $response->meta->after_cursor : null;

return $response->{$this->resourcesKey};
}

public function shouldGetPage($position) {
return !$this->started || $this->afterCursor;
}

public function params()
{
$result = array_merge($this->params, $this->paginationParams(), $this->sortParams());
$result = $this->unsetObpParams($result);

return $result;
}
/**
* The params that are needed to ordering in CBP (eg: ["sort" => "-age"])
* If OBP params are passed, they are converted to CBP
*
* OBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort_order=desc&sort_by=updated_at&per_page=2
* CBP: https://{subdomain}.zendesk.com/api/v2/tickets?sort=-updated_at&page[size]=2
*
* @return array all params with CBP sorting order
*/
private function sortParams()
{
if (isset($this->params['sort_by']) && !isset($this->params['sort'])) {
$direction = (isset($this->params['sort_order']) && strtolower($this->params['sort_order']) === 'desc') ? '-' : '';
return array_merge($this->params, ['sort' => $direction . $this->params['sort_by']]);
} else {
return [];
}
}
/**
* The params that are needed to for pagination (eg: ["page[size]" => "100"])
* If OBP params are passed, they are converted to CBP
*
* @return array all params with CBP sorting order
*/
private function paginationParams()
{
$result = isset($this->afterCursor) ? ['page[after]' => $this->afterCursor] : [];

return array_merge(['page[size]' => $this->pageSize()], $result);
}

private function unsetObpParams($params)
{
unset(
$params['page'],
$params['per_page'],
$params['sort_by'],
$params['sort_order']
);
return $params;
}
Copy link
Collaborator Author

@ecoologic ecoologic Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so now the CbpStrategy knows a lot about OBP. I thought of extracting it, and I did,
but it leads to a lot of indirection. In the end I prefer to accept some duplication and imperfect encapsulation in favour of simpler code.

}
12 changes: 8 additions & 4 deletions src/Zendesk/API/Traits/Utility/Pagination/ObpStrategy.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,20 @@ class ObpStrategy extends AbstractStrategy
{
private $pageNumber = 0;

public function getPage($pageFn)
public function page($getPageFn)
{
++$this->pageNumber;
$params = ['page' => $this->pageNumber, 'page_size' => $this->pageSize];
$response = $pageFn($params);
$response = $getPageFn();

return $response->{$this->resourcesKey};
}

public function shouldGetPage($position) {
return $this->pageNumber == 0 || $position >= $this->pageNumber * $this->pageSize;
return $this->pageNumber == 0 || $position >= $this->pageNumber * $this->pageSize();
}
public function paramsWithPagination()
{
return ['page' => $this->pageNumber, 'per_page' => $this->pageSize()];

}
}
22 changes: 11 additions & 11 deletions src/Zendesk/API/Traits/Utility/Pagination/PaginationIterator.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@

namespace Zendesk\API\Traits\Utility\Pagination;

const DEFAULT_PAGE_SIZE = 100;

use Iterator;

class PaginationIterator implements Iterator
{
private $position = 0;
private $page = [];
private $strategy;
private $params;

/**
* @var mixed use trait FindAll. The object handling the list, Ie: `$client->{clientList}()`
* @var mixed using trait FindAll. The object handling the list, Ie: `$client->{clientList}()`
* Eg: `$client->tickets()` which uses FindAll
*/
private $clientList;
private $strategy;
private $position = 0;
private $page = [];

public function __construct($clientList, AbstractStrategy $strategy, $params = [])
public function __construct($clientList, AbstractStrategy $strategy)
{
$this->clientList = $clientList;
$this->strategy = $strategy;
$this->params = $params;
}

public function key()
Expand Down Expand Up @@ -59,10 +59,10 @@ private function getPageIfNeeded()
return;
}

$pageFn = function ($paginationParams = []) {
return $this->clientList->findAll(array_merge($this->params, $paginationParams));
$getPageFn = function () {
return $this->clientList->findAll($this->strategy->params());
};

$this->page = array_merge($this->page, $this->strategy->getPage($pageFn));
$this->page = array_merge($this->page, $this->strategy->page($getPageFn));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ class SinglePageStrategy extends AbstractStrategy
{
protected $started = false;

public function getPage($pageFn)
public function page($getPageFn)
{
$this->started = true;
$response = $pageFn();
$response = $getPageFn();

return $response->{$this->resourcesKey};
}
Expand Down
Loading
Loading