Published Content Query with skip > 0 returns incorrect results

[matthewcare]

Which exact Umbraco version are you using? For example: 8.13.1 - don't just write v8

9.0.0 (also the contrib branch)

Bug summary

Strange interaction with PublishedContentQuery when searching with skip and take. I'm experiencing unexpected results, and no results in some cases.

Specifics

Here is the output of an identical set of searches with different skip and take values
For reference, I had 5 pages containing the word page, A Page, Sub Page, More page, Further sub page, Pages

Skip 0 Take 0 total results 5
A Page
Sub Page
More page
Further sub page
Pages

Skip 0 Take 1 total results 5
A Page

Skip 0 Take 100 total results 5
A Page
Sub Page
More page
Further sub page
Pages

Skip 1 Take 1 total results 0

Skip 1 Take 2 total results 5

Skip 1 Take 100 total results 5
More page
Further sub page
Pages

Skipped 2 Taken 0 total results 0

Skipped 2 Taken 2 total results 5

Skipped 2 Taken 100 total results 5
Pages

In the cases where there is a skip of 1, the first instance said there were no results, the second instance said there were 5 results but showed none of them, and the last instance again said there were 5 results, but appeared to skip the first 2, despite saying to only skip 1.

Similar results for when the skip is set to 2.

@Shazwazza
I'm not sure if this relates to Shazwazza/Examine#234

Digging through Umbraco.Cms.Infrastructure.PublishedContentQuery, it certainly looks like this is an issue with whatever happens inside the QueryExecutor

Steps to reproduce

Create a number of pages in the CMS with a name that contains "page"

Put this in a view (for ease), or implement a controller.

@using Umbraco.Cms.Core
@using Examine
@using Umbraco.Cms.Infrastructure.Examine
@inject IPublishedContentQuery _publishedContentQuery;
@inject IExamineManager _examineManager;

@{
	_examineManager.TryGetIndex(Constants.UmbracoIndexes.ExternalIndexName, out var index);

	const string searchTerm = "page";

    var query = index.Searcher
        .CreateQuery(IndexTypes.Content)
        .ManagedQuery(searchTerm);

	var searchTests = new List<(int, int)> { (0, 0), (0, 1), (0, 100), (1, 1), (1, 2), (1, 100) };
	
    foreach (var (skip, take) in searchTests)
    {
        var results = _publishedContentQuery.Search(query, skip, take, out var totalItems);

        <h4>Skipped @skip Taken @take total results @totalItems</h4>
        foreach (var result in results)
        {
            <p>@result.Content.Name</p>
        }

    }
}

Expected result / actual result

I would expect correct paging when using skip

[matthewcare]

After running the code again with PR #11378 the following results are achieved, which is a little more correct.

Skipped 0 Taken 0 total results 5
A Page
Sub Page
More page
Further sub page
Pages

Skipped 0 Taken 1 total results 5
A Page

Skipped 0 Taken 100 total results 5
A Page
Sub Page
More page
Further sub page
Pages

Skipped 1 Taken 1 total results 0

Skipped 1 Taken 2 total results 5
Sub Page

Skipped 1 Taken 100 total results 5
Sub Page
More page
Further sub page
Pages

Skipped 2 Taken 0 total results 0

Skipped 2 Taken 2 total results 5
More page
Further sub page

Skipped 2 Taken 100 total results 5
More page
Further sub page
Pages

[matthewcare]

By no means a well put together test, but if someone wants some tests to run against. You can update
Umbraco.Cms.Tests.Integration.Umbraco.Examine.Lucene.UmbracoExamine.SearchTests with
https://gist.github.com/matthewcare/a2e4c0db9c45a3aa2b0d5d0ea8f807f6

Current failures are

Skip 1 Take 1
Skip 1 Take 2
Skip 1 Take 3
Skip 1 Take 4

Where each returns one fewer result than expected.

[matthewcare]

@Shazwazza Located the problem in Examine and opened a PR
Examine will need updating and PR #11378 applying for search paging to work properly.
@nul800sebastiaan Anything more I can do here?

[nul800sebastiaan]

Apologies for the delay @matthewcare - thanks very much for the help much appreciated. Look forward to v9.1.0! 👍

[nul800sebastiaan]

Hmmm, guess we need to merge #11378 as well for this to be fully fixed? 🤔

[matthewcare]

@nul800sebastiaan Yeah, the newer Examine search returns the already paged results, so the additional Skip was causing the first page to be skipped.