Searching Sentinel 1 RTC produces different number of results with sort parameter

[PCunninghamML]

I'm running a search of Sentinel 1 RTC via URL which is filtered to roughly the lower 48 US states. I'm getting what appears to be a correct response that can be paged via next links to completion. However when sorting by title, the results of the second page are presented as the end of the dataset (# results less than limit param, no next link). I wouldn't think that adding a sort parameter ought to affect the number of features/items.

Searching "ALOS PALSAR Annual Mosaic" (alos-palsar-mosaic) with the same parameters and sorting appears to produce the expected results.

Thank you for your help.

First call with sorting by title:
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313%2c22.917922936146%2c-63.7481689453125%2c50.6250730634144&sortby=%2bproperties.title
{
    "type": "FeatureCollection",
    "features": [
		{ "snip": "250 items here" }
	],
    "links": [
        {
            "rel": "next",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&sortby=+properties.title&token=next:sentinel-1-rtc:S1B_IW_GRDH_1SSV_20161012T014253_20161012T014318_002467_004297_rtc"
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/"
        },
        {
            "rel": "self",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313%2c22.917922936146%2c-63.7481689453125%2c50.6250730634144&sortby=%2bproperties.title"
        }
    ]
}

Second call with sorting, URL taken verbatim from "next" link of first call:
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&sortby=+properties.title&token=next:sentinel-1-rtc:S1B_IW_GRDH_1SSV_20161012T014253_20161012T014318_002467_004297_rtc
{
    "type": "FeatureCollection",
    "features": [
		{ "snip": "249 items here" }
	],
    "links": [
        {
            "rel": "previous",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&sortby=+properties.title&token=prev:sentinel-1-rtc:S1B_IW_GRDH_1SSV_20170516T133947_20170516T134012_005624_009D99_rtc"
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/"
        },
        {
            "rel": "self",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&sortby=+properties.title&token=next:sentinel-1-rtc:S1B_IW_GRDH_1SSV_20161012T014253_20161012T014318_002467_004297_rtc"
        }
    ]
}


Repeated test without sorting produces expected results.
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313%2c22.917922936146%2c-63.7481689453125%2c50.6250730634144
{
    "type": "FeatureCollection",
    "features": [
		{ "snip": "250 items here" }
	],
    "links": [
        {
            "rel": "next",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231201T003639_20231201T003704_051454_0635BF_rtc"
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/"
        },
        {
            "rel": "self",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313%2c22.917922936146%2c-63.7481689453125%2c50.6250730634144"
        }
    ]
}

Subsequent API calls taken from "next" link of the prior response:
call 2:
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231201T003639_20231201T003704_051454_0635BF_rtc
{
    "type": "FeatureCollection",
    "features": [
		{ "snip": "250 items here" }
	],
    "links": [
        {
            "rel": "next",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231124T221301_20231124T221326_051365_0632C4_rtc"
        },
        {
            "rel": "previous",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=prev:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231201T003614_20231201T003639_051454_0635BF_rtc"
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/"
        },
        {
            "rel": "self",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231201T003639_20231201T003704_051454_0635BF_rtc"
        }
    ]
}

call 3:
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231124T221301_20231124T221326_051365_0632C4_rtc
{
    "type": "FeatureCollection",
    "features": [
		{ "snip": "250 items here" }
	],
    "links": [
        {
            "rel": "next",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231119T124040_20231119T124105_051286_06300A_rtc"
        },
        {
            "rel": "previous",
            "type": "application/geo+json",
            "method": "GET",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=prev:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231124T221236_20231124T221301_051365_0632C4_rtc"
        },
        {
            "rel": "root",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/"
        },
        {
            "rel": "self",
            "type": "application/json",
            "href": "https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=sentinel-1-rtc&limit=250&bbox=-128.787231445313,22.917922936146,-63.7481689453125,50.6250730634144&token=next:sentinel-1-rtc:S1A_IW_GRDH_1SDV_20231124T221301_20231124T221326_051365_0632C4_rtc"
        }
    ]
}

...and so on and so forth for thousands of more features/items

[TomAugspurger]

Thanks for the report, and apologies for not getting back to you earlier.

I thought this was stac-utils/pgstac#177 (which is very similar) but we have that deployed and are still seeing this issue. We'll look into it and will get back to you.

For reference, here's the pystac-client version:

import pystac_client
catalog = pystac_client.Client.open("[https://planetarycomputer.microsoft.com/api/stac/v1")](https://planetarycomputer.microsoft.com/api/stac/v1%22))
 
a = catalog.search(
    collections="sentinel-1-rtc",
    # limit=250,
    bbox=[-128.787231445313, 22.917922936146, -63.7481689453125, 50.6250730634144],
    # sortby=["properties.title"],
    datetime="2020-01-01/2020-01-15"
)
 
 
b = catalog.search(
    collections="sentinel-1-rtc",
    # limit=250,
    bbox=[-128.787231445313, 22.917922936146, -63.7481689453125, 50.6250730634144],
    sortby=["title"],
    datetime="2020-01-01/2020-01-15"
)
 
 
aa = a.item_collection()
bb = b.item_collection()
assert len(aa) == len(bb), (len(aa), len(bb))

That's giving 943 vs. 499

[gadomski]

Similar/same issue reported on stac-fastapi: stac-utils/stac-fastapi#620 (reply in thread)

[PCunninghamML]

Has the status on this issue changed? I noticed the linked stac-fastapi comment mentions the issue possibly originating in stac-utils/pgstac but I didn't see any open issues over there describing the same behavior.

[TomAugspurger]

No updates here yet. We'll make an issue on pgstac once we have a reproducer that's isolated to just it (if indeed that's where the problem is).

[ghidalgo3]

Closed due to inactivity, feel free to reopen if you would like to continue this discussion.