[QUERY] Workaround for Java Data Lake SDK 5000 max results limit?

[al-v-in]

Query/Question
The List function will return a maximum of 5000 results, as stated here: https://docs.microsoft.com/en-us/java/api/com.azure.storage.file.datalake.models.listpathsoptions?view=azure-java-stable

Assuming I have a directory that has more than 5000 files, how can I list them all?

The rest API for "list" (https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/list) has a continuation token to get around this which doesn't seem to be exposed in the Java SDK. Is that correct?

If it's true that I can't use the Java Data Lake SDK for this, what should I use, the rest API directly?

Why is this not a Bug or a feature Request?
Because before I make a feature request I want to make sure I've not missed something obvious.

Setup (please complete the following information if applicable):
Not tested in code yet, just reviewing options from looking at the documentation.
com.azure:azure-storage-file-datalake:12.4.1

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Query Added
• Setup information Added

[joshfree]

@gapra-msft could you please take a look

[gapra-msft]

Hi @al-v-in

Thank you for your question. I will take a look at the behavior of listPaths and let you know

[gapra-msft]

Hi @al-v-in

The listPaths API should support automatic pagination by continuation token for you if you iterate over the PagedIterable returned, are you only seeing 5000 paths?

[al-v-in]

Thank you, @gapra-msft! I thought I must of have been missing something. As I mentioned I haven't tested in code, I was trying to figure out which method was best to interact with the data lake for our requirements.

I'm not sure if you would agree that it would be appropriate in this case, but perhaps it would clarify the matter if somewhere in the ListPathsOptions Class documentation it said something along the lines of "...max result of 5000 items per page"?

[gapra-msft]

Yeah! I can make this a docs bug and we will pick it up when we have time!