Handling ThreadPool Exhaustion with High Concurrency in PnP Core SDK

[bajce]

In our .NET application we are reading significant amount of data from SharePoint Online and in order to optimize performance, I've implemented parallel tasks to read data concurrently. However, after processing data for some time, I', getting the following exception:

There were not enough free threads in the ThreadPool to complete the operation.

This exception is thrown by the PnP Core SDK, and as it's part of the library's internal workings, I have limited control over the thread management within the SDK itself.

The amount of data I'm working with is dynamic, varying based on different SharePoint servers, as performance of servers hosting our application which makes it difficult to set a constant limit on the number of threads or parallel tasks.

I'm reaching out to ask for guidance or suggestions on how to best handle this situation. Specifically, I'm interested in:

1. Any best practices for managing ThreadPool resources effectively when using PnP Core SDK in a highly concurrent environment.
2. Strategies to prevent ThreadPool exhaustion without significantly compromising the data processing performance.
3. If there are any configurations or patterns within the PnP Core SDK that could help in optimizing thread usage or any known issues that I should be aware of.

I appreciate any insights or experiences you could share about handling similar scenarios or any advice on how to approach this problem.

Thank you in advance for your support and suggestions!

[jansenbe]

@bajce : can you share a stack trace?