Discussion: Making the threadpool its own package

[mratsim]

@nayr7 in #281

I'm not sure if this is the right place to ask (sorry if it's not) but is there any interest to make constantine/threadpool into it's own nimble package like Weave and nim-taskpools ? I am not sure if there are technical hurdles that prevents this (for example shared code between the threading and cryptographic part) or if it's just a lack of interest. I personally think there might be some merit to be able to use constantine's threadpool without bringing in the all the crypto baggage with it. Also thanks for your work ! :)

[mratsim]

@nayr7

Constantine's threadpool is standalone and does not depend on the cryptographic side of Constantine.

The only thing it uses in common with Constantine is log2 and the allocators like allocHeap.

What I can do is make it so that import constantine/threadpool works. However there is no plan to make it a separate package, beyond just a "monorepo", though maybe it can be a package that depends on Constantine and is only used for documentation and examples.

The reason why is that in cryptography and security-focused applications, having as few dependencies as possible or zero is a very strong feature, especially now that supply-chain attacks are rampant. I have been approached by teams telling me that they require zero dependencies from their crypto providers.

[mratsim]

The future Weave-IO is bootstraped using Constantine's threadpool.

https://github.com/mratsim/weave-io/tree/v0.2.0

It will evolve separately due to differences in goals but at the moment it's a 1-1 port.

The differences will boil down to:

• Constantine optimizes for throughput, hence uses LIFO scheduling to reuse data hot in cache.
• Weave IO will optimize for latency and fairness, hence FIFO scheduling to ensure that a task does not monopolize resources by spawning new tasks over and over.

cc @nayr7

[nayr7]

Hey, thanks for the follow-up! I have been using a private fork of Constantine in the meantime but I greatly appreciate this, this is an amazing addition to Nim's ecosystem and I thank you for it. I'll be switching to it when I have a little time since it seems more nicely organized... :-)