Create Rucio APIs for MicroServices with multi pycurl

[amaltaro]

Impact of the new feature
MicroServices

Is your feature request related to a problem? Please describe.
The MS Transferor implementation is not yet fully supporting Rucio. One of the reasons is because we still have to find out what are the exact APIs that best suit our needs.

Describe the solution you'd like
Implement the Common wrapper APIs for Rucio requests, using the concurrent pycurl getdata
function such that we can speed up any REST API requests.
If there are no Rucio APIs providing the information that is needed, e.g.: given a dataset, return its size, we might have to fallback to DBS (likely meaning a slower response too).

Describe alternatives you've considered
We could use the Services/Rucio module, but then we cannot take advantage of the concurrent requests. In the future we can reorganize those functions under the Common.py module and make it at least more clear and reuse as much code as we can.

Additional context
This issue is meant to address what didn't get implemented in this PR: #9457

[amaltaro]

@ericvaandering Eric, I'm planning to use the Rucio REST APIs because then we can profit of the concurrency offered by the WMCore/pycurl module. However, we have the extra complexity in the sense that we need to manage the token, and make sure that we use a valid token; and handle requests that failed due to its "unexpected" expiration.

If I'm not wrong, I think I heard you saying in one of these meetings that a "special setup" has been provided to DAS, to avoid this extra burden with the token(?) Perhaps we could look into the same thing for these CMSWEB services?

[ericvaandering]

No, there is not a special setup. The issue we had to solve with DAS was allowing DAS to be able to omit the Rucio account name, but that doesn’t help you. It’s going to be a pain this way. :-)

…

On Jun 8, 2020, at 5:54 AM, Alan Malta Rodrigues ***@***.***> wrote: @ericvaandering <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ericvaandering&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=HwTp2yl0DgquBcdmWcdoJybTeG6aUotB0E0v4ovpB-4&s=XNQvQv3W2f_64uXNgwrKtZWaUlr7TGs7gpUKH0GRauw&e=> Eric, I'm planning to use the Rucio REST APIs because then we can profit of the concurrency offered by the WMCore/pycurl module. However, we have the extra complexity in the sense that we need to manage the token, and make sure that we use a valid token; and handle requests that failed due to its "unexpected" expiration. If I'm not wrong, I think I heard you saying in one of these meetings that a "special setup" has been provided to DAS, to avoid this extra burden with the token(?) Perhaps we could look into the same thing for these CMSWEB services? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_issues_9461-23issuecomment-2D640529950&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=HwTp2yl0DgquBcdmWcdoJybTeG6aUotB0E0v4ovpB-4&s=ACoL5LP_1681tWnSF3g7URCRf1wCOLaqLN3ShCIBTrQ&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAMYJLQPTOHFGDAKSXXCCO3RVS7PFANCNFSM4J4A6P4Q&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=HwTp2yl0DgquBcdmWcdoJybTeG6aUotB0E0v4ovpB-4&s=xCVsjVaI-Q0sKN1xKGQdpmAgR3m2xbqSwxO6KjLN6nw&e=>.

[vkuznet]

@amaltaro , if you need to understand REST Rucio APIs you may use dasgoclient with proper site query and use -verbose 3 option which will print all HTTP request info such that you may understand all REST calls DAS is doing behind the scene.

[amaltaro]

Thanks Eric, Valentin. I have a prototype using Rucio REST APIs, so the basics should all be clear by now. It's just the extra burden of renewing/verifying the token.