MSTransferor: support input data placement with Rucio #9759
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amaltaro
commented
Jun 19, 2020
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
For some reason, the it must be missing some subtle detail of the import behavior. @todor-ivanov I believe there will be further changes to this PR. But just in case you can have a look at these changes by tomorrow morning, it would be great! Thanks |
|
Jenkins results:
|
|
Jenkins results:
|
todor-ivanov
requested changes
Jul 1, 2020
|
Todor, thanks for your review. Following your review, I have made further changes in my last commit. Please have another look, and if you agree, mark those conversations as resolved. |
|
Thanks @amaltaro I took a quick look at your new commit and I think you addressed everything I mentioned. I think the changes are good to go now. |
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
@nsmith- and @ericvaandering I think I have finally converged on this PR and tested the most common use cases. I might merge it today, but feel free to leave questions/comments at any time and I can follow up on those in a different PR, if needed. |
|
Jenkins results:
|
|
Jenkins results:
|
Implement whole logic of input data placement with Rucio fix import from PycurlRucio clean init file minor aesthetic changes import absolute_import one more absolute_import copy cert/ckey methods over to pycurlRucio useRucio flag might not be in the configuration file apply Todors suggestions use getattr for self.phedex set self.phedex instead Rucio initialization message complete RSE limits/usage logic fix key name name -> rse bytes_remaining is always 0, set it to quota fix getRucioToken getdata has no decode parameter fix getPileupDatasetSizesRucio function fix getDatasetBlocksRucio and getPileupSubscriptionsRucio fix getBlockReplicasAndSizeRucio method New pycurl function listReplicationRules fix listReplicationRule fail case API to fetch blocks and their sizes given a container Make the correct calls for PhEDEx and Rucio integrate pickRSE update long=1 to long=True fix logic for renewing the rucio token RequestInfo child class of MSCore; skip some service initialization fix keyword arguments fix MSMonitor NoneType; add debug rule log bugfixes to rule monitoring; rule creation; others support lifetime parameter for rule replicas fix MSOutput initialization more fixes to the pycurl rucio function; print grouping of the rules Make Rucio rule completion consistent with PhEDEx, percentage stuck_at returns a string datetime format minor logging fix
cleanup init.py module more fixes to unit tests unit test for blocks/sizes per container fix unit tests
|
Jenkins results:
|
|
All right, it should be tested enough by now... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #9725
Fixes #9461
Status
In development
Description
Summary of changes are:
__init__method to support sub-classes to provide service names that they do not want to be initialized;wmcore_transferorask_approvalalways False;lifetimeandasynchronousparameters for replication rule creation;In addition to that, there is:
It depends on the following new configuration parameters:
useRucio: to enable/disable Rucio as a DM service (if disabled, PhEDEx is used)rulesLifetime: defines an expiration time for all rules created by MSTransferor. Production rules (against the production Rucio server) have no expiration time.Is it backward compatible (if not, which system it affects?)
yes
Related PRs
none
External dependencies / deployment changes
Deployment changes:
dmwm/deployment#924
dmwm/deployment#942