Use spack in wrappers #536
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* If GROUP is not set in environment when fake_ifdh.getRole is called, use getExp to guess the group. Also use fermilab group for test * Split out getRole to make it clearer what the logic is in discovering the role * Replaced instances where we set the GROUP env with monkeypatch so things get set back after tests are done * Strip off any spaces/newlines in the default getExp case * Reorganize tests into classes, since we have so many tests for each function
* Fixed pre-commit pylint stanza so we're actually running pylint on the repo * Added disabled checks * Fixed bin/ files to make pylint happy or disabled checks as needed * Fixed lib/ files to make pylint happy or disabled checks as needed - part 1 * Fixed lib/ files to make pylint happy or disabled checks as needed - part 2 * After rebasing, correct outstanding pylint errors or suppress them * PR #508 introduced a cyclic import. I've suppressed it here, but we need to fix this before the next release * Moved pylint check to local install * Skip pylint only for CI --------- Co-authored-by: Marc Mengel <mengel@fnal.gov>
* Moved pylint check to local install * Skip pylint only for CI * Added pylint github action * Testing workflow * Move ubuntu version back to support python 3.6 * Fixed pylint calls * Disabled OrderedDict subscriptable check. This disabling can be taken out when we upgrade python versions * Ignore condor_vault_storer in pylint check * Ignore condor_vault_storer in pylint check - fixed * Remove workflow_dispatch event
* Added capability for TarfilePublisherHandler to use a fixed RCDS server. The default is still to switch with each operation
* Between first RCDS check for existence and publish action if necessary, use the same RCDS server
* Changes to retry logic for TarfilePublisherHandler
1. Changed TarfilePublisherHandler decorator pubapi_operation so it
accepts a parameter, always_switch_servers. If always_switch_servers
is set to True, then upon retries, the handler will attempt to switch
servers, even if the previous state of the handler was to keep the
server fixed.
2. TarfilePublisherHandler.publish uses (1) so that it always attempts
different servers upon retrying.
3. For all operations, if the first attempt is a failure, we will
retry immediately with the next server, according to (1). If that
next retry fails, then we will wait for RETRY_INTERVAL_SEC before the
next and any subsequent retries.
* For the fixed_server case, make sure we don't switch to an iterator that returns the wrong server constantly
* Needed to implement the retry logic in the outer loop too
* Simplified logic for whether or not we change/restore the fixed server behavior in the pubapi_operation decorators
* Make restore_fixed_server_behavior a private function
* Added some verbose statements
* Fixed bug in __restore_fixed_server_behavior_func where we were not properly storing the old dropbox server. Also cleaned up some comments
) * Added github action that checks that we can run make and build RPM * Added pull_request to events that trigger this workflow * Added build status badge for build-al9 workflow * Run workflow at push and merged PR Originally the plan was to run this workflow at PR open, but that doesn't seem possible due to: https://github.blog/changelog/2021-02-19-github-actions-workflows-triggered-by-dependabot-prs-will-run-with-read-only-permissions/ We need a repo secret to access fermitools/jobsub_lite_config. Since we can't do that, for now, at least run after a PR merge or any other push to master. * Made build steps into custom action * Move container prep to its own action * Revert "Move container prep to its own action" This reverts commit fd091ff.
* Split build workflows into PR and push workflows * Updated status badge
ubuntu-20.04 runner
|
Note, we should definitely squash-merge this, as it has been rebased/merged and looks like a lot of commits that way. |
Closed
shreyb
reviewed
Feb 21, 2024
shreyb
requested changes
Feb 21, 2024
|
I just tried running all three of those commands on dunegpvm01, and it didn't work: Is python missing from those ifdh environments? |
|
Sigh. I just renamed the environment directories, but there was a symlink in there that still had the old path...
…--
Marc W. Mengel
Computer Services Senior Developer
Data Management and Applications
Fermi National Accelerator Laboratory
630 840 8256 office
www.fnal.gov<https://email.fnal.gov/owa/redir.aspx?C=JkVsWjSyHEuWV2kFJWNPyC9OyMKf7dEIp_yxPk9yfhfadE3mdLfI6T8Ztf8BVaJjzoIe8oJ5H0M.&URL=http%3a%2f%2fwww.fnal.gov>
<https://email.fnal.gov/owa/redir.aspx?C=JkVsWjSyHEuWV2kFJWNPyC9OyMKf7dEIp_yxPk9yfhfadE3mdLfI6T8Ztf8BVaJjzoIe8oJ5H0M.&URL=http%3a%2f%2fwww.fnal.gov>
Connect with us!
Newsletter<https://email.fnal.gov/owa/redir.aspx?C=-29AtpIe40SAvGF4aiNr86KvVjap7dEIuxIdg8Q5bYa9sT7QTFVhxhVnl8zwXkEuBSuAQLRkEfg.&URL=http%3a%2f%2fwww.fnal.gov%2fpub%2ftoday> | Facebook<https://email.fnal.gov/owa/UrlBlockedError.aspx> | Twitter<https://email.fnal.gov/owa/redir.aspx?C=-29AtpIe40SAvGF4aiNr86KvVjap7dEIuxIdg8Q5bYa9sT7QTFVhxhVnl8zwXkEuBSuAQLRkEfg.&URL=https%3a%2f%2ftwitter.com%2fFermilabToday>
________________________________
From: Shreyas Bhat ***@***.***>
Sent: Wednesday, February 21, 2024 1:44 PM
To: fermitools/jobsub_lite ***@***.***>
Cc: Marc W Mengel ***@***.***>; Author ***@***.***>
Subject: Re: [fermitools/jobsub_lite] Use spack in wrappers (PR #536)
[EXTERNAL] – This message is from an external sender
I just tried running all three of those commands on dunegpvm01, and it didn't work:
-bash-4.2$ . /cvmfs/fermilab.opensciencegrid.org/packages/common/setup-env.sh
-bash-4.2$ spack env list
==> 6 environments
ifdh_env_almalinux9_current ifdh_env_almalinux9_v2_6_20 ifdh_env_almalinux9_v2_7 ifdh_env_scientific7_current ifdh_env_scientific7_v2_6_20 ifdh_env_scientific7_v2_7
-bash-4.2$ spack env activate ifdh_env_scientific7_current
==> Warning: could not load runtime environment due to RuntimeError: Unable to locate python command in /cvmfs/fermilab.opensciencegrid.org/packages/common/spack/v0.21.0-fermi/NULL/var/spack/environments/ifdh_env_scientific7_current/.spack-env/view/./bin
Is python missing from those ifdh environments?
—
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fermitools_jobsub-5Flite_pull_536-23issuecomment-2D1957778740&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=mA4qFs7Hfq-joWhWXnsXfA&m=VTnADIkh-Nz7EyC4mt55YudX0L-qLQHxPvaJ9NVLhEzxzlpBLMzN9d8YjlOZEQFn&s=PfJO_lUUDqgzjBxxb14xxq2d4CECBh0Zju9n8LyDlvI&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAB6GV4R2FL7NU7NQR2KE2DYUZFA5AVCNFSM6AAAAABDSAEENKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJXG43TQNZUGA&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=mA4qFs7Hfq-joWhWXnsXfA&m=VTnADIkh-Nz7EyC4mt55YudX0L-qLQHxPvaJ9NVLhEzxzlpBLMzN9d8YjlOZEQFn&s=dV4ZzvZXrScDykQUDLxWfu5LGaqsMr74mulFxW1q1bw&e=>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
No worries - it looks good to me! I'll go ahead and squash-merge this then. |
|
Actually - doing two quick test submissions to make sure this works onsite and offsite: 8439421.0@jobsub04.fnal.gov (Onsite) |
|
Both tests worked. This is good to go. |
shreyb
approved these changes
Feb 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change uses the /cvmfs/fermilab.opensciencegrid.org/packages/common Spack area to find ifdhc, via a spack environment,
rather than using the UPS pacakges in /cvmfs/fermilab.opensciencegrid.org/products/common, or looking in /grid/fermiapp
which hasn't actually worked for some time.