Codebase git mirroring #718

sgfost · 2024-05-07T23:21:26Z

part 1 (1-way mirror):

adds a button that allows model submitters to create an auto-updating, read-only git repository archive which is hosted on a central organization

additions

include CITATION.cff and LICENSE files in archive packages (resolves comses/planning#234)
- CITATION.cff is translated from codemeta
- LICENSE file is built from license text templates in the License model
library.fs.CodebaseGitRepositoryApi: functionality for building/updating a git repository from a Codebase
library.github.GithubApi: provides an interface over PyGithub for interacting with repositories on github
library.github.GithubRepoNameValidator: provides validate() to make sure a repo name is valid and unused
huey for async task processing which runs on the server container (resolves comses/planning#231)
- mirror_codebase() and update_mirrored_codebase() huey tasks which call the CodebaseGitRepositoryApi to build the git repo on the file system and then GithubApi to create/push to the remote
feature overview page at /github/
button + form on the release detail page for mirroring a codebase

configuration steps

create an app on the comses-model-library organization with the following permissions:
- Administration: read and write
- Contents: read and write
- Metadata: read only
generate client secret and private key
install the app on the organization and add the installation id to .env
add the app id, client id, and organization name to .env
add the private key and client secret to secrets/

django/library/views.py

* fix release ordering to sort by semantic version number rather than by string

this API is responsible for managing a local git repository mirror for a comses codebase. PUBLIC release archives are commits/tags in the history `build()` and `append_releases()` are the two main API methods which construct (or rebuild) a git repo and add new releases to the repo, respectively

* indicate that ordered releases method on codebase now returns a list and add public_releases() which returns a queryset

currently these are not retroactively inserted into archive packages since that would require rebuilding everything. Generating git repos, however, will add them if they are missing ** includes an experimental refactor of metadata transformations which is used to implement the citation file format generation resolves comses/planning#234

currently a synchronous process with 0 error handling * add file size checking to the git repo fs api

this allows for retries regardless of where the failure occurred the main points are: - only build the git repo if it doesn't exist - only create the github repo if it doesn't exist - pushing changes that exist remotely is already handled gracefully

https://huey.readthedocs.io/ The huey service is essentially a mirror of the server with a connection to the same db and redis service that runs the huey consumer process **setup is currently only for development**

* now only attempt to create github releases if they don't already exist * fixed bug where local_releases were being overwritten with only the last updated releases instead of adding to the set

this is considerably easier to manage than creating a clone service, not sure if there is any potential downside to not isolating the two processes

checks that repo names are valid and available TODO: add a /github page for more in depth information about the feature and a summary in the action modal

will need to really clean up metadata coversions, especially before adding syncing but the one-way transformers idea likely won't hold up ideas for a better approach: - pydantic for validation/structuring - codemeta as an intermediate format

adding these manually was an easily forgotten step that wouldn't be noticed in dev but would fail to build in prod

mirroring strategy is not currently planned to be used with user repositories * use absolute url in github description

[no ci]

alee

looks great, thanks for pushing this forward @sgfost!

some minor notes on the overall structure included as inline comments. the circular dependency from tasks <-> models is tricky, not sure how to get around it without some other layer of indirection like a service layer but it's probably fine to just keep the hyperlocal import, and I tend to agree with the no-service-layer django argument https://www.b-list.org/weblog/2020/mar/16/no-service/

alee · 2024-12-12T23:43:47Z

django/core/tests/base.py

@@ -173,5 +173,13 @@ def initialize_test_shared_folders():
    )


+def clear_test_shared_folder(dir=settings.REPOSITORY_ROOT):
+    for fs in os.scandir(dir):


is this cleaner if we move to pathlib like the other file where you used iterdir()? e.g.,

folder = Path(dir) for item in folder.iterdir(): if item.is_file(): item.unlink() elif item.is_dir(): shutil.rmtree(item, ignore_errors=True)

django/library/fs.py

django/library/github.py

django/library/fs.py

alee · 2024-12-13T01:02:20Z

django/library/fs.py

+                    r=True,
+                )
+        # copy over files from the sip storage and add to the index
+        for file in sip_storage.list(absolute=True):


would CodebaseReleaseStorage benefit from a copy_all convenience method that takes an optional callback method to be invoked on each file that's copied, or returns a list of all copied files' paths? That might simplify this to:

files = sip_storage.copy_all(dest_path) self.repo.index.add(files)

django/library/models.py

alee · 2024-12-13T02:45:05Z

django/library/metadata.py

+    }
+
+
+def release_to_codemeta(release: CodebaseRelease):


I haven't looked closely at the codemeticulous pydantic data classes yet but it might be cleaner to still have wrapper DataCiteSchema and CodeMetaSchema classes that also encapsulate the schema-specific transformations needed to properly populate these data classes

Yeah, there will probably end up being more than just codemeta generation in this module so it would at least scope the functions

django/requirements.txt

alee · 2024-12-13T02:57:29Z

frontend/src/components/GithubMirrorModal.vue

+      <template #body>
+        <p>
+          This will transform your model into a git repository and archive it on GitHub under a
+          central CoMSES Model Library organization.


How about updating this text to:

This will transform your model into a git repository and create a mirror of it on GitHub under the central CoMSES Model Library organization. Mirroring your model onto GitHub comes with many advantages; (list a few of the most compelling use cases we can think of, and also reassure them they will still retain ownership of the repository).

alee · 2024-12-13T02:58:31Z

frontend/vite.config.ts


 const { resolve } = require("path");
 const resolvePath = (relativePath: string) => {
  return resolve(__dirname, relativePath);
 };

+const getAppEntries = () => {


great refactor! need to document that the name of the file will be the expected bundle name on the django side.

alee · 2024-12-13T16:48:39Z

looks great, thanks for pushing this forward @sgfost!

some minor notes on the overall structure included as inline comments. the circular dependency from tasks <-> models is tricky, not sure how to get around it without some other layer of indirection like a service layer but it's probably fine to just keep the hyperlocal import, and I tend to agree with the no-service-layer django argument https://www.b-list.org/weblog/2020/mar/16/no-service/

For the updated metadata situation, we could consider pushing the metadata changes as a single commit on the release branch. If the release branch is also head then it also advances the main branch, otherwise it only lives on that release branch... could get messy though

sgfost · 2024-12-13T18:54:06Z

For the updated metadata situation, we could consider pushing the metadata changes as a single commit on the release branch. If the release branch is also head then it also advances the main branch, otherwise it only lives on that release branch... could get messy though

I think that is by far the best way to handle it. Question is whether adding release branches at all introduces too much complexity. For this (mirroring), probably not, but my thinking up until now was that it would for the fully synced mode

I may have been totally wrong though, its definitely a bit more complex but may solve more problems than it creates. I don't think we'd have to expose authors to the complexity, i.e. require that they make release branches, as the CML side could just generate one from the release tag

Sample workflow:

start sync from a codebase with releases 1.0.0 and 1.1.0
generate 1.0.0 1.1.0 and main branches
push to remote
author commits any number of times to main
author cuts a release on github, 2.0.0
CML pulls in changes, makes a release branch for 2.0.0 from the 2.0.0 tag
author decides to make a release on the CML now, 2.1.0 so we make a branch 2.1.0 and point main there
push to remote, which adds branches 2.0.0, 2.1.0, and updates main

The one major problem I can think of is handling updates to the release branches incoming from github. Metadata changes might be able to be resolved but source code, etc. should be frozen like they are in the CML, but there's no way to enforce this

sgfost added feature in progress labels May 7, 2024

sgfost force-pushed the git-mirror branch 4 times, most recently from 35de762 to a12a8c1 Compare July 30, 2024 19:17

sgfost force-pushed the git-mirror branch from 4334357 to 6ff1b56 Compare August 7, 2024 00:03

github-advanced-security bot found potential problems Aug 16, 2024

View reviewed changes

django/library/views.py Fixed Show fixed Hide fixed

sgfost force-pushed the git-mirror branch 2 times, most recently from ca4ade6 to 4be9693 Compare September 11, 2024 23:08

github-advanced-security bot found potential problems Sep 18, 2024

View reviewed changes

django/library/views.py Fixed Show fixed Hide fixed

sgfost force-pushed the git-mirror branch 2 times, most recently from 6b4e25f to 5a356fb Compare October 4, 2024 19:34

github-advanced-security bot found potential problems Oct 4, 2024

View reviewed changes

django/library/views.py Dismissed Show dismissed Hide dismissed

sgfost removed the in progress label Oct 18, 2024

sgfost marked this pull request as ready for review October 18, 2024 22:47

sgfost mentioned this pull request Oct 24, 2024

provide automated DOI registration #699

Merged

sgfost force-pushed the git-mirror branch from 4d4ddc9 to b88a22a Compare October 26, 2024 03:12

sgfost force-pushed the git-mirror branch from 10fa0ae to b22313c Compare December 3, 2024 19:58

sgfost added 11 commits December 5, 2024 11:37

feat: add model for tracking codebase git mirror

21eb1b0

* fix release ordering to sort by semantic version number rather than by string

test: add coverage for git repo api

6a57046

* indicate that ordered releases method on codebase now returns a list and add public_releases() which returns a queryset

fix: only format/access license text when needed

8b5e2c9

feat(WIP): basic github mirroring

f509c84

currently a synchronous process with 0 error handling * add file size checking to the git repo fs api

fix: use proper default for integer env settings

cb48570

feat(WIP): add huey for async task processing

ecfc97a

https://huey.readthedocs.io/ The huey service is essentially a mirror of the server with a connection to the same db and redis service that runs the huey consumer process **setup is currently only for development**

refactor: use huey task for github mirroring

0f5e5ab

feat: update github mirror on publish

eb73d6f

* now only attempt to create github releases if they don't already exist * fixed bug where local_releases were being overwritten with only the last updated releases instead of adding to the set

sgfost added 11 commits December 5, 2024 11:37

build: run huey consumer as a runit deamon in the server service

91b6a3c

this is considerably easier to manage than creating a clone service, not sure if there is any potential downside to not isolating the two processes

feat: better mirroring UI + repo name validation

2cdc58d

checks that repo names are valid and available TODO: add a /github page for more in depth information about the feature and a summary in the action modal

feat: add feature page for github integration

7024976

refactor: only make private repos when settings.DEBUG is True

2b52e89

build: dynamically add vue apps to vite config

df8ef62

adding these manually was an easily forgotten step that wouldn't be noticed in dev but would fail to build in prod

refactor: simplify gh mirroring task

9efef4d

mirroring strategy is not currently planned to be used with user repositories * use absolute url in github description

refactor: rename github integration app configs for clarity

99a30f9

refactor: minor adjustment to github integration UI

bfbf72d

fix: indent/format codemeta.json

2475fee

chore: append git mirroring migrations

2d48eab

sgfost force-pushed the git-mirror branch from f297ac9 to 2d48eab Compare December 5, 2024 18:44

refactor(WIP): codemeta generation to use codemeticulous

61438b8

[no ci]

alee reviewed Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Codebase git mirroring #718

Codebase git mirroring #718

sgfost commented May 7, 2024 •

edited

Loading

alee left a comment

alee Dec 12, 2024 •

edited

Loading

alee Dec 13, 2024

alee Dec 13, 2024

sgfost Dec 13, 2024

alee Dec 13, 2024 •

edited

Loading

alee Dec 13, 2024

alee commented Dec 13, 2024

sgfost commented Dec 13, 2024 •

edited

Loading

Codebase git mirroring #718

Are you sure you want to change the base?

Codebase git mirroring #718

Conversation

sgfost commented May 7, 2024 • edited Loading

part 1 (1-way mirror):

additions

configuration steps

alee left a comment

Choose a reason for hiding this comment

alee Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

alee Dec 13, 2024

Choose a reason for hiding this comment

alee Dec 13, 2024

Choose a reason for hiding this comment

sgfost Dec 13, 2024

Choose a reason for hiding this comment

alee Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

alee Dec 13, 2024

Choose a reason for hiding this comment

alee commented Dec 13, 2024

sgfost commented Dec 13, 2024 • edited Loading

sgfost commented May 7, 2024 •

edited

Loading

alee Dec 12, 2024 •

edited

Loading

alee Dec 13, 2024 •

edited

Loading

sgfost commented Dec 13, 2024 •

edited

Loading