Enhancement: eclean-dist should additionally handle git checkouts in $DISTDIR/git3-src #33

hyprsyd · 2023-09-21T20:27:20Z

thesamesam · 2023-09-26T21:10:41Z

@floppym Would you mind taking a look?

floppym · 2023-09-27T19:12:56Z

pym/gentoolkit/eclean/search.py

+    def git_check(self, distdir):
+        cpvs = [
+            i.split("/")[1][:-5]
+            for i in set(self.vardb.cpv_all())
+            if "live" in self.vardb.aux_get(i, ["PROPERTIES"])
+        ]
+        git_src = os.path.join(distdir, "git3-src")
+        gitdir = dict(map(lambda i: (i.split("_")[1][:-4], i), os.listdir(git_src)))
+        deprecated_git = [
+            os.path.join(git_src, gitdir[i]) for i in gitdir.keys() if i not in cpvs
+        ]
+        return deprecated_git


Please add a comment to explain what this is doing. It's very hard to read.

I suspect you are matching directory names in git3-src to ebuild package names. That seems incorrect to me: git-r3.eclass does not use the ebuild package name to calculate the repo_name variable, which is what determines the directory name within git3-src. The repo_name variable is calculated based on the git source URI.

Yes @floppym I was doing that now I'm using the env file for the path.

hyprsyd · 2023-09-27T23:50:39Z

thesamesam · 2023-09-28T04:55:36Z

pym/gentoolkit/eclean/search.py

+        ]
+        git_src = os.path.join(distdir, "git3-src")
+        gitdir = [os.path.join(git_src, i) for i in os.listdir(git_src)]
+        deprecated_git = [i for i in gitdir if i not in expected_dirs]


Do you mean obsolete instead of deprecated?

Yes I'll change that

floppym

I wonder how this will work for ebuilds that utilize multiple git repos, either directly or indirectly via git submodules. It's probably worth testing that.

It might also be good to give the user a way to opt out of cleaning git repos via a command line option.

floppym · 2023-09-28T14:52:29Z

pym/gentoolkit/eclean/search.py

+    def git_check(self, distdir):
+        """Checks $DISTDIR/git3-src for checkouts which are not in the vardb"""
+        expected_dirs = [
+            self.vardb._aux_env_search(i, ["EGIT_DIR"])["EGIT_DIR"]


EGIT_DIR contains an absolute path that starts with DISTDIR as it was defined at the time the package was installed.

It is possible for the user to move DISTDIR to another location by copying/moving the files and updating make.conf. In this case, eclean-dist would end up removing all git repos since the paths would not match in the new DISTDIR.

A simple workaround might be to take the last component of EGIT_DIR, and append it to the current value of ${DISTDIR}/git3-src.

Yes you are right @floppym ! I'll change that

… fetched in the gir3_src see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

… fetched in the gir3_src ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing GDIR_NAME to address this limitation. see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

…eing fetched in the gir3_src ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing EGIT_DIR_NAME to address this limitation. see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

hyprsyd · 2023-09-30T20:25:26Z

@thesamesam @floppym

Ebuild that fetches single repo

Ebuild that fetches multiple repos

with pretend

thesamesam · 2023-10-08T03:32:35Z

@mgorny, please take a look

pym/gentoolkit/eclean/cli.py

…eing fetched in the gir3_src ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing EGIT_DIR_NAME to address this limitation. see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing EVCS_STORE_DIRS to address this limitation. see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

mgorny · 2023-10-16T14:37:43Z

pym/gentoolkit/eclean/search.py

+                            .strip("()")
+                            .split()


This won't handle whitespace correctly. The whole point of using an array is to handle it correctly.

The eclass currently does not use whitespace in repo_name, so this is problem in theory only.

Are you aware of any way to properly parse a bash array in Python?

The eclass currently does not use whitespace in repo_name, so this is problem in theory only.

The variable is supposed to contain the full path.

Are you aware of any way to properly parse a bash array in Python?

Perhaps shlex would be good enough.

The variable is supposed to contain the full path.

~~No, it's not. The full path is useless since the user could change DISTDIR at any time.~~

We can ignore this edge case for now.

Perhaps shlex would be good enough.

Yeah, shlex.split() will work after stripping the parens.

ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing EVCS_STORE_DIRS to address this limitation. see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

hyprsyd · 2023-10-18T12:24:10Z

@mgorny @floppym thanks for the reviews hope this looks better.

mgorny

One more idea: let's make it more "generic", i.e. at least make options about "vcs" rather than "git". While you don't have to immediately support other VCS stores, let's have a single CLI that'll work if someone adds more in the future.

hyprsyd · 2023-11-01T18:19:56Z

One more idea: let's make it more "generic", i.e. at least make options about "vcs" rather than "git". While you don't have to immediately support other VCS stores, let's have a single CLI that'll work if someone adds more in the future.

Done

A new feature for eclean-dist to clean git3-src. Optionally, cleaning the vcs-src can be skipped with --skip-vcs. Bug: https://bugs.gentoo.org/622938 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

mgorny

LGTM from eclasses standpoint. I can't judge all the eclean code though.

ebuilds which calls git-r3_fetch multiple times for diffrent repos. While EGIT_DIR stores a single repo path, mapping all repos to such packages is currently unfeasible. Introducing EVCS_STORE_DIRS to address this limitation. See-Also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com> Closes: #33133 Signed-off-by: Michał Górny <mgorny@gentoo.org>

thesamesam · 2023-12-03T07:52:08Z

Thank you! Merged, and appreciate the patience.

zmedico · 2024-05-02T16:32:32Z

pym/gentoolkit/eclean/search.py

@@ -148,7 +149,7 @@ def findDistfiles(
                + "%s remaining candidates to clean" % len(clean_me)
            )
            clean_me, saved = self._check_excludes(exclude, clean_me)
-        return clean_me, saved, deprecated
+        return clean_me, saved, deprecated, vcs


This is also needed on line 117 in order to fix https://bugs.gentoo.org/928138.

hyprsyd force-pushed the master branch from 2eacabb to 8a637ef Compare September 21, 2023 21:08

hyprsyd force-pushed the master branch from 8a637ef to b76bdfb Compare September 27, 2023 18:58

floppym suggested changes Sep 27, 2023

View reviewed changes

hyprsyd force-pushed the master branch 2 times, most recently from 4d8dccf to 9c0ebf0 Compare September 27, 2023 23:48

hyprsyd requested a review from floppym September 27, 2023 23:51

thesamesam reviewed Sep 28, 2023

View reviewed changes

hyprsyd force-pushed the master branch from 9c0ebf0 to 5f1b823 Compare September 28, 2023 12:12

hyprsyd requested a review from thesamesam September 28, 2023 12:21

floppym suggested changes Sep 28, 2023

View reviewed changes

hyprsyd force-pushed the master branch from 5273dbc to a45909d Compare September 28, 2023 18:49

hyprsyd requested a review from floppym September 28, 2023 19:05

hyprsyd force-pushed the master branch from a45909d to d31bccf Compare September 29, 2023 15:38

hyprsyd added a commit to hyprsyd/gentoo that referenced this pull request Sep 30, 2023

git-r3.eclass: Inroducing GDIR_NAME var to record all the repos being…

fa4cbb6

… fetched in the gir3_src see_also: gentoo/gentoolkit#33 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

hyprsyd mentioned this pull request Sep 30, 2023

git-r3.eclass: Inroducing EGIT_DIR_NAME var to record all the repos being fetched in the gir3_src gentoo/gentoo#33133

Closed

hyprsyd force-pushed the master branch from d31bccf to 41ff304 Compare September 30, 2023 08:37

hyprsyd force-pushed the master branch from 41ff304 to d5a1981 Compare September 30, 2023 11:50

hyprsyd force-pushed the master branch 4 times, most recently from 04154be to f36db1e Compare September 30, 2023 20:19

hyprsyd force-pushed the master branch 2 times, most recently from efab605 to 3633a71 Compare October 5, 2023 19:46

thesamesam reviewed Oct 8, 2023

View reviewed changes

pym/gentoolkit/eclean/cli.py Outdated Show resolved Hide resolved

hyprsyd force-pushed the master branch 2 times, most recently from 449448c to d387f4a Compare October 8, 2023 15:46

hyprsyd requested a review from thesamesam October 10, 2023 14:31

hyprsyd force-pushed the master branch from d387f4a to 2a48054 Compare October 15, 2023 12:40

mgorny reviewed Oct 16, 2023

View reviewed changes

hyprsyd force-pushed the master branch 2 times, most recently from 6e8097d to 9b1e476 Compare October 18, 2023 12:24

hyprsyd requested a review from mgorny October 18, 2023 12:25

mgorny reviewed Nov 1, 2023

View reviewed changes

hyprsyd force-pushed the master branch from 9b1e476 to 6fc6290 Compare November 1, 2023 18:12

hyprsyd requested a review from mgorny November 1, 2023 18:20

Enhancement: eclean-dist handle git checkouts

31813b6

A new feature for eclean-dist to clean git3-src. Optionally, cleaning the vcs-src can be skipped with --skip-vcs. Bug: https://bugs.gentoo.org/622938 Signed-off-by: Siddhanth Rathod <xsiddhanthrathod@gmail.com>

hyprsyd force-pushed the master branch from 6fc6290 to 31813b6 Compare November 1, 2023 18:27

mgorny approved these changes Nov 2, 2023

View reviewed changes

floppym approved these changes Nov 9, 2023

View reviewed changes

gentoo-bot closed this in 87912b4 Dec 3, 2023

zmedico reviewed May 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement: eclean-dist should additionally handle git checkouts in $DISTDIR/git3-src #33

Enhancement: eclean-dist should additionally handle git checkouts in $DISTDIR/git3-src #33

hyprsyd commented Sep 21, 2023

thesamesam commented Sep 26, 2023

floppym Sep 27, 2023 •

edited

Loading

hyprsyd Sep 27, 2023

hyprsyd commented Sep 27, 2023

thesamesam Sep 28, 2023

hyprsyd Sep 28, 2023

floppym left a comment

floppym Sep 28, 2023

hyprsyd Sep 28, 2023

hyprsyd commented Sep 30, 2023 •

edited

Loading

thesamesam commented Oct 8, 2023

mgorny Oct 16, 2023

floppym Oct 16, 2023

mgorny Oct 16, 2023

floppym Oct 16, 2023 •

edited

Loading

floppym Oct 16, 2023

hyprsyd commented Oct 18, 2023

mgorny left a comment

hyprsyd commented Nov 1, 2023

mgorny left a comment

thesamesam commented Dec 3, 2023

zmedico May 2, 2024

Enhancement: eclean-dist should additionally handle git checkouts in $DISTDIR/git3-src #33

Enhancement: eclean-dist should additionally handle git checkouts in $DISTDIR/git3-src #33

Conversation

hyprsyd commented Sep 21, 2023

thesamesam commented Sep 26, 2023

floppym Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyprsyd commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

floppym left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyprsyd commented Sep 30, 2023 • edited Loading

thesamesam commented Oct 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

floppym Oct 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyprsyd commented Oct 18, 2023

mgorny left a comment

Choose a reason for hiding this comment

hyprsyd commented Nov 1, 2023

mgorny left a comment

Choose a reason for hiding this comment

thesamesam commented Dec 3, 2023

Choose a reason for hiding this comment

floppym Sep 27, 2023 •

edited

Loading

hyprsyd commented Sep 30, 2023 •

edited

Loading

floppym Oct 16, 2023 •

edited

Loading