Clone repo instead of init + fetch and take advantage of git-autoshare #55

joao-p-marques · 2021-10-05T12:22:23Z

Closes #45

As mentioned in the issue discussion, the aggregation process could be improved to start with a git clone instead of a init followed by fetch. This PR implements that.

That also opens the possibility to use git-autoshare to speed up the process and save space. This PR also adds that functionality, whilst keeping the original setup working when you don't have git-autoshare installed in your system.

@Tecnativa
TT32042

ping @sbidoul @yajo

git_aggregator/repo.py

sbidoul · 2021-10-06T12:43:02Z

git-aggregator should not have to know about git-autoshare.

sbidoul · 2021-10-06T12:52:13Z

BTW if you are playing with this, you could try the --filter=blob:none git option to optimize download times, it might help a lot.

joao-p-marques · 2021-10-06T12:53:02Z

git-aggregator should not have to know about git-autoshare.

Ok, but then how can it take advantage of git-autoshare? As we are launching git commands from Python, the suggested bash script would not work, I think...

sbidoul · 2021-10-06T12:54:31Z

If you do a subprocess.call(["git",...]) it should look in PATH, I think.

joao-p-marques · 2021-10-06T12:55:12Z

BTW if you are playing with this, you could try the --filter=blob:none git option to optimize download times, it might help a lot.

Ok, I will set this do Draft while I fix the tests and look into this option... Thanks for the suggestions!

yajo · 2021-10-08T08:54:58Z

git-aggregator should not have to know about git-autoshare.

Do you mean that git-autoshare is used automatically on some git commands?

codecov · 2021-10-08T13:11:51Z

Codecov Report

Merging #55 (6e2ecdd) into master (1047659) will increase coverage by 0.77%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #55      +/-   ##
==========================================
+ Coverage   81.44%   82.22%   +0.77%     
==========================================
  Files           6        6              
  Lines         388      405      +17     
==========================================
+ Hits          316      333      +17     
  Misses         72       72

Impacted Files	Coverage Δ
git_aggregator/repo.py	`73.04% <100.00%> (+2.15%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1047659...6e2ecdd. Read the comment docs.

joao-p-marques · 2021-10-08T13:12:39Z

CI should be fixed now...

I removed the git-autoshare part for now, and will try to work around it with the bash script as mentioned, but I still need some testing to find if it is the best solution...

sbidoul · 2021-10-08T14:29:04Z

git_aggregator/repo.py

+            repository,
+            target_dir,
+        )
+        cmd = ('git', 'clone')


Suggested change

cmd = ('git', 'clone')

cmd = ('git', 'clone', '--filter=blob:none')

And maybe you don't even need git-autoshare.

I read the docs but don't understand. What does that do?

@yajo have a look at this article.

We've been doing tests and indeed we're gonna use it. It's a great feature. However, it's also true that it doesn't fix all the problems we want to fix, so we'll use it in combination with git-autoshare.

If this suggestion is applied, IMHO the PR would be complete, as the rest of using git-autoshare is doable with the wrapper script and/or can be done in a later PR.

So could we move forward with this? Thanks.

This is the only debatable thing. Should I add it?

OK, thanks.

yajo · 2021-10-19T07:49:03Z

Can't the git command wrapper be included in git-autoshare as an extension? So we pipx install git-autoshare[git-wrapper] and get it working out of the box?

sbidoul · 2021-10-19T08:47:45Z

Can't the git command wrapper be included in git-autoshare as an extension? So we pipx install git-autoshare[git-wrapper] and get it working out of the box?

Must think about it - probably not trivial. What is sure is that it must not be a python script because of python startup time. That's why there is a bash example.

yajo · 2021-10-19T09:01:55Z

I don't think python time is a real problem, TBH. After all, you're gonna use it for git-autoshare itself and it doesn't seem to be a problem, right? You're not cloning all day after all. And bash is less maintainable and portable.

sbidoul · 2021-10-19T12:14:16Z

I don't think python time is a real problem, TBH. After all, you're gonna use it for git-autoshare itself and it doesn't seem to be a problem, right? You're not cloning all day after all. And bash is less maintainable and portable.

Trust me, adding 100ms to the startup time of your git command is a problem. I've tried.

sebastienbeau

LGTM Tested
@sbidoul let's merge it ?

sbidoul · 2022-04-06T08:34:47Z

The PR is still in draft state. @joao-p-marques is it ready ?

yajo · 2022-04-07T06:15:29Z

ping @Tardo

sbidoul · 2022-09-07T11:01:30Z

@joao-p-marques gentle nudge. Is this PR ready (it is still draft)?

pedrobaeza · 2022-09-07T12:01:56Z

Joao is no longer at Odoo world, so he can't say. Checking the PR, seems in good state and tested by several actors, so I put it as ready.

pedrobaeza · 2022-09-07T12:02:30Z

It seems I'm not able to change the PR status.

sbidoul · 2022-09-20T18:05:06Z

I added the --filter=blob:none option and merged manually.

pedrobaeza · 2022-09-20T18:08:52Z

Thanks

joao-p-marques force-pushed the use-git-autoshare branch from d745040 to 465d84c Compare October 5, 2021 12:24

yajo approved these changes Oct 6, 2021

View reviewed changes

git_aggregator/repo.py Outdated Show resolved Hide resolved

joao-p-marques force-pushed the use-git-autoshare branch 2 times, most recently from cf42778 to d99b6b9 Compare October 6, 2021 12:50

joao-p-marques marked this pull request as draft October 6, 2021 16:13

joao-p-marques force-pushed the use-git-autoshare branch from d99b6b9 to a05991e Compare October 6, 2021 16:18

Clone repo instead of init + fetch

6e2ecdd

joao-p-marques force-pushed the use-git-autoshare branch from 94caf15 to 6e2ecdd Compare October 8, 2021 13:11

sbidoul reviewed Oct 8, 2021

View reviewed changes

joao-p-marques mentioned this pull request Oct 18, 2021

Use git-autoshare with git-aggregator Tecnativa/doodba#453

Open

2 tasks

joao-p-marques mentioned this pull request Oct 19, 2021

Package suggested git script with Python package acsone/git-autoshare#15

Open

sebastienbeau approved these changes Apr 6, 2022

View reviewed changes

sbidoul merged commit 72c28c5 into acsone:master Sep 20, 2022

pedrobaeza deleted the use-git-autoshare branch September 20, 2022 18:08

PabloEForgeFlow mentioned this pull request Jun 19, 2023

[FIX] Update gitaggregator version + minor fixes ForgeFlow/odoo-filter-addons#2

Merged

sbidoul mentioned this pull request Jul 22, 2023

git-autoshare never triggered? #26

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clone repo instead of init + fetch and take advantage of git-autoshare #55

Clone repo instead of init + fetch and take advantage of git-autoshare #55

joao-p-marques commented Oct 5, 2021

sbidoul commented Oct 6, 2021

sbidoul commented Oct 6, 2021

joao-p-marques commented Oct 6, 2021

sbidoul commented Oct 6, 2021

joao-p-marques commented Oct 6, 2021

yajo commented Oct 8, 2021

codecov bot commented Oct 8, 2021 •

edited

Loading

joao-p-marques commented Oct 8, 2021

sbidoul Oct 8, 2021

yajo Oct 19, 2021

sbidoul Oct 19, 2021

yajo Nov 4, 2021

yajo Jan 12, 2022

pedrobaeza Sep 7, 2022

yajo Sep 7, 2022

pedrobaeza Sep 7, 2022

yajo commented Oct 19, 2021

sbidoul commented Oct 19, 2021

yajo commented Oct 19, 2021

sbidoul commented Oct 19, 2021

sebastienbeau left a comment

sbidoul commented Apr 6, 2022

yajo commented Apr 7, 2022

sbidoul commented Sep 7, 2022

pedrobaeza commented Sep 7, 2022

pedrobaeza commented Sep 7, 2022

sbidoul commented Sep 20, 2022

pedrobaeza commented Sep 20, 2022

	cmd = ('git', 'clone')
	cmd = ('git', 'clone', '--filter=blob:none')

Clone repo instead of init + fetch and take advantage of git-autoshare #55

Clone repo instead of init + fetch and take advantage of git-autoshare #55

Conversation

joao-p-marques commented Oct 5, 2021

sbidoul commented Oct 6, 2021

sbidoul commented Oct 6, 2021

joao-p-marques commented Oct 6, 2021

sbidoul commented Oct 6, 2021

joao-p-marques commented Oct 6, 2021

yajo commented Oct 8, 2021

codecov bot commented Oct 8, 2021 • edited Loading

Codecov Report

joao-p-marques commented Oct 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yajo commented Oct 19, 2021

sbidoul commented Oct 19, 2021

yajo commented Oct 19, 2021

sbidoul commented Oct 19, 2021

sebastienbeau left a comment

Choose a reason for hiding this comment

sbidoul commented Apr 6, 2022

yajo commented Apr 7, 2022

sbidoul commented Sep 7, 2022

pedrobaeza commented Sep 7, 2022

pedrobaeza commented Sep 7, 2022

sbidoul commented Sep 20, 2022

pedrobaeza commented Sep 20, 2022

codecov bot commented Oct 8, 2021 •

edited

Loading