Git tools that enable easy bidirectional sync between multiple repositories
mgt
is a command line tool that makes it easier to maintain a monorepo and contribute code back and forth with other git repositories. It only depends on git
being installed, and works on Windows, Mac, and Linux.
Most commands in mgt
depend on a repo_file
which is a toml file that defines how your repository should be mapped.
[repo]
remote = "https://github.com/nikita-skobov/gitfilter"
[include_as]
"lib/gitfilter/" = " "
"lib/exechelper/" = "exechelper/"
"lib/die/" = "die/"
"lib/gumdrop/" = "gumdrop/"
The above would create the following mapping between your local code, and code that you would push somewhere else (such as Github for example):
Local repo Github repo
================================================
lib/ die/
gitfilter/ exechelper/
README.md gumdrop/
src/ src/
exechelper/ README.md
die/
gumdrop/
other/
libs/
doc/
Note that there are other folders in the local repo that do not get mapped to the Github repo because they were not defined in the [include_as]
section. mgt
allows you to pick and choose what part of your git history you want to share publicly
- Easily add remote projects to your private codebase
- Add only parts of remote projects to your private codebase
- Split out part of your repository to share publicly
- Conveniently push recent changes to a public repo
If you have a local repository, and you want to bring in an external dependency, but you want the actual source code of the dependency (instead of just consuming it as a package from a package manager), then you can use mgt
to conveniently add the whole remote repository to a subdirectory of your local repository:
mgt split-in-as --as lib/animals/cat/ https://github.com/user/supercat
The split-in-as
command will take a remote repository, in this case a Github URL, and it takes a --as <FOLDER>
argument which maps the entirety of the user/supercat
repository to a folder that we define. If this folder doesn't exist, it will be created for you. Here are a before and after diagram of what this command would do:
# Before running split-in-as:
Local repo Github supercat
================================================
lib/ index.js
private/ README.md
LICENCE
# After running split-in-as:
Local repo Github supercat
================================================
lib/ index.js
animals/ README.md
cat/ LICENSE
README.md
LICENSE
index.js
private/
We can also add the --gen-repo-file
option to generate a file that contains the mapping that was performed. This repo file can then be used to modify how we want our future mapping to work, or to perform bidirectional sync by specifying a file instead of typing out the --as lib/animals/cat <github url>
every time. If we ran the following command:
mgt split-in-as --gen-repo-file --as lib/animals/cat https://github.com/user/supercat
Then mgt
would perform the same operation as earlier, but it would also generate a repo file in our current directory called supercat.rf
(it uses the name of the repository being split in). This repo file would have the following contents:
[repo]
name = "supercat"
remote = "https://github.com/user/supercat"
[include_as]
"lib/animals/cat/" = " "
If we wanted to sync in the future, we could just specify this file, instead of specifying the extra command line arguments we did before. ie: we could get latest updates with:
# notice it is not split-in-as:
mgt split-in supercat.rf
Many repositories out there have tons of code in them. Often times there are repositories that are really many projects combined into one. We sometimes might want to only take parts of remote projects without getting parts that we don't care about. To do this, we have to explicitly define a repo file that contains exactly the mapping that we want. Let's write one below:
# bigproject.rf
[repo]
remote = "https://github.com/user/bigproject
[include_as]
"lib/sound/" = "src/sound/"
exclude = ["src/sound/samples/"]
We can then use this repo file to split-in
the remote project into our local repo. We can do that via:
mgt split-in bigproject.rf --rebase
Note that the --rebase
command will do a simple git rebase
for us after the split-in
happens. There should not be any rebase conflicts as long as the lib/sound/
folder does not already exist with git history. Otherwise, you would have to resolve those conflicts.
Let's look at a before and after diagram of what this command would do:
# Before running split-in:
Local repo Github bigproject
================================================
lib/ src/
private/ projectA/
projectB/
sound/
code/
samples/
bigfile.mp3
README.md
# After running split-in:
Local repo Github bigproject
================================================
lib/ src/
private/ projectA/
sound/ projectB/
code/ sound/
README.md code/
samples/
bigfile.mp3
README.md
We notice that we only take the src/sound/
folder, and ignore the other projects from the bigproject
repo. And also, we explicitly exclude samples/
because it has big audio files that we don't care about.
Consider a repository that has some private code as well as some code you pulled in from an open source project.
# apples.rf
[repo]
remote = "https://github.com/user/my-fork-of-apples"
[include_as]
"apples/" = " "
exclude = ["apples/private-notes.txt"]
By running: mgt split-out apples.rf
mgt
will create a new branch called my-fork-of-apples
and it will have entirely new git history where the file structure will now look like:
Local repo Github my-fork-of-apples
================================================
private-folder/ README.md
apples/ index.js
README.md
index.js
private-notes.txt
private-file.txt
Note that your private files and folders do not get shared because they were not included in the repo file mapping. Also it is interesting to note that apples/private-notes.txt
does not get included because we explicitly exclude it in our repo file, but everything else in the apples/
folder did get included.
Continuing from our previous example, we now have a my-fork-of-apples
branch, and it has some changes in it that we want to share with our public Github fork. We could do the following:
git pull --rebase https://github.com/user/my-fork-of-apples
But that would only work in limited cases where our recently split-out branch is compatible with the remote version. Consider if we have the following git history:
Local repo Github my-fork-of-apples
================================================
aaaaaaaa fixed bug 1
bbbbbbbb improved docs yyyyyyyy improved docs
cccccccc copied apples zzzzzzzz initial commit
Our local repository has different commit hashes for the "improved docs" but we see that the commit message is the same. Internally, we can analyze the git blobs and trees and find out that really that commit has the same blobs but maybe different trees, which would cause the commit hash to be different. If we know that bbbb
and yyyy
are essentially the same commit, we can then say: "let's only apply the most recent aaaa commit on top of yyyy", which we can do by interactively rebasing, and then only picking the commit that we want (ie: `git pull --rebase=interactive https://github.com/user/my-fork-of-apples)
Alternatively mgt
provides a convenient way to calculate this fork point, and do this interactive rebase for you. Consider if instead of running the mgt split-out apples.rf
command from before, we ran this instead:
mgt split-out apples.rf --topbase
the --topbase
flag will tell mgt
to fetch my-fork-of-apples
into a temporary branch, compute the fork point from the split-out version of apples, and then apply an interactive rebase using the calculated fork point. it is called topbase
because it calculates the top point of where the rebase should happen.
If we ran the topbase
command shown above, we would now have the following history on the my-fork-of-apples
local branch:
Local repo Github my-fork-of-apples
================================================
aaaaaaaa fixed bug 1
yyyyyyyy improved docs yyyyyyyy improved docs
zzzzzzzz initial commit zzzzzzzz initial commit
Now we see that our local repository is compatible with the github repository, and we are ahead by 1 commit, so we can just do a normal push:
git push https://github.com/user/my-fork-of-apples HEAD:some-branch
# and then you can do a pull request in Github to merge some-branch
# or alternatively you can just push directly to the main branch
You will need rust installed
git clone https://github.com/nikita-skobov/monorepo-git-tools
cd monorepo-git-tools
cargo build --release
chmod +x ./target/release/mgt
cp ./target/release/mgt /usr/bin/mgt
This project builds and tests every release on GitHub
To get the latest binary, go to the release page and download the zip file that is appropriate for your machine.
To install it, simply extract the zip, and copy the executable to a directory that is on your system's path.
The full command line usage documentation can be found here