Implement `jj sync` command #1039

dbarnett · 2023-01-14T22:06:22Z

jj should have a command to sync all of my branches (or some subset) with a remote via merge or rebase, in a single operation.

Since that could cause a lot of big changes at once, it'd be nice to show a preview of changes that will be performed and prompt to confirm and/or output a very clear summary of what all changed from that sync operation.

dbarnett · 2023-01-14T22:16:12Z

I could also imagine some related curation tools for e.g. moving a local branch to "draft mode" to not be published or synced yet.

dbarnett · 2023-01-14T22:33:12Z

This is also one of the areas jj was dinged in the comparison table at https://blog.waleedkhan.name/git-ui-features/#data.

MForster · 2023-01-15T09:34:42Z

IMHO sync is ambiguous about the direction. Does it only pull or also push?

PhilipMetzger · 2023-01-15T17:16:01Z

I think jj sync should only pull, like its Perforce predecessor. If that isn't clear enough, we can alias the command for Git users to jj pull

martinvonz · 2023-01-15T17:25:03Z

IMHO sync is ambiguous about the direction. Does it only pull or also push?

Hmm, that's a good point. On the other hand, I don't think we've had any confusion about that at Google with our hg sync command, which does pull+rebase. I'm not sure what a better name would be.

we can alias the command for Git users to jj pull

I think Git's use of "pull" for "fetch+rebase/merge" is quite confusing. I prefer hg's use of hg pull for just pulling (like git fetch). That is more symmetric with hg/git push.

MForster · 2023-01-15T17:27:45Z

I don't think we've had any confusion about that at Google with our hg sync command

That's not representative, because ~everybody at Google has used Perforce/Piper.

FWIW, VSCode has a "Sync" button that pulls and pushes...

martinvonz · 2023-01-15T17:35:46Z

That's not representative, because ~everybody at Google has used Perforce/Piper.

True. (I have practically not used it, but I still have run p4 sync several times.)

FWIW, VSCode has a "Sync" button that pulls and pushes...

Oh, that's a good data point!

We'd still have to think of a better name then... There's hg fetch that does this. I don't know if they named it that as a joke, given that it means that hg pull == git fetch and hg fetch == git pull. So I don't really think we should call it jj fetch.

PhilipMetzger · 2023-01-15T17:42:57Z

IMHO, jj sync should be good enough. We then should mention, that it's inspired by Perforce's p4 sync. If we then want to accommodate Mercurial and Git users, we can alias it to their equivalent.
I think we should approach Editor integration with our own tools and assumptions and disregard the current state for Git and Mercurial.

dbarnett · 2023-01-16T16:32:08Z

Graphite also uses "sync" to refer to this same kind of "pull-and-restack" operation: https://docs.graphite.dev/guides/graphite-cli/syncing-and-resolving-conflicts

Do we envision jj ever would support that kind of bidirectional "pull + push"? If so, maybe we have a command called sync that defaults to --in mode with eventual support for --in-out mode, or something like that? (Hard for me to imagine wanting such a shotgun command though, personally.)

Ralith · 2023-01-16T19:25:46Z

That's not representative, because ~everybody at Google has used Perforce/Piper.

I skipped Piper and went straight to Fig, and found hg sync to be plenty intuitive. Agreed that doing both at once seems weird and error-prone.

ilyagr · 2023-02-20T08:30:42Z

Now that #1288 is in, you can get part of the way to jj sync with a command like jj rebase -s oldmain+~:main -d main -L. This is, of course, not particularly pretty and requires finding the oldmain commit.

Another, simpler, alternative if you're OK with rebasing everything not on main: jj rebase -s roots(main..) -d main.

PhilipMetzger · 2023-09-28T10:21:27Z

We discussed jj sync multiple times in the recent months in Discord as there is one big issue to solve: backend independence.

Summary:
Ideally jj sync should be a un-namespaced command (without a backend prefix) but this makes it difficult to determine which implementation to use. @martinvonz's suggestion in an internal chat and on Discord was to make the command forge specific, e.g jj git sync, jj gerrit sync or jj piper sync and then allow the unnamespaced sync command to wrap it. This means that for the most users, jj sync would be jj git sync. Another option is to make that a buiiltin alias for each backend.

While I was initially sceptical of the namespaced approach, I now think it is the right solution for this problem. The only issue I see, if we take the forge-specific approach, is that some layering issue may pop up. But if we implement something like jj submit or jj mail, a forge trait is needed anyway.

We will need this in the `jj git sync` command to perform a fetch that behaves the same as the `jj git fetch` command. Fetch extract the function in this commit, so diffs are easier to review. A later commit will move this function (and associated functions) out to cli/src/git_util.rs Part of: #1039

We will need this in the `jj git sync` command to perform a fetch that behaves the same as the `jj git fetch` command. Part of: #1039

* map old heads -> new heads * rebase * roots are candidates with parents with updated heads. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

* First commit that just sets up scaffolding. * `jj git sync` is hidden as it's not yet ready. Issue: #1039

* map old heads -> new heads * rebase * roots are candidates with parents with updated heads. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [ ] Rebase * [X] Build old -> new map * [ ] transform descendants ## Details Build a record of old branch heads mapped to the new branch heads. This record will be used to derive the candidates that need rebasing. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants ## Details * Add tests * Unhide the command, now that it's complete. * Update CHANGELOG Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [ ] Rebase * [X] Build old -> new map * [ ] transform descendants ## Details Build a record of old branch heads mapped to the new branch heads. This record will be used to derive the candidates that need rebasing. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants ## Details * implement CommitRewritter::repo() to return an immutable reference. * transform_descendants() roots are candidates with parents in update_record. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants ## Details * Add tests * Unhide the command, now that it's complete. * Update CHANGELOG Issue: #1039

Similarly to `git_fetch`, this will be used in `jj git sync`, so moving this out to maintain the same behaviour across `fetch` command and `sync`. * Refactor out `get_fetch_remotes` to `git_util.rs`. * inline the different `get_*_remote*` functions into `get_fetch_remotes`; The resulting function is still small enough to be easily readable. * Move `get_single_remote` into `git_util.rs` and update call sites. * Use common FetchArgs to pass arguments both to `get_fetch_remotes` and `git_fetch` * Update use references All tests still pass. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants # git sync: Setup scaffolding ## Details * First commit that just sets up scaffolding. * `jj git sync` is hidden as it's not yet ready. * Set up some tracing guards which will help debugging when run with --debug. # git sync: Get branch heads and fetch from remote. ## Details * For branch heads, we grab local branches matching the pattern specified in args. * Pre-fetch heads query tx.base_repo() while post-fetch heads query tx.repo(); tx.repo() is the in memory Mutable repo which is updated after a fetch in a transaction. * We fetch from all branches from all configured remotes. --all_remotes affects if we use `git.fetch` or all repo remotes. # git sync: Build candidate commits. ## Details * Candidate commits (parent_id, child_id) are: * branch commits * and their descendants * which are not found in untracked remote branches # git sync: Build update record ## Details Build a record of old branch heads mapped to the new branch heads. This record will be used to derive the candidates that need rebasing. # git sync: unhide `jj git sync` ## Details * Add tests * Unhide the command, now that it's complete. * Update CHANGELOG # git sync: Rebase commits onto new branch heads. ## Details * implement CommitRewritter::repo() to return an immutable reference. * transform_descendants() roots are candidates with parents in update_record. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

Similarly to `git_fetch`, this will be used in `jj git sync`, so moving this out to maintain the same behaviour across `fetch` command and `sync`. * Refactor out `get_fetch_remotes` to `git_util.rs`. * inline the different `get_*_remote*` functions into `get_fetch_remotes`; The resulting function is still small enough to be easily readable. * Move `get_single_remote` into `git_util.rs` and update call sites. * Use common FetchArgs to pass arguments both to `get_fetch_remotes` and `git_fetch` * Update use references All tests still pass. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants # git sync: Setup scaffolding ## Details * First commit that just sets up scaffolding. * `jj git sync` is hidden as it's not yet ready. * Set up some tracing guards which will help debugging when run with --debug. # git sync: Get branch heads and fetch from remote. ## Details * For branch heads, we grab local branches matching the pattern specified in args. * Pre-fetch heads query tx.base_repo() while post-fetch heads query tx.repo(); tx.repo() is the in memory Mutable repo which is updated after a fetch in a transaction. * We fetch from all branches from all configured remotes. --all_remotes affects if we use `git.fetch` or all repo remotes. # git sync: Build candidate commits. ## Details * Candidate commits (parent_id, child_id) are: * branch commits * and their descendants * which are not found in untracked remote branches # git sync: Build update record ## Details Build a record of old branch heads mapped to the new branch heads. This record will be used to derive the candidates that need rebasing. # git sync: unhide `jj git sync` ## Details * Add tests * Unhide the command, now that it's complete. * Update CHANGELOG # git sync: Rebase commits onto new branch heads. ## Details * implement CommitRewritter::repo() to return an immutable reference. * transform_descendants() roots are candidates with parents in update_record. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

Similarly to `git_fetch`, this will be used in `jj git sync`, so moving this out to maintain the same behaviour across `fetch` command and `sync`. * Refactor out `get_fetch_remotes` to `git_util.rs`. * inline the different `get_*_remote*` functions into `get_fetch_remotes`; The resulting function is still small enough to be easily readable. * Move `get_single_remote` into `git_util.rs` and update call sites. * Use common FetchArgs to pass arguments both to `get_fetch_remotes` and `git_fetch` * Update use references All tests still pass. Issue: #1039

## Summary * [X] Get branch pre-fetch heads * [X] Build candidates from prefetch heads * [X] Fetch from remotes * [X] Get branch post-fetch heads * [X] Rebase * [X] Build old -> new map * [X] transform descendants # git sync: Setup scaffolding ## Details * First commit that just sets up scaffolding. * `jj git sync` is hidden as it's not yet ready. * Set up some tracing guards which will help debugging when run with --debug. # git sync: Get branch heads and fetch from remote. ## Details * For branch heads, we grab local branches matching the pattern specified in args. * Pre-fetch heads query tx.base_repo() while post-fetch heads query tx.repo(); tx.repo() is the in memory Mutable repo which is updated after a fetch in a transaction. * We fetch from all branches from all configured remotes. --all_remotes affects if we use `git.fetch` or all repo remotes. # git sync: Build candidate commits. ## Details * Candidate commits (parent_id, child_id) are: * branch commits * and their descendants * which are not found in untracked remote branches # git sync: Build update record ## Details Build a record of old branch heads mapped to the new branch heads. This record will be used to derive the candidates that need rebasing. # git sync: unhide `jj git sync` ## Details * Add tests * Unhide the command, now that it's complete. * Update CHANGELOG # git sync: Rebase commits onto new branch heads. ## Details * implement CommitRewritter::repo() to return an immutable reference. * transform_descendants() roots are candidates with parents in update_record. * simplify parent merge * drop newly emptied commits * update new parents from the updated heads set if the old parents are ancestors. Issue: #1039

dbarnett added the enhancement New feature or request label Jan 14, 2023

rslabbert mentioned this issue Mar 23, 2023

Add jj git push --stack to push all working branches #1415

Open

PhilipMetzger added the polish🪒🐃 Make existing features more convenient and more consistent label Sep 28, 2023

martinvonz mentioned this issue Sep 28, 2023

Drop commits that were made empty by rebase #229

Closed

martinvonz mentioned this issue Nov 2, 2023

FR: make fetch automatic rebase behavior optional #2504

Closed

martinvonz added the good first issue Good for newcomers label Nov 13, 2023

martinvonz self-assigned this Jan 22, 2024

PhilipMetzger mentioned this issue Feb 4, 2024

FR: add jj git fetch --quiet #2944

Open

jyn514 mentioned this issue Feb 7, 2024

FR: abandon changes that have been merged into trunk #2978

Closed

PhilipMetzger mentioned this issue Feb 12, 2024

FR: --advance-branches flag for jj commit #2338

Open

PhilipMetzger unassigned martinvonz Oct 10, 2024

essiene mentioned this issue Oct 14, 2024

FR: Implement jj git sync #4641

Closed

essiene self-assigned this Oct 14, 2024

arxanas mentioned this issue Oct 24, 2024

FR: Generalizable and reusable "rebase behavior" #4708

Open

essiene mentioned this issue Nov 4, 2024

cli: Refactor a reusable git_fetch function from the command. #4767

Merged

essiene added a commit that referenced this issue Nov 4, 2024

git fetch: Refactor a reusable git_fetch to git_util.rs

3a16b88

We will need this in the `jj git sync` command to perform a fetch that behaves the same as the `jj git fetch` command. Part of: #1039

essiene mentioned this issue Nov 19, 2024

git sync: Build candidate commits. #4910

Closed

5 tasks

essiene added a commit that referenced this issue Nov 19, 2024

git sync: Setup scaffolding

4e46350

* First commit that just sets up scaffolding. * `jj git sync` is hidden as it's not yet ready. Issue: #1039

essiene mentioned this issue Nov 19, 2024

git sync: Build update record #4911

Closed

7 tasks

This was referenced Nov 19, 2024

git sync: Rebase commits onto new branch heads. #4913

Closed

git sync: unhide jj git sync #4914

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `jj sync` command #1039

Implement `jj sync` command #1039

dbarnett commented Jan 14, 2023

dbarnett commented Jan 14, 2023

dbarnett commented Jan 14, 2023 •

edited

Loading

MForster commented Jan 15, 2023

PhilipMetzger commented Jan 15, 2023

martinvonz commented Jan 15, 2023

MForster commented Jan 15, 2023

martinvonz commented Jan 15, 2023

PhilipMetzger commented Jan 15, 2023

dbarnett commented Jan 16, 2023

Ralith commented Jan 16, 2023

ilyagr commented Feb 20, 2023 •

edited

Loading

PhilipMetzger commented Sep 28, 2023 •

edited

Loading

Implement jj sync command #1039

Implement jj sync command #1039

Comments

dbarnett commented Jan 14, 2023

dbarnett commented Jan 14, 2023

dbarnett commented Jan 14, 2023 • edited Loading

MForster commented Jan 15, 2023

PhilipMetzger commented Jan 15, 2023

martinvonz commented Jan 15, 2023

MForster commented Jan 15, 2023

martinvonz commented Jan 15, 2023

PhilipMetzger commented Jan 15, 2023

dbarnett commented Jan 16, 2023

Ralith commented Jan 16, 2023

ilyagr commented Feb 20, 2023 • edited Loading

PhilipMetzger commented Sep 28, 2023 • edited Loading

Implement `jj sync` command #1039

Implement `jj sync` command #1039

dbarnett commented Jan 14, 2023 •

edited

Loading

ilyagr commented Feb 20, 2023 •

edited

Loading

PhilipMetzger commented Sep 28, 2023 •

edited

Loading