GitLab repository with 8638 commits ahead can't be handled

[Byron]

Version

0.13.1

Operating System

macOS

Distribution Method

dmg (Apple Silicon)

Describe the issue

After not touching the GitLab repository for a while, it's very far behind.

However, the initial spinner takes about 30 minutes to disappear so the virtual branch is shown.
I did not manage to update the workspace successfully, and the new 'incoming changes' popup seemed to be taking a lot of resources from the browser window so it was pegged at 1CPU at all times. Note that it really is only busy while the popup is up (and spinning), even though it just disappeared when I clicked outside of it, despite still spinning.

It's notable that when it finishes, it shows a lot of changed files (~7800 according to git status | wc -l)

To show ~8000 files, the frontend also uses about 4.5GB of RAM.

I think this is a regression as even dealing with remotes being 5000 commits ahead of the local branches was absolutely possible previously.

Additional information

• The backend profile showing that diff-tree-to-tree takes most of the time.
• WebContent rendering once the backend is done
• The profiles themselves from which the screenshots were taken macos-instruments-profiles.zip
• After bringing the initial processing time down to ~4m (the time when only the spinner is visible), I pressed the Update button once again. This time, after waiting a while at 100% CPU it came back with this (Chosen resolution do not match…):
• The memory usage is very high in the frontend, nearly 4 real GB, even though this seems to be a small improvement

How to reproduce

Use the GitLab repository (https://gitlab.com/gitlab-org/gitlab), create a workspace, and wait until the remote is ahead - it will get slower and slower presumably the more commits the remote is ahead.
It should be possible to unzip the following archive into .git/gitbutler for the same effect, maybe after fiddling with the local branches. I copied them here as well.

• dot-git-gitbutler.zip
• dot-git-refs-and-index.zip
• It's indeed enough to set the target branch hash to d19faa23359d2d19bd7dace707d80210de74f0a6 for example to have a decent amount of commits ahead to see the lagging.

Expected behavior

The UI be fast enough to work on a repo of that size.

Relevant log output

Notable is list_virtual_branches() with 10 minutes on the clock: list_virtual_branches [ 360s | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a.

❯ GITBUTLER_PERFORMANCE_LOG=1 /Applications/GitButler.app/Contents/MacOS/GitButler
INFO     ｉ [info]: system git executable for fetch/push: "git"
INFO     ｉ [info]: starting app | version: 0.13.1 | name: GitButler
2024-10-16 14:01:24.387 GitButler[70367:17766425] +[IMKClient subclass]: chose IMKClient_Legacy
INFO     list_projects [ 129µs | 100.00% ]
INFO     get_user [ 38.4ms | 100.00% ]
2024-10-16 14:01:24.488 GitButler[70367:17766425] +[IMKInputSession subclass]: chose IMKInputSession_Legacy
INFO     get_project [ 984µs | 100.00% ] id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | no_validation: None
INFO     git_head [ 1.44ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     operating_mode [ 298µs | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     list_branches [ 260ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | filter: Some(BranchListingFilter { local: None, applied: None })
INFO     get_base_branch_data [ 340ms | 0.03% / 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     ┕━ get_base_branch_data [ 339ms | 99.97% ]
INFO     get_base_branch_data [ 367ms | 0.03% / 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     ┕━ get_base_branch_data [ 367ms | 99.97% ]
2024-10-16 14:01:26.222 GitButler[70367:17766425] _TIPropertyValueIsValid called with 16 on nil context!
2024-10-16 14:01:26.222 GitButler[70367:17766425] imkxpc_getApplicationProperty:reply: called with incorrect property value 16, bailing.
2024-10-16 14:01:26.222 GitButler[70367:17766425] Text input context does not respond to _valueForTIProperty:
INFO     list_local_branches [ 227ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     list_branches [ 251ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | filter: Some(BranchListingFilter { local: None, applied: None })
INFO     get_branch_listing_details [ 3.02s | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | branch_names: ["Virtual-branch", "482733-add-planner-role", "479586-resolve-cross-join-issues-in-gitlab-ee-app-services-vulnerabilities-create_service-rb", "477558-resolve-vulnerability_reads-cross-join-issues", "494294_update_pull_mirror_endpoint", "463854-import-sharding-bulk_import_exports", "master-i18n", "461787-finalize-project-id-sharding-key", "add-concurrency-limit-to-sidekiq-dev-docs"]
INFO     handle [ 176µs | 100.00% ] event: GitFileChange(a4dfef20-d58e-47d8-984d-9f78eb012c2a, FETCH_HEAD, logs/HEAD)
INFO     get_base_branch_data [ 318ms | 0.05% / 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     ┕━ get_base_branch_data [ 318ms | 99.95% ]
INFO     list_local_branches [ 221ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     list_branches [ 246ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | filter: Some(BranchListingFilter { local: None, applied: None })
INFO     get_branch_listing_details [ 170ms | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | branch_names: ["nd/work-item-epics-related-links-ssot", "458197-ignore-confidence-columns-on-vulnerability", "473207-create-multiple-indices-on-big-namespace-rollout"]
INFO     fetch_from_remotes [ 15.7s | 98.04% / 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | action: Some("auto")
INFO     ┕━ get_base_branch_data [ 308ms | 1.96% ]
INFO     list_virtual_branches [ 360s | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a
INFO     normalize_branch_name [ 7.38µs | 100.00% ] name: "Virtual branch"
INFO     normalize_branch_name [ 1.38µs | 100.00% ] name: "Virtual branch"
INFO     git_get_global_config [ 205µs | 100.00% ] key: "gitbutler.aiModelProvider"
INFO     git_get_global_config [ 229µs | 100.00% ] key: "gitbutler.aiModelProvider"
INFO     list_branches [ 2.43s | 100.00% ] project_id: a4dfef20-d58e-47d8-984d-9f78eb012c2a | filter: Some(BranchListingFilter { local: None, applied: None })
INFO     normalize_branch_name [ 8.21µs | 100.00% ] name: "Virtual branch"
INFO     secret_get_global [ 10.6ms | 100.00% ] handle: "aiOpenAIKey"
INFO     secret_get_global [ 10.6ms | 100.00% ] handle: "aiOpenAIKey"
INFO     git_get_global_config [ 562µs | 100.00% ] key: "gitbutler.aiOpenAIKey"
INFO     git_get_global_config [ 514µs | 100.00% ] key: "gitbutler.aiOpenAIKey"
INFO     secret_get_global [ 4.57ms | 100.00% ] handle: "aiAnthropicKey"
INFO     secret_get_global [ 4.59ms | 100.00% ] handle: "aiAnthropicKey"
INFO     git_get_global_config [ 238µs | 100.00% ] key: "gitbutler.aiAnthropicKey"
INFO     git_get_global_config [ 170µs | 100.00% ] key: "gitbutler.aiAnthropicKey"
INFO     git_get_global_config [ 155µs | 100.00% ] key: "gitbutler.aiOllamaEndpoint"
INFO     git_get_global_config [ 104µs | 100.00% ] key: "gitbutler.aiOllamaEndpoint"
INFO     git_get_global_config [ 150µs | 100.00% ] key: "gitbutler.aiOllamaModelName"
INFO     git_get_global_config [ 248µs | 100.00% ] key: "gitbutler.aiOllamaModelName"
INFO     git_get_global_config [ 196µs | 100.00% ] key: "gitbutler.aiModelProvider"
INFO     git_get_global_config [ 118µs | 100.00% ] key: "gitbutler.aiModelProvider"
INFO     git_get_global_config [ 120µs | 100.00% ] key: "gitbutler.aiOpenAIKeyOption"
INFO     git_get_global_config [ 100µs | 100.00% ] key: "gitbutler.aiOpenAIKeyOption"
INFO     git_get_global_config [ 88.5µs | 100.00% ] key: "gitbutler.aiAnthropicKeyOption"
INFO     git_get_global_config [ 109µs | 100.00% ] key: "gitbutler.aiAnthropicKeyOption"
2024-10-16 14:08:53.924 GitButler[70367:17766425] _TIPropertyValueIsValid called with 16 on nil context!
2024-10-16 14:08:53.924 GitButler[70367:17766425] imkxpc_getApplicationProperty:reply: called with incorrect property value 16, bailing.
2024-10-16 14:08:53.924 GitButler[70367:17766425] Text input context does not respond to _valueForTIProperty:

Related PRs

• use gitoxide for merging trees #5411
  • reduces the time to show a status from 86m to ~3m5s

Other TODOs

• Check if multiple merge-bases alleviate the time-based cutoff in this repository, see actions/branch.rs:575 for the revwalk to alter.

[nir-tsabari]

Having similar issues, very large repo, initial load takes long, 500 commits can't be handled, and sometimes even 30 commits can't be handled.
Github, Windows 11 x64, 0.13.8

[Byron]

I hope that this weekend I have some new numbers to share here :).

[Byron]

@nir-tsabari In the latest nighty build there are improvements that will significantly speed up some operations (~30x). However, it's still difficult to get the workspace update to work at least in my case, as it usually errors out.

This is particular issue is very much related to the frontend, and to my mind the introduction of the new "Incoming Changes" popup is causing this. CC @mtsgrd who might now how to accelerate the UI - maybe it's rendering a list of 5k items unconditionally into the DOM? The browser seems very busy computing the layout, somehow.

Edit: I was able to select Stash in the conflict popup which took a lot of time to finish… but it did finish, maybe 3 minutes later. Now my GItLab repo is uptodate again. On another note, after squashing (which seems to be unapplying the virtual branch) the virtual-branches.toml file has 5.5MB). It's indeed enough to set the target branch hash to d19faa23359d2d19bd7dace707d80210de74f0a6 for example to have a decent amount of commits ahead to see the lagging.