Convert the submodule star-sys/STAR to a git subtree #29

sjackman · 2020-08-13T14:58:09Z

Remove the Git submodule star-sys/STAR.
Add the Git subtree star-sys/STAR/source.

Thanks to Lance for this suggestion!

Files under star-sys/STAR that are not under star-sys/STAR/source have been removed, in particular the large binaries in star-sys/STAR/bin. There's thankfully no need to rewrite the history of Orbit. Since star-sys/STAR was a Git submodule, once it has been removed, its Git objects will not be fetched by default. The Git commit history of STAR, and the Orbit-specific changes to STAR, has been preserved.

This change addresses two issues:

the ~500 MB of large binaries in star-sys/STAR/bin
the cargo bug git submodules are not cached git submodules are not cached rust-lang/cargo#7987

The challenges are that git subtree is an uncommon tool and will be unfamiliar to most developers, and merging upstream STAR into Orbit is somewhat more involved. https://gist.github.com/tswaters/542ba147a07904b1f3f5#fetching-upstram

See https://www.atlassian.com/git/tutorials/git-subtree
and https://stackoverflow.com/questions/23937436/add-subdirectory-of-remote-repo-with-git-subtree
and https://jrsmith3.github.io/merging-a-subdirectory-from-another-repo-via-git-subtree.html
and https://gist.github.com/tswaters/542ba147a07904b1f3f5

…ctory path) for the file names in --readFilesIn . Implemented read group ID output as the last column of the Chimeric.out.junction file.

Don't exit when fastq contains a zero-length read

…alized memory access. The chimeric output may change for a very small number of reads.

…r read.

…e values.

…nique (i.e. NH=1).

…ing bins. Icnreasing this number reduces the amount of RAM required for sorting.

…led executables.

…it, use --peOverlapNbasesMin and --peOverlapMMp oprions.

…E reads.

…ends of the mates.

Add -Werror to CXXFLAGS. Remove register keyword. Comment out unused function arguments.

…8ebe672d5ae7' git-subtree-dir: star-sys/STAR/source git-subtree-mainline: 05d6c8b git-subtree-split: 85c4977

…' into sj/subtree

pmarks

I will have to learn the details at some but, but definitely seems like this is worth a shot.

pmarks

Oh -- we should move the license file from STAR to the top of the subtree. Don't want to elide the license accidentally.

nlhepler · 2020-08-13T15:36:22Z

I agree w/ Pat re: License. I may also be easier to just copy into place. Otherwise, dope!

sjackman · 2020-08-13T15:58:17Z

Good catch, Pat. I've added star-sys/STAR/LICENSE.

nlhepler

👌 SHIP IT 👌

sjackman · 2020-08-13T16:11:40Z

$ time git clone https://github.com/10XGenomics/orbit
Cloning into 'orbit'...
remote: Enumerating objects: 3997, done.
remote: Counting objects: 100% (3997/3997), done.
remote: Compressing objects: 100% (969/969), done.
remote: Total 4378 (delta 2942), reused 3988 (delta 2940), pack-reused 381
Receiving objects: 100% (4378/4378), 7.36 MiB | 3.73 MiB/s, done.
Resolving deltas: 100% (3112/3112), done.
git clone https://github.com/10XGenomics/orbit orbit-subtree  0.39s user 0.16s system 19% cpu 2.787 total 13504 MB
$ du -hs orbit
 10M	orbit-subtree

😁

adam-azarchs · 2020-08-13T19:51:59Z

LGTM, though it would be good to include a script for updating the subtree.

sjackman · 2020-08-13T20:21:26Z

# Extract the source/ subdirectory of STAR into a Git subtree.
orbit-subtree:
	git clone https://github.com/10XGenomics/orbit orbit-subtree
	git -C orbit-subtree remote add star https://github.com/10XGenomics/STAR
	git -C orbit-subtree fetch star

	# Remove the STAR submodule.
	git -C orbit-subtree switch -c orbit/subtree remotes/origin/master
	git -C orbit-subtree rm star-sys/STAR .gitmodules
	git -C orbit-subtree commit -m 'Remove the submodule star-sys/STAR'

	# Create the subtree of star/master.
	git -C orbit-subtree switch -c star/master remotes/star/master
	git -C orbit-subtree subtree split -P source -b star/subtree

	# Add the subtree of star/master to orbit.
	git -C orbit-subtree switch orbit/subtree
	git -C orbit-subtree subtree add -P star-sys/STAR/source star/subtree

	# Create the subtree of star/orbit.
	git -C orbit-subtree switch -c star/orbit remotes/star/orbit
	git -C orbit-subtree subtree split -P source -b star/subtree

	# Add the subtree of star/orbit to orbit.
	git -C orbit-subtree switch orbit/subtree
	git -C orbit-subtree subtree merge -P star-sys/STAR/source star/subtree

	# Check for large files.
	git -C orbit-subtree lfs migrate info --include-ref=orbit/subtree --above=1MB
	git -C orbit-subtree rev-list --objects orbit/subtree \
	| git -C orbit-subtree cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(rest)' \
	| awk '$$2 == "blob" && $$3 >= 1000000 { print substr($$1, 1, 7), $$3, $$4; fail=1 } END { exit fail }'

Here's the script used to create the subtree.

sjackman · 2020-08-13T20:23:33Z

# Check for large files.
$ git -C orbit-subtree lfs migrate info --include-ref=orbit/subtree --above=1MB
migrate: Sorting commits: ..., done.                                            
migrate: Examining commits: 100% (671/671), done.                               
STAR      	7.4 MB	4/4 files(s)	100%
*.a       	6.2 MB	2/2 files(s)	100%
STARstatic	3.6 MB	1/1 files(s)	100%
$ git -C orbit-subtree rev-list --objects orbit/subtree \
	| git -C orbit-subtree cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(rest)' \
	| awk '$2 == "blob" && $3 >= 1000000 { print substr($1, 1, 7), $3, $4; fail=1 } END { exit fail }'
40b79fe 3106550 liborbit.a
9b28e14 3097906 orbit.a
8a26056 1793054 STAR
5f27ff2 1792798 STAR
43e95fd 3646356 STARstatic
e532a98 1792790 STAR
9c8e339 2019903 STAR

Note that there are 17 MB of large binary files stored in STAR/source/. It's not in my opinion worth rewriting history to either remove these large files or store them in Git LFS, as it would significantly complicate the process of merging STAR/master into Orbit.

sjackman · 2020-08-13T20:24:21Z

LGTM, though it would be good to include a script for updating the subtree.

Yes, agreed. I'll do that in a subsequent PR. Those commands scan be extracted from #29 (comment)

evolvedmicrobe

Because sometimes the bin is a trash bin, I love it!

sjackman · 2020-08-14T17:20:17Z

Opened PR https://github.com/10XDev/cellranger/pull/4302

alexdobin and others added 30 commits December 1, 2017 11:43

Passes chr22 TestSuite tests.

72ae23d

Merged master changes in to var.

b294afa

Implemented --readFilesPrefix option for specifying prefix (e.g. dire…

3e199f0

…ctory path) for the file names in --readFilesIn . Implemented read group ID output as the last column of the Chimeric.out.junction file.

Merge pull request #334 from drpowell/master

c150910

Don't exit when fastq contains a zero-length read

Merged master changes (RG for chimeric junction output) into var.

6986377

Fixed a bug in chimeric detection code which sometimes led to uniniti…

4e21924

…alized memory access. The chimeric output may change for a very small number of reads.

Implementing WASP algorithm.

815e922

Implementing WASP algorithm.

a3a4a8e

Merged chimeric bug-fix from master. WASP works for single variant pe…

4995412

…r read.

Implementing WASP algorithm.

498bdbf

Implementing WASP algorithm.

04c1f11

Added variant allele=4 for variant read base=N. Added various waspTyp…

210deae

…e values.

Initial test release of WASP filtering.

7316edc

For BAM to signal conversion, process alignments without NH tags as u…

dd7566e

…nique (i.e. NH=1).

Implemented --outBAMsortingBinsN option to control the number of sort…

bbd1139

…ing bins. Icnreasing this number reduces the amount of RAM required for sorting.

Fixed vW:i:2 type to mark multimappers that overlap variants. Recompi…

c889d3f

…led executables.

2.5.4a

3f51800

Fixed a problem with non-default --sjdbOverhang genome generation.

ea90fe0

Merged 2.5.4a master into var.

d7a903b

spelling fixes

de4bffb

Implemented merging and re-mapping for overlapping mates. To control …

30a79fd

…it, use --peOverlapNbasesMin and --peOverlapMMp oprions.

2.5.4b

9c6df13

Fixed a problem in the chimeric detection algorithm for overlapping P…

35e5360

…E reads.

Fixed some issues with the overlapping mates algorithm.

ba0325e

Fixed some issues with the overlapping mates algorithm.

920bbc5

Fixed some issues with the overlapping mates algorithm.

347e063

Fixed a bug in the overlapping mates algorithm.

048d9e7

Fixed a bug in the overlapping mates algorithm related to protruding …

a26ce03

…ends of the mates.

Fixed a bug in ReadAlign_stitchPieces.cpp.

dfad0be

Fixed bugs in the algorithm that finds the best alignment.

364456b

sjackman and others added 6 commits May 12, 2020 11:51

Fix warnings and treat warnings as errors (#10)

b864977

Add -Werror to CXXFLAGS. Remove register keyword. Comment out unused function arguments.

re-instate reference version check

e059bc7

forfeit STARsolo to bring back quantMode

f49e976

Remove the submodule star-sys/STAR

05d6c8b

Add 'star-sys/STAR/source/' from commit '85c4977512e8454b503c11d49570…

ab8c1ae

…8ebe672d5ae7' git-subtree-dir: star-sys/STAR/source git-subtree-mainline: 05d6c8b git-subtree-split: 85c4977

star-sys/STAR: Merge commit 'f49e976d9330e5306753c1f296093d79d6260169…

cb1ccec

…' into sj/subtree

sjackman self-assigned this Aug 13, 2020

sjackman added the enhancement New feature or request label Aug 13, 2020

sjackman requested review from nlhepler, evolvedmicrobe, pmarks and adam-azarchs August 13, 2020 14:58

pmarks approved these changes Aug 13, 2020

View reviewed changes

pmarks requested changes Aug 13, 2020

View reviewed changes

Add star-sys/STAR/LICENSE

6e5be8d

nlhepler approved these changes Aug 13, 2020

View reviewed changes

sjackman marked this pull request as ready for review August 13, 2020 16:50

Add LICENSE

07e9bbd

sjackman force-pushed the sj/subtree branch from c40e38a to 07e9bbd Compare August 13, 2020 19:03

evolvedmicrobe approved these changes Aug 14, 2020

View reviewed changes

sjackman merged commit b7914fb into master Aug 14, 2020

sjackman deleted the sj/subtree branch August 14, 2020 19:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert the submodule star-sys/STAR to a git subtree #29

Convert the submodule star-sys/STAR to a git subtree #29

sjackman commented Aug 13, 2020 •

edited

Loading

pmarks left a comment

pmarks left a comment

nlhepler commented Aug 13, 2020

sjackman commented Aug 13, 2020

nlhepler left a comment

sjackman commented Aug 13, 2020 •

edited

Loading

adam-azarchs commented Aug 13, 2020

sjackman commented Aug 13, 2020 •

edited

Loading

sjackman commented Aug 13, 2020

sjackman commented Aug 13, 2020

evolvedmicrobe left a comment

sjackman commented Aug 14, 2020

Convert the submodule star-sys/STAR to a git subtree #29

Convert the submodule star-sys/STAR to a git subtree #29

Conversation

sjackman commented Aug 13, 2020 • edited Loading

pmarks left a comment

Choose a reason for hiding this comment

pmarks left a comment

Choose a reason for hiding this comment

nlhepler commented Aug 13, 2020

sjackman commented Aug 13, 2020

nlhepler left a comment

Choose a reason for hiding this comment

sjackman commented Aug 13, 2020 • edited Loading

adam-azarchs commented Aug 13, 2020

sjackman commented Aug 13, 2020 • edited Loading

sjackman commented Aug 13, 2020

sjackman commented Aug 13, 2020

evolvedmicrobe left a comment

Choose a reason for hiding this comment

sjackman commented Aug 14, 2020

sjackman commented Aug 13, 2020 •

edited

Loading

sjackman commented Aug 13, 2020 •

edited

Loading

sjackman commented Aug 13, 2020 •

edited

Loading