Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use git subtree to graft Pkg3, etc. into stdlib #25942

Closed
wants to merge 974 commits into from
Closed

Conversation

StefanKarpinski
Copy link
Member

This was astonishingly easy to do with git subtree. It really might be a great solution for developing stdlib packages outside of the main repo but having the code available without any additional futzing upon checkout of julia. On this branch, you can do using Pkg3 from the julia prompt and it works:

julia> using Pkg3
[ Info: Precompiling module Pkg3

pkg> init
[ Info: Initialized environment in /Users/stefan/projects/julia by creating the file Project.toml

pkg> add JSON
[ Info: Resolving package versions
[ Info: Updating "Project.toml"
 [682c06a0] + JSON v0.16.4
[ Info: Updating "Manifest.toml"
 [34da2185] + Compat v0.49.0
 [682c06a0] + JSON v0.16.4
 [4d1e1d77] + Nullables v0.0.3

Resulting project and manifest files.

Project.toml:

[deps]
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

Manifest.toml

[[Compat]]
git-tree-sha1 = "e74e175d5b9c7411f17d7db90ccdfdf492bde9e9"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "0.49.0"

[[JSON]]
deps = ["Compat", "Nullables"]
git-tree-sha1 = "e4b57d438f523a2ad5f9f3510598f00961d4bc08"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.16.4"

[[Nullables]]
deps = ["Compat"]
git-tree-sha1 = "05176cebbb49ab979006b0bd12008c3f9a54f60c"
uuid = "4d1e1d77-625e-5b40-9113-a560ec7a8ecd"
version = "0.0.3"

Upshot

julia> using JSON
[ Info: Precompiling module JSON
ERROR: LoadError: LoadError: ArgumentError: Module Unicode not found in current path.
Run `Pkg.add("Unicode")` to install the Unicode package.

Ok, so there are some rough edges to smooth out, but this finally basically works.

StefanKarpinski and others added 30 commits October 17, 2017 15:01
otherwise the build process doesn't pick up correct versions
The miniconda installer creates the correct shebang line no problem
but conda update encounters long shebang lines and decides that they
are dangerous, so it silently replaces a "long" (≥128 bytes) path to
`python` with `/usr/bin/env python`. What could go wrong?!? This works
around that behavior by using a 16-byte "slug" in package/version
paths instead of using the full "$uuid/$sha1" string, which, to be
fair, is an excessive 77 characters in the middle of paths.

The issue was that Conda looks for long shebang lines (>127) and
silently replaces the or
otherwise the SHA package would have to be internalized
when the build takes a while, you can tail the file to see what's up
This allows Pkg3 loading to work even if it's loaded with `include`
instead of by being on the `LOAD_PATH` and running `using Pkg3`.
Support more forms of at-nospecialize.
remove precompile from internalized toml
add information I was missing when installing and remove remark about precompilation, which doesn't seem to be true anymore.
no precompilation remark, add pkg3 clone, pkg dir
ararslan and others added 9 commits February 5, 2018 19:13
Add occursin and find* methods replacing search and findin
* Add size(::LinearIndices)

Fixes indexing LinearIndices with an array. This is useful in particular
to convert cartesian indices that find() now returns to linear indices.

* Add size(::LinearIndices) on older 0.7 versions
…a389a'

git-subtree-dir: stdlib/Compat
git-subtree-mainline: 2017524
git-subtree-split: 95db156
git-subtree-dir: stdlib/SHA
git-subtree-mainline: 331bf19
git-subtree-split: 2958883
…483'

git-subtree-dir: stdlib/Pkg3
git-subtree-mainline: 278ede2
git-subtree-split: 3c0c9e2
@vchuravy
Copy link
Member

vchuravy commented Feb 8, 2018 via email

@StefanKarpinski
Copy link
Member Author

Yes, the things in Pkg3/ext should become proper stdlib modules, but I wasn't quite ready to add those right now – they are at the moment, implementation details of Pkg3. I'm not 100% sold on git subtree, but so far it's been a pleasure to use. The main advantages seem to be:

  1. See history of stdlib packages in the same repo as Base.
  2. Develop stdlib packages together with Base, upstream the changes when they're ready.

@StefanKarpinski StefanKarpinski changed the title use git subtree to graft Pkg3, SHA and Compat into stdlib use git subtree to graft Pkg3, etc. into stdlib Feb 8, 2018
@StefanKarpinski StefanKarpinski added this to the 1.0 milestone Feb 8, 2018
@StefanKarpinski StefanKarpinski added the triage This should be discussed on a triage call label Feb 8, 2018
@StefanKarpinski
Copy link
Member Author

Milestoning, not because we have to do it this way but because we have to do something like this.

@o314
Copy link
Contributor

o314 commented Feb 8, 2018

Drupal followed a similar way a few month ago

Subtree splits are now being automated by drupal infrastructure. We have a jenkins job that is polling git.drupal.org/project/drupal.git every minute and runs a script that traverses a git checkout of drupal, looking for composer.json files. If it discovers a composer.json (except in test directories), it will:

  • create a subtree split of that directory
  • parse composer.json
  • If it doesnt exist, it will create a new repo at github.com/drupal/{composer-project-name}
  • It pushes the subtree split to github.com/drupal/{composer-project-name}
  • If it doesnt exist, it will create a new entry at packagist.org
  • If the packagist entry does exist, it is instructed to update its entry from github.

So new components or new composer.json parts that we wish to expose outside the drupal ecosystem should automatically appear on packagist.

@ https://www.drupal.org/project/infrastructure/issues/2352091#comment-12056883

The symfony framework (~ new drupal kernel, and more) guys gave a talk about this at drupalcon 2016 : a monorepo vs manyrepos (slides), (video), and (code)

@StefanKarpinski
Copy link
Member Author

Triage dixit: no Compat in stdlib, so excise that dependency. Pkg3 + SHA in base is fine, but favors @vchuravy's approach to use @vtjnash's "git external" mechanism. The TerminalMenus is a different story since it should be merged into REPL; for that, using subtree to import and do a temporary bridge between internal and external is fine.

@StefanKarpinski StefanKarpinski removed the triage This should be discussed on a triage call label Feb 8, 2018
@JeffBezanson
Copy link
Member

In triage several of us prefer @vchuravy 's approach (#25714). I think that will make it clearer that the packages are separate, and owned by different repos. There's something to be said for being able to update Base and stdlibs all at once, but we're already used to having many packages in external repos that might need to be updated in sync with Base.

@KristofferC
Copy link
Member

KristofferC commented Feb 8, 2018

That seems very(!!!) unfortunate in my view. Having stuff in the same repo tremendously ease development and coherence. All other stdlibs live in the repo. Why should Pkg3 be the only one getting punished!?

I think that will make it clearer that the packages are separate, and owned by different repos.

I am all for getting rid of the Pkg3 repo and just have it here if it would mean not having to do this. Just develop it here then.

but we're already used to having many packages in external repos that might need to be updated in sync with Base.

And this is typically very annoying and time-consuming. Run make docs now and see how well it works. Presumably, we want to run the Pkg3 test as part of Base CI (to make sure we don't break the whole ecosystem on master at once) which would make this even worse than the Documenter situation.

Also, multiple stdlib packages depend on Pkg right now. Are they going to be in this repo and depend on a repo located somewhere else?

@KristofferC
Copy link
Member

KristofferC commented Feb 8, 2018

I didn't manage to get to the triage call in time for this, but if the discussion was only between keeping the Pkg3 repo separate and using subtree to sync changes vs using git-external then my suggestion here is just to move Pkg3 to a stdlib and delete the old Pkg3 repo. Just like any other stdlib we currently have. Whenever we decide to use some other method for dealing with stdlibs, we do it for all at once. Using Pkg3 as the guinea pig for this is not a good choice regarding getting to 1.0 as fast as possible, in my opinion.

@JeffBezanson
Copy link
Member

If Compat is out of the picture, then I'm much more ok with using git subtrees. I also wouldn't object to entirely moving the Pkg3 code here.

@KristofferC
Copy link
Member

KristofferC commented Feb 9, 2018

Yes, Compat is out of the picture.

Great, I can update this PR, with #25956 and #25953 in the picture.

@StefanKarpinski StefanKarpinski removed this from the 1.0 milestone Feb 9, 2018
@StefanKarpinski
Copy link
Member Author

I'll close this then since @KristofferC's PRs supersede it.

@KristofferC
Copy link
Member

I was going to update this one but can make a new one.

@StefanKarpinski
Copy link
Member Author

Eh, just make a new one. There's too much junk in here.

@vtjnash
Copy link
Member

vtjnash commented Feb 9, 2018

Why should Pkg3 be the only one getting punished!?

Being frozen in stdlib is a punishment, being free to develop separately should be a positive.

Develop stdlib packages together with Base, upstream the changes when they're ready.

Regardless of where this code lives, I think we should encourage development to happen upstream, with separate release cycles, issue tracker, CI, etc., then just import the stable releases here.

@DilumAluthge DilumAluthge deleted the sk/pkg3 branch March 25, 2021 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.