configuration shim for poorly-constructed package structures

[disruptek]

Basically, the user cannot simply --nimblePath=/dist whatfer Nimble reasons -- it demands directories formed with -x.y.z (or whatever) in their name. This runs counter to the goal of a single namespace which denies possible collisions.

As this later feature is more important than --nimblePath, we may need a way to shim the distribution into paths, especially in the case where a src/ subdirectory exists needlessly. There are several ways to do this; I'm interested to hear your preferences.

Another alternative is to fork packages and fix their structures. Any other ideas?

[c-blake]

I agree on the need. I did some whole-nimbleverse testing when doing nimp. I mention some statistics there. At least within the transitive closure with roots starting at "packages that produce a binary executable", there are currently very few namespace conflicts just adding the src/ subdir and the like to the nim module search path. Many more problems arise just from the Nim compiler itself evolving away from how the packages use the language.

I also thought of a perhaps simpler way to describe the poor package construction. Starting point: what does a package manager mostly do? Fetch and install dependencies. What is the primary way for one package to depend upon another in Nim? import (or maaaabye include). This kind of dependency is immediately satisfied by search path. src/ requires either augmenting the path to squash it or the user to say import foo/src/bar. The former creates more (currently unrealized) opportunity for collision while the latter is verbose/ugly. Conclusion - src/ is a terrible default for any package that is a "library". It solves no immediate problem and creates problems down the line. { Honestly, this seems almost like a math theorem and unlikely to be controversial except to aboriginal nimble authors and people who are perhaps more easily suggestible by the first example they see of a solution. }

There are a handful of dependencies with a secondary way "to depend" on an installed package as a program, not as a library. E.g., build tools & code generators { things like nake and closure_compiler. nimx is a good source for these as I think it needs 4 ultimately. :-) }

Almost all namespace collisions actually come from the stdlib itself not using std/ and a package reusing some stdlib module name (things like "random" and "cpuinfo") . This is not surprising. stdlib is both the biggest namespace populater by far and also an example of standard "ways to name things". So, I would say one initial step on this journey might be to get stdlib using std/ everywhere internally. I'm not sure how popular/unpopular that would be.

Anyway, the way nimp does its "shim" dance is largely via just patching up/hacking up the origin package. This was mostly inspired by the way Gentoo lets you have local /etc/portage/(patches|env|...) "shadow trees" that add tweaks to the global shared default. It is a little clunky, but also pretty general. No-op shims are the vast majority {and would be even less necessary without src/ defaults}. Even so, dependees can always have bugs/misfeatures that a user may want to patch (rather than forking the repo and pointing elsewhere), whether that user is just one endpoint person or someone(s) organizing a collection on the behalf of a community.

[disruptek]

Okay, but this doesn't get us closer to solving the problem -- especially with our submodule-based architecture. Unless you're proposing that we fork or patch the packages with silly structure?

[c-blake]

My comment was more observation than solution/preference. I guess I was not sure what "shim into paths" meant, exactly. The observation more briefly was just taking "shim into paths" as just "constructing a nim.cfg/config.nims that sets up a deep path" leads to a problem of a (rare) need to package-qualify some imports (either in the stdlib or in the package). Patching could add qualification. Renaming modules could avoid the need. Either could be upstreamed eventually.

Anyway, sorry if this is off track. I think it would probably be helpful for you to list the "several ways" you have in mind to get people to comment upon them.

[disruptek]

Well, I don't want to prevent anyone from coming up with a better idea by supplying them with a weaker one.

One thought I had was to go ahead and rename frosty to frosty-0.4.4. While "correct by construction" may not hold for the repository, it could be the problem most readily solved -- write a test and make it pass. The benefit is that we can use --nimblePath naturally and to good effect. It works with all the tooling, Nimph, and nimbledeps as well.

Another idea would be to import the distribution into a project via a local config.nims that adds path statements. Since this may be required for poorly-structured packages in any event, it seems like the best first step.

I don't love either of these solutions. What I'd really like, honestly, is to just store the distribution submodules as bare repositories and clone them, locally, shallowly, and directly into my project via Nimph. Failing that, maybe just use a symlink to connect /dist/stupid/src/ to /project/stupid/.

These don't feel like solutions that others will want to use, though, and it's unlikely the Windows folks will tolerate a symlink, let alone a bare repository. Plus, it doesn't further the goal of the distribution -- we don't want Nimph rolling your dependencies for you anymore -- we want everyone using the same, tested, stable versions.

So, yeah, I'm still trying to figure out how this can work for everyone (and myself) and while it feels like it can work better than fusion, it's far from a tight design. 😁

[c-blake]

fision bears mentioning - just in case you/any reader was unaware, as it gets very little discussion/exposure on IRC/Forum/etc. (I guess @juancarlospaco dropped the usual double-s on purpose..something about Spanish spelling.) That's more on the opposite side of the version stability space you seem to be exploring, but otherwise seems a similar "back reaction" to fusion. (All these physics puns are making my head acquire angular momentum, erm I mean spin!)

[timotheecour]

Another alternative is to fork packages and fix their structures. Any other ideas?

@disruptek IMO nim-lang/RFCs#291 fixes all those issues; with this RFC, no need for src prefix (but also works if a package has it), no risk of namespace collision. It also makes it easy to override at arbitrary granularity (package, directory, module), and even enables depending on 2 versions of a given package, although this needs further explanation.

eg:

# dist/config.nims:
const dist = getCurrentSourcePath.parentDir
--path:gram:dist/gram
--path:untar:dist/untar/src
... # add all packages tracked by dist here with the correct path

[disruptek]

Right, but we want to use this without regard to Nim version.

Basically, the best solution we've come up with so far is to have gitnim write a new nim.cfg for the distribution when it runs during tests; it can vet the namespace and compose the cfg automatically using ups.

But I'm open to hearing other ideas. I'm down on config.nims because it's slow and I think dynamicism is the last thing we want here.

gitnim is a rather key ingredient because there's no easy way to include the config otherwise; gitnim can pull it into the compiler's config trivially during installation of the compiler binaries, according to each version...

[timotheecour]

that feature in the RFC can be implemented and backported all the way to 1.0.0

[disruptek]

If you build it, there will be no reason not to use it.