jqnpm - a package manager built for jq as an example implementation

[joelpurra]

@nicowilliams, @pkoppstein, @wtlangford, others:

Hacked together a package manager for you to play with:
https://github.com/joelpurra/jqnpm

It works within the current import limitations such as:

• jq not being aware of there being an official package manager, as npm is treated as for node.
• Library search paths only being relative to the initial script called, not the script calling import.

cd tests/execute-flat-dependencies/local-project/
                                    # Set a local package cache folder for the tests in this folder.
export JQNPM_PACKAGES_CACHE="$PWD/../package-cache"
jqnpm install                       # Install recursively in ./.jq/
tree -a                             # See installation results.
jqnpm execute -n                    # Temporary wrapper for jq, but with managed dependencies.

• No remote repo cloning/fetching yet.
• Does not respect dependency versions yet.

See also previous discussions in #491, #566, #535 regarding my preferred package manager style. Let me know if you have any questions, fixes, workarounds!

[nicowilliams]

@joelpurra Yes!!! I'll take a closer look this evening. An external pkg manager will make a jq 1.5 release a beautiful thing. Thank you!

[nicowilliams]

Skimming the README I see you've put a lot of effort into this. I greatly appreciate it.

Feel free to file a new issue for jq 1.6 or whatever milestone later than 1.5 for pkg-related features you want natively in jq in the future.

[nicowilliams]

(Small and urgent extensions might still get into 1.5 though. Now is the time to request them though.)

[joelpurra]

Oops. Had a bug which ended execution if debugging wasn't enabled through the undocumented JQNPM_DEBUG_LEVEL; fixed.

[joelpurra]

@nicowilliams: glad to see some enthusiasm!

Keeping the discussion in this issue thread for now, as my proposals will be related to jqnpm. This is what I came up with at the moment - I feel most of it is about removing unnecessary fluff from the core and letting the package manager deal with it.

Regarding import

Make the format strictly import "<package>" as <alias>;.

• Make package a proper, quoted string. This way :: can be avoided as "path separators"; the more commonly used / path separator won't add extra syntax character hassle when wrapped in double quotes.
• ./ is already used as a (cross-platform) trigger for paths in { search: "..." }; re-use the package string instead.
• Remove the "extra" options hash. It has been replaced with external package metadata in ./jq.json and the package string.
• Require an alias value that is unique per package. No automatic "file merging" or "namespace merging", no null for top level replacements etcetera. I dislike namespace merging surprises where ordering matters and there are no warnings by design. Library System #472 (comment)
• Have jq follow the node- and npm-inspired import resolving algorithm. See the node module link for inspiring details; changes should be small enough as much of the code already exists in some form.

Regarding module

• Remove module altogether. "It may well be utterly useless." List of publicly released modules? #535 (comment)
• Let the package manager manage the metadata in the per-package ./js.json.

Bonus requests

• I've included import "./resources/my-data.json" as myData; imports in the algorithm as well, since you've mentioned it and I'd like to use it.
• 56ae88d says default (system-dependent, #ifdef WIN32) paths are replaced with jq -L. Don't, as they are annoying to regenerate externally. Either:
  • Prepend.
  • Add a new flag which adds to the path.
  • Drop the -L functionality altogether. Remove ~/.jq support when module system integrates #485

---

Sorry for being absent from the discussion since the summer. Hope my suggestions aren't too drastic.

[nicowilliams]

On Mon, Dec 29, 2014 at 02:40:27PM -0800, Joel Purra wrote:

@nicowilliams: glad to see some enthusiasm!

Well hey, it's a nice gift from you. Incidentally, happy holidays!

Make the format strictly import "<package>" as <alias>;.

• Make package a proper, quoted string. This way :: can be avoided as "path separators"; the more commonly used / path separator won't add extra syntax character hassle when wrapped in double quotes.

Is this because mapping ::<->/ is a pain for the pkg manager?

With the as clause being required I no longer would need to ensure that the module name be a valid IDENT... So, yeah, a string would be better.

Does the .jq extension need to be included in the string? That would seem weird.

How do I tell if the path is absolute? Or should I require that it be relative (because i always append it to a path that ends in '/')? I prefer the latter so I don't have to have any OS-dependent code to check if the path is relative. I wouldn't want to try the path as-is last because that would make the current directory always part of the search path, and that's not something I'm fond of.

# Import foo/bar.jq or foo/bar/bar.jq as foo.
#
# If there's associated data files, read them and bind them to
# $foo::name (see notes below about `module`).
#
import "foo/bar" as foo;

# Just read some stuff (relative to the containing module's
# containing directory).
import "stuff.json" as $stuff;

• ./ is already used as a (cross-platform) trigger for paths in { search: "..." }; re-use the package string instead.

I'm not sure I follow. Can you show me an example?

• Remove the "extra" options hash. It has been replaced with external package metadata in ./jq.json and the package string.

It might be useful for single-file modules though -- sure, it's a pain for some kinds of metadata (e.g., commit hashes), so you might not use it; it costs nothing to keep anyways. For the same reason I'd like to keep module.

• Require an alias value that is unique per package. No automatic "file merging" or "namespace merging", no null for top level replacements etcetera. I dislike namespace merging surprises where ordering matters and there are no warnings by design. Library System #472 (comment)

I may do that, yeah.

@pkoppstein seemed to like being able to import into the caller's namespace. @wtlangford didn't like defaulting to that. You don't like it in any case. Is there a compromise to be made? I'm leaning to just requiring the as part.

Regarding module

• Remove module altogether. "It may well be utterly useless." List of publicly released modules? #535 (comment)

It may be, but for simple single-file modules (which jqnpm might well not support ever) it could be useful (no commit hashes and so on, but maybe a homepage, mailto, ...). Is it harmful?

Actually, I have a use for something like a module directive: indicate whether there are associated JSON data files. If we import "foo/bar" and find a .jq file, that doesn't necessarily mean that there must be a data file to go with it, but if one is added, it should be listed in the module's in-band metadata (this is not a PITA to keep up to date, as it's not something that changes with every commit/build/...).

Perhaps it should be something like:

# Take the containint module's .jq file's path, s/\.jq/.json/, read
# as $mydata:
#
metadata ".json" as $mydata;

and even:

# Take the containint module's .jq file's path, remove the last path
# component, add "/some_stuff.json", read as $mydata:
#
metadata "some_stuff.json" as $some_stuff;

metadata "other_stuff.json" as $other_stuff;

• Let the package manager manage the metadata in the per-package ./js.json.

It can do that regardless :) Especially once I add support for reading it automatically. See above.

Bonus requests

• I've included import "./resources/my-data.json" as myData; imports in the algorithm as well, since you've mentioned it and I'd like to use it.

Yeah, I need to make that work. That's probably urgent.

• 56ae88d says default (system-dependent, #ifdef WIN32) paths are replaced with jq -L. Don't, as they are annoying to regenerate externally. Either:
  • Prepend.

Can I assume you mean "prepend the user's -L... path to the default?

If the problem is OS-dependence then I can always add a token for "default system path here".

• Add a new flag which adds to the path.

Prepend or append?

• Drop the -L functionality altogether. Remove ~/.jq support when module system integrates #485

It's quite useful for testing; I could drop it from the docs but keep it. It could also be useful for keeping multiple universes of jq modules lying around. (Think NixOS-style packaging.) I'd rather keep it altogether.

[pkoppstein]

@joelpurra - Your package manager seems to be quite complex to use. Who is responsible for writing jq.json?

In the julia-inspired view of things, the end-user would not have to do anything beyond:

1. installing the package manager (git clone ....; jqpm install)
2. adding the particular packages of interest (jqpm add Shazam)
3. use the packages within jq (e.g. import Shazam;)

Zero configuration. All dependencies taken care of automagically.

(For per-project installation, the end-user would use jqpm --project FOLDER ..., and would have to invoke jq with a suitable option to cause it to look in FOLDER.)

[nicowilliams]

I don't know when I'll have enough time to implement the desired changes to import. I'm still finishing the labels work.

[joelpurra]

@joelpurra - Your package manager seems to be quite complex to use. Who is responsible for writing jq.json?

I'm surprised to see a developer of a json tool see a json file this small as a complex problem. Of course the package developer is responsible for it - especially if the package will be published. That said, the jq.json basics are easy to generate, but I wrote this package manager yesterday and the night before, between playing boardgames and watching tv-series, and didn't quite get to ad hoc jq projects yet.

For now the jq.json file is required, because jq doesn't handle packages the way I'd like it to. The lines that require jq.json for execution can actually be removed, but then I'll always overwrite the default library search paths with jq -L - see issue comments.

In the julia-inspired view of things, the end-user would not have to do anything beyond:

Yes, that is also possible with jqnpm - except import Shazam, because jq lacks support for that syntax at the moment (see import resolving algorithm) for this exact style of package manager (which just had its first release).

Every time I read about an npm package, the author says the equivalent of jq install --save cool-package, meaning it'll be saved to the jq.jsonhash of dependencies. Thinking of making --save the default and generate jq.json every time a new package is installed from the command line.

Zero configuration. All dependencies taken care of automagically.

That is not zero configuration.

(For per-project installation, the end-user would use jqpm --project FOLDER ..., and would have to invoke jq with a suitable option to cause it to look in FOLDER.)

That is also configuration, except you seem to have to do it every time you execute code.

[pkoppstein]

Of course the package developer is responsible for it

On the README.md page, it would perhaps help to have separate sections for package developers and package end-users.

That is not zero configuration.

Could you please explain? If you are saying that declarative statements such as "add NAME" and "import NAME" (with no reference to location or the like) are "configuration parameters", then you are using a very wide net. There is an important and useful distinction to be made between such "what-is-needed" declarations and "how-to-do-it" configuration parameters.

One of the features of Julia-like package managers is that for registered packages (irrespective of where they are located) the end-user need not know where they are located. Since that does not seem to be a feature of jqnpm, it would be helpful if the end-user documentation could also explain the process of package discovery.

Thanks.

[nicowilliams]

I think I'm done with the labels work now (not pushed yet), which would leave just this and #454 as the outstanding issues for a 1.5 release, plus the work of producing the release. I'll know later this week when I can realistically finish this.

[joelpurra]

Of course the package developer is responsible for it

On the README.md page, it would perhaps help to have separate sections for package developers and package end-users.

Ah, yes, you are right. I'll look into it.

That is not zero configuration.

Could you please explain? If you are saying that declarative statements such as "add NAME" and "import NAME" (with no reference to location or the like) are "configuration parameters", then you are using a very wide net. There is an important and useful distinction to be made between such "what-is-needed" declarations and "how-to-do-it" configuration parameters.

I see install NAME as system configuration as it is required and changes the state of the system. "Configure your system to use package NAME." The "system" in this case may be system-global, user-global or per project.

One of the features of Julia-like package managers is that (for registered packages, irrespective of where they are located) the end-user need not know where they are located. Since that does not seem to be a feature of jqnpm, it would be helpful if the end-user documentation could also explain the process of package discovery.

Are you referring to setting the JQNPM_PACKAGES_CACHE? That is for test execution, which is all there is at the moment. Of course users don't have to know where their packages are located - they just jqnpm install and let jqnpm and jq deal with it.

Could add a simple explanation of the default install location as well as import resolving algorithm for the end-user as well. The current readme is obviously written in way too much detail.

[joelpurra]

@nicowilliams wrote:

On Mon, Dec 29, 2014 at 02:40:27PM -0800, Joel Purra wrote:

Regarding import

• Make package a proper, quoted string. This way :: can be avoided as "path separators"; the more commonly used / path separator won't add extra syntax character hassle when wrapped in double quotes.

Is this because mapping ::<->/ is a pain for the pkg manager?

No, the package manager has no reason to parse any.jq files. It's just for jq's convenience as the current core always ends up concatenating :: separated this::that with / separated paths - either JQ_LIBRARY_PATH, -L paths, { search: "./path" } or the default. I feel jq has ended up writing a double syntax.

With the as clause being required I no longer would need to ensure that the module name be a valid IDENT... So, yeah, a string would be better.

Excellent!

Does the .jq extension need to be included in the string? That would seem weird.

Not for package names joelpurra/jq-some-package nor directory references ../another-package/, only for direct file paths ./my-local-import.jq. All cases are handled by the import resolving algorithm.

How do I tell if the path is absolute? Or should I require that it be relative (because i always append it to a path that ends in '/')? I prefer the latter so I don't have to have any OS-dependent code to check if the path is relative.

Not a fan of absolut paths in import, but would use starting with / as in /somewhere/on/disk/ for absolute paths. Forward slashes should work well with the file:// scheme even on windows, seeing it as file:// + /c:/dev/folder/ -> file:///c:/dev/folder/.

• ./ is already used as a (cross-platform) trigger for paths in { search: "..." }; re-use the package string instead.

I'm not sure I follow. Can you show me an example?

56ae88d#diff-d2fa3f3320a35df5eca0fe665267bd2fR2343
56ae88d#diff-0c8955d25f69d48ea0457eb0c5cdd88eR3

• Remove the "extra" options hash. It has been replaced with external package metadata in ./jq.json and the package string.

It might be useful for single-file modules though -- sure, it's a pain for some kinds of metadata (e.g., commit hashes), so you might not use it; it costs nothing to keep anyways. For the same reason I'd like to keep module.

I would not like to be the maintainer of that syntax, when there are external package managers doing (aiming to do) it better.

• Require an alias value that is unique per package. No automatic "file merging" or "namespace merging", no null for top level replacements etcetera. I dislike namespace merging surprises where ordering matters and there are no warnings by design. Library System #472 (comment)

I may do that, yeah.

Great!

Regarding module

• Remove module altogether. "It may well be utterly useless." List of publicly released modules? #535 (comment)

Oh, and I would also remove modulemeta as well, if that wasn't clear. That is an example of functionality jq shouldn't have to care about.

It may be, but for simple single-file modules (which jqnpm might well not support ever) it could be useful (no commit hashes and so on, but maybe a homepage, mailto, ...). Is it harmful?

Harmful as in splitting efforts to modularize the jq ecological system perhaps, as there would be two competing systems - and harmful to time and effort spent developing duplicate/parallel systems. I strongly prefer packages that consist of a single "module" for extreme/atomical reusability - though jqnpm requires a whooping two files - jq.json for the metadata and at least one.jq.

jqnpm is currently trying to be strict about having a jq.json file in place. As there is no central repository, the hard requirements on this file at this stage are actually more relaxed than I'd like to see moving forward. Am a proponent of having author name and licensing etcetera listed as jq.json metadata (for public packages), as it reduces ambiguity.

Actually, I have a use for something like a module directive: indicate whether there are associated JSON data files. If we import "foo/bar" and find a .jq file, that doesn't necessarily mean that there must be a data file to go with it, but if one is added, it should be listed in the module's in-band metadata (this is not a PITA to keep up to date, as it's not something that changes with every commit/build/...).

Very much unnecessary. The jq.json metadata property main path indicates if the main purpose of the package is ./src/some.jq or ./data/file.json; import "joelpurra/jq-useful-data" as joelsData.

If ./src/some.jq then does import "../data/file.json" as myData it is no longer metadata - that is code.

Referencing a namespaced package import "joelpurra/jq-useful-data/data/another-file.json" as joelsOtherData is also possible, it's just that I don't usually pick single files from others' packages very often.

Bonus requests

• I've included import "./resources/my-data.json" as myData; imports in the algorithm as well, since you've mentioned it and I'd like to use it.

Yeah, I need to make that work. That's probably urgent.

Not super urgent for me, but it would be very nice to have later on. I already have at least a couple cases where I already use external json files strictly for lookups, and they are loaded as --argfile in v1.4.

• 56ae88d says default (system-dependent, #ifdef WIN32) paths are replaced with jq -L. Don't, as they are annoying to regenerate externally. Either:
  • Prepend.

Can I assume you mean "prepend the user's -L... path to the default?

Correct.

If the problem is OS-dependence then I can always add a token for "default system path here".

That could work; would add some prepend/append flexibility.

• Add a new flag which adds to the path.

Prepend or append?

Haven't decided which is more useful, if there's only once choice.

• Drop the -L functionality altogether. Remove ~/.jq support when module system integrates #485

It's quite useful for testing; I could drop it from the docs but keep it. It could also be useful for keeping multiple universes of jq modules lying around. (Think NixOS-style packaging.) I'd rather keep it altogether.

I don't really see the usability of -L once the import resolving algorithm is in place. A "local universe" can already be defined in the jq.json dependencies.

[nicowilliams]

On Tue, Dec 30, 2014 at 01:26:53AM -0800, Joel Purra wrote:

@nicowilliams wrote:

On Mon, Dec 29, 2014 at 02:40:27PM -0800, Joel Purra wrote:

Regarding import

• Make package a proper, quoted string. This way :: can be avoided as "path separators"; the more commonly used / path separator won't add extra syntax character hassle when wrapped in double quotes.

Is this because mapping ::<->/ is a pain for the pkg manager?

No, the package manager has no reason to parse any.jq files. It's just for jq's convenience as the current core always ends up concatenating :: separated this::that with / separated paths - either JQ_LIBRARY_PATH, -L paths, { search: "./path" } or the default. I feel jq has ended up writing a double syntax.

Refusing to use modulemeta to read a pkg's necessarily-declared dependencies seems a bit odd. Suppose I had a module or program like this:

import "..." as foo {git: "http://github.com/jqmodules/..."};
...

Now, you could use jq to get that metadata, use it to learn how to find the dependency foo (git clone the given home), and then update your pkg database.

Also, ISTM that the pkg mgr's JSON database ought to be more like a cache: if damaged, you should be able to re-create it. Here, once more, the import metadata of installed .jq files seems authoritative.

Does the .jq extension need to be included in the string? That would seem weird.

Not for package names joelpurra/jq-some-package nor directory references ../another-package/, only for direct file paths ./my-local-import.jq. All cases are handled by the import resolving algorithm.

There wouldn't be package names, just the as SYMBOL, the filename, search list, ...

How do I tell if the path is absolute? Or should I require that it be relative (because i always append it to a path that ends in '/')? I prefer the latter so I don't have to have any OS-dependent code to check if the path is relative.

Not a fan of absolut paths in import, but would use starting with / as in /somewhere/on/disk/ for absolute paths. Forward slashes should work well with the file:// scheme even on windows, seeing it as file:// + /c:/dev/folder/ -> file:///c:/dev/folder/.

There's an Internet-Draft for the file: URI scheme. It's... not that simple...

• Remove the "extra" options hash. It has been replaced with external package metadata in ./jq.json and the package string.

It might be useful for single-file modules though -- sure, it's a pain for some kinds of metadata (e.g., commit hashes), so you might not use it; it costs nothing to keep anyways. For the same reason I'd like to keep module.

I would not like to be the maintainer of that syntax, when there are external package managers doing (aiming to do) it better.

There'd be no need to maintain it if unused. Still no harm. (But see above.)

Regarding module

• Remove module altogether. "It may well be utterly useless." List of publicly released modules? #535 (comment)

Oh, and I would also remove modulemeta as well, if that wasn't clear. That is an example of functionality jq shouldn't have to care about.

It was.

More later. GTG.

[nicowilliams]

@joelpurra

jq programs must import their dependencies. Why not ask jq what dependencies a module wants? It seems like a fine thing to do to check any external metadata for consistency.

Also, I still think that putting metadata in the import directive about how to locate a missing dependency is a good idea. Sure, you could repeat this in a metadata file, but it will be less work for developers if there is less duplication, and if in the common case the metadata file can be generated for them by tooling.

[nicowilliams]

This is the only issue left for 1.5 now. I'll try to get around to it before the weekend (otherwise it will have to be three weeks from now). I don't think I'll remove all the things you don't want to use @joelpurra, but you can always not use them; the other changes I will make.

[nicowilliams]

Implementing import "stuff" as $d; turned out to be pretty tricky. Not too hard, just tricky. In the end I needed a new instruction: because at the linking stage the libraries have to be made up of only "binder" instructions, but variable bindings from the command-line (which are^H^H^Hwere the only true globals, weren't single-instruction code sequences each, nor are normal ... as $var sequences single-instruction either -- nor could they be. But for global constants, it could be done, and indeed, had to be done. There was no other way to make the compiler and linker do the right thing.

(Briefly, the linker tries to avoid compiling the same file twice. What the linker does then is bind (pointer-wise) dependents to the "blocks" generated by the parser for their dependencies, changing the dependecie's symbol names on the fly to deal with the as SYMBOL part of import. Then later on the linker puts all of the loaded libraries' and the top program's blocks together, removes anything left unreferenced, then it assembles the final bytecode. Dropping the unreferenced blocks helps performance, and so should (eventually, at least as to builtins) the dedup effort. But this all becomes infinitely harder if a "binder" can be a sequence of multiple instructions instead of just one.)

[nicowilliams]

Anyways, this has:

• string instead of symbol for path to module to import

  import "relative/path" as SYMBOL; looks for "relative/path.jq", compiles it, and loads its symbols into the caller's SYMBOL::... sub-namespace

  NOTE: two modules using the same sub-namespaces will NOT collide; they never could before, still can't now.

• as SYMBOL or as $SYMBOL is always required

• import "relative/path" as $SYMBOL; looks for "relative/path.json", loads it if it finds it (careful! it slurps), and binds it to a global $SYMBOL (finally; this is great)

• import "relative/path" as $SYMBOL {raw: true}; reads the whole file as a single string

  Perhaps I'll add more options for reading data (just the first value instead of slurp, an array of lines vs. one big string).

• unified the builtin search path

• updated the docs

Other things:

• so far I didn't change how -L works; I either didn't get around to it, or something;
• module lost the symbol name; it retains the metadata;
• it was already the case that two modules importing into apparently the same namespace didn't actually. There's nothing to do to keep one module from importing multiple modules into the same namespaces, since the programmer can just... read the code, as there's only one file to look at) (see above)
• modulemeta remains.

Bang on this guys, please. Don't be upset if I missed something: this took a lot of doing. I'm done for the night.

[nicowilliams]

BTW, @joelpurra, you got most of what you asked for. I didn't remove modulemeta because being able to ask what modules a module imports seems very important to me. The metadata for modules and dependencies does seem harmless to me, and I don't think it will engender competing pkg managers. For imports in particular, I have a variety of what I think must be unobjectionable uses for metadata (see above, and the docs). Only the module metadata has no use at this time, and that directive might yet go.

Thanks for your comments and ideas!

[nicowilliams]

Also, the metadata on import is utterly optional. The right thing happens as to the search path (search the caller's directory first).

[nicowilliams]

On Tue, Dec 30, 2014 at 01:26:53AM -0800, Joel Purra wrote:

• ./ is already used as a (cross-platform) trigger for paths in { search: "..." }; re-use the package string instead.

I'm not sure I follow. Can you show me an example?

56ae88d#diff-d2fa3f3320a35df5eca0fe665267bd2fR2343
56ae88d#diff-0c8955d25f69d48ea0457eb0c5cdd88eR3

Ah, the right thing happens. No need to even include metadata on
import, or the "search" key.

If ./src/some.jq then does import "../data/file.json" as myData it is no longer metadata - that is code.

Right. Check the latest.

Bonus requests

• I've included import "./resources/my-data.json" as myData; imports in the algorithm as well, since you've mentioned it and I'd like to use it.

Yeah, I need to make that work. That's probably urgent.

Not super urgent for me, but it would be very nice to have later on. I already have at least a couple cases where I already use external json files strictly for lookups, and they are loaded as --argfile in v1.4.

Dependencies couldn't force --argfile. So this was rather urgent, and took the lion's share of the work.

• 56ae88d says default (system-dependent, #ifdef WIN32) paths are replaced with jq -L. Don't, as they are annoying to regenerate externally. Either:
  • Prepend.

Can I assume you mean "prepend the user's -L... path to the default?

Correct.

I'll make it so.

If the problem is OS-dependence then I can always add a token for "default system path here".

That could work; would add some prepend/append flexibility.

I just merged the two things.

• Add a new flag which adds to the path.

Prepend or append?

Haven't decided which is more useful, if there's only once choice.

I haven't either. I think overriding the builtin setting is the thing that the user wants. It may not be what everyone wants, but it should be the 80/20, and acceptable enough.

• Drop the -L functionality altogether. Remove ~/.jq support when module system integrates #485

It's quite useful for testing; I could drop it from the docs but keep it. It could also be useful for keeping multiple universes of jq modules lying around. (Think NixOS-style packaging.) I'd rather keep it altogether.

I don't really see the usability of -L once the import resolving algorithm is in place. A "local universe" can already be defined in the jq.json dependencies.

Sure it's useful! Suppose i have a jq program I'm working on and I want to try it with different installed universes of modules: with -L I can do that as just a small CLI change. Without -L I'd have to move the .jq file around, or change its imports' search metadata. -L is a big hammer; sometimes that's what you need.

Now, $JQ_LIBRARY_PATH, that I might remove: I have that sort of environment variable's existence. It generally causes nothing but trouble. Because jq has dependent-relative searching by default, I suspect no one will use $JQ_LIBRARY_PATH anyways, so that I should remove, yes.

[nicowilliams]

Oh, actually, -L replaces the builtin path completely. I think that's probably a desirable result, though a builtin for printing the path would be nice (DONE). Also, I did remove the JQ_LIBRARY_PATH.

And now, I'm done for the night. Again, the import_659 branch of my clone has these bits.

[nicowilliams]

$ ./jq -cn -L /tmp -L /nowhere 'get_search_list'
["/tmp","/nowhere"]
$ 

[nicowilliams]

@joelpurra

I had some bugs in the linker left. I pushed a new HEAD (sorry for the rebasing) to import_659 that makes the deep case almost work: the only problem left is your having an "src" directory in the way but not named in the imports...

For example, renaming tests/execute-deep-dependencies/package-source/anotheruser/pack1/src/main.jq to tests/execute-deep-dependencies/package-source/anotheruser/pack1/pack1.jq makes it possible to import it:

$ git mv tests/execute-deep-dependencies/package-source/anotheruser/pack1/src/main.jq tests/execute-deep-dependencies/package-source/anotheruser/pack1/pack1.jq
$ ../jq/jq -n -L $PWD/tests/execute-deep-dependencies/package-source/ 'import "anotheruser/pack1" as pack1; .'
jq: error: module not found: someuser/pack4

jq: 1 compile error
$ 

Now fix pack4 and it works:

$ git mv tests/execute-deep-dependencies/package-source/someuser/pack4/src/main.jq tests/execute-deep-dependencies/package-source/someuser/pack4/pack4.jq
$  ../jq/jq -n -L $PWD/tests/execute-deep-dependencies/package-source/ 'import "anotheruser/pack1" as pack1; pack1::f'
"abc123"
$

Do you really want that "src" directory in the way? Seems gratuitous. Granted, if I'm running from a source tree that I'm building, and I'm "building" the .jq files, then I might want a directory in the way, but probably "build", not "src" (jq aside I prefer not building in place, but I digress).

[nicowilliams]

Similarly renaming pack3's and pack4's .jq files finally makes running the deep example work:

$ ../jq/jq -n -L $PWD/tests/execute-deep-dependencies/package-source/ -f tests/execute-deep-dependencies/local-project/src/main.jq
"abc123def456ghi789"
$ 

[nicowilliams]

Ditto flat.

[nicowilliams]

@joelpurra Sent a PR :)

[nicowilliams]

Eh, I didn't close this issue, I just pushed my changes to master so I can make a 1.5rc1 tag.

EDIT: Yes, I know that github closed this automatically because the commit said "fix #659".

[joelpurra]

@nicowilliams: jq's make check tests fail for nicowilliams/jq@86bc662, the current HEAD of nicowilliams@import_659

$ git clean -fdX :/ && autoreconf -i && ./configure && make clean && make -j8 && make check && ./jq --version

$ ./jq --version
jq-1.4-214-g86bc662

By the way: I see the clean up routines in setup.sh, but it seems hacky. git clean -fdX :/ is the way to clean out generated/ignored files from a git repo. (I use git reset --hard to clean out non-ignored files, but it's risky for active developers.)

---

I'll start looking at the rest now.

[joelpurra]

@nicowilliams: Oh, I see now that stedolan@master is newer than import_659. I'll use that, as tests pass.

[joelpurra]

@nicowilliams wrote:

For example, renaming [...] makes it possible to import it:

There have been ways for jqnpm to (in my opinion) work around most obstacles in previous jq builds too.

Sorry, but I don't agree with repeating the last part of a package name to resolve the main .jq file -- either in the root nor repeating the package name twice as a subfolder like jq-my-package/jq-my-package.jq (which is inside the package folder with the name jq-my-package/). Prefer the main configuration option in jq.json with the default/fallback src/main.jq. See jqnpm/jqnpm#5.

Do you really want that "src" directory in the way?

Yes -- I like separating code from the root directory -- it is a sane default for public open source packages. Separation of concerns, considering how many README.md/LICENSE/.gitignore/.travis.yml/jq.json files there are in the root already. (Without repeating the package name multiple times.) See jqnpm/jqnpm#5.

Keep in mind that my suggested import resolving algorithm is what I hope to see implemented in jq.

---

@nicowilliams: I'm very thankful for your help. Sorry that I'm not up to speed with C and the jq internals enough to make the changes myself.

[joelpurra]

@nicowilliams: added an override for you to test jq with: https://github.com/joelpurra/jqnpm/blob/master/BUILDING.md#override-for-the-temporary-jq-wrapper-jqnpm-execute - please do, so you can see what parts I expect from jq when I remove the temporary jqnpm execute wrapper.

If you want to see commands as executed by jqnpm execute, you can also use export JQNPM_DEBUG_LEVEL=5

New tests -- primarily to test jq

Wrote the failing tests to match my expectations and wishes, as a way forward for jq -- not to work around the currently implemented jq import resolving algorithm.

• Renamed the test folders from execute-* to jq-*, so as to make it clear that jqnpm execute is not the way I expect things to work.
• Added a test for direct file paths.
• Added tests for other main paths in jq.json.

[joelpurra]

@nicowilliams wrote:

@joelpurra Hey, thanks for teaching me a bashism I didn't know.

Haha, glad my copy-paste bash hacks are useful outside of their current, narrow scope. More hacks in my shell-based music queue manager npshell - check it out =)

[nicowilliams]

On Thu, Jan 01, 2015 at 01:21:41AM -0800, Joel Purra wrote:

By the way: I see the clean up routiens in setup.sh, but it seems hacky. git clean -fdX :/ is the way to clean out generated/ignored files from a git repo. (I use git reset --hard to clean out non-ignored files, but it's risky for active developers.)

Yeah, I never use git clean in active workspaces. It's very risky.

Instead I keep a separate workspace for release work, but I get lazy and don't make check in that workspace before every push. I do use it around releases, and then I realize that I forgot to git add some file and so on; it's a bit embarrassing. What I should do is rig my dev workspace to always fetch and make check in the clean workspace when either making with no extant changes or when pushing.

[nicowilliams]

Let's come up with a compromise for the "/src/" thing that works for us both and, barring any new bugs, we'll be ready for a 1.5 release.

Do keep banging on jq. If you find any bugs, let me know!

[joelpurra]

Note: the jqnpm src/main.jq conversation continued in jqnpm/jqnpm#5.

[joelpurra]

Glad to let you know that jqnpm now works with single-level dependencies! Use the jq command wrapper jqnpm execute until jq catches up regarding some path handling issues. Check it out, and create some packages!
https://github.com/joelpurra/jqnpm
https://github.com/joelpurra/jqnpm/wiki/

Edit: jqnpm has been able to install "infinite" multi-level dependencies a while, it's just they can't be import ...-ed properly. Yet. Work is in progress.

[joelpurra]

Do check out the command jqnpm generate, which makes it easy to generate skeletons for packages, such as Zeros.
https://github.com/joelpurra/jq-zeros

Happy package generating =)

[joelpurra]

I've created proof-of-concept patch showing how to find the package root for a given path, and how to use package root-relative P/ paths for importing packages. See my package-root branch (diff against stedolan/jq master). This means that deep, "infinite level" dependencies jqnpm install'ed in P/.jq/packages/ work -- without jq -L!

• Use the jqnpm branch jq-package-root if you want a jq -f workaround as well.
• Haven't tested the code on Windows.
• Can't install Valgrind on Mac OS X 10.10, so haven't tested for leaks -- presuming there's more than one, so please help me here.
• jq_find_package_root(...) might be useful in other places as well, such as when looking for the package's jq.json.
• Not familiar with the jq core, please have a look and send pull-requests to my branch.

Together with @nicowilliams' patch 6e7cf81 for jq/main.jq things are looking pretty good =)

Next steps:

• Detect the main script automatically, if none was given as jq -f or jq '...'.
  • Use jq '.main' P/jq.json to get the path, if available in the package root from $PWD.
  • Use the default path P/jq/main.jq, if available in the package root from $PWD.
  • A jq -f workaround is still in place in jqnpm execute, but it should be deleted.

Yay, very happy things are moving forward =)

[joelpurra]

In case you're using Mac OS X with Homebrew, please try out the new jqnpm formula:

brew tap joelpurra/joelpurra
brew install --HEAD jq          # Use --HEAD to get jq v1.5rc1+.
brew install jqnpm

Let me know if it works!

[pkoppstein]

@joelpurra wrote:

Let me know if it works!

Hmmm. I typed brew install joelpurra/joelpurra/jqnmp but I don't think things are quite ready yet. Here's why:

==> Installing dependencies for jqnpm: bash, bison, git, shunit2

But I already have recent homebrew versions of bash, bison and git!

(Specifically: bash 4.3.26 ; bison 3.0.2; and git 2.0.0.)

2. bison was installed keg-only, so making jq is likely to fail.

3. Regarding jqnpm, brew said:

Note: checking out '7d4174350a141a526a8a8758d12e55463a679b12'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

My target audience will be frightened off by talk about "detached HEADs".

[joelpurra]

==> Installing dependencies for jqnpm: bash, bison, git, shunit2
But I already have recent homebrew versions of bash, bison and git!

(Specifically: bash 4.3.26 ; bison 3.0.2; and git 2.0.0.)

I'm all brew and see it as the primary alternative on Mac. This is not a jqnpm issue; you can try brew install --ignore-dependencies jqnpm if you want. Or even better - please create a package for your preferred package manager, it'd be great!

2. bison was installed keg-only, so making jq is likely to fail.

It doesn't break brew install jq nor brew install --HEAD jq for me¹. Did you try it?

3. Regarding jqnpm, brew said:
   [---]
   My target audience will be frightened off by talk about "detached HEADs".

Switched to using git and tags instead of <tag>.tar.gz and sha256 checksums about an hour ago. Not sure why brew has chosen not to hide that message, so I asked them in Homebrew/legacy-homebrew#36173. Seems a bunch of other formulae are using the same style in brew, around 40 or so. I'm not worried - but if your users are, send them to me and I'll talk to them.

---

¹ Seems brew takes care of the bison keg path during brew install jq. If I build jq manually from the git repository I do:

brew info bison
# Look for the path: /usr/local/Cellar/bison/3.0.3
export "/usr/local/Cellar/bison/3.0.3/bin:$PATH"

# jq build steps here

[joelpurra]

@pkoppstein: oh, and thanks for trying it out =)

[nicowilliams]

I don't mean to be AWOL on this. I just am :(

I hope to find time for jq. But there's a lot on my plate at this time.

[joelpurra]

@nicowilliams: no worries, I believe you gave a heads-up in December =)