Basic lockfile support

[travisperson]

This is basic support for a lock-file concept discussed between @Stebalien and @whyrusleeping this week.

See also: whyrusleeping/gx-go#49

[Stebalien]

Per our discussion:

type Lock struct {
    // Empty for the top level package.
    // In the future, we could use IPNS!
    Ref string
    Language string

    // maps language namespace -> name in language -> dependency
    Deps map[string]map[string]Lock
}

[Stebalien]

(or we could keep it non-recursive, that was just a thought)

[Stebalien]

Why the "double map": The initial proposal was "dep name in language" -> definition. For example:

{
    "github.com/ipfs/go-ipfs": {
        "ref": "/ipfs/....",
        "language": "go"
    }
}

Unfortunately, this falls apart when we need multiple namespaces. For example, with CGO, we might want (not well thought out):

{
    "github.com/ipfs/go-ipfs": {
        "ref": "/ipfs/....",
        "language": "go"
    },
    "libs/libsnark.a": {
        "ref": "/ipfs/....",
        "language": "rust"
    }
}

package thing

// #cgo LDFLAGS: -L${SRCDIR}/libs -llibsnark
import "C"

The problem here is that there's the "go package" namespace and the "cgo" namespace (really, just the filesystem) and we need some way to disambiguate.

[Stebalien]

The double map would be:

{
    "go": {
        "github.com/ipfs/go-ipfs": {
            "ref": "/ipfs/....",
            "language": "go"
        }
    },
    "cgo": { // files?
        "libs/libsnark.a": {
            "ref": "/ipfs/....",
            "language": "rust"
        }
    }
}

[whyrusleeping]

Added a version field here. Let's bikeshed a wee bit more, and then just pick something that works well enough for now.

[whyrusleeping]

Alright, now lock-install fetches all the dependencies to their gx paths inside the cache directory (which is currently just $PWD/vendor) and then symlinks everything into place. The files written into the cache directory are set read only. This works to build both ipfs and filecoin now!

[schomatis]

This seems to have pretty much the same functionality as fetchDependencies with the difference that it traverses a LockFile instead of a Package, could we leverage it? (maybe having a more abstract function that receives a closure that knows how to extract the dependency hashes and install paths?)

[travisperson]

Yes, absolutely.

[schomatis]

I'm assuming that this naming model was inherited from LoadPackageFile but could we avoid naming the main -> gxutil functions the same? (Or add some comments that clearly distinguish them both?) This can be a bit confusing for a new user.

[schomatis]

Could we document what's the difference between a LockDep and a Dependency?

I mean, why are we using this new structure instead of extending Dependency or including it in this structure hierarchy? I'm not saying it should, but reading this automatically makes me thing of what we now call a dependency (inside the package.json), and now I'll be seeing a different file that also talks about dependencies but with a different structure which can be confusing.

I understand that conceptually both are referring to a package (that is a dep of another package) but with a different representation in a different context so it would be nice to make that link stronger.

[travisperson]

I updated this branch to use the recursive structure and implemented a rather crude worker pattern that can probably be updated a bit. It was the only way I could see doing it that wouldn't run into lock contention (though that might be able to be solved with extending the workers). Both the package install and lock install might be able to share some code if we supply a postFetch callback method as pointed out by @schomatis. I may look into this a bit further.

However, a few things immediately become apparent as I started to work on this code a bit.

1. Making gx packages read-only breaks some hooks as they write a file to keep track
   I'm noting this because it was breaking some tests. However, this might not actually be an issue, and instead is just an artifact of not cleaning up tests. I believe only the post-install hook does this, which may not be an issue.

2. Recursive / transient deps
   While we can keep track of the entire dependency tree in the lock-file, it currently has to be flattened entirely if we want to keep gx deps entirely read-only, as we wouldn't be able to use the dependencies "vendor" directory to store it's own / conflicting dependencies.

   In the go case I don't think this is an issue. However, if we want to support more languages we'd need to handle this case. This might be as simple as allowing the "vendor" directory to be writable. Doing this though breaks some ideas down the road, such as building directly from /ipfs / /ipns.

Regardless I think this definitely shows that the language extensions of gx need to be more in the loop with regard to how the lock file is installed.

[schomatis]

1. Making gx packages read-only breaks some hooks as they write a file to keep track
   However, this might not actually be an issue, and instead is just an artifact of not cleaning up tests.

Yes, gx drops a .gx/post-install file to mark the package as installed: #208.

[whyrusleeping]

@Stebalien Are we happy with the lockfile format? I'd like to get things moving forward here.

If the lockfile format looks good, we can change the particulars of install (and lockfile generation) later.

[Stebalien]

I am. Let's merge this.

[whyrusleeping]

Thanks a bunch @travisperson ! This is exciting :)