Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secure Distributed Registry for deno #3616

Closed
ameerthehacker opened this issue Jan 7, 2020 · 19 comments
Closed

Secure Distributed Registry for deno #3616

ameerthehacker opened this issue Jan 7, 2020 · 19 comments

Comments

@ameerthehacker
Copy link

Deno is goal to create a distributed package management solutions but

  1. How do we ensure the immutability of those packages?
  2. How to we take care about the security vulnerability in them (or we don't do anything about it)
  3. Since we have a set of distributed packages from various packages, how do deno users know which source to use for the same package

Thoughts on the solution

Is is possible that we create a registry system that anyone can host to share their packages (we do security checks, immutability support in the service) and is there is a way by which deno can detect that we are downloading packages from our service hosted registries securely without being fooled

@lucacasonato
Copy link
Member

I'm going to compare this with Go which has a similarly distributed module system as Deno:

  1. Go has a service that handles module immutability: https://proxy.golang.org/. It is enabled by default in Go 1.13.
  2. Go doesn't have security reviews for modules.
  3. Always download it from the location the module author is publishing it. If it is on deno.land/x then that is the place to download it from. You can also use raw.githubusercontent.com URLs with commit hashes, that is pretty immutable. There are also services like https://denolib.com.

Also Go uses standard https:// and ssh:// git operations to fetch modules. That means if you trust the domain you can trust that the communication to that domain is secure because of https.

@ameerthehacker
Copy link
Author

thanks @lucacasonato that gives a lot of clarity. Are we planning to build similar proxy system for deno too or is it already built?

@lucacasonato
Copy link
Member

It doesn't exist right now, and I don't know if it is something that is planned. @ry?

@kitsonk
Copy link
Contributor

kitsonk commented Jan 7, 2020

https://deno.land/x/ is effectively the equivalent to the proxy and documents on that page how other packages can be added. The optional support for lock files (#3231) was the solution for sub-resource integrity to compliment this.

@lucacasonato
Copy link
Member

lucacasonato commented Jan 7, 2020 via email

@rsp
Copy link
Contributor

rsp commented Jan 11, 2020

@ameerthehacker @lucacasonato, those are interesting issues worth thinking about.

The lock files can ensure that I always get the same dependencies that I get the first time when I installed them, but it doesn't ensure that I get the right files for the first time to begin with.

Let's say that a malicious user creates a useful library with no bad code. Something that helps with handling credit cards. It's all nice, people audit the code and use it with lock files. One day, the author decides to steal the data and suddenly publishes a new version and changes the tags on GitHub so that it looks like the old version. From that point in time the following happens:

  1. For those who used lock files - their code stops working on CI and deploys.
  2. For those who didn't use lock files - their data is stolen.
  3. New users who create lock files now - they have data stolen and a false sense of security.
  4. For new users who don't use lock files are equally screwed as No. 3.

This is a problem of mutability of GitHub version tags.

Possible solution would be to have deno.land/x or a service like deno.land/x to either:

  1. Cache the contents of files instead of redirecting to GitHub
  2. Redirect to URLs with commit hashes instead of version tags

Currently, deno.land/x seems to return the file contents instead of redirecting to GitHub. I am almost sure that it used to redirect to GitHub before.

I don't know if the current behavior is getting the original file once with a guarantee of immutability, or is it just a caching for convenience. I suppose it has no guarantees or otherwise the code for master branches would never get updated.

In any case, what you are proposing here might be done as a third party service with no need to be centralized. In fact, it would be interesting to see other services like deno.land/x to demonstrate different ideas to implement a service like this.

@fakoua
Copy link

fakoua commented Jan 11, 2020

Also it is very important to add statistics to help developers to pick top modules

@kitsonk
Copy link
Contributor

kitsonk commented Jan 12, 2020

The quality or usefulness of a module is not determined by its statistics, for example https://www.npmjs.com/package/isarray.

@hayd
Copy link
Contributor

hayd commented Jan 13, 2020

Screen Shot 2020-01-12 at 17 08 15

😮

@rsp
Copy link
Contributor

rsp commented Jan 14, 2020

@kitsonk Good point 😄

return toString.call(arr) == '[object Array]'

https://github.com/juliangruber/isarray/blob/master/index.js#L4

@mudivili
Copy link

mudivili commented May 11, 2020

I install git(hub/lab), etc. hosted modules over SSH. Is it possible for deno now or later, to provide such ability to load modules that are protected by SSH keys?

@kitsonk
Copy link
Contributor

kitsonk commented May 11, 2020

Using an ssh key only validates that you are who you say you are. Why is this a requirement for being able to download a module?

@mudivili
Copy link

@kitsonk, the requirement is to download private modules. In a distributed software architecture, some modules could need to be reused in many or all services.

So far, I had used github, gitlab, bitbucket along with ssh keys to reuse those modules.

I am wondering how to securely distribute private packages across services in deno.

@kitsonk
Copy link
Contributor

kitsonk commented May 12, 2020

SSH really isn't conducive for single file fetches. GitHub supports passing an auth token as part of the URL which would give you access to a private repo (and I assume it would work with other services as well). Still that is likely not sufficient as it requires you embedding your token in the code. I opened #5239 which would mean it would be passible on the command line.

@ghost
Copy link

ghost commented May 21, 2020

We discussed options for a distributed / decentralized registry for Deno in a different issue, if you're interested, have a look at denoland/dotland#406.

I've started working on a registry a while ago, which should solve most of the problems we talked about. For a more detailed explanation, have a look at denoland/dotland#406 (comment) or check out the website https://deno.to/.

It's still a work in progress, but I hope to get a first version up and running soon.

@boskiv
Copy link

boskiv commented May 27, 2020

Please don't forget about private registries. Many companies want to use their own private modules in projects.
There is an important part of the development process.
Will Deno support authorisation to load modules from private repositories ?

@t8
Copy link

t8 commented Jun 29, 2020

I might be a little late to this conversation, and I apologize for that. I think that it would be helpful for me to mention a module registry and CDN (of sorts) that my team and I developed: https://nest.land.
Nest.land solves several of the reasons why this issue was brought about. Let me address each one:

Q: How do we ensure the immutability of those packages?
A: Nest.land's core lies on the blockchain. All module code is sent straight to the Arweave blockchain through something the Arweave team likes to call the "permaweb." We're looking at code as bundles of transactions now, rather than just as files. In a blockchain, deleting this transaction data is physically impossible, as it is distributed across hundreds of nodes around the world, and counting.

Q: How to we take care about the security vulnerability in them (or we don't do anything about it)
A: Module code is uploaded to the permaweb, but registry data is not. This gives us a certain degree of control over what users see on our registry. Though a malicious module would be permanently accessible, we don't have to display it to the user. We're also working on archive and audit systems. The nest.land archive will be a place where registry data is "backed up" at a set interval so that if our service goes down, the registry can continue to function. The audit feature that can warn a user if a module they are importing is malicious from our CLI.

Q: Since we have a set of distributed packages from various packages, how do Deno users know which source to use for the same package
A: This isn't necessarily a problem that any one person or entity can solve. In fact, I think that a major goal of Deno is to give the user flexibility with the module that they want to import and from where they want to import it. Similar to capitalism, different registries can and will coexist. I believe that the competition between this coexistence will bring better services, encourage innovation, and will benefit the end user for the lifespan of Deno.

A free, open source, and decentralized module registry/CDN is simply my take on what the Deno ecosystem could benefit from.

@lucacasonato
Copy link
Member

deno.land/x has gotten a major upgrade recently (https://deno.land/posts/registry2), with the main change being that all code being served is now immutable. Also there are other registries like nest.land that provide even stricter immutability guarantees. I think this is mostly a userland issue and deno provides the necessary tools to prevent rogue modules (secure by default, lock files). @ameerthehacker can this be closed?

@ameerthehacker
Copy link
Author

awesome @lucacasonato this looks greats, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants