Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have Package Managers Support a User Provided pre-populated cache or other endpoint. #282

Open
gfs opened this issue Nov 18, 2021 · 7 comments
Labels
enhancement New feature or request

Comments

@gfs
Copy link
Contributor

gfs commented Nov 18, 2021

No description provided.

@gfs gfs added the enhancement New feature or request label Nov 18, 2021
@jpinz
Copy link
Member

jpinz commented Nov 18, 2021

So the idea for us is that we want to override the existing check. We have our own cosmos db, and our own registry. We have also discussed the idea of using a bloom filter.
Basically, I think the idea is that we would be able to override the PackageExists method, to first check our bloom filter, then our registry/cosmos db or something, and finally as a last resort, check the public registry for the package manager.
We also have discussed the idea of checking private registries along with the public one to help with Dependency Confusion attacks a la https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

@scovetta
Copy link
Member

We already support using environment variables to override the default registry endpoints, and support local caching of some results, but extending this would make lookups much faster, especially for typosquatting as the character set or package name length grows.

Having a publicly accessible endpoint to rapidly look up multiple packages would be interesting -- I'd rather not generate 1000s of requests when one would do.

GET /PackageExists
name: ["foo", "fo0", "f0o", "e0o"...]

Response:

{ "exists": ["foo", "e0o" ...]}

Having a separate oss-dependency-confusion tool might be reasonable too:

oss-dependency-confusion --repo-list=registry.npmjs.com,registry.contoso.com pkg:npm/foobar

@gfs
Copy link
Contributor Author

gfs commented Nov 18, 2021

support local caching of some results

Note that the current cache implementation does not cache 404's. For most typo-squatting checks you are going to be querying things that do not exist so most will be a cache miss in the OSSGadget implementation.

@gfs
Copy link
Contributor Author

gfs commented Nov 18, 2021

We already support using environment variables to override the default registry endpoints

It sounds like they are not implementing the same API surface as the original registry so this likely will not work in this case.

@gfs
Copy link
Contributor Author

gfs commented Nov 18, 2021

It sounds like the solution that you might employ would be to have a "CosmosProjevtManager" that you can implement the custom behavior in the ProejctExists method. If the calls to your cache fail to surface anything you could then subinstantiate the correct package manager based on the Purl and call into the default behavior.

I believe this is possible with the current implementation of of the lib in #277.

@jpinz
Copy link
Member

jpinz commented Nov 18, 2021

It sounds like the solution that you might employ would be to have a "CosmosProjevtManager" that you can implement the custom behavior in the ProejctExists method. If the calls to your cache fail to surface anything you could then subinstantiate the correct package manager based on the Purl and call into the default behavior.

I believe this is possible with the current implementation of of the lib in #277.

Sounds good! I will give it a shot once it is published to NuGet, it's hard to tell exactly just from reviewing the code on GitHub.

@gfs
Copy link
Contributor Author

gfs commented Nov 19, 2021

Here's a skeleton I worked up that might help. I believe all you have to do below is implement the CosmosHas method to be compatible with find squats.

namespace Microsoft.CST.OpenSource.Shared
{
    using System;
    using System.Threading.Tasks;
    public class CosmosProjectManager : BaseProjectManager
    {
        public CosmosProjectManager(string destinationDirectory) : base(destinationDirectory)
        {
        }

        public override async Task<bool> PackageExists(PackageURL purl, bool useCache = true)
        {
            if (await CosmosHas(purl))
            {
                return true;
            }
            else
            {
                BaseProjectManager? manager = ProjectManagerFactory.CreateProjectManager(purl);
                if (manager is not null && await manager.PackageExists(purl))
                {
                    return true;
                }
            }

            return false;
        }

        private async Task<bool> CosmosHas(PackageURL purl)
        {
            // Your code here
            throw new NotImplementedException();
        }
    }
}

And then you can just.

using Microsoft.CST.OpenSource.FindSquats.ExtensionMethods;

IEnumerable<PackageURL> urls = { your package URLs to check }
CosmosProjectManager cpm = new("some/path");
foreach(PackageURL targetPackageURL in urls)
{
    await foreach (FindPackageSquatResult potentialSquat in cpm.EnumerateSquats(purl))
    {
        // Process the squat.
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants