Have Package Managers Support a User Provided pre-populated cache or other endpoint.

[gfs]

No description provided.

[jpinz]

So the idea for us is that we want to override the existing check. We have our own cosmos db, and our own registry. We have also discussed the idea of using a bloom filter.
Basically, I think the idea is that we would be able to override the PackageExists method, to first check our bloom filter, then our registry/cosmos db or something, and finally as a last resort, check the public registry for the package manager.
We also have discussed the idea of checking private registries along with the public one to help with Dependency Confusion attacks a la https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

[scovetta]

We already support using environment variables to override the default registry endpoints, and support local caching of some results, but extending this would make lookups much faster, especially for typosquatting as the character set or package name length grows.

Having a publicly accessible endpoint to rapidly look up multiple packages would be interesting -- I'd rather not generate 1000s of requests when one would do.

GET /PackageExists
name: ["foo", "fo0", "f0o", "e0o"...]

Response:

{ "exists": ["foo", "e0o" ...]}

Having a separate oss-dependency-confusion tool might be reasonable too:

oss-dependency-confusion --repo-list=registry.npmjs.com,registry.contoso.com pkg:npm/foobar

[gfs]

support local caching of some results

Note that the current cache implementation does not cache 404's. For most typo-squatting checks you are going to be querying things that do not exist so most will be a cache miss in the OSSGadget implementation.

[gfs]

We already support using environment variables to override the default registry endpoints

It sounds like they are not implementing the same API surface as the original registry so this likely will not work in this case.

[gfs]

It sounds like the solution that you might employ would be to have a "CosmosProjevtManager" that you can implement the custom behavior in the ProejctExists method. If the calls to your cache fail to surface anything you could then subinstantiate the correct package manager based on the Purl and call into the default behavior.

I believe this is possible with the current implementation of of the lib in #277.

[jpinz]

It sounds like the solution that you might employ would be to have a "CosmosProjevtManager" that you can implement the custom behavior in the ProejctExists method. If the calls to your cache fail to surface anything you could then subinstantiate the correct package manager based on the Purl and call into the default behavior.

I believe this is possible with the current implementation of of the lib in #277.

Sounds good! I will give it a shot once it is published to NuGet, it's hard to tell exactly just from reviewing the code on GitHub.

[gfs]

Here's a skeleton I worked up that might help. I believe all you have to do below is implement the CosmosHas method to be compatible with find squats.

namespace Microsoft.CST.OpenSource.Shared
{
    using System;
    using System.Threading.Tasks;
    public class CosmosProjectManager : BaseProjectManager
    {
        public CosmosProjectManager(string destinationDirectory) : base(destinationDirectory)
        {
        }

        public override async Task<bool> PackageExists(PackageURL purl, bool useCache = true)
        {
            if (await CosmosHas(purl))
            {
                return true;
            }
            else
            {
                BaseProjectManager? manager = ProjectManagerFactory.CreateProjectManager(purl);
                if (manager is not null && await manager.PackageExists(purl))
                {
                    return true;
                }
            }

            return false;
        }

        private async Task<bool> CosmosHas(PackageURL purl)
        {
            // Your code here
            throw new NotImplementedException();
        }
    }
}

And then you can just.

using Microsoft.CST.OpenSource.FindSquats.ExtensionMethods;

IEnumerable<PackageURL> urls = { your package URLs to check }
CosmosProjectManager cpm = new("some/path");
foreach(PackageURL targetPackageURL in urls)
{
    await foreach (FindPackageSquatResult potentialSquat in cpm.EnumerateSquats(purl))
    {
        // Process the squat.
    }
}