Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an API to generate registry index data #4497

Open
4 tasks
carols10cents opened this issue Sep 15, 2017 · 25 comments
Open
4 tasks

Add an API to generate registry index data #4497

carols10cents opened this issue Sep 15, 2017 · 25 comments
Labels
A-cargo-api Area: cargo-the-library API and internal code issues A-registries Area: registries C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. T-cargo Team: Cargo

Comments

@carols10cents
Copy link
Member

carols10cents commented Sep 15, 2017

See RFC 2141 for more details/context.

Currently, the knowledge of what goes in the registry index in what format lives in crates.io:

In order to support minimal registries, we'd like cargo to be able to generate the JSON information that should go into a registry for the current crate.

Cargo does already understand how to read these entries! We need to teach it how to create them as well :)

So as I understand it, this would involve:

@carols10cents carols10cents added A-new-subcommand Area: new subcommand C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` E-mentor T-cargo Team: Cargo A-registries Area: registries labels Sep 15, 2017
@carols10cents
Copy link
Member Author

In order to test out #4506, I've created a minimal registry! The part that this proposed command would create is this file's contents, given the current crate.

I don't think we envision people running this command and doing something manually with the output, though that will be possible.

The main goal is to enable any tool to be able to create the same registry entries that crates.io creates right now, without needing to run all of crates.io.

@stephanbuys
Copy link

@carols10cents from what I can tell #4506 deals with cargo support for alt-registries, not this issue (ie. generating the actual minimal registry data). I'd like to have a stab at it if this command is still up for grabs.
Does it make sense to abstract some of this work out into a 3rd party crate to be used by crates.io and other registry implementations?

@carols10cents
Copy link
Member Author

@stephanbuys

@carols10cents from what I can tell #4506 deals with cargo support for alt-registries, not this issue (ie. generating the actual minimal registry data).

Yes, #4506 is only tangentially related-- someone on gitter was asking for an example of what a "minimal registry" would look like and what this command needs to generate, and #4506 had just happened to prompt me to do just that :)

Does it make sense to abstract some of this work out into a 3rd party crate to be used by crates.io and other registry implementations?

Cargo can be used as a dependency, and someone on gitter also suggested adding a public API function that returns the text this command would output, so I think that would work. Does that sound ok?

@liranringel
Copy link

I started to look into it.
RegistryPackage contains DependencyList which manually implements Deserialize, so I guess it's also necessary to manually implement Serialize.
Good chance to learn some serde.

@liranringel
Copy link

liranringel commented Oct 3, 2017

What would be the best way to calculate the checksum?
I thought to use cargo::ops::package to get the .crate file.

Edit: I noticed that the hash changes on every cargo package command, perhaps a timestamp is inserted, so this is meaningless to get the checksum.

@withoutboats
Copy link
Contributor

withoutboats commented Oct 4, 2017

@carols10cents Do you have strong opinions on whether this should live as a cargo subcommand or a separate package we publish to crates.io? My only concern is that cargo could become one of those tools where you type cargo --help and you're drowned in dozens of niche subcommands & you can't find what you're looking for. Do you think its too much to ask people to cargo install this?

@carols10cents
Copy link
Member Author

I guess I don't feel super strongly about whether this is part of cargo or external, but it should be in the rust-lang org and it's seeming like having a library with a public api function as well as a binary would be useful...

@withoutboats
Copy link
Contributor

I definitely agree about both of those points.

@liranringel
Copy link

So where should it be?
To make it clear, I interpret the task as getting a .crate file and generating it's index metadata.
If it won't be in cargo, it will probably be in a crate that depends on cargo, or cargo will depend on it (large refactor).

@cswindle
Copy link
Contributor

cswindle commented Nov 9, 2017

Is anyone actively working on this, if not then I may make a start on a new crate to add the interface, this way it makes it a lot easier for versioning the interface for external crates to use (not to mention the reduction in time as cargo takes ages to compile when you are only needing a tiny amount of the function).

@liranringel
Copy link

I'd like to but no one answered me here or in gitter, and I don't want to make such important decisions by myself.

@cswindle
Copy link
Contributor

After taking a look it would seem to me more useful to have a command which adds the entry in a git registry directly, for example this code. It just seems to me like anyone who uses a command to generate the fragment of index would need to also perform the following:

  • update the git repo for the index
  • calculate the index file location for the crate
  • open the index file and append the new data to it
  • commit the changes to the git repo
  • push the changes to the git repo (I guess this could be optional for local git repos for testing)

That seems like a bunch of code which has already been written that anyone who uses the new command will also need to write (or copy). Is there some use case which I am missing?

@withoutboats
Copy link
Contributor

We don't want it to do things that are highly contextual to specific users. Possibly it could make a git commit for you, but definitely not pull or push, since those involve questions of which remote is the source of truth, and what authentication do you use.

@cswindle
Copy link
Contributor

Leaving out the pull and push and just updating a git repo would still make for quite a clean split, rather than spreading the knowledge of the registry layout around different crates.

@synek317
Copy link

I have implemented a working prototype here: https://github.com/synek317/cargo/tree/synek317_generate_index_metadata

I decided to create generate-index-metadata subcommand despite I'm not sure if it should be included in cargo. Anyway, this subcommand is just a 'working stub' (poor error message, almost no parameters).

@hallfox
Copy link

hallfox commented Jun 21, 2018

Any updates on this?

@synek317
Copy link

@hallfox have you checked my branch? I think it is ready but unfortunately I had no time to push it.

@ninrod
Copy link

ninrod commented Jun 21, 2018

with this issue resolved, would we entreprise users, trying to use rust behind a firewall, with access to an artifactory server, be finally able to use rust? we have no access whatsoever to crates.io.

@hallfox
Copy link

hallfox commented Jun 21, 2018 via email

@synek317
Copy link

Actually, I was blocked with two things (+ time):

  1. I'm not sure if this command should really be part of cargo (maybe cargo should only export some public items and generate-index-metadata should be external command?)

  2. In order to check if everything works fine, I was building a tool that allows user to create crate registry in any github repository. I was close to finish a draft, but there are still some small things to do: https://github.com/synek317/cargo-git-registry

@repnop
Copy link

repnop commented Oct 28, 2018

@synek317 have you had time to continue with this?

@ehuss
Copy link
Contributor

ehuss commented Jan 15, 2019

Rather than implementing this in-tree, I have posted an experimental subcommand which relies on cargo metadata instead. It is available at https://crates.io/crates/cargo-index (requires the latest 1.33 nightly). If anyone is interested in it, feel free to send feedback on its issue tracker or take any of the code from within. In general it's a fairly straightforward translation of the cargo metadata output into the JSON structure needed in the index.

@synek317
Copy link

@ehuss this is not a good solution. I mean, I was also concerned if generate-index-metadata should be included in cargo or in subcommand as you can see in my comment above. Anyway registry structs are currently defined in crates.io and in cargo, as carols10cents pointed it out in the first comment. Instead of unifying it, you are adding another copy.

I've also some personal objections because now someone's effort (mine or yours, probably mine) will be wasted. If anyone had answered my previous question, I could just move it to the subcommand.

Not even mentioning that your implementation is something different than @carols10cents was asking in the first comment.

@ehuss
Copy link
Contributor

ehuss commented Jan 15, 2019

@synek317 I apologize for the lack of coordination. The code I posted is not intended to be a permanent solution. It was intended to explore what would be needed to support this functionality outside of Cargo, and to gauge the utility.

The Cargo team is actively discussing the possibility of reorganizing some of the internals to make them more easily accessible in isolation and in a more stable form. However, it is not entirely clear, yet, exactly how that should appear, or how useful it would be. Different projects have different needs, so it's not always obvious how it should look and who it would serve.

@ehuss ehuss removed the E-mentor label Aug 22, 2019
@gilescope
Copy link
Contributor

Given time has gone by, is there some feedback from people using https://crates.io/crates/cargo-index ? I have to say the docs didn't happen to mention it or I would have tried it out months ago. Maybe cargo local-registry should mention it in the alternatives section on their readme?

@epage epage added A-cargo-api Area: cargo-the-library API and internal code issues S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. and removed A-new-subcommand Area: new subcommand labels Oct 17, 2023
@epage epage changed the title Add a command to generate registry index data Add an API to generate registry index data Oct 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cargo-api Area: cargo-the-library API and internal code issues A-registries Area: registries C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. T-cargo Team: Cargo
Projects
None yet
Development

No branches or pull requests