panicked at 'could not get packages from registry; ids=...'

[emk]

To reproduce: Create a project using rust new testdummy and replace the Cargo.toml file with:

[package]

name = "testdummy"
version = "0.0.1"
authors = ["Example <example@example.com>"]

[dependencies]
uchardet = "0.0.3"

Run cargo build. You will see:

task '

' panicked at 'could not get packages from registry; ids=[testdummy v0.0.1 (file:///home/YOURUSER/src/rust-uchardet/testdummy), pkg-config v0.0.1, uchardet v0.0.3, uchardet-sys v0.0.2]; ret=[pkg-config v0.0.1, uchardet v0.0.3, uchardet-sys v0.0.2, uchardet-sys v0.0.2, testdummy v0.0.1 (file:///home/YOURUSER/src/rust-uchardet/testdummy)]', src/cargo/core/registry.rs:100

This seems to affect uchardet, but not the other packages I've tried.

[emk]

It might be worth mentioning that:

curl https://crates.io/api/v1/crates | grep uchardet

…returns nothing. But both of these return data:

curl https://crates.io/api/v1/crates/uchardet
curl https://crates.io/api/v1/crates/uchardet-sys

Not sure what's going on here.

[alexcrichton]

This is quite interesting! A few things:

• It's ok if /api/v1/crates doesn't show your crate, it's paginated by default (may be on another page).
• The crate uploaded for uchardet 0.0.3 includes the uchardet-sys directory (which it shouldn't).

What version of cargo did you use to publish the crate? When I run cargo package inside of a checkout of https://github.com/emk/rust-uchardet, it doesn't package up the uchardet-sys directory, so I'm kinda curious how that made it in there! If you run cargo package at this point, does it still include the uchardet-sys directory?

[emk]

I get my cargo distributions from rustup. I ran it yesterday afternoon, and again this morning, and I could reproduce this problem with each. I checked target/package manually, and verified that uchardet-sys was getting included in the parent package as of this morning:

$ ls target/package/uchardet-0.0.4/
Cargo.lock     .gitignore     src/           uchardet-sys/
Cargo.toml     README.md      .travis.yml    UNLICENSE.txt

However, I was suspicious about my uchardet-sys/target directory (which I had cleaned several times yesterday afternoon while packaging uchardet-sys, but maybe it was confusing things), so I ran the following command:

cargo clean && (cd uchardet-sys/ && cargo clean)

Now that this has been done, I can't find any way to reproduce the problem. I've tried repackaging uchardet-sys, etc., and everything comes up clean. However, I've added the following to my Cargo.toml out of paranoia:

exclude = ["uchardet-sys"]

I've uploaded a working uchatdet package under a new version number, too.

[emk]

...and it just broke again, even with the exclude in place. I have no idea what triggers this. I do have a uchardet-sys/target/ directory this time around, however.

Looks like I'll be manually checking this for now before uploading.

[alexcrichton]

Aha! I have found the bug. The problem here is that cargo package will leave "three packages" in the repository:

1. rust-uchardet/Cargo.toml
2. rust-uchardet/uchardet-sys/Cargo.toml
3. rust-uchardet/uchardet-sys/target/package/uchardet-sys-0.0.1/Cargo.toml

Right now there are weird interactions with this causing problems, I'll try to fix pronto!

[alexcrichton]

ooooooh, another thing which I believe is why I've never seen this before, is that your dependency is a crates.io-based dependency, not a path-based dependency. This line should be a path= dependency instead of a crates.io-dependency.

[emk]

How does that work when the crates get uploaded? Right now, I'm just
treating them as basically independent crates that happen to live in the
same git repo. Does it insert a version dependency somehow? I'll go look at
some of your packages and try to figure out how that works. Thank you!

Le Fri Nov 21 2014 at 15:10:34, Alex Crichton notifications@github.com a
écrit :

ooooooh, another thing which I believe is why I've never seen this before,
is that your dependency is a crates.io-based dependency, not a path-based
dependency. This line
https://github.com/emk/rust-uchardet/blob/master/Cargo.toml#L20-L21
should be a path= dependency instead of a crates.io-dependency.

—
Reply to this email directly or view it on GitHub
#937 (comment).

[alexcrichton]

All path dependencies are rewritten to crates.io-based dependencies on a publish operation. Each path dependency can also have a version = "0.2.0" value which will be ignored for a normal cargo build but is required for a cargo publish

[emk]

This definitely seems to have fixed the problems we were seeing yesterday.

But I've found another way to abuse Cargo and confuse it with nested target/package directories:

1. Create a package foo.
2. In a subdirectory of foo, create a package foo-extras that depends on foo using path = ".." to point to the parent directory.
3. Package foo-extras.
4. Package foo.

For me, this yields an error of the form:

Could not find `Cargo.toml` in `/home/emk/src/rust-uchardet/uchardet-encoding/target/package

I'm using last night's cargo build, as usual.

I admit that I'm doing something weird, but my thought process went as follows:

• The uchardet package is the important one in this library.
  *.But it would be nice to have a uchardet-encoding package that was fully integrated with the encoding package, so users could just say, "Get me the Encoding object for this &[u8]", or "Decode this &[u8]" or "Decode this byte stream".
• The uchardet-encoding package is really the secondary package here, and I might have a couple of them if another encoding library appears, so it makes sense to keep it in a subdirectory.
• I could create yet another GitHub project, but I actually want to develop and test these libraries together.

Now, maybe the right answer here is to say, "Please stop creatively abusing Cargo already!" And I'd be OK with that. :-) But if this seems like a vaguely rational usecase, I thought you'd be interested in knowing about the error. And cargo probably ought to stay out of target/package directories when scanning for Cargo.toml files in any case.

Thanks for putting up with my creative cargo misuse. :-)

[alexcrichton]

@emk that looks vaguely along the lines of #797, which should be fixed with a git add . to ensure that the files are in the git index (which is itself a bug). This use case is definitely quite reasonable, though, and should definitely be fixed!

[emk]

Hmm, it's fully reproducible after git add (after a rustup, of course). My guess is that it's confused by the fact that rust-uchardet/uchardet-sys/target/package/uchardet-sys-0.0.1/Cargo.toml isn't in git, if it's related to #797 at all.

[alexcrichton]

@emk I just landed a whole mess of bug fixes onto master, would you mind updating and trying again (the updates haven't hit nightlies yet sadly).

Also, do you have a set of commands to reproduce the bug? It'd help me diagnose it over on my end.

[emk]

I'm running off nightlies, so I'll add this to my to-do list for tomorrow.

Anyway, I can't provide you with a step-by step repro this morning, but here's how to set one up:

1. cargo new foo; cd foo; cargo new foo-ext; cp .gitignore foo-ext/; git add .; git commit -m "Initial"
2. Set up foo/foo-ext/Cargo.toml as described below.
3. Try to package foo-ext.
4. Try to package foo.

Here's the foo/foo-ext/Cargo.toml:

[package]

name = "foo-ext"
version = "0.0.1"
authors = ["Example <git@example.com>"]

[dependencies.foo]
path = ".."

This will fail with various errors, though the details vary from day-to-day. I've tested it against the most recent nightlies.

Sorry I can't do more until tomorrow. :-( But thank you, and please let me know if you have any questions.

[alexcrichton]

I'm unable to reproduce with master, but I'll wait to close this until nightlies are available and you can confirm as well.

[emk]

OK, this is really weird. It's still easily reproducible for me after rustup. Here's how:

git clone https://github.com/emk/rust-uchardet.git
cd rust-uchardet/uchardet-encoding
cargo package
cd ..
cargo package

This gives some warnings spew from my work-in-progress, and then just fails:

Could not find `Cargo.toml` in `/home/emk/scratch/rust-uchardet/uchardet-encoding/target/package`

If the answer to this is "Please don't try that; packages shouldn't contain their own add-on packages", I'll understand. :-)

[alexcrichton]

What version of cargo are you using? I tried that sequence and it worked ok for me:

$ rustc -v
rustc 0.13.0-nightly (eedfc0779 2014-11-25 22:36:59 +0000)
$ cargo -V
cargo 0.0.1-pre-nightly (e1dc4bc 2014-11-25 21:59:15 +0000)

[emk]

Hmm. This is what I had after running rustup.sh this morning:

$ rustc -v
rustc 0.13.0-nightly (395901393 2014-11-24 00:46:30 +0000)
$ cargo -V
cargo 0.0.1-pre-nightly (1bdfb25 2014-11-20 19:29:22 +0000)

I don't know why that was stale—I've been running rustup every day, pretty
much.

Re-running rustup now gets me a newer build:

$ cargo -V
cargo 0.0.1-pre-nightly (e1dc4bc 2014-11-25 21:59:15 +0000)

And this one works! Thank you for your help tracking this down, and my
apology for giving you the run-around with a stale cargo binary.

Le Wed Nov 26 2014 at 12:37:33, Alex Crichton notifications@github.com a
écrit :

What version of cargo are you using? I tried that sequence and it worked
ok for me:

$ rustc -v
rustc 0.13.0-nightly (eedfc0779 2014-11-25 22:36:59 +0000)
$ cargo -V
cargo 0.0.1-pre-nightly (e1dc4bc 2014-11-25 21:59:15 +0000)

—
Reply to this email directly or view it on GitHub
#937 (comment).

[alexcrichton]

No worries at all! Glad to hear that it's working :)

Closing for now, but please feel free to reopen if you run into any more issues!