-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove DocBuilder, part 1 #1016
Conversation
Those options aren't used at all anywhere in the codebase, so it's safe to remove them.
As far as I'm aware this flag was never used since the Rust project started to take care of docs.rs, and it's not much useful. Basically, every time a build happened the DocBuilder recorded the crate name and version in a text file, and the flag prevented a build if the related names and versions were present in such file. Checking if the build is successful in the database is more reliable.
@@ -48,21 +48,13 @@ pub fn queue_builder( | |||
|
|||
if status.count() >= 10 { | |||
// periodically, we need to flush our caches and ping the hubs | |||
debug!("10 builds in a row; flushing caches"); | |||
debug!("10 builds in a row; pinging pubsubhubhub"); | |||
status = BuilderState::QueueInProgress(0); | |||
|
|||
match pubsubhubbub::ping_hubs() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't know we had this 😆 do we know anyone using this? I'd rather have #809 I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dunno why it exists, but let's not remove it for now until we get more insights on who's using it.
"SELECT 1 FROM crates, releases, builds | ||
WHERE crates.id = releases.crate_id AND releases.id = builds.rid |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs an INNER JOIN
or you'll get duplicates (the id won't match the crate that succeeded).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm? This is working as expected when testing manually. Why would you expect it to fail?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh, I expected it to fail when you have three different crates, each of which satisfies one of these conditions.
CREATE TEMPORARY TABLE crates (
id SERIAL PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL
);
CREATE TEMPORARY TABLE releases (
id SERIAL PRIMARY KEY,
crate_id INT NOT NULL REFERENCES crates(id),
version VARCHAR(100)
);
CREATE TEMPORARY TABLE builds (
id SERIAL,
rid INT NOT NULL REFERENCES releases(id),
build_status BOOL NOT NULL
);
insert into crates (name) values ('a'), ('b');
insert into releases (crate_id, version) values (1, '0.1.0'), (2, '0.1.0');
insert into builds (rid, build_status) values (1, true), (2, true);
SELECT 1 FROM crates, releases, builds
WHERE crates.id = releases.crate_id AND releases.id = builds.rid
AND crates.name = 'a' AND releases.version = '0.1.0' AND builds.build_status = TRUE;
But that doesn't happen - the WHERE id = crate_id AND id = rid
filters out the duplicates. It's more clear when you select everything instead of 1
:
select * from crates, releases, builds WHERE crates.id = releases.crate_id AND releases.id = builds.rid;
id | name | id | crate_id | version | id | rid | build_status
----+------+----+----------+---------+----+-----+--------------
1 | a | 1 | 1 | 0.1.0 | 1 | 1 | f
2 | b | 2 | 2 | 0.1.0 | 2 | 2 | t
joshua=> select * from crates, releases, builds WHERE crates.name = 'a' AND releases.version = '0.1.0' AND builds.build_status = TRUE;
id | name | id | crate_id | version | id | rid | build_status
----+------+----+----------+---------+----+-----+--------------
1 | a | 1 | 1 | 0.1.0 | 2 | 2 | t
1 | a | 2 | 2 | 0.1.0 | 2 | 2 | t
joshua=> select * from crates, releases, builds WHERE crates.name = 'a' AND releases.version = '0.1.0' AND builds.build_status = TRUE AND crates.id = releases.crate_id AND releases.id = builds.rid;
id | name | id | crate_id | version | id | rid | build_status
----+------+----+----------+---------+----+-----+--------------
(0 rows)
So this works, it's just confusing.
Co-authored-by: Joshua Nelson <joshua@yottadb.com>
This PR is the first of the two PRs I plan to send to remove
DocBuilder
andDocBuilderOptions
from the docs.rs codebase: both those structs serve as "intermediaries" betweenRustwideBuilder
and the rest of the application, but now that we haveConfig
andContext
that's not really needed anymore. Not having the middle layer will help with building buildtests a lot.Notable changes:
cratesfyi build --skip-if-log-exists <command>
was removed: the--skip-if-log-exists
flag was really fragile, relying on a text log of all the builds updated every 10 builds. There is also no need I can see for it, as the practically equivalent (and better)--skip-if-exists
already exists.cratesfyi build --skip-if-exists <command>
was improved: now it checks if a successful build is present, instead of checking whether the release metadata is stored in the database.RustwideBuilder
configuration was moved over toConfig
.DocBuilderOptions
was moved over toConfig
.Index
is now stored inContext
and a single instance is reused across the whole application. This won't affect the file descriptors problem, as the underlying git instance is still re-created every time it is accessed.In the next PR I'll move away all the rest of
DocBuilder
. This PR is best reviewed commit-by-commit.