Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Prover CLI): status batch --verbose #1899

Merged
merged 126 commits into from
May 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
17e564c
fmt
ColoCarletti Apr 8, 2024
245b0a4
wip
ColoCarletti Apr 8, 2024
8e84243
add cargo.toml cargo.lock
ColoCarletti Apr 8, 2024
17517b1
go back to anihow
ColoCarletti Apr 8, 2024
4ab7ce8
fmt
ColoCarletti Apr 8, 2024
4655201
rm tool workspace
ColoCarletti Apr 9, 2024
c32d1aa
fmt
ColoCarletti Apr 10, 2024
8e49e04
zk fmt
ColoCarletti Apr 10, 2024
80e13da
fix typo
ColoCarletti Apr 10, 2024
0f2ebd8
move query to prover_dal
ColoCarletti Apr 12, 2024
69d2104
remove --all flag
Apr 15, 2024
c9e6e5e
change command name
ColoCarletti Apr 16, 2024
83fab40
merge main
ColoCarletti Apr 16, 2024
cdbb467
fmt
ColoCarletti Apr 16, 2024
d1f3994
add sqlx query json
ColoCarletti Apr 16, 2024
4ee3ff2
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 16, 2024
fff3910
update README
ColoCarletti Apr 16, 2024
ef906b9
zk fmt
ColoCarletti Apr 16, 2024
3758547
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
ColoCarletti Apr 16, 2024
b3051b2
remove eol
ColoCarletti Apr 16, 2024
ee9d35b
Move jobs cmd to status cmd module
Apr 16, 2024
82faf32
Refactor status cmd
Apr 16, 2024
aa44864
Update Cargo.lock
Apr 16, 2024
3fe32b6
zk fmt
Apr 16, 2024
d43dd15
Merge pull request #1705 from matter-labs/il-refactor-prover-cli-stat…
ilitteri Apr 17, 2024
70b0348
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 17, 2024
585059b
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 17, 2024
9977e95
Refactor
Apr 18, 2024
ee88cfe
Merge branch 'main' into prover_cli_status_command
ilitteri Apr 18, 2024
075e6dc
Merge branch 'main' into prover_cli_status_command
ColoCarletti Apr 19, 2024
0120a09
fix typo
ColoCarletti Apr 19, 2024
9a11a94
modify query to include aggregation_round
ColoCarletti Apr 19, 2024
1c92e95
change query return type
ColoCarletti Apr 24, 2024
05e1636
Update Cargo files
Apr 24, 2024
58d47ad
Setle status cmd baseline
Apr 24, 2024
f778759
Add status cmd utils
Apr 24, 2024
e945df2
Add status utils
Apr 24, 2024
6585378
zk fmt
Apr 24, 2024
47df898
Document enums
Apr 24, 2024
ceae289
Add BatchDataBuilder struct
Apr 24, 2024
f6d7b9a
Add TaskStatus::Custom enum variant
Apr 24, 2024
8e16d35
Fix BatchDataBuilder
Apr 24, 2024
a76cb32
Rollback
Apr 24, 2024
c460029
impl From<ProofCompressionJobStatus> for TaskStatus
Apr 24, 2024
e76da1f
Fix TaskStatus::Custom fmt
Apr 24, 2024
4be214e
Add query for getting proof compression job info for a batch
Apr 25, 2024
e9e197f
Add query for getting proof compression job info for a batch
Apr 25, 2024
7c9a499
Handle proof compression job status for batches
Apr 25, 2024
17b5875
Fix get_proof_compression_job_status_for_batch
Apr 25, 2024
3fdc055
Remove BatchDataBuilder struct
Apr 25, 2024
a2ad3f9
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
Apr 25, 2024
3093cde
Fix
Apr 25, 2024
fde3be1
Move prover_dal types to basic_types::prover_dal module
ilitteri Apr 25, 2024
efd70ac
Refactor query
ilitteri Apr 25, 2024
6ec0ac3
Merge pull request #1789 from matter-labs/prover_cli_compressor_status
ilitteri Apr 25, 2024
1830ced
add bwg query
ColoCarletti Apr 25, 2024
a6105b8
merge compresor job
ColoCarletti Apr 25, 2024
bbe5dc3
add status for prover jobs
ColoCarletti Apr 26, 2024
111884b
add title for prover jobs
ColoCarletti Apr 26, 2024
9fef23d
Refactor
ilitteri Apr 26, 2024
01ecd2f
add leaf query
ColoCarletti Apr 29, 2024
25e089b
Merge branch 'prover_cli_status_command' into prover_cli_status_comma…
ColoCarletti Apr 29, 2024
590c58b
fix merge
ColoCarletti Apr 29, 2024
e08b05e
add querys
ColoCarletti Apr 29, 2024
cf069b9
fix witnes jobs status
ColoCarletti Apr 30, 2024
6ad5566
rm comments
ColoCarletti Apr 30, 2024
a3f1938
Fix & refactors
ilitteri Apr 30, 2024
78c7cd8
zk lint rust
ilitteri Apr 30, 2024
33729b8
Fix import
ilitteri Apr 30, 2024
1efde33
Merge branch 'prover_cli_status_command' of github.com:matter-labs/zk…
ilitteri Apr 30, 2024
112ae93
fix import
ColoCarletti Apr 30, 2024
e1854fa
zk lint rust
ilitteri Apr 30, 2024
ccab3d3
Merge branch 'prover_cli_status_command_lwg' of github.com:matter-lab…
ColoCarletti Apr 30, 2024
fa26c26
fix wg task types
ColoCarletti Apr 30, 2024
dbaf3b4
Merge pull request #1830 from matter-labs/prover_cli_status_command_lwg
ColoCarletti Apr 30, 2024
4124f63
Merge main
ColoCarletti Apr 30, 2024
a4fde1f
update README
ColoCarletti Apr 30, 2024
10fdcb9
fix EIP Blob use
ColoCarletti Apr 30, 2024
8a1101f
fix typos
ColoCarletti Apr 30, 2024
c1282a5
Merge branch 'main' into prover_cli_status_command
ilitteri May 2, 2024
1bbde93
Fix typo
ilitteri May 2, 2024
16be778
Merge branch 'main' into prover_cli_status_command
ilitteri May 2, 2024
24a0247
refactor no verbose
ColoCarletti May 2, 2024
57beedb
add verbose flag
ColoCarletti May 7, 2024
5cfbdb0
fix prints
ColoCarletti May 7, 2024
a650383
fix status bug
ColoCarletti May 8, 2024
e0e484f
print prover_jobs global status
ColoCarletti May 8, 2024
8afa497
change return for wg querys
ColoCarletti May 8, 2024
2ff0d00
fix verbose flag
ColoCarletti May 8, 2024
c8c3511
fix lines indentation
ColoCarletti May 8, 2024
2ffb7f7
fix spacing
ColoCarletti May 8, 2024
4eaeccc
order prover_jobs by circuit_id
ColoCarletti May 8, 2024
97ff273
align digits
ColoCarletti May 8, 2024
fd68fd1
change status emojis
ColoCarletti May 8, 2024
1ec9877
improve format
ColoCarletti May 8, 2024
10afa17
capitalize status print
ColoCarletti May 8, 2024
3142ce9
merge main
ColoCarletti May 9, 2024
d6ebe02
fix merge errors
ColoCarletti May 9, 2024
20fd720
fix prover jobs display order
ColoCarletti May 9, 2024
ea03118
map hashmap to vector
ColoCarletti May 9, 2024
5b5d7ac
Print refactor
ilitteri May 9, 2024
5ef5eea
add recursion tip query
ColoCarletti May 9, 2024
67406c1
add recursion tip status
ColoCarletti May 9, 2024
ed535ec
merge
ColoCarletti May 9, 2024
c3d61e0
force (circuit_id - 2) for node wg
ColoCarletti May 10, 2024
4c45845
add function to correct circuit_id in node wg
ColoCarletti May 10, 2024
cb41861
rm unused imports
ColoCarletti May 10, 2024
182948a
merge main
ColoCarletti May 10, 2024
0bb3d8a
fix spellcheck
ColoCarletti May 10, 2024
b48e2fc
add status for gpu prover
ColoCarletti May 10, 2024
e82ec1f
fix status match
ColoCarletti May 10, 2024
9dcf8e1
Remove use of ensure
ilitteri May 10, 2024
4eb178a
Minor refactors
ilitteri May 10, 2024
2257f6a
fix no verbose status
ColoCarletti May 10, 2024
d76469d
Merge branch 'prover_cli_status_verbose' of github.com:matter-labs/zk…
ColoCarletti May 10, 2024
fc09ccf
Merge branch 'main' into prover_cli_status_verbose
ilitteri May 13, 2024
275dc9b
Merge branch 'main' into prover_cli_status_verbose
ilitteri May 13, 2024
c36713a
Merge branch 'main' into prover_cli_status_verbose
ilitteri May 13, 2024
bdba814
refactor
ColoCarletti May 14, 2024
1572f97
Merge branch 'main' into prover_cli_status_verbose
ColoCarletti May 14, 2024
80c4e74
fix small print error
ColoCarletti May 14, 2024
67beeb9
zk fmt
ColoCarletti May 14, 2024
a1bd89e
fix imports
ColoCarletti May 14, 2024
4194e29
fix typo
ColoCarletti May 14, 2024
f5b34a3
Merge branch 'main' into prover_cli_status_verbose
ColoCarletti May 14, 2024
6d83e78
Merge branch 'main' into prover_cli_status_verbose
ilitteri May 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions core/lib/basic_types/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ serde_json.workspace = true
chrono.workspace = true
strum = { workspace = true, features = ["derive"] }
num_enum.workspace = true
sqlx = { workspace = true, feature= ["derive"]}
anyhow.workspace = true
url = { workspace = true, features = ["serde"] }

Expand Down
43 changes: 38 additions & 5 deletions core/lib/basic_types/src/prover_dal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,13 @@ pub struct JobPosition {
pub sequence_number: usize,
}

#[derive(Debug, Default, PartialEq)]
#[derive(Debug, Default, PartialEq, Clone)]
pub struct ProverJobStatusFailed {
pub started_at: DateTime<Utc>,
pub error: String,
}

#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Clone)]
pub struct ProverJobStatusSuccessful {
pub started_at: DateTime<Utc>,
pub time_taken: Duration,
Expand All @@ -120,7 +120,7 @@ impl Default for ProverJobStatusSuccessful {
}
}

#[derive(Debug, Default, PartialEq)]
#[derive(Debug, Default, PartialEq, Clone)]
pub struct ProverJobStatusInProgress {
pub started_at: DateTime<Utc>,
}
Expand All @@ -146,7 +146,7 @@ pub struct WitnessJobStatusFailed {
pub error: String,
}

#[derive(Debug, strum::Display, strum::EnumString, strum::AsRefStr, PartialEq)]
#[derive(Debug, strum::Display, strum::EnumString, strum::AsRefStr, PartialEq, Clone)]
pub enum ProverJobStatus {
#[strum(serialize = "queued")]
Queued,
Expand Down Expand Up @@ -238,6 +238,7 @@ impl FromStr for GpuProverInstanceStatus {
}
}

#[derive(Debug, Clone)]
pub struct ProverJobFriInfo {
pub id: u32,
pub l1_batch_number: L1BatchNumber,
Expand All @@ -260,6 +261,7 @@ pub struct ProverJobFriInfo {
pub picked_by: Option<String>,
}

#[derive(Debug, Clone)]
pub struct BasicWitnessGeneratorJobInfo {
pub l1_batch_number: L1BatchNumber,
pub merkle_tree_paths_blob_url: Option<String>,
Expand All @@ -276,6 +278,7 @@ pub struct BasicWitnessGeneratorJobInfo {
pub eip_4844_blobs: Option<Eip4844Blobs>,
}

#[derive(Debug, Clone)]
pub struct LeafWitnessGeneratorJobInfo {
pub id: u32,
pub l1_batch_number: L1BatchNumber,
Expand All @@ -294,6 +297,7 @@ pub struct LeafWitnessGeneratorJobInfo {
pub picked_by: Option<String>,
}

#[derive(Debug, Clone)]
pub struct NodeWitnessGeneratorJobInfo {
pub id: u32,
pub l1_batch_number: L1BatchNumber,
Expand All @@ -312,6 +316,22 @@ pub struct NodeWitnessGeneratorJobInfo {
pub picked_by: Option<String>,
}

#[derive(Debug, Clone)]
pub struct RecursionTipWitnessGeneratorJobInfo {
pub l1_batch_number: L1BatchNumber,
pub status: WitnessJobStatus,
pub attempts: u32,
pub processing_started_at: Option<NaiveDateTime>,
pub time_taken: Option<NaiveTime>,
pub error: Option<String>,
pub created_at: NaiveDateTime,
pub updated_at: NaiveDateTime,
pub number_of_final_node_jobs: Option<i32>,
pub protocol_version: Option<i32>,
pub picked_by: Option<String>,
}

#[derive(Debug, Clone)]
pub struct SchedulerWitnessGeneratorJobInfo {
pub l1_batch_number: L1BatchNumber,
pub scheduler_partial_input_blob_url: String,
Expand All @@ -326,7 +346,7 @@ pub struct SchedulerWitnessGeneratorJobInfo {
pub picked_by: Option<String>,
}

#[derive(Debug, EnumString, Display)]
#[derive(Debug, EnumString, Display, Clone)]
pub enum ProofCompressionJobStatus {
#[strum(serialize = "queued")]
Queued,
Expand All @@ -342,6 +362,7 @@ pub enum ProofCompressionJobStatus {
Skipped,
}

#[derive(Debug, Clone)]
pub struct ProofCompressionJobInfo {
pub l1_batch_number: L1BatchNumber,
pub attempts: u32,
Expand All @@ -355,3 +376,15 @@ pub struct ProofCompressionJobInfo {
pub time_taken: Option<NaiveTime>,
pub picked_by: Option<String>,
}

// This function corrects circuit IDs for the node witness generator.
//
// - Circuit IDs in the node witness generator are 2 higher than in other rounds.
// - The `EIP4844Repack` circuit (ID 255) is an exception and is set to 18.
pub fn correct_circuit_id(circuit_id: i16, aggregation_round: AggregationRound) -> u32 {
match (circuit_id, aggregation_round) {
(18, AggregationRound::NodeAggregation) => 255,
(circuit_id, AggregationRound::NodeAggregation) => (circuit_id as u32) - 2,
_ => circuit_id as u32,
}
}
2 changes: 2 additions & 0 deletions prover/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions prover/prover_cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,4 @@ zksync_dal.workspace = true
strum.workspace = true
colored.workspace = true
sqlx.workspace = true
circuit_definitions.workspace = true
Loading
Loading