send duplicate shred proofs for conflicting shred scenarios #32965

AshwinSekar · 2023-08-23T23:18:17Z

These are multiple last_shred_in_slot shreds and
coding shreds with conflicting erasure metas.

Problem

These scenarios are detected and stored in blockstore, but never sent to gossip or the state machine.

Summary of Changes

Fix the plumbing so these are sent to gossip and the state machine.

Split from #32862

codecov · 2023-08-24T03:14:01Z

Codecov Report

Merging #32965 (3812210) into master (79c02c8) will increase coverage by 0.0%.
Report is 8 commits behind head on master.
The diff coverage is 89.9%.

❗ Current head 3812210 differs from pull request most recent head 5907204. Consider uploading reports for the commit 5907204 to get more accurate results

@@           Coverage Diff            @@
##           master   #32965    +/-   ##
========================================
  Coverage    81.9%    82.0%            
========================================
  Files         785      784     -1     
  Lines      213295   212931   -364     
========================================
- Hits       174794   174704    -90     
+ Misses      38501    38227   -274

gossip/src/duplicate_shred.rs

gossip/src/duplicate_shred_handler.rs

ledger/src/blockstore.rs

bw-solana

LGTM

AshwinSekar · 2023-08-25T18:31:47Z

Will wait for behzad before merging.

behzadnouri · 2023-08-28T14:54:32Z

gossip/src/duplicate_shred.rs

+    if shred1.index() != shred2.index() && shred1.shred_type() == ShredType::Code {
+        if shred1.fec_set_index() != shred2.fec_set_index() {
+            return Err(Error::InvalidErasureMetaConflict);
+        }
+        let erasure_meta =
+            ErasureMeta::from_coding_shred(shred1).expect("Shred1 should be a coding shred");
+        if erasure_meta.check_coding_shred(shred2) {
+            return Err(Error::InvalidErasureMetaConflict);


This is not technically exhaustive.
Any 2 shreds with different but overlapping erasure sets can be considered duplicate.
In particular doesn't even need to have the same fec_set_index.

Technically the overlapping erasure sets might result in non-duplicate shreds, but a leader shouldn't generate overlapping erasure sets unless malicious or buggy.

We don't currently identify those cases because we just lookup by the erasure set:
https://github.com/solana-labs/solana/blob/5de27d9a0/ledger/src/blockstore.rs#L1197-L1206
but I think that has to be fixed at some point.

Good to know, right now i've only replicated the cases that we catch in blockstore.
I created #33037, and will extend the checks to catch these types of conflicts in a follow up pr.

Can you put a comment reference to #33037 here?

behzadnouri · 2023-08-28T15:11:49Z

gossip/src/duplicate_shred.rs

+    // Shred index of the first shred in the proof
+    pub(crate) shred_index: u32,


Is there a use to this any more?
Can we just change this to _unused?

behzadnouri · 2023-08-28T15:12:21Z

gossip/src/duplicate_shred.rs

+    #[error("invalid last index conflict")]
+    InvalidLastIndexConflict,
+    #[error("invalid erasure meta conflict")]
+    InvalidErasureMetaConflict,


Can you keep this enum sorted?

behzadnouri · 2023-08-28T15:13:25Z

gossip/src/duplicate_shred.rs

+            (ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
+            (ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),


What if both are last_in_slot but ix1 != ix2?

It will get caught by the _ => () case as this is a valid last index conflict proof.

This part (and now the function as a whole) has become too confusing.
We want to check if the shreds indeed indicate a duplicate slot.
So we need to check that the shreds have the same slot and type and then either of these conditions hold:

same index, different payload.

one has last_in_slot but the other has higher index.

the erasure configs are conflicting.

It is more readable to check if these conditions hold than check if one of them does not hold.

behzadnouri · 2023-08-29T13:38:10Z

gossip/src/duplicate_shred.rs

+
+    #[allow(dead_code)]
    shred_index: u32,
+


Just rename this to _unused.
You will need to update frozen-abi digest in a couple of places but that is fine.
Also please don't add unnecessary blank lines:
https://google.github.io/styleguide/cppguide.html#Vertical_Whitespace

behzadnouri · 2023-08-29T13:42:29Z

gossip/src/duplicate_shred.rs

+        return Err(Error::ShredTypeMismatch);
+    }
+
+    if shred1.index() == shred2.index() && shred1.payload() == shred2.payload() {


If shred1.index() != shred2.index() then the payload cannot be the same.
The index is encoded in the payload.
So the first check is unnecessary.

behzadnouri · 2023-08-29T13:44:40Z

gossip/src/duplicate_shred.rs

+    if shred1.index() != shred2.index() && shred1.shred_type() == ShredType::Code {
+        if shred1.fec_set_index() != shred2.fec_set_index() {
+            return Err(Error::InvalidErasureMetaConflict);
+        }
+        let erasure_meta =
+            ErasureMeta::from_coding_shred(shred1).expect("Shred1 should be a coding shred");
+        if erasure_meta.check_coding_shred(shred2) {
+            return Err(Error::InvalidErasureMetaConflict);


Can you put a comment reference to #33037 here?

behzadnouri · 2023-08-29T13:55:19Z

gossip/src/duplicate_shred.rs

+            (ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
+            (ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),


This part (and now the function as a whole) has become too confusing.
We want to check if the shreds indeed indicate a duplicate slot.
So we need to check that the shreds have the same slot and type and then either of these conditions hold:

same index, different payload.

one has last_in_slot but the other has higher index.

the erasure configs are conflicting.

It is more readable to check if these conditions hold than check if one of them does not hold.

behzadnouri · 2023-08-29T14:02:22Z

ledger/src/blockstore.rs

        ));
        assert!(blockstore.has_duplicate_shreds_in_slot(0));
+        assert_eq!(duplicate_shreds.len(), 1);
+        matches!(


assert_matches

matches by itself does not do any assertions here.

behzadnouri · 2023-08-29T14:02:54Z

ledger/src/blockstore.rs

        ));
+
+        assert_eq!(duplicate_shreds.len(), 1);
+        matches!(


behzadnouri · 2023-08-29T14:06:35Z

ledger/src/blockstore_meta.rs

@@ -344,7 +344,7 @@ impl ErasureMeta {

    // Returns true if the erasure fields on the shred
    // are consistent with the erasure-meta.
-    pub(crate) fn check_coding_shred(&self, shred: &Shred) -> bool {
+    pub fn check_coding_shred(&self, shred: &Shred) -> bool {


To minimize the public api surface, instead of making these 2 functions public, can you add a method which takes two shreds and returns false if their erasure configs mismatch.
That function can call these 2 functions here internally.

behzadnouri · 2023-08-29T14:10:57Z

core/src/window_service.rs

+                if !blockstore.has_duplicate_shreds_in_slot(shred_slot) {
+                    if let Some(existing_shred_payload) = blockstore.is_shred_duplicate(&shred) {
+                        blockstore.store_duplicate_slot(
+                            shred_slot,
+                            existing_shred_payload.clone(),
+                            shred.clone().into_payload(),
+                        )?;
+                        (shred, existing_shred_payload)
+                    } else {
+                        // Shred is not duplicate
+                        return Ok(());
+                    }
+                } else {
+                    // Shred has already been handled
+                    return Ok(());
+                }


This would be more readable if you avoid nested branches:

if blockstore.has_duplicate_shreds_in_slot(shred_slot) { return Ok(()); // A duplicate is already recorded. } let Some(existing_shred_payload) = blockstore.is_shred_duplicate(&shred) else { return Ok(()); // Not a duplicate. } blockstore.store_duplicate_slot( shred_slot, existing_shred_payload.clone(), shred.clone().into_payload(), )?; (shred, existing_shred_payload)

The scenarios are multiple last_shred_in_slot shreds and coding shreds with conflicting erasure metas.

behzadnouri

lgtm, aside from that extraneous #[allow(dead_code)].

behzadnouri · 2023-08-31T12:36:25Z

gossip/src/duplicate_shred.rs

@@ -29,7 +29,8 @@ pub struct DuplicateShred {
    pub(crate) from: Pubkey,
    pub(crate) wallclock: u64,
    pub(crate) slot: Slot,
-    shred_index: u32,
+    #[allow(dead_code)]


When you start the field name with underscore, don't need #[allow(dead_code)] anymore.

thanks for the review! I've removed the #[allow(dead_code)]

mergify · 2023-08-31T18:22:53Z

automerge label removed due to a CI failure

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch from 5c90b2c to 3812210 Compare August 24, 2023 00:33

AshwinSekar requested review from carllin and bw-solana August 24, 2023 04:01

bw-solana reviewed Aug 25, 2023

View reviewed changes

gossip/src/duplicate_shred.rs Show resolved Hide resolved

bw-solana reviewed Aug 25, 2023

View reviewed changes

gossip/src/duplicate_shred_handler.rs Outdated Show resolved Hide resolved

bw-solana reviewed Aug 25, 2023

View reviewed changes

ledger/src/blockstore.rs Show resolved Hide resolved

AshwinSekar requested a review from behzadnouri August 25, 2023 18:22

bw-solana previously approved these changes Aug 25, 2023

View reviewed changes

behzadnouri reviewed Aug 28, 2023

View reviewed changes

AshwinSekar dismissed bw-solana’s stale review via cfa2d69 August 28, 2023 21:18

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch 2 times, most recently from cfa2d69 to f45864c Compare August 28, 2023 22:05

AshwinSekar requested a review from behzadnouri August 28, 2023 23:52

behzadnouri reviewed Aug 29, 2023

View reviewed changes

AshwinSekar added 3 commits August 30, 2023 18:20

send duplicate shred proofs for conflicting shred scenarios

1192973

The scenarios are multiple last_shred_in_slot shreds and coding shreds with conflicting erasure metas.

Pr feedback: deprecate shred_index for gossip duplicate shred proofs

4059787

Pr feedback

8ee8631

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch from f45864c to 8ee8631 Compare August 30, 2023 18:54

AshwinSekar requested a review from behzadnouri August 30, 2023 20:04

behzadnouri previously approved these changes Aug 31, 2023

View reviewed changes

Remove extraneous dead code

5907204

AshwinSekar dismissed behzadnouri’s stale review via 5907204 August 31, 2023 17:46

behzadnouri approved these changes Aug 31, 2023

View reviewed changes

AshwinSekar added the automerge Merge this Pull Request automatically once CI passes label Aug 31, 2023

mergify bot removed the automerge Merge this Pull Request automatically once CI passes label Aug 31, 2023

AshwinSekar merged commit 6d2dae6 into solana-labs:master Aug 31, 2023

AshwinSekar self-assigned this Nov 29, 2023

AshwinSekar mentioned this pull request Dec 7, 2023

Add feature flag for LastIndex and Erasure duplicate proofs #34360

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

send duplicate shred proofs for conflicting shred scenarios #32965

send duplicate shred proofs for conflicting shred scenarios #32965

AshwinSekar commented Aug 23, 2023 •

edited

Loading

codecov bot commented Aug 24, 2023 •

edited

Loading

bw-solana left a comment

AshwinSekar commented Aug 25, 2023

behzadnouri Aug 28, 2023

AshwinSekar Aug 28, 2023

behzadnouri Aug 29, 2023

behzadnouri Aug 28, 2023

behzadnouri Aug 28, 2023

behzadnouri Aug 28, 2023

AshwinSekar Aug 28, 2023

behzadnouri Aug 29, 2023 •

edited

Loading

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023 •

edited

Loading

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023

behzadnouri Aug 29, 2023

behzadnouri left a comment

behzadnouri Aug 31, 2023

AshwinSekar Aug 31, 2023

mergify bot commented Aug 31, 2023

		// Shred index of the first shred in the proof
		pub(crate) shred_index: u32,

		(ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
		(ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),

send duplicate shred proofs for conflicting shred scenarios #32965

send duplicate shred proofs for conflicting shred scenarios #32965

Conversation

AshwinSekar commented Aug 23, 2023 • edited Loading

Problem

Summary of Changes

codecov bot commented Aug 24, 2023 • edited Loading

Codecov Report

bw-solana left a comment

Choose a reason for hiding this comment

AshwinSekar commented Aug 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

behzadnouri Aug 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

behzadnouri Aug 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

behzadnouri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Aug 31, 2023

AshwinSekar commented Aug 23, 2023 •

edited

Loading

codecov bot commented Aug 24, 2023 •

edited

Loading

behzadnouri Aug 29, 2023 •

edited

Loading

behzadnouri Aug 29, 2023 •

edited

Loading