Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rpc): implement Filecoin.EthGetMessageCidByTransactionHash #4402

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
70e0019
Add support for Eth mappings
elmattic Jun 7, 2024
63711e1
Appease clippy
elmattic Jun 7, 2024
0907b35
Merge branch 'main' into elmattic/eth-mappings
elmattic Jun 7, 2024
92cf525
Move info trace
elmattic Jun 7, 2024
a99c93d
Implement EthGetMessageCidByTransactionHash method
elmattic Jun 7, 2024
f38bbdc
Add tests
elmattic Jun 7, 2024
ab84529
Handle error
elmattic Jun 7, 2024
fd70514
Update CHANGELOG
elmattic Jun 7, 2024
d36243b
Fix db dev mode tests
elmattic Jun 10, 2024
adf4158
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 10, 2024
633a372
Use Hash column indexes for eth mappings
elmattic Jun 10, 2024
161d914
Demote info logs to debug
elmattic Jun 10, 2024
07f093d
Move back to Btree column indexes
elmattic Jun 10, 2024
c6feb43
Merge branch 'main' into elmattic/eth-mappings
elmattic Jun 10, 2024
0566486
Add doc comments
elmattic Jun 10, 2024
64ed199
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 10, 2024
6f60fed
Fix doc build
elmattic Jun 10, 2024
59c7667
Use Hash column indexes for eth mappings
elmattic Jun 12, 2024
65dfcd7
Merge branch 'main' into elmattic/eth-mappings
elmattic Jun 12, 2024
81cf85a
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 12, 2024
af71e97
Merge branch 'main' into elmattic/eth-mappings
elmattic Jun 13, 2024
ffc96b6
Add timestamp when putting mapping
elmattic Jun 13, 2024
3eda7aa
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 13, 2024
7cffeb8
Fix method to discard the timestamp
elmattic Jun 13, 2024
5851b00
Appease clippy
elmattic Jun 13, 2024
541db9f
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 13, 2024
32a49e5
Add task to retrieve eth txs older than a given duration
elmattic Jun 13, 2024
8c50d3a
Change method name
elmattic Jun 13, 2024
0c76277
Add deletion of Hash to Cid entries
elmattic Jun 14, 2024
f6a0b28
Merge branch 'main' into elmattic/eth-mappings
elmattic Jun 17, 2024
12fdc79
Merge branch 'elmattic/eth-mappings' into elmattic/elmattic/eth-get-m…
elmattic Jun 17, 2024
b372c0a
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 18, 2024
6131239
Remove todos
elmattic Jun 18, 2024
3b4a9e4
Remove newline
elmattic Jun 18, 2024
2360e4a
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 18, 2024
fe714e8
Refactor
elmattic Jun 18, 2024
9ce6bfe
Appease clippy
elmattic Jun 18, 2024
876ef45
Add ttl to client Config
elmattic Jun 19, 2024
4c8ddcf
Write eth tx hashes when persisting tipsets
elmattic Jun 19, 2024
de39542
Refactor
elmattic Jun 19, 2024
2f53517
Fix doc and lint errors
elmattic Jun 19, 2024
869afe6
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 19, 2024
7f1ea25
Fix persisting eth entries
elmattic Jun 20, 2024
a0abbef
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 20, 2024
39aee92
Add tipset key persistence
elmattic Jun 20, 2024
ffc4800
Add test script
elmattic Jun 20, 2024
a9f7386
Add job
elmattic Jun 20, 2024
c7e64d0
Fix shellcheck errors
elmattic Jun 20, 2024
dfe93e4
Rename job
elmattic Jun 20, 2024
18ea5ad
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 20, 2024
5653afc
Try fix script
elmattic Jun 20, 2024
e6ceb3a
Add executable right
elmattic Jun 20, 2024
d9e0f3b
Fix script
elmattic Jun 20, 2024
82aeaa6
Test result and return error in case of missing data
elmattic Jun 21, 2024
651356a
Increase number of tipsets
elmattic Jun 21, 2024
4fa644b
Fix script to check Hash to TipsetKey
elmattic Jun 21, 2024
9a9aa5a
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 21, 2024
a79308c
Add message persistence
elmattic Jun 21, 2024
ea5bbff
Rename and move function to a method
elmattic Jun 21, 2024
d12419d
Add better logs
elmattic Jun 21, 2024
0e9b7d0
Fix tipset keys not always persisted
elmattic Jun 21, 2024
936962d
Remove method
elmattic Jun 21, 2024
6e54ed6
Fix tipset key persistence
elmattic Jun 24, 2024
1aab758
Fix tipset key persistence
elmattic Jun 24, 2024
98037b3
Remove useless code
elmattic Jun 24, 2024
cab25c2
Start to add testing of ttl
elmattic Jun 24, 2024
f5acb98
Use block timestamp when writing the mapping
elmattic Jun 24, 2024
cc1fd73
Fix bash script
elmattic Jun 24, 2024
c21670f
Add more slack
elmattic Jun 24, 2024
5b9546f
Add debug function to print timestamp
elmattic Jun 24, 2024
23068e0
Fix shellcheck errors
elmattic Jun 24, 2024
8b2537f
Appease shellcheck
elmattic Jun 24, 2024
3d1e78a
Fix mapping persistence
elmattic Jun 25, 2024
a06777f
Remove some comment and comment out echo
elmattic Jun 25, 2024
018e3be
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 25, 2024
286e7dd
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jun 25, 2024
7a33f6a
Change how test is done
elmattic Jun 25, 2024
9bf4d07
Add two minutes of slack
elmattic Jun 25, 2024
df76ac2
Add newlines
elmattic Jun 25, 2024
f71bee6
Simplify test script
elmattic Jun 25, 2024
dbd1f44
Remove lines
elmattic Jun 25, 2024
f73f6a5
Construct option later
elmattic Jun 25, 2024
2acb7a2
Remove testing code
elmattic Jun 26, 2024
00aa606
Refactor and add unit test
elmattic Jun 27, 2024
c4257a9
Add ttl collector and test
elmattic Jul 1, 2024
0f63b83
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jul 1, 2024
910c212
Refactor
elmattic Jul 1, 2024
b2527fe
Fix spellcheck errors
elmattic Jul 1, 2024
f5dc010
Fix spellcheck error
elmattic Jul 1, 2024
86d8fa5
Merge branch 'main' into elmattic/elmattic/eth-get-message-cid-by-tra…
elmattic Jul 2, 2024
eed5449
Remove comments
elmattic Jul 2, 2024
ef86ba8
Refactor bash script using jq
elmattic Jul 2, 2024
971ba17
Remove dead code
elmattic Jul 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/forest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,32 @@ jobs:
chmod +x ~/.cargo/bin/forest*
- run: ./scripts/tests/calibnet_kademlia_check.sh
timeout-minutes: '${{ fromJSON(env.SCRIPT_TIMEOUT_MINUTES) }}'
calibnet-eth-mapping-check:
needs:
- build-ubuntu
name: Calibnet eth mapping check
runs-on: ubuntu-latest
steps:
- run: lscpu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean lscpu?

- uses: actions/cache@v4
with:
path: '${{ env.FIL_PROOFS_PARAMETER_CACHE }}'
key: proof-params-keys
- name: Checkout Sources
uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: 'forest-${{ runner.os }}'
path: ~/.cargo/bin
- uses: actions/download-artifact@v4
with:
name: 'forest-${{ runner.os }}'
path: ~/.cargo/bin
- name: Set permissions
run: |
chmod +x ~/.cargo/bin/forest*
- run: ./scripts/tests/calibnet_eth_mapping_check.sh
timeout-minutes: '${{ fromJSON(env.SCRIPT_TIMEOUT_MINUTES) }}'
db-migration-checks:
needs:
- build-ubuntu
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@
- [#4381](https://github.com/ChainSafe/forest/pull/4381) Add support for the
`Filecoin.StateSectorPartition` RPC method.

- [#4368](https://github.com/ChainSafe/forest/issues/4368) Add support for the
`Filecoin.EthGetMessageCidByTransactionHash` RPC method.

- [#4167](https://github.com/ChainSafe/forest/issues/4167) Add support for the
`Filecoin.EthGetBlockByHash` RPC method.

Expand Down
73 changes: 73 additions & 0 deletions scripts/tests/calibnet_eth_mapping_check.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
# This script is checking the correctness of the ethereum mapping feature
# It requires both the `forest` and `forest-cli` binaries to be in the PATH.

set -eu

source "$(dirname "$0")/harness.sh"

forest_init

FOREST_URL='http://127.0.0.1:2345/rpc/v1'

NUM_TIPSETS=600

echo "Get Ethereum block hashes and transactions hashes from the last $NUM_TIPSETS tipsets"

OUTPUT=$($FOREST_CLI_PATH info show)

HEAD_EPOCH=$(echo "$OUTPUT" | sed -n 's/.*epoch: \([0-9]*\).*/\1/p')
EPOCH=$((HEAD_EPOCH - 1))

ETH_BLOCK_HASHES=()
ETH_TX_HASHES=()

for ((i=0; i<=NUM_TIPSETS; i++)); do
EPOCH_HEX=$(printf "0x%x" $EPOCH)
JSON=$(curl -s -X POST "$FOREST_URL" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg epoch "$EPOCH_HEX" '{jsonrpc: "2.0", id: 1, method: "Filecoin.EthGetBlockByNumber", params: [$epoch, false]}')")


HASH=$(echo "$JSON" | jq -r '.result.hash')
ETH_BLOCK_HASHES+=("$HASH")

if [[ $(echo "$JSON" | jq -e '.result.transactions') != "null" ]]; then
TRANSACTIONS=$(echo "$JSON" | jq -r '.result.transactions[]')
for tx in $TRANSACTIONS; do
ETH_TX_HASHES+=("$tx")
done
else
echo "No transactions found for block hash: $EPOCH_HEX"
fi

EPOCH=$((EPOCH - 1))
done

ERROR=0
echo "Testing Ethereum mapping"

for hash in "${ETH_BLOCK_HASHES[@]}"; do
JSON=$(curl -s -X POST "$FOREST_URL" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg hash "$hash" '{jsonrpc: "2.0", id: 1, method: "Filecoin.EthGetBalance", params: ["0xff38c072f286e3b20b3954ca9f99c05fbecc64aa", $hash]}')")

if [[ $(echo "$JSON" | jq -e '.result') == "null" ]]; then
echo "Missing tipset key for hash $hash"
ERROR=1
fi
done

for hash in "${ETH_TX_HASHES[@]}"; do
JSON=$(curl -s -X POST "$FOREST_URL" \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg hash "$hash" '{jsonrpc: "2.0", id: 1, method: "Filecoin.EthGetMessageCidByTransactionHash", params: [$hash]}')")

if [[ $(echo "$JSON" | jq -e '.result') == "null" ]]; then
echo "Missing cid for hash $hash"
ERROR=1
fi
done

echo "Done"
exit $ERROR
121 changes: 114 additions & 7 deletions src/chain/store/chain_store.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ use crate::interpreter::BlockMessages;
use crate::interpreter::VMTrace;
use crate::libp2p_bitswap::{BitswapStoreRead, BitswapStoreReadWrite};
use crate::message::{ChainMessage, Message as MessageTrait, SignedMessage};
use crate::networks::ChainConfig;
use crate::rpc::eth;
use crate::networks::{ChainConfig, Height};
use crate::rpc::eth::{self, eth_tx_from_signed_eth_message};
use crate::shim::clock::ChainEpoch;
use crate::shim::{
address::Address, econ::TokenAmount, executor::Receipt, message::Message,
Expand Down Expand Up @@ -78,6 +78,9 @@ pub struct ChainStore<DB> {

/// Ethereum mappings store
eth_mappings: Arc<dyn EthMappingsStore + Sync + Send>,

/// Needed by the Ethereum mapping.
chain_config: Arc<ChainConfig>,
}

impl<DB> BitswapStoreRead for ChainStore<DB>
Expand Down Expand Up @@ -131,12 +134,13 @@ where
let cs = Self {
publisher,
chain_index,
tipset_tracker: TipsetTracker::new(Arc::clone(&db), chain_config),
tipset_tracker: TipsetTracker::new(Arc::clone(&db), chain_config.clone()),
db,
settings,
genesis_block_header,
validated_blocks,
eth_mappings,
chain_config,
};

Ok(cs)
Expand All @@ -162,10 +166,10 @@ where
pub fn put_tipset(&self, ts: &Tipset) -> Result<(), Error> {
persist_objects(self.blockstore(), ts.block_headers().iter())?;

self.put_tipset_key(ts.key())?;

// Expand tipset to include other compatible blocks at the epoch.
let expanded = self.expand_tipset(ts.min_ticket_block().clone())?;
self.put_tipset_key(expanded.key())?;

self.update_heaviest(Arc::new(expanded))?;
Ok(())
}
Expand All @@ -177,6 +181,20 @@ where
Ok(())
}

/// Writes the delegated message `Cid`s to the blockstore for `EthAPI` queries.
pub fn put_delegated_message_hashes<'a>(
&self,
headers: impl Iterator<Item = &'a CachingBlockHeader>,
) -> Result<(), Error> {
tracing::debug!("persist eth mapping");

// The messages will be ordered from most recent block to less recent
let delegated_messages = self.headers_delegated_messages(headers)?;

self.process_signed_messages(&delegated_messages)?;
Ok(())
}

/// Reads the `TipsetKey` from the blockstore for `EthAPI` queries.
pub fn get_required_tipset_key(&self, hash: &eth::Hash) -> Result<TipsetKey, Error> {
let tsk = self
Expand All @@ -188,12 +206,19 @@ where
}

/// Writes with timestamp the `Hash` to `Cid` mapping to the blockstore for `EthAPI` queries.
pub fn put_mapping(&self, k: eth::Hash, v: Cid) -> Result<(), Error> {
let timestamp = chrono::Utc::now().timestamp() as u64;
pub fn put_mapping(&self, k: eth::Hash, v: Cid, timestamp: u64) -> Result<(), Error> {
self.eth_mappings.write_obj(&k, &(v, timestamp))?;
Ok(())
}

/// Reads the `Cid` from the blockstore for `EthAPI` queries.
pub fn get_mapping(&self, hash: &eth::Hash) -> Result<Option<Cid>, Error> {
Ok(self
.eth_mappings
.read_obj::<(Cid, u64)>(hash)?
.map(|(cid, _)| cid))
}

/// Expands tipset to tipset with all other headers in the same epoch using
/// the tipset tracker.
fn expand_tipset(&self, header: CachingBlockHeader) -> Result<Tipset, Error> {
Expand Down Expand Up @@ -354,6 +379,88 @@ where
.map_err(|e| Error::Other(format!("Could not get tipset from keys {e:?}")))?;
Ok((lbts, *next_ts.parent_state()))
}

/// Filter [`SignedMessage`]'s to keep only the most recent ones, then write corresponding entries to the Ethereum mapping.
pub fn process_signed_messages(&self, messages: &[(SignedMessage, u64)]) -> anyhow::Result<()>
where
DB: fvm_ipld_blockstore::Blockstore,
{
let eth_txs: Vec<(eth::Hash, Cid, u64, usize)> = messages
.iter()
.enumerate()
.filter_map(|(i, (smsg, timestamp))| {
if let Ok(tx) = eth_tx_from_signed_eth_message(smsg, self.chain_config.eth_chain_id)
{
if let Ok(hash) = tx.eth_hash() {
// newest messages are the ones with lowest index
Some((hash, smsg.cid().unwrap(), *timestamp, i))
} else {
None
}
} else {
None
}
})
.collect();
let filtered = filter_lowest_index(eth_txs);
let num_entries = filtered.len();

// write back
for (k, v, timestamp) in filtered.into_iter() {
tracing::trace!("Insert mapping {} => {}", k, v);
self.put_mapping(k, v, timestamp)?;
}
tracing::debug!("Wrote {} entries in Ethereum mapping", num_entries);
Ok(())
}

pub fn headers_delegated_messages<'a>(
&self,
headers: impl Iterator<Item = &'a CachingBlockHeader>,
) -> anyhow::Result<Vec<(SignedMessage, u64)>>
where
DB: fvm_ipld_blockstore::Blockstore,
{
let mut delegated_messages = vec![];

// Hygge is the start of Ethereum support in the FVM (through the FEVM actor).
// Before this height, no notion of an Ethereum-like API existed.
let filtered_headers =
headers.filter(|bh| bh.epoch >= self.chain_config.epoch(Height::Hygge));

for bh in filtered_headers {
if let Ok((_, secp_cids)) = block_messages(self.blockstore(), bh) {
let mut messages: Vec<_> = secp_cids
.into_iter()
.filter(|msg| msg.is_delegated())
.map(|m| (m, bh.timestamp))
.collect();
delegated_messages.append(&mut messages);
}
}

Ok(delegated_messages)
}
}

fn filter_lowest_index(values: Vec<(eth::Hash, Cid, u64, usize)>) -> Vec<(eth::Hash, Cid, u64)> {
let map: HashMap<eth::Hash, (Cid, u64, usize)> = values.into_iter().fold(
HashMap::default(),
|mut acc, (hash, cid, timestamp, index)| {
acc.entry(hash)
.and_modify(|&mut (_, _, ref mut min_index)| {
if index < *min_index {
*min_index = index;
}
})
.or_insert((cid, timestamp, index));
acc
},
);

map.into_iter()
.map(|(hash, (cid, timestamp, _))| (hash, cid, timestamp))
.collect()
}

/// Returns a Tuple of BLS messages of type `UnsignedMessage` and SECP messages
Expand Down
3 changes: 3 additions & 0 deletions src/chain_sync/chain_muxer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,9 @@ where
block.persist(&chain_store.db)?;
}

// This is needed for the Ethereum mapping
chain_store.put_tipset_key(tipset.key())?;

// Update the peer head
network
.peer_manager()
Expand Down
18 changes: 16 additions & 2 deletions src/chain_sync/tipset_syncer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -761,15 +761,20 @@ async fn sync_tipset_range<DB: Blockstore + Sync + Send + 'static>(
return Err(why.into());
};

// Sync and validate messages from the tipsets
// Persist tipset keys
for ts in parent_tipsets.iter() {
chain_store.put_tipset_key(ts.key())?;
}

// Sync and validate messages from the tipsets
tracker.write().set_stage(SyncStage::Messages);
if let Err(why) = sync_messages_check_state(
tracker.clone(),
state_manager,
network,
chain_store.clone(),
&bad_block_cache,
parent_tipsets,
parent_tipsets.clone(),
&genesis,
InvalidBlockStrategy::Forgiving,
)
Expand All @@ -780,6 +785,9 @@ async fn sync_tipset_range<DB: Blockstore + Sync + Send + 'static>(
return Err(why);
};

// Call only once messages persisted
chain_store.put_delegated_message_hashes(headers.into_iter())?;

// At this point the head is synced and it can be set in the store as the
// heaviest
debug!(
Expand Down Expand Up @@ -961,6 +969,9 @@ async fn sync_tipset<DB: Blockstore + Sync + Send + 'static>(
proposed_head.block_headers().iter(),
)?;

// Persist tipset key
chain_store.put_tipset_key(proposed_head.key())?;

// Sync and validate messages from the tipsets
if let Err(e) = sync_messages_check_state(
// Include a dummy WorkerState
Expand All @@ -979,6 +990,9 @@ async fn sync_tipset<DB: Blockstore + Sync + Send + 'static>(
return Err(e);
}

// Call only once messages persisted
chain_store.put_delegated_message_hashes(proposed_head.block_headers().iter())?;
LesnyRumcajs marked this conversation as resolved.
Show resolved Hide resolved

// Add the tipset to the store. The tipset will be expanded with other blocks
// with the same [epoch, parents] before updating the heaviest Tipset in
// the store.
Expand Down
3 changes: 3 additions & 0 deletions src/cli_shared/cli/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ pub struct Client {
pub token_exp: Duration,
/// Load actors from the bundle file (possibly generating it if it doesn't exist)
pub load_actors: bool,
/// `TTL` to set for Ethereum `Hash` to `Cid` entries or `None` to never reclaim them.
pub eth_mapping_ttl: Option<u32>,
}

impl Default for Client {
Expand Down Expand Up @@ -103,6 +105,7 @@ impl Default for Client {
),
token_exp: Duration::try_seconds(5184000).expect("Infallible"), // 60 Days = 5184000 Seconds
load_actors: true,
eth_mapping_ttl: None,
}
}
}
Loading
Loading