Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package Relay Draft 2 (Orphanage Eviction Approach) #9

Closed
wants to merge 59 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
c50859c
[rpc] allow submitpackage to be called outside of regtest
glozow Apr 11, 2023
d6a52d6
[validation] call AcceptSingleTransaction when only 1 package tx left
glozow Dec 15, 2022
3005f35
MOVEONLY: move package checks into helper functions
glozow Sep 16, 2022
feb39aa
scripted-diff: rename CheckPackage to IsPackageWellFormed
glozow Jan 19, 2023
298c09c
[packages] AncestorPackage sorts and builds ancestor subsets
glozow Sep 16, 2022
ee8d481
[fuzz] AncestorPackage
glozow Feb 16, 2023
c5e16ee
[test util] multi-input multi-output CreateValidMempoolTransaction
glozow Oct 3, 2022
08189a8
[validation] skip and pre-fill txns with irreversibly invalid ancestors
glozow Dec 14, 2022
1241c5e
[validation] skip last tx in AcceptPackage "submit individually" loop
glozow Dec 16, 2022
704ecd1
[validation] validate packages by submitting each tx's ancestor sub-p…
glozow Dec 15, 2022
161c61a
[policy] allow any ancestor package, not just child-with-unconfirmed-…
glozow Feb 17, 2023
70adade
[refactor] rename AcceptPackage to AcceptAncestorPackage
glozow Apr 17, 2023
8cf5291
-- Part 1: Orphan Resolution Module --
glozow Feb 20, 2023
9977fa6
[p2p] add tx package tracker
glozow May 1, 2022
2d002cc
[refactor] wrap TxOrphanage inside TxPackageTracker
glozow Oct 6, 2022
73656ff
[txorphange] GetTx by wtxid
glozow Jan 24, 2023
2947755
[txorphanage] EraseTx by wtxid
glozow Feb 20, 2023
14e2f3e
[txorphanage] return erased wtxids from EraseForBlock
glozow Feb 20, 2023
80d756d
[log] add category TXPACKAGES for orphanage and package relay
glozow Mar 6, 2023
f604dc3
[txorphanage] impose a maximum total size of orphans
glozow Mar 31, 2023
f030a4e
[txorphanage] track bytes provided per peer
glozow Apr 18, 2023
412876f
[txorphanage] when peers are overloaded, favor their orphans for evic…
glozow Apr 18, 2023
809e2c4
[txorphanage] handle AddTx(nullptr)
glozow Apr 18, 2023
2d5e167
[txpackagetracker] add orphan resolution tracker
glozow Jan 24, 2023
b41a471
[refactor] use txpackagetracker for orphan resolution
glozow Oct 6, 2022
31c556d
[txpackagetracker] delete unused OrphanageAddTx
glozow Apr 18, 2023
0d693fb
[txrequest] GetCandidatePeers and ResetRequestTimeout
glozow Mar 20, 2023
dffdc15
[txorphanage] support multiple announcers
glozow Apr 18, 2023
808fe20
[unit test] orphanage: multiple announcers, eviction
glozow Apr 18, 2023
427fe67
[p2p] use all orphan announcers as potential parent sources
glozow Oct 6, 2022
17a88be
[doc] no notfound handling for orphan resolution by parent txid
glozow Apr 19, 2023
ee88683
[functional test] orphan handling with multiple announcers and dosers
glozow Apr 19, 2023
e46daca
-- Part 2: Signal SENDPACKAGES --
glozow Feb 20, 2023
8909aec
[txpackagetracker] negotiation logic
glozow Feb 20, 2023
e2ce8cc
[unit test] txpackagetracker tests
glozow Feb 20, 2023
0544ec6
[p2p] signal support for package relay
glozow Oct 7, 2022
eda5593
[rpc] expose package relay on rpc
glozow Sep 6, 2022
cab6f5a
-- Part 3: Request Ancestor Package Info For Orphans --
glozow Feb 20, 2023
80d688b
[packages] hash multiple transactions
glozow Jan 9, 2023
be2e540
[validation/p2p] separate TxValidationResult and rejects filter for l…
glozow Sep 12, 2022
20ca067
[p2p] respond to getdata(ancpkginfo) requests with ancpkginfo
glozow Sep 6, 2022
d7cea47
[txpackagetracker] create ancpkginfo requests using orphan resolution…
glozow Jan 24, 2023
9e3f816
[unit test] txpackagetracker orphan tracking
glozow Feb 20, 2023
0850bc9
[p2p] Resolve orphans by requesting ancpkginfo and requesting txdata
glozow Oct 6, 2022
c71b4f4
-- Part 4: Download and Validate Packages --
glozow Feb 20, 2023
9eb814a
[p2p] respond to getpkgtxns with pkgtxns or notfound MSG_PKGTXNS
glozow Oct 3, 2022
165b930
[txpackagetracker] track packages to download
glozow Jan 5, 2023
f38d199
[txpackagetracker] handle pkgtxns/notfound, return validation work items
glozow Jan 9, 2023
68e8897
[unit test] PkgInfoAllowed, ReceivedAncPkgInfo, ReceivedPkgTxns
glozow Feb 20, 2023
df737db
[p2p] use ancpkginfo to make getpkgtxns requests
glozow Jan 6, 2023
3a67c66
[p2p] handle pkgtxns and validate packages
glozow Jan 9, 2023
9d2a7f7
[p2p] don't inv fee-bumping children to non-pkgrelay peers
glozow Apr 4, 2023
d172b7b
-- Part 5: Nice to Haves --
glozow Feb 20, 2023
3b461e8
[txpackagetracker] delay overloaded peers and expire orphans faster
glozow May 3, 2023
9d9e1de
[mempool] evict everything below min relay fee in TrimToSize()
glozow Jan 17, 2023
1d2b4cb
[test framework] return txhex from create_lots_of_big_transactions
glozow Jan 19, 2023
d7ed479
[test] raise wallet_abandonconflict -minrelaytxfee settings
glozow Jan 19, 2023
82a2cdd
[mempool] persist packages across restart
glozow Jan 19, 2023
7ee77ad
wip [test] orphanage DoSing
glozow May 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[p2p] handle pkgtxns and validate packages
  • Loading branch information
glozow committed May 9, 2023
commit 3a67c661deb218264b65111b5cb051e42af341be
99 changes: 98 additions & 1 deletion src/net_processing.cpp
Original file line number Diff line number Diff line change
@@ -604,6 +604,10 @@ class PeerManagerImpl final : public PeerManager
bool ProcessOrphanTx(Peer& peer)
EXCLUSIVE_LOCKS_REQUIRED(!m_peer_mutex, g_msgproc_mutex);

/** Validate package if any */
void ProcessPackage(CNode& node, const node::TxPackageTracker::PackageToValidate& package)
EXCLUSIVE_LOCKS_REQUIRED(!m_peer_mutex, g_msgproc_mutex);

/** Process a single headers message from a peer.
*
* @param[in] pfrom CNode of the peer
@@ -3134,6 +3138,78 @@ bool PeerManagerImpl::ProcessOrphanTx(Peer& peer)
return false;
}

void PeerManagerImpl::ProcessPackage(CNode& node, const node::TxPackageTracker::PackageToValidate& package)
{
AssertLockHeld(g_msgproc_mutex);
LOCK(cs_main);
// We won't re-validate the exact same transaction or package again.
if (m_recent_rejects_reconsiderable.contains(GetPackageHash(package.m_unvalidated_txns))) {
// Should we do anything else here?
return;
}
const auto package_result{ProcessNewPackage(m_chainman.ActiveChainstate(), m_mempool,
package.m_unvalidated_txns, /*test_accept=*/false)};
if (package_result.m_state.IsInvalid()) {
// If another peer sends the same packageinfo again, we can immediately reject it without
// re-downloading the transactions. Note that state.IsInvalid() doesn't mean all
// transactions have been rejected.
m_recent_rejects_reconsiderable.insert(package.m_pkginfo_hash);
}
std::set<uint256> successful_txns;
std::set<uint256> invalid_final_txns;
for (const auto& tx : package.m_unvalidated_txns) {
const auto& txid = tx->GetHash();
const auto& wtxid = tx->GetWitnessHash();
const auto result{package_result.m_tx_results.find(wtxid)};
if (package_result.m_state.IsValid() ||
package_result.m_state.GetResult() == PackageValidationResult::PCKG_TX) {
// If PCKG_TX or valid, every tx should have a result.
Assume(result != package_result.m_tx_results.end());
}
if (result == package_result.m_tx_results.end()) break;
if (result->second.m_result_type == MempoolAcceptResult::ResultType::VALID) {
LogPrint(BCLog::MEMPOOL, "\nProcessPackage: tx %s from peer=%d accepted\n", txid.ToString(), node.GetId());
successful_txns.insert(wtxid);
m_txrequest.ForgetTxHash(txid);
m_txrequest.ForgetTxHash(wtxid);
RelayTransaction(txid, wtxid);
node.m_last_tx_time = GetTime<std::chrono::seconds>();
for (const CTransactionRef& removedTx : result->second.m_replaced_transactions.value()) {
AddToCompactExtraTransactions(removedTx);
}
} else if (result->second.m_state.GetResult() != TxValidationResult::TX_WITNESS_STRIPPED) {
if (result->second.m_state.GetResult() == TxValidationResult::TX_LOW_FEE) {
m_recent_rejects_reconsiderable.insert(wtxid);
// FIXME: also cache subpackage failure
} else {
m_recent_rejects.insert(wtxid);
if (result->second.m_state.GetResult() == TxValidationResult::TX_INPUTS_NOT_STANDARD && wtxid != txid) {
m_recent_rejects.insert(txid);
m_txrequest.ForgetTxHash(txid);
} else if (result->second.m_state.GetResult() == TxValidationResult::TX_CONSENSUS) {
invalid_final_txns.insert(wtxid);
}
}
m_txrequest.ForgetTxHash(wtxid);
if (RecursiveDynamicUsage(*tx) < 100000) {
AddToCompactExtraTransactions(tx);
}
LogPrint(BCLog::MEMPOOLREJ, "\nProcessPackage: %s from peer=%d was not accepted: %s\n",
wtxid.ToString(), node.GetId(), result->second.m_state.ToString());
MaybePunishNodeForTx(wtxid == package.m_rep_wtxid ? node.GetId() : package.m_info_provider, result->second.m_state);
}
m_txpackagetracker->EraseOrphanTx(wtxid);
}
m_txpackagetracker->FinalizeTransactions(successful_txns, invalid_final_txns);
// Do this last to avoid adding children that were already validated within this package.
for (const auto& tx : package.m_unvalidated_txns) {
auto iter{package_result.m_tx_results.find(tx->GetWitnessHash())};
if (iter != package_result.m_tx_results.end() && iter->second.m_result_type == MempoolAcceptResult::ResultType::VALID) {
m_txpackagetracker->AddChildrenToWorkSet(*tx);
}
}
}

bool PeerManagerImpl::PrepareBlockFilterRequest(CNode& node, Peer& peer,
BlockFilterType filter_type, uint32_t start_height,
const uint256& stop_hash, uint32_t max_height_diff,
@@ -4907,7 +4983,26 @@ void PeerManagerImpl::ProcessMessage(CNode& pfrom, const std::string& msg_type,
}

if (msg_type == NetMsgType::PKGTXNS) {
LogPrint(BCLog::NET, "pkgtxns received from peer=%d", pfrom.GetId());
if (RejectIncomingTxs(pfrom)) {
LogPrint(BCLog::NET, "\npkgtxns sent in violation of protocol peer=%d\n", pfrom.GetId());
pfrom.fDisconnect = true;
return;
}
if (!m_txpackagetracker) return;
unsigned int num_txns = ReadCompactSize(vRecv);
if (num_txns == 0) return;
if (num_txns > MAX_PACKAGE_COUNT) {
Misbehaving(*peer, 100, strprintf("pkgtxns size = %u", num_txns));
return;
}
std::vector<CTransactionRef> package_txns;
package_txns.resize(num_txns);
for (unsigned int n = 0; n < num_txns; n++) {
vRecv >> package_txns[n];
}
if (const auto package_to_validate{m_txpackagetracker->ReceivedPkgTxns(pfrom.GetId(), package_txns)}) {
ProcessPackage(pfrom, package_to_validate.value());
}
return;
}

@@ -5102,6 +5197,8 @@ void PeerManagerImpl::ProcessMessage(CNode& pfrom, const std::string& msg_type,
pfrom.fDisconnect = true;
}
m_txpackagetracker->ForgetPkgInfo(pfrom.GetId(), inv.hash, node::PKG_RELAY_ANCPKG);
} else if (inv.IsMsgPkgTxns() && m_enable_package_relay) {
m_txpackagetracker->ReceivedNotFound(pfrom.GetId(), inv.hash);
}
}
}
17 changes: 16 additions & 1 deletion test/functional/p2p_package_relay.py
Original file line number Diff line number Diff line change
@@ -365,6 +365,8 @@ def test_orphan_announcer_memory(self):
peer_package_relay2 = node.add_p2p_connection(PackageRelayer())
# Sends an inv for the orphan while the node is requesting ancpkginfo
peer_package_relay3 = node.add_p2p_connection(PackageRelayer())
# Sends an inv for the orphan while the node is requesting txdata using the ancpkginfo
peer_package_relay4 = node.add_p2p_connection(PackageRelayer())

peer_package_relay1.send_and_ping(msg_inv([orphan_inv]))
self.fastforward(NONPREF_PEER_TX_DELAY + 1)
@@ -392,11 +394,21 @@ def test_orphan_announcer_memory(self):
peer_package_relay2.wait_for_getancpkginfo(int(orphan_wtxid, 16))

self.log.info("Test that the node requests ancpkginfo from a different peer if peer disconnects")
# Peer 2 disconnected before responding
# Peer2 disconnected before responding
peer_package_relay2.peer_disconnect()
self.fastforward(1)
peer_package_relay3.sync_with_ping()
peer_package_relay3.wait_for_getancpkginfo(int(orphan_wtxid, 16))
# Peer3 provides ancpkginfo but doesn't respond to getpkgtxns request
ancpkginfo_message = msg_ancpkginfo([int(wtxid, 16) for wtxid in package_wtxids])
peer_package_relay3.send_and_ping(ancpkginfo_message)
self.fastforward(1)
peer_package_relay3.wait_for_getpkgtxns([int(wtxid, 16) for wtxid in package_wtxids[:-1]])
peer_package_relay4.send_and_ping(msg_inv([orphan_inv]))
# The getpkgtxns request to peer3 expires after 60sec.
self.fastforward(60)
peer_package_relay3.sync_with_ping()
peer_package_relay4.wait_for_getancpkginfo(int(orphan_wtxid, 16))

@cleanup
def test_ancpkginfo_received(self):
@@ -430,6 +442,9 @@ def test_ancpkginfo_received(self):
self.fastforward(NONPREF_PEER_TX_DELAY + 1)
# The orphan should not be re-requested.
peer2.wait_for_getpkgtxns([int(wtxid, 16) for wtxid in package_wtxids[:-1]])
peer2.send_and_ping(msg_pkgtxns(txns=package_txns[:-1]))
peer2.sync_with_ping()
assert_equal(set(node.getrawmempool()), set([tx.rehash() for tx in package_txns]))

@cleanup
def test_ancpkginfo_invalid_ancestors(self):