-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(core): multi-pubkey DB support #2093
base: dev
Are you sure you want to change the base?
Conversation
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Next review iteration!
mm2src/mm2_main/src/lp_native_dex.rs
Outdated
#[cfg(not(target_arch = "wasm32"))] | ||
{ | ||
let dbdir = ctx.dbdir(); | ||
fs::create_dir_all(&dbdir).map_to_mm(|e| MmInitError::ErrorCreatingDbDir { | ||
path: dbdir.clone(), | ||
error: e.to_string(), | ||
})?; | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the base db dir should be created here to allow the below to work
pub(super) async fn save_encrypted_passphrase( | |
ctx: &MmArc, | |
wallet_name: &str, | |
encrypted_passphrase_data: &EncryptedData, | |
) -> WalletsStorageResult<()> { | |
let wallet_path = ctx.wallet_file_path(wallet_name); | |
ensure_file_is_writable(&wallet_path).map_to_mm(WalletsStorageError::FsWriteError)?; | |
mm2_io::fs::write_json(encrypted_passphrase_data, &wallet_path, true) | |
.await | |
.mm_err(|e| WalletsStorageError::FsWriteError(e.to_string())) | |
} |
komodo-defi-framework/mm2src/mm2_main/src/lp_native_dex.rs
Lines 544 to 545 in a78704d
// This either initializes the cryptographic context or sets up the context for "no login mode". | |
initialize_wallet_passphrase(&ctx).await?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the catch !
@@ -5378,12 +5418,11 @@ pub struct HistoricalOrder { | |||
conf_settings: Option<OrderConfirmationsSettings>, | |||
} | |||
|
|||
pub async fn orders_kick_start(ctx: &MmArc) -> Result<HashSet<String>, String> { | |||
pub async fn orders_kick_start(ctx: &MmArc, db_id: Option<&str>) -> Result<HashSet<String>, String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to swaps kickstarts, we need to only kickstart orders if the coin is activated with the same db_id
as the order's db_id
ref.
komodo-defi-framework/mm2src/mm2_main/src/lp_swap.rs
Lines 1545 to 1601 in 9a8494e
let taker_coin = loop { | |
match lp_coinfind(&ctx, &taker_coin_ticker).await { | |
Ok(Some(c)) => { | |
if taker_coin_db_id == c.account_db_id().await.as_deref() { | |
break c; | |
}; | |
info!( | |
"Can't kickstart the swap {} until the coin {} is activated with pubkey: {}", | |
swap.uuid(), | |
taker_coin_ticker, | |
taker_coin_db_id.unwrap_or(&ctx.rmd160.to_string()) | |
); | |
Timer::sleep(5.).await; | |
}, | |
Ok(None) => { | |
info!( | |
"Can't kickstart the swap {} until the coin {} is activated", | |
swap.uuid(), | |
taker_coin_ticker | |
); | |
Timer::sleep(5.).await; | |
}, | |
Err(e) => { | |
error!("Error {} on {} find attempt", e, taker_coin_ticker); | |
return; | |
}, | |
}; | |
}; | |
let maker_coin = loop { | |
match lp_coinfind(&ctx, &maker_coin_ticker).await { | |
Ok(Some(c)) => { | |
if maker_coin_db_id == c.account_db_id().await.as_deref() { | |
break c; | |
}; | |
info!( | |
"Can't kickstart the swap {} until the coin {} is activated with pubkey: {}", | |
swap.uuid(), | |
maker_coin_ticker, | |
maker_coin_db_id.unwrap_or(&ctx.rmd160.to_string()) | |
); | |
Timer::sleep(5.).await; | |
}, | |
Ok(None) => { | |
info!( | |
"Can't kickstart the swap {} until the coin {} is activated", | |
swap.uuid(), | |
maker_coin_ticker | |
); | |
Timer::sleep(5.).await; | |
}, | |
Err(e) => { | |
error!("Error {} on {} find attempt", e, maker_coin_ticker); | |
return; | |
}, | |
}; | |
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry if I wasn't clear about this. I meant that if the other coin of the order was with a different db_id
. Similar to what we done with swaps before if you remember.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some more review notes related to your latest fixes!
mm2src/coins/eth/v2_activation.rs
Outdated
DerivationMethod::HDWallet(hd_wallet) => hd_wallet.get_enabled_address().await.map(|addr| { | ||
// Skip the first byte of the uncompressed public key before converting to the eth address. | ||
let pubkey = Public::from_slice(&addr.pubkey().as_bytes()[1..]); | ||
public_to_address(&pubkey).to_string() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should use display_eth_address
function and strip the 0x
at the start
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should use display_eth_address function and strip the 0x at the start
I think the below would be much simpler than what you implemented
let address = display_eth_address(&addr.address());
address.trim_start_matches("0x").to_string()
mm2src/coins/eth/v2_activation.rs
Outdated
let pubkey = dhash160(activated_key.public().as_bytes()).to_string(); | ||
// Skip the first byte of the uncompressed public key before converting to the eth address. | ||
let pubkey = Public::from_slice(&activated_key.public().as_bytes()[1..]); | ||
let pubkey = public_to_address(&pubkey).to_string(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here
mm2src/mm2_main/src/lp_native_dex.rs
Outdated
#[cfg(not(target_arch = "wasm32"))] | ||
{ | ||
let dbdir = ctx.dbdir(None); | ||
fs::create_dir_all(&dbdir).map_to_mm(|e| MmInitError::ErrorCreatingDbDir { | ||
path: dbdir.clone(), | ||
error: e.to_string(), | ||
})?; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be before initialize_wallet_passphrase
as it was done before
komodo-defi-framework/mm2src/mm2_main/src/lp_native_dex.rs
Lines 553 to 554 in 1675a53
// This either initializes the cryptographic context or sets up the context for "no login mode". | |
initialize_wallet_passphrase(&ctx).await?; |
You don't need to create the default db dir, only the global one. We probably also need to
ensure_dir_is_writable
, just in case.
@@ -5378,12 +5418,11 @@ pub struct HistoricalOrder { | |||
conf_settings: Option<OrderConfirmationsSettings>, | |||
} | |||
|
|||
pub async fn orders_kick_start(ctx: &MmArc) -> Result<HashSet<String>, String> { | |||
pub async fn orders_kick_start(ctx: &MmArc, db_id: Option<&str>) -> Result<HashSet<String>, String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry if I wasn't clear about this. I meant that if the other coin of the order was with a different db_id
. Similar to what we done with swaps before if you remember.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks you for the fixes! Next review iteration where I only focus on lp_ordermatch.rs
!
Please merge with latest dev to get docker tests to pass. We should also start writing all the testcases we discussed before related to swaps / orders when we activate one of the coins with a different db_id
(whether it's the one that has the swap/orders data or not). I remember we discussed a lot of these cases.
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
base_coin_account_id: Option<String>, | ||
rel_coin_account_id: Option<String>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add doc comments for these. And similar others.
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
coins.insert(order.request.base.clone()); | ||
coins.insert(order.request.rel.clone()); | ||
taker_orders.insert(order.request.uuid, order); | ||
let order_base_coin_account_id = order.base_coin_account_id.clone().unwrap_or(ctx.default_db_id()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Taker order is not always loaded from rel coin db_id
as it depends on if the order is buy or sell ref
komodo-defi-framework/mm2src/mm2_main/src/lp_ordermatch.rs
Lines 1688 to 1692 in 117aa44
pub fn account_id(&self) -> &Option<String> { | |
match self.request.action { | |
TakerAction::Buy => &self.rel_coin_account_id, | |
TakerAction::Sell => &self.base_coin_account_id, | |
} |
Maybe we should add a function called
other_coin_account_id
to TakerOrder
and MakerOrder
and use it here and here let order_rel_coin_db_id = order.rel_coin_account_id.clone().unwrap_or(ctx.default_db_id()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
#[cfg(target_arch = "wasm32")] | ||
ordermatch_db: ConstructibleDb<OrdermatchDb>, | ||
} | ||
|
||
#[allow(unused)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you add this? I can see that this is actually used.
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
@@ -2970,12 +2999,13 @@ fn lp_connect_start_bob(ctx: MmArc, maker_match: MakerMatch, maker_order: MakerO | |||
}, | |||
}; | |||
|
|||
let account_db_id = maker_coin.account_db_id().await; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we use the orders db_id
to have swap files/data saved in the same place as the order files/data?
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
@@ -3121,6 +3153,7 @@ fn lp_connected_alice(ctx: MmArc, taker_order: TakerOrder, taker_match: TakerMat | |||
); | |||
|
|||
let now = now_sec(); | |||
let account_db_id = taker_coin.account_db_id().await; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
storage.load_order_from_history(req.uuid, db_id.as_deref()).await, | ||
&storage.select_order_status(req.uuid, db_id.as_deref()).await, | ||
) { | ||
info!("Order with UUID=({})", req.uuid); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need for this info
mm2src/mm2_main/src/lp_ordermatch.rs
Outdated
maker_orders_ctx.add_order(ctx.weak(), order.clone(), None); | ||
let order_rel_coin_db_id = order.rel_coin_account_id.clone().unwrap_or(ctx.default_db_id()); | ||
|
||
loop { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This loop and the other one will never end if the other coin is not enabled with the right address / db_id
. We should rethink this whole kickstarting approach. This is where the code looks for if the coin for an order is enabled or not
komodo-defi-framework/mm2src/mm2_main/src/lp_ordermatch.rs
Lines 4644 to 4654 in 117aa44
if let Ok(Some(coin)) = lp_coinfind(&ctx, &ticker).await { | |
let balance = match coin.my_spendable_balance().compat().await { | |
Ok(balance) => balance, | |
Err(_) => continue, | |
}; | |
if Some(&balance) != current_balance.as_ref() { | |
let coins_ctx = CoinsContext::from_ctx(&ctx).unwrap(); | |
coins_ctx.balance_updated(&coin, &balance).await; | |
current_balance = Some(balance); | |
} | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huge work!
Here is the next review iteration
#[cfg(not(target_arch = "wasm32"))] | ||
{ | ||
let pubkey = { | ||
let pubkey = Public::from_slice(activated_key.public().as_bytes()); | ||
let addr = display_eth_address(&public_to_address(&pubkey)); | ||
addr.trim_start_matches("0x").to_string() | ||
}; | ||
|
||
run_db_migration_for_new_pubkey(ctx, pubkey) | ||
.await | ||
.map_to_mm(EthActivationV2Error::InternalError)?; | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please clarify why this block is only for non-wasm target? is it bcz IDB automatically handles schema changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WASM doesn't need theses operations only SQL needs running migration when opening db and also additional directory checks which of course WASM doesn't need
} | ||
|
||
/// Test method for initializing a single-user database connection in-memory. | ||
pub fn init_test(ctx: &MmArc) -> Result<(), String> { Self::init_impl_test(ctx, None, DbIdConnKind::Single) } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
} | ||
|
||
/// Initialize a database connection. | ||
pub async fn init_test(ctx: &MmArc, db_id: Option<&str>) -> Result<(), String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here too, lets add #[cfg(all(feature = "for-tests", not(target_arch = "wasm32")))]
annotation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had some problem doing this(init_test
isn't really used directly in tests and some other functions also uses it which aren't annotated with tests cfg) but I guess I can pick it up again after working on more important stuffs in this PR.. thanks for the catch !
#[derive(Debug, Serialize)] | ||
pub struct NftsTransferHistoryLists { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add doc comm for new NftsTransferHistoryLists structure
@@ -653,7 +662,7 @@ pub struct NftTransferHistoryFilters { | |||
} | |||
|
|||
/// Contains parameters required to update NFT transfer history and NFT list. | |||
#[derive(Debug, Deserialize)] | |||
#[derive(Debug, Deserialize, Clone)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
its not critical but please make Clone
first, as we try to keep alphabetical order in derive
pub async fn find_unique_nft_account_ids( | ||
ctx: &MmArc, | ||
chains: Vec<Chain>, | ||
) -> Result<HashMap<String, Vec<Chain>>, String> { | ||
let cctx = try_s!(CoinsContext::from_ctx(ctx)); | ||
let coins = cctx.coins.lock().await; | ||
let coins = coins.values().collect::<Vec<_>>(); | ||
|
||
let mut active_id_chains = HashMap::new(); | ||
for coin in coins.iter() { | ||
if coin.is_available() { | ||
// Use default if no db_id | ||
let db_id = coin | ||
.inner | ||
.account_db_id() | ||
.await | ||
.unwrap_or_else(|| ctx.rmd160.to_string()); | ||
let entry = active_id_chains.entry(db_id).or_insert_with(Vec::new); | ||
if let Ok(chain) = Chain::from_ticker(coin.inner.ticker()) { | ||
if chains.contains(&chain) { | ||
entry.push(chain); | ||
} | ||
} | ||
} | ||
} | ||
|
||
Ok(active_id_chains) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could be simplified
async fn find_unique_nft_account_ids(
ctx: &MmArc,
chains: Vec<Chain>,
) -> Result<HashMap<String, Vec<Chain>>, String> {
let cctx = try_s!(CoinsContext::from_ctx(ctx));
let coins = cctx.coins.lock().await;
let mut active_id_chains: HashMap<String, Vec<Chain>> = HashMap::new();
for coin in coins.values() {
if coin.is_available() {
// Use default if no db_id
let db_id = coin
.inner
.account_db_id()
.await
.unwrap_or_else(|| ctx.rmd160.to_string());
if let Ok(chain) = Chain::from_ticker(coin.inner.ticker()) {
if chains.contains(&chain) {
active_id_chains.entry(db_id).or_default().push(chain);
}
}
}
}
Ok(active_id_chains)
}
You can avoid collecting values into a Vec, and directly iterate over coins.values()
.
instead of creating entry object, you can call .entry(db_id).or_default()
.
Also seems like there is no need in pub
let coins = coins.values().collect::<Vec<_>>(); | ||
|
||
for coin in coins.iter() { | ||
if coin.is_available() { | ||
// Use default if no db_id | ||
let db_id = coin | ||
.inner | ||
.account_db_id() | ||
.await | ||
.unwrap_or_else(|| ctx.rmd160.to_string()); | ||
if let Ok(chain) = Chain::from_ticker(coin.inner.ticker()) { | ||
if chains == chain { | ||
return Ok(Some((db_id, chain))); | ||
} | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here you also dont have to collect coins into vector
let coins = cctx.coins.lock().await;
for coin in coins.values() {
if coin.is_available() {
// Use default if no db_id
let db_id = coin
.inner
.account_db_id()
.await
.unwrap_or_else(|| ctx.rmd160.to_string());
if let Ok(chain) = Chain::from_ticker(coin.inner.ticker()) {
if chains == chain {
return Ok(Some((db_id, chain)));
}
}
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have question
@@ -744,7 +746,7 @@ mod wasm_test { | |||
// async fn create_to_address_fails_on_unverified_notes() { | |||
// // init blocks_db | |||
// let ctx = mm_ctx_with_custom_db(); | |||
// let blockdb = BlockDbImpl::new(&ctx, TICKER.to_string(), PathBuf::new()).await.unwrap(); | |||
// let blockdb = BlockDbImpl::new(&ctx, TICKER.to_string(), PathBuf::new(), None).await.unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a todo above Todo: Uncomment after improving tx creation time
could you tell what the problem with this?
I remember that there was an issue with blocks synchronisation, which was fixed by adding sync params. but what about tx creation time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generating zcoin txs uses lot of resource and takes time to run...there're some optimization for native targets that cannot be achieved with WASM yet
This pull request enables support for managing multiple public keys in mm2 databases. Initially, a single database instance is generated using the startup public key rmd160. With this update, the
mm_ctx
will manage multiple database instances by leveraging the public key fromMmCoin::account_db_id
if defined, or defaulting to the mm2 startup public key.Completed
sql_connection_pool
for managing all instances of shared and sqlite connections from one placeasync_sql_connection_pool
for managing all instances of async sqlite connections from one placeindexeddb
driver implementation to allow multi pubkey supportSwapContext
,MySwapStorage
,MakerSwapStorage
,TakerSwapStorage
NftCtx
,UtxoBlockHeaderStorage
,Zcoin storages(web only)
,lp_ordermatch
and other storage impl to support multipubkeyswap_kick_start
andorders_kick_start
unfinished swaps/orders upon new database instance creation or account activation using a custom public key.Checklists for next PR
get_public_key
,get_public_key_hash
,show_priv_key
rpc for multi pubkey dbs in a another PR