Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Api improvements #131

Merged
merged 32 commits into from
Oct 6, 2022
Merged

Api improvements #131

merged 32 commits into from
Oct 6, 2022

Conversation

nepet
Copy link
Contributor

@nepet nepet commented Sep 14, 2022

This is a wip PR to address some API improvements.

It so far contains:

  • UTC in peerswapd (lnd) logs
  • reloadpolicy shows min_swap_amount_msat
  • Move min_swap_amount to policy (still hard-coded)
  • Rework of the dev env (it was a mess)
  • Validate addpeer addsuspeer inputs (need to be a hex[66])
  • Unix timestamp in responses
  • Rework of the internal handling of allowswaprequests. This now persists in the policy. Also allowswaprequests now responds with the policy struct.
  • Remove unused and unimplemented listnodes request
  • Added Swagger file for easy api exploration
  • Serialize Enums

@nepet nepet linked an issue Sep 14, 2022 that may be closed by this pull request
We already use a lower boundary in sat to perform a swap due to
economical reasons. We want to be able to return this value via the
api, so this commit moves this value to the policy where it belongs.
The MinSwapAmountMsat is now in msat, set to a default of
100000000 msat and can NOT be overridden in the policy file.
We might change this in the future.

This might change at some point in the future.
We missed some mutex locks on the getters.
GLightning returned the struct in camel case but we want to use
snake case for the returned values / structs.
We want a cleaner toolkit for developing and an easy way to setup
a linear network of 2 cln nodes.
We want a cleaner toolkit for developing and an easy way to setup
a linear network of 2 lnd nodes.
We want to return the same data for the listswap calls on both
peerswapd for lnd and the cln peerswap plugin. We use the protos
and the generated stubs to ensure the same return values on the
listswaps, listactiveswaps, swapin and swapout commands.
We want a cleaner toolkit for developing and an easy way to setup
a linear network of 2 lnd nodes.
We want to ensure that the string that is written to the policy
file lists (suspicious_peers and allowlisted_peers) is not empty
or not a valid hex representation of a pubkey.
We used to handle the global allowance of swaps non persistent in
the swap service. It is better to have all these kind of
constraints (this is a kill-switch) in one place.

This also adds some methods to set and get the value.
When we generate the stubs from the proto the structs gets tagged
with json:"omitempty". When we pass this message to a json
marshaler for e.g. printing to stdout, the marshaler omits empty
fields or fields that contain the default value for their type.
What we want is to print all fields even if they are empty. This
adds a custom marshaler that is configured to emit unpopulated
values.
We want to return the policy struct. This uses the generated struct
from the peerswaprpc protos.
We now use the policy instead of the inmemory map to determine if
we can start a new swap or not.

In the case of a incoming swap request from a peer, this check is
moved to the CheckRequestWrapperAction as we also want to send our
peer a cancel message if we do not allow for swaps atm.

Prior this commit the check was performed in the service methods
and did not send a cancel message to the peer so that the peer
could get stuck.
@ShahanaFarooqui
Copy link

ShahanaFarooqui commented Sep 27, 2022

Hi Peter, Please check my inline comments below for API improvement PR review:

  1. Date format: created_at field response returns its values in PDT format. However, returning the date/time fields in epoch format is usually a recommended practice as it is easier to convert them in desired formats in the UI. Also, all other core lightning RPC calls return dates in epoch format.

[Shahana]: It is fixed.

  1. resendMsg response: Successful resendMsg doesn't return any value which can be confusing. A string/boolean response will be more useful.

[Shahana]: Throws error now:
{ "error": { "type": "lightning", "name": "LightningError", "message": "swap.NextMessage is nil", "code": -1, "fullType": "lightning" } }

  1. Response case styling: Most of the calls are returning response fields in snake case which is core lightning standard as well. But reloadpolicy, addpeer, and removepeer response fields are in pascal case.

[Shahana]: Mostly fixed except error object's field fullType.

  1. Enumeration values: There are two fields, 'role' and 'type', which are returned as integers in the getswap call while all others return them as string values.

[Shahana]: getSwap returns numeric values while 'listswaps', 'listactiveswaps' etc return strings for role & type both.

  1. Swap states subscriptions: We should have an RPC command to subscribe to state update for any active swap like waitsendpay, waitanyinvoice, etc.

[Shahana]: Not available.

  1. List swap filter (swap-in, swap-out and cancelled swap), sort and pagination

[Shahana]: Not available.

  1. Get the minimum required amount: We do not have any way to know the minimum required amount for swap-out and swap-in. Currently, min amount validation in the UI has a hard coded value of 100,000 sats. Ideally it should be provided by the API not hardcoded.

[Shahana]: Fixed, receiving it in reloadpolicy as min_swap_amount_msat now.

  1. Get allowSwapRequests current status: I am unable to get current 'allowSwapRequests' status. We cannot expose a feature in the UI to update something without knowing its current status.

[Shahana]: Fixed, receiving it in reloadpolicy as allow_new_swaps now.

  1. Field name for 'opening_tx-hex': ActiveSwap's 'data' object has a field named 'opening_tx-hex' which does not follow any naming convension.

[Shahana]: Fixed with opening_tx_id now.

  1. API for updating AcceptAllPeers and ReserveOnchainMsat: I understand that it might not be possible to update these crucial fields without CLN restart. However, I am adding it for the sake of completeness. There is no API to update 'AcceptAllPeers' or 'ReserveOnchainMsat' fields of reloadPolicy. Node ids should also be returned with Aliases in various APIs like listSwaps, PeerAllowList, SuspiciousPeerList, etc.

[Shahana]: Not available.

  1. Unlock loop: lnd and peerswapd get into an unlock wallet loop if the user will exit lnd but keep lnd-peerswap running. Once lnd restarts, peerswapd tries to connect but lnd asks to unlock the wallet. Thus both run into a loop.

[Shahana]: Fixed, does not get into the loop. It backs off for 30 seconds but does not reconnect even if lnd is up again within these 30 seconds. Peerswap restart required if lnd restarts.
2022/09/26 15:54:05 [DEBUG] [grpc_conn]: grpc_retry attempt: 1, backoff for 30s
2022/09/26 15:54:35 [INFO] Lnd synced, continue...
2022/09/26 15:54:35 [DEBUG] [grpc_conn]: grpc_retry attempt: 0, got err: rpc error: code = Canceled desc = context canceled
2022/09/26 15:54:35 [DEBUG] [grpc_conn]: grpc_retry attempt: 0, parent context error: context canceled
2022/09/26 15:54:35 rpc error: code = Canceled desc = context canceled

  1. Adds and removes undefined to policy lists: User is allowed to add 'undefined' in Allower and Suspicious Peers lists.

[Shahana]: Fixed with error message <invalid reflect.Value> is not a valid public key

  1. Error message when the amount is missing: The RPC command does not throw any error if the amount is missing in swap-out and swap-in requests.

[Shahana]: Error from RPC call "Plugin terminated before replying to RPC call.",

[CLN logs, first run, SwapIn without amount]:
2022-09-26T23:08:50.855Z INFO plugin-peerswap-plugin: Killing plugin: exited during normal operation

[CLN logs, second run, SwapIn without amount]:
2022-09-26T23:23:05.359Z DEBUG plugin-peerswap-plugin: reloading policy \u003cnil\u003e
2022-09-26T23:23:05.364Z INFO plugin-peerswap-plugin: Killing plugin: exited during normal operation

[CLN logs, third run, SwapOut without amount]:
2022-09-27T00:09:46.853Z DEBUG plugin-peerswap-plugin: [Messenger] From: 0391cd7ae2eca090c5f807d71ab600bea5590796b56b21b44de596f9be42b7fd96 got msgtype: a463 payload: {"version":0,"assets":["lbtc","btc"],"peer_allowed":true}
2022-09-27T00:09:46.854Z INFO plugin-peerswap-plugin: [Swap:96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1]: Recovering from state State_SwapOutSender_AwaitAgreement
2022-09-27T00:09:46.855Z DEBUG lightningd: Plugin peerswap-plugin returned from custommsg hook call
2022-09-27T00:09:46.857Z DEBUG plugin-peerswap-plugin: [FSM] event:id: 96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1, Event_ActionFailed on State_SwapOutSender_AwaitAgreement
2022-09-27T00:09:46.857Z INFO plugin-peerswap-plugin: plugin quitting, error: event rejected
2022-09-27T00:09:46.858Z INFO plugin-peerswap-plugin: Killing plugin: exited during normal operation


  1. This one is just a new observation. Did we intentionally design 5 different RPC methods (addpeer, addsuspeer, allowswaprequests, removepeer, removesuspeer) only to update policy fields? What happens if two RPC methods are called simultaneously?

@wtogami
Copy link
Contributor

wtogami commented Sep 27, 2022

  1. Swap states subscriptions: We should have an RPC command to subscribe to state update for any active swap like waitsendpay, waitanyinvoice, etc.

Do not expect subscriptions to happen (too many variants of interfaces and we can't prioritize this.) The UI should periodically poll or have a manual refresh button.

  1. List swap filter (swap-in, swap-out and cancelled swap), sort and pagination

Will not happen anytime soon due to priority. The amount of data is super tiny compared to how quickly other things like forwarding grow which are cases that need filtering.

  1. API for updating AcceptAllPeers and ReserveOnchainMsat: I understand that it might not be possible to update these crucial fields without CLN restart. However, I am adding it for the sake of completeness. There is no API to update 'AcceptAllPeers' or 'ReserveOnchainMsat' fields of reloadPolicy.

ElementsProject/lightning#5294
Let's defer deciding what to do about this until after this CLN bug is fixed. Otherwise behavior too easily becomes inconsistent and confusing.

Node ids should also be returned with Aliases in various APIs like listSwaps, PeerAllowList, SuspiciousPeerList, etc.

It sounds nice to have. Please file in a new issue?

  1. This one is just a new observation. Did we intentionally designed 5 different RPC methods (addpeer, addsuspeer, allowswaprequests, removepeer, removesuspeer) only to update policy fields? What happens if two RPC methods are called simultaneously?

@nepet that reminds me allowswaprequests or addsuspeer should only prevent NEW swaps, but if it is already happening when swaps become disallowed it is allowed to complete or fail?

If this is the case then I'm not concerned about the race condition potential of these commands because they are used almost exclusively by humans and have only an indirect effect on whether swaps can start or not.

@nepet
Copy link
Contributor Author

nepet commented Sep 27, 2022

@wtogami You are correct, allowswaprequests and addsuspeer only prevent new swaps. Also, we use a mutex to prevent concurrent writing to the policy.

@nepet
Copy link
Contributor Author

nepet commented Sep 29, 2022

2. resendMsg response: Successful resendMsg doesn't return any value which can be confusing. A string/boolean response will be more useful.

[Shahana]: Throws error now: { "error": { "type": "lightning", "name": "LightningError", "message": "swap.NextMessage is nil", "code": -1, "fullType": "lightning" } }

3. Response case styling: Most of the calls are returning response fields in snake case which is core lightning standard as well. But reloadpolicy, addpeer, and removepeer response fields are in pascal case.

[Shahana]: Mostly fixed except error object's field fullType.

@ShahanaFarooqui fyi the fullType is produced by rtl's c-lightning-REST

https://github.com/Ride-The-Lightning/c-lightning-REST/blob/487d5ffa4448faebb4d0b614cb9224f6325b74c0/controllers/payments.js#L113-L115

This was a command for debugging purposes but somehow made it to
the public api.
When writing data to the store fails, we return a failure. If this
happens on swap creation we want the swap to be finished so that
we can start a new swap
The enums role and type have been returned as int values. For
a better human readability we now return them as string
@nepet
Copy link
Contributor Author

nepet commented Sep 29, 2022

``

13. Error message when the amount is missing: The RPC command does not throw any error if the amount is missing in swap-out and swap-in requests.

[Shahana]: Error from RPC call "Plugin terminated before replying to RPC call.",

I also observed this weirdness when ever I call peerswap too soon on startup. I will figure out how we can handle this. -> #137

@nepet
Copy link
Contributor Author

nepet commented Oct 4, 2022

The bug of #137 is resolved in #138

@nepet nepet marked this pull request as ready for review October 4, 2022 20:28
@ShahanaFarooqui
Copy link

  • Swap states subscriptions & list swap filter (swap-in, swap-out and cancelled swap), sort and pagination
    [Shahana]: These will be introduced in future releases as per Warren and Peter :).

  • No API to update 'AcceptAllPeers' or 'ReserveOnchainMsat'.
    [Shahana]: Warren suggested to defer until CLN bug is fixed.

  • listswaprequests: I am unable to test it yet. I tried by adding peer into suspicious list/peer not running peerswap/setting allowswaprequests to false/etc., but I am always getting empty array (CLN) or empty object (LND) in response.

  • Reload policy (LND): Missing "accept_all_peers" and "reserve_onchain_msat" fields.

  • Add/remove peers: CLN's reload policy responds with [] if allowed/suspicious peers are not available. However, LND's reload policy doesn't return these keys at all (allowlisted_peers & suspicious_peers will be missing)

  • Allow requests: LND's allowrequests as false removes allow_new_swaps field from the response. It should rather be returned as "allow_new_swaps": false.

  • created_at & amount (LND): Fields should be numeric.

  • Short channel ID format (LND): For LND users short channel id should look like "119107897613915137" but swaps lists are returning them as "108228:3:0"/"104770x2x1".

  • Context Canceled when LND stops before stopping peerswapd: This is an inconsistent error. It randomly occurs when LND stops and peerswapd's grpc port closes but rest does not (lsof -i | grep LISTEN). Terminal shows below message:

2022/10/05 18:11:24 [DEBUG] [RedundantSender]	SendMessageWithRetry: rpc error: code = Canceled desc = context canceled

Peerswapd is mostly terminating successfully with below message and needs manual restart after LND's restart.

2022/10/05 18:17:23 [INFO] [PaymentWatcher] Stop called, cancel context and wait for subscriptions to close
  • Peerswap plugin is killed on startup for my second node with below error (CLN): Unable to start peerswap on this node anymore.
2022-10-05T17:16:56.975Z DEBUG   plugin-peerswap-plugin: poll_service: could not send request_poll msg: -32600:Peer is not connected
2022-10-05T17:16:56.975Z DEBUG   0391cd7ae2eca090c5f807d71ab600bea5590796b56b21b44de596f9be42b7fd96-connectd: peer_out INVALID 42085
2022-10-05T17:16:56.979Z INFO    plugin-peerswap-plugin: [Swap:96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1]: Recovering from state State_SwapOutSender_AwaitAgreement
2022-10-05T17:16:56.980Z DEBUG   plugin-peerswap-plugin: [FSM] event:id: 96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1, Event_ActionFailed on State_SwapOutSender_AwaitAgreement
2022-10-05T17:16:56.980Z INFO    plugin-peerswap-plugin: plugin quitting, error: event rejected
2022-10-05T17:16:56.981Z INFO    plugin-peerswap-plugin: Killing plugin: exited during normal operation
  • Error message when the 'amt_sat' is undefined (CLN): The RPC command does not throw any error if the amount is missing in swap-out and swap-in requests. Error from RPC call, peerswap plugin killed.
2022-10-05T18:36:21.266Z DEBUG   lightningd: Plugin peerswap-plugin returned from custommsg hook call
2022-10-05T18:36:31.261Z DEBUG   034e317df18aef6c1d8f468b051d10653f36eaaaf514f0b961a7a4d91e9a54c124-connectd: peer_out INVALID 42077
2022-10-05T18:36:33.548Z INFO    plugin-peerswap-plugin: Killing plugin: exited during normal operation
2022-10-05T18:36:33.549Z UNUSUAL plugin-clrest.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Plugin terminated before replying to RPC call.\",\"code\":-4,\"fullType\":\"lightning\"}

@nepet
Copy link
Contributor Author

nepet commented Oct 6, 2022

Thanks for your review.

* Reload policy (LND): Missing "accept_all_peers" and "reserve_onchain_msat" fields.

* Add/remove peers: CLN's reload policy responds with `[]` if allowed/suspicious peers are not available. However, LND's reload policy doesn't return these keys at all (`allowlisted_peers` & `suspicious_peers` will be missing)

* Allow requests: LND's `allowrequests` as false removes `allow_new_swaps` field from the response. It should rather be returned as `"allow_new_swaps": false`.

Fixed 86628e4

* `created_at` & `amount` (LND): Fields should be numeric.

The internal representation of these fields is of type uint64. It is convenient to represent a 64bit type as string rather than a number as javascripts number is only safe to use up to Number.MAX_SAVE_INTEGER is 2^53-1, larger values require a BigInt.

If it is okay for you to parse the string to a number I would not change anything about this, otherwise we would need to parse every struct that contains uint64 fields to a copy that uses uint32 before we marshal the json.

* Short channel ID format (LND): For LND users short channel id should look like "119107897613915137" but swaps lists are returning them as "108228:3:0"/"104770x2x1".

short_channel_id is the short channel id as defined in Bolt#7 so the return values are expected. It might have been a bad decision to use scids at all instead of normal channel ids, but a change now would need a protocol update. This is out of scope for now but will change eventually.

* Context Canceled when LND stops before stopping peerswapd: This is an inconsistent error. It randomly occurs when LND stops and peerswapd's grpc port closes but rest does not (lsof -i | grep LISTEN). Terminal shows below message:
2022/10/05 18:11:24 [DEBUG] [RedundantSender]	SendMessageWithRetry: rpc error: code = Canceled desc = context canceled

Peerswapd is mostly terminating successfully with below message and needs manual restart after LND's restart.

2022/10/05 18:17:23 [INFO] [PaymentWatcher] Stop called, cancel context and wait for subscriptions to close
* Peerswap plugin is killed on startup for my second node with below error (CLN): Unable to start peerswap on this node anymore.
2022-10-05T17:16:56.975Z DEBUG   plugin-peerswap-plugin: poll_service: could not send request_poll msg: -32600:Peer is not connected
2022-10-05T17:16:56.975Z DEBUG   0391cd7ae2eca090c5f807d71ab600bea5590796b56b21b44de596f9be42b7fd96-connectd: peer_out INVALID 42085
2022-10-05T17:16:56.979Z INFO    plugin-peerswap-plugin: [Swap:96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1]: Recovering from state State_SwapOutSender_AwaitAgreement
2022-10-05T17:16:56.980Z DEBUG   plugin-peerswap-plugin: [FSM] event:id: 96500ce1c7824710450a325ee0e9894d619ecfe0d666abe8300e8ef53183aea1, Event_ActionFailed on State_SwapOutSender_AwaitAgreement
2022-10-05T17:16:56.980Z INFO    plugin-peerswap-plugin: plugin quitting, error: event rejected
2022-10-05T17:16:56.981Z INFO    plugin-peerswap-plugin: Killing plugin: exited during normal operation

These should not be related to this PR, could you file issues for this and provide more information?

* Error message when the 'amt_sat' is undefined (CLN): The RPC command does not throw any error if the amount is missing in swap-out and swap-in requests. Error from RPC call, peerswap plugin killed.
2022-10-05T18:36:21.266Z DEBUG   lightningd: Plugin peerswap-plugin returned from custommsg hook call
2022-10-05T18:36:31.261Z DEBUG   034e317df18aef6c1d8f468b051d10653f36eaaaf514f0b961a7a4d91e9a54c124-connectd: peer_out INVALID 42077
2022-10-05T18:36:33.548Z INFO    plugin-peerswap-plugin: Killing plugin: exited during normal operation
2022-10-05T18:36:33.549Z UNUSUAL plugin-clrest.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Plugin terminated before replying to RPC call.\",\"code\":-4,\"fullType\":\"lightning\"}

As said before, this is an issue that arises when peerswap is called too early after startup (whil still waiting for lnd to sync) and is part of #137

Except for some minor issues that we wont address right now I'd say we are good to go and can build on top of this pr.

@ShahanaFarooqui
Copy link

ShahanaFarooqui commented Oct 6, 2022

If it is okay for you to parse the string to a number I would not change anything about this, otherwise we would need to parse every struct that contains uint64 fields to a copy that uses uint32 before we marshal the json.

We already format amounts & dates, so converting it into number is not an issue for RTL.

short_channel_id is the short channel id as defined in Bolt#7 so the return values are expected. It might have been a bad decision to use scids at all instead of normal channel ids, but a change now would need a protocol update. This is out of scope for now but will change eventually.

This can be an issue for the UI. If LND's short channel Id is "119134283892916224" and Peerswapd is sending it as "108352:2:0", how will the UI map them together?

These should not be related to this PR, could you file issues for this and provide more information?

Created an issue here. I tried to provide as much information as I can but logs are not showing much details.

As said before, this is an issue that arises when peerswap is called too early after startup (whil still waiting for lnd to sync) and is part of #137

This issue is persistent and happening well after cln is synced, policy loaded and peerswap initialized etc. I tested it with commit id 81ea6ec9cab385ce050cbf4f966d61c861974b26.

@wtogami
Copy link
Contributor

wtogami commented Oct 6, 2022

ACK, merging #131 as this is holding up fixing #137 among others.

@wtogami wtogami merged commit c99c266 into ElementsProject:master Oct 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

All timestamps should be stored as UTC? Add chain in listswaps "addpeer" command works with blank string
3 participants