Merge branch 'develop' into fixPayChanV1

.github/pull_request_template.md

@@ -33,10 +33,27 @@ Please check [x] relevant options, delete irrelevant ones.
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
 - [ ] Refactor (non-breaking change that only restructures code)
-- [ ] Tests (You added tests for code that already exists, or your new feature included in this PR)
-- [ ] Documentation Updates
+- [ ] Tests (you added tests for code that already exists, or your new feature included in this PR)
+- [ ] Documentation update
+- [ ] Chore (no impact to binary, e.g. `.gitignore`, formatting, dropping support for older tooling)
 - [ ] Release
 
+### API Impact
+
+<!--
+Please check [x] relevant options, delete irrelevant ones.
+
+* If there is any impact to the public API methods (HTTP / WebSocket), please update https://github.com/xrplf/rippled/blob/develop/API-CHANGELOG.md
+  * Update API-CHANGELOG.md and add the change directly in this PR by pushing to your PR branch.
+* libxrpl: See https://github.com/XRPLF/rippled/blob/develop/docs/build/depend.md
+* Peer Protocol: See https://xrpl.org/peer-protocol.html
+-->
+
+- [ ] Public API: New feature (new methods and/or new fields)
+- [ ] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
+- [ ] `libxrpl` change (any change that may affect `libxrpl` or dependents of `libxrpl`)
+- [ ] Peer protocol change (must be backward compatible or bump the peer protocol version)
+
 <!--
 ## Before / After
 If relevant, use this section for an English description of the change at a technical level.

.github/workflows/doxygen.yml

@@ -11,6 +11,8 @@ concurrency:
 jobs:
   job:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
     container:
       image: docker://rippleci/rippled-ci-builder:2944b78d22db
     steps:

.github/workflows/windows.yml

@@ -89,7 +89,7 @@ jobs:
           generator: '${{ matrix.generator }}'
           configuration: ${{ matrix.configuration }}
           # Hard code for now. Move to the matrix if varied options are needed
-          cmake-args: '-Dassert=ON -Dreporting=OFF -Dunity=OFF'
+          cmake-args: '-Dassert=ON -Dreporting=OFF -Dunity=ON'
       - name: test (permitted to silently fail)
         shell: bash
         # Github runners are resource limited, which causes unit tests to fail

API-CHANGELOG.md

@@ -26,6 +26,14 @@ In `api_version: 2`, the `signer_lists` field [will be moved](#modifications-to-
 
 The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it was not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode).
 
+## Unreleased
+
+### Additions
+
+Additions are intended to be non-breaking (because they are purely additive).
+
+- `server_definitions`: A new RPC that generates a `definitions.json`-like output that can be used in XRPL libraries.
+
 ## XRP Ledger version 1.12.0
 
 [Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023.
@@ -46,27 +54,27 @@ Additions are intended to be non-breaking (because they are purely additive).
     - `Account`: The issuer of the asset being clawed back. Must also be the sender of the transaction.
     - `Amount`: The amount being clawed back, with the `Amount.issuer` being the token holder's address.
 - Adds [AMM](https://github.com/XRPLF/XRPL-Standards/discussions/78) ([#4294](https://github.com/XRPLF/rippled/pull/4294), [#4626](https://github.com/XRPLF/rippled/pull/4626)) feature:
-    - Adds `amm_info` API to retrieve AMM information for a given tokens pair.
-    - Adds `AMMCreate` transaction type to create `AMM` instance.
-    - Adds `AMMDeposit` transaction type to deposit funds into `AMM` instance.
-    - Adds `AMMWithdraw` transaction type to withdraw funds from `AMM` instance.
-    - Adds `AMMVote` transaction type to vote for the trading fee of `AMM` instance.
-    - Adds `AMMBid` transaction type to bid for the Auction Slot of `AMM` instance.
-    - Adds `AMMDelete` transaction type to delete `AMM` instance.
-    - Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
-    - Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
-    - Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
-      `tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination 
-      for `AMMDeposit` and `AMMWithdraw` transactions.
-    - Adds new transaction result codes:
-       - tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
-       -  tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
-       -  tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
-       -  tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
-       -  tecAMM_EMPTY: AMM is in empty state. Transaction expects AMM in non-empty state (LP tokens > 0).
-       -  tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction expects AMM in empty state (LP tokens == 0).
-       -  tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
-       -  tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
+  - Adds `amm_info` API to retrieve AMM information for a given tokens pair.
+  - Adds `AMMCreate` transaction type to create `AMM` instance.
+  - Adds `AMMDeposit` transaction type to deposit funds into `AMM` instance.
+  - Adds `AMMWithdraw` transaction type to withdraw funds from `AMM` instance.
+  - Adds `AMMVote` transaction type to vote for the trading fee of `AMM` instance.
+  - Adds `AMMBid` transaction type to bid for the Auction Slot of `AMM` instance.
+  - Adds `AMMDelete` transaction type to delete `AMM` instance.
+  - Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
+  - Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
+  - Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
+    `tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination
+    for `AMMDeposit` and `AMMWithdraw` transactions.
+  - Adds new transaction result codes:
+    - tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
+    - tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
+    - tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
+    - tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
+    - tecAMM_EMPTY: AMM is in empty state. Transaction expects AMM in non-empty state (LP tokens > 0).
+    - tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction expects AMM in empty state (LP tokens == 0).
+    - tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
+    - tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
 
 ## XRP Ledger version 1.11.0
 
@@ -118,13 +126,21 @@ Changes below this point are in development.
 At the time of writing, this version is expected to be introduced in `rippled` version 2.0.
 
 Currently (prior to the release of 2.0), it is available as a "beta" version, meaning it can be enabled with a config setting in `rippled.cfg`:
+
 ```
 [beta_rpc_api]
 1
 ```
 
 Since `api_version` 2 is in "beta", breaking changes to V2 can currently be made at any time.
 
+#### Removed methods
+
+In API version 2, the following methods are no longer available:
+
+- `tx_history` - Instead, use other methods such as `account_tx` or `ledger` with the `transactions` field set to `true`.
+- `ledger_header` - Instead, use the `ledger` method.
+
 #### Modifications to account_info response in V2
 
 - `signer_lists` is returned in the root of the response. Previously, in API version 1, it was nested under `account_data`. (https://github.com/XRPLF/rippled/pull/3770)
@@ -149,6 +165,15 @@ Since `api_version` 2 is in "beta", breaking changes to V2 can currently be made
 
 - Attempting to use a non-boolean value (such as a string) for the `transactions` parameter returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
 
+##### In progress
+
+- In `Payment` transaction type, JSON RPC field `Amount` is renamed to `DeliverMax`. To enable smooth client transition, `Amount` is still handled, as described below:
+  - On JSON RPC input (e.g. `submit_multisigned` etc. methods), `Amount` is recognized as an alias to `DeliverMax` for both API version 1 and version 2 clients.
+  - On JSON RPC input, submitting both `Amount` and `DeliverMax` fields is allowed _only_ if they are identical; otherwise such input is rejected with `rpcINVALID_PARAMS` error.
+  - On JSON RPC output (e.g. `subscribe`, `account_tx` etc. methods), `DeliverMax` is present in both API version 1 and version 2.
+  - On JSON RPC output, `Amount` is only present in API version 1 and _not_ in version 2.
+
+
 # Unit tests for API changes
 
 The following information is useful to developers contributing to this project:

Builds/CMake/RippledCore.cmake

@@ -217,6 +217,7 @@ install (
 install (
   FILES
     src/ripple/json/JsonPropertyStream.h
+    src/ripple/json/MultivarJson.h
     src/ripple/json/Object.h
     src/ripple/json/Output.h
     src/ripple/json/Writer.h
@@ -467,6 +468,7 @@ target_sources (rippled PRIVATE
   src/ripple/app/misc/detail/impl/WorkSSL.cpp
   src/ripple/app/misc/impl/AccountTxPaging.cpp
   src/ripple/app/misc/impl/AmendmentTable.cpp
+  src/ripple/app/misc/impl/DeliverMax.cpp
   src/ripple/app/misc/impl/LoadFeeTrack.cpp
   src/ripple/app/misc/impl/Manifest.cpp
   src/ripple/app/misc/impl/Transaction.cpp
@@ -519,6 +521,7 @@ target_sources (rippled PRIVATE
   src/ripple/app/tx/impl/CreateTicket.cpp
   src/ripple/app/tx/impl/DeleteAccount.cpp
   src/ripple/app/tx/impl/DepositPreauth.cpp
+  src/ripple/app/tx/impl/DID.cpp
   src/ripple/app/tx/impl/Escrow.cpp
   src/ripple/app/tx/impl/InvariantCheck.cpp
   src/ripple/app/tx/impl/NFTokenAcceptOffer.cpp
@@ -785,6 +788,7 @@ if (tests)
     src/test/app/DeliverMin_test.cpp
     src/test/app/DepositAuth_test.cpp
     src/test/app/Discrepancy_test.cpp
+    src/test/app/DID_test.cpp
     src/test/app/DNS_test.cpp
     src/test/app/Escrow_test.cpp
     src/test/app/FeeVote_test.cpp
@@ -916,6 +920,7 @@ if (tests)
     src/test/json/Output_test.cpp
     src/test/json/Writer_test.cpp
     src/test/json/json_value_test.cpp
+    src/test/json/MultivarJson_test.cpp
     #[===============================[
        test sources:
          subdir: jtx
@@ -937,6 +942,7 @@ if (tests)
     src/test/jtx/impl/check.cpp
     src/test/jtx/impl/delivermin.cpp
     src/test/jtx/impl/deposit.cpp
+    src/test/jtx/impl/did.cpp
     src/test/jtx/impl/envconfig.cpp
     src/test/jtx/impl/fee.cpp
     src/test/jtx/impl/flags.cpp
@@ -1056,6 +1062,7 @@ if (tests)
     src/test/rpc/KeyGeneration_test.cpp
     src/test/rpc/LedgerClosed_test.cpp
     src/test/rpc/LedgerData_test.cpp
+    src/test/rpc/LedgerHeader_test.cpp
     src/test/rpc/LedgerRPC_test.cpp
     src/test/rpc/LedgerRequestRPC_test.cpp
     src/test/rpc/ManifestRPC_test.cpp
@@ -1079,6 +1086,7 @@ if (tests)
     src/test/rpc/ValidatorInfo_test.cpp
     src/test/rpc/ValidatorRPC_test.cpp
     src/test/rpc/Version_test.cpp
+    src/test/rpc/Handler_test.cpp
     #[===============================[
        test sources:
          subdir: server

Builds/levelization/results/ordering.txt

@@ -135,6 +135,7 @@ test.csf > ripple.protocol
 test.csf > test.jtx
 test.json > ripple.beast
 test.json > ripple.json
+test.json > ripple.rpc
 test.json > test.jtx
 test.jtx > ripple.app
 test.jtx > ripple.basics

src/ripple/app/ledger/BookListeners.cpp

@@ -39,7 +39,7 @@ BookListeners::removeSubscriber(std::uint64_t seq)
 
 void
 BookListeners::publish(
-    Json::Value const& jvObj,
+    MultiApiJson const& jvObj,
     hash_set<std::uint64_t>& havePublished)
 {
     std::lock_guard sl(mLock);
@@ -54,7 +54,8 @@ BookListeners::publish(
             // Only publish jvObj if this is the first occurence
             if (havePublished.emplace(p->getSeq()).second)
             {
-                p->send(jvObj, true);
+                p->send(
+                    jvObj.select(apiVersionSelector(p->getApiVersion())), true);
             }
             ++it;
         }

src/ripple/app/ledger/BookListeners.h

@@ -20,7 +20,9 @@
 #ifndef RIPPLE_APP_LEDGER_BOOKLISTENERS_H_INCLUDED
 #define RIPPLE_APP_LEDGER_BOOKLISTENERS_H_INCLUDED
 
+#include <ripple/json/MultivarJson.h>
 #include <ripple/net/InfoSub.h>
+
 #include <memory>
 #include <mutex>
 
@@ -58,7 +60,7 @@ class BookListeners
 
     */
     void
-    publish(Json::Value const& jvObj, hash_set<std::uint64_t>& havePublished);
+    publish(MultiApiJson const& jvObj, hash_set<std::uint64_t>& havePublished);
 
 private:
     std::recursive_mutex mLock;

src/ripple/app/ledger/OrderBookDB.cpp

@@ -250,7 +250,7 @@ void
 OrderBookDB::processTxn(
     std::shared_ptr<ReadView const> const& ledger,
     const AcceptedLedgerTx& alTx,
-    Json::Value const& jvObj)
+    MultiApiJson const& jvObj)
 {
     std::lock_guard sl(mLock);
 

src/ripple/app/ledger/OrderBookDB.h

@@ -23,6 +23,8 @@
 #include <ripple/app/ledger/AcceptedLedgerTx.h>
 #include <ripple/app/ledger/BookListeners.h>
 #include <ripple/app/main/Application.h>
+#include <ripple/json/MultivarJson.h>
+
 #include <mutex>
 
 namespace ripple {
@@ -63,7 +65,7 @@ class OrderBookDB
     processTxn(
         std::shared_ptr<ReadView const> const& ledger,
         const AcceptedLedgerTx& alTx,
-        Json::Value const& jvObj);
+        MultiApiJson const& jvObj);
 
 private:
     Application& app_;

src/ripple/app/ledger/impl/LedgerToJson.cpp

@@ -19,6 +19,7 @@
 
 #include <ripple/app/ledger/LedgerToJson.h>
 #include <ripple/app/main/Application.h>
+#include <ripple/app/misc/DeliverMax.h>
 #include <ripple/app/misc/TxQ.h>
 #include <ripple/basics/base_uint.h>
 #include <ripple/core/Pg.h>
@@ -123,6 +124,7 @@ fillJsonTx(
     else
     {
         copyFrom(txJson, txn->getJson(JsonOptions::none));
+        RPC::insertDeliverMax(txJson, txnType, fill.context->apiVersion);
         if (stMeta)
         {
             txJson[jss::metaData] = stMeta->getJson(JsonOptions::none);

src/ripple/app/main/Main.cpp

@@ -169,6 +169,7 @@ printHelp(const po::options_description& desc)
            "     peer_reservations_list\n"
            "     ripple ...\n"
            "     ripple_path_find <json> [<ledger>]\n"
+           "     server_definitions [<hash>]\n"
            "     server_info [counters]\n"
            "     server_state [counters]\n"
            "     sign <private_key> <tx_json> [offline]\n"

src/ripple/app/misc/DeliverMax.h

@@ -0,0 +1,53 @@
+//------------------------------------------------------------------------------
+/*
+    This file is part of rippled: https://github.com/ripple/rippled
+    Copyright (c) 2023 Ripple Labs Inc.
+
+    Permission to use, copy, modify, and/or distribute this software for any
+    purpose  with  or without fee is hereby granted, provided that the above
+    copyright notice and this permission notice appear in all copies.
+
+    THE  SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+    WITH  REGARD  TO  THIS  SOFTWARE  INCLUDING  ALL  IMPLIED  WARRANTIES  OF
+    MERCHANTABILITY  AND  FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+    ANY  SPECIAL ,  DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+    WHATSOEVER  RESULTING  FROM  LOSS  OF USE, DATA OR PROFITS, WHETHER IN AN
+    ACTION  OF  CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+    OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+//==============================================================================
+
+#ifndef RIPPLE_APP_MISC_DELIVERMAX_H_INCLUDED
+#define RIPPLE_APP_MISC_DELIVERMAX_H_INCLUDED
+
+#include <ripple/protocol/TxFormats.h>
+
+#include <functional>
+#include <memory>
+
+namespace Json {
+class Value;
+}
+
+namespace ripple {
+
+namespace RPC {
+
+/**
+   Copy `Amount` field to `DeliverMax` field in transaction output JSON.
+   This only applies to Payment transaction type, all others are ignored.
+
+   When apiVersion > 1 will also remove `Amount` field, forcing users
+   to access this value using new `DeliverMax` field only.
+   @{
+ */
+
+void
+insertDeliverMax(Json::Value& tx_json, TxType txnType, unsigned int apiVersion);
+
+/** @} */
+
+}  // namespace RPC
+}  // namespace ripple
+
+#endif

src/ripple/app/misc/FeeEscalation.md

@@ -190,11 +190,27 @@ lower) fee to get into the same position as a reference transaction.
 
 ### Consensus Health
 
-For consensus to be considered healthy, the consensus process must take
-less than 5 seconds. This time limit was chosen based on observed past
-behavior of the network. Note that this is not necessarily the time between
-ledger closings, as consensus usually starts some amount of time after
-a ledger opens.
+For consensus to be considered healthy, the peers on the network
+should largely remain in sync with one another. It is particularly
+important for the validators to remain in sync, because that is required
+for participation in consensus. However, the network tolerates some
+validators being out of sync.  Fundamentally, network health is a
+function of validators reaching consensus on sets of recently submitted
+transactions.
+
+Another factor to consider is
+the duration of the consensus process itself. This generally takes
+under 5 seconds on the main network under low volume. This is based on
+historical observations. However factors such as transaction volume
+can increase consensus duration. This is because rippled performs
+more work as transaction volume increases. Under sufficient load this
+tends to increase consensus duration. It's possible that relatively high
+consensus duration indicates a problem, but it is not appropriate to
+conclude so without investigation. The upper limit for consensus
+duration should be roughly 20 seconds. That is far above the normal.
+If the network takes this long to close ledgers, then it is almost
+certain that there is a problem with the network. This circumstance
+often coincides with new ledgers with zero transactions.
 
 ### Other Constants