Merge branch 'master' of https://github.com/input-output-hk/plutus in…

…to effectfully/builtins/both-MajorProtocolVersion-and-multiple-CostModels

.github/workflows/haddock.yml

@@ -13,7 +13,7 @@ jobs:
       name: github-pages
     steps:
       - uses: actions/checkout@v4
-      - uses: nixbuild/nix-quick-install-action@v26
+      - uses: nixbuild/nix-quick-install-action@v27
         with:
           nix_conf: |
             experimental-features = nix-command flakes

doc/read-the-docs-site/howtos/asdata.rst

@@ -0,0 +1,128 @@
+.. highlight:: haskell
+.. _asdata:
+
+How to use ``AsData`` to optimize scripts
+=========================================
+
+The Plutus libraries contain a ``PlutusTx.asData`` module that contains Template Haskell (TH) code for encoding algebraic data types (ADTs) as ``Data`` objects in Plutus Core, as opposed to sums-of-products terms.
+In general, ``asData`` pushes the burden of a computation nearer to where a value is used, in a crude sense making the evaluation less strict and more lazy.
+This is intended for expert Plutus developers.
+
+Purpose
+-------
+
+Values stored in datums or redeemers need to be encoded into ``Data`` objects.
+When writing and optimizing a Plutus script, one of the challenges is finding the right approach to handling ``Data`` objects and how expensive that method will be.
+To make an informed decision, you may need to benchmark and profile your smart contract code to measure its actual resource consumption.
+The primary purpose of ``asData`` is to give you more options for how you want to handle ``Data``.
+
+Choice of two approoaches
+-------------------------
+
+When handling ``Data`` objects, you have a choice of two pathways.
+It is up to you to determine which pathway to use depending on your particular use case.
+There are trade offs in performance and where errors occur.
+
+Approach one: proactively do all of the parsing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first approach is to parse the object immediately (using ``fromBuiltinData``) into a native Plutus Core datatype, which will also identify any problems with the structuring of the object.
+However, this performs all the work up front.
+
+This is the normal style that has been promoted in the past.
+
+Approach two: only do the parsing if and when necessary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the second approach, the script doesn't do any parsing work immediately, and instead does it later, when it needs to.
+It might be that this saves you a lot of work, because you may never need to parse the entire object.
+Instead, the script will just carry the item around as a ``Data`` object.
+
+Using this method, every time the script uses the object, it will look at it to find out if it has the right shape.
+If it does have the right shape, it will deconstruct the ``Data`` object and do its processing; if not, it will throw an error.
+This work may be repeated depending on how your script is written.
+In some cases, you might do less work, in some cases you might do more work, depending on your specific use case.
+
+The Plutus Tx library provides some helper functions to make this second style easier to do, in the form of the ``asData`` function.
+
+Using ``asData``
+------------------
+
+The ``asData`` function takes the definition of a data type and replaces it with an equivalent definition whose representation uses ``Data`` directly.
+
+For example, if we wanted to use it on the types from the :ref:`auction example<simple_example>`, we would put the datatype declarations inside a Template Haskell quote and call ``asData`` on it.
+
+.. literalinclude:: ../tutorials/AuctionValidator.hs
+   :start-after: BLOCK9
+   :end-before: BLOCK10
+
+This is normal Template Haskell that just generates new Haskell source, so you can see the code that it generates with `{-# OPTIONS_GHC -ddump-splices #-}`, but it will look something like this:
+
+.. code-block::
+
+        PlutusTx.asData
+        [d| data Bid'
+                = Bid' {bBidder' :: PubKeyHash, bAmount' :: Lovelace}
+                deriving newtype (Eq, Ord, ToBuitinData, FromBuiltinData, UnsafeFromBuiltinData)
+            data AuctionRedeemer' = NewBid' Bid | Payout'
+                deriving newtype (Eq, Ord, ToBuitinData, FromBuiltinData, UnsafeFromBuiltinData) |]
+
+        ======>
+
+        newtype Bid' = Bid'2 BuiltinData
+        deriving newtype (Eq, Ord, PlutusTx.ToData, FromData, UnsafeFromData)
+
+        {-# COMPLETE Bid' #-}
+        pattern Bid' :: PubKeyHash -> Lovelace -> Bid'
+        pattern Bid' ...
+
+        newtype AuctionRedeemer' = AuctionRedeemer'2 BuiltinData
+        deriving newtype (Eq, Ord, PlutusTx.ToData, FromData, UnsafeFromData)
+
+        {-# COMPLETE NewBid', Payout' #-}
+        pattern NewBid' :: Bid -> AuctionRedeemer'
+        pattern NewBid' ...
+        pattern Payout' :: AuctionRedeemer'
+        pattern Payout' ...
+
+That is:
+
+- It creates a newtype wrapper around ``BuiltinData``
+- It creates pattern synonyms corresponding to each of the constructors you wrote
+
+This lets you write code "as if" you were using the original declaration that you wrote, while in fact the pattern synonyms are handling conversion to/from ``Data`` for you.
+But any values of this type actually are represented with ``Data``.
+That means that when we newtype-derive the instances for converting to and from ``Data`` we get the instances for ``BuiltinData`` - which are free!
+
+Nested fields
+~~~~~~~~~~~~~
+
+The most important caveat to using ``asData`` is that ``Data`` objects encoding datatypes must also encode the *fields* of the datatype as ``Data``.
+However, ``asData`` tries to make the generated code a drop-in replacement for the original code, which means that when using the pattern synonyms they try to give you the fields as they were originally defined, which means *not* encoded as ``Data``.
+
+For example, in the ``Bid`` case above the ``bAmount`` field is originally defined to have type ``Lovelace`` which is a newtype around a Plutus Core builtin integer.
+However, since we are using ``asData``, we need to encode the field into ``Data`` in order to store it.
+That means that when you construct a ``Bid`` object you must take the ``Integer`` that you start with and convert it to ``Data``, and when you pattern match on a ``Bid`` object you do the reverse conversion.
+
+These conversions are potentially expensive!
+If the ``bAmount`` field was a complex data structure, then every time we constructed or deconstructed a ``Bid`` object we would need to convert that datastructure to or from ``Data``.
+Whether or not this is a problem depends on the precise situation, but in general:
+
+- If the field is a builtin integer or bytestring or a wrapper around those, it is probably cheap
+- If the field is a datatype which is itself defined with ``asData`` then it is free (since it's already ``Data``!)
+- If the field is a complex or large datatype then it is potentially expensive
+
+Therefore ``asData`` tends to work best when you use it for a type and also for all the types of its fields.
+
+Choosing an approach
+--------------------
+
+There are a number of tradeoffs to consider:
+
+1. Plutus Tx's datatypes are faster to work with and easier to optimize than ``Data``, so if the resulting object is going to be processed in its entirety (or have parts of it repeatedly processed) then it can be better to parse it up-front.
+2. If it is important to check that the entire structure is well-formed, then it is better to parse it up-front, since the conversion will check the entire structure for well-formedness immediately, rather than checking only the parts that are used when they are used.
+3. If you do not want to use ``asData`` for the types of the fields, then it may be better to not use it at all in order to avoid conversion penalties at the use sites.
+
+Which approach is better is an empirical question and may vary in different cases.
+A single script may wish to use different approaches in different places.
+For example, your datum might contain a large state object which is usually only inspected in part (a good candidate for ``asData``), whereas your redeemer might be a small object which is inspected frequently to determine what to do (a good candidate for a native Plutus Tx datatype).

doc/read-the-docs-site/howtos/index.rst

@@ -7,5 +7,6 @@ How-to guides
    :maxdepth: 3
    :titlesonly:
 
+   asdata
    exporting-a-script
    profiling-scripts

doc/read-the-docs-site/reference/cardano/language-changes.rst

@@ -8,12 +8,12 @@ Language versions
 
 See the documentation on :ref:`language versions <what_are_plutus_language_versions>` for an explanation of what they are.
 
-Plutus V1
+PlutusV1
 ~~~~~~~~~~
 
 ``PlutusV1`` was the initial version of Plutus, introduced in the Alonzo hard fork.
 
-Plutus V2
+PlutusV2
 ~~~~~~~~~~
 
 ``PlutusV2`` was introduced in the Vasil hard fork.
@@ -29,7 +29,7 @@ The ``ScriptContext`` was extended to include the following information:
 Examples
 ------------
 
-- `Plutus V2 functionalities <https://github.com/IntersectMBO/cardano-node/blob/master/doc/reference/plutus/babbage-script-example.md>`_
+- `PlutusV2 functionalities <https://github.com/input-output-hk/cardano-node-wiki/blob/main/docs/reference/plutus/babbage-script-example.md>`_
 - `How to use reference inputs <https://github.com/perturbing/vasil-tests/blob/main/reference-inputs-cip-31.md>`_
 - `How to use inline datums <https://github.com/perturbing/vasil-tests/blob/main/inline-datums-cip-32.md>`_
 - `How to reference scripts <https://github.com/perturbing/vasil-tests/blob/main/referencing-scripts-cip-33.md>`_
@@ -53,6 +53,48 @@ Vasil
 
 All of the built-in types and functions from ``PlutusV1`` were added to ``PlutusV2``.
 
-The following built-in function was added to ``PlutusV2`` only (i.e., it is not available in ``PlutusV1``).
+The following built-in function was added to ``PlutusV2`` only (ie, it is not available in ``PlutusV1``).
 
 - ``serializeData`` (proposed in `CIP-42 <https://cips.cardano.org/cips/cip42/>`_)
+
+PlutusV3
+~~~~~~~~~
+
+Plutus and cryptography teams at IOG, in collaboration with `MLabs <https://mlabs.city/>`_, continue to develop Plutus capabilities. Starting with the release of `Cardano node v.8.8.0-pre <https://github.com/IntersectMBO/cardano-node/releases/tag/8.8.0-pre>`_, ``PlutusV3`` is available on `SanchoNet <https://sancho.network/>`_, introducing the Cardano community to governance features from `CIP-1694 <https://cips.cardano.org/cip/CIP-1694#goal>`_ in a controlled testnet environment. 
+
+``PlutusV3`` is the new ledger language that enhances Plutus Core's cryptographic capabilities, offering the following benefits for the smart contract developer community: 
+
+- Providing an updated script context that will let users see `CIP-1694 <https://cips.cardano.org/cip/CIP-1694#goal>`_  governance-related entities and voting features
+- Interoperability between blockchains
+- Advanced Plutus primitives
+- Well-known and optimal cryptographic algorithms
+- Support for porting of smart contracts from Ethereum
+- Creating sidechain bridges
+- Improving performance by adding a sums of products (SOPs) feature to support the direct encoding of differrent data types.
+
+Sums of products
+~~~~~~~~~~~~~~~~
+
+``PlutusV3`` introduces sums of products - a way of encoding data types that leads to smaller and cheaper scripts compared with `Scott encoding <https://en.wikipedia.org/wiki/Mogensen%E2%80%93Scott_encoding>`_, a common way of encoding data types in Plutus Core. 
+
+The sums of products approach aims to boost script efficiency and improve code generation for Plutus Core compilers. The changes involve new term constructors for packing fields into constructor values and efficient tag inspection for case branches, potentially running programs 30% faster. For an in-depth discussion, see `CIP-85 <https://cips.cardano.org/cip/CIP-0085>`_. 
+
+New cryptographic primitives
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``PlutusV3`` provides new built-in primitives that expand the language's capabilities. 
+
+- **BLS12-381**: A curve pairing that includes 17 primitives that support cryptographic curves. This is a benefit to sidechain specification implementation and `Mithril <https://iohk.io/en/blog/posts/2023/07/20/mithril-nears-mainnet-release/>`_ integration. 
+- **Blake2b-224**: A cryptographic hash function for on-chain computation of public-key hashes for the validation of transaction signatures. Supports community projects and contributes to Cardano's versatility. 
+- **Keccak-256**: A cryptographic hash function that produces a 256-bit (32-byte) hash value, commonly used for secure data verification. Supports Ethereum signature verification within scripts and cross-chain solutions. 
+
+Bitwise primitives
+~~~~~~~~~~~~~~~~~~~
+
+PlutusV3 initially brings several new bitwise primitives (with more to come at later stages). The introduction of `CIP-58 <https://cips.cardano.org/cip/CIP-0058>`_ bitwise primitives will enable the following features: 
+
+- Very low-level bit manipulations within Plutus, supporting the ability to execute high-performance data manipulation operations. 
+- Supporting the implementation of secure and robust cryptographic algorithms within Plutus. 
+- Facilitating standard, high-performance implementations for conversions between integers and bytestrings. 
+
+``PlutusV3`` adds two bitwise primitives: ``integerToByteString`` and ``byteStringToInteger``. The remaining primitives will be added to ``PlutusV3`` gradually and will not require a new ledger language. 

doc/read-the-docs-site/tutorials/AuctionValidator.hs

@@ -1,7 +1,9 @@
 {-# LANGUAGE DataKinds             #-}
+{-# LANGUAGE DerivingStrategies    #-}
 {-# LANGUAGE ImportQualifiedPost   #-}
 {-# LANGUAGE MultiParamTypeClasses #-}
 {-# LANGUAGE OverloadedStrings     #-}
+{-# LANGUAGE PatternSynonyms       #-}
 {-# LANGUAGE ScopedTypeVariables   #-}
 {-# LANGUAGE Strict                #-}
 {-# LANGUAGE TemplateHaskell       #-}
@@ -28,6 +30,7 @@ import PlutusLedgerApi.V2 (Datum (..), OutputDatum (..), ScriptContext (..), TxI
                            TxOut (..), from, to)
 import PlutusLedgerApi.V2.Contexts (getContinuingOutputs)
 import PlutusTx
+import PlutusTx.AsData qualified as PlutusTx
 import PlutusTx.Prelude qualified as PlutusTx
 import PlutusTx.Show qualified as PlutusTx
 
@@ -206,3 +209,24 @@ auctionValidatorScript params =
   $$(PlutusTx.compile [||auctionUntypedValidator||])
     `PlutusTx.unsafeApplyCode` PlutusTx.liftCode plcVersion100 params
 -- BLOCK9
+PlutusTx.asData [d|
+  data Bid' = Bid'
+    { bBidder' :: PubKeyHash,
+      -- ^ Bidder's wallet address.
+      bAmount' :: Lovelace
+      -- ^ Bid amount in Lovelace.
+    }
+    -- We can derive instances with the newtype strategy, and they
+    -- will be based on the instances for 'Data'
+    deriving newtype (Eq, Ord, PlutusTx.ToData, FromData, UnsafeFromData)
+
+  -- don't do this for the datum, since it's just a newtype so
+  -- simply delegates to the underlying type
+
+  -- | Redeemer is the input that changes the state of a smart contract.
+  -- In this case it is either a new bid, or a request to close the auction
+  -- and pay out the seller and the highest bidder.
+  data AuctionRedeemer' = NewBid' Bid | Payout'
+    deriving newtype (Eq, Ord, PlutusTx.ToData, FromData, UnsafeFromData)
+  |]
+-- BLOCK10

plutus-benchmark/bls12-381-costs/test/9.6/bls12-381-costs.golden

@@ -103,7 +103,7 @@ VRF example
 
     n     Script size             CPU usage               Memory usage
   ----------------------------------------------------------------------
-    -     715   (4.4%)      1303368563  (13.0%)           49449   (0.4%) 
+    -     712   (4.3%)      1303299563  (13.0%)           49149   (0.4%) 
 
 G1 Verify
 

plutus-benchmark/marlowe/src/PlutusBenchmark/Marlowe/Core/V1/Semantics.hs

@@ -328,12 +328,12 @@ refundOne accounts = case Map.toList accounts of
     -- Isabelle semantics in that it returns the least-recently
     -- added account-token combination rather than the first
     -- lexicographically ordered one. Also, the sequence
-    -- `Map.fromList . tail . Map.toList` preserves the
+    -- `Map.unsafeFromList . tail . Map.toList` preserves the
     -- invariants of order and non-duplication.
     ((accId, token), balance) : rest ->
         if balance > 0
-        then Just ((accId, token, balance), Map.fromList rest)
-        else refundOne (Map.fromList rest)
+        then Just ((accId, token, balance), Map.unsafeFromList rest)
+        else refundOne (Map.unsafeFromList rest)
 
 
 -- | Obtains the amount of money available an account.

plutus-benchmark/marlowe/test/semantics/9.6/0000020002010200020101020201000100010001020101020201010000020102.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 401412988
-| mem: 1451263})
+({cpu: 400791988
+| mem: 1448563})

plutus-benchmark/marlowe/test/semantics/9.6/0001000101000000010101000001000001010101010100000001000001010000.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 519999134
-| mem: 1821564})
+({cpu: 519677134
+| mem: 1820164})

plutus-benchmark/marlowe/test/semantics/9.6/0003040402030103010203030303000200000104030002040304020400000102.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1368245687
-| mem: 5135402})
+({cpu: 1367463687
+| mem: 5132002})

plutus-benchmark/marlowe/test/semantics/9.6/004025fd712d6c325ffa12c16d157064192992faf62e0b991d7310a2f91666b8.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 971935150
-| mem: 3459445})
+({cpu: 970348150
+| mem: 3452545})

plutus-benchmark/marlowe/test/semantics/9.6/0101010001010101010101000100010100000001010000010001000001000101.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 842825112
-| mem: 2477998})
+({cpu: 842043112
+| mem: 2474598})

plutus-benchmark/marlowe/test/semantics/9.6/0101020201010201010200010102000201000201010102000102010201010000.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 376467639
-| mem: 1373721})
+({cpu: 376191639
+| mem: 1372521})

plutus-benchmark/marlowe/test/semantics/9.6/0101080808040600020306010000000302050807010208060100070207080202.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 997974739
-| mem: 3653124})
+({cpu: 997652739
+| mem: 3651724})

plutus-benchmark/marlowe/test/semantics/9.6/0104010200020000040103020102020004040300030304040400010301040303.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 985076123
-| mem: 3606481})
+({cpu: 984570123
+| mem: 3604281})

plutus-benchmark/marlowe/test/semantics/9.6/04000f0b04051006000e060f09080d0b090d0104050a0b0f0506070f0a070008.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 913587161
-| mem: 3290999})
+({cpu: 912690161
+| mem: 3287099})

plutus-benchmark/marlowe/test/semantics/9.6/0543a00ba1f63076c1db6bf94c6ff13ae7d266dd7544678743890b0e8e1add63.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1335886976
-| mem: 4623885})
+({cpu: 1333655976
+| mem: 4614185})

plutus-benchmark/marlowe/test/semantics/9.6/0705030002040601010206030604080208020207000101060706050502040301.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1410834502
-| mem: 4711774})
+({cpu: 1407637502
+| mem: 4697874})

plutus-benchmark/marlowe/test/semantics/9.6/07070c070510030509010e050d00040907050e0a0d06030f1006030701020607.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1328135567
-| mem: 4842543})
+({cpu: 1326939567
+| mem: 4837343})

plutus-benchmark/marlowe/test/semantics/9.6/0bcfd9487614104ec48de2ea0b2c0979866a95115748c026f9ec129384c262c4.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1461455571
-| mem: 5309691})
+({cpu: 1460719571
+| mem: 5306491})

plutus-benchmark/marlowe/test/semantics/9.6/0be82588e4e4bf2ef428d2f44b7687bbb703031d8de696d90ec789e70d6bc1d8.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 1773166290
-| mem: 6446866})
+({cpu: 1771924290
+| mem: 6441466})

plutus-benchmark/marlowe/test/semantics/9.6/0f1d0110001b121d051e15140c0c05141d151c1f1d201c040f10091b020a0e1a.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 610770250
-| mem: 2243067})
+({cpu: 610494250
+| mem: 2241867})

plutus-benchmark/marlowe/test/semantics/9.6/119fbea4164e2bf21d2b53aa6c2c4e79414fe55e4096f5ce2e804735a7fbaf91.budget.golden

@@ -1,2 +1,2 @@
-({cpu: 937067950
-| mem: 3383163})
+({cpu: 935480950
+| mem: 3376263})