Add new interim governance commands: {create, answer, verify}-poll

[KtorZ]

The Cardano Foundation proposes a mechanism for polling Cardano stake pool operators on specific topics. Polls are done on-chain through transaction metadata and authenticated through stake pool credentials (either VRF public key similar to what's described in CIP-0022 or Ed25519 cold key).

The goal is to gather opinions on governance matters such as protocol parameters updates. This standard is meant to be an inclusive interim solution while the work on a larger governance framework such as CIP-1694 continues.

See proposed CIP for details.

---

This commits adds three new commands:

• create-poll: For the current governing entities, as a means to create new polls.

• answer-poll: For participants who want to answer a given poll.

• verify-poll: For anyone who seek to verify a poll entry (e.g. explorers)

The commands are built to fit and play nicely within the cardano-cli. The poll and answers structures are based on transaction metadata and require to be embedded in an actual transaction. The added commands however only works from metadata and raw "GovernancePoll" envelopes.

---

Tutorial

Pre-requisites

Here's a quick tutorial on how to interactively submit an answer to an SPO
survey. To start, you must have received a valid survey in the form of a JSON
files properly constructed using the create-poll command. The file should
look roughly like the following:

poll.json

{
    "type": "GovernancePoll",
    "description": "An on-chain poll for SPOs: Pineapples on pizza?",
    "cborHex": "a1185ea2007450696e656170706c6573206f6e2070697a7a613f018263796573626e6f"
}

Warning

Surveys will be make 'official' using a signature from the genesis delegate
key; the signature is however not present in metadata but used as extra
signatory on the originating transaction.

Creating answer

From there, you can create a metadata entry to answer the survey using the
governance answer-poll command as such:

$ cardano-cli governance answer-poll --poll-file poll.json --signing-key-file key.secret

Note that:

1. The key specified as --signing-key-file can/must be either a VRF secret
   key or a stake pool cold private key (Ed25519) in the JSON/Text-Envelope
   usual format expected by the cardano-cli. Here are an example of each:

   VRF key

   {
       "type": "VrfSigningKey_PraosVRF",
       "description": "VRF Signing Key",
       "cborHex": "5840b23fa897c1fc869d081e4818ea0ac533c1efaccb888cb57d8a40f6582783045d2dc2fa217af8b52251c4cdf538fa106cbf0b5beac3e74d05f97ceb33c0147a2c"
   }

   Cold key

   {
       "type": "StakePoolSigningKey_ed25519",
       "description": "Stake Pool Operator Signing Key",
       "cborHex": "58201d298ffa1544da0a5b2ea544728fc1ba7d2ae7c60e1d37da03895019740dd00a"
   }

2. This command will prompt you to answer interactively. If you do not wish to
   be prompt interactively, you can use --answer with the index of the
   answer.

Running this command will display the survey in a human-readable form, and
prompt you for an answer, as shown below:

An on-chain poll for SPOs: Pineapples on pizza?
[0] Yes
[1] No

Please indicate an answer (by index): _

You can proceed by typing one of the possible answer index (here, 0 or 1)
and a newline. This will print witnessed metadata in the form of a JSON
detailed schema that shall then be posted on-chain in any transaction: the
easiest being to build a simple transaction to yourself carrying the metadata.

Here's an example of metadata using a VRF key and choosing the answer 0:

answer.json

{
  "94": {
    "map": [
      {
        "k": { "int": 2 },
        "v": { "bytes": "c9a077a1098d73498d17e9ea27045af820c311ced91f8c2bb9b5c7f446379063" }
      },
      {
        "k": { "int": 3 },
        "v": { "int": 0 }
      },
      {
        "k": { "int": 4 },
        "v": {
          "list": [
            { "bytes": "2dc2fa217af8b52251c4cdf538fa106cbf0b5beac3e74d05f97ceb33c0147a2c" },
            {
              "list": [
                { "bytes": "3606afd8d437a69a0c94ebff7fe57638bdb54bd03193c3c830808187a5d740222ab26fa3f1b7df8b22e846196dda1ee16d4d6e64c062129e07273ec36c06f453" },
                { "bytes": "06c46025c18deaa04aa45913e1c91906" }
              ]
            }
          ]
        }
      }
    ]
  }
}

Publishing answer

From there, you can use the transaction build command to create a transaction
to post on-chain. You'll need a signing key associated to a UTxO with enough
funds to carry the transaction (~0.2 Ada should you make a basic transaction to
yourself).

Assuming you have saved metadata produced from the previous step in a file
called answer.json, the command for building the transaction shall look like:

$ cardano-cli transaction build \
    --babbage-era \
    --cardano-mode \
    --mainnet \
    --tx-in ...#... \
    --change-address addr1... \
    --metadata-json-file answer.json \
    --json-metadata-detailed-schema \
    --out-file answer.tx

You'll need to fill-in --tx-in & --change-address with their corresponding
values. From there, you can sign answer.tx and submit the result as usual. If
everything goes well, the cardano-cli should display a transaction id that you
can track on-chain to ensure your reply to the survey was properly published.

Verifying Answers

Finally, it's possible to verify answers seen on-chain using the governance verify-poll command. What 'verify' means here is two-folds:

• It checks that an answer is valid within the context of a given survey
• It controls the witness (i.e. proof or signature) provided with it

Assuming you still have the original poll.json file, and some witnessed
metadata (as provided by the governance answer-poll command, and as found in
transactions on-chain) as answer.json, you can verify its validity via:

$ cardano-cli governance verify-poll --poll-file poll.json --metadata-file answer.json

On success, this should outputs:

Ok.

Otherwise, the command will formulate an issue with the answer and/or poll.

[newhoggy]

I'm wondering if instead of verb-object we should instead do object verb like most of the rest of the CLI.

For example poll create instead of create-poll?

[KtorZ]

@newhoggy I actually tried to keep it consistent with other cli commands already 😅 ...

• transaction build-raw
• transaction calculate-min-fee
• transaction hash-script-data
• governance create-mir-certificate
• governance create-genesis-key-delegation-certificate
• governance create-update-proposal
• address key-gen
• address build-script

etc...

Not saying I am big fan, overall I do agree with you; but I value consistency more than my own preferences and it seems that this current PR is a better fit with the current state.

[newhoggy]

As there is no such test, can you add a comment to the PR showing example interactive session that goes from start to finish and includes on-chain interaction as well?

[KtorZ]

@newhoggy added a quick tutorial as PR description; it should help people get through the steps :)

[newhoggy]

Nice! Can you finish off with a section on verify-poll as well?

[Jimbo4350]

@KtorZ Please break this PR up into smaller commits, starting with the API changes and then propagating those changes to the cli

[KtorZ]

@Jimbo4350 what's the added value? It's take-all-or-nothing situation. There's no point having the API extension without the CLI commands. So it's one big atomic change.

It's not that I can't, but if I can avoid spending half an hour breaking into smaller commits that I'll probably have to amend individually because there are further discussions on the CIP. It is a lot easier to manage as a single commit; especially for conflict resolutions. So unless there's really a strong argument to "break it down", I won't do it.

[Jimbo4350]

@Jimbo4350 what's the added value? It's take-all-or-nothing situation. There's no point having the API extension without the CLI commands. So it's one big atomic change.

It's not that I can't, but if I can avoid spending half an hour breaking into smaller commits that I'll probably have to amend individually because there are further discussions on the CIP. It is a lot easier to manage as a single commit; especially for conflict resolutions. So unless there's really a strong argument to "break it down", I won't do it.

Breaking it up in to manageable commits (with good commit messages) makes it easier for reviewers (especially new hires) to follow the changes.

smaller commits that I'll probably have to amend individually because there are further discussions on the CIP

You don't need to do an interactive rebase and amend the commits in this case, you can just append commits e.g "Review feedback ..."

[KtorZ]

@Jimbo4350 done.

As a side-note:

You can just append commits e.g "Review feedback ..."

I would strongly advise against doing that. This is a nightmare for people coming after that try to make sense of how changes were introduced. Instead, amend commits and make it one atomic changes. I am not fond of over splitting commits into small de-correlated chunks for the same reasons. When features are introduced in one block, it often is the one block that makes sense; because some changes in module X may only be truly justified because of another change in module Y. So looking at them as two separate changes removes context -- and it's often all about context.

If that's the preferred way to manage changes on this repository; so be it. I've made a split that hopefully makes sense and makes the review easier. I'll amend commit and force-push the branch if there's any edits.

---

• 📍 Expose bytes and text limit constraints on metadata value.
  Useful to have accessible from external modules that need to construct metadata values.

• 📍 Add 'meta{Bytes,Text}chunks helper smart-constructors for TxMetadataValue
  It is quite common to need to construct long text or bytestring and, it is annoying to have to handle that at call-site every single time. Instead, we can provide smart constructors that takes care of splitting the text or byte string into reasonably-sized chunks. Note that we could also implement a version of those functions that is more flexible and only constructs chunks when needed; otherwise returning a plain MetaText or MetaBytes when they fit.

  For example, in CDDL, we would represent such a text string as:

  arbitrary_text =
    text .size (0..64) / [ * text .size (0..64) ]
  

  For the sake of keeping things simple however, those functions only implement the list variation.

• 📍 Define new Governance.Poll types and high-level interface
  This module is really meant to be driven by the cardano-cli or any client implementation that seeks to (re-)implement the SPO on-chain poll functionality.

• 📍 Implement (de)serialization methods for GovernancePoll objects
  This commit also introduces a new type-class 'AsTxMetadata' to hint to the fact that the chosen representation on-the-wire for those various types is a transaction metadata value. The serialization to CBOR becomes then straightforward once we've converted the type into a 'MetadataValue'. Similarly, the deserialization is made simpler by first deserializing an opaque 'MetadataValue', and then inspecting it to see if it has the expected shape.

• 📍 Define commands behavior for {create,answer,verify}-poll
  These commands are pretty straightforward to write by leveraging the newly introduced Cardano.Api.Governance.Poll API.
  One may ask why use sometimes files, sometimes stderr and sometimes stdout in implementing those commands.

  As a rule of thumb:

  • stdout = relevant content that should be structured to be piped into other tools or send to files
  • stderr = debug information useful to communicate context and details to users

  A command-line that outputs interactive debug information on stdout is arguably doing something wrong; unless it's the main terminal output of the application (e.g. printing structured logs on stdout).

  Then, why stdout rather than a file? Because I find the UX a lot better that way. Command lines with too many params are arguably hard to process; and using file as the medium of exchanges makes it harder / prevent piping into other tools easily. When printing structured results on stdout, one can always redirect the output to a file should they want it; so IMO stdout should always be the default; and files used only when necessary.

  Here I am only using an output file in the case of create-poll as a "build artifact". It allows to produce two outputs with distinct purpose; the file is meant to be shared as file, and thus it makes sense to treat it as such from the CLI as well. It also makes it clearer for users (even though that's going to be only super users here) what is meant to be shared and what is metadata.

• 📍 Wire newly introduce governance commands in the CLI
  Mostly plumbing and parser implementation following what already exists.

• 📍 Introduce a new test helper function: tryExecCardanoCLI
  This is meant as a way to assert on expected failures! Sadly, try or other exception handling mechanisms do not work inside of the TestT monad, so I had to extract and lift the error to be able to catch it and assert on it. Yet, I need to assert on failures and thus, failures should not crash the test early but be assertable as a possible execution outcome.

  There's maybe something more clever to do but I only had a day and a half to spend on all this so I'd rather "get it done".

• 📍 Write automated tests to cover newly introduced SPO on-chain poll commands
  Fixture keys were generated using the command-line itself. The set of tests cover quite extensively the various commands, as well as a few 'negative' test scenarios. It is more complicated to cover the interactive part of the 'answer-poll' command through those tests; and this is therefore left as manual test. Instructions for executing the sequence will also be provided with the introduction of the commands (e.g. in the description of PR#5050.

• 📍 Add roundtrip serialization property tests for GovernancePoll{Answer, Witness}

  roundtrip GovernancePoll CBOR:                        OK (0.09s)
    ✓ roundtrip GovernancePoll CBOR passed 100 tests.
  roundtrip GovernancePollAnswer CBOR:                  OK
    ✓ roundtrip GovernancePollAnswer CBOR passed 100 tests.
  roundtrip GovernancePollWitness CBOR:                 OK (0.01s)
    ✓ roundtrip GovernancePollWitness CBOR passed 100 tests.
  

• 📍 Add property tests for 'chunks', and fix 'metaTextChunks'

  Cardano.Api
  Test.Cardano.Api.Metadata
    valid & rountrip text chunks:  OK (0.03s)
        ✓ valid & roundtrip text chunks passed 100 tests.
          Empty chunks   3% ▌··················· ✓  1%
          Single chunks 26% █████▏·············· ✓  5%
          Many chunks   71% ██████████████▏····· ✓ 25%
    valid & rountrip bytes chunks: OK
        ✓ valid & roundtrip bytes chunks passed 100 tests.
          Empty chunks   3% ▌··················· ✓  1%
          Single chunks 55% ███████████········· ✓  5%
          Many chunks   42% ████████▍··········· ✓ 25%
  

  Turns out there were two issues:

  • Empty {text,byte}strings would generate a singleton chunk with an
    empty value; which is okay semantically but ugly; empty strings now
    generate an empty chunk.

  • Metadata values measure the length of UTF-8-encoded strings, which
    means we can't rely on default text functions to split a text
    string. This is likely an overkill in many situation in the context
    of PR#5050 since most questions / answers will be in plain english.

    However, we can now put emojis and crazy unicode characters in there
    without problems.

[newhoggy]

Thanks for adding the quick tutorials.

So if I understand correctly, someone can put up a poll somewhere off-chain that they've specified in a poll.json and any number of people can answer the poll by submitting their own transaction with answer.json.

And finally verify-poll is used to determine if a signed transaction for the poll.json/answer.json exists on-chain.

I'd like to additionally ask if this is for voting by many people?

If so, I have these questions:

• How do you specify who is eligible to vote?
• How do you find all of the answer.json files in order to tally up the responses?
• How do you determine the cut-off after which the voting is finished?
• How do you prevent someone from voting twice or voting for more than one answer per poll?

If not, I may be misunderstanding the purpose of the PR.

Perhaps my questions point to the misunderstanding I have and you can help clear that up.

[newhoggy]

I'm interested to see the following:

• adding round trip tests
• fixing the hlint warnings

I'm okay with adding the write to file functionality myself later if it isn't included in this PR, but you're welcome to do that too.

[KtorZ]

@newhoggy Done.

Note: there's no instructions whatsoever about what linter, formatter, etc.., are expected; if there are some I couldn't find them. I did find a Makefile which sounded useful with a make lint relying on Nix behind the scene but it doesn't work on arm architecture (e.g. MacOS M1 / M2), so no luck. I did run a local version of hlint, but hints could differ based on what version is expected from the CI and it's simply impossible to figure out what version the CI uses.

[newhoggy]

Ah, yes, I see your problem now.

The linter is defined here with the version number:

https://github.com/input-output-hk/cardano-node/blob/master/.github/workflows/check-hlint.yml

But of course, PR does not trigger the CI for forks for security reasons, so it is not obvious that linting is necessary.

We need to document that better. We need to consider how to improve the experience for contributions outside of IOG.

A good start could be in Github PR template.

Thanks for pointing this out.

[newhoggy]

I have triggered the build to see if it passes in CI

[newhoggy]

Much thanks for the property tests!

[newhoggy]

Don't worry about the ci/pr failures.

[newhoggy]

Typo accaptable

[newhoggy]

Okay CI passes now. I triggered CI by pushing the commit on my own branch to the repo.

I have approved the PR.

[Jimbo4350]

Subsumed by #5112