plugins: unique command line options, small fixes and plugin documentation

[SimonVrouwe]

Some issues I came across looking at #5294 (that discussion is ongoing, this PR adds some context)

Converts plugin->cmd to a full path immediately (couldn't find a reason not to), ensures only one instance can run.
Plugin command line options should have unique names (and no aliases).
Update plugin documentation.

edit Last 2 8be7693 and cf2dbbb commits is an idea for plugins that do configs internally, not fully worked out.

edit Add "dynamic" field to plugin list result, some builtin plugins are dynamic !

[vincenzopalazzo]

Concept ACK.

I need to return on it!

[SimonVrouwe]

About halve of the builtin plugins are "dynamic", is this deliberate?
Calling plugin stop on them will shutdown lightningd.

[cdecker]

Great cleanup of some longstanding issues 👍

[rustyrussell]

Sorry, these comments were sitting pending in GH for a while :(

[SimonVrouwe]

Applied the feedback and some minor fixes, squashed and rebased.
The addendum stuff has been dropped (for now).
I added some relative links and a footnote to PLUGINS.md, hopefully these work in readthedocs.

[SimonVrouwe]

The most serious test fail seems a memleak in test_openchannel_hook_chaining:

E            - Node /tmp/ltests-y28mxbp5/test_openchannel_hook_chaining_1/lightning-2/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "ccan/ccan/io/io.c:91 (io_new_conn_)",
E                       "lightningd/subd.c:774 (new_subd)",
E                       "lightningd/subd.c:828 (new_channel_subd_)",
E                       "lightningd/opening_control.c:872 (peer_start_openingd)",
E                       "lightningd/peer_control.c:1311 (peer_active)",
E                       "lightningd/connect_control.c:458 (connectd_msg)",
E                       "lightningd/subd.c:557 (sd_msg_read)",
E                       "lightningd/subd.c:357 (read_fds)",
E                       "ccan/ccan/io/io.c:59 (next_plan)",
E                       "ccan/ccan/io/io.c:407 (do_plan)",
E                       "ccan/ccan/io/io.c:417 (io_ready)",
E                       "ccan/ccan/io/poll.c:453 (io_loop)",
E                       "lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
E                       "lightningd/lightningd.c:1182 (main)",
E                       "../csu/libc-start.c:308 (__libc_start_main)"
E                   ],
E                   "label": "ccan/ccan/io/io.c:91:struct io_conn",
E                   "parents": [
E                       "lightningd/lightningd.c:107:struct lightningd"
E                   ],
E                   "value": "0x55d72cfc3058"

2022-07-04T17:12:53.278Z DEBUG   lightningd: openchannel hook rejects and says 'reject on principle'
2022-07-04T17:12:53.278Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-openingd-chan#2: peer_out WIRE_ERROR
2022-07-04T17:12:53.279Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-openingd-chan#2: peer_in WIRE_ERROR
2022-07-04T17:12:53.279Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-openingd-chan#2: aborted opening negotiation: They sent error channel ae288ec6893205a5f369f192a924252d31c7cec52c2f05da658d1ace512ae795: You gave bad parameters: reject on principle
2022-07-04T17:12:53.279Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-openingd-chan#2: billboard perm: They sent error channel ae288ec6893205a5f369f192a924252d31c7cec52c2f05da658d1ace512ae795: You gave bad parameters: reject on principle
2022-07-04T17:12:53.280Z DEBUG   plugin-spenderp: mfc 33, dest 0: failed! fundchannel_start 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59: {\"code\":-1,\"message\":\"They sent error channel ae288ec6893205a5f369f192a924252d31c7cec52c2f05da658d1ace512ae795: You gave bad parameters: reject on principle\"}.
2022-07-04T17:12:53.280Z DEBUG   plugin-spenderp: mfc 33: parallel channel starts done.
2022-07-04T17:12:53.280Z DEBUG   plugin-spenderp: mfc 33: trying redo despite 'fundchannel_start' failure (They sent error channel ae288ec6893205a5f369f192a924252d31c7cec52c2f05da658d1ace512ae795: You gave bad parameters: reject on principle); will cleanup for now.
2022-07-04T17:12:53.280Z DEBUG   plugin-spenderp: mfc 33: cleanup!
2022-07-04T17:12:53.280Z DEBUG   plugin-spenderp: mfc 33: unreserveinputs task.
2022-07-04T17:12:53.282Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: destroy_subd: 1 subds, to_peer conn 0x55782e6644b8, read_to_die = 0
2022-07-04T17:12:53.282Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: disconnect
2022-07-04T17:12:53.282Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_conn_closed
2022-07-04T17:12:53.282Z DEBUG   0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: destroy_subd: 1 subds, to_peer conn (nil), read_to_die = 0

Note the nil logged by L2 in the last line, it seems io_conn was already taken/moved ?

Will try to reproduce, but I don't think the problem lies in this PR but maybe it triggered a race.

edit I cannot reproduce above memleak
The only thing I notice from logs is the in the 2nd case (with an openchannel hook chain of 3 ), openingd is killed about 6ms later compared to the the other case, i.e.:

022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_conn_closed
2022-07-04T17:22:17.3525818Z 2022-07-04T17:12:53.282Z DEBUG   in_L2-connectd: destroy_subd: 1 subds, to_peer conn (nil), read_to_die = 0
...
2022-07-04T17:22:17.3536336Z 2022-07-04T17:12:53.288Z DEBUG   in_L2-openingd-chan#2: Status closed, but not exited. Killing

compared to

2022-07-04T17:22:17.3445616Z 2022-07-04T17:12:53.253Z DEBUG   in_L2-openingd-chan#1: Status closed, but not exited. Killing
2022-07-04T17:22:17.3446370Z 2022-07-04T17:12:53.253Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: destroy_subd: 1 subds, to_peer conn 0x55782e662a68, read_to_die = 0

[niftynei]

The most serious test fail seems a memleak in test_openchannel_hook_chaining:

This is failing elsewhere, is a general regression on master.

[niftynei]

ACK d267f27

Test failures are unrelated, i've saved the logs from the fetchinvoice error and will add a separate issue to track the flakes.