feat: new ctl #395

hugomrdias · 2019-10-11T14:30:50Z

Adds interface-core common tests setup to ipfsd-ctl so we can dedupe this code from js-ipfs (https://github.com/ipfs/js-ipfs/blob/master/test/utils/interface-common-factory.js#L71) and http-client (https://github.com/ipfs/js-ipfs-http-client/blob/master/test/utils/interface-common-factory.js).

Also adds jsdocs to be easier to transition to this new async setup.

It works basically the same as the createAsync version with one less function call and a new method to create an unmanaged node to be used inside tests you want to stop manually.

PRs to use this in js-ipfs, http-client and interface-core will follow

This PR evolved from adding some helper functions for tests into a full rewrite (sorry about that 😅). While making the supporting PRs i found some problems and inconsistencies that needed to be fix to improve our own and contributors DX.

Problems:

Browsers tests skipped cause ctl didn't support proper connectivity to remote nodes
We weren't able to tell ctl to use a specific commit of http-client, js-ipfs or cli
Options/config between the 3 types of daemons weren't consistent
Ctl didn't support remote "in process" daemon
IPFS options were handled manually inside ctl, so any change in js-ipfs would require a PR in ctl to support the new options or change to an option

Related issues:

Improvements:

better errors
DEBUG='ipfsd-ctl:*' everywhere
factories for tests with good defaults
options are properly merged everywhere
safer child_process exit stop()
faster stop()
IPFS Options are now the same format as https://github.com/ipfs/js-ipfs/blob/master/README.md#ipfs-constructor
Ctl can init, start and set config in one cmd (at least with js-ipfs)
better docs and jsdocs
we can now be sure which http-client, ipfs or go-ipfs is being used
utils functions actually work in the browser now
works in webworkers now
simpler and faster overall
disposable node actually clean themselves in the browser
better tests
...
support electron
test in electron

New:

new method createController returns a spawned controller
createFactory as a second parameter to override options per type

Changes:

create change to createFactory
createFactory options changed

Old

- `options` - optional object with:
  - `remote` bool - use remote endpoint to spawn the nodes.
  - `port` number - remote endpoint port. Defaults to 43134.
  - `exec` - IPFS executable path. `ipfsd-ctl` will attempt to locate it by default. If you desire to spawn js-ipfs instances in the same process, pass the ref to the module instead (e.g `exec: require('ipfs')`)
  - `type` - the daemon type, see below the options
    - `go` - spawn go-ipfs daemon
    - `js` - spawn js-ipfs daemon
    - `proc` - spawn in-process js-ipfs instance. Needs to be called also with exec. Example: `DaemonFactory.create({type: 'proc', exec: require('ipfs') })`.
  - `IpfsClient` - A custom IPFS API constructor to use instead of the packaged one

New

-   `remote` [boolean] Use remote endpoint to spawn the nodes. Defaults to `true` when not in node.
-   `test` [test=false] - Flag to activate custom config for tests.
-   `endpoint` [endpoint] - Endpoint URL to manage remote Controllers. (Defaults: 'http://localhost:43134').
-   `disposable` [Boolean] A new repo is created and initialized for each invocation, as well as cleaned up automatically once the process exits.
-   `type` [string] The daemon type, see below the options:-   go - spawn go-ipfs daemon
    -   js - spawn js-ipfs daemon
    -   proc - spawn in-process js-ipfs instance
-   `env` [Object] Additional environment variables, passed to executing shell. Only applies for Daemon controllers.
-   `args` [Array] Custom cli args.
-   `ipfsHttp` [Object] Setup IPFS HTTP client to be used by ctl.
    -   `ipfsHttp.ref` [Object] Reference to a IPFS HTTP Client object. (defaults to the local require(`ipfs-http-client`))
    -   `ipfsHttp.path` [string] Path to a IPFS HTTP Client to be required. (defaults to the local require.resolve('ipfs-http-client'))
-   `ipfsApi` [Object] Setup IPFS API to be used by ctl.
    -   `ipfsApi.ref` [Object] Reference to a IPFS API object. (defaults to the local require(`ipfs`))
    -   `ipfsApi.path` [string] Path to a IPFS API implementation to be required. (defaults to the local require.resolve('ipfs'))
-   `ipfsBin` [String] Path to a IPFS exectutable . (defaults to the local 'js-ipfs/src/bin/cli.js')
-   `ipfsOptions` [IpfsOptions] Options for the IPFS instance

Previous default ipfs config is only applied when test options equals true
defaultAddrs option was removed
Spawn options are the same as createFactory

Old

- `options` is an optional object the following properties:
  - `init` bool (default true) or Object - should the node be initialized
  - `initOptions` object - should be of the form `{bits: <size>}`, which sets the desired key size
  - `start` bool (default true) - should the node be started
  - `repoPath` string - the repository path to use for this node, ignored if node is disposable
  - `disposable` bool (default true) - a new repo is created and initialized for each invocation, as well as cleaned up automatically once the process exits
  - `defaultAddrs` bool (default false) - use the daemon default `Swarm` addrs
  - `args` - array of cmd line arguments to be passed to ipfs daemon
  - `config` - ipfs configuration options

NEW
Same as js-ipfs constructor https://github.com/ipfs/js-ipfs#ipfs-constructor

ipfsd.killProcess removed not needed anymore
ipfsd.getConfig removed call ipfsd.api.config.get instead
ipfsd.setConfig removed, call ipfsd.api.config.set instead

TODO:

update readme
finish tests
update interface-core feat: add support new ctl ipfs-inactive/interface-js-ipfs-core#541
update http-client fix: use new ipfsd-ctl setup ipfs-inactive/js-ipfs-http-client#1127
update js-ipfs feat: support new ctl api js-ipfs#2529
update interop
update desktop @hacdias

alanshaw

Looks good, just some minor things. Lets get this merged and released! 🚀

package.json

src/factory-in-proc.js

src/factory-daemon.js

src/index.js

src/factory-client.js

src/ipfsd-daemon.js

achingbrain · 2019-11-15T19:12:08Z

@hugomrdias are you going to polish off the last few nits in this? It'd be good to get it in to unblock a bunch of other stuff.

alanshaw · 2019-11-26T10:05:18Z

🙏 pretty please can we have this?

hacdias · 2019-11-26T10:47:19Z

@alanshaw +1

hugomrdias · 2019-12-03T14:26:28Z

PR description updated explaining whats new and whats changed.

achingbrain

Minor nits, looks good though

achingbrain · 2019-12-03T16:14:35Z

README.md

-### Remote endpoint - `const server = IPFSFactory.createServer([options])`
-
-`IPFSFactory.createServer` starts a IPFSFactory endpoint.
+### `create([options])`


This might be better named createFactory([options])?

damn it was like that i just changed it back yesterday to reduce the diff for the end user.
do you think it's worth it ?

I think it's good to be explicit. The diff is going to be hefty anyway, right?

In the top level api nothing actually changed just the options are different

README.md

achingbrain · 2019-12-03T16:18:57Z

src/endpoint/server.browser.js

+   */
+  start () {
+    console.warn('Server not implemented in the browser')
+    return Promise.resolve(this)


Should this (and the other methods in this class) throw instead?

alanshaw · 2019-12-04T12:36:27Z

README.md

@@ -58,11 +58,11 @@ await ipfsd.stop()
 // Start a remote disposable node, and get access to the api
 // print the node id, and stop the temporary daemon

-const IPFSFactory = require('ipfsd-ctl')
+const { create, createServer } = require('ipfsd-ctl')


I prefer the previous require style, using create() in code makes me think "what am I creating?" whereas IPFSFactory.create() is obvious.

alanshaw · 2019-12-04T12:39:08Z

README.md

-
-console.log('endpoint has stopped')
-```
+### `createNodeTests([options])`


Why not just createNode({ test: true })?

i can do that sure

alanshaw · 2019-12-04T12:40:59Z

README.md

+#### TestsInterface
+`nodes` **Array** List of the created controlled nodes.   
+`node()` **Function** Creates a standalone node, this node will **NOT** be stopped by `teardown()`.   
+`setup(options)` **Function(**[FactoryOptions](#FactoryOptions)**)** Creates a controlled node.   


What does "controlled" mean? Just that they get stopped when teardown is called?

alanshaw · 2019-12-04T12:42:34Z

README.md

+Returns a **[TestsInterface](#TestsInterface)**
+#### TestsInterface
+`nodes` **Array** List of the created controlled nodes.   
+`node()` **Function** Creates a standalone node, this node will **NOT** be stopped by `teardown()`.   


How about just using the same API but providing an option? setup({ controlled: false })? setup({ managed: false })?

This needs better wording

node() returns the a ctl node
setup() returns directly an ipfs core api

Should both return the same thing ?

i would rather add a new param than to use an option just to keep FactoryOptions clean. Cause this would be specific to the testInterface.

I also think using the word node here is wrong... We use daemon for process nodes and node for in process etc... but here it's actually a Controller for a IPFS node either process or in-proc.

Would you agree if i change the naming to Controller instead of Node ?

Then ipfsd-ctl would have:

Factory to wrap spawn, version and tmpDir
Controller to wrap the 3 different types of node
plus some methods that return an IPFS Core API.

alanshaw · 2019-12-04T12:43:25Z

README.md


-Get the address (multiaddr) of connected IPFS HTTP Gateway. Returns a multiaddr.
+### `createServer([options])`
+Create an Endpoint Server.


Needs elaboration! What is the server for? Why would you create one? What can it do?

alanshaw · 2019-12-04T12:46:10Z

README.md

+    -   proc - spawn in-process js-ipfs instance
+-   `env` **[Object]** Additional environment variables, passed to executing shell. Only applies for Daemon controllers.
+-   `args` **[Array]** Custom cli args.
+-   `ipfsHttp` **[Object]** Setup IPFS HTTP client to be used by ctl.


If I pass this option are the properties both mandatory? Please can the documentation be clarified?

nothing is mandatory, but if you really want to make sure you are using a specific version you need to specify both!
ill update the readme with that info !

alanshaw · 2019-12-04T12:48:57Z

README.md

+-   `ipfsHttp` **[Object]** Setup IPFS HTTP client to be used by ctl.
+    -   `ipfsHttp.ref` **[Object]** Reference to a IPFS HTTP Client object. (defaults to the local require(`ipfs-http-client`))
+    -   `ipfsHttp.path` **[string]** Path to a IPFS HTTP Client to be required. (defaults to the local require.resolve('ipfs-http-client'))
+-   `ipfsApi` **[Object]** Setup IPFS API to be used by ctl.


I'm not convinced this is the best name for this...especially considering IPFS HTTP Client used to be called ipfs-api and both js-ipfs and js-ipfs-http-client expose the same API.

What about ipfsModule and ipfsHttpClientModule for these?

alanshaw · 2019-12-04T12:51:27Z

README.md

+    -   `ipfsApi.ref` **[Object]** Reference to a IPFS API object. (defaults to the local require(`ipfs`))
+    -   `ipfsApi.path` **[string]** Path to a IPFS API implementation to be required. (defaults to the local require.resolve('ipfs'))
+-   `ipfsBin` **[string]** Path to a IPFS exectutable . (defaults to the local 'js-ipfs/src/bin/cli.js')
+-   `ipfsOptions` **[IpfsOptions]** Options for the IPFS instance


Specify when these are applicable? I assume they do not work for HTTP client or go daemon?

they work for all 3 types as far as possible exactly like the old version.

proc its passed directly

daemon it translates as much as possible to args (plus you can manually define and extra args option)

client same as daemon cause its just a proxy

this change was to normalise and have just one set possible of options

alanshaw · 2019-12-04T12:53:19Z

README.md


-const server = IPFSFactory.createServer({ port: 12345 })
+### `createNode([options])`
+Creates a node.


Can you explain why you'd use createNode and not create?

create creates a factory and you get .tmpDir(), .version() and .spawn()

createNode return the node directly basically calls spawn() for you which is what you want 90% of the time.

i didn't made createNode default because there are some use cases that you really need Factory.tmpDir() to create a temp dir for the repo when using a remote node.

src/ipfsd-daemon.js

hugomrdias · 2019-12-06T14:23:53Z

Moved a few things around to integrate your feedback. Please review again.

alanshaw · 2019-12-10T10:55:51Z

README.md


-Get the version without spawning a daemon
+### `createServer([options])`
+Create an Endpoint Server. This server is used by a client node to control a remote node. Example: Spawning a go-ipfs node from a browser.


Are there any docs for the API endpoints that are available?

No, its mostly to be used by the remote Controller not for public use.

we supported ['--pass some-string'] which is not very common in any process spawning lib so now we only support the common syntax which is ['--pass', 'some-string']

Co-Authored-By: Alex Potsides <alex@achingbrain.net>

hugomrdias requested a review from achingbrain October 11, 2019 14:31

This was referenced Oct 11, 2019

feat: add support new ctl ipfs-inactive/interface-js-ipfs-core#541

Merged

fix: use new ipfsd-ctl setup ipfs-inactive/js-ipfs-http-client#1127

Merged

feat: support new ctl api ipfs/js-ipfs#2529

Merged

hugomrdias force-pushed the feat/interface-tests branch from e2d2841 to bbda5a9 Compare October 16, 2019 14:26

hugomrdias changed the title ~~feat: add interface-core tests setup~~ feat: new ctl Nov 5, 2019

hugomrdias mentioned this pull request Nov 5, 2019

Improve overall state of js-ipfs tests ipfs/js-ipfs#2276

Closed

42 tasks

alanshaw requested changes Nov 6, 2019

View reviewed changes

package.json Outdated Show resolved Hide resolved

src/factory-in-proc.js Outdated Show resolved Hide resolved

src/factory-daemon.js Outdated Show resolved Hide resolved

src/index.js Outdated Show resolved Hide resolved

achingbrain requested changes Nov 6, 2019

View reviewed changes

src/factory-client.js Outdated Show resolved Hide resolved

src/ipfsd-daemon.js Outdated Show resolved Hide resolved

src/ipfsd-daemon.js Outdated Show resolved Hide resolved

src/ipfsd-daemon.js Outdated Show resolved Hide resolved

hugomrdias changed the title ~~feat: new ctl~~ [WIP] feat: new ctl Nov 8, 2019

hugomrdias requested a review from hacdias November 8, 2019 11:57

PedroMiguelSS mentioned this pull request Nov 20, 2019

chore: async await refactor ipfs-inactive/interface-js-ipfs-core#562

Merged

hugomrdias requested review from alanshaw and achingbrain December 3, 2019 14:25

hugomrdias changed the title ~~[WIP] feat: new ctl~~ feat: new ctl Dec 3, 2019

achingbrain requested changes Dec 3, 2019

View reviewed changes

alanshaw requested changes Dec 4, 2019

View reviewed changes

hugomrdias requested review from achingbrain and alanshaw December 6, 2019 14:23

alanshaw approved these changes Dec 10, 2019

View reviewed changes

hugomrdias added 5 commits December 10, 2019 14:18

feat: add interface-core tests setup

5b4abe7

fix: add preload false to the defaults

c70fd4e

fix: fix the args handling

e27310c

we supported ['--pass some-string'] which is not very common in any process spawning lib so now we only support the common syntax which is ['--pass', 'some-string']

feat: complete rewrite

1307f85

fix: try to fix windows

c27bce8

hugomrdias and others added 25 commits December 10, 2019 14:18

fix: update deps

35e9240

docs: readme

bea75f6

fix: try to fix timeout issues

05d5b77

docs: fix typo

a6704f4

Co-Authored-By: Alex Potsides <alex@achingbrain.net>

feat: feedback

0c4cd1a

fix: typo

20c9ca6

fix: fix timeout

dcc6d5f

chore: make travis happy

3be7ac1

fix: cant stop this one

da2ec63

fix: fix stop tests

694cf2b

chore: debug travis

66e251e

chore: comment travis stuff for debuggin

93be66e

chore: more debug

ef6c91f

chore: more debug 1

961c896

chore: more debug 2

e87008c

chore: more debug 3

4cc33dd

chore: more debug 4

dbef8a4

chore: more debug 5

f4d162d

chore: more debug 6

a9bdd9a

chore: more debug 7

bae9bba

chore: remove debug

bc532d8

chore: remove debug 2

34e4e6f

chore: ignore undefined

c477b5f

fix: factory change signature

2152176

chore: default to the proper type

982bb12

hugomrdias force-pushed the feat/interface-tests branch from a04b555 to 982bb12 Compare December 10, 2019 14:20

fix: increase bundle size

eae8e8f

hugomrdias merged commit 3dae276 into master Dec 11, 2019

hugomrdias deleted the feat/interface-tests branch December 11, 2019 15:59

hugomrdias mentioned this pull request Mar 10, 2020

Make sure we have interop with differents types of secondary nodes ipfs/js-ipfs#2881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new ctl #395

feat: new ctl #395

hugomrdias commented Oct 11, 2019 •

edited

Loading

alanshaw left a comment

achingbrain commented Nov 15, 2019

alanshaw commented Nov 26, 2019

hacdias commented Nov 26, 2019 •

edited

Loading

hugomrdias commented Dec 3, 2019

achingbrain left a comment

achingbrain Dec 3, 2019

hugomrdias Dec 3, 2019

achingbrain Dec 3, 2019

hugomrdias Dec 3, 2019

achingbrain Dec 3, 2019

alanshaw Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019 •

edited

Loading

alanshaw Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019 •

edited

Loading

alanshaw Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019

alanshaw Dec 4, 2019

hugomrdias Dec 4, 2019

hugomrdias commented Dec 6, 2019

alanshaw Dec 10, 2019

hugomrdias Dec 10, 2019

feat: new ctl #395

feat: new ctl #395

Conversation

hugomrdias commented Oct 11, 2019 • edited Loading

alanshaw left a comment

Choose a reason for hiding this comment

achingbrain commented Nov 15, 2019

alanshaw commented Nov 26, 2019

hacdias commented Nov 26, 2019 • edited Loading

hugomrdias commented Dec 3, 2019

achingbrain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugomrdias Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugomrdias Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugomrdias commented Dec 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugomrdias commented Oct 11, 2019 •

edited

Loading

hacdias commented Nov 26, 2019 •

edited

Loading

hugomrdias Dec 4, 2019 •

edited

Loading

hugomrdias Dec 4, 2019 •

edited

Loading