diff --git a/docs/source/release_process.rst b/docs/source/contributing.rst similarity index 72% rename from docs/source/release_process.rst rename to docs/source/contributing.rst index 4b2e3b102..371df7fe0 100644 --- a/docs/source/release_process.rst +++ b/docs/source/contributing.rst @@ -1,3 +1,24 @@ +.. _contributing: + +****************** +Contributor guide +****************** + +Receptor is an open source project that lives at https://github.com/ansible/receptor + +.. contents:: + :local: + +Code of conduct +================ + +All project contributors must abide by the `Ansible Code of Conduct `_. + +Contributing +============= + +Receptor welcomes community contributions! See the :ref:`dev_guide` for details. + Release process =============== diff --git a/docs/source/developer_guide.rst b/docs/source/developer_guide.rst index e1c9f20db..9dfe96c00 100644 --- a/docs/source/developer_guide.rst +++ b/docs/source/developer_guide.rst @@ -1,10 +1,18 @@ +.. _dev_guide: + Developer guide ================ Receptor is an open source project that lives at https://github.com/ansible/receptor +.. contents:: + :local: + +See the :ref:`contributing` for more general details. + + Testing -^^^^^^^ +-------- Pull requests must pass a suite of integration tests before being merged into ``devel``. @@ -39,12 +47,12 @@ For example, to create/update mocks for Workceptor, we can run: mockgen -source=pkg/workceptor/workceptor.go -destination=pkg/workceptor/mock_workceptor/workceptor.go Source code -^^^^^^^^^^^ +----------- The next couple of sections are aimed to orient developers to the receptor codebase and provide a starting point for understanding how receptor works. Parsing receptor.conf -""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^ Let's see how items in the config file are mapped to Golang internals. @@ -126,7 +134,7 @@ This gets a new TCP dialer object and passes it to the netceptor AddBackend meth In general, when studying how the start up process works in receptor, take a look at the Init, Prepare, and Run methods throughout the code, as these are the entry points to running those specific components of receptor. Ping -"""" +^^^^ Studying how pings work in receptor will provide a useful glimpse into the internal workings of netceptor -- the main component of receptor that handles connections and data traffic over the mesh. diff --git a/docs/source/index.rst b/docs/source/index.rst index 5ef9216eb..b5da23a9d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,129 +3,16 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Introduction -============ +############################### +Ansible Receptor documentation +############################### Receptor is an overlay network intended to ease the distribution of work across a large and dispersed collection of workers. Receptor nodes establish peer-to-peer connections with each other via existing networks. Once connected, the receptor mesh provides datagram (UDP-like) and stream (TCP-like) capabilities to applications, as well as robust unit-of-work handling with resiliency against transient network failures. -Terminology and Concepts -^^^^^^^^^^^^^^^^^^^^^^^^ - -- `receptor`: The receptor application taken as a whole, which typically runs as a daemon. -- `receptorctl`: A user-facing command line used to interact with receptor, typically over a Unix domain socket. -- `node`: A single running instance of receptor. -- `node ID`: An arbitrary string identifying a single node, analogous to an IP address. -- `backend`: A type of connection that receptor nodes can pass traffic over. Current backends include TCP, UDP and websockets. -- `control service`: A built-in service that usually runs under the name `control`. Used to report status and to launch and monitor work. -- `netceptor`: The component of receptor that handles all networking functionality. -- `workceptor`: The component of receptor that handles work units. - -Installation -^^^^^^^^^^^^ - -Download and extract precompiled binary for your OS and platform from `the releases page on GitHub `_ - -Alternatively, you can compile from source code (Golang 1.19+ required) - -.. code:: - - make receptor - -Test the installation with - -.. code:: - - receptor --help - receptor --version - -The preferred way to interact with receptor nodes is to use the receptorctl command line tool - -.. code:: - - pip install receptorctl - -receptorctl will be used in various places throughout this documentation. - -Basic usage -^^^^^^^^^^^ - -Run the following command in a terminal to start a node called `foo`, - -.. code:: - - receptor --node id=foo --local-only --log-level Debug - -The log shows the receptor node started successfully - -``INFO 2021/07/22 22:40:36 Initialization complete`` - -Supported log levels, in increasing verbosity, are Error, Warning, Info and Debug. - -Note: stop the receptor process with ``ctrl-c`` - -Config file -^^^^^^^^^^^ - -Receptor can be configured on the command-line, exemplified above, or via a yaml config file. All actions and parameters shown in ``receptor --help`` can be written to a config file. - -.. code-block:: yaml - - --- - - node: - id: foo - - - local-only - - - log-level: - level: Debug - -Start receptor using the config file - -.. code:: - - receptor --config foo.yml - -Changing the configuration file does take effect until the receptor process is restarted. - -Container image -^^^^^^^^^^^^^^^ - -.. code:: - - podman pull quay.io/ansible/receptor - -Start a container, which automatically runs receptor with the default config located at ``/etc/receptor/receptor.conf`` - -.. code:: - - podman run -it --rm --name receptor quay.io/ansible/receptor - -In another terminal, issue a basic "status" command to the running receptor process - -.. code:: - - $ podman exec receptor receptorctl status - Node ID: d9b5a8e3c156 - Version: 1.0.0 - System CPU Count: 8 - System Memory MiB: 15865 - - Node Service Type Last Seen Tags Work Types - d9b5a8e3c156 control Stream 2021-08-04 19:26:14 - - - -Note: the config file does not specify a node ID, so the hostname (on the container) is chosen as the node ID. - .. toctree:: - :maxdepth: 3 - :caption: Contents: + :maxdepth: 2 - self - connecting_nodes - interacting_with_nodes - workceptor - k8s - tls - firewall - edge_networks + installation + user_guide/index developer_guide - release_process + contributing diff --git a/docs/source/installation.rst b/docs/source/installation.rst new file mode 100644 index 000000000..634a40714 --- /dev/null +++ b/docs/source/installation.rst @@ -0,0 +1,28 @@ +.. _installing: + +******************* +Installation guide +******************* + +Download and extract precompiled binary for your OS and platform from `the releases page on GitHub `_ + +Alternatively, you can compile from source code (Golang 1.19+ required) + +.. code:: + + make receptor + +Test the installation with + +.. code:: + + receptor --help + receptor --version + +The preferred way to interact with receptor nodes is to use the receptorctl command line tool + +.. code:: + + pip install receptorctl + +receptorctl will be used in various places throughout this documentation. \ No newline at end of file diff --git a/docs/source/user_guide/basic_usage.rst b/docs/source/user_guide/basic_usage.rst new file mode 100644 index 000000000..18f4a36b7 --- /dev/null +++ b/docs/source/user_guide/basic_usage.rst @@ -0,0 +1,74 @@ +Using Receptor +=============== + +. contents:: + :local: + +Configuring Receptor with the CLI +----------------------------------- + +Run the following command in a terminal to start a node called `foo`, + +.. code:: + + receptor --node id=foo --local-only --log-level Debug + +The log shows the receptor node started successfully + +``INFO 2021/07/22 22:40:36 Initialization complete`` + +Supported log levels, in increasing verbosity, are Error, Warning, Info and Debug. + +Note: stop the receptor process with ``ctrl-c`` + +Configuring Receptor with a config file +---------------------------------------- + +Receptor can be configured on the command-line, exemplified above, or via a yaml config file. All actions and parameters shown in ``receptor --help`` can be written to a config file. + +.. code-block:: yaml + + --- + - node: + id: foo + + - local-only + + - log-level: + level: Debug + +Start receptor using the config file + +.. code:: + + receptor --config foo.yml + +Changing the configuration file does take effect until the receptor process is restarted. + +Use Receptor through a container image +--------------------------------------- + +.. code:: + + podman pull quay.io/ansible/receptor + +Start a container, which automatically runs receptor with the default config located at ``/etc/receptor/receptor.conf`` + +.. code:: + + podman run -it --rm --name receptor quay.io/ansible/receptor + +In another terminal, issue a basic "status" command to the running receptor process + +.. code:: + + $ podman exec receptor receptorctl status + Node ID: d9b5a8e3c156 + Version: 1.0.0 + System CPU Count: 8 + System Memory MiB: 15865 + + Node Service Type Last Seen Tags Work Types + d9b5a8e3c156 control Stream 2021-08-04 19:26:14 - - + +Note: the config file does not specify a node ID, so the hostname (on the container) is chosen as the node ID. diff --git a/docs/source/connecting_nodes.rst b/docs/source/user_guide/connecting_nodes.rst similarity index 97% rename from docs/source/connecting_nodes.rst rename to docs/source/user_guide/connecting_nodes.rst index a9378d702..1c920a587 100644 --- a/docs/source/connecting_nodes.rst +++ b/docs/source/user_guide/connecting_nodes.rst @@ -1,9 +1,14 @@ Connecting nodes ================ +.. contents:: + :local: + + Connect nodes via receptor backends. TCP, UDP, and websockets are currently supported. For example, ``tcp-peer`` can be used to connect to another node's ``tcp-listener``, and ``ws-peer`` can be used to connect to another node's ``ws-listener``. .. image:: mesh.png + :alt: Connected nodes as netceptor peers foo.yml @@ -80,7 +85,7 @@ Logs from `fish` shows a successful connection to `bar` via `foo`. Configuring backends -^^^^^^^^^^^^^^^^^^^^ +-------------------- ``redial`` If set to true, receptor will automatically attempt to redial and restore connections that are lost. diff --git a/docs/source/edge.png b/docs/source/user_guide/edge.png similarity index 100% rename from docs/source/edge.png rename to docs/source/user_guide/edge.png diff --git a/docs/source/edge_networks.rst b/docs/source/user_guide/edge_networks.rst similarity index 95% rename from docs/source/edge_networks.rst rename to docs/source/user_guide/edge_networks.rst index 0ff8c0938..eaf9068da 100644 --- a/docs/source/edge_networks.rst +++ b/docs/source/user_guide/edge_networks.rst @@ -3,12 +3,16 @@ Support for Edge Networks Recptor out-of-the-box has the ability to support complicated networking environments including edge networks. +.. contents:: + :local: + Consider the following environment: .. image:: edge.png + :alt: Network diagram with netceptor peers to edge network Configurable-Items -^^^^^^^^^^^^^^^^^^ +------------------- Receptor encapsulates the concepts of `below-the-mesh` and `above-the-mesh` connections. Please refer to :doc:`tls` for a better understanding of these networking layers. diff --git a/docs/source/firewall.rst b/docs/source/user_guide/firewall.rst similarity index 100% rename from docs/source/firewall.rst rename to docs/source/user_guide/firewall.rst diff --git a/docs/source/user_guide/index.rst b/docs/source/user_guide/index.rst new file mode 100644 index 000000000..29d14e3d3 --- /dev/null +++ b/docs/source/user_guide/index.rst @@ -0,0 +1,47 @@ +.. _user_guide: + +******************* +User guide +******************* + +This guide describes how to use receptor in multiple environments and uses the following terms: + +.. glossary:: + + receptor + The receptor application taken as a whole, which typically runs as a daemon. + + receptorctl + A user-facing command line used to interact with receptor, typically over a Unix domain socket. + + node + A single running instance of receptor. + + node ID + An arbitrary string identifying a single node, analogous to an IP address. + + backend + A type of connection that receptor nodes can pass traffic over. Current backends include TCP, UDP and websockets. + + control service + A built-in service that usually runs under the name `control`. Used to report status and to launch and monitor work. + + netceptor + The component of receptor that handles all networking functionality. + + workceptor + The component of receptor that handles work units. + +.. toctree:: + :maxdepth: 2 + + basic_usage + connecting_nodes + interacting_with_nodes + workceptor + k8s + tls + firewall + edge_networks + + diff --git a/docs/source/interacting_with_nodes.rst b/docs/source/user_guide/interacting_with_nodes.rst similarity index 98% rename from docs/source/interacting_with_nodes.rst rename to docs/source/user_guide/interacting_with_nodes.rst index c21b13867..13ead5ad9 100644 --- a/docs/source/interacting_with_nodes.rst +++ b/docs/source/user_guide/interacting_with_nodes.rst @@ -1,6 +1,12 @@ + + Interacting with nodes ====================== +.. contents:: + :local: + + The ``control-service`` allows the user to issue commands like "status" or "work submit" to a receptor node. foo.yml @@ -65,7 +71,7 @@ The "status" command will display helpful information about mesh, including know bar control Stream 2021-07-22 23:32:35 - - ReceptorControl -^^^^^^^^^^^^^^^ +---------------- For a more programmatic way to interact with receptor nodes, use the ReceptorControl python class. @@ -79,7 +85,7 @@ For a more programmatic way to interact with receptor nodes, use the ReceptorCon .. _connect_to_csv: Connect to control service -^^^^^^^^^^^^^^^^^^^^^^^^^^ +--------------------------- Use the "connect" command to connect to any receptor control service running on the mesh. From here, issue a series of commands and examine the output, without disconnecting. @@ -104,7 +110,7 @@ Once connected to a control service, one can issue commands like "status" or "wo Keep in mind that a "work submit" command will require a payload. Type out the payload contents and press ctrl-D to send the EOF signal. The socket will then close and work will begin. See :ref:`workceptor` for more on submitting work via receptor. Control service commands -^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------- A ``control-service`` can accept commands in two formats; a space-delimited string or JSON. In some cases, JSON accepts arguments that are not supported in the string format and are marked with `json-only` in the table below. @@ -170,7 +176,7 @@ The order of the parameters (from left to right) in the following table matter, The above table does not apply the receptorctl command-line tool. For the exact usage of the various receptorctl commands, type ``receptorctl --help``, or to see the help for a specific command, ``receptorctl work submit --help``. Reload -^^^^^^ +------- In general, changes to a receptor configuration file do not take effect until the receptor process is restarted. diff --git a/docs/source/k8s.rst b/docs/source/user_guide/k8s.rst similarity index 98% rename from docs/source/k8s.rst rename to docs/source/user_guide/k8s.rst index 8c381edb8..1f4e7c236 100644 --- a/docs/source/k8s.rst +++ b/docs/source/user_guide/k8s.rst @@ -1,6 +1,9 @@ Kubernetes work =============== +.. contents:: + :local: + Workceptor has the ability to launch Kubernetes pods to perform work. foo.yml @@ -70,8 +73,8 @@ Now we can submit a kubernetes work unit. ``secret_kube_pod`` The contents of a pod definition. The "@" tells receptorctl to read in a file name and pass the contents on. -Runtime Params -^^^^^^^^^^^^^^ +Runtime params +--------------- Additional parameters can be passed in when issuing a "work submit" command, using "--param" in receptorctl. These params must have the correct ``allowruntime*`` fields specified in the ``work-kubernetes`` definition. diff --git a/docs/source/mesh.png b/docs/source/user_guide/mesh.png similarity index 100% rename from docs/source/mesh.png rename to docs/source/user_guide/mesh.png diff --git a/docs/source/remote.png b/docs/source/user_guide/remote.png similarity index 100% rename from docs/source/remote.png rename to docs/source/user_guide/remote.png diff --git a/docs/source/tls.rst b/docs/source/user_guide/tls.rst similarity index 97% rename from docs/source/tls.rst rename to docs/source/user_guide/tls.rst index 8311b3741..be9aee172 100644 --- a/docs/source/tls.rst +++ b/docs/source/user_guide/tls.rst @@ -4,9 +4,11 @@ TLS support Receptor supports mutual TLS authentication and encryption for above and below the mesh connections. - +.. contents:: + :local: + Configuring TLS -^^^^^^^^^^^^^^^ +--------------- Add ``tls-server`` and ``tls-client`` definitions to receptor config files. @@ -60,7 +62,7 @@ bar.yml ``myclient`` is referenced in ``tcp-peer``. Once started, `foo` and `bar` will authenticate each other, and the connection will be fully encrypted. Generating certs -^^^^^^^^^^^^^^^^ +----------------- Receptor supports X.509 compliant certificates. Although numerous tools can be used to generate valid certificates, receptor has a built-in tool to help with this process. Running receptor with the ``cert-init``, ``cert-makereq``, and ``cert-signreq`` actions will create certificate authorities, make requests, and sign requests, respectively. @@ -78,8 +80,8 @@ makecerts.sh The above script will create a CA, and for each node `foo` and `bar`, create a certificate request and sign it with the CA. These certs and keys can then be used to create ``tls-server`` and ``tls-client`` definitions in the receptor config files. -Pinned Certificates -^^^^^^^^^^^^^^^^^^^ +Pinned certificates +-------------------- In a case where a TLS connection is only ever going to be made between two well-known nodes, it may be preferable to require a specific certificate rather than accepting any certificate signed by a CA. Receptor supports certificate @@ -116,15 +118,15 @@ SHA256 and SHA512 fingerprints are supported. SHA1 fingerprints are not support Above the mesh TLS -^^^^^^^^^^^^^^^^^^ +------------------- Below-the-mesh TLS deals with connections that are being made to an IP address or DNS name, and so it can use normal X.509 certificates which include DNS names or IP addresses in their subjectAltName field. However, above-the-mesh TLS deals with connections whose endpoint addresses are receptor node IDs. This requires generating certificates that include receptor node IDs as names in the subjectAltName extension. To do this, the otherName field of subjectAltName can be utilized. This field is designed to accept arbitrary names of any type, and includes an ISO Object Identifier (OID) that defines what type of name this is, followed by arbitrary data that is meaningful for that type. Red Hat has its own OID namespace, which is controlled by RHANANA, the Red Hat Assigned Names And Number Authority. Receptor has an assignment within the overall Red Hat namespace. If you decide to consume TLS authentication in your mesh, the certificates OIDs (1.3.6.1.4.1.2312.19.1) will be verified against the `node.id` specified in the configuration file. If there is no match, the receptor binary will hard exit. If you need to get around this check, visit the `Skip Certificate Validation`_ section for more details. -Skip Certificate Validation -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Skip certificate validation +---------------------------- Depending on the specifics of your environment(s), if you need to turn off certificate validation, you can add a `skipreceptornamescheck` key-value pair in your configuration file for `tls-server`, `tls-config`, or both. The default behaviour for this option is `false` which means that the certificate's OIDs will be verified against the node ID. diff --git a/docs/source/workceptor.rst b/docs/source/user_guide/workceptor.rst similarity index 97% rename from docs/source/workceptor.rst rename to docs/source/user_guide/workceptor.rst index 5f5b894fb..b84af0e38 100644 --- a/docs/source/workceptor.rst +++ b/docs/source/user_guide/workceptor.rst @@ -3,6 +3,9 @@ Workceptor ========== +.. contents:: + :local: + Workceptor is a component of receptor that handles units of work. ``work-command`` defines a type of work that can run on the node. @@ -59,7 +62,7 @@ bar.yml Configuring work commands -^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------- ``worktype`` User-defined name to give this work definition @@ -69,7 +72,7 @@ Configuring work commands Local work -^^^^^^^^^^ +----------- Start the work by connecting to the ``control-service`` and issuing a "work submit" command @@ -82,7 +85,7 @@ Start the work by connecting to the ``control-service`` and issuing a "work subm Receptor started an instance of this work type, and labeled it with a unique "Unit ID" Work results -^^^^^^^^^^^^ +------------- Use the "Unit ID" to get work results @@ -102,7 +105,8 @@ Use the "Unit ID" to get work results Remote work -^^^^^^^^^^^ +------------ + Although connected to `foo`, by providing the "--node" option the work can be started on node `bar`. The work type must be defined on the node it is intended to run on, e.g. `bar` must have a ``work-command`` called "echoint", in this case. @@ -126,7 +130,8 @@ Sequence of events for remote work submission .. _work_payload: Payload -^^^^^^^^^^^^ +-------- + in `bar.yml` .. code-block:: yaml @@ -153,7 +158,7 @@ Note: "-f" instructs receptorctl to follow the work unit immediately, i.e. strea Runtime Parameters -^^^^^^^^^^^^^^^^^^ +------------------- Work commands can be configured to allow parameters to be passed to commands when work is submitted: @@ -194,7 +199,8 @@ extend the ``params`` work command setting. The ``--all`` flag can be passed to Work list -^^^^^^^^^ +---------- + "work list" returns information about all work units that have ran on this receptor node. The following shows two work units, ``12L8s8h2`` and ``T0oN0CAp`` .. code:: @@ -226,22 +232,22 @@ Notice that ``T0oN0CAp`` was a remote work submission, therefore its work type i Work cancel -^^^^^^^^^^^ +------------ Cancel will stop any running work unit. Upon canceling a "remote" work unit, the local node will attempt to connect to the remote node's control service and issue a work cancel. If the remote node is down, receptor will periodically attempt to connect to the remote node to do the cancellation. Work release -^^^^^^^^^^^^ +------------- Release will cancel the work and then delete files on disk associated with that work unit. For remote work submission, release will attempt to delete files both locally and on the remote machine. Like work cancel, the release can be pending if the remote node is down. In that situation, the local files will remain on disk until the remote node can be contacted. Work force-release -^^^^^^^^^^^^^^^^^^ +-------------------- It might be preferable to force a release, using the ``work force-release`` command. This will do a one-time attempt to connect to the remote node and issue a work release there. After this one attempt, it will then proceed to delete all local files associated with the work unit. States -^^^^^^^^^^^ +--------- A unit of work can be in Pending, Running, Succeeded, or Failed state @@ -250,7 +256,7 @@ For local work, transitioning from Pending to Running occurs the moment the ``co For remote work, transitioning from Pending to Running occurs when the status reported from the remote node has a Running state. Signed work -^^^^^^^^^^^^^^^^^^ +------------ Remote work submissions can be digitally signed by the sender. The target node will verify the signature of the work command before starting the work unit. @@ -295,7 +301,7 @@ Use the "--signwork" parameter to sign the work. $ receptorctl --socket /tmp/foo.sock work submit echoint --node bar --no-payload --signwork Units on disk -^^^^^^^^^^^^^^^^^^ +-------------- Netceptor, the main component of receptor that handles mesh connectivity and traffic, operates entirely in memory. That is, it does not store any state information on disk. However, Workceptor functionality is designed to be persistent across receptor restarts. Work units might be running commands that could take hours to complete, and as such needs to store some relevant information on disk in case the receptor process restarts. @@ -378,5 +384,6 @@ The main purpose of work unit ``BsAjS4wi`` on `foo` is to copy stdin, stdout, an .. image:: remote.png + :alt: sequence of events during work remote submission The sequence of events during a work remote submission. Blue lines indicate moments when receptor writes files to disk.