Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documentation for cluster mode #1646

Merged
merged 3 commits into from
Dec 18, 2020
Merged

Documentation for cluster mode #1646

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6b3ce8b
add cluster doc, and packaging improvements
pm47 Dec 16, 2020
c2f62de
fixup! add cluster doc, and packaging improvements
pm47 Dec 17, 2020
aeb4947
make system props take precedence over eclair.conf
pm47 Dec 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 177 additions & 0 deletions docs/Cluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
## How to _clusterize_ your Eclair node

Eclair allows you to scale up one _logical_ lightning node across multiple servers.

Front servers take care of routing table related gossip and syncing requests from peers, which is cpu/bandwidth intensive. The backend server can focus on core channel management. So, BOLT 1&7 messages are handled in the frontend, while BOLT 2 messages go through and are processed in the backend.

Front servers are stateless, they can be stopped/killed at will. The node will remain operational and reachable as long as there is at least one `frontend` available.

```
+---+ +-----------+
| | | +-------+ |
| |-----|-| | |
P -----| L |-----|-| FRONT |-|---,
U | O |-----|-| | | \
B ---| A | | +-------+ | \
L | D | | | \
I | | | +-------+ | \ +------+
C -----| B |-----|-| | | `| |
| A |-----|-| FRONT |-|---------| BACK |<-- channels management
N | L |-----|-| | | ,| | relay logic
E ----| A | | +-------+ | / +------+
T | N | | | /
W | C | | +-------+ | /
O | E |-----|-| | | /
R -----| R |-----|-| FRONT |-|---'
K | |-----|-| |<------- connection management
| | | +-------+ | routing table sync
+---+ +-----------+
```

The goal is to offload your node from connection and routing table management:
- incoming connections
- outgoing connections
- gossip queries + pings
- incoming gossip aggregation
- outgoing gossip dispatch (rebroadcast)

### Prerequisite

You already have a node up and running in a standalone setup (with Bitcoin Core properly configured, etc.).

You know what your `node id` is.

In the following, what we previously called `eclair-node` will be called `backend`. It is to be launched, configured and backed-up exactly like in a standalone setup.

### Minimal/Demo setup

Use this if you want to experiment with the cluster mode on a single local server.

Set the following values in `.eclair/eclair.conf`:
```
akka.actor.provider = cluster
akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"]

// replace this with your node id
// if you don't know what your node id is, you should probably stop right here
eclair.front.pub = 03...............
```

Start the `backend`:
```shell
$ ./eclair-node.sh
```

Then run an instance of `frontend`:
```shell
$ ./eclair-front.sh -Dakka.remote.artery.canonical.port=25521 -Declair.server.port=9736
```

NB: we override the ports, otherwise they would conflict since in this example everything runs on the same server. You can run multiple `frontend`s on the same server, just make sure to change the ports.

### Production setup

In production you should:
- run multiple `frontend`s
- run one app per server
- enable `tcp-tls` to encrypted between members of the cluster with your own generated certificate (see below)
pm47 marked this conversation as resolved.
Show resolved Hide resolved
- use a load balancer to hide all your `frontend` servers under the same ip address
- set firewall rules to disable lightning connections (port 9735) on your `backend` server, so all connections go through the `frontend`
- enable [monitoring](Monitoring.md)
- on AWS, use AWS Secrets Manager (see [AWS deployment](#aws-deployment))

#### Enable encrypted communication for the cluster

We use a self-signed certificate, which offers a good compromise. More advanced options are available, see [akka doc](https://doc.akka.io/docs/akka/current/remoting-artery.html#remote-security).
> Have a single set of keys and a single certificate for all nodes and disable hostname checking
> - The single set of keys and the single certificate is distributed to all nodes. The certificate can be self-signed as it is distributed both as a certificate for authentication but also as the trusted certificate.
> - If the keys/certificate are lost, someone else can connect to your cluster.
> - Adding nodes to the cluster is simple as the key material can be deployed / distributed to the new node.

Generate a self-signed certificate (set a strong password):
```shell
$ keytool -genkeypair -v \
-keystore akka-cluster-tls.jks \
-dname "O=ACME, C=FR" \
-keypass:env <password> \
-storepass:env <password> \
-keyalg RSA \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you use RSA and not elliptic curves (which are safer, faster and create smaller certificates)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used one of the recommended algorithms in akka doc, but advanced users can change if they prefer.

-keysize 4096 \
-validity 9999
```

Copy the resulting certificate to your `.eclair` directory:
```shell
$ cp akka-cluster-tls.jks ~/.eclair
```

Add this to your `eclair.conf`:
```
AKKA_TLS_PASSWORD=<password>
akka.remote.artery.transport = "tls-tcp"
```

### AWS Deployment

For convenience, we provide a prebuilt AWS Beanstalk bundle for the `frontend` (choose a WebServer environment type, and Java platform).

You can run it as-is for testing.

#### TLS encryption

If you intend to use it in production, you need to enable encryption with your own certificate:
1. Follow the procedure above to generate your `akka-tls.jks`

2. We recommend forking the project and build your own bundle:
pm47 marked this conversation as resolved.
Show resolved Hide resolved
```shell
$ git clone git@github.com:ACINQ/eclair.git
$ vi eclair-core/src/main/reference.conf # set akka.remote.artery.transport = "tls-tcp"
$ cp akka-cluster-tls.jks eclair-front/modules/awseb/ # copy the file you generated
$ vi eclair-front/modules/awseb.xml # uncomment the relevant parts
$ mvn package -DskipTests
```
Alternatively, you can also edit the existing bundle and manually add the `akka-cluster-tls.jks` file to the root of the zip archive. You will also need to set `akka.remote.artery.transport=tls-tcp` at runtime.

#### Private key

In production, we highly recommend using AWS Secrets manager to provide the node private key. This is done by setting `eclair.front.priv-key-provider=aws-sm`. Default secret name is "node-priv-key", but it is configurable with `eclair.front.aws-sm.aws-sm.priv-key-name`
pm47 marked this conversation as resolved.
Show resolved Hide resolved

#### Configuration

We recommend using Beanstalk environment variables for `AKKA_TLS_PASSWORD`, `BACKEND_IP`, and `NODE_PUB_KEY`. Other configuration keys should be set in the `AKKA_CONF` environment variable, semicolon separated. Example:
- `AKKA_CONF`: `eclair.enable-kamon=true; akka.remote.artery.transport=tls-tcp; eclair.front.priv-key-provider=aws-sm...`
- `AKKA_TLS_PASSWORD`: `xxxxxxxx`
- `BACKEND_IP`: `1.2.3.4`
- `NODE_PUB_KEY`: `03933884aaf1d6b108397e5efe5c86bcf2d8ca8d2f700eda99db9214fc2712b134`

Port `25520` needs to be open, within the Beanstalk security group, and between the Beanstalk security group and your `backend` node.

### Tor

We recommend running your Tor hidden service on a separate server, and use `eclair.tor.targets` to redirect clearnet(*) connections to your `frontend` servers.

(*) Clearnet for Tor, but BOLT 8 encrypted.

Here is the resulting architecture:

```
+---+ +-----------+
| | | +-------+ |
| |-----|-| | |
P -----| L |-----|-| FRONT |-|---,
U | O |-----|-| | | \
B ---| A | | +-------+ | \
L | D | | | \
I | | | +-------+ | \ +------+
C -------| B |-----|-| | | `| |
| A |-----|-| FRONT |-|---------| BACK |<-- channels management
N | L |-----|-| | | ,| | relay logic
E ----| A | | +-------+ | / +------+
T | N | | | /
W +-------+ | C | | +-------+ | /
O | | | E |-----|-| | | /
R ---| Tor |------| R |-----|-| FRONT |-|---'
K | | | |-----|-| |<------- connection management
+-------+ | | | +-------+ | routing table sync
+---+ +-----------+
```
36 changes: 5 additions & 31 deletions eclair-front/modules/assembly.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>${git.commit.id.abbrev}-awseb_bundle</id>
<id>bin</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets> <!-- include dependencies -->
<dependencySet>
<outputDirectory>lib</outputDirectory>
Expand All @@ -22,39 +21,14 @@
<include>LICENSE*</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>Procfile</include>
</includes>
<lineEnding>unix</lineEnding>
</fileSet>
<!-- uncomment if you want to include a keystore in the awseb package for TLS -->
<!--fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>*.jks</include>
</includes>
<fileMode>0400</fileMode>
<lineEnding>keep</lineEnding>
</fileSet-->
<fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<fileSet> <!-- Include the launcher scripts -->
<directory>${project.basedir}/src/main/resources</directory>
<outputDirectory>bin</outputDirectory>
<includes>
<include>run.sh</include>
<include>eclair-front.sh</include>
</includes>
<fileMode>0755</fileMode>
<lineEnding>unix</lineEnding>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>application.jar</include>
</includes>
</fileSet>
</fileSets>
</assembly>
46 changes: 46 additions & 0 deletions eclair-front/modules/awseb.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>${git.commit.id.abbrev}-awseb_bundle</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet> <!-- release zip -->
<directory>${project.build.directory}</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>eclair-front-*-bin.zip</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>Procfile</include>
<include>logback_eb.xml</include>
</includes>
<lineEnding>unix</lineEnding>
</fileSet>
<!-- uncomment if you want to include a keystore in the awseb package for TLS -->
<!--fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>*.jks</include>
</includes>
<fileMode>0400</fileMode>
<lineEnding>keep</lineEnding>
</fileSet-->
<fileSet>
<directory>${project.basedir}/modules/awseb</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>run.sh</include>
</includes>
<fileMode>0755</fileMode>
<lineEnding>unix</lineEnding>
</fileSet>
</fileSets>
</assembly>
45 changes: 45 additions & 0 deletions eclair-front/modules/awseb/logback_eb.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2019 ACINQ SAS
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<configuration>

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<!-- this filters out AWS ELB probes -->
<evaluator>
<expression>
return formattedMessage.contains("connected to /10.") ||
(formattedMessage.contains("connection closed") &amp;&amp; !mdc.containsKey("nodeId")) ||
(formattedMessage.contains("transport died") &amp;&amp; !mdc.containsKey("nodeId")) ||
(formattedMessage.contains("stopping") &amp;&amp; !mdc.containsKey("nodeId"));
</expression>
</evaluator>
<OnMismatch>NEUTRAL</OnMismatch>
<OnMatch>DENY</OnMatch>
</filter>
<target>System.out</target>
<withJansi>false</withJansi>
<encoder>
<pattern>${HOSTNAME} %d %-5level %logger{24}%X{category}%X{nodeId}%X{channelId}%X{paymentHash}%.-11X{parentPaymentId}%.-11X{paymentId} - %msg%ex{12}%n</pattern>
</encoder>
</appender>

<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>

</configuration>
10 changes: 9 additions & 1 deletion eclair-front/modules/awseb/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
export LOCAL_IP=$(curl -s 169.254.169.254/latest/meta-data/local-ipv4)
export HOSTNAME=$(hostname)
exec java -javaagent:lib/kanela-agent-1.0.5.jar -jar application.jar

# make the eclair home directory
mkdir -p /home/webapp/.eclair

# if provided, copy the certificate to the proper directory
[[ -e akka-cluster-tls.jks ]] && cp -v akka-cluster-tls.jks /home/webapp/.eclair/

unzip -o eclair-front-*-bin.zip
exec ./eclair-front-*/bin/eclair-front.sh -with-kanela -Dlogback.configurationFile=logback_eb.xml
5 changes: 2 additions & 3 deletions eclair-front/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,9 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<finalName>application</finalName>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>fr.acinq.eclair.Boot</mainClass>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
</manifest>
<manifestEntries>
Expand All @@ -55,8 +52,10 @@
<artifactId>maven-assembly-plugin</artifactId>
<version>3.2.0</version>
<configuration>
<finalName>${project.name}-${project.version}-${git.commit.id.abbrev}</finalName>
<descriptors>
<descriptor>modules/assembly.xml</descriptor>
<descriptor>modules/awseb.xml</descriptor>
</descriptors>
</configuration>
<executions>
Expand Down
Loading