Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Makefile for automating localnet setup #3718

Merged
merged 14 commits into from
Dec 6, 2019
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true

[Makefile]
indent_style = tab

[*.bat]
end_of_line = crlf

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,4 @@ deploy
/monitor/TorHiddenServiceStartupTimeTests/*
/monitor/monitor-tor/*
.java-version
.localnet
251 changes: 251 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
#
# INTRODUCTION
#
# This makefile is designed to help Bisq contributors get up and running
# as quickly as possible with a local regtest Bisq network deployment,
# or 'localnet' for short. A localnet is a complete and self-contained
# "mini Bisq network" suitable for development and end-to-end testing
# efforts.
#
#
# REQUIREMENTS
#
# You'll need the following to proceed:
#
# - Linux, macOS or similar *nix with standard tools like `make`
# - bitcoind and bitcoin-cli (`brew install bitcoin` on macOS)
# - JDK 10 to build and run Bisq binaries; see
# https://www.oracle.com/java/technologies/java-archive-javase10-downloads.html
#
#
# USAGE
#
# The following commands (and a couple manual instructions) will get
# your localnet up and running quickly.
#
# STEP 1: Build all Bisq binaries and set up localnet resources. This
# will take a few minutes the first time through.
#
# $ make
#
# Notes:
#
# - When complete, you'll have a number of scripts available in the
# root directory. They will be used in the make targets below to start
# the various Bisq seed and desktop nodes that will make up your
# localnet:
#
# $ ls -1 bisq-*
# bisq-desktop
# bisq-monitor
# bisq-pricenode
# bisq-relay
# bisq-seednode
# bisq-statsnode
#
# - You will see a new '.localnet' directory containing the data dirs
# for your regtest Bitcoin and Bisq nodes. Once you've deployed them in
# the step below, the directory will look as follows:
#
# $ tree -d -L 1 .localnet
# .localnet
# ├── alice
# ├── bitcoind
# ├── bob
# ├── mediator
cbeams marked this conversation as resolved.
Show resolved Hide resolved
# ├── seednode
# └── seednode2
#
# STEP 2: Deploy the Bitcoin and Bisq nodes that make up the localnet.
# Run each of the following in a SEPARATE TERMINAL WINDOW, as they are
# long-running processes.
#
# $ make bitcoind
# $ make seednode
# $ make seednode2
# $ make mediator
# $ make alice
# $ make bob
#
# Tip: Those familiar with the `screen` terminal multiplexer can
# automate the above by running the `deploy` target found below.
#
# Notes:
#
# - The 'seednode' targets launch headless Bisq nodes that help
# desktop nodes discover other peers, as well as storing and
# forwarding p2p network messages for nodes as they go on and
# offline.
#
# - As you run the 'mediator', 'alice' and 'bob' targets above,
# you'll see a Bisq desktop node window appear for each. The Alice
# and Bob instances represent two traders who can make and take
# offers with one another. The Mediator instance represents a Bisq
# contributor who can help resolve any technical problems or disputes
# that come up between the two traders.
#
# STEP 3: Configure the mediator Bisq node. In order to make and take
# offers, Alice and Bob will need to have a mediator and a refund agent
# registered on the network. Follow the instructions below to complete
# that process:
#
# a) Go to the Account screen in the Mediator instance and press CMD+N
# and a popup will appear. Click 'Unlock' and then click 'Register' to
# register the instance as a mediator.
#
# b) While still in the Account screen, press CMD+D and follow the same
# steps as above to register the instance as a refund agent.
#
# When the steps above are complete, your localnet should be up and
# ready to use. You can now test in isolation all Bisq features and use
# cases.
#

# Set up everything necessary for deploying your localnet. This is the
# default target.
setup: build .localnet

clean: clean-build clean-localnet

clean-build:
./gradlew clean

clean-localnet:
rm -rf .localnet ./dao-setup

# Build Bisq binaries and shell scripts used in the targets below
build: seednode/build desktop/build

seednode/build:
./gradlew :seednode:build

desktop/build:
./gradlew :desktop:build

# Unpack and customize a Bitcoin regtest node and Alice and Bob Bisq
# nodes that have been preconfigured with a blockchain containing the
# BSQ genesis transaction
.localnet:
# Unpack the old dao-setup.zip and move things around for more
# concise and intuitive naming. This is a temporary measure until we
# clean these resources up more thoroughly.
unzip docs/dao-setup.zip
mv dao-setup .localnet
mv .localnet/Bitcoin-regtest .localnet/bitcoind
mv .localnet/bisq-BTC_REGTEST_Alice_dao .localnet/alice
mv .localnet/bisq-BTC_REGTEST_Bob_dao .localnet/bob
# Remove the preconfigured bitcoin.conf in favor of explicitly
# parameterizing the invocation of bitcoind in the target below
rm -v .localnet/bitcoind/bitcoin.conf
# Avoid spurious 'runCommand' errors in the bitcoind log when nc
# fails to bind to one of the listed block notification ports
echo exit 0 >> .localnet/bitcoind/blocknotify

# Alias '.localnet' to 'localnet' so the target is discoverable in tab
# completion
localnet: .localnet

# Deploy a complete localnet by running all required Bitcoin and Bisq
# nodes, each in their own named screen window. If you are not a screen
# user, you'll need to manually run each of the targets listed below
# commands manually in a separate terminal or as background jobs.
deploy: setup
# create a new screen session named 'localnet'
screen -dmS localnet
# deploy each node in its own named screen window
targets=('bitcoind' 'seednode' 'seednode2' 'alice' 'bob' 'mediator'); \
Copy link
Contributor

@julianknutsen julianknutsen Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Makefile didn't specify the SHELL variable so mine defaulted to /bin/sh which doesn't support the array syntax here. Not sure if there is a shell agnostic way to do it, but adding SHELL=/bin/bash to the Makefile fixed it locally.

julian@dev:~/bisq$ make deploy
# create a new screen session named 'localnet'
screen -dmS localnet
# deploy each node in its own named screen window
targets=('bitcoind' 'seednode' 'seednode2' 'alice' 'bob' 'mediator'); \
for t in "${targets[@]}"; do \
	screen -S localnet -X screen -t $t; \
	screen -S localnet -p $t -X stuff "make $t\n"; \
done;
/bin/sh: 1: Syntax error: "(" unexpected
make: *** [Makefile:156: deploy] Error 2

The iterative single-node behavior also doesn't seem to work quite right. I checked out an older version of the code, killed bob, ran make bob and it just restarted without a build.

I also ran make desktop/build and no changes, but ./gradlew :desktop:build had something to do.

dependency_issue

Even making alice & bob have a dependency on desktop/build instead of just setup didn't work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Makefile didn't specify the SHELL variable so mine defaulted to /bin/sh which doesn't support the array syntax here. Not sure if there is a shell agnostic way to do it, but adding SHELL=/bin/bash to the Makefile fixed it locally.

Good catch, thanks @julianknutsen. Commit 234c228 removes the offending bashism such that /bin/sh should run without error.

The iterative single-node behavior also doesn't seem to work quite right. I checked out an older version of the code, killed bob, ran make bob and it just restarted without a build.

You're right, and I should have mentioned this in #3718 (comment). The rationale for why things need to be this way is laid out in 5fb4b21. Search for the first occurrence of the word 'contention' there. I'm updating the comment above now.

Copy link
Contributor

@julianknutsen julianknutsen Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the commit link and it makes sense w.r.t. contention. Verified the bashism issue is fixed as well and will do 1.2.4 testing today with your latest code and call out any other glaring issues. So expect an ACK by EOD from me if everything works out.

Copy link

@bodymindarts bodymindarts Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbeams thanks for your instructive commit messages!
I understand your reasoning behind removing build from PHONY. I'm just wondering wether there is a better solution. Doing a clean-rebuild takes a very long time and is not practical for quick iteration cycles. Essentially the problem you have solved is a race condition. Couldn't we ensure somehow that the build runs just once before deploying the individual nodes. Perhaps running the individual node commands shouldn't depend on the setup/build commands. That way we could have both a 1 time setup + iteration.
Is there a significant advantage of having all the individual node commands depend on setup/build that I'm missing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing a clean-rebuild takes a very long time and is not practical for quick iteration cycles.

Here's what I'm doing in practice as I develop stuff. The following assumes I have already done a make deploy, and that now I'm iterating on something, using my alice desktop node to see my changes live:

# quit my running `alice` node, e.g. with CMD-Q in the UI or ^C on the running process
./gradlew :desktop:build && make alice

This picks up and rebuilds just the changes I've made and then deploys those new bits as alice.

Couldn't we ensure somehow that the build runs just once before deploying the individual nodes.

It does effectively run just once. The individual node deployment targets, e.g. alice and bob depend on build. Note that build is, in the latest commits, actually included once again in PHONY, so it does run every time, but that target just depends on the more specific desktop/build and seednode/build targets, which in turn run only if their respective directories do not already exist. The effect here is that when we deploy more than one node or all of them via make deploy, the {desktop|seednode}/build targets get run once and only once, avoiding the above mentioned race condition and inefficient contention for Gradle resources. If you want to cause a fast (incremental) rebuild, it's simply necessary to drop down to calling gradle directly like I've shown above. calling make clean-build should not be necessary in any case, unless you actually want to blow away all the build directories.

Is there a significant advantage of having all the individual node commands depend on setup/build that I'm missing?

It just ensures that deploying any given node causes build and localnet to run if they have not already done so. So someone can come along in a clean checkout and run only make bitcoind and make seednode and everything will work as they expect, meaning that the .localnet dir will get created and the seednode build will run. If that's already happened, then those targets are no-ops.

for t in "$${targets[@]}"; do \
screen -S localnet -X screen -t $$t; \
screen -S localnet -p $$t -X stuff "make $$t\n"; \
done;
# give bitcoind rpc server time to start
sleep 5
# generate a block to ensure Bisq nodes get dao-synced
make block

bitcoind: .localnet
bitcoind \
-regtest \
-prune=0 \
-txindex=1 \
-peerbloomfilters=1 \
-server \
-rpcuser=bisqdao \
cbeams marked this conversation as resolved.
Show resolved Hide resolved
-rpcpassword=bsq \
-datadir=.localnet/bitcoind \
-blocknotify='.localnet/bitcoind/blocknotify %s'

seednode: seednode/build
./bisq-seednode \
--baseCurrencyNetwork=BTC_REGTEST \
--useLocalhostForP2P=true \
--useDevPrivilegeKeys=true \
--fullDaoNode=true \
--rpcUser=bisqdao \
--rpcPassword=bsq \
--rpcBlockNotificationPort=5120 \
--nodePort=2002 \
--userDataDir=.localnet \
--appName=seednode

seednode2: seednode/build
./bisq-seednode \
--baseCurrencyNetwork=BTC_REGTEST \
--useLocalhostForP2P=true \
--useDevPrivilegeKeys=true \
--fullDaoNode=true \
--rpcUser=bisqdao \
--rpcPassword=bsq \
--rpcBlockNotificationPort=5121 \
--nodePort=3002 \
--userDataDir=.localnet \
--appName=seednode2

mediator: desktop/build
./bisq-desktop \
--baseCurrencyNetwork=BTC_REGTEST \
--useLocalhostForP2P=true \
--useDevPrivilegeKeys=true \
--nodePort=4444 \
--appDataDir=.localnet/mediator \
--appName=Mediator

alice: setup
./bisq-desktop \
--baseCurrencyNetwork=BTC_REGTEST \
--useLocalhostForP2P=true \
--useDevPrivilegeKeys=true \
--nodePort=5555 \
--fullDaoNode=true \
--rpcUser=bisqdao \
--rpcPassword=bsq \
--rpcBlockNotificationPort=5122 \
--genesisBlockHeight=111 \
--genesisTxId=30af0050040befd8af25068cc697e418e09c2d8ebd8d411d2240591b9ec203cf \
--appDataDir=.localnet/alice \
--appName=Alice

bob: setup
./bisq-desktop \
--baseCurrencyNetwork=BTC_REGTEST \
--useLocalhostForP2P=true \
--useDevPrivilegeKeys=true \
--nodePort=6666 \
--appDataDir=.localnet/bob \
--appName=Bob

# Generate a new block on your Bitcoin regtest network. Requires that
# bitcoind is already running. See the `bitcoind` target above.
block:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could use another sentence or so in the preamble. You end up needing to create blocks to test features like governance so helping new users fix common errors like "I created a proposal from Alice, but it isn't visible on Bob. Why not?" may help the onboarding.

bitcoin-cli \
-regtest \
-rpcuser=bisqdao \
-rpcpassword=bsq \
getnewaddress \
| xargs bitcoin-cli \
-regtest \
-rpcuser=bisqdao \
-rpcpassword=bsq \
generatetoaddress 1

.PHONY: build seednode