2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

ChristopherA · 2020-06-06T22:14:47Z

Currently the linux scripts for Bitcoin Standup Scripts are fairly simple, and establish just a single node.

This script was originally based on the script in Learning Bitcoin from the Command Line as I found it very inexpensive and easy for a novice to install a Bitcoin testnet full-node with tx-index on Linode by simply pasting a single stackscript and waiting a half-day. This node would be free for the first month (and Blockchain Commons gets a $10 credit if they use the referral code), and only $5 - $10 a month thereafter. When I had taught it before with people laptops we had all kinds of problems, and this solved it.

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

This will also help us in the future as hosting a Lightning Network requires a full node. See Issue #7.

There are other improvements that can be made to the Bitcoin Standup scripts, including:

Review @ NicolasDorier scripts in the @btcpayserver community, both code and docs https://docs.btcpayserver.org/ManualDeploymentExtended/ for best practices for our use as well as submit changes back.
Consider adding option for Tor .onion based BitcoinFibre https://github.com/bitcoinfibre to allow faster sync between Bitcoin Standup full nodes (this is also used by Blockstream Satellite).
Review Blockstream Satellite https://github.com/Blockstream/bitcoinsatellite/ code and docs. * [ ] hosting multiple bitcoin networks like the yesterday's version of Bitcoin Standup MacOS.
Allowing alternative versions of Bitcoin-core to be installed, such as @luke-jr's Bitcoin Knots https://github.com/bitcoinknots/bitcoin or @kallewoof's Signet https://github.com/kallewoof/signet-platform. In particular, the latter can help simulate adversarial conditions on the Bitcoin network to better test future versions of FullyNoded against.
Supporting some form of pre-signed download of pruned full-node data like BTCPay offers as an option.
The ability to host some other simple bitcoin-related services using the nodes .onion
Can the new bitcoin-core Hardware Wallet Interface (HWI) https://github.com/bitcoin-core/HWI code can't be used on a VPS (I don't think) but could be useful for people installing on their own machine.
We need some process for testing and keeping these scripts updated as new releases are announced, as well as updating existing nodes.

We also should release scripts to work standalone for Mac (for those who don't want to use the Bitcoin Standup Mac app) and Windows (a big challenge).

Eventually, after thorough testing, these scripts should audited formally by people in the bitcoin-core community.

What other features am I missing?

-- Christopher Allen

chiselinc · 2020-06-08T00:03:26Z

As per our discussion @ChristopherA , I'm "splitting out" the task of "release scripts to work standalone for Windows" into its own designated issue, and claiming "point person" for that deliverable.

I feel confident enough poking around existing scripts, and whether I like it or not, some of my grad work has required me to commit in-depth to Windows literacy. I'll still always fall back to Linux when I have a choice, but the wrangling experience will likely prove valuable!

gg2001 · 2020-06-08T00:36:12Z

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

Linode has block storage which can be attached to a linode. 320GiB costs $32/month and 640GiB costs $64/month ($0.10/GiB per month). Using 640GiB attached to a 4GB RAM VM would cost $84 a month. DigitalOcean also has block storage at a similar price.

OVH seems to have cheaper options: VPS/block storage, but I haven't looked into the specifics of the pricing. Also AWS has HDD storage, which may turn out to be cheaper than the previous options, but the unpredictable pricing would be a problem.

jodobear · 2020-06-08T18:35:03Z

Proposal

Tasks

Improve Standup Script with best practices inspired by BTCPay Server.
Add pruned node setup options.
Add options to get deployment environment to make it deployable on VPS/Desktop/RPi with one script.
Add LND script with RTL(for node management) and then add option for c-lightning (It would be better to add them both together but, haven't looked at c-lighting. Need to assess.).
Break up the script for better management of code.
Add option for luke-jr's bitcoin knots and kallewoof's signet
Add option to use pre-signed blockchain snapshot for faster sync.

Enhancements and Other Services

Add BTCPay, Mempool.space, Electrs setup options based on discussed priority.
If possible within the timeframe, dockerize the setup of each service.
Create a simple GUI for installation.

Questions

What other services are desirable e.g. Joinmarket, Whirlpool, Ronin Dojo? What other .onion services would be a nice addition?
Is Tor-to-IP tunnel and Reverse SSH useful addition?
Ideas about how to adapt this script to be deployable on Windows and ~~Linux~~ Mac OS.

Hosting Data

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

AWS S3 upto 50 TB:

Frequent access/Intelligent tiering = $0.023/GB ~ $8.05/350GB
Infrequent access i.e. min 30 days of no change in data = $0.0125/GB
Glacier min 90 days no change, 1min-12 hour access to data = $0.004/GB
Deep Glacier 1-2 times access/yr & 12 hr retreival time of data = $0.00099/GB
Keybase has 250GB/individual user storage space available for free.
Q: Does it make sense to create a torrent with pre-signed hash by and getting it signed by many people in the ecosystem?

jodobear · 2020-06-08T19:55:47Z

Additionally, scope how to incorporate BitcoinFiber for faster sync between nodes and Blockstream Sattelite for hosting multiple networks.

ChristopherA · 2020-06-08T20:38:15Z

Hosting Data

BTW, to be clear, Blockchain Commons is willing to pay for full hosting. In fact, I'd like to eventually also offer full Esplora (aka Blockstream.info). I just want to allow others to be able to follow us by offering redundant services more cheaply.

Another point of interest. Every time someone uses my Bitcoin course and sets up a Linode to give it a try, we get a $10 credit. So I have a bunch of unused credit on Linode. That, and that is is very easy for a new developer to set up a StackScript (as opposed to the much more complicated deployment systems), is why we currently use Linode. However, we don't have to.

Ultimately I'd like Blockchain Commons to be able to have 3 redundant hosts around the world.

-- Christopher Allen

ChristopherA · 2020-06-08T20:40:57Z

What other services are desirable?

Good question.

I know that we already use Esplora via Blockstream.info, and it is probably good for us to host an Esplora instance of our own.

Maybe a survey of each of these requirements? Also how stable are they? So far our scripts for self-hosting and our devops are an all volunteer group, so we have to consider ongoing support costs.

-- Christopher Allen

jodobear · 2020-06-10T01:22:35Z

Maybe a survey of each of these requirements? Also how stable are they?

Esplora needs 440GB after compaction (~1TB while indexing) and 270GB at 620k block height in --lightmode,

jodobear · 2020-06-10T23:32:31Z

Fonta1n3 · 2020-06-11T03:31:37Z

@jodobear

Proposed Milestones

Add Help and merge Linode & Linux scripts and document the environment variables with additional prune mode and FastSync (if we agree on using BTCPay data).

This should certainly be number 1 priority.
Number two priority should be:

align the scripts with Standup.app MacOS. As of the latest update there was a major change to how we create hidden services, we now create a seperate HS for all three rpcports so a user can run all three simultaneously and connect to all three remotely via independent HS. See the macos linux script here line 70 - 108. Currently the scripts for linux/vps create one HS that controls all three rpcports.

Add Esplora, HWI and BTCPay to the Stackscript.

I feel Esplora is too cumbersome for a VPS, can we make it optional?

Add update.sh for the node.

Incorporate LN integration (LND & c-lightning). (Dependency 2020 Summer Internship Project: Lightning Support for Bitcoin Standup #7 )

I personally prefer c-lightning so would like to see that take priority, but beggars can't be choosers ;-)

Add the Price History API service once it is complete. (Dependency 2020 Summer Internship Project: Bitcoin Price & History Data #6)

Add option for luke-jr's bitcoin knots and kallewoof's signet.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

Question: Do we want to add BTCPay Server to this script?

I think that would be great.

jodobear · 2020-06-14T19:03:03Z

align the scripts with Standup.app MacOS. As of the latest update there was a major change to how we create hidden services, we now create a seperate HS for all three rpcports so a user can run all three simultaneously and connect to all three remotely via independent HS. See the macos linux script here line 70 - 108. Currently the scripts for linux/vps create one HS that controls all three rpcports.

Meaning modify how we create HS for linux accordingly? If so, sure!

I feel Esplora is too cumbersome for a VPS, can we make it optional?

Agreed and yes it will be optional. Infact, the way I plan to implement it is only bitcoind will be the base installation, everything else will be optional based on user preference.

I personally prefer c-lightning so would like to see that take priority, but beggars can't be choosers ;-)

I have never worked with c-lightning, and this will be a great motivation for me to learn how to c-lightning. Since, LND is very easy, i will start with that and then also integrate c-lightning option or the other way round. I will look into it.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

I only added those since @ChristopherA mentioned it. But, i feel these are low priority and there's a lot to do before that. We can take a call on them at a later date.

What other services would you like to be added to the scripts?

ChristopherA · 2020-06-14T23:35:13Z

I feel Esplora is too cumbersome for a VPS, can we make it optional?

Yes, it is unlikely our typical BitcoinStandup VPS users will want to do Esplora, but I know we will want to do our own as it is needed for some future things like BTCR.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

Knots has the lowest priority, but I'd love to be able demonstrate that we can, and get @luke-jr's endorsement.

Signet is potential much more useful for us, as I have concerns about FN2 performing well under adversarial conditions or if full nodes are tied up in forks and such. Signet can simulate those, making me feel much more confident about FN2 on mainnet.

Fibre could potentially also be very useful once we have more people doing BitcoinStandup, as we can lower latency between associated nodes. It also is a prerequisite for Blockstream Satellite.

--Christopher Allen

jodobear · 2020-06-14T23:52:52Z

Knots has the lowest priority, but I'd love to be able demonstrate that we can, and get @luke-jr's endorsement.

I like this idea, will be done! 😁

javiervargas · 2020-06-24T15:37:01Z

When i installed standup.sh for using FullyNoded2 with my own mainnet node I don't needed perform IBD, so in addition to the first option to install all stuff (including two steps for c-lightning),

sudo apt-get install -y software-properties-common
sudo add-apt-repository -u ppa:lightningnetwork/ppa
sudo apt-get install lightningd

I think it is a good idea to give the user different options depending on their situation. Perhaps creating a menu that calls different functions and executes the pre-checks is an ideal approach.

Something like this:

~~~~~~~~~~~~~~~~~~~~~
  BitcoinStandup.sh  
~~~~~~~~~~~~~~~~~~~~~
0. Install all
1. Install Tor
2. Install tools
3. Debian updates
4. Install ufw
5. Install StandUp User
6. Install Bitcoin core
7. Create and show QR Code
8. Install c-lightning
9. Exit
Enter choice [ 1 - 9]

Script standup.sh

show_menus() {
 clear
 echo "~~~~~~~~~~~~~~~~~~~~~"
 echo "  BitcoinStandup.sh  "
 echo "~~~~~~~~~~~~~~~~~~~~~"
 echo "0. Install all"
 echo "1. Install Tor"
 echo "2. Install tools"
 echo "3. Debian updates"
 echo "4. Install ufw"
 echo "5. Install StandUp User"
 echo "6. Install Bitcoin core"
 echo "7. Create and show QR Code"
 echo "8. Install c-lightning"
 echo "9. Exit"
}

read_options(){
 local choice
 read -p "Enter choice [ 1 - 9] " choice
 case $choice in
  1) one ;;
  2) user_info "who" ;;
  3) debian_updates ;;
  9) exit 0;;
  *) echo -e "${RED}Error...${STD}" && sleep 2
 esac
}

debian_updates(){
 echo "debian"
        pause
}

Waiting your comments.
J.

jodobear · 2020-06-24T16:53:47Z

@javiervargas Would this require displaying the main menu repeatedly to install each service/item?

Here's how i'm thinking to implement the structure:

UX

Display default installation info and give option for that or configure install
If default then install.
Else give one by one options for:
1. Tor
2. SSH
3. Bitcoin - version (core, knots, signet), chain, prune, FastSync, txindex
4. LN - LND, c-lightning
5. HWI
6. Esplora
7. BTCPay Server
Core components no matter the config:
1. UFW
2. QuickConnect

All of these options can be given as arguments to the script for a one liner.

Default Installation

All communication defaults through Tor. Services:

UFW
Tor
SSH
Bitcoin- bitcoin-core, full node, mainnet, FastSync=False, txindex=1
QuickConnect
LN - c-lightning
HWI
BTCPay Server

javiervargas · 2020-06-24T18:33:22Z

@jodobear Not exactly, the menu is in a do while cycle that you can interrupt at any time with ctrl-c.

jodobear · 2020-06-24T21:28:45Z

Ok, I'm currently deconstructing it, laying down best practices and putting the variables in place. I'll probably push a basic structure by this weekend and we can take it from there

jodobear · 2020-07-06T15:07:27Z

I've merged the scripts, added help, documentation, checks for each step and installing bitcoin from onion site:

ss.sh

It's tested on Debian 10 VM. Everything works and is setup properly.

Still need to test on VPS, don't have access at the moment.

Please provide feedback if i can do it better or making any mistakes.

shannona · 2020-11-04T00:05:22Z

Closing out summer 2020 issues. Thanks to @jodobear for work on the Standup Scripts!

ChristopherA changed the title ~~2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Improvements~~ 2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements Jun 6, 2020

ChristopherA added intern project This issue is in regards to an internship project. status:scoping This issue is in the scoping and discovery phase of project development labels Jun 6, 2020

jodobear removed the status:scoping This issue is in the scoping and discovery phase of project development label Jul 6, 2020

shannona closed this as completed Nov 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

ChristopherA commented Jun 6, 2020 •

edited by jodobear

Loading

chiselinc commented Jun 8, 2020

gg2001 commented Jun 8, 2020

jodobear commented Jun 8, 2020 •

edited

Loading

jodobear commented Jun 8, 2020

ChristopherA commented Jun 8, 2020

ChristopherA commented Jun 8, 2020

jodobear commented Jun 10, 2020

jodobear commented Jun 10, 2020 •

edited

Loading

Fonta1n3 commented Jun 11, 2020 •

edited

Loading

Proposed Milestones

jodobear commented Jun 14, 2020

ChristopherA commented Jun 14, 2020

jodobear commented Jun 14, 2020

javiervargas commented Jun 24, 2020

jodobear commented Jun 24, 2020 •

edited

Loading

javiervargas commented Jun 24, 2020

jodobear commented Jun 24, 2020

jodobear commented Jul 6, 2020

shannona commented Nov 4, 2020

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

Comments

ChristopherA commented Jun 6, 2020 • edited by jodobear Loading

chiselinc commented Jun 8, 2020

gg2001 commented Jun 8, 2020

jodobear commented Jun 8, 2020 • edited Loading

Proposal

Tasks

Enhancements and Other Services

Questions

Hosting Data

jodobear commented Jun 8, 2020

ChristopherA commented Jun 8, 2020

ChristopherA commented Jun 8, 2020

jodobear commented Jun 10, 2020

jodobear commented Jun 10, 2020 • edited Loading

Proposed Milestones

Tasks

Scope

Fonta1n3 commented Jun 11, 2020 • edited Loading

Proposed Milestones

jodobear commented Jun 14, 2020

ChristopherA commented Jun 14, 2020

jodobear commented Jun 14, 2020

javiervargas commented Jun 24, 2020

jodobear commented Jun 24, 2020 • edited Loading

UX

Default Installation

javiervargas commented Jun 24, 2020

jodobear commented Jun 24, 2020

jodobear commented Jul 6, 2020

shannona commented Nov 4, 2020

ChristopherA commented Jun 6, 2020 •

edited by jodobear

Loading

jodobear commented Jun 8, 2020 •

edited

Loading

jodobear commented Jun 10, 2020 •

edited

Loading

Fonta1n3 commented Jun 11, 2020 •

edited

Loading

jodobear commented Jun 24, 2020 •

edited

Loading