2020 Summer Internship Project: Bitcoin Price & History Data #6

ChristopherA · 2020-06-06T20:32:38Z

To fully function as a mainnet Bitcoin mobile wallet, FullyNoded 2 needs to have price data. Right now this means we've had to turn off locking the wallet down to only speak to hidden .onion addresses, as there doesn't appear to be any free APIs for price data.

In addition, we'd like to users being able to support adding price data history to your transactional data in your personal wallet. For instance, to allow export to a personal Beancount accounting database.

Thus at minimum we'd like to host our own hidden .onion service with current price data, and in the principle of self-sovereignty, allow people to host this service themselves if they want on their own VPS, VM or personal node.

Other features are the price history, possibly scraped from multiple sources, various forms of validation (public lists of price servers, data is signed by the .onion servers keys, is this the same in 3 or more sites, etc.)

Right now Cooper Kernan (@cprkrn) has the most background about price data, and is likely the intern lead on this project. But he'll need support to design how to do this portably so that we can can potentially host this at multiple .onion sites at multiple locations.

As with all of these, this conversation is open to better ideas and solutions.

cc/ @fakhrulKhir @TheBlueMatt

gg2001 · 2020-06-07T00:46:35Z

This library seems useful for this project: https://github.com/ccxt/ccxt

The CCXT library consists of a public part and a private part. Anyone can use the public part immediately after installation. Public APIs provide unrestricted access to public information for all exchange markets without the need to register a user account or have an API key.

To reduce the time it takes to retrieve the price, the ideal approach would be to have the backend of the hidden service connect to an exchange via a Websocket API. However, I can't seem to find a free Websocket pricing API.

I also have experience hosting hidden services, so I can help with this project.

ChristopherA · 2020-06-07T00:54:52Z

@Fonta1n3 What price feed are we using now? What others have we investigated and we can't use because of API or other limitations?

Also, if we do our own API, any thoughts on your preferences? We already are doing some connections with https://blockstream.info via their .onion service — I know that they said they may offer it someday (Blockstream has the data, but they may be restricted in sharing it due to their contract with ICE/NYSE). Have they published any price API thoughts?

@wolfmcnally, you also looked at a bunch of bitcoin APIs at one point — if we do our own APIs, do you have any strong preferences?

-- Christopher Allen

Fonta1n3 · 2020-06-07T01:24:17Z

@Fonta1n3 What price feed are we using now? What others have we investigated and we can't use because of API or other limitations?

Unfortunately we are using blockchain.info/ticker currently. We were using blockchair.com/api but quickly found it is not free.

gg2001 · 2020-06-08T06:01:56Z

I'm volunteering to lead the Bitcoin price data project.

Proposal

First, build a Flask REST API that returns the current price of Bitcoin, along with historical price data. The API would be accessible as a .onion v3 hidden service.

GET /usd/now returns the current Bitcoin USD price
GET /usd/date/<year>-<month>-<date>-<hour>-<minute>-<second> returns the Bitcoin USD price at a specific date and time

Most likely the ccxt/ccxt library will be used to retrieve price data from multiple exchanges.

Tasks

See if there is a free Websocket API to reduce the time it takes the Flask server to retrieve the price
Read the source code of FullyNoded-2 and see how to make GET requests to a hidden service
Look into how the price data should be validated (signing the data on the flask server, and then verifying the signature on FullyNoded-2)
Display the current price in the FullyNoded-2 UI (display the value of the BTC held in the wallet and the current price of 1 BTC)
Add support for multiple fiat currencies on the API
Add a setting to FullyNoded-2 to select which fiat currency you use (USD will be selected by default)
Possibly automatically select the currency of the users locale by default (via Locale.current)
Save the selected fiat currency, and reflect the selection across the app (e.g. if AUD is selected, FullyNoded-2 should request GET /aud/now instead)
When sending BTC in FullyNoded-2, add a way to input a fiat currency value, and send the BTC equivalent of it (have to look into handling slippage etc.)
Allow users to save historical price data to transaction history, either manually or automatically (by requesting the API)
Write a linux shell script that simplifies deployment of the Flask API and sets up a v3 .onion hidden service
Allow users to change the pricing data server to one of their choice in the FullyNoded-2 UI
Include a setting in FullyNoded-2 to export the transaction history, along with prices
Possibly have FullyNoded-2 request pricing data from multiple servers in order to validate the data
Possibly create a Websocket connection between FullyNoded-2 and the Flask server to allow real time price changes to be displayed (have to look into how to do websockets on a hidden service, and how to connect to websockets from FullyNoded-2)
Possibly add caching to the Flask API

ChristopherA · 2020-06-08T08:55:16Z

I would suggest separating out this into separate sub-projects.

First is to implement the core API and storage of data, and a simple curl or other cli terminal command to request it. This has no dependencies on anyone else.
Next is how does the tool scrape and validate the data, and maybe some signed checkpoints that you can download from various places. This also has no dependencies.
Write a script to install the above on at least one service (probably linode to start). Again, no dependencies.
Then we need to wrap this into our own live server somewhere (probably linode) under tor. Also offer our own signed checkpoints. You’ll need to coordinate this with others doing Blockchain Commons devops for our servers & onion.
Finally there is integration with FullyNoded2 which requires coordination with @Fonta1n3

— Christopher Allen

watersnake1 · 2020-06-08T20:12:40Z

I'd like to offer my help on this project - I have experience with flask, python generally, and data mining. I don't have a mac so can't do any of the FN2 related stuff but could certainly be helpful with the API.

Here is a websocket service that aggregates data from several exchanges: https://docs.cryptoapis.io/websockets/index#websockets
It requires a key and limits the number of new connections in a 24 hour period (and limits connection time to 24 hours, so the client would need to automatically reconnect every day), but might not be a bad way to get a lot of exchanges data at once.
It also wouldn't be that bad to write a quick websocket client for a list of popular exchanges and aggregate everything together ourselves. Most exchanges have totally free data websockets and the code required to receive the data isn't too long. I've written websocket clients for Coinbase, Bitfinex, Binance, Kraken; I would be happy to assist here.

gg2001 · 2020-06-09T09:56:40Z

@watersnake1 Cool, I think we can split this project into 2 parts, the API and FN2.

You can work on the API and I can work on integrating it with FN2. I have experience in hosting .onions so I can help with writing a setup script for running the API behind a hidden service.

watersnake1 · 2020-06-09T15:57:16Z

Great. It looks like the cctx library you mentioned is a great starting point.

FN2-API Proposal

Summary

Create a local web server that provides API access to bitcoin price data across many exchanges. Store selected in a local db for quicker access. Once running, the server can be accessed locally or remotely (i.e. FN2) to provide updated price information for btc and historical price data.

Proposal

First, build a Flask REST API that returns the current price of Bitcoin, along with historical price data. The API would be accessible as a .onion v3 hidden service.

GET /usd/now returns the current Bitcoin USD price

GET /usd/date/----- returns the Bitcoin USD price at a specific date and time

I would also add on here that the server should store some data in a local db. Keeping historical data cached will allow the server to serve historical feeds without being ratelimited (most exchange APIs limit to ~30 calls / minute). Even if everything ran on websocket data this could still be an issue if nothing is cached, because websockets only give you new data.

jodobear · 2020-06-09T17:47:09Z

Great. It looks like the cctx library you mentioned is a great starting point.

FN2-API Proposal

Summary

Create a local web server that provides API access to bitcoin price data across many exchanges. Store selected in a local db for quicker access. Once running, the server can be accessed locally or remotely (i.e. FN2) to provide updated price information for btc and historical price data.

Proposal

First, build a Flask REST API that returns the current price of Bitcoin, along with historical price data. The API would be accessible as a .onion v3 hidden service.

GET /usd/now returns the current Bitcoin USD price

GET /usd/date/----- returns the Bitcoin USD price at a specific date and time

I would also add on here that the server should store some data in a local db. Keeping historical data cached will allow the server to serve historical feeds without being ratelimited (most exchange APIs limit to ~30 calls / minute). Even if everything ran on websocket data this could still be an issue if nothing is cached, because websockets only give you new data.

This is perfect, once we have this running we can add this service to the Standup Script.

cprkrn · 2020-06-09T18:13:05Z

I'm volunteering to lead the Bitcoin price data project.

Proposal

First, build a Flask REST API that returns the current price of Bitcoin, along with historical price data. The API would be accessible as a .onion v3 hidden service.

* `GET /usd/now` returns the current Bitcoin USD price

* `GET /usd/date/<year>-<month>-<date>-<hour>-<minute>-<second>` returns the Bitcoin USD price at a specific date and time

Most likely the ccxt/ccxt library will be used to retrieve price data from multiple exchanges.

Tasks

* See if there is a free Websocket API to reduce the time it takes the Flask server to retrieve the price

* Read the source code of FullyNoded-2 and see how to make GET requests to a hidden service

* Look into how the price data should be validated (signing the data on the flask server, and then verifying the signature on FullyNoded-2)

* Display the current price in the FullyNoded-2 UI (display the value of the BTC held in the wallet and the current price of 1 BTC)

* Add support for multiple fiat currencies on the API

* Add a setting to FullyNoded-2 to select which fiat currency you use (USD will be selected by default)

* Possibly automatically select the currency of the users locale by default (via Locale.current)

* Save the selected fiat currency, and reflect the selection across the app (e.g. if AUD is selected, FullyNoded-2 should request `GET /aud/now` instead)

* When sending BTC in FullyNoded-2, add a way to input a fiat currency value, and send the BTC equivalent of it (have to look into handling slippage etc.)

* Allow users to save historical price data to transaction history, either manually or automatically (by requesting the API)

* Write a linux shell script that simplifies deployment of the Flask API and sets up a v3 .onion hidden service

* Allow users to change the pricing data server to one of their choice in the FullyNoded-2 UI

* Include a setting in FullyNoded-2 to export the transaction history, along with prices

* Possibly have FullyNoded-2 request pricing data from multiple servers in order to validate the data

* Possibly create a Websocket connection between FullyNoded-2 and the Flask server to allow real time price changes to be displayed (have to look into how to do websockets on a hidden service, and how to connect to websockets from FullyNoded-2)

* Possibly add caching to the Flask API

You certainly have the technical prowess to take lead on this! Need to do a little research on some of the technologies you mentioned to have proper input, but here to help in any way. Have previous experience w/ python + sql.

ChristopherA · 2020-06-09T19:56:41Z

Ok, it looks like we are pretty close to a plan. We have two leads, @watersnake1 for a new repo for the service, @jodobear for a feature branch to FN2. Additional support and review of API from @cprkrn, @Fonta1n3 and myself, and @cprkrn also as a key tester.

@Fonta1n3 in particular should review the API proposals above and tasks and give a +1.

What we need now are ~3 reasonable top-level milestone for these two projects with the last being public service, docs, and publicity. Then we add that to the internship agreements, sign them, and we are a go!

-- Christopher Allen

jodobear · 2020-06-09T23:44:05Z

@jodobear for a feature branch to FN2.

FN2 is written in Swift, i have never written Swift so, how do i contribute?

gg2001 · 2020-06-09T23:56:42Z

@ChristopherA @jodobear I have experience with Swift, I can work on a feature branch for FN2.

Fonta1n3 · 2020-06-10T03:26:18Z

GET /usd/now returns the current Bitcoin USD price

I would put it on the wishlist that this returns an average price over a certain time period (30 mins for example) from an array of exchanges and is not just limited to USD but includes all major currencies in the response, sorted by ticker. That way we can easily support many fx rates.

Read the source code of FullyNoded-2 and see how to make GET requests to a hidden service

You can see it here

Display the current price in the FullyNoded-2 UI (display the value of the BTC held in the wallet and the current price of 1 BTC)

FN2 already does this.

Possibly automatically select the currency of the users locale by default (via Locale.current)

Would certainly prefer this approach to using USD by default.

When sending BTC in FullyNoded-2, add a way to input a fiat currency value, and send the BTC equivalent of it (have to look into handling slippage etc.)

I imagine this could open up some liability issues and many complaints from customers (see why Samourai abandoned fiat) if slippage occurs, probably best to just keep it in BTC and simply display balances in fiat.

Allow users to save historical price data to transaction history, either manually or automatically (by requesting the API)

Currently we have the ability to save transaction data but no ability to look up historic fx rates (only current), it would be nice if we do include this functionality to automatically parse all txid which are not yet saved locally and add the historic fx rates to them as well as saving them.

cc @ChristopherA (apologies had not seen your request for me to review yet, these are my initial requests/feedback)

Cheers,
Fontaine

gg2001 · 2020-06-10T11:24:40Z

FN2 Branch Proposal

First add a setting that allows a user to enter what API server they want to access, and save it locally on the device (core data)
Then, detect the users locale, based on Locale.current at startup to choose which fiat currency to request
Possibly create a setting to override the Locale.current defaults to a fiat currency of the users choice
Once the API is ready, change the default API server to the .onion one setup by us
Create a feature that allows automatic request of historical price data for transaction history
Possible - Don't allow users to input tx amount in fiat, but allow users to see the value of their tx amount in fiat before sending
Look into ways to export transaction history along with historical price
Look into possibility of requesting data from multiple servers

Fonta1n3 · 2020-06-10T13:31:53Z

FN2 Branch Proposal

First add a setting that allows a user to enter what API server they want to access, and save it locally on the device (core data)

Then, detect the users locale, based on Locale.current at startup to choose which fiat currency to request

Possibly create a setting to override the Locale.current defaults to a fiat currency of the users choice

Once the API is ready, change the default API server to the .onion one setup by us

Create a feature that allows automatic request of historical price data for transaction history

Possible - Don't allow users to input tx amount in fiat, but allow users to see the value of their tx amount in fiat before sending

Look into ways to export transaction history along with historical price

Look into possibility of requesting data from multiple servers

sgtm +1

Look into ways to export transaction history along with historical price

The foundation for this has already been added, currently when you broadcast a transaction in FN2 the app will save the transaction along with an optional user added memo, we include the current fx rate (usd/btc) with this meta data (core data).

When you tap the "info" button on the home screen in the transaction cell we display that data to the user. If that particular tx was not broadcast with FN2 it won't be stored locally and no historic fiat data will be available. The user may add a memo to the transaction at which point it will be saved locally.

Once we have a means of fetching historic price data (for transactions that were not broadcast with FN2) we can add a function that parses all historic transactions that way users who want to "do the right thing" and utilize FN2 as an accounting tool can easily tap an "export to beancount" button (@ChristopherA is more familiar with beancount then I am). The only reason they need to be saved locally is for the ability to add memos, other than that we can just fetch everything we need from bitcoind on the fly.

If you want to start working on this just let me know so we can coordinate, I am currently working off of this branch on my fork, it would probably be smoothest if you waited till we merge it into Blockchain Commons FN2 development branch that way we are working from the same code base.

watersnake1 · 2020-06-12T01:04:19Z

Here is my first attempt at a design spec for the api server. This version should focus on using existing libraries or plain GET requests to obtain data. Then, support for websockets should be added. The server should run in the background as a system service.

FN-Server Design Spec

Language: Python3
Major Dependencies: Flask, Sqlite3, cctx
FN-Server shall run in the background and act as a local API that can be queried to obtain price information. FN-Server will use GET requests to obtain Bitcoin price information of exchanges and store this data locally. The server will be accessible locally and remotely. The server will automatically query relevant data sources periodically to keep information as updated as possible. When the number of rows becomes too large, the server will prune old data from local storage.

API Routes

API responses are returned as JSON
GET /currency/now

Return the latest price in listed currency from local db
GET /currency/hist/YYYY-MM-DD-HH-mm-SS
Return the price at the specified time (or closest available time) from local db
GET /status/
Return the status of the server

Methods

install(dataDir)

create sqlite db and config file in the specified directory.

prune(pruneDepth)

remove rowCount - pruneDepth oldest rows from the db

request(currency, exchanges)

make a GET request to given exchanges to obtain current price in given currency

request_historical(datetime, currency, exchanges)

make a GET request to given exchanges to obtain price in given currency at time datetime

read_config()

re-load configuration settings from the config file

query_db(datetime, currency, exchanges)

query local db for closest row to given time. Return price in given currency, time and exchange as JSON

Config File

The config file should have the following settings to be defined by the user

pruneDepth: the maximum number of rows to store
dataDir: location of the database
exchanges: list of exchanges the user wishes to keep data for. This will always be a subset of the exchanges that the server supports because every exchange API is structured slightly differently and cannot be supported automatically.

gg2001 · 2020-06-14T22:46:23Z

Internship milestones

Write Node.js and Go sections for LBTCftCL
- Latest Completion date*: June 26th 11:59PM AEST
- Dependencies: Fill out work order form before committing changes on my fork and submitting a pull request
FN2 features proposal
- Latest Completion date*: July 24th 11:59PM AEST
- Dependencies:
  - Merging of @Fonta1n3's FN2 develop branch to the FN2 development branch
  - Completion of a very basic version of the FN2-API
- First add a setting that allows a user to enter what API server they want to access, and save it locally on the device (core data)
- Then, detect the users locale, based on Locale.current at startup to choose which fiat currency to request
- Create a setting to override the Locale.current defaults to a fiat currency of the users choice
- Once the API is ready, change the default API server to the .onion one setup by us
- Create a feature that allows automatic request of historical price data for transaction history
Finish FN2 features proposal and work on setup script with onion support for the FN2-API
- Latest Completion date*: August 21st 11:59PM AEST
- Dependencies:
  - Completion of milestone 2
  - Need to coordinate with people working on Bitcoin standup
- Add export with historical prices to beancount
- develop setup script with onion support for the FN2-API
- integrate setup script with Bitcoin standup Linux and MacOS

*The latest completion date is an upper bound, I will probably finish before it

ChristopherA · 2020-06-14T23:20:23Z

This looks a good start.

It mostly lacks some rough completion dates, and connecting those dates with some dependences (milestone 2 is dependent on some version of the API server being available, and 2nd part of milestone 3 that that code ready to be installable by itself, or possibly using some BitcoinStandup plugin method.)

-- Christopher Allen

ChristopherA · 2020-06-14T23:27:33Z

BTW, an archive of some details on what I learned about using Beancount for Bitcoin Accounting:

This feature on a per Bitcoin account in FN2 I think would be an incredibly useful tool for someone trying to "do the right thing" with bitcoin accounting. It uses the "brew install beancount" command-line ledger system.

The following file would be available in each FN2 account, created from the transaction data. The top would have some default bookkeeping accounts (someday changeable in the settings for other account name preferences).

Example of a two in events (receiving bitcoin), and then an outgoing event which should be an expense, but in fact it also generates a small income event has the bitcoin price went up. Input put this into beancount and it will properly show it.

An important thing I learned is that if you know the cost basis for the UTXO's being spent (i.e. have been using our wallet the whole time, or at least until we find a historical price feed), you need to store the cost basis of the change output as well, using HIFO (Highest cost base UTXO is spent first, until you have the lowest cost one as the price for change.

; Beancount Ledger for Account "XXXXX" [fingerprint]path;xpub

2019-01-01 open Assets:Cryptocurrency:BTC       BTC                             ; My Account
2019-01-01 open Expenses:Value-Sent-BTC         USD                             ; The value sent from the transaction in USD when I sent it
2019-01-01 open Expenses:Fees:Transaction-BTC   USD                             ; Fees paid in BTC to miners for transactions
2019-01-01 open Income:Value-Received-BTC       USD                             ; The value recieved from the transaction in USD when I sent it

; Transactions from X to Y, using HIFO (Highest In First Out)

2019-01-15 * "Received Bitcoin" "memo"                                          ; txid = XXXX
  Assets:Cryptocurrency:BTC          0.77930358 BTC {2821.04 USD} @ 3619.95 USD ; on EXCHANGE as of DATE:TIME
  Income:Value-Received-BTC                                                     ; Transacton value 2,821.04 USD

2019-06-01 * "Received Bitcoin" "memo"                                          ; txid = XXXX
  Assets:Cryptocurrency:BTC          0.28938957 BTC {2481.18 USD} @ 8573.84 USD ; on EXCHNGE as of DATE:TIME
  Income:Value-Received-BTC                                                     ; Transacton value 2481.18 USD

2019-12-29 * "Sent Bitcoin" "memo"                                              ; txid = XXXX
  Assets:Cryptocurrency:BTC         -0.28938957 BTC {2481.18 USD} @ 7296.10 USD ; / BTC on EXCHANGE as of DATE:TIME
  Assets:Cryptocurrency:BTC         -0.61065411 BTC {2821.04 USD} @ 7296.10 USD ; / BTC on EXCHANGE as of DATE:TIME
  Expenses:Fees:Transaction-BTC           -0.32 USD                             ; 0.00004368 at XX satoshi per byte
  Expenses:Value-Sent-BTC                                                       ; Transaction value $6,566.81 USD
                                                                                ; change 1.38995769 BTC cost basis {2821.04 USD}

Account	Other
Assets
Cryptocurrency
BTC	0.16864947 BTC
Equity
Conversions
Previous	-0.00 USD
Expenses
Fees
Transaction-BTC	-0.32 USD
Value-Sent-BTC	2441.03 USD
Income
Value-Received-BTC	-2916.47 USD
Liabilities
	-475.77 USD

This was a HUGE pain in the ass for me in 2019, and will be worse in 2020, so I'm sure I'm not the only one dealing with this as a problem.

gg2001 · 2020-06-15T00:32:42Z

@ChristopherA thanks for the info about beancount, I'll be looking into how to design the exporting feature soon.

I've also added rough completion dates along with dependencies for each milestone (the dates are an upper bound).

ChristopherA · 2020-06-15T01:02:50Z

@gg2001 Looks good.

Here is some example text toward a signed work order:

This is a Work Order under the most recently released version of Open Development Intern Contract published at https://github.com/BlockchainCommons/Open-Development/tree/master/internship-contract.

Blockchain Commons and Intern agree:

Start Date

2020-06-15

End Date

2020-08-21

Repositories

[list of existing or new repositories you will be contributing to]

Milestones

[your list of milestones above]

Total Hours

40 hours. If intern exceeds 40 hours, any additional effort toward these milestones will not be compensated by Blockchain Commons. All such contributions after 40 hours will at Intern's own choice as part of any normal open source contribution under the standard Contributor License Agreement (aka CLA) in that repo for that project.

Fee

Flat fee of US$564.00, in three installments of US$188. Rate will be USD/BTC based on the Gemini exchange at time of transfer.

Currency

Bitcoin

Intern's Payment Instructions

Intern will provide Blockchain Commons a new bitcoin address for each payment over a secure channel.

Blockchain Commons' Billing Instructions

First payment to be paid in Bitcoin for US$188 in Bitcoin within 5 days of digital signature of this work order and terms.

2nd payment for completion of 2nd milestone. Payment US$188 in Bitcoin within 5 days of notice to Blockchain Commons.

3rd payment on completion of 3rd milestone, or completion of 40 hours of effort by intern, no later than August 21st. Payment of US$188 in Bitcoin within 5 days of notice to Blockchain Commons.

ChristopherA · 2020-06-17T06:20:35Z

See discussion about onionbalance, my conflating requirements of different hosted projects, and a need for understanding risk models in #5 (comment)

-- Christopher Allen

NicolasDorier · 2020-06-18T00:28:15Z

BTCPay is actually integrating with many exchanges and has its own rating script DSL. (Where you can define more complex rules of calculation)
For example:

X_X=bitflyer(X_X)

means that every pair will use bitflyer

X_JPY=bitflyer(X_JPY)
BTC_USD=coinbase(BTC_USD)

Mean that usd will use coinbase and any pair with JPY will use bitflyer. Then you can compose it.

X_JPY=bitflyer(X_JPY)
BTC_USD=coinbase(BTC_USD)
X_X=kraken(X_X)
LTC_USD=LTC_BTC * BTC_USD

which mean the LTC_USD pair will use kraken(LTC_BTC) * coinbase(BTC_USD).

We already provide a public endpoint for those rate on your own instance. We just need to document it better.

We use a mix a direct API integration, and for some exchanges, coingecko.

We do not provide history.

ChristopherA · 2020-06-18T01:16:01Z

@NicolasDorier — thank you for that information!

Any chance that you'd be willing to make that public endpoint also available at a Tor v3 hidden service, the way blockstream.info also has a hidden service version?

-- Christopher Allen

ChristopherA · 2020-06-18T01:17:32Z

@NicolasDorier, we are also investigating using Onion Balance so that we can host anycast-like services from multiple places around the world. This might be something of interest to your community as well.

https://onionbalance.readthedocs.io/en/latest/v3/tutorial-v3.html#tutorial-v3

ChristopherA · 2020-06-18T01:23:07Z

So that it is easier to follow the conversation in this thread, here is an excerpt from another threat that should be here: #5

You are correct, I've conflating the requirements of the two (or more) projects. Especially given your onion balance suggestion, we can try to have greater security in one place, and be less rigorous in the others. But in the long run I really want our users not to even depend on us for these services, instead give them a choice of many.

We are definitely need to do significantly more puzzling through the current price API risk models. Anything we do will be likely be better than the current state of the art where mobile wallets use a single source of current price information, typically the wallet vendor or a single exchange's API. Not only is this model vulnerable to DOS attacks, I suspect there are other tricks an attacker can do if they can make your remote wallet think the price has changed when it has not. Offering this services via Tor v3 also means we add a new attack surface that is less well known.

I don't necessarily want to replicate what Blockstream is doing with their current price data, but more desire to enable those who want to source it themselves for their own full node / remote wallet combos, that they can install an app alongside their full node that sources its own current data from APIs that user chooses. If they want to point their remote app to us, another reputable clone of our service, or directly to the API of their favorite exchange, or even test against multiple exchanges, it is up to them.

Our key requirement for this service is as much self-sovereignty for the full-node / remote wallet user as possible.

The historical data is also useful, but finding some way to distribute it safely is a different risk model than that for current price data.

NicolasDorier · 2020-06-18T01:33:04Z

@ChristopherA all btcpay installs (via the docker install method) come with a hidden service properly set up.

ChristopherA · 2020-06-18T05:00:22Z

@NicolasDorier wrote:

all btcpay installs (via the docker install method) come with a hidden service properly set up.

What I meant was a hidden service for your:

We already provide a public endpoint for those rate on your own instance. We just need to document it better.

ChristopherA · 2020-06-18T05:57:07Z

@slush0 wrote in a telegram group:

Blockbook (https://github.com/trezor/blockbook/) has already such interface implemented, including historical data feed and websocket API.

I’ve asked if there is an onion feed.

NicolasDorier · 2020-06-18T09:53:04Z

Our https://mainnet.demo.btcpayserver.org/ that you can access to http://fc4booz5wmoq3knc63gf4sn7oisz45sc7zxtlgnqkeqb2pvzqow6ydqd.onion/ support it.

This is our demo server, so everybody installing BTCPay has their own service as well.
You need to create a store and then hit /api/rates?storeId=yourStoreId to get the rates you chose.

watersnake1 · 2020-06-20T14:49:35Z

In regards to the local database, I think sqlite3 is a good option. It is an easy to use / integrate db thats pretty lightweight. Although it is not the best choice for use as a webserver, I think that it is a safe assumption that most server instances anyone runs will not be used by more than a few people (and probably just one person a lot of the time). Does anyone have any other db suggestions, or resources on hardening / making sqlite3 as secure as possible?

As for the database design, I am not sure whether to store all price entries in one table, like this:

timestamp	exchange	currency	open	high	low	close

or to separate each supported exchange into its own table:

timestamp	currency	open	high	low	close

Since the actual table structure would be abstracted away from the user via the API, I suppose that the specific table setup doesn't effect anything too much. But any input is appreciated. I will be integrating the db design into the final spec that I will complete by the deadline for my project phase one goal.

watersnake1 · 2020-06-26T01:14:49Z

fn-server spec (draft 2, final for milestone 1)

This is my final version of the spec for this project. I will use this as my reference when implementing during milestone 2.
Language: Python3
Major Dependencies: Flask, Sqlite3, cctx, Celery
FN-Server shall run in the background and act as an API service that can be queried to obtain price information. FN-Server will use GET requests to obtain Bitcoin price information of exchanges and store this data locally in the users chosen currency. The server will be accessible locally and remotely. The server will automatically query relevant data sources periodically to keep information as updated as possible. A celery task-queue will be used to optimize the performance of the server. When the number of rows becomes too large (according to user definition), the server will prune old data from local storage.

API Routes

API responses are returned as JSON
GET /now/[currency]/[exchange]

Return the latest price in listed currency from local db. If there is no information present in the database in the specified currency, then make a GET request to download it.

GET /hist/[currency]/[exchange]/[date_start]/[date_end]

Return the rows with timestamp values between date_start and date_end. These params are provided in YYYY-MM-DDTHH-mm-SS format (similar to ISO8601 time format, but easier to parse).

GET /status/

Return the status of the server

Configuration

The user will configure the program via a config file, which will be located in a hidden subdirectory of the users home folder. The following settings will be included:

pruneDepth: the max number of rows in the table.
exchanges: exchanges whose data will be stored by the program.
currencies: currencies that will be supported by the program.
The syntax of the file will be field=value. Multiple values can be specified for exchanges and currencies. These will be separated by spaces after the equals sign. At runtime the program will read this file and store all fields in memory.

Database

The database will be an sqlite3 db. One database will be created. Inside of it, each exchange will have a separate table. Entries from different currencies will be in different rows. The columns of each table shall be timestamp (string) currency (string) open (float) high (float) low (float) close (float). Note, there are no native datetime types in sqlite3 as in mySQL, but sqlite3 can still compare dates stored as string.

Methods

install()

read the config file
load config options into memory
create a sqlite3 database if one does not exist already, with the design specified above.

prune(pruneDepth)

drop rowCount-pruneDepth rows from each table.

request(currency, exchanges)

connect to the local database.
perform a GET request for each pair of exchange and currency specified in the config file.
insert these data into the database at regular intervals.
space out requests to avoid API ratelimits.

request_historical(date_start, date_end, exchanges, currencies)

Between date_start, and date_end, make GET requests to obtain price data in given currencies for given exchanges.

read_config()

re-load configuration settings from the config file

now(currency, exchange)

function for the now api route.
query database for the latest row in currency currency on exchange exchange.
if there is no data, request it directly.

hist(date_start, date_end, currency, exchange)

function for the hist api route.
query database for all rows between date_start and date_end. If there is not informaion available, return nothing.

Execution Flow

At first run, the install() function will setup the user's database
the request process and the Flask api will be run as two separate threads so that new data can be downloaded in the background while the API is running.
Once flask begins running it is ready to accept incoming API requests.

shannona · 2020-11-03T23:52:07Z

Thanks to @watersnake1 with support from @jodobear and @javiervargas, we now have spotbit, an amazing bit of progress for a mere summer's work. Thanks folks!

ChristopherA changed the title ~~2020 Internship Project: Bitcoin Price & History Data~~ 2020 Summer Internship Project: Bitcoin Price & History Data Jun 6, 2020

ChristopherA added intern project This issue is in regards to an internship project. status:scoping This issue is in the scoping and discovery phase of project development labels Jun 6, 2020

jodobear mentioned this issue Jun 10, 2020

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

Closed

8 tasks

ChristopherA mentioned this issue Jun 15, 2020

2020 Summer Internship Project: Additional Tor Exit Nodes, Support Tor #5

Closed

11 tasks

shannona closed this as completed Nov 3, 2020

ChristopherA added the program: lbtcftcl associated with the Learning Bitcoin from the Command Line projects label Oct 4, 2022

2020 Summer Internship Project: Bitcoin Price & History Data #6

2020 Summer Internship Project: Bitcoin Price & History Data #6

Comments

ChristopherA commented Jun 6, 2020

gg2001 commented Jun 7, 2020 • edited Loading

ChristopherA commented Jun 7, 2020

Fonta1n3 commented Jun 7, 2020 • edited Loading

gg2001 commented Jun 8, 2020 • edited Loading

Proposal

Tasks

ChristopherA commented Jun 8, 2020

watersnake1 commented Jun 8, 2020

gg2001 commented Jun 9, 2020

watersnake1 commented Jun 9, 2020

FN2-API Proposal

Summary

Proposal

jodobear commented Jun 9, 2020

FN2-API Proposal

Summary

Proposal

cprkrn commented Jun 9, 2020

Proposal

Tasks

ChristopherA commented Jun 9, 2020

jodobear commented Jun 9, 2020

gg2001 commented Jun 9, 2020

Fonta1n3 commented Jun 10, 2020 • edited Loading

gg2001 commented Jun 10, 2020

FN2 Branch Proposal

Fonta1n3 commented Jun 10, 2020

FN2 Branch Proposal

watersnake1 commented Jun 12, 2020 • edited Loading

FN-Server Design Spec

API Routes

Methods

Config File

gg2001 commented Jun 14, 2020 • edited Loading

Internship milestones

ChristopherA commented Jun 14, 2020

ChristopherA commented Jun 14, 2020

gg2001 commented Jun 15, 2020

ChristopherA commented Jun 15, 2020

Start Date

End Date

Repositories

Milestones

Total Hours

Fee

Currency

Intern's Payment Instructions

Blockchain Commons' Billing Instructions

ChristopherA commented Jun 17, 2020

NicolasDorier commented Jun 18, 2020 • edited Loading

ChristopherA commented Jun 18, 2020

ChristopherA commented Jun 18, 2020

ChristopherA commented Jun 18, 2020 • edited Loading

NicolasDorier commented Jun 18, 2020 • edited Loading

ChristopherA commented Jun 18, 2020

ChristopherA commented Jun 18, 2020

NicolasDorier commented Jun 18, 2020

watersnake1 commented Jun 20, 2020

watersnake1 commented Jun 26, 2020 • edited Loading

fn-server spec (draft 2, final for milestone 1)

API Routes

Configuration

Database

Methods

Execution Flow

shannona commented Nov 3, 2020

gg2001 commented Jun 7, 2020 •

edited

Loading

Fonta1n3 commented Jun 7, 2020 •

edited

Loading

gg2001 commented Jun 8, 2020 •

edited

Loading

Fonta1n3 commented Jun 10, 2020 •

edited

Loading

watersnake1 commented Jun 12, 2020 •

edited

Loading

gg2001 commented Jun 14, 2020 •

edited

Loading

NicolasDorier commented Jun 18, 2020 •

edited

Loading

ChristopherA commented Jun 18, 2020 •

edited

Loading

NicolasDorier commented Jun 18, 2020 •

edited

Loading

watersnake1 commented Jun 26, 2020 •

edited

Loading