backfill not working on binance, slow/not working on gdax

[dakreepy1]

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/zenbot

If you open a GitHub issue, here is our policy:

1. It must be a bug or a feature request.
2. The form below must be filled out.

Here's why we have that policy: Zenbot developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies hundreds of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• Have I written custom code (as opposed to using zenbot vanilla): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 16.04
• Zenbot version: latest unstable
• NodeJS version: 8.9.4
• Python version: 2.7.12
• Exact command to reproduce:
  ./zenbot.sh trade binance.ETH-BTC or ./zenbot.sh backfill binance.ETH-BTC

Describe the problem

after pulling the latest unstable it seems that backfill is broken. There is an error using binance as shown in the the logs section below, seems like the request is missing the startTime parameter.

For gdax ./zenbot.sh backfill gdax.ETH-USD --days 1 i am seeing it get trades but it sure is taking a while, not sure if it was due to the high volume recently or what. Output shows 100 trades processed after about 3 minutes or so.

When using the trade option (./zenbot trade gdax.BTC-USD) on gdax, the output says 'Backfilling...' and sits there for a while not doing anything, no output at all (waited about 10 minutes). This was after killing the command and doing a pull and also an npm update. So there should not be too many trades to download.

Source code / logs

ryan@coin:~/zenbot$ ./zenbot.sh backfill binance.ETH-USDT

---

Hitting up the exchange 'binance' for trades within the past 7 days.

---

Backfilling...

An error occurred { Error: binance GET https://api.binance.com/api/v1/aggTrades?symbol=ETHUSDT&startTime=&endTime=3600000 400 Bad Request {"code":-1105,"msg":"Parameter 'startTime' was was empty."} (possible reasons: invalid API keys, bad or old nonce, exchange is down or offline, on maintenance, DDoS protection, rate-limiting)
at binance.defaultErrorHandler (/home/ryan/zenbot/node_modules/ccxt/js/base/Exchange.js:434:15)
at response.text.then.responseBody (/home/ryan/zenbot/node_modules/ccxt/js/base/Exchange.js:452:18)
at
at process._tickCallback (internal/process/next_tick.js:188:7) constructor: [Function: ExchangeNotAvailable] }

And for gdax:
./zenbot.sh trade gdax.BTC-USD
fetching pre-roll data:

---

Hitting up the exchange 'gdax' for trades within the past 1 day.

---

Backfilling...

[phatwila]

Same here...

[root@cpaneldev zenbot]# ./zenbot.sh backfill binance.BNB-BTC --days 10
An error occurred { Error: binance {"code":-1127,"msg":"More than 1 hours between startTime and endTime."}
    at binance.handleErrors (/home/btctrader/zendev/zenbot/node_modules/ccxt/js/binance.js:809:27)
    at response.text.then.text (/home/btctrader/zendev/zenbot/node_modules/ccxt/js/base/Exchange.js:443:18)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) constructor: [Function: ExchangeError] }
An error occurred { Error: binance {"code":-1127,"msg":"More than 1 hours between startTime and endTime."}
    at binance.handleErrors (/home/btctrader/zendev/zenbot/node_modules/ccxt/js/binance.js:809:27)
    at response.text.then.text (/home/btctrader/zendev/zenbot/node_modules/ccxt/js/base/Exchange.js:443:18)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) constructor: [Function: ExchangeError] }

[dakreepy1]

i've reverted to a few commits ago and things seem to be working. looks like this one caused the issue.
Refactor backfill for unit testing (#1214)

[Haehnchen]

fixed it 2 workarounds needed for now: see #1214 just revert for More than 1 hours between issue and see #1268 for Parameter 'startTime'

[dakreepy1]

#1268 worked for binance, thanks!
i still ended up reverting back to before #1214 for the time being as i use on both gdax and binance and this made both work for now. i'll see if i can skip over 1214 somehow while getting the other commits. Tried to use cherry-pick but that did not work out. I'm not the best at git but i'll figure it out. Any input will be welcome!

[nslocomotives]

Im still seeing this issue even after reading this and making sure im up to date.

User@User-PC MINGW64 ~/repos/zenbot (unstable)
$ docker pull deviavir/zenbot:unstable
unstable: Pulling from deviavir/zenbot
Digest: sha256:80e1a202aef77d17b931933b7ee312b4e00f7109471beefa73de1fbf5f361fbd
Status: Image is up to date for deviavir/zenbot:unstable

User@User-PC MINGW64 ~/repos/zenbot (unstable)
$ docker-compose --file=docker-compose-windows.yml up -d
Starting zenbot_mongodb-data_1 ... done
Starting zenbot_mongodb_1 ... done
Recreating zenbot_server_1 ... done

User@User-PC MINGW64 ~/repos/zenbot (unstable)
$ docker-compose exec server zenbot trade binance.trx-eth --paper
fetching pre-roll data:

---

Hitting up the exchange 'binance' for trades within the past 1 day.

---

Backfilling...

An error occurred { Error: binance {"code":-1127,"msg":"More than 1 hours betwee
n startTime and endTime."}
at binance.handleErrors (/app/node_modules/ccxt/js/binance.js:809:27)
at response.text.then.text (/app/node_modules/ccxt/js/base/Exchange.js:443:1
8)
at
at process._tickCallback (internal/process/next_tick.js:188:7) constructor:
[Function: ExchangeError] }
An error occurred { Error: binance {"code":-1127,"msg":"More than 1 hours betwee
n startTime and endTime."}
at binance.handleErrors (/app/node_modules/ccxt/js/binance.js:809:27)
at response.text.then.text (/app/node_modules/ccxt/js/base/Exchange.js:443:1
8)

Id really like to get this working.

[brinley]

I created #1281 to fix the api errors "More than 1 hours between startTime and endTime.". See if this works for you.

[dakreepy1]

not sure what my issue may be or if a fix is in place yet. I just did a pull since there were a bunch of recent commits. restarted my bots after taking 5 mins to pull and update, and now they're sitting there backfilling. It really shouldnt be taking this long as it only missed like 5-10 minutes of trade data during the update. tested it out on binance and gdax with the exact same results.
tried using the trade command and just regular backfilling, doesnt make a difference.

[dakreepy1]

decided to start from scratch. The first test with a fresh git pull and npm install gave the same results. So i decided to change the database name. After the change in the conf.js i see trades are backfilling way faster. So perhaps i may just need to purge my database and start again.

[dakreepy1]

So purging the database and starting from scratch did help a little. I did a test where i had zenbot running, all the trades were downloaded and things were updating as they should be. I killed it and started it back immediately. It still takes a very long time to backfill before trades can start, while i'm writing this its been over 20 minutes and its still backfilling. If i had to guess it may be downloading the whole day's trades again into the database. I do notice some high database cpu usage during this backfill.

[dakreepy1]

latest pull seems to have resolved this. ive only tested gdax so far.