Example scripts error #68

jgrabek · 2021-11-02T21:40:30Z

Description

As in the title. Provided examples throw errors during execution
Tested files: bot.py (generated by 'blankly init' command), examples/rsi.py, examples/mlp_model.py

settings.json

{
  "settings": {
    "account_update_time": 5000,
    "use_sandbox": true,
    "use_sandbox_websockets": true,
    "websocket_buffer_size": 10000,
    "test_connectivity_on_auth": true,
    "coinbase_pro": {
      "cash": "USD"
    },
    "binance": {
      "cash": "USDT",
      "binance_tld": "us"
    },
    "alpaca": {
      "websocket_stream": "iex",
      "cash": "USD"
    }
  }
}

backtest.json

{
  "price_data": {
    "assets": []
  },
  "settings": {
    "use_price": "close",
    "smooth_prices": false,
    "GUI_output": true,
    "show_tickers_with_zero_delta": false,
    "save_initial_account_value": true,
    "show_progress_during_backtest": true,
    "cache_location": "./price_caches",
    "continuous_caching": true,
    "resample_account_value_for_metrics": "1d",
    "quote_account_value_in": "USDT",
    "ignore_user_exceptions": false,
    "risk_free_return_rate": 0.0
  }
}

Error

rsi.py

Traceback (most recent call last):
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 692, in run
    raise e
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 685, in run
    self.price_events[0]['function'](self.interface.get_price(self.price_events[0]['asset_id']),
  File "/home/jakub/PycharmProjects/bot/blankly/rsi.py", line 7, in price_event
    rsi = blankly.indicators.rsi(state.variables['history'])
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/indicators/oscillators.py", line 33, in rsi
    rsi_values = ti.rsi(data, period)
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/tulipy/__init__.py", line 1003, in rsi
    return lib.rsi([real], [period])
  File "tulipy/lib/__init__.pyx", line 105, in tulipy.lib._Indicator.__call__
tulipy.lib.InvalidOptionError

mlp_model.py

Traceback (most recent call last):
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 692, in run
    raise e
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 685, in run
    self.price_events[0]['function'](self.interface.get_price(self.price_events[0]['asset_id']),
  File "/home/jakub/PycharmProjects/bot/blankly/mlp_model.py", line 38, in price_event
    rsi_values = rsi(variables['history'], period=14).reshape(-1, 1)
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/indicators/oscillators.py", line 33, in rsi
    rsi_values = ti.rsi(data, period)
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/tulipy/__init__.py", line 1003, in rsi
    return lib.rsi([real], [period])
  File "tulipy/lib/__init__.pyx", line 105, in tulipy.lib._Indicator.__call__
tulipy.lib.InvalidOptionError

bot.py

Traceback (most recent call last):
  File "/home/jakub/PycharmProjects/bot/blankly2/bot.py", line 39, in <module>
    results = strategy.backtest(to='1y', initial_values={'USDT': 10000})
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/frameworks/strategy/strategy_base.py", line 495, in backtest
    results = self.backtesting_controller.run()
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 543, in run
    self.interface.receive_price(k, v[use_price].iloc[0])
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/pandas/core/indexing.py", line 879, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/pandas/core/indexing.py", line 1496, in _getitem_axis
    self._validate_integer(key, axis)
  File "/home/jakub/anaconda3/envs/cbot/lib/python3.9/site-packages/pandas/core/indexing.py", line 1437, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds

Platform Info

Python version: 3.9 Anaconda based
Platform: Ubuntu 20.04.3 LTS

Additional context
Library installed as proposed in https://github.com/Blankly-Finance/Blankly#quickstart

All example scripts and json files were modified to use sandbox account and API for Binance, also USD was changed to USDT in symbols and portfolio.

EmersonDove · 2021-11-02T21:50:08Z

Interesting, I will look into this. This is a dependency problem inside the metrics library that we use. We haven't tested the code on an Anaconda distribution so if I were to guess it might be an issue building the extension modules that tuliply requires. I will test it out and respond asap. I believe that the Binance library we use might also have issues with Anaconda.

Thanks for opening an issue!

jgrabek · 2021-11-02T22:24:18Z

Thanks for the reply. Unfortunately, the standalone virtual environment on python 3.9 (pip install blankly) throws the same errors

EmersonDove · 2021-11-02T22:31:59Z

Interesting, are there any errors with the install or does it run smoothly (besides version conflicts)?

jgrabek · 2021-11-02T22:50:38Z

The pip log looks clean both in venv and anaconda

EmersonDove · 2021-11-02T23:02:43Z

Ah so I figured it out. The Binance sandbox deletes their historical data at the beginning of the month. There is only 15 hours of data inside the sandbox exchange which won't work with price events at a day resolution. We use data keys inside our tests to get around this issue (read only no trade on the live exchange).

You're just getting really unlucky - I would recommend creating low permission live keys for binance data.

jgrabek · 2021-11-02T23:55:24Z

Thank you. The mentioned errors in rsi.py and mlp_model.py are gone when I use a standard Binance account. But bot.py error is still present.

Also when I use rsi.py and mlp_model.py the metrics returns a lot of nan's and zeros (associated output and error/warning below)

No cached data found for BTC-USDT from: 1635897059 to 1635897125 at a resolution of 60 seconds.

Initializing...

Backtesting...
Progress: [##########] 98.81% /home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/metrics/portfolio.py:31: RuntimeWarning: divide by zero encountered in double_scalars
  return (end_value / start_value) ** (1 / years) - 1
/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/metrics/portfolio.py:68: RuntimeWarning: Degrees of freedom <= 0 for slice.
  return np.nanvar(returns) * np.sqrt(n) if n else np.nanvar(returns)

Historical Dataframes: 
Account History: 
           USDT     BTC          time  Account Value (USDT)
0     100.00000  0.0000  1.635835e+09            100.000000
1       0.56640  0.0016  1.635835e+09             99.382400
2       0.56640  0.0016  1.635835e+09             99.316912
3       0.56640  0.0016  1.635835e+09             99.385776
4       0.56640  0.0016  1.635835e+09             99.282112
...         ...     ...           ...                   ...
1032  612.02581  0.0000  1.635897e+09            612.025810
1033  612.02581  0.0000  1.635897e+09            612.025810
1034  612.02581  0.0000  1.635897e+09            612.025810
1035  612.02581  0.0000  1.635897e+09            612.025810
1036  612.02581  0.0000  1.635897e+09            612.025810

[1037 rows x 4 columns]
Account Returns: 
           time  value
0  1.635835e+09    NaN
Resampled Account Value: 
           time  value
0  1.635835e+09  100.0
Blankly Metrics: 
Compound Annual Growth Rate (%):   0.0%
Cumulative Returns (%):            0.0%
Max Drawdown (%):                  nan%
Variance (%):                      nan%
Sortino Ratio:                     nan
Sharpe Ratio:                      nan
Calmar Ratio:                      nan
Volatility:                        nan
Value-at-Risk:                     nan
Conditional Value-at-Risk:         nan

EmersonDove · 2021-11-03T16:42:44Z

Yes these should both be easy fixes!

We cache the prices to avoid slow downloads, the empty sandbox data is inside the cache right now, so try deleting the ./price_caches folder.

For the metrics, did you switch the quote_account_value_in to USD? The nan results are common for quoting a currency the exchange does not use. I see from the backtest.json provided its in USDT but I want to double check!

I'm going to improve the IndexError: single positional indexer is out-of-bounds error because it is impossible debug without context.

jgrabek · 2021-11-03T19:47:11Z

Cleaning cache folders helped - all errors are gone. Thank you very much for your advice. The library looks great and I hope I'll use it in near future to test my models.

Regarding quote_account_value_in I left USDT because there are no nans in metrics anymore and for Binance USD throws error

Traceback (most recent call last):
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/exchanges/interfaces/paper_trade/paper_trade_interface.py", line 306, in get_account
    return trade_local.get_account(symbol)
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/exchanges/interfaces/paper_trade/local_account/trade_local.py", line 125, in get_account
    return copy.deepcopy(utils.AttributeDict(local_account.account[asset_id]))
KeyError: 'USD'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jakub/PycharmProjects/bot/blankly/rsi.py", line 39, in <module>
    results = strategy.backtest(to='1y', initial_values={'USDT': 10000})
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/frameworks/strategy/strategy_base.py", line 495, in backtest
    results = self.backtesting_controller.run()
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 637, in run
    available_dict, no_trade_dict = self.format_account_data(self.initial_time)
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/exchanges/interfaces/paper_trade/backtest_controller.py", line 448, in format_account_data
    true_account[i] = self.interface.get_account(i)
  File "/home/jakub/PycharmProjects/bot/venv/lib/python3.8/site-packages/blankly/exchanges/interfaces/paper_trade/paper_trade_interface.py", line 309, in get_account
    raise KeyError("Symbol not found. This can be caused by an invalid quote currency "
KeyError: 'Symbol not found. This can be caused by an invalid quote currency in backtest.json.'

EmersonDove self-assigned this Nov 2, 2021

EmersonDove mentioned this issue Nov 8, 2021

Backtest crashing depending on time-frame #71

Closed

EmersonDove added a commit that referenced this issue Nov 11, 2021

Address difficult to understand error in #68

ceb2d3c

EmersonDove mentioned this issue Nov 19, 2021

v1.13.0-beta Update #67

Merged

EmersonDove added a commit that referenced this issue Nov 20, 2021

Address difficult to understand error in #68

39966bd

EmersonDove closed this as completed Aug 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example scripts error #68

Example scripts error #68

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 2, 2021 •

edited

Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 2, 2021 •

edited

Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 2, 2021 •

edited

Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 3, 2021 •

edited

Loading

jgrabek commented Nov 3, 2021

Example scripts error #68

Example scripts error #68

Comments

jgrabek commented Nov 2, 2021

Description

settings.json

backtest.json

Error

Platform Info

EmersonDove commented Nov 2, 2021 • edited Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 2, 2021 • edited Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 2, 2021 • edited Loading

jgrabek commented Nov 2, 2021

EmersonDove commented Nov 3, 2021 • edited Loading

jgrabek commented Nov 3, 2021

EmersonDove commented Nov 2, 2021 •

edited

Loading

EmersonDove commented Nov 2, 2021 •

edited

Loading

EmersonDove commented Nov 2, 2021 •

edited

Loading

EmersonDove commented Nov 3, 2021 •

edited

Loading