GPU enabled run doesn't discover bitcoin password

[mclate]

When enabling GPU, passwords not being discovered in bitcoin wallet, while same CPU-enabled run does find correct password.

Environment:

• OS: Mac OS X 10.12.6
• Python: 2.7.10
• pyopencl: 2017.2
• pycrypto: 2.6.1
• GPU: OpenCL GPU AMD Radeon Pro 455 Compute Engine
• Wallet: bitcoin, Wallet difficulty: 185,184 SHA-512 iterations (different difficulties have same result)
• btcrecover.py: 0.17.5

Steps to reproduce:

1. Create new wallet (i've used Bitcoin Core) and set some simple passphrase (say, 123)
2. Add simple token to tokenfile: %3d
3. Run btcrecover (simple run, with only wallet and token arguments): password is found on 123 iteration as expected.
4. Run same btcrecover command, add --enable-gpu: Password search exhausted

For sake of testing tried to run on integrated GPU and CPU via OpenCL:

• Using OpenCL GPU Intel(R) HD Graphics 530 - password was found as expected.
• Using OpenCL CPU Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz - password found as expected

Let me know if any extra info is needed

[gurnec]

Thanks for the detailed bug report, it's very much appreciated!

Unfortunately, I don't have access to an OS X box with a GPU to reproduce this, so it may be difficult or impossible for me to figure this out. It may require a bunch of trial-and-error on both our parts, if/when you can spare the time to help me....

First off, could you run the run-all-tests.py script to check that the only tests which fail are the GPU tests (I'd guess 5 of them will fail, all with cl in the name)? Before you do, there's a chance running this could crash the video drivers... I've never tested this on OS X with a GPU before, and there are a couple of tests which are skipped on Windows because they crash the video drivers, but they're not currently skipped on OS X... so please save anything you're working on first!

Next, can you try removing this pragma directive from line 235 of the sha512-bc-kernel.cl file and trying your test wallet again? If this doesn't help, can you try removing all of the #pragma unroll lines and try again?

Thanks!

[mclate]

======================================================================
ERROR: test_bitcoincore_cl (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/test/test_passwords.py", line 1566, in test_bitcoincore_cl
    btcrpass.loaded_wallet.init_opencl_kernel([dev], [4], [4], 200)
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 836, in init_opencl_kernel
    error_exit("--local-ws of", local_ws[i], "exceeds max of", max_local_ws, "for GPU '"+device.name.strip()+"' with Bitcoin Core wallets")
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 2522, in error_exit
    sys.exit(b" ".join(map(str, _do_safe_print(prog+": error:", *messages))))
SystemExit: run-all-tests.py: error: --local-ws of 4 exceeds max of 1 for GPU 'Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz' with Bitcoin Core wallets

======================================================================
ERROR: test_bitcoincore_cl_no_interrupts (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/test/test_passwords.py", line 1599, in test_bitcoincore_cl_no_interrupts
    btcrpass.loaded_wallet.init_opencl_kernel([dev], [4], [4], 1)
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 836, in init_opencl_kernel
    error_exit("--local-ws of", local_ws[i], "exceeds max of", max_local_ws, "for GPU '"+device.name.strip()+"' with Bitcoin Core wallets")
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 2522, in error_exit
    sys.exit(b" ".join(map(str, _do_safe_print(prog+": error:", *messages))))
SystemExit: run-all-tests.py: error: --local-ws of 4 exceeds max of 1 for GPU 'Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz' with Bitcoin Core wallets

======================================================================
ERROR: test_bitcoincore_cl_unicode (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/test/test_passwords.py", line 1584, in test_bitcoincore_cl_unicode
    btcrpass.loaded_wallet.init_opencl_kernel([dev], [4], [4], 200)
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 836, in init_opencl_kernel
    error_exit("--local-ws of", local_ws[i], "exceeds max of", max_local_ws, "for GPU '"+device.name.strip()+"' with Bitcoin Core wallets")
  File "/Users/o_o/bitcoin/btcrecover/btcrecover/btcrpass.py", line 2522, in error_exit
    sys.exit(b" ".join(map(str, _do_safe_print(prog+": error:", *messages))))
SystemExit: run-all-tests.py: error: --local-ws of 4 exceeds max of 1 for GPU 'Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz' with Bitcoin Core wallets

----------------------------------------------------------------------

Tests do fail, but notice that they use CPU as gpu unit (regardless of what gpu is selected in Mac settings)

Result is the same with 235 pragma being removed as well as all pragmas being removed.

P.S. No crashes were detected while running on integrated video as well as on AMD one.

[gurnec]

Thanks for running the tests (and sorry for the delay).

A couple of things here...

First off, the unit tests test all available OpenCL devices in whatever order they're returned by the OpenCL ICD loader. I'm guessing the CPU is returned first on your system, and when it fails, the GPU tests are skipped. The failing tests above are due to a bug in the unit tests--I think this is now fixed. Would you mind grabbing the most recent version and running the tests again?

Next, can you confirm that there are a total of 5 tests which fail (3 grouped together near the bottom, and 2 farther up)? If there are fewer than 5 tests which fail, are they under the earlier ASCII test section, or the later Unicode one?

I made a debug version of btcrecover which outputs a bunch of intermediate results. After the tests above, could you download this version (from this branch if you want to see the source) and run it like this:

echo 123 | ./btcrecover.py --enable-gpu --gpu-names Radeon --global-ws 1 --passwordlist --wallet btcrecover\test\test-wallets\bitcoincore-wallet.dat > debug.txt

and attach the resulting debug.txt file to a response here?

Thank you!

[mclate]

Running tests against provided master branch:

ASCII mode:

======================================================================
FAIL: test_bitcoincore_cl (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover2/btcrecover-master/btcrecover/test/test_passwords.py", line 1583, in test_bitcoincore_cl
    dev.name.strip() + " failed to find password")
AssertionError: AMD Radeon Pro 455 Compute Engine failed to find password

======================================================================
FAIL: test_bitcoincore_cl_no_interrupts (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover2/btcrecover-master/btcrecover/test/test_passwords.py", line 1614, in test_bitcoincore_cl_no_interrupts
    [tstr("btcr-wrong-password-3"), tstr("btcr-test-password"), tstr("btcr-wrong-password-4")]), (tstr("btcr-test-password"), 2))
AssertionError: Tuples differ: (False, 3) != ('btcr-test-password', 2)

First differing element 0:
False
'btcr-test-password'

- (False, 3)
+ ('btcr-test-password', 2)

Unicode mode:

======================================================================
FAIL: test_bitcoincore_cl (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover2/btcrecover-master/btcrecover/test/test_passwords.py", line 1583, in test_bitcoincore_cl
    dev.name.strip() + " failed to find password")
AssertionError: AMD Radeon Pro 455 Compute Engine failed to find password

======================================================================
FAIL: test_bitcoincore_cl_no_interrupts (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover2/btcrecover-master/btcrecover/test/test_passwords.py", line 1614, in test_bitcoincore_cl_no_interrupts
    [tstr("btcr-wrong-password-3"), tstr("btcr-test-password"), tstr("btcr-wrong-password-4")]), (tstr("btcr-test-password"), 2))
AssertionError: Tuples differ: (False, 3) != (u'btcr-test-password', 2)

First differing element 0:
False
u'btcr-test-password'

- (False, 3)
+ (u'btcr-test-password', 2)

======================================================================
FAIL: test_bitcoincore_cl_unicode (btcrecover.test.test_passwords.Test08KeyDecryption)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/o_o/bitcoin/btcrecover2/btcrecover-master/btcrecover/test/test_passwords.py", line 1598, in test_bitcoincore_cl_unicode
    dev.name.strip() + " failed to find password")
AssertionError: AMD Radeon Pro 455 Compute Engine failed to find password

----------------------------------------------------------------------

Debug file for debug version (with exhausted password as well):
debug.txt

[gurnec]

Thanks!

The debug file differs from what it should be at the very beginning... for some reason, the initial data that's sent to the GPU isn't what it should be/gets corrupted right away. My initial guess is an issue with OpenCL or the GPU drivers.

Are these the stock OS X drivers? What, if anything, did you have to do to get OpenCL and PyOpenCL running on your box? Are you using brew/gcc, xcode, or something else?

I'll have to think about next steps, I'm not sure at the moment....