R4R: PrivKeyLedgerSecp256k1: lost connection #3431

jleni · 2019-01-29T18:13:25Z

This PR solves issue #3335

This PR upgrades ledger-cosmos-go and brings the following improvements:

Migrates USB HID library to a better maintained implementation
Fixes a problem in MacOS that was resulting in "lost connection" messages
Fixes issues in Windows clients
Additional unit tests in each of the layers
better resource handling in gaia. Ledger connections are not kept open, etc.
signature fixes: conversion from DER to BER encoding
additional tests
new make target to run ledger integration tests (with real device) make test_ledger

Technical description

This upgrade also results in multiple changes/fixes in several dependencies:

HID https://github.com/ZondaX/hid

Moved from a fork of https://github.com/brejski/hid to https://github.com/karalabe/hid
There were some issues in OSX in the other hid wrapper that were affecting gaia.
Fixes a problem upstream in karalabe's hid that was affecting Windows:
Incorrect number of written bytes karalabe/hid#16
Resolves gcc Windows warning affecting upstream
Zondax/hid@302fd40

Ledger-go https://github.com/ZondaX/ledger-go

Upgrades HID dependency and adjusts to the new API Fixing connection issues in OSX Zondax/ledger-go#1
Incorporates fixes for issues in karalabe's HID (applicable to Linux & Windows) Fixes for Linux and Windows Zondax/ledger-go#2
Avoid case mismatches in gopkg.toml vendor deps. This was affecting downstream repos. avoid case mismatches in vendor deps Zondax/ledger-go#5
Support better resource handling Improve resource handling Zondax/ledger-go#6

Ledger-cosmos-go https://github.com/ZondaX/ledger-cosmos-go

Fixes OSX connection issues https://github.com/ZondaX/ledger-cosmos-go/pull/21/files
Improves discovery in Linux. Update to get Windows fix. Additional unit tests (https://github.com/ZondaX/ledger-cosmos-go/pull/22/files)
Avoid case mismatches in gopkg.toml vendor deps. This was affecting downstream repos. avoid case mismatches in vendor deps Zondax/ledger-cosmos-go#23
Support better resource handling Improve resource handling Zondax/ledger-cosmos-go#24

Linked to github-issue with discussion and accepted design OR link to spec that describes this work.
Wrote tests
Updated relevant documentation (docs/)
Added entries in PENDING.md with issue #
rereviewed Files changed in the github PR explorer

For Admin Use:

Added appropriate labels to PR (ex. wip, ready-for-review, docs)
Reviewers Assigned
Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

jackzampolin · 2019-01-29T22:22:44Z

Hey @jleni can you also link to a PR to ZondaX/ledger-cosmos-go with the changes that you are importing into the SDK? We can't really reason about the changes if we don't have that. Also this PR needs a rebase.

jleni · 2019-01-30T09:24:15Z

Yes, the conflict appeared once another PR was merged last night. No worries, the PR is still WIP.

@jackzampolin I have now included a longer technical explanation for all the changes that were made with links to each respective layer and PR.

codecov · 2019-01-31T21:40:25Z

Codecov Report

Merging #3431 into develop will increase coverage by 6.23%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #3431      +/-   ##
===========================================
+ Coverage    52.97%   59.21%   +6.23%     
===========================================
  Files          142      134       -8     
  Lines        10363     9940     -423     
===========================================
+ Hits          5490     5886     +396     
+ Misses        4550     3680     -870     
- Partials       323      374      +51

jleni · 2019-02-01T15:25:59Z

As discussed with @cwgoes @jackzampolin

I have noticed something that may result in problems. At the moment, when a key is requested to a ledger device, a PrivKeyLedgerSecp256k1 object keeps the device together with the keys. However, this object is actually a USB resource that is never released.

At the moment, there are a few problems in gaia and still in this PR:

discovery opens the resource but never closes it... calling discovery twice fails as the resource has not been released.
the cached usb resource can fail in many ways (disconnects, etc.)
we might need to add code to reconnect, verify that it is the same device, etc.
PrivKeyLedgerSecp256k1 gets serialized to amino I am sure it is skipping the connection object, but still.. it feels odd.

We agreed on refactoring gaia's code, so resources are released ASAP and pks are rechecked before signing. As PKs are checked, it should be possible to detect if devices are swapped or there are multiple devices and the incorrect one is re-discovered.

jleni · 2019-02-01T21:20:55Z

In addition to these changes, I have noticed that we have a mix of DER vs BER signatures. I will try to unify this a bit too to avoid confusion.

jackzampolin · 2019-02-01T22:10:45Z

This PR just updated dependancies.

jleni · 2019-02-01T22:16:36Z

True, for now the changes are in other layers as described above.
I will soon push all the other local changes I made that apply to the SDK.
Once the fixes in the dependencies got integrated, there are other reasons why there might be connection issues. Also unit tests for ledger in the SDK are quite weak. Actually, they are skipped at the moment.

jackzampolin · 2019-02-01T22:17:54Z

@jleni Looking forward to unit testing for this really crucial usecase. Thank you for adding them!

jleni · 2019-02-02T11:44:17Z

I would say this is not the PR/issue for that.. but once #3461 and #3431 are merged, I would open another issue to add cli tests that check a real ledger device too.

jleni · 2019-02-02T14:07:55Z

@jackzampolin Given that these fixes have uncovered an existing incompatibility between DER and BER signatures, I would suggest considering this PR for prelaunch: without these changes, signatures will be invalid.

alexanderbez

Thanks @jleni! I left a few remarks 👍

crypto/ledger_integration_test.go

crypto/ledger_secp256k1.go

removing case mismatches in dependencies

Co-Authored-By: jleni <juan.leni@zondax.ch>

jleni · 2019-02-04T07:52:21Z

It seems an unrelated issue hit integration_tests here:

    gobash.go:80: Running gaiad start --home=/tmp/gaia_integration_TestGaiaCLISubmitProposal_210313979/.gaiad --rpc.laddr=tcp://0.0.0.0:32893 --p2p.laddr=tcp://0.0.0.0:38125
panic: Get http://localhost:32893/block: read tcp 127.0.0.1:53250->127.0.0.1:32893: read: connection reset by peer [recovered]
	panic: Get http://localhost:32893/block: read tcp 127.0.0.1:53250->127.0.0.1:32893: read: connection reset by peer

Should I open another issue for that? I can try rerunning but there is a panic to analyse there anyway.

jleni · 2019-02-04T07:54:47Z

Here is the link to the CI error:
https://circleci.com/gh/ZondaX/cosmos-sdk/343

jleni · 2019-02-04T08:01:22Z

Running integration_tests again and they passed. There is definitely a sporadic unrelated issue in integration_tests that needs to be looked at.

jleni · 2019-02-05T06:40:00Z

@jackzampolin @alessio Fixed a conflict in Pending.md.
Hopefully, this is also ready to go.

alessio

Very good indeed. Nihil obstat from me. @cwgoes fancy taking a look at this?

jleni · 2019-02-05T09:35:19Z

@jackzampolin @alessio This PR and #3461 have changes in similar areas. Whichever you merge first will result in changes on the other side.
I would say, we merge #3461 first before it goes stale and I adjust this PR accordingly.

jackzampolin · 2019-02-05T16:24:22Z

@jleni Per your suggestion I've merged #3461 first. This now has merge conflicts.

jleni · 2019-02-05T16:25:11Z

Yes, thanks! I will adjust this in the next hour

jackzampolin

Thank you @jleni! LGTM

jleni requested review from cwgoes, ebuchman and rigelrozanski as code owners January 29, 2019 18:13

jleni added the C:Ledger Issues and features related Ledger integration and functionality label Jan 29, 2019

jleni added the wip label Jan 30, 2019

jleni force-pushed the fix/lost-conn branch from e6da6b0 to 985564a Compare February 1, 2019 15:16

jleni force-pushed the fix/lost-conn branch from 1a19d4b to 36951ce Compare February 2, 2019 13:12

jleni changed the title ~~WIP: PrivKeyLedgerSecp256k1: lost connection~~ R4R: PrivKeyLedgerSecp256k1: lost connection Feb 2, 2019

jleni force-pushed the fix/lost-conn branch from 36951ce to 12134b0 Compare February 2, 2019 13:45

jleni added the ready-for-review label Feb 2, 2019

alexanderbez reviewed Feb 3, 2019

View reviewed changes

crypto/ledger_integration_test.go Outdated Show resolved Hide resolved

crypto/ledger_integration_test.go Outdated Show resolved Hide resolved

crypto/ledger_secp256k1.go Show resolved Hide resolved

crypto/ledger_secp256k1.go Outdated Show resolved Hide resolved

alessio reviewed Feb 3, 2019

View reviewed changes

crypto/ledger_secp256k1.go Outdated Show resolved Hide resolved

jleni added 8 commits February 4, 2019 07:47

Upgrade cosmos-ledger-go to v0.9.4

1d7fcd8

upgrading to cosmos-ledger-go v0.9.5

a0b517a

upgrading to v0.9.6

ac1ceb6

removing case mismatches in dependencies

wip

bd7d674

improving tests

ca1cac1

Upgrading to v0.9.7

f48f177

Convert signatures DER->BER + tests

612c8d3

refactoring device handling

651169a

jleni and others added 9 commits February 4, 2019 07:47

moving to existing types for consistency

6c4840c

make target + more tests

5100162

Removing old redundant types

eff4dcd

fixing linter issue in defer calls

8bc4736

updating pending.md

fbf5869

fixing tagging issue

e3b2a21

Update crypto/ledger_secp256k1.go

1bfcf51

Co-Authored-By: jleni <juan.leni@zondax.ch>

Update ledger_integration_test.go

58f3060

Making explicit this tag is test related only

16e0362

jleni force-pushed the fix/lost-conn branch from b150e63 to 16e0362 Compare February 4, 2019 06:53

jleni added 2 commits February 4, 2019 19:01

Merge branch 'develop' into fix/lost-conn

7e9ade0

Merge branch 'develop' into fix/lost-conn

d1dbd4f

alessio approved these changes Feb 5, 2019

View reviewed changes

Merging changes from develop

3c34033

jackzampolin approved these changes Feb 5, 2019

View reviewed changes

jackzampolin merged commit 174ec0c into cosmos:develop Feb 5, 2019

jleni deleted the fix/lost-conn branch February 5, 2019 18:22

zmanian mentioned this pull request Feb 9, 2019

Sending a transfer transaction from the Ledger Nano S fails #3573

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

R4R: PrivKeyLedgerSecp256k1: lost connection #3431

R4R: PrivKeyLedgerSecp256k1: lost connection #3431

jleni commented Jan 29, 2019 •

edited

Loading

jackzampolin commented Jan 29, 2019

jleni commented Jan 30, 2019

codecov bot commented Jan 31, 2019 •

edited

Loading

jleni commented Feb 1, 2019

jleni commented Feb 1, 2019 •

edited

Loading

jackzampolin commented Feb 1, 2019

jleni commented Feb 1, 2019

jackzampolin commented Feb 1, 2019

jleni commented Feb 2, 2019

jleni commented Feb 2, 2019 •

edited

Loading

alexanderbez left a comment

jleni commented Feb 4, 2019

jleni commented Feb 4, 2019

jleni commented Feb 4, 2019 •

edited

Loading

jleni commented Feb 5, 2019

alessio left a comment

jleni commented Feb 5, 2019

jackzampolin commented Feb 5, 2019

jleni commented Feb 5, 2019

jackzampolin left a comment

R4R: PrivKeyLedgerSecp256k1: lost connection #3431

R4R: PrivKeyLedgerSecp256k1: lost connection #3431

Conversation

jleni commented Jan 29, 2019 • edited Loading

Technical description

jackzampolin commented Jan 29, 2019

jleni commented Jan 30, 2019

codecov bot commented Jan 31, 2019 • edited Loading

Codecov Report

jleni commented Feb 1, 2019

jleni commented Feb 1, 2019 • edited Loading

jackzampolin commented Feb 1, 2019

jleni commented Feb 1, 2019

jackzampolin commented Feb 1, 2019

jleni commented Feb 2, 2019

jleni commented Feb 2, 2019 • edited Loading

alexanderbez left a comment

Choose a reason for hiding this comment

jleni commented Feb 4, 2019

jleni commented Feb 4, 2019

jleni commented Feb 4, 2019 • edited Loading

jleni commented Feb 5, 2019

alessio left a comment

Choose a reason for hiding this comment

jleni commented Feb 5, 2019

jackzampolin commented Feb 5, 2019

jleni commented Feb 5, 2019

jackzampolin left a comment

Choose a reason for hiding this comment

jleni commented Jan 29, 2019 •

edited

Loading

codecov bot commented Jan 31, 2019 •

edited

Loading

jleni commented Feb 1, 2019 •

edited

Loading

jleni commented Feb 2, 2019 •

edited

Loading

jleni commented Feb 4, 2019 •

edited

Loading