Verified Deals not transferring & unable to cancel in data-transfers

[cwhiggins]

I was sent five verified deals all got stuck in transferring. I am also unable to remove them from data-transfers using cancel.

miner id f015927 over 24 hours

Storage Deals List

...zmac7vke  0        StorageDealTransferring             f3ufzpudvsjqyiholpxiqoomsd2svy26jvy4z4pzodikgovkhkp6ioxf5p4jbpnf7tgyg67dny4j75e7og7zeq  16GiB   0.0000016601248 FIL       1037578
...ujxoprpm  0        StorageDealTransferring             f3ufzpudvsjqyiholpxiqoomsd2svy26jvy4z4pzodikgovkhkp6ioxf5p4jbpnf7tgyg67dny4j75e7og7zeq  16GiB   0.0000016596848 FIL       1037303
...4alrdv6m  0        StorageDealTransferring             f3ufzpudvsjqyiholpxiqoomsd2svy26jvy4z4pzodikgovkhkp6ioxf5p4jbpnf7tgyg67dny4j75e7og7zeq  16GiB   0.0000016594256 FIL       1037141
...7rskdode  0        StorageDealTransferring             f3ufzpudvsjqyiholpxiqoomsd2svy26jvy4z4pzodikgovkhkp6ioxf5p4jbpnf7tgyg67dny4j75e7og7zeq  16GiB   0.0000016637936 FIL       1039871
...4g2do2ai  0        StorageDealTransferring             f3ufzpudvsjqyiholpxiqoomsd2svy26jvy4z4pzodikgovkhkp6ioxf5p4jbpnf7tgyg67dny4j75e7og7zeq  8GiB    0.0000016635632 FIL       1039727

Data Transfers List

3641  Ongoing  ...tiMLWaNJ     ...46uhljsa  N           0B           ...ylahwpfofne6dxgb23o5nij6uszmac7vke"}}  
3650  Ongoing  ...tiMLWaNJ     ...cck2ywmw  N           0B           ...sjq5b5uozexdcksuq5p45aeoslujxoprpm"}}  
3656  Ongoing  ...tiMLWaNJ     ...7pd2uwdg  N           0B           ...n2ta4gat5xlsaomokghbv5gtbd4alrdv6m"}}  
3662  Ongoing  ...tiMLWaNJ     ...4w37u75s  N           0B           ...f7tz4yesxisms6xshil2ixyun77rskdode"}}  
3669  Ongoing  ...tiMLWaNJ     ...4czdlu5s  N           0B           ...ewiavjnrpcq74gounfujsahka74g2do2ai"}}

Daemon:  1.2.2+git.93d26195f+api1.0.0
Local: lotus version 1.2.2+git.93d26195f

[dirkmc]

Thanks for the bug report. I believe this will be fixed by filecoin-project/go-data-transfer#127

I am also unable to remove them from data-transfers using cancel

What happens when you try to remove them? Do you get an error message? I'm wondering if this is related to an existing issue about a confusing error message: #5146

[cwhiggins]

Upon restarting miner at one point, storage deals did go to error state. Deals are however still in data transfer list. One cancelled no issue now, others gave error, but I think with restart will also clear. Will try tomorrow, if possible.

[dirkmc]

Ok sounds good - whenever you see an error message please include it in any new issue / comment as it helps us track down what the problem is

[cwhiggins]

2020-12-15T09:41:56.936-0500	WARN	dt-impl	impl/impl.go:263	unable to close channel: %!w(datatransfer.errorType=channel not found)
2020-12-15T09:42:01.936-0500	WARN	dt-impl	impl/events.go:236	channel 12D3KooWA9KBMzTVBaYjo8jV6BBTaagy9hbiF85PuaGqtiMLWaNJ-12D3KooWQGPq3PGYwecDrdb8Xv8N8KXqsVs4VP7EXBh671nttjpT-3641 has stalled or disconnected
2020-12-15T09:42:01.936-0500	WARN	dt-impl	impl/impl.go:270	Unable to send cancel message: context canceled
2020-12-15T09:42:01.936-0500	ERROR	storagemarket_impl	dtutils/dtutils.go:66	processing dt event: %!w(*xerrors.errorString=&{normal shutdown of state machine {[7548558 19095176 5755594]}})
2020-12-15T09:42:01.937-0500	WARN	rpc	go-jsonrpc@v0.1.2/handler.go:279error in RPC call to 'Filecoin.MarketCancelDataTransfer': Unable to send cancel message: context canceled
2020-12-15T09:42:01.938-0500	ERROR	storagemarket_impl	dtutils/dtutils.go:66	processing dt event: %!w(*xerrors.errorString=&{normal shutdown of state machine {[7548558 19095176 5755594]}})

this was in miner log after running a data-transfer cancel

ERROR: Unable to send cancel message: context canceled

this was displayed in terminal issuing command

[cwhiggins]

In this instance the data transfer is now no longer in list

[cwhiggins]

2020-12-15T09:46:03.140-0500	WARN	dt-impl	impl/impl.go:263	unable to close channel: %!w(datatransfer.errorType=channel not found)
2020-12-15T09:46:03.140-0500	WARN	rpc	go-jsonrpc@v0.1.2/handler.go:279error in RPC call to 'Filecoin.MarketCancelDataTransfer': unable to send cancel to channel FSM: %!w(<nil>):
    github.com/filecoin-project/go-data-transfer/impl.(*manager).CloseDataTransferChannel
        /home/chris/go/pkg/mod/github.com/filecoin-project/go-data-transfer@v1.2.2/impl/impl.go:278

miner logs

ERROR: unable to send cancel to channel FSM: %!w(<nil>)

terminal display

In this instance transfer remains in list. Rerunning command returns same error. Will try again after a miner restart

[dirkmc]

Ok sounds like an instance of #5146

Thanks for the bug report. I'm going to close this so that we can track the issue in one place.