SPs continue to see commP errors on larger Estuary deals #303
Assignees
Labels
Comments
|
Hi @stuberman - thanks for your report. Quick clarifying question:
So if you use |
|
That is not correct. |
|
Bidbot requires either Boost or Marketv1 as does Estuary. |
|
I see exact same problem. All larger deals from Estuary contains incorrect data. This seems to be an estuary only problem, so it looks like the data preparation is broken.
|
|
Another storage provider seeing the CommP error: Confirmed in logs: |
|
I think I just got couple of larger deals here for troubleshooting. 0% success rate :(
|
|
Steps to replicate:
cc @stuberman |
|
Upon receiving the staging file ( package main
import (
"fmt"
m "github.com/filecoin-project/boost/storagemarket"
"github.com/filecoin-project/go-state-types/abi"
"github.com/ipfs/go-cid"
)
func main() {
InboundFilePath := "1e347bf0-ce28-47a2-bf15-59472f4f87cf.download"
var PieceSize abi.PaddedPieceSize = 34359738368
// expected from Estuary
clientPieceCid, err := cid.Decode("baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii")
if err != nil {
panic(err)
}
pieceCid, err := m.GeneratePieceCommitment(InboundFilePath, PieceSize)
if err != nil {
fmt.Printf("failed to generate CommP: %s\n", err)
}
fmt.Println(pieceCid.String())
if pieceCid != clientPieceCid {
fmt.Printf("commP mismatch, expected=%s, actual=%s\n", clientPieceCid, pieceCid)
}
}giving us the expected output, that also came out of commP mismatch, expected=baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii, actual=baga6ea4seaqmjjmizj5fywuonasf2kozxfumae54nthan42u2ha3idrtfl35epqHowever, when I ran the same code against the original CAR file, there was no error, the piece commitment was generated correctly ( Upon further investigation, I noticed that the .download file received by boost and the original CAR file were identical in size, however their contents differed, confirmed with a simple sum call: boost's download file: % ls -slap
33756598209 30 Jul 22:51 1e347bf0-ce28-47a2-bf15-59472f4f87cf.download
% shasum -a 256 1e347bf0-ce28-47a2-bf15-59472f4f87cf.download
6465fcb4b9f4641a8572635a6e14d4d9ec5751078f7e2b9325a619cd63a6fc1foriginal CAR file: % ls -salp ~/test-cars/output.car
33756598209 Jul 30 19:23 /root/test-cars/output.car
% shasum -a 256 ~/test-cars/output.car
7d1896d0b758cfc3f4951d5adf66cc1114517f753276c55d2b903aa16e3aa2e7This implies that somehow the CAR's contents are mutated in transit/process from Estuary to the filesystem through Boost, explaining the sum / commP discrepancies. |
|
@jlogelin can you use the go-car cli util to inspect the blocks for both car files and do a diff |
|
@en0ma Confirmed that the block CIDS are identical, they just aren't in the same order: list1 = []
list2 = []
with open('output.car.out') as car_out:
for line in car_out:
list1.append(line.rstrip('\n').split(','))
with open('1e347bf0-ce28-47a2-bf15-59472f4f87cf.out') as car_out:
for line in car_out:
list2.append(line.rstrip('\n').split(','))
print("Unsorted equality: ", list1 == list2)
print("Sorted equality: ",sorted(list1) == sorted(list2))Output |
|
@stuberman are you still experiencing this, so I know if we can close this ticket |
|
I am unable to tell, most Estuary deals fail before they download for the last month. |
|
I agree with @stuberman. Download errors from Estuary is now so dominant that it's quite hard to test for this issue. I don't know how it can run so broken, but apparently it keeps going like this for weeks. I think most SPs are probably not complaining, as this is like a never ending story :( See my resent boost logs here (Its somewhat impressive):
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
SPs receiving large deals continue to see commP failures such as the one below:
Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii, actual=baga6ea4seaqmjjmizj5fywuonasf2kozxfumae54nthan42u2ha3idrtfl35epqTo Reproduce
Accept Estuary deals of all sizes
I am running Lotus v1.16.0 and separate Boost v1.1.0
I do not see this behavior with other large deals from tools like bidbot.
Expected behavior
I do not expect to see any commP failures after downloading any files.
Actual behavior
Deal failures after download on larger deals (see table below)
Additional context
Here is a list of recent Estuary deals as shown in Boost GUI from client ID
f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36aNotice the deal sizes.
The text was updated successfully, but these errors were encountered: