Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): upgrade to celestia-app v2.0.0 #3453

Merged
merged 87 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from 85 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
63bc288
fix: imports
rootulp May 31, 2024
926e5a3
fix: error message
rootulp May 31, 2024
8c75d6a
wip: adapt to v2
rootulp May 31, 2024
8c9fc08
fix: copy broadcastTx from v1.x celestia-app
rootulp Jun 6, 2024
c3e84e1
go mod tidy
rootulp Jun 6, 2024
fc8dac3
refactor: error msg
rootulp Jun 6, 2024
44e65e0
refactor: rename to namespace
rootulp Jun 6, 2024
e1286a4
test: validate extended header
rootulp Jun 6, 2024
866cd1e
test: extended header validate
rootulp Jun 6, 2024
69a05a9
Merge branch 'main' into rp/support-v2
rootulp Jun 6, 2024
99b0b29
fix: most lint errors
rootulp Jun 7, 2024
b0401f3
refactor: rename b to appBlob
rootulp Jun 7, 2024
54eadb7
fix: last linter warning about copying mutexes
rootulp Jun 7, 2024
bcde3f7
Merge branch 'main' into rp/support-v2
rootulp Jun 25, 2024
9c9ff4e
bump to celestia-app v2.0.0-rc2
rootulp Jun 25, 2024
a59a373
fix: split txClient from signer
rootulp Jun 25, 2024
ae72d69
fix: lint
rootulp Jun 25, 2024
b7633c9
chore(deps): upgrade to Go 1.22.4
rootulp Jun 25, 2024
dff037e
fix: lint
rootulp Jun 25, 2024
efb6c9d
try to fix lint
rootulp Jun 27, 2024
2bde950
Merge branch 'main' into rp/support-v2
rootulp Jun 27, 2024
5542aa7
Merge branch 'main' into rp/support-v2
rootulp Jul 2, 2024
f95d9c7
refactor: update core_access.go implementation
rootulp Jul 2, 2024
bccc2b4
test: remove min gas price assertion b/c TODO
rootulp Jul 2, 2024
f147fb6
fix: tests by converting squareblob to nodeblob
rootulp Jul 2, 2024
3e3fedd
refactor: stop using convert
rootulp Jul 4, 2024
8b5ef8f
test: attempt to fix TestCoreExchange_RequestHeaders
rootulp Jul 4, 2024
383490c
fix: da_test comma
rootulp Jul 4, 2024
4857abe
fix: TestBlob
rootulp Jul 4, 2024
a5650ac
HACKHACK: I think bug in celestia-app testnode that prevents custom c…
rootulp Jul 4, 2024
e2eb8e9
Revert "HACKHACK: I think bug in celestia-app testnode that prevents …
rootulp Jul 4, 2024
c4be6a8
resolve merge conflicts
rootulp Jul 9, 2024
9e0f501
test: attempt to fix integration tests
rootulp Jul 9, 2024
c120fd2
wip: fix some lint issues
rootulp Jul 9, 2024
4eddd43
fix: build
rootulp Jul 9, 2024
b8fbd4b
fix: manual re-review
rootulp Jul 9, 2024
daf1a81
attempt to fix import order
rootulp Jul 9, 2024
0f48e13
chore(deps): bump to v2.0.0-rc3
rootulp Jul 16, 2024
8b8efd2
fix: core access test
rootulp Jul 16, 2024
6bd366f
refactor: add comment
rootulp Jul 16, 2024
9db2d2c
go mod tidy
rootulp Jul 16, 2024
69be2c9
make fmt
rootulp Jul 16, 2024
587914a
fix: TestGetBalance
rootulp Jul 16, 2024
8fb0d04
fix: lint
rootulp Jul 16, 2024
4df4240
debug: attempt to initialize tx client mtx
rootulp Jul 16, 2024
a2b808d
chore(deps): upgrade to celestia-core v1.38.0
rootulp Jul 16, 2024
f325abf
wip: need another change in celestia-app
rootulp Jul 16, 2024
99265ec
chore(deps): upgrade to v2.0.0-rc4
rootulp Jul 17, 2024
383380a
fix: exchange_test
rootulp Jul 17, 2024
4d51a78
refactor: IntegrationTestSuite
rootulp Jul 17, 2024
1d82662
go mod tidy
rootulp Jul 17, 2024
dc369f2
docs: add comments
rootulp Jul 17, 2024
9acbbc8
fix: TestListenerWithNonEmptyBlocks
rootulp Jul 17, 2024
052cf18
refactor: DefaultTestConfig
rootulp Jul 17, 2024
38350c3
refactor: DefaultTestConfig more
rootulp Jul 17, 2024
5f263a1
fix: TestRemoteClient_StartBlockSubscription_And_GetBlock
rootulp Jul 17, 2024
125d766
fix: TestBlockFetcherHeaderValues
rootulp Jul 17, 2024
e3dda78
fix: TestBlockFetcher_GetBlock_and_SubscribeNewBlockEvent
rootulp Jul 17, 2024
e2be6fa
fix: nd_test by bumping timeout
rootulp Jul 17, 2024
db1d7cd
debug: double fraud proof test timeoutgs
rootulp Jul 17, 2024
53a114e
Merge branch 'main' into rp/support-v2
rootulp Jul 17, 2024
d286ad2
Merge branch 'main' into rp/support-v2
rootulp Jul 18, 2024
d016d2b
attempt to resolve merge conflicts
rootulp Jul 24, 2024
cf05699
fix: convert types between app and core
rach-id Jul 25, 2024
a664f30
refactor: small refactor to use a single loop
rach-id Jul 25, 2024
198d02e
fix: remove big test cases
rach-id Jul 25, 2024
b254dc4
Merge pull request #1 from rach-id/rp/support-v2
rootulp Jul 25, 2024
87e8612
fix: lint
rootulp Jul 25, 2024
33fafc9
fix: correctly setup the subtreeRoot threshold and re-introduce the n…
rach-id Jul 26, 2024
8e92874
Merge pull request #2 from rach-id/rp/support-v2
rootulp Jul 30, 2024
d8cdef3
Merge branch 'main' into rp/support-v2
rootulp Jul 31, 2024
037aa75
refactor: one account for FillBlocks
rootulp Aug 1, 2024
2a8d98a
refactor: const for appVersion
rootulp Aug 1, 2024
79d8458
revert: header support for v2
rootulp Aug 1, 2024
7e31762
refactor: bump swamp test timeout to 10 mins
rootulp Aug 1, 2024
6c12f7b
refactor: remove celestia-node.sh script
rootulp Aug 1, 2024
f52886c
refactor: provide len to make()
rootulp Aug 1, 2024
35f5164
revert: len in make
rootulp Aug 1, 2024
c284dc2
Revert "revert: header support for v2"
rootulp Aug 1, 2024
6a92027
docs: comment for convert
rootulp Aug 2, 2024
96e1bcd
refactor: use Namespace.Compare() and unit test
rootulp Aug 2, 2024
cc3ee1d
fix: ToAppBlobs
rootulp Aug 2, 2024
9a59f26
refactor: use convert
rootulp Aug 2, 2024
26da487
chore(deps): upgrade to nmt v0.22.1
rootulp Aug 2, 2024
77d0477
revert: convert refactor
rootulp Aug 2, 2024
167496a
refactor: remove overflow check inside convertBlobs
rootulp Aug 5, 2024
e9eee2a
Merge branch 'main' into rp/support-v2
renaynay Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/gateway/share.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (

"github.com/gorilla/mux"

"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/go-square/shares"

"github.com/celestiaorg/celestia-node/share"
)
Expand Down
16 changes: 8 additions & 8 deletions api/gateway/share_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ import (
"testing"

"github.com/stretchr/testify/require"
coretypes "github.com/tendermint/tendermint/types"

"github.com/celestiaorg/celestia-app/pkg/appconsts"
"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/celestia-app/v2/pkg/appconsts"
"github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/shares"

"github.com/celestiaorg/celestia-node/share/sharetest"
)
Expand All @@ -23,13 +23,13 @@ func Test_dataFromShares(t *testing.T) {
ns := sharetest.RandV0Namespace()
sss := shares.NewSparseShareSplitter()
for _, data := range testData {
b := coretypes.Blob{
b := blob.Blob{
Data: data,
NamespaceID: ns.ID(),
NamespaceVersion: ns.Version(),
ShareVersion: appconsts.ShareVersionZero,
NamespaceId: ns.ID(),
NamespaceVersion: uint32(ns.Version()),
ShareVersion: uint32(appconsts.ShareVersionZero),
}
err := sss.Write(b)
err := sss.Write(&b)
require.NoError(t, err)
}

Expand Down
40 changes: 23 additions & 17 deletions blob/blob.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,21 @@ import (
"errors"
"fmt"

tmproto "github.com/tendermint/tendermint/proto/tendermint/types"
"github.com/tendermint/tendermint/crypto/merkle"

"github.com/celestiaorg/celestia-app/pkg/appconsts"
"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/celestia-app/x/blob/types"
"github.com/celestiaorg/celestia-app/v2/pkg/appconsts"
v2 "github.com/celestiaorg/celestia-app/v2/pkg/appconsts/v2"
"github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/inclusion"
"github.com/celestiaorg/go-square/shares"
"github.com/celestiaorg/nmt"

"github.com/celestiaorg/celestia-node/share"
)

// appVersion is the current application version of celestia-app.
const appVersion = v2.Version

var errEmptyShares = errors.New("empty shares")

// The Proof is a set of nmt proofs that can be verified only through
Expand Down Expand Up @@ -56,7 +61,7 @@ func (p Proof) equal(input Proof) error {

// Blob represents any application-specific binary data that anyone can submit to Celestia.
type Blob struct {
types.Blob `json:"blob"`
*blob.Blob `json:"blob"`
renaynay marked this conversation as resolved.
Show resolved Hide resolved

Commitment Commitment `json:"commitment"`

Expand Down Expand Up @@ -84,18 +89,18 @@ func NewBlob(shareVersion uint8, namespace share.Namespace, data []byte) (*Blob,
return nil, err
}

blob := tmproto.Blob{
blob := blob.Blob{
NamespaceId: namespace.ID(),
Data: data,
ShareVersion: uint32(shareVersion),
NamespaceVersion: uint32(namespace.Version()),
}

com, err := types.CreateCommitment(&blob)
com, err := inclusion.CreateCommitment(&blob, merkle.HashFromByteSlices, appconsts.SubtreeRootThreshold(appVersion))
if err != nil {
return nil, err
}
return &Blob{Blob: blob, Commitment: com, namespace: namespace, index: -1}, nil
return &Blob{Blob: &blob, Commitment: com, namespace: namespace, index: -1}, nil
}

// Namespace returns blob's namespace.
Expand Down Expand Up @@ -157,18 +162,19 @@ func (b *Blob) MarshalJSON() ([]byte, error) {
}

func (b *Blob) UnmarshalJSON(data []byte) error {
var blob jsonBlob
err := json.Unmarshal(data, &blob)
var jsonBlob jsonBlob
err := json.Unmarshal(data, &jsonBlob)
if err != nil {
return err
}

b.Blob.NamespaceVersion = uint32(blob.Namespace.Version())
b.Blob.NamespaceId = blob.Namespace.ID()
b.Blob.Data = blob.Data
b.Blob.ShareVersion = blob.ShareVersion
b.Commitment = blob.Commitment
b.namespace = blob.Namespace
b.index = blob.Index
b.Blob = &blob.Blob{}
b.Blob.NamespaceVersion = uint32(jsonBlob.Namespace.Version())
b.Blob.NamespaceId = jsonBlob.Namespace.ID()
b.Blob.Data = jsonBlob.Data
b.Blob.ShareVersion = jsonBlob.ShareVersion
b.Commitment = jsonBlob.Commitment
b.namespace = jsonBlob.Namespace
b.index = jsonBlob.Index
return nil
}
26 changes: 19 additions & 7 deletions blob/blob_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,18 @@ package blob

import (
"bytes"
"fmt"
"math"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/tendermint/tendermint/types"

apptypes "github.com/celestiaorg/celestia-app/x/blob/types"
"github.com/celestiaorg/celestia-app/v2/pkg/appconsts"
apptypes "github.com/celestiaorg/celestia-app/v2/x/blob/types"
"github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/inclusion"
"github.com/celestiaorg/go-square/merkle"

"github.com/celestiaorg/celestia-node/blob/blobtest"
)
Expand Down Expand Up @@ -36,7 +41,11 @@ func TestBlob(t *testing.T) {
{
name: "compare commitments",
expectedRes: func(t *testing.T) {
comm, err := apptypes.CreateCommitment(&blob[0].Blob)
comm, err := inclusion.CreateCommitment(
blob[0].Blob,
merkle.HashFromByteSlices,
appconsts.SubtreeRootThreshold(appVersion),
)
require.NoError(t, err)
assert.Equal(t, blob[0].Commitment, Commitment(comm))
},
Expand Down Expand Up @@ -78,7 +87,7 @@ func TestBlob(t *testing.T) {

newBlob := &Blob{}
require.NoError(t, newBlob.UnmarshalJSON(data))
require.True(t, bytes.Equal(blob[0].Blob.Data, newBlob.Data))
require.True(t, bytes.Equal(blob[0].Blob.GetData(), newBlob.Data))
require.True(t, bytes.Equal(blob[0].Commitment, newBlob.Commitment))
},
},
Expand All @@ -89,10 +98,13 @@ func TestBlob(t *testing.T) {
}
}

func convertBlobs(appBlobs ...types.Blob) ([]*Blob, error) {
func convertBlobs(appBlobs ...*blob.Blob) ([]*Blob, error) {
blobs := make([]*Blob, 0, len(appBlobs))
for _, b := range appBlobs {
blob, err := NewBlob(b.ShareVersion, append([]byte{b.NamespaceVersion}, b.NamespaceID...), b.Data)
for _, appBlob := range appBlobs {
if shareVersion := appBlob.GetShareVersion(); shareVersion > math.MaxUint8 {
return nil, fmt.Errorf("share version %d is greater than max share version %d", shareVersion, math.MaxUint8)
rootulp marked this conversation as resolved.
Show resolved Hide resolved
}
blob, err := NewBlob(uint8(appBlob.GetShareVersion()), appBlob.Namespace().Bytes(), appBlob.GetData())
if err != nil {
return nil, err
}
Expand Down
18 changes: 9 additions & 9 deletions blob/blobtest/testing.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,30 @@ package blobtest

import (
tmrand "github.com/tendermint/tendermint/libs/rand"
"github.com/tendermint/tendermint/types"

"github.com/celestiaorg/celestia-app/pkg/appconsts"
"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/celestia-app/test/util/testfactory"
"github.com/celestiaorg/celestia-app/v2/pkg/appconsts"
"github.com/celestiaorg/celestia-app/v2/test/util/testfactory"
"github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/shares"

"github.com/celestiaorg/celestia-node/share"
)

// GenerateV0Blobs is a test utility producing v0 share formatted blobs with the
// requested size and random namespaces.
func GenerateV0Blobs(sizes []int, sameNamespace bool) ([]types.Blob, error) {
blobs := make([]types.Blob, 0, len(sizes))
func GenerateV0Blobs(sizes []int, sameNamespace bool) ([]*blob.Blob, error) {
blobs := make([]*blob.Blob, 0, len(sizes))

for _, size := range sizes {
size := rawBlobSize(appconsts.FirstSparseShareContentSize * size)
appBlob := testfactory.GenerateRandomBlob(size)
if !sameNamespace {
nid, err := share.NewBlobNamespaceV0(tmrand.Bytes(7))
namespace, err := share.NewBlobNamespaceV0(tmrand.Bytes(7))
if err != nil {
return nil, err
}
appBlob.NamespaceVersion = nid[0]
appBlob.NamespaceID = nid[1:]
appBlob.NamespaceVersion = uint32(namespace[0])
appBlob.NamespaceId = namespace[1:]
}

blobs = append(blobs, appBlob)
Expand Down
7 changes: 3 additions & 4 deletions blob/commitment_proof.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,9 @@ import (
"bytes"
"fmt"

coretypes "github.com/tendermint/tendermint/types"

"github.com/celestiaorg/celestia-app/pkg/appconsts"
"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/celestia-app/v2/pkg/proof"
"github.com/celestiaorg/nmt"
"github.com/celestiaorg/nmt/namespace"

Expand All @@ -33,8 +32,8 @@ type CommitmentProof struct {
NamespaceID namespace.ID `json:"namespace_id"`
// RowProof is the proof of the rows containing the blob's data to the
// data root.
RowProof coretypes.RowProof `json:"row_proof"`
NamespaceVersion uint8 `json:"namespace_version"`
RowProof proof.RowProof `json:"row_proof"`
rootulp marked this conversation as resolved.
Show resolved Hide resolved
NamespaceVersion uint8 `json:"namespace_version"`
}

func (com Commitment) String() string {
Expand Down
36 changes: 16 additions & 20 deletions blob/helper.go
Original file line number Diff line number Diff line change
@@ -1,33 +1,29 @@
package blob

import (
"bytes"
"sort"

"github.com/tendermint/tendermint/types"

"github.com/celestiaorg/celestia-app/pkg/shares"
apptypes "github.com/celestiaorg/celestia-app/x/blob/types"
squareblob "github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/shares"

"github.com/celestiaorg/celestia-node/share"
)

// BlobsToShares accepts blobs and convert them to the Shares.
func BlobsToShares(blobs ...*Blob) ([]share.Share, error) {
b := make([]types.Blob, len(blobs))
for i, blob := range blobs {
namespace := blob.Namespace()
b[i] = types.Blob{
NamespaceVersion: namespace[0],
NamespaceID: namespace[1:],
Data: blob.Data,
ShareVersion: uint8(blob.ShareVersion),
func BlobsToShares(nodeBlobs ...*Blob) ([]share.Share, error) {
b := make([]*squareblob.Blob, len(nodeBlobs))
for i, nodeBlob := range nodeBlobs {
namespace := nodeBlob.Namespace()
b[i] = &squareblob.Blob{
NamespaceVersion: uint32(namespace[0]),
NamespaceId: namespace[1:],
Data: nodeBlob.Data,
ShareVersion: nodeBlob.ShareVersion,
}
}

sort.Slice(b, func(i, j int) bool {
val := bytes.Compare(b[i].NamespaceID, b[j].NamespaceID)
return val < 0
return b[i].Namespace().Compare(b[j].Namespace()) < 0
})

rawShares, err := shares.SplitBlobs(b...)
Expand All @@ -37,11 +33,11 @@ func BlobsToShares(blobs ...*Blob) ([]share.Share, error) {
return shares.ToBytes(rawShares), nil
}

// ToAppBlobs converts node's blob type to the blob type from celestia-app.
func ToAppBlobs(blobs ...*Blob) []*apptypes.Blob {
appBlobs := make([]*apptypes.Blob, 0, len(blobs))
// ToAppBlobs converts node's blob type to the blob type from go-square.
func ToAppBlobs(blobs ...*Blob) []*squareblob.Blob {
appBlobs := make([]*squareblob.Blob, len(blobs))
for i := range blobs {
appBlobs[i] = &blobs[i].Blob
appBlobs[i] = blobs[i].Blob
rootulp marked this conversation as resolved.
Show resolved Hide resolved
}
return appBlobs
}
Expand Down
57 changes: 57 additions & 0 deletions blob/helper_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package blob

import (
"bytes"
"testing"

"github.com/celestiaorg/celestia-app/v2/pkg/appconsts"
squareblob "github.com/celestiaorg/go-square/blob"
"github.com/celestiaorg/go-square/namespace"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestBlobsToShares(t *testing.T) {
t.Run("should sort blobs by namespace in ascending order", func(t *testing.T) {
namespaceA := namespace.MustNewV0(bytes.Repeat([]byte{0xa}, 10)).Bytes()
namespaceB := namespace.MustNewV0(bytes.Repeat([]byte{0xb}, 10)).Bytes()

blobA, err := NewBlob(appconsts.ShareVersionZero, namespaceA, []byte("dataA"))
require.NoError(t, err)
blobB, err := NewBlob(appconsts.ShareVersionZero, namespaceB, []byte("dataB"))
require.NoError(t, err)

got, err := BlobsToShares(blobB, blobA)
require.NoError(t, err)
assert.Equal(t, got[0][:appconsts.NamespaceSize], namespaceA)
assert.Equal(t, got[1][:appconsts.NamespaceSize], namespaceB)
assert.IsIncreasing(t, got)
})
}

func TestToAppBlobs(t *testing.T) {
namespaceA := namespace.MustNewV0(bytes.Repeat([]byte{0xa}, 10))
namespaceB := namespace.MustNewV0(bytes.Repeat([]byte{0xb}, 10))

blobA, err := NewBlob(appconsts.ShareVersionZero, namespaceA.Bytes(), []byte("dataA"))
require.NoError(t, err)
blobB, err := NewBlob(appconsts.ShareVersionZero, namespaceB.Bytes(), []byte("dataB"))
require.NoError(t, err)

got := ToAppBlobs(blobA, blobB)
want := []*squareblob.Blob{
{
NamespaceId: blobA.NamespaceId,
NamespaceVersion: blobA.NamespaceVersion,
Data: blobA.Data,
ShareVersion: blobA.ShareVersion,
},
{
NamespaceId: blobB.NamespaceId,
NamespaceVersion: blobB.NamespaceVersion,
Data: blobB.Data,
ShareVersion: blobB.ShareVersion,
},
}
assert.Equal(t, want, got)
}
2 changes: 1 addition & 1 deletion blob/parser.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"errors"
"fmt"

"github.com/celestiaorg/celestia-app/pkg/shares"
"github.com/celestiaorg/go-square/shares"
)

// parser helps to collect shares and transform them into a blob.
Expand Down
Loading
Loading