Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API: Experimental full assets for account endpoint. #5948

Merged
merged 18 commits into from
May 3, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
6993d7f
Initial cut of support for a full assets for account endpoint. Suppor…
gmalouf Apr 30, 2024
e846d55
Patches for cleanly returning no results when next token is set above…
gmalouf Mar 6, 2024
d2d1b8e
Unit test of the AccountAssets endpoint. Updated pagination logic to …
gmalouf Apr 1, 2024
f6de85c
Update lookup limited resources sql query to conditionally/left join …
gmalouf Apr 4, 2024
55667f3
Set asset max on account assets pagination query to 1000.
gmalouf Apr 4, 2024
7bc9aa1
Add support for invoking the account assets information endpoint from…
gmalouf Apr 6, 2024
f694aba
Introduce new database target version, adding a ctype column to the r…
gmalouf Apr 9, 2024
6726399
Remove join from resources table ctype migration and add a check if d…
gmalouf Apr 10, 2024
6c3d282
OAS cleanup.
gmalouf Apr 10, 2024
7722f80
Update AccountsInitTest for latest migration.
gmalouf Apr 10, 2024
a810c49
Decrease migration runtime of ctype resources migration by 2.5x lever…
gmalouf Apr 22, 2024
5d87b80
Address test failures (k-v store version has to match sqlite, even wi…
gmalouf Apr 23, 2024
5ab49fd
Refactor resources ctype migration to use a default value, avoiding h…
gmalouf Apr 25, 2024
6bb6442
Test adjustments to properly simulate initializing a ledger from a ca…
gmalouf Apr 30, 2024
963027d
Code review tweaks.
gmalouf Apr 30, 2024
2db2669
Changes based on additional CR feedback.
gmalouf May 2, 2024
778cadf
Basic exercise of goal account assetdetails in goal-account-asset.sh
gmalouf May 2, 2024
737c178
Update test/scripts/e2e_subs/goal-account-asset.sh
gmalouf May 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions daemon/algod/api/server/v2/handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
ConsensusParams(r basics.Round) (config.ConsensusParams, error)
Latest() basics.Round
LookupAsset(rnd basics.Round, addr basics.Address, aidx basics.AssetIndex) (ledgercore.AssetResource, error)
LookupAssets(rnd basics.Round, addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error)
LookupAssets(addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error)
LookupApplication(rnd basics.Round, addr basics.Address, aidx basics.AppIndex) (ledgercore.AppResource, error)
BlockCert(rnd basics.Round) (blk bookkeeping.Block, cert agreement.Certificate, err error)
LatestTotals() (basics.Round, ledgercore.AccountTotals, error)
Expand Down Expand Up @@ -1113,93 +1113,92 @@

// AccountAssetsInformation looks up an account's asset holdings.
// (GET /v2/accounts/{address}/assets)
func (v2 *Handlers) AccountAssetsInformation(ctx echo.Context, address string, params model.AccountAssetsInformationParams) error {
if !v2.Node.Config().EnableExperimentalAPI {
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
return ctx.String(http.StatusNotFound, "/v2/accounts/{address}/assets was not enabled in the configuration file by setting the EnableExperimentalAPI to true")

Check warning on line 1118 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1116-L1118

Added lines #L1116 - L1118 were not covered by tests
}

addr, err := basics.UnmarshalChecksumAddress(address)
if err != nil {
return badRequest(ctx, err, errFailedToParseAddress, v2.Log)

Check warning on line 1123 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1121-L1123

Added lines #L1121 - L1123 were not covered by tests
}

var assetGreaterThan uint64 = 0
if params.Next != nil {
agt, err0 := strconv.ParseUint(*params.Next, 10, 64)
if err0 != nil {
return badRequest(ctx, err0, fmt.Sprintf("%s: %v", errUnableToParseNext, err0), v2.Log)

Check warning on line 1130 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1126-L1130

Added lines #L1126 - L1130 were not covered by tests
}
assetGreaterThan = agt

Check warning on line 1132 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1132

Added line #L1132 was not covered by tests
}

if params.Limit != nil {
if *params.Limit <= 0 {
return badRequest(ctx, errors.New(errInvalidLimit), errInvalidLimit, v2.Log)

Check warning on line 1137 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1135-L1137

Added lines #L1135 - L1137 were not covered by tests
}

if *params.Limit > MaxAssetResults {
limitErrMsg := fmt.Sprintf("limit %d exceeds max assets single batch limit %d", *params.Limit, MaxAssetResults)
return badRequest(ctx, errors.New(limitErrMsg), limitErrMsg, v2.Log)

Check warning on line 1142 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1140-L1142

Added lines #L1140 - L1142 were not covered by tests
}
} else {

Check warning on line 1144 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1144

Added line #L1144 was not covered by tests
// default limit
l := DefaultAssetResults
params.Limit = &l

Check warning on line 1147 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1146-L1147

Added lines #L1146 - L1147 were not covered by tests
}

ledger := v2.Node.LedgerForAPI()

Check warning on line 1150 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1150

Added line #L1150 was not covered by tests

// Logic
// 1. Get the account's asset holdings subject to limits
// 2. Handle empty response
// 3. Prepare JSON response
lastRound := ledger.Latest()

// We intentionally request one more than the limit to determine if there are more assets.
records, lookupRound, err := ledger.LookupAssets(lastRound, addr, basics.AssetIndex(assetGreaterThan), *params.Limit+1)
records, lookupRound, err := ledger.LookupAssets(addr, basics.AssetIndex(assetGreaterThan), *params.Limit+1)

Check warning on line 1158 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1158

Added line #L1158 was not covered by tests

if err != nil {
return internalError(ctx, err, errFailedLookingUpLedger, v2.Log)

Check warning on line 1161 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1160-L1161

Added lines #L1160 - L1161 were not covered by tests
}

// prepare JSON response
response := model.AccountAssetsInformationResponse{Round: uint64(lookupRound)}

Check warning on line 1165 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1165

Added line #L1165 was not covered by tests

// If the total count is greater than the limit, we set the next token to the last asset ID being returned
if uint64(len(records)) > *params.Limit {

Check warning on line 1168 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1168

Added line #L1168 was not covered by tests
// we do not include the last record in the response
records = records[:*params.Limit]
nextTk := strconv.FormatUint(uint64(records[len(records)-1].AssetID), 10)
response.NextToken = &nextTk

Check warning on line 1172 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1170-L1172

Added lines #L1170 - L1172 were not covered by tests
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
}

assetHoldings := make([]model.AccountAssetHolding, 0, len(records))

Check warning on line 1175 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1175

Added line #L1175 was not covered by tests

for _, record := range records {
if record.AssetHolding == nil {
v2.Log.Warnf("AccountAssetsInformation: asset %d has no holding - should not be possible", record.AssetID)
continue

Check warning on line 1180 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1177-L1180

Added lines #L1177 - L1180 were not covered by tests
}

aah := model.AccountAssetHolding{
AssetHolding: model.AssetHolding{
Amount: record.AssetHolding.Amount,
AssetID: uint64(record.AssetID),
IsFrozen: record.AssetHolding.Frozen,
},

Check warning on line 1188 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1183-L1188

Added lines #L1183 - L1188 were not covered by tests
}

if !record.Creator.IsZero() {
asset := AssetParamsToAsset(record.Creator.String(), record.AssetID, record.AssetParams)
aah.AssetParams = &asset.Params

Check warning on line 1193 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1191-L1193

Added lines #L1191 - L1193 were not covered by tests
}

assetHoldings = append(assetHoldings, aah)

Check warning on line 1196 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1196

Added line #L1196 was not covered by tests
}

response.AssetHoldings = &assetHoldings

Check warning on line 1199 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1199

Added line #L1199 was not covered by tests

return ctx.JSON(http.StatusOK, response)

Check warning on line 1201 in daemon/algod/api/server/v2/handlers.go

View check run for this annotation

Codecov / codecov/patch

daemon/algod/api/server/v2/handlers.go#L1201

Added line #L1201 was not covered by tests
}

// PreEncodedSimulateTxnResult mirrors model.SimulateTransactionResult
Expand Down
49 changes: 25 additions & 24 deletions daemon/algod/api/server/v2/test/handlers_resources_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,10 +109,10 @@ func (l *mockLedger) LookupAsset(rnd basics.Round, addr basics.Address, aidx bas
return ar, nil
}

func (l *mockLedger) LookupAssets(rnd basics.Round, addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
func (l *mockLedger) LookupAssets(addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
ad, ok := l.accounts[addr]
if !ok {
return nil, rnd, nil
return nil, basics.Round(0), nil
}

var res []ledgercore.AssetResourceWithIDs
Expand All @@ -132,7 +132,7 @@ func (l *mockLedger) LookupAssets(rnd basics.Round, addr basics.Address, assetID
res = append(res, apr)
}
}
return res, rnd, nil
return res, basics.Round(0), nil
}

func (l *mockLedger) LookupApplication(rnd basics.Round, addr basics.Address, aidx basics.AppIndex) (ar ledgercore.AppResource, err error) {
Expand Down Expand Up @@ -428,7 +428,6 @@ func accountInformationResourceLimitsTest(t *testing.T, accountMaker func(int) b

func accountAssetInformationResourceLimitsTest(t *testing.T, handlers v2.Handlers, addr basics.Address,
acctData basics.AccountData, params model.AccountAssetsInformationParams, inputNextToken int, maxResults int, expectToken bool) {
fakeLatestRound := basics.Round(10)

ctx, rec := newReq(t)
err := handlers.AccountAssetsInformation(ctx, addr.String(), params)
Expand All @@ -437,7 +436,6 @@ func accountAssetInformationResourceLimitsTest(t *testing.T, handlers v2.Handler
var ret model.AccountAssetsInformationResponse
err = json.Unmarshal(rec.Body.Bytes(), &ret)
require.NoError(t, err)
assert.Equal(t, fakeLatestRound, basics.Round(ret.Round))

if expectToken {
nextRaw, err0 := strconv.ParseUint(*ret.NextToken, 10, 64)
Expand Down Expand Up @@ -477,43 +475,46 @@ func TestAccountAssetsInformation(t *testing.T) {
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData, model.AccountAssetsInformationParams{},
0, int(v2.DefaultAssetResults), false)

// 2. Query with limit<total resources, no next - should get the first (lowest asset id to highest) limit results back
rawLimit := 100
limit := uint64(rawLimit)
// 2. Query with limit<total resources, no next - should get the first (lowest asset id to highest) limit results back
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData,
model.AccountAssetsInformationParams{Limit: &limit}, 0, rawLimit, true)

// 3. Query with limit, next
rawNext := 100
nextTk := strconv.FormatUint(uint64(rawNext), 10)
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData,
model.AccountAssetsInformationParams{Limit: &limit, Next: &nextTk}, rawNext, rawLimit, true)

//4. Query with limit, next to retrieve final batch
rawNext = 1019
nextTk = strconv.FormatUint(uint64(rawNext), 10)
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData,
model.AccountAssetsInformationParams{Limit: &limit, Next: &nextTk}, rawNext, totalAssetHoldings-rawNext, false)
// 3. Loop through all assets in the account in batches of 100, ensure we get all assets back.
// Exercises limit and next combined.
for rawNext := 0; rawNext < totalAssetHoldings; rawNext += rawLimit {
nextTk := strconv.FormatUint(uint64(rawNext), 10)
// We expect a next token for all but the last batch
expectToken := true
expectedResultsCount := rawLimit
if rawNext+rawLimit >= totalAssetHoldings {
expectToken = false
expectedResultsCount = totalAssetHoldings - rawNext
}
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData,
model.AccountAssetsInformationParams{Limit: &limit, Next: &nextTk}, rawNext, expectedResultsCount, expectToken)
}

// 5. Query with limit, next to provide batch, but no data in that range
rawNext = 1025
nextTk = strconv.FormatUint(uint64(rawNext), 10)
// 4. Query with limit, next to provide batch, but no data in that range
rawNext := 1025
nextTk := strconv.FormatUint(uint64(rawNext), 10)
accountAssetInformationResourceLimitsTest(t, handlers, addr, acctData,
model.AccountAssetsInformationParams{Limit: &limit, Next: &nextTk}, rawNext, totalAssetHoldings-rawNext, false)

algorandskiy marked this conversation as resolved.
Show resolved Hide resolved
// 6. Malformed address
// 5. Malformed address
ctx, rec := newReq(t)
err := handlers.AccountAssetsInformation(ctx, "", model.AccountAssetsInformationParams{})
require.NoError(t, err)
require.Equal(t, 400, rec.Code)
require.Equal(t, "{\"message\":\"failed to parse the address\"}\n", rec.Body.String())

// 7. Unknown address (200 returned, just no asset data)
// 6. Unknown address (200 returned, just no asset data)
unknownAddress := basics.Address{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}
accountAssetInformationResourceLimitsTest(t, handlers, unknownAddress, basics.AccountData{}, model.AccountAssetsInformationParams{},
0, 0, false)

// 8a. Invalid limits - larger than configured max
// 7a. Invalid limits - larger than configured max
ctx, rec = newReq(t)
err = handlers.AccountAssetsInformation(ctx, addr.String(), model.AccountAssetsInformationParams{
Limit: func() *uint64 {
Expand All @@ -525,7 +526,7 @@ func TestAccountAssetsInformation(t *testing.T) {
require.Equal(t, 400, rec.Code)
require.Equal(t, "{\"message\":\"limit 1001 exceeds max assets single batch limit 1000\"}\n", rec.Body.String())

// 8b. Invalid limits - zero
// 7b. Invalid limits - zero
ctx, rec = newReq(t)
err = handlers.AccountAssetsInformation(ctx, addr.String(), model.AccountAssetsInformationParams{
Limit: func() *uint64 {
Expand Down
113 changes: 39 additions & 74 deletions ledger/acctupdates.go
Original file line number Diff line number Diff line change
Expand Up @@ -341,8 +341,8 @@
return au.lookupResource(rnd, addr, aidx, ctype, true /* take lock */)
}

func (au *accountUpdates) LookupAssetResources(rnd basics.Round, addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
return au.lookupAssetResources(rnd, addr, assetIDGT, limit, true /* take lock */)
func (au *accountUpdates) LookupAssetResources(addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
return au.lookupAssetResources(addr, assetIDGT, limit)

Check warning on line 345 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L344-L345

Added lines #L344 - L345 were not covered by tests
}

func (au *accountUpdates) LookupKv(rnd basics.Round, key string) ([]byte, error) {
Expand Down Expand Up @@ -1212,88 +1212,53 @@
}
}

func (au *accountUpdates) lookupAssetResources(rnd basics.Round, addr basics.Address, assetIDGT basics.AssetIndex, limit uint64, synchronized bool) (data []ledgercore.AssetResourceWithIDs, validThrough basics.Round, err error) {
needUnlock := false
if synchronized {
au.accountsMu.RLock()
needUnlock = true
// lookupAllResources returns all the resources for a given address, solely based on what is persisted to disk. It does not
// take into account any in-memory deltas; the round number returned is the latest round number that is known to the database.
func (au *accountUpdates) lookupAssetResources(addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) (data []ledgercore.AssetResourceWithIDs, validThrough basics.Round, err error) {

Check warning on line 1217 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1217

Added line #L1217 was not covered by tests
// Look for resources on disk
persistedResources, resourceDbRound, err0 := au.accountsq.LookupLimitedResources(addr, basics.CreatableIndex(assetIDGT), limit, basics.AssetCreatable)
if err0 != nil {
return nil, basics.Round(0), err0

Check warning on line 1221 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1219-L1221

Added lines #L1219 - L1221 were not covered by tests
}
defer func() {
if needUnlock {
au.accountsMu.RUnlock()
}
}()

for {
currentDbRound := au.cachedDBRound

// This checks that the specified round is >= au.cachedDBRound - if not, it will return an error.
_, err = au.roundOffset(rnd)
if err != nil {
return
}

if synchronized {
au.accountsMu.RUnlock()
needUnlock = false
}

// Look for resources on disk
persistedResources, resourceDbRound, err0 := au.accountsq.LookupLimitedResources(addr, basics.CreatableIndex(assetIDGT), limit, basics.AssetCreatable)
if err0 != nil {
return nil, basics.Round(0), err0
}

if resourceDbRound == currentDbRound || len(persistedResources) == 0 { // db round will return 0 in this case
data = make([]ledgercore.AssetResourceWithIDs, 0, len(persistedResources))
for _, pd := range persistedResources {
ah := pd.Data.GetAssetHolding()

var arwi ledgercore.AssetResourceWithIDs
if !pd.Creator.IsZero() {
ap := pd.Data.GetAssetParams()

arwi = ledgercore.AssetResourceWithIDs{
AssetID: basics.AssetIndex(pd.Aidx),
Creator: pd.Creator,
data = make([]ledgercore.AssetResourceWithIDs, 0, len(persistedResources))
for _, pd := range persistedResources {
ah := pd.Data.GetAssetHolding()

Check warning on line 1226 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1224-L1226

Added lines #L1224 - L1226 were not covered by tests

AssetResource: ledgercore.AssetResource{
AssetHolding: &ah,
AssetParams: &ap,
},
}
} else {
arwi = ledgercore.AssetResourceWithIDs{
AssetID: basics.AssetIndex(pd.Aidx),
var arwi ledgercore.AssetResourceWithIDs
if !pd.Creator.IsZero() {
ap := pd.Data.GetAssetParams()

Check warning on line 1230 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1228-L1230

Added lines #L1228 - L1230 were not covered by tests

AssetResource: ledgercore.AssetResource{
AssetHolding: &ah,
},
}
}
arwi = ledgercore.AssetResourceWithIDs{
AssetID: basics.AssetIndex(pd.Aidx),
Creator: pd.Creator,

Check warning on line 1234 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1232-L1234

Added lines #L1232 - L1234 were not covered by tests

data = append(data, arwi)
AssetResource: ledgercore.AssetResource{
AssetHolding: &ah,
AssetParams: &ap,
},

Check warning on line 1239 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1236-L1239

Added lines #L1236 - L1239 were not covered by tests
}
// We've found all the resources we could find for this address.
return data, currentDbRound, nil
}
} else {
arwi = ledgercore.AssetResourceWithIDs{
AssetID: basics.AssetIndex(pd.Aidx),

Check warning on line 1243 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1241-L1243

Added lines #L1241 - L1243 were not covered by tests

if synchronized {
if resourceDbRound < currentDbRound {
au.log.Errorf("accountUpdates.lookupAssetResources: resource database round %d is behind in-memory round %d", resourceDbRound, currentDbRound)
return nil, basics.Round(0), &StaleDatabaseRoundError{databaseRound: resourceDbRound, memoryRound: currentDbRound}
}
au.accountsMu.RLock()
needUnlock = true
for currentDbRound >= au.cachedDBRound {
au.accountsReadCond.Wait()
AssetResource: ledgercore.AssetResource{
AssetHolding: &ah,
},

Check warning on line 1247 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1245-L1247

Added lines #L1245 - L1247 were not covered by tests
}
} else {
// in non-sync mode, we don't wait since we already assume that we're synchronized.
au.log.Errorf("accountUpdates.lookupAssetResources: database round %d mismatching in-memory round %d", resourceDbRound, currentDbRound)
return nil, basics.Round(0), &MismatchingDatabaseRoundError{databaseRound: resourceDbRound, memoryRound: currentDbRound}
}

data = append(data, arwi)

Check warning on line 1251 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1251

Added line #L1251 was not covered by tests
}
// We've found all the resources we could find for this address.
currentDbRound := resourceDbRound

Check warning on line 1254 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1254

Added line #L1254 was not covered by tests
// The resourceDbRound will not be set if there are no persisted resources
if len(data) == 0 {
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
au.accountsMu.RLock()
currentDbRound = au.cachedDBRound
au.accountsMu.RUnlock()

Check warning on line 1259 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1256-L1259

Added lines #L1256 - L1259 were not covered by tests
}
return data, currentDbRound, nil

Check warning on line 1261 in ledger/acctupdates.go

View check run for this annotation

Codecov / codecov/patch

ledger/acctupdates.go#L1261

Added line #L1261 was not covered by tests
}

func (au *accountUpdates) lookupStateDelta(rnd basics.Round) (ledgercore.StateDelta, error) {
Expand Down
4 changes: 2 additions & 2 deletions ledger/ledger.go
Original file line number Diff line number Diff line change
Expand Up @@ -584,9 +584,9 @@
}

// LookupAssets loads asset resources that match the request parameters from the ledger.
func (l *Ledger) LookupAssets(rnd basics.Round, addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
resources, lookupRound, err := l.accts.LookupAssetResources(rnd, addr, assetIDGT, limit)
func (l *Ledger) LookupAssets(addr basics.Address, assetIDGT basics.AssetIndex, limit uint64) ([]ledgercore.AssetResourceWithIDs, basics.Round, error) {
resources, lookupRound, err := l.accts.LookupAssetResources(addr, assetIDGT, limit)
return resources, lookupRound, err

Check warning on line 589 in ledger/ledger.go

View check run for this annotation

Codecov / codecov/patch

ledger/ledger.go#L587-L589

Added lines #L587 - L589 were not covered by tests
}

// lookupResource loads a resource that matches the request parameters from the accounts update
Expand Down
18 changes: 16 additions & 2 deletions ledger/store/trackerdb/sqlitedriver/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -948,77 +948,91 @@
var creatableTypeOnResourcesRun bool
err := e.QueryRow("SELECT 1 FROM pragma_table_info('resources') WHERE name='ctype'").Scan(&creatableTypeOnResourcesRun)
if err == nil {
// Already exists.
// Check if any ctypes are invalid
var count uint64
err0 := e.QueryRow("SELECT COUNT(*) FROM resources WHERE ctype NOT IN (0, 1)").Scan(&count)
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
if err0 != nil {
return err0

Check warning on line 955 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L952-L955

Added lines #L952 - L955 were not covered by tests
}
if count > 0 {

Check warning on line 957 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L957

Added line #L957 was not covered by tests
// Invalid ctypes found, return an error
return fmt.Errorf("invalid ctypes found in resources table; database is corrupted and needs to be rebuilt")

Check warning on line 959 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L959

Added line #L959 was not covered by tests
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
}
// Column exists, no ctypes are invalid, no migration needed so return clean
return nil

Check warning on line 962 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L962

Added line #L962 was not covered by tests
} else if !errors.Is(err, sql.ErrNoRows) {
return err

Check warning on line 964 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L964

Added line #L964 was not covered by tests
} // A sql.ErrNoRows error means the column does not exist, so we need to create it
} // A sql.ErrNoRows error means the column does not exist, so we need to create it/run the migration

// If we reached here, a sql.ErrNoRows error was returned, so we need to create the column

// Add ctype column
createStmt := `ALTER TABLE resources ADD COLUMN ctype INTEGER NOT NULL DEFAULT -1`
gmalouf marked this conversation as resolved.
Show resolved Hide resolved

_, err = e.ExecContext(ctx, createStmt)
if err != nil {
return err

Check warning on line 974 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L974

Added line #L974 was not covered by tests
}

if populateColumn {
// Populate the new ctype column with the corresponding creatable type from assetcreators where available
updateStmt := `UPDATE resources SET ctype = (
SELECT COALESCE((SELECT ac.ctype FROM assetcreators ac WHERE ac.asset = resources.aidx),-1)
) WHERE ctype = -1`

Check warning on line 981 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L979-L981

Added lines #L979 - L981 were not covered by tests

_, err0 := e.ExecContext(ctx, updateStmt)
if err0 != nil {
return err0

Check warning on line 985 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L983-L985

Added lines #L983 - L985 were not covered by tests
}

updatePrepStmt, err0 := e.PrepareContext(ctx, "UPDATE resources SET ctype = ? WHERE addrid = ? AND aidx = ?")
if err0 != nil {
return err0

Check warning on line 990 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L988-L990

Added lines #L988 - L990 were not covered by tests
}
defer updatePrepStmt.Close()

Check warning on line 992 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L992

Added line #L992 was not covered by tests

// Pull resource entries into memory where ctype is not set
rows, err0 := e.QueryContext(ctx, "SELECT addrid, aidx, data FROM resources r WHERE ctype = -1")
if err0 != nil {
return err0

Check warning on line 997 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L995-L997

Added lines #L995 - L997 were not covered by tests
}
defer rows.Close()

Check warning on line 999 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L999

Added line #L999 was not covered by tests

// Update the ctype column for subset of resources where ctype was not resolved from assetcreators
for rows.Next() {
var addrid int64
var aidx int64
var encodedData []byte
err0 = rows.Scan(&addrid, &aidx, &encodedData)
if err0 != nil {
return err0

Check warning on line 1008 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1002-L1008

Added lines #L1002 - L1008 were not covered by tests
}

var rd trackerdb.ResourcesData
err0 = protocol.Decode(encodedData, &rd)
if err0 != nil {
return err0

Check warning on line 1014 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1011-L1014

Added lines #L1011 - L1014 were not covered by tests
}

var ct basics.CreatableType
if rd.IsAsset() && rd.IsApp() {

Check warning on line 1018 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1017-L1018

Added lines #L1017 - L1018 were not covered by tests
// This should never happen!
return fmt.Errorf("unable to discern creatable type for addrid %d, resource %d", addrid, aidx)
} else if rd.IsAsset() {
ct = basics.AssetCreatable
} else if rd.IsApp() {
ct = basics.AppCreatable
} else { // This should never happen!
return fmt.Errorf("unable to discern creatable type for addrid %d, resource %d", addrid, aidx)

Check warning on line 1026 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1020-L1026

Added lines #L1020 - L1026 were not covered by tests
}

_, err0 = updatePrepStmt.ExecContext(ctx, ct, addrid, aidx)
if err0 != nil {
return err0

Check warning on line 1031 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1029-L1031

Added lines #L1029 - L1031 were not covered by tests
}
}

return rows.Err()

Check warning on line 1035 in ledger/store/trackerdb/sqlitedriver/schema.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/schema.go#L1035

Added line #L1035 was not covered by tests
}

return nil
jasonpaulos marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
29 changes: 25 additions & 4 deletions ledger/store/trackerdb/sqlitedriver/sql.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,27 @@
return nil, err
}

qs.lookupLimitedResourcesStmt, err = q.Prepare("SELECT ab.rowid, ar.rnd, r.aidx, ac.creator, r.data, cr.data FROM acctrounds ar JOIN accountbase ab ON ab.address = ? JOIN resources r ON r.addrid = ab.addrid LEFT JOIN assetcreators ac ON r.aidx = ac.asset LEFT JOIN " +
"accountbase cab ON ac.creator = cab.address LEFT JOIN resources cr ON cr.addrid = cab.addrid AND cr.aidx = r.aidx WHERE ar.id = 'acctbase' AND r.ctype = ? AND r.aidx > ? ORDER BY r.aidx ASC LIMIT ?")
qs.lookupLimitedResourcesStmt, err = q.Prepare(
`SELECT ab.rowid,
ar.rnd,
r.aidx,
ac.creator,
r.data,
cr.data
FROM acctrounds ar
JOIN accountbase ab ON ab.address = ?
JOIN resources r ON r.addrid = ab.addrid
LEFT JOIN assetcreators ac ON r.aidx = ac.asset
LEFT JOIN accountbase cab ON ac.creator = cab.address
LEFT JOIN resources cr ON cr.addrid = cab.addrid
AND cr.aidx = r.aidx
WHERE ar.id = 'acctbase'
AND r.ctype = ?
AND r.aidx > ?
ORDER BY r.aidx ASC
LIMIT ?`)
if err != nil {
return nil, err

Check warning on line 110 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L110

Added line #L110 was not covered by tests
}

qs.lookupKvPairStmt, err = q.Prepare("SELECT acctrounds.rnd, kvstore.key, kvstore.value FROM acctrounds LEFT JOIN kvstore ON key = ? WHERE id='acctbase';")
Expand Down Expand Up @@ -174,7 +191,7 @@
return
}

w.insertResourceStmt, err = e.Prepare("INSERT INTO resources(addrid, aidx, data, ctype) VALUES(?, ?, ?, ?)")

Check warning on line 194 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L194

Added line #L194 was not covered by tests
if err != nil {
return
}
Expand Down Expand Up @@ -385,9 +402,9 @@
func (qs *accountsDbQueries) LookupAllResources(addr basics.Address) (data []trackerdb.PersistedResourcesData, rnd basics.Round, err error) {
err = db.Retry(func() error {
// Query for all resources
rows, err0 := qs.lookupAllResourcesStmt.Query(addr[:])
if err0 != nil {
return err0

Check warning on line 407 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L405-L407

Added lines #L405 - L407 were not covered by tests
}
defer rows.Close()

Expand All @@ -396,7 +413,7 @@
data = nil
var buf []byte
for rows.Next() {
err = rows.Scan(&addrid, &dbRound, &aidx, &buf)

Check warning on line 416 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L416

Added line #L416 was not covered by tests
if err != nil {
return err
}
Expand Down Expand Up @@ -427,83 +444,87 @@
return
}

func (qs *accountsDbQueries) LookupLimitedResources(addr basics.Address, minIdx basics.CreatableIndex, maxCreatables uint64, ctype basics.CreatableType) (data []trackerdb.PersistedResourcesDataWithCreator, rnd basics.Round, err error) {
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
err = db.Retry(func() error {
rows, err0 := qs.lookupLimitedResourcesStmt.Query(addr[:], ctype, minIdx, maxCreatables)
if err0 != nil {
return err0

Check warning on line 451 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L447-L451

Added lines #L447 - L451 were not covered by tests
}
defer rows.Close()

Check warning on line 453 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L453

Added line #L453 was not covered by tests

var addrid, aidx sql.NullInt64
var dbRound basics.Round
data = nil
var actAssetBuf []byte
var crtAssetBuf []byte
var creatorAddrBuf []byte
for rows.Next() {
err = rows.Scan(&addrid, &dbRound, &aidx, &creatorAddrBuf, &actAssetBuf, &crtAssetBuf)
if err != nil {
return err

Check warning on line 464 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L455-L464

Added lines #L455 - L464 were not covered by tests
}
if !addrid.Valid || !aidx.Valid {

Check warning on line 466 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L466

Added line #L466 was not covered by tests
// we received an entry without any index. This would happen only on the first entry when there are no resources for this address.
// ensure this is the first entry, set the round and return
if len(data) != 0 {
return fmt.Errorf("LookupLimitedResources: unexpected invalid result on non-first resource record: (%v, %v)", addrid.Valid, aidx.Valid)

Check warning on line 470 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L469-L470

Added lines #L469 - L470 were not covered by tests
}
rnd = dbRound
break

Check warning on line 473 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L472-L473

Added lines #L472 - L473 were not covered by tests
}
var actResData trackerdb.ResourcesData
var crtResData trackerdb.ResourcesData
err = protocol.Decode(actAssetBuf, &actResData)
if err != nil {
return err

Check warning on line 479 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L475-L479

Added lines #L475 - L479 were not covered by tests
}

var prdwc trackerdb.PersistedResourcesDataWithCreator
if len(crtAssetBuf) > 0 {
err = protocol.Decode(crtAssetBuf, &crtResData)
if err != nil {
return err

Check warning on line 486 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L482-L486

Added lines #L482 - L486 were not covered by tests
}

// Update asset holding data in the creator resource data
// Since there is a creator, we want to return all of the asset params along with the asset holdings.
// The most simple way to do this is to set the necessary asset holding data on the creator resource data
// retrieved from the database. Note that this is unique way of setting resource flags, making this structure
// not suitable for use in other contexts (where the params would only be present colocated with the asset holding
// of the creator).
crtResData.Amount = actResData.Amount
crtResData.Freeze = actResData.Freeze
crtResData.Frozen = actResData.Frozen
crtResData.ResourceFlags = actResData.ResourceFlags

Check warning on line 496 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L494-L496

Added lines #L494 - L496 were not covered by tests
algorandskiy marked this conversation as resolved.
Show resolved Hide resolved

creatorAddr := basics.Address{}
copy(creatorAddr[:], creatorAddrBuf)

Check warning on line 499 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L498-L499

Added lines #L498 - L499 were not covered by tests

prdwc = trackerdb.PersistedResourcesDataWithCreator{
PersistedResourcesData: trackerdb.PersistedResourcesData{
AcctRef: sqlRowRef{addrid.Int64},
Aidx: basics.CreatableIndex(aidx.Int64),
Data: crtResData,
Round: dbRound,
},
Creator: creatorAddr,

Check warning on line 508 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L501-L508

Added lines #L501 - L508 were not covered by tests
}
} else { // no creator found, asset was likely deleted, will not have asset params
prdwc = trackerdb.PersistedResourcesDataWithCreator{
PersistedResourcesData: trackerdb.PersistedResourcesData{
AcctRef: sqlRowRef{addrid.Int64},
Aidx: basics.CreatableIndex(aidx.Int64),
Data: actResData,
Round: dbRound,
},

Check warning on line 517 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L510-L517

Added lines #L510 - L517 were not covered by tests
}
}

data = append(data, prdwc)

Check warning on line 521 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L521

Added line #L521 was not covered by tests

rnd = dbRound

Check warning on line 523 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L523

Added line #L523 was not covered by tests
}
return nil

Check warning on line 525 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L525

Added line #L525 was not covered by tests
})
return

Check warning on line 527 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L527

Added line #L527 was not covered by tests
}

// LookupAccount looks up for a the account data given it's address. It returns the persistedAccountData, which includes the current database round and the matching
Expand Down Expand Up @@ -719,17 +740,17 @@
return nil, fmt.Errorf("no account could be found for rowid = nil: %w", err)
}
addrid := accountRef.(sqlRowRef).rowid
var ctype basics.CreatableType
if data.IsAsset() && data.IsApp() {
return nil, fmt.Errorf("unable to resolve single creatable type for account ref %d, creatable idx %d", addrid, aidx)
} else if data.IsAsset() {
ctype = basics.AssetCreatable
} else if data.IsApp() {
ctype = basics.AppCreatable
} else {
return nil, fmt.Errorf("unable to resolve creatable type for account ref %d, creatable idx %d", addrid, aidx)

Check warning on line 751 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L743-L751

Added lines #L743 - L751 were not covered by tests
}
result, err := w.insertResourceStmt.Exec(addrid, aidx, protocol.Encode(&data), ctype)

Check warning on line 753 in ledger/store/trackerdb/sqlitedriver/sql.go

View check run for this annotation

Codecov / codecov/patch

ledger/store/trackerdb/sqlitedriver/sql.go#L753

Added line #L753 was not covered by tests
if err != nil {
return
}
Expand Down
Loading