You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
yyforyongyu opened this issue
Aug 24, 2023
· 1 comment
Assignees
Labels
databaseRelated to the database/storage of LNDepicIssues created to track large feature developmentP1MUST be fixed or reviewedpaymentsRelated to invoices/paymentssql
This is the master issue that tracks the progress of adding native SQL support for payment db, which is split into three steps.
Support native SQL config
The first step is to allow users to configure SQL backends. There are two approaches,
use existing config flags.
add new config flags.
Since we already support multiple database backends, current nodes running with SQL would already have tables filled with key-value data. Thus going with option 1 creates more complexity as now we need to deal with mixed data types in the same database. Option 2 allows managing the native SQL in a clean state. With a few flag tricks, we can easily separate the way the database is configured. The idea is expressed in
The second step is to build SQL queries for payment data using sqlc. We already have the interface ControlTower that covers db operations. Ideally, we just need to create a new package payments to hold the data structs and db operations, to avoid we end up having a big package like channeldb. The tracking PR is,
Going SQL means we are giving up the flexibility provided by TLV, any updates to the scheme now need a migration. Future payment format should define its own struct and save the data in a new table.
Migrate kvdb data into SQL
In #6469 we introduced optional migrations. For existing nodes, we'd need to migrate their current key-value data into SQL rows. This migration needs to be paid attention to OOM as payment data can be large.
Finalize the optional migration scheme. The current optional migration scheme is half-finish as it only works for a single migration. To enable multiple migrations, we'd need to decide the relationship among the potential optional migrations - whether it's ok to only perform one or a few of them? Can any of the migrations be skipped? To reduce complexity, we could make it all or nothing - either we optionally run them all, or we don't run at all.
Add optional migration for payments. We'll need to start two database backends to move the data from kvdb to SQL. We need futher investigate this as our current migration scheme only supports one database.
The text was updated successfully, but these errors were encountered:
Going SQL means we are giving up the flexibility provided by TLV, any updates to the scheme now need a migration. Future payment format should define its own struct and save the data in a new table.
If relevant, we do have the ability to sort of mix things a bit, by having a column that actually stores TLV data. The upside of just using the normal migrations there though as we have to maintain less specific migration code for the various data structures.
databaseRelated to the database/storage of LNDepicIssues created to track large feature developmentP1MUST be fixed or reviewedpaymentsRelated to invoices/paymentssql
Depends on #6288.
This is the master issue that tracks the progress of adding native SQL support for payment db, which is split into three steps.
Support native SQL config
The first step is to allow users to configure SQL backends. There are two approaches,
Since we already support multiple database backends, current nodes running with SQL would already have tables filled with key-value data. Thus going with option 1 creates more complexity as now we need to deal with mixed data types in the same database. Option 2 allows managing the native SQL in a clean state. With a few flag tricks, we can easily separate the way the database is configured. The idea is expressed in
Use native SQL for payments
The second step is to build SQL queries for payment data using
sqlc
. We already have the interfaceControlTower
that covers db operations. Ideally, we just need to create a new packagepayments
to hold the data structs and db operations, to avoid we end up having a big package likechanneldb
. The tracking PR is,Going SQL means we are giving up the flexibility provided by TLV, any updates to the scheme now need a migration. Future payment format should define its own struct and save the data in a new table.
Migrate kvdb data into SQL
In #6469 we introduced optional migrations. For existing nodes, we'd need to migrate their current key-value data into SQL rows. This migration needs to be paid attention to OOM as payment data can be large.
The text was updated successfully, but these errors were encountered: