[PAY-2178] Initial implementation of payment router indexing #6892

schottra · 2023-12-08T05:42:02Z

Description

This adapts some existing work from index_user_bank.py to create a payment router indexer. Things work a little differently for payment router:

We don't have to index "Create" instructions, and the inner transfers are wrapped in a "Route" instruction.
The sender in the route instruction will always be an ATA owned by the payment router PDA.
The sender in the (technically optional) first Transfer instruction might be a user bank, but isn't required to be one.
There is a variable amount of recipients, each of which may or may not be a user bank and need to have a transfer transaction logged
Indexing a purchase is similar to the user bank indexing process: check the balance change of each recipient against the amount they should have received as defined in the splits from the premium content metadata. All recipients must receive at least their assigned split
Extra amount is a slightly trickier computation (total of all balance changes - track price)
In order to determine which mint was used, we have to check the sender in the Route instruction against known values for the AUDIO and USDC ATAs which are owned by payment router. These are derived in a bit of a convoluted way using the payment router program id & a seed string ("payment_router") to compute the expected base address followed by a get_associated_token_address using our known mint addresses for AUDIO/USDC.

So! Given all that, here's what we're doing:

Check for transactions which reference the payment router program id
For each transaction, look for a Route instruction to be present and extract it
For the Route instruction, check the sender against our known token accounts. If it's USDC, index each recipient's balance change if said recipient is a user bank.
If there is a memo transaction, attempt to parse it as purchase metadata. If we succeed, check the recipient balance changes against the expected splits. Index the purchase if it's valid. Note: For payment router transactions, we can't rely on account addresses to determine the userbank to which the purchase should be assigned. That is now passed as part of the memo transaction metadata.
Process any audio matching challenges that would result for recipients based on the transaction

Things we are explicitly not doing:

We are ignoring payment router transactions which deal with AUDIO. That indexing will be added a later date
We do not detect if the initial sender (the address sending tokens into payment router before the route instruction) is a userbank and add a USDC transaction record for that user. This will be added in follow up PRs, but requires some additional logic.

How Has This Been Tested?

This has been tested locally using an updated solana-test-validator running a development deployment of the payment router program. I submitted transactions which match the shape of USDC content purchases and verified that we correctly generate transaction records, purchase records, and balance changes.

…ment-router * origin/main: (30 commits) [INF-557] Convert libs to esm so vite HMR works (#6882) [C-3410] Native password page + Common rollup changes (#6874) Remove extraneous icon imports (#6885) [C-3423] Add Avatar component to native harmony (#6884) feat: Add text align to Text component props (#6856) [C-3431] Add universal harmony icons (#6877) [Sign Up Redesign] Fix sign up allowed routes [C-3488] (#6879) Bump version to 0.5.33 PROTO-1364: fix relay heavy load (#6878) [PAY-2197] Check Redis cache for transactions (#6812) [PAY-2194] Update summary tables (#6871) [C-3458] Sign Up QA round 2 (#6851) [C-3479] 404 on invalid collection url (#6875) [C-3332] Fix playlist edit after create (#6873) [C-2959] Fix creating playlist from track (#6868) [C-3480] Fix null description in Edit Playlist (#6870) fix openresty version (#6869) [Mobile] [C-3413] Fix verified sign up on mobile via Twitter/IG (#6855) PROTO-1474: Discovery uses db tempalte for integration tests (#6853) Bump version to 0.5.32 ...

gitguardian · 2023-12-08T05:42:07Z

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
688750	Generic High Entropy Secret	`17c6d3c`	monitoring/uptime/node-ui/.env.stage.local	View secret
2416684	Generic High Entropy Secret	`17c6d3c`	monitoring/uptime/node-ui/.env.stage.local	View secret
2416685	Generic High Entropy Secret	`17c6d3c`	monitoring/uptime/node-ui/.env.stage.local	View secret
2416686	Generic High Entropy Secret	`17c6d3c`	monitoring/uptime/node-ui/.env.stage.local	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

schottra · 2023-12-08T05:46:28Z

packages/discovery-provider/src/tasks/index_payment_router.py

@@ -0,0 +1,859 @@
+import concurrent.futures


Might be worth comparing this file to index_user_bank.py if you're not familiar with the indexing. A lot of this logic was copied and modified from there. It could use cleanup later on and we could potentially move all of the logic in the second half of the file around fetching and caching transactions/slots into a shared wrapper of some sort.

there have been many talks about templatizing indexers...

packages/discovery-provider/src/utils/redis_constants.py

packages/discovery-provider/src/tasks/index_payment_router.py

dharit-tan · 2023-12-08T19:44:14Z

packages/discovery-provider/src/tasks/index_payment_router.py

+
+
+# Return highest payment router slot that has been processed
+def get_highest_payment_router_tx_slot(session: Session):


nit and you don't need to do it now, but i think we should prefer using the system that index_spl_token.py follows where it stores a single row containing the latest transaction.
ideally we update user bank and reward manager indexers to do this too.

Oh interesting. Agreed, that would probably be better.

monitoring/uptime/main.go

packages/discovery-provider/src/tasks/index_payment_router.py

raymondjacobson

No major issues jump out at me. Seems pretty safe to merge to main IMO!

packages/discovery-provider/default_config.ini

raymondjacobson · 2023-12-08T19:47:57Z

packages/discovery-provider/src/app.py

@@ -370,6 +371,10 @@ def configure_celery(celery, test_config=None):
                "task": "index_user_bank",
                "schedule": timedelta(seconds=5),
            },
+            "index_payment_router": {
+                "task": "index_payment_router",
+                "schedule": timedelta(seconds=5),


I actually don't recall the original reason for 5s interrupts in these on solana, but might improve our overall indexing speed if we lower it... (not this PR)

Worth a test! I would imagine the volume being processed on prod is much higher so maybe we don't want to get overwhelmed? But as it was, I was seeing up to 10 seconds locally for the transaction to be picked up.

raymondjacobson · 2023-12-08T19:50:25Z

packages/discovery-provider/src/tasks/index_payment_router.py

+                purchaser_user_id = int(purchaser_user_id_str)
+                blocknumber = int(blocknumber_str)
+
+                # TODO: Wait for blocknumber to be indexed by ACDC


@rickyrombo TODO!!!!!!!!!!

raymondjacobson · 2023-12-08T19:53:00Z

packages/discovery-provider/src/tasks/index_payment_router.py

+                        )
+                        continue
+                    query = session.query(TrackPriceHistory)
+                    if env != "dev":


hopefully solder's bankrun replacement for solana-test-validator helps here eventually in dev
https://github.com/kevinheavey/solders

packages/discovery-provider/src/tasks/index_payment_router.py

raymondjacobson · 2023-12-08T19:57:16Z

packages/discovery-provider/src/tasks/index_payment_router.py

+                        raise
+
+            # Sort by slot
+            # Note: while it's possible (even likely) to have multiple tx in the same slot,


I like the idea of enforcing determinism here (I know you just copied this code) so discovery nodes are consistent between eachother

You thinking just sort on an arbitrary value so that each discovery node is guaranteed to process in the same order? Maybe signature? Thinking it's something that could be added in a future cleanup PR.

packages/discovery-provider/src/tasks/index_payment_router.py

dharit-tan

fire!

sddioulde

sick PR! love to see it

sddioulde · 2023-12-08T20:49:01Z

packages/discovery-provider/ddl/migrations/0041_add_payment_router_txs.sql

@@ -0,0 +1,11 @@
+begin;


by the way 0041 is already in main. you'll have to update this file name

ooo good catch thank you!

…ment-router * origin/main: Commit package-lock changes (#6895) [PAY-2221] Purchase content through payment router (#6880) Re-remove copy . . from discovery dev dockerfile (#6889) [C-3456] Improve mobile user-list loading (#6845) Address mobile icon issues (#6893) Polish social media flow, other QA [C-3418] (#6887) relay: fix error on no metadata (#6883) Bump version to 0.5.34 [Harmony] TextInput - show red error if input too long (#6886)

[952c141] [PAY-2278] Disable coinflow apple/gpay on desktop (#6961) Raymond Jacobson [aab8f30] Notify Slack when stage nodes aren't updating (#6922) Theo Ilie [6a71a1a] Make finding slow queries compatible with pg 13+ (#6950) Theo Ilie [bf24278] [PAY-2283] Disable purchase button immediately after click (#6960) Marcus Pasell [509efd7] [PAY-2274] Update minWidths of pay extra pills to prevent clipping (#6959) Randy Schott [7a8d979] Use retry logic for coinflow (#6957) Marcus Pasell [bc346fd] Bump mobile versions for full release (#6958) Dylan Jeffers [a336fc1] [PAY-2286] Increase payment router indexer frequency (#6956) Raymond Jacobson [4d674d0] [C-3449] Add native welcome modal & consolidate messages to common (#6939) JD Francis [4d1e378] Scheduled release mobile (#6954) Isaac Solo [c954108] Bump version to 0.6.3 audius-infra [54e3a24] [C-3485, C-3487, C-3486] Add Track to Album (web) (#6936) Andrew Mendelsohn [551902e] fix: Fix native select genres pill performance issue (#6953) JD Francis [801d075] [PAY-2224] Add ExploreTransaction callout to sale/purchase details (#6944) Raymond Jacobson [7fe4adf] Fix pay extra (#6952) Raymond Jacobson [f02009d] Revert "Extra amount is in dollars (#6949)" (#6951) Marcus Pasell [c4c8b6b] Extra amount is in dollars (#6949) Marcus Pasell [b193f23] Fix primary key for DashboardWalletUsers (#6934) nicoback2 [9e33796] [C-3506] Upgrade selectable-pill to use gestures (#6938) Dylan Jeffers [96bb775] [C-3435] Add CompletionCheck to native harmony (#6947) Kyle Shanks [4b4328f] [PAY-2276] Selectable amounts for coinflow add funds web (#6946) Reed [b6a82b9] [PAY-2277][PAY-2272] Fix purchase modaldrawer scroll and safe areas (#6941) Raymond Jacobson [fe21499] [PAY-2273] Fix payExtra for coinflow and stripe (#6943) Raymond Jacobson [215a36a] [PAY-2270] Show vendor button only if more than 1 vendor (#6945) Reed [9c2e79c] Use 0.0.0.0 default host for vite app (#6942) Raymond Jacobson [8f2f0f6] Bump version to 0.6.2 audius-infra [c45aa43] [PAY-2249] Adds tests for payment router indexing (#6932) Randy Schott [f18f78e] [C-3509] Fix sign-in legacy styles (#6940) Dylan Jeffers [ac8aa98] [C-3475] Allow editing album track order (#6876) Andrew Mendelsohn [5b9d32c] [C-3505] Upgrade react-native-gesture-handler (#6923) Dylan Jeffers [3b24a93] Fix feature flag lint issue (#6937) Dylan Jeffers [38670b9] Add dashboard wallet users table C-3497 (#6933) nicoback2 [3d8a5cd] Scheduled release web client (#6866) Isaac Solo [8213ebb] [PAY-2271] Skip preflight in coinflow checkout (#6930) Raymond Jacobson [354a5b6] [PAY-2269] Fix mobile config (#6931) Raymond Jacobson [ad95c08] [C-3429] Add FollowButton to native harmony (#6928) Kyle Shanks [b18a138] rm does_follow_current_user from API response (#6836) Steve Perkins [88bde40] Fix defaults when coinflow disabled + android purchase vendor drawer (#6926) Reed [76bede3] Remove CTA on withdrawals table empty state (#6924) Reed [683748f] Fix mobile web purchase drawer layout (#6927) Reed [92ad724] Add uptime to node overview tables in protocol dashboard (#6900) Michelle Brier [be58789] [PROTO-1481] Hardcode mediorum dbUrl to localhost (#6918) Theo Ilie [b6207d8] [C-3446] Native Select Genres Page (#6916) JD Francis [cbb0508] Bump version to 0.6.1 audius-infra [59f5b5c] Bump version to 0.6.0 (#6925) Raymond Jacobson [ca78cdf] [PROTO-1480] Expose hosted DBs in healthz (#6908) Theo Ilie [e10f870] Fix prices add types for purchases (#6921) Marcus Pasell [6b6ebb6] [PAY-2246][PAY-2257][PAY-2264] Fix mobile coinflow support (#6919) Raymond Jacobson [d6de701] [PAY-2263] Always Allow Deactivations on DN Relay (#6920) Marcus Pasell [7c379ff] [PAY-2263] Decode ABI for blocked users and relay if they're deactivation requests (#6917) Marcus Pasell [dc91844] [PAY-2241][PAY-2259] Add Funds via Coinflow (#6915) Marcus Pasell [0f571a1] [C-3504] Enable native animated icons (#6914) Dylan Jeffers [15c7909] Add sign in route aliases C-3503 (#6912) nicoback2 [5f7e980] Fix timestamp comparison for sched releases (#6913) Isaac Solo [46ce24f] [C-3445] Add native finish-profile-screen (#6909) Dylan Jeffers [ad8463d] [C-3436] Add SelectablePill component to native harmony (#6903) Kyle Shanks [a9df266] Bump version to 0.5.36 audius-infra [768e3b5] trpc test setup (#6872) Steve Perkins [57348ab] Separate web / mobile trpc client (#6906) Steve Perkins [8c6d628] [C-3452 C-3448] Native sign-on screen (#6905) Dylan Jeffers [7929dd0] [PAY-2128] Implement general coinflow scaffolding (#6847) Raymond Jacobson [ab24471] Add memo for purchaser user id (#6846) Reed [68d9f71] [C-3444] Native Pick Handle Page (#6890) JD Francis [1a54363] Fix missing string format on payment router indexing (#6907) Randy Schott [730423c] [C-3440] Add Hint component to native harmony (#6904) Kyle Shanks [aab9814] [PAY-2132] Premium content explore page denylist (#6902) Reed [eecf247] Bump version to 0.5.35 audius-infra [1f99498] [PAY-2178] Initial implementation of payment router indexing (#6892) Randy Schott

schottra added 5 commits December 5, 2023 14:41

checkpoint

af53bed

wire up payment router program address

10c95da

initial pass at indexing payment router transactions, needs cleanup

6b57f6f

cleanup

be2045d

pull-request-size bot added the size/XL label Dec 8, 2023

schottra commented Dec 8, 2023

View reviewed changes

cleanup

943db0f

schottra requested review from rickyrombo, raymondjacobson, dharit-tan and sddioulde December 8, 2023 05:47

schottra changed the base branch from feature-index-payment-router to main December 8, 2023 17:49

schottra added 2 commits December 8, 2023 12:58

fix lint errors

609c667

more lint fixes

0087933

dharit-tan reviewed Dec 8, 2023

View reviewed changes

packages/discovery-provider/src/utils/redis_constants.py Outdated Show resolved Hide resolved

remove unused redis keys

b4bc778

dharit-tan reviewed Dec 8, 2023

View reviewed changes

packages/discovery-provider/src/tasks/index_payment_router.py Outdated Show resolved Hide resolved

dharit-tan reviewed Dec 8, 2023

View reviewed changes

packages/discovery-provider/src/tasks/index_payment_router.py Outdated Show resolved Hide resolved

dharit-tan reviewed Dec 8, 2023

View reviewed changes

raymondjacobson reviewed Dec 8, 2023

View reviewed changes

monitoring/uptime/main.go Show resolved Hide resolved

dharit-tan reviewed Dec 8, 2023

View reviewed changes

packages/discovery-provider/src/tasks/index_payment_router.py Outdated Show resolved Hide resolved

raymondjacobson approved these changes Dec 8, 2023

View reviewed changes

dharit-tan reviewed Dec 8, 2023

View reviewed changes

packages/discovery-provider/src/tasks/index_payment_router.py Outdated Show resolved Hide resolved

dharit-tan approved these changes Dec 8, 2023

View reviewed changes

schottra added 3 commits December 8, 2023 15:36

PR feedback

12a0ad1

final cleanup

71eeb6d

actually use the exception, oops

4995137

sddioulde approved these changes Dec 8, 2023

View reviewed changes

schottra added 2 commits December 8, 2023 16:26

rename migration file

0bb48d7

schottra enabled auto-merge (squash) December 8, 2023 21:27

schottra merged commit 1f99498 into main Dec 11, 2023
10 checks passed

schottra deleted the rs-pay-2178-index-payment-router branch December 11, 2023 15:56

raymondjacobson mentioned this pull request Dec 11, 2023

Add payment router program ids AudiusProject/audius-docker-compose#419

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PAY-2178] Initial implementation of payment router indexing #6892

[PAY-2178] Initial implementation of payment router indexing #6892

schottra commented Dec 8, 2023 •

edited

Loading

gitguardian bot commented Dec 8, 2023 •

edited

Loading

schottra Dec 8, 2023

dharit-tan Dec 8, 2023

dharit-tan Dec 8, 2023

schottra Dec 8, 2023

raymondjacobson left a comment

raymondjacobson Dec 8, 2023

schottra Dec 8, 2023

raymondjacobson Dec 8, 2023

raymondjacobson Dec 8, 2023

raymondjacobson Dec 8, 2023

schottra Dec 8, 2023

dharit-tan left a comment

sddioulde left a comment

sddioulde Dec 8, 2023

schottra Dec 8, 2023



		# Return highest payment router slot that has been processed
		def get_highest_payment_router_tx_slot(session: Session):

[PAY-2178] Initial implementation of payment router indexing #6892

[PAY-2178] Initial implementation of payment router indexing #6892

Conversation

schottra commented Dec 8, 2023 • edited Loading

Description

How Has This Been Tested?

gitguardian bot commented Dec 8, 2023 • edited Loading

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondjacobson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dharit-tan left a comment

Choose a reason for hiding this comment

sddioulde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schottra commented Dec 8, 2023 •

edited

Loading

gitguardian bot commented Dec 8, 2023 •

edited

Loading