Squashed commit of the following:

commit 3ab0989 Author: Pablo Fernandez <p@f7z.io> Date: Fri Aug 2 13:34:22 2024 +0100 handle events coming in with the same created_at and different IDs commit 08b3a2e Author: Pablo Fernandez <p@f7z.io> Date: Fri Aug 2 13:33:24 2024 +0100 emit insufficient balance commit b85cbae Author: Pablo Fernandez <p@f7z.io> Date: Fri Aug 2 11:57:40 2024 +0100 handle the same event being incorrectly being published multiple times before it finishes publishing commit ab187ba Author: Pablo Fernandez <p@f7z.io> Date: Thu Aug 1 16:33:34 2024 +0100 wrap string in error commit 9fb7f50 Author: Pablo Fernandez <p@f7z.io> Date: Thu Aug 1 15:57:10 2024 +0100 fix LN payments commit b47e829 Author: Pablo Fernandez <p@f7z.io> Date: Thu Aug 1 15:56:22 2024 +0100 Accept from that return undefined commit 81784d5 Author: Pablo Fernandez <p@f7z.io> Date: Thu Aug 1 14:07:14 2024 +0100 improve nutzaps commit a013d28 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:55:28 2024 +0100 rename meta-data because its bet-ter commit 942afe9 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:54:09 2024 +0100 debug commit dbf0e79 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:50:42 2024 +0100 be explicit about the timeout commit c8d9fc8 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:50:31 2024 +0100 fix order of getting reference tags so we don't add markers to h tags commit d3cfdb0 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:47:36 2024 +0100 refactor NDKCashuWallet a bit to be more explicit on what tags are private or public commit 5217f01 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:47:11 2024 +0100 remove dead code commit f233ad8 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:46:55 2024 +0100 fetch main user's relay lists in order to get a better clue of where the user's wallets are commit 341a6fd Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 22:46:26 2024 +0100 Read events' onRelays from the subscription manager, this will give a better idea of where events exists commit 701d546 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 20:28:00 2024 +0100 normalize relay urls to connect to explicit relays commit 68f4a60 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 14:09:34 2024 +0100 delete unused file commit f1e5624 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 11:12:26 2024 +0100 Add documentation on subscription lifecycle commit 79d2333 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 10:12:40 2024 +0100 lint commit 1a181a4 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 10:12:05 2024 +0100 format commit 7fe4ca2 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 10:05:29 2024 +0100 changeset commit 8aae145 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 10:03:23 2024 +0100 Massive refactor of how subscriptions are fingerprinted, grouped, ungrouped and their internal lifecycle commit 2dd41ec Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 09:53:42 2024 +0100 correct NDKSubscriptionTier kind commit 2286630 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 09:50:06 2024 +0100 inherit relay auth policy commit 21a4ac6 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 09:45:08 2024 +0100 add ndk reference to relays commit 69cc6f9 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 31 09:44:21 2024 +0100 k-tag deletions commit 26e2da9 Author: Pablo Fernandez <p@f7z.io> Date: Thu Jul 25 10:08:18 2024 +0100 h-tag when getting reference tags commit 99d413a Author: Pablo Fernandez <p@f7z.io> Date: Mon Jul 22 21:30:34 2024 +0100 wip commit 158b65b Author: Pablo Fernandez <p@f7z.io> Date: Mon Jul 22 20:16:00 2024 +0100 wip commit 8ad0be0 Merge: f7eaec3 8185295 Author: Pablo Fernandez <p@f7z.io> Date: Mon Jul 22 20:15:34 2024 +0100 merge with all-your-relays-are-mine commit f7eaec3 Author: Pablo Fernandez <p@f7z.io> Date: Mon Jul 22 20:11:26 2024 +0100 abstract relay stuff commit 8185295 Author: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Mon Jul 22 17:33:57 2024 +0200 Remove returns from methods in auth-policies commit 4c47449 Author: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Mon Jul 22 17:25:51 2024 +0200 Remove dependency on Nostr Tools Relay commit 741035c Author: Pablo Fernandez <p@f7z.io> Date: Sat Jul 20 08:17:30 2024 +0100 pnpm commit 32495e9 Author: Pablo Fernandez <p@f7z.io> Date: Sat Jul 20 08:17:18 2024 +0100 change the zapping interface commit 585fa41 Author: Pablo Fernandez <p@f7z.io> Date: Sat Jul 20 08:13:51 2024 +0100 event wrappers commit 74c5b08 Author: Pablo Fernandez <p@f7z.io> Date: Sat Jul 20 08:12:40 2024 +0100 prevent writing to closed websockets commit a64ad45 Author: Pablo Fernandez <p@f7z.io> Date: Sat Jul 20 08:10:43 2024 +0100 ensure we only eose once per subscription commit 337f841 Author: Pablo Fernandez <p@f7z.io> Date: Fri Jul 19 00:24:38 2024 +0100 wip commit 848abda Author: Pablo Fernandez <p@f7z.io> Date: Fri Jul 19 00:22:50 2024 +0100 inherir properties from ndkevent when using `from` commit 709f9d7 Author: Pablo Fernandez <p@f7z.io> Date: Thu Jul 18 11:46:25 2024 +0100 refactor ndk-wallet commit 415ac67 Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 17 12:32:18 2024 +0100 NIP-29 work commit 191d82a Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 17 12:07:12 2024 +0100 wip commit 85cf1da Author: Pablo Fernandez <p@f7z.io> Date: Wed Jul 17 12:07:07 2024 +0100 wip commit 0b9bf8e Author: Pablo Fernandez <p@f7z.io> Date: Mon Jul 15 10:47:42 2024 +0100 wip
nostr-dev-kit · Aug 2, 2024 · cdb656f · cdb656f
1 parent 4dd675a
commit cdb656f
Show file tree

Hide file tree

Showing 133 changed files with 7,783 additions and 2,702 deletions.
diff --git a/.changeset/big-rivers-deliver.md b/.changeset/big-rivers-deliver.md
diff --git a/.changeset/clever-bears-knock.md b/.changeset/clever-bears-knock.md
diff --git a/.changeset/curly-dolphins-speak.md b/.changeset/curly-dolphins-speak.md
diff --git a/.changeset/kind-news-sin.md b/.changeset/kind-news-sin.md
diff --git a/.changeset/old-cats-attack.md b/.changeset/old-cats-attack.md
diff --git a/.changeset/three-hats-drop.md b/.changeset/three-hats-drop.md
diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts
@@ -1,7 +1,8 @@
 import { defineConfig } from 'vitepress'
+import { withMermaid } from "vitepress-plugin-mermaid";
 
 // https://vitepress.dev/reference/site-config
-export default defineConfig({
+export default withMermaid(defineConfig({
   title: "NDK",
   description: "NDK Docs",
   base: "/ndk/",
@@ -29,6 +30,7 @@ export default defineConfig({
           { text: 'Local-first', link: '/tutorial/local-first' },
           { text: 'Publishing', link: '/tutorial/publishing' },
           { text: "Subscription Management", link: '/tutorial/subscription-management' },
+          { text: "Speed", link: '/tutorial/speed' },
         ]
       },
       {
@@ -44,10 +46,16 @@ export default defineConfig({
           { text: 'NDK Svelte', link: '/wrappers/svelte' },
         ]
       },
+      {
+        text: "Internals",
+        items: [
+          { text: "Subscription Lifecycle", link: '/internals/subscriptions' },
+        ]
+      }
     ],
 
     socialLinks: [
       { icon: 'github', link: 'https://github.com/nostr-dev-kit/ndk' }
     ]
   }
-})
+}))
diff --git a/docs/internals/subscriptions.md b/docs/internals/subscriptions.md
@@ -0,0 +1,205 @@
+# Subscriptions Lifecycle
+When an application creates a subscription a lot of things happen under the hood.
+
+Say we want to see `kind:1` events from pubkeys `123`, `456`, and `678`.
+
+```ts
+const subscription = ndk.subscribe({ kinds: [1], authors: [ "123", "456", "678" ]})
+```
+
+Since the application level didn't explicitly provide a relay-set, which is the most common use case, NDK will calculate a relay set based on the outbox model plus a variety of some other factors.
+
+So the first thing we'll do before talking to relays is, decide to *which* relays we should talk to.
+
+The `calculateRelaySetsFromFilters` function will take care of this and provide us with a map of relay URLs and filters for each relay.
+
+This means that the query, as specified by the client might be broken into distinct queries specialized for the different relays.
+
+For example, if we have 3 relays, and the query is for `kind:1` events from pubkeys `a` and `b`, the `calculateRelaySetsFromFilters` function might return something like this:
+
+```ts
+{
+  "wss://relay1": { kinds: [1], authors: [ "a" ] },
+  "wss://relay2": { kinds: [1], authors: [ "b" ] },
+}
+```
+
+```mermaid
+flowchart TD
+    Client -->|"kinds: [1], authors: [a, b]"| Subscription1
+    Subscription1 -->|"kinds: [1], authors: [a]"| wss://relay1
+    Subscription1 -->|"kinds: [1], authors: [b]"| wss://relay2
+```
+
+## Subscription bundling
+Once the subscription has been split into the filters each relay should receive, the filters are sent to the individual `NDKRelay`'s `NDKRelaySubscriptionManager` instances.
+
+`NDKRelaySubscriptionManager` is responsible for keeping track of the active and scheduled subscriptions that are pending to be executed within an individual relay.
+
+This is an important aspect to consider:
+
+> `NDKSubscription` have a different lifecycle than `NDKRelaySubscription`. For example, a subscription that is set to close after EOSE might still be active within the `NDKSubscription` lifecycle, but it might have been already been closed within the `NDKRelaySubscription` lifecycle, since NDK attempts to keep the minimum amount of open subscriptions at any given time.
+
+## NDKRelaySubscription
+Most NDK subscriptions (by default) are set to be executed with a grouping delay. Will cover what this looks like in practice later, but for now, let's understand than when the `NDKRelaySubscriptionManager` receives an order, it might not execute it right away.
+
+The different filters that can be grouped together (thus executed as a single `REQ` within a relay) are grouped within the same `NDKRelaySubscription` instance and the execution scheduler is computed respecting what each individual `NDKSubscription` has requested.
+
+(For example, if a subscription with a `groupingDelay` of `at-least` 500 millisecond has been grouped with another subscription with a `groupingDelay` of `at-least` 1000 milliseconds, the `NDKRelaySubscriptionManager` will wait 1000 ms before sending the `REQ` to this particular relay).
+
+### Execution
+Once the filter is executed at the relay level, the `REQ` is submitted into that relay's `NDKRelayConnectivity` instance, which will take care of monitoring for responses for this particular REQ and communicate them back into the `NDKRelaySubscription` instance.
+
+Each `EVENT` that comes back as a response to our `REQ` within this `NDKRelaySubscription` instance is then compared with the filters of each `NDKSubscription` that has been grouped and if it matches, it is sent back to the `NDKSubscription` instance.
+
+
+# Example
+
+If an application requests `kind:1` of pubkeys `123`, `456`, and `789`. It creates an `NDKSubscription`:
+
+```ts
+ndk.subscribe({ kinds: [1], authors: [ "123", "456", "789" ]}, { groupableDelay: 500, groupableDelayType: 'at-least' })
+// results in NDKSubscription1 with filters { kinds: [1], authors: [ "123", "456", "789" ] }
+```
+
+Some other part of the application requests a kind:7 from pubkey `123` at the same time.
+
+```ts
+ndk.subscribe({ kinds: [7], authors: [ "123" ]}, { groupableDelay: 500, groupableDelayType: 'at-most' })
+// results in NDKSubscription2 with filters { kinds: [7], authors: [ "123" ] }
+```
+
+```mermaid
+flowchart TD
+    subgraph Subscriptions Lifecycle
+        A[Application] -->|"kinds: [1], authors: [123, 456, 678], groupingDelay: at-least 500ms"| B[NDKSubscription1]
+        
+        A2[Application] -->|"kinds: [7], authors: [123], groupingDelay: at-most 1000ms"| B2[NDKSubscription2]
+    end
+```
+
+Both subscriptions have their relayset calculated by NDK and, the resulting filters are sent into the `NDKRelaySubscriptionManager`, which will decide what, and how filters can be grouped.
+
+```mermaid
+flowchart TD
+    subgraph Subscriptions Lifecycle
+        A[Application] -->|"kinds: [1], authors: [123, 456, 678], groupingDelay: at-least 500ms"| B[NDKSubscription1]
+        B --> C{Calculate Relay Sets}
+        
+        A2[Application] -->|"kinds: [7], authors: [123], groupingDelay: at-most 1000ms"| B2[NDKSubscription2]
+        B2 --> C2{Calculate Relay Sets}
+    end
+
+    subgraph Subscription Bundling
+        C -->|"kinds: [1], authors: [123]"| E1[wss://relay1 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [456]"| E2[wss://relay2 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [678]"| E3[wss://relay3 NDKRelaySubscriptionManager]
+
+        C2 -->|"kinds: [7], authors: [123]"| E1
+    end
+```
+
+The `NDKRelaySubscriptionManager` will create `NDKRelaySubscription` instances, or add filters to them if `NDKRelaySubscription` with the same filter fingerprint exists.
+
+```mermaid
+flowchart TD
+    subgraph Subscriptions Lifecycle
+        A[Application] -->|"kinds: [1], authors: [123, 456, 678], groupingDelay: at-least 500ms"| B[NDKSubscription1]
+        B --> C{Calculate Relay Sets}
+        
+        A2[Application] -->|"kinds: [7], authors: [123], groupingDelay: at-most 1000ms"| B2[NDKSubscription2]
+        B2 --> C2{Calculate Relay Sets}
+    end
+
+    subgraph Subscription Bundling
+        C -->|"kinds: [1], authors: [123]"| E1[wss://relay1 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [456]"| E2[wss://relay2 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [678]"| E3[wss://relay3 NDKRelaySubscriptionManager]
+        
+        C2 -->|"kinds: [7], authors: [123]"| E1
+
+        E1 -->|"Grouping Delay: at-most 1000ms"| F1[NDKRelaySubscription]
+        E2 -->|"Grouping Delay: at-least 500ms"| F2[NDKRelaySubscription]
+        E3 -->|"Grouping Delay: at-least 500ms"| F3[NDKRelaySubscription]
+    end
+```
+
+Each individual `NDKRelaySubscription` computes the execution schedule of the filters it has received and sends them to the `NDKRelayConnectivity` instance, which in turns sends the `REQ` to the relay.
+
+```mermaid
+flowchart TD
+    subgraph Subscriptions Lifecycle
+        A[Application] -->|"kinds: [1], authors: [123, 456, 678], groupingDelay: at-least 500ms"| B[NDKSubscription1]
+        B --> C{Calculate Relay Sets}
+        
+        A2[Application] -->|"kinds: [7], authors: [123], groupingDelay: at-most 1000ms"| B2[NDKSubscription2]
+        B2 --> C2{Calculate Relay Sets}
+    end
+
+    subgraph Subscription Bundling
+        C -->|"kinds: [1], authors: [123]"| E1[wss://relay1 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [456]"| E2[wss://relay2 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [678]"| E3[wss://relay3 NDKRelaySubscriptionManager]
+        
+        C2 -->|"kinds: [7], authors: [123]"| E1
+
+        E1 -->|"Grouping Delay: at-most 1000ms"| F1[NDKRelaySubscription]
+        E2 -->|"Grouping Delay: at-least 500ms"| F2[NDKRelaySubscription]
+        E3 -->|"Grouping Delay: at-least 500ms"| F3[NDKRelaySubscription]
+
+        F1 -->|"REQ: kinds: [1, 7], authors: [123]"| G1[NDKRelayConnectivity]
+        F2 -->|"REQ: kinds: [1], authors: [456]"| G2[NDKRelayConnectivity]
+        F3 -->|"REQ: kinds: [1], authors: [678]"| G3[NDKRelayConnectivity]
+    end
+
+    subgraph Execution
+        G1 -->|"Send REQ to wss://relay1 after 1000ms"| R1[Relay1]
+        G2 -->|"Send REQ to wss://relay2 after 500ms"| R2[Relay2]
+        G3 -->|"Send REQ to wss://relay3 after 500ms"| R3[Relay3]
+    end
+```
+
+As the events come from the relays, `NDKRelayConnectivity` will send them back to the `NDKRelaySubscription` instance, which will compare the event with the filters of the `NDKSubscription` instances that have been grouped together and send the received event back to the correct `NDKSubscription` instance.
+
+```mermaid
+flowchart TD
+    subgraph Subscriptions Lifecycle
+        A[Application] -->|"kinds: [1], authors: [123, 456, 678], groupingDelay: at-least 500ms"| B[NDKSubscription1]
+        B --> C{Calculate Relay Sets}
+        
+        A2[Application] -->|"kinds: [7], authors: [123], groupingDelay: at-most 1000ms"| B2[NDKSubscription2]
+        B2 --> C2{Calculate Relay Sets}
+    end
+
+    subgraph Subscription Bundling
+        C -->|"kinds: [1], authors: [123]"| E1[wss://relay1 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [456]"| E2[wss://relay2 NDKRelaySubscriptionManager]
+        C -->|"kinds: [1], authors: [678]"| E3[wss://relay3 NDKRelaySubscriptionManager]
+        
+        C2 -->|"kinds: [7], authors: [123]"| E1
+
+        E1 -->|"Grouping Delay: at-most 1000ms"| F1[NDKRelaySubscription]
+        E2 -->|"Grouping Delay: at-least 500ms"| F2[NDKRelaySubscription]
+        E3 -->|"Grouping Delay: at-least 500ms"| F3[NDKRelaySubscription]
+
+        F1 -->|"REQ: kinds: [1, 7], authors: [123]"| G1[NDKRelayConnectivity]
+        F2 -->|"REQ: kinds: [1], authors: [456]"| G2[NDKRelayConnectivity]
+        F3 -->|"REQ: kinds: [1], authors: [678]"| G3[NDKRelayConnectivity]
+    end
+
+    subgraph Execution
+        G1 -->|"Send REQ to wss://relay1 after 1000ms"| R1[Relay1]
+        G2 -->|"Send REQ to wss://relay2 after 500ms"| R2[Relay2]
+        G3 -->|"Send REQ to wss://relay3 after 500ms"| R3[Relay3]
+
+        R1 -->|"EVENT: kinds: [1]"| H1[NDKRelaySubscription]
+        R1 -->|"EVENT: kinds: [7]"| H2[NDKRelaySubscription]
+        R2 -->|"EVENT"| H3[NDKRelaySubscription]
+        R3 -->|"EVENT"| H4[NDKRelaySubscription]
+
+        H1 -->|"Matched Filters: kinds: [1]"| I1[NDKSubscription1]
+        H2 -->|"Matched Filters: kinds: [7]"| I2[NDKSubscription2]
+        H3 -->|"Matched Filters: kinds: [1]"| I1
+        H4 -->|"Matched Filters: kinds: [1]"| I1
+    end
+```
diff --git a/docs/tutorial/speed.md b/docs/tutorial/speed.md
@@ -0,0 +1,48 @@
+# Built for speed
+
+NDK makes multiple optimizations possible to create a performant client.
+
+## Signature Verifications
+Signature validation is typically the most computationally expensive operation in a nostr client. Thus, NDK attempts to reduce the number of signature verifications that need to be done as much as possible.
+
+### Service Worker signature validation
+In order to create performant clients, it's very useful to offload this computation to a service worker, to avoid blocking the main thread.
+
+```ts
+// Using with vite
+const sigWorker = import.meta.env.DEV ?
+		new Worker(new URL('@nostr-dev-kit/ndk/workers/sig-verification?worker', import.meta.url), { type: 'module' }) : new NDKSigVerificationWorker();
+
+const ndk = new NDK();
+ndk.signatureVerificationWorker = worker
+```
+
+Since signature verification will thus be done asynchronously, it's important to listen for invalid signatures and handle them appropriately; you should
+always warn your users when they are receiving invalid signatures from a relay and/or immediately disconnect from an evil relay.
+
+```ts
+ndk.on("event:invalid-sig", (event) => {
+    const { relay } = event;
+    console.error("Invalid signature coming from relay", relay.url);
+});
+```
+
+### Signature verification sampling
+Another parameter we can tweak is how many signatures we verify. By default, NDK will verify every signature, but you can change this by setting a per-relay verification rate.
+
+```ts
+ndk.initialValidationRatio = 0.5; // Only verify 50% of the signatures for each relay
+ndk.lowestValidationRatio = 0.01; // Never verify less than 1% of the signatures for each relay
+```
+
+NDK will then begin verifying signatures from each relay and, as signatures as verified, it will reduce the verification rate for that relay.
+
+### Custom validation ratio function
+If you need further control on how the verification rate is adjusted, you can provide a validation ratio function. This function will be called periodically and the returning value will be used to adjust the verification rate.
+
+```ts
+ndk.validationRatioFn = (relay: NDKRelay, validatedEvents: number, nonValidatedEvents: number): number => {
+    // write your own custom function here
+    return validatedEvents / (validatedEvents + nonValidatedEvents);
+}
+```
diff --git a/docs/tutorial/zaps/index.md b/docs/tutorial/zaps/index.md
@@ -0,0 +1,38 @@
+# Zaps
+
+NDK comes with an interface to make zapping as simple as possible.
+
+```ts
+const user = await ndk.getUserFromNip05("pablo@f7z.io");
+const zapper = await ndk.zap(user, 1000)
+```
+
+## Connecting to WebLN
+Advanced users might have a webln extension available in their browsers. To attempt to use their WebLN to pay, you can connect webln with NDK.
+
+```ts
+import { requestProvider } from "webln";
+let weblnProvider;
+requestProvider().then(provider => weblnProvider = provider });
+
+// whenever the user wants to pay for something, and using LN is an option for the payment
+// this function will be called
+ndk.walletConfig.onLnPay = async ({pr: string, amount: number, target?: NDKEvent | NDKUser}) => {
+    if (weblnProvider) {
+        if (confirm("Would you like to pay with your WebLN provider?")) {
+            await weblnProvider.sendPayment(pr);
+        }
+    } else {
+        // show a QR code to the user or handle in some way
+    }
+});
+```
+
+Now from anywhere in the app, you can:
+
+```ts
+event.zap(1000); // zap an event 1 sat
+```
+
+## Configuring a wallet
+NDK provides an `ndk-wallet` package that makes it very simple to use a NIP-60 wallet.