Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: customize ipns dnsResolvers #445

Merged
merged 14 commits into from
Feb 26, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 39 additions & 4 deletions packages/verified-fetch/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,29 @@
* })
* ```
*
* ### Custom DNS resolvers
*
* If you don't want to leek DNS queries to the default resolvers, you can provide your own list of DNS resolvers to `createVerifiedFetch`.
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved
*
* Note that you do not need to provide both a DNS-over-HTTPS and a DNS-over-JSON resolver, and you should prefer `dnsJsonOverHttps` resolvers for usage in the browser for a smaller bundle size. See https://github.com/ipfs/helia/tree/main/packages/ipns#example---using-dns-json-over-https for more information.
*
* @example Customizing DNS resolvers
*
* ```typescript
* import { createVerifiedFetch } from '@helia/verified-fetch'
* import { dnsJsonOverHttps, dnsOverHttps } from '@helia/ipns/dns-resolvers'
*
* const fetch = await createVerifiedFetch({
* gateways: ['https://trustless-gateway.link'],
* routers: ['http://delegated-ipfs.dev']
* }, {
* dnsResolvers: [
* dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
* dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
* ]
* })
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another question here. Do we want to add dnsResolvers to the first object?

to stick to the pattern of "components in first object, options that alter how things work in the second" I think it would make sense in either place.. 🤔

Copy link
Member

@2color 2color Feb 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think dnsResolvers are deserving of the same importance as routers which makes the case to move it to the first object.

routers is currently used for IPNS routing (and CID routing in the future). Assuming that we see IPNS and DNSLink as being of equal importance dnsResolvers should live beside routers in my humble opinion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, customising dnsResolvers requires importing the constructors for dnsOverHttps and dnsJsonOverHttps which does reduce ergonomics.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'll still be optional either way, but I think I agree with you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe in the first argument, keeping the routers/gateways/resolvers together makes sense to me.

The only thing we have in the second arg is contentTypeParser, not 100% on why we have two args right now anyway!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at this for a bit and starting to change both args to a single object, but contentTypeParser has to be in the second obj, or we need to accept a possible .helia option in the first obj, or else users can't pass both a custom Helia & contentTypeParser

* ```
*
* ### IPLD codec handling
*
* IPFS supports several data formats (typically referred to as codecs) which are included in the CID. `@helia/verified-fetch` attempts to abstract away some of the details for easier consumption.
Expand Down Expand Up @@ -439,7 +462,7 @@ import { createHeliaHTTP } from '@helia/http'
import { delegatedHTTPRouting } from '@helia/routers'
import { VerifiedFetch as VerifiedFetchClass } from './verified-fetch.js'
import type { Helia } from '@helia/interface'
import type { IPNSRoutingEvents, ResolveDnsLinkProgressEvents, ResolveProgressEvents } from '@helia/ipns'
import type { DNSResolver, IPNSRoutingEvents, ResolveDnsLinkProgressEvents, ResolveProgressEvents } from '@helia/ipns'
import type { GetEvents } from '@helia/unixfs'
import type { CID } from 'multiformats/cid'
import type { ProgressEvent, ProgressOptions } from 'progress-events'
Expand Down Expand Up @@ -479,8 +502,22 @@ export interface CreateVerifiedFetchOptions {
* provide will be passed the first set of bytes we receive from the network,
* and should return a string that will be used as the value for the
* `Content-Type` header in the response.
*
* @default undefined
*/
contentTypeParser?: ContentTypeParser

/**
* In order to parse DNSLink records, we need to resolve DNS queries. You can
* pass a list of DNS resolvers that we will provide to the @helia/ipns
* instance for you. You must construct them using the `dnsJsonOverHttps` or
* `dnsOverHttps` functions exported from `@helia/ipns/dns-resolvers`.
*
* We use cloudflare and google's dnsJsonOverHttps resolvers by default.
*
* @default [dnsJsonOverHttps('https://mozilla.cloudflare-dns.com/dns-query'),dnsJsonOverHttps('https://dns.google/resolve')]
*/
dnsResolvers?: DNSResolver[]
}

/**
Expand Down Expand Up @@ -524,8 +561,6 @@ export interface VerifiedFetchInit extends RequestInit, ProgressOptions<BubbledP
* Create and return a Helia node
*/
export async function createVerifiedFetch (init?: Helia | CreateVerifiedFetchInit, options?: CreateVerifiedFetchOptions): Promise<VerifiedFetch> {
const contentTypeParser: ContentTypeParser | undefined = options?.contentTypeParser

if (!isHelia(init)) {
init = await createHeliaHTTP({
blockBrokers: [
Expand All @@ -537,7 +572,7 @@ export async function createVerifiedFetch (init?: Helia | CreateVerifiedFetchIni
})
}

const verifiedFetchInstance = new VerifiedFetchClass({ helia: init }, { contentTypeParser })
const verifiedFetchInstance = new VerifiedFetchClass({ helia: init }, options)
async function verifiedFetch (resource: Resource, options?: VerifiedFetchInit): Promise<Response> {
return verifiedFetchInstance.fetch(resource, options)
}
Expand Down
5 changes: 3 additions & 2 deletions packages/verified-fetch/src/verified-fetch.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { ipns as heliaIpns, type IPNS } from '@helia/ipns'
import { ipns as heliaIpns, type DNSResolver, type IPNS } from '@helia/ipns'
import { dnsJsonOverHttps } from '@helia/ipns/dns-resolvers'
import { unixfs as heliaUnixFs, type UnixFS as HeliaUnixFs, type UnixFSStats } from '@helia/unixfs'
import { code as dagCborCode } from '@ipld/dag-cbor'
Expand Down Expand Up @@ -32,6 +32,7 @@ interface VerifiedFetchComponents {
*/
interface VerifiedFetchInit {
contentTypeParser?: ContentTypeParser
dnsResolvers?: DNSResolver[]
}
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved

interface FetchHandlerFunctionArg {
Expand Down Expand Up @@ -86,7 +87,7 @@ export class VerifiedFetch {
this.helia = helia
this.log = helia.logger.forComponent('helia:verified-fetch')
this.ipns = ipns ?? heliaIpns(helia, {
resolvers: [
resolvers: init?.dnsResolvers ?? [
dnsJsonOverHttps('https://mozilla.cloudflare-dns.com/dns-query'),
dnsJsonOverHttps('https://dns.google/resolve')
]
Expand Down
20 changes: 20 additions & 0 deletions packages/verified-fetch/test/index.spec.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/* eslint-env mocha */
import { createHeliaHTTP } from '@helia/http'
import { dnsJsonOverHttps, dnsOverHttps } from '@helia/ipns/dns-resolvers'
import { expect } from 'aegir/chai'
import { createHelia } from 'helia'
import { createVerifiedFetch, verifiedFetch } from '../src/index.js'
Expand Down Expand Up @@ -51,4 +52,23 @@ describe('createVerifiedFetch', () => {
expect(verifiedFetch.stop).to.be.a('function')
expect(verifiedFetch.start).to.be.a('function')
})

it('can be passed a contentTypeParser', async () => {
const contentTypeParser = (_bytes: Uint8Array): string => 'application/json'
const verifiedFetch = await createVerifiedFetch(undefined, {
contentTypeParser
})
expect(verifiedFetch).to.be.ok()
await verifiedFetch.stop()
})
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved

it('can be passed custom dnsResolvers', async () => {
const dnsResolver = dnsOverHttps('https://example.com/dns-query')
const dnsJsonResolver = dnsJsonOverHttps('https://example.com/dns-json')
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved
const verifiedFetch = await createVerifiedFetch(undefined, {
dnsResolvers: [dnsResolver, dnsJsonResolver]
})
expect(verifiedFetch).to.be.ok()
await verifiedFetch.stop()
})
})
19 changes: 19 additions & 0 deletions packages/verified-fetch/test/verified-fetch.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -523,4 +523,23 @@ describe('@helia/verifed-fetch', () => {
await expect(resp.text()).to.eventually.equal('hello world')
})
})

describe('custom dns-resolvers', () => {
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved
it('uses custom dnsResolvers if provided', async () => {
const customDnsResolver = Sinon.stub()

customDnsResolver.returns(Promise.resolve('/ipfs/QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg'))

const verifiedFetch = new VerifiedFetch({
helia
}, {
dnsResolvers: [customDnsResolver]
})
// error of walking the CID/dag because we haven't actually added the block to the blockstore
await expect(verifiedFetch.fetch('ipns://some-non-cached-domain.com')).to.eventually.be.rejected.with.property('errors').that.has.lengthOf(0)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this aggregate error should probably have a length greater than zero 🤔

Looks like we're logging the error, but not returning them:

this.log.error('Error walking path %s', path, err)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, AggregateError has really poor DX as the default message logged to the console doesn't include any of the actual errors which can lead to spurious issues being reported and a lot of wasted time.

In other places where there's only one entry in .errors we unwind it to just throw that error instead.

In this case, if there's an error, it should be made available, otherwise no error should be thrown?


expect(customDnsResolver.callCount).to.equal(1)
expect(customDnsResolver.getCall(0).args).to.deep.equal(['some-non-cached-domain.com', { onProgress: undefined }])
})
})
})
Loading