Support Custom Protocols in WebExtension

[lidel]

(living summary: updated @ 2018-09)

This issue tracks browser extension support for various protocol schemes and URIs according to four stages of the upgrade path for path addressing and IPFS Addressing in Web Browsers memo, namely:

URLs:

ipfs://{cidv1b32} → http://127.0.0.1:8080/ipfs/{cidv1b32}
ipns://{hash} → http://127.0.0.1:8080/ipns/{hash}

URI:

dweb:/ip[f|n]s/{hash} → http://127.0.0.1:8080/ip[f|n]s/{hash}

This issue also tracks (currently nonexistent) ways to address/workaround how Origin is calculated (Problem #2)

WebExtension APIs

Universal API

No such thing yet (but see WIP work in comments)

• BUT! We use the most insane workaround in Support Custom Protocols in WebExtension #164 (comment). It works in both Firefox (Desktop-only – dweb:/ipfs/$cid not detected on Android #348) and Chrome and is the best thing we can do as of late 2017. It was merged with feat: Poor Man's Protocol Handlers #283.

Firefox

• Support for simplified redirect-based protocol handler (simple redirect, no origin support) landed in Firefox 54 (Bug #1310427).
  • Prefixed, basic web+fs protocol is supported as a redirect: Support Custom Protocols in WebExtension #164 (comment)
  • Removal of web+ prefix
    • a thread at Potential design problem with the protocol handlers API
    • Whitelist custom protocol handlers for work on decentralization technologies with WebExtensions
• More advanced API to implement programmable custom protocol handler in Firefox is currently discussed/designed:
  • Should provide a way to inject responses (Problem #1)

    • Rob Wu wrote up a draft concept of having a webRequest.respondWithcapability [but it went nowhere so far]
      – https://mail.mozilla.org/pipermail/dev-addons/2017-December/003466.html

  • Should provide control over how Origin is calculated (Problem #2)
  • Bugzilla #1271553 starting at comment #16 (REOPENED 🙃 )
    • I described our use case in comment #47
  • Experimental Protocol Handler API for WebExtensions is being designed as a part of mozilla/libdweb:

    The Protocol API allows you to handle custom protocols from your Firefox extension. This is different from the existing WebExtensions protocol handler API in that it does not register a website for handling corresponding URLs but rather allows your WebExtension to implement the handler.
    More: https://github.com/mozilla/libdweb/#protocol-api

    • Still experimental, requires a special build, but the goal is for this to land in regular builds of Firefox Nightly at some point in the future ✨
    • We are working on a PoC handler that loads data over raw IPFS and keeps ipfs:// in address bar.
      It can be found in libdweb branch. More info in PR [PoC] libdweb experiment: protocol handler API #533 and /libdweb/docs/libdweb.md

Chrome / Chromium

• no API for this yet: Chrome 67 requires web+ prefix for non-safelisted protocols
• Chrome published intent-to-implement support for registering HTTP-based handlers for DWeb protocols without web+ prefix (context: Safelisting DWeb Protocols arewedistributedyet/arewedistributedyet#23)
• @lidel filled Extensions API should implement manifest.json/protocol_handlers for feature-parity with Firefox

Brave

Muon-based (deprecated in 2018)

• browser.protocol.registerStringProtocol available to trusted extensions
• missing api for Buffer/Stream-based protocols (Muon-based Brave Browser Integrations  #312 (comment))

Chromium-based

• Basically same challenges as Chromium right now

Related discussions

• The four stages of the upgrade path for path addressing: Background on Address Scheme discussions (dweb, ipfs:// and more) specs#152 (comment)
• Old discussions about browser Origins: Tackle identifying origins with (or without?) dweb: paths in-web-browsers#6
  • Suborigins
  • Subdomains

[lidel]

Support for simplified redirect-based protocol handler (simple redirect, no origin support) landed in Firefox 54:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1310427

This a a thin UX on top of Navigator.registerProtocolHandler():

• https://developer.mozilla.org/en-US/docs/Web/API/Navigator/registerProtocolHandler

• https://blog.mozilla.org/addons/2017/03/13/webextensions-firefox-54/:

  The protocol_handler API lets you easily map protocols to actions in your extension. For example: we use irccloud at Mozilla, so we can map ircs:// links to irccloud by adding this into an extension:

   "protocol_handlers": [
     {
       "protocol": "ircs",
       "name": "IRC Mozilla Extension",
       "uriTemplate": "https://irccloud.mozilla.com/#!/%s"
     }
   ]
  

  When a user clicks on an IRC link, it shows the application selector with the IRC Mozilla Extension visible:
  

Note that this is just a redirect:

This means the address in location bar changes (Origin barrier is broken) and there are different security context for public and local gateways (different cookies, etc).

And yes, this is Firefox-only.

[lidel]

It seems that this new API has the same naming limitation as Chrome, firefox-54.0a2 says:

Reading manifest: Error processing protocol_handlers.0.protocol: Value must either: be one of ["bitcoin", "geo", "im", "irc", "ircs", "magnet", "mailto", "mms", "news", "nntp", "sip", "sms", "smsto", "ssh", "tel", "urn", "webcal", "wtai", "xmpp"], or match the pattern /^(ext|web)+[a-z0-9.+-]+$/

This means we can't support ipfs://hash anymore and are forced to use web+ipfs://hash 👎

[Kubuxu]

When we get official registration of the schema we should be able to request the exemption to be added.

[lidel]

I've just tested a ridiculous PoC that restores support for ipfs:// dweb: etc in WebExtension:

• We know that:
  • permission <all_urls> fires onBeforeRequest for every URL
  • when URL starting with an unknown protocol is entered in location bar, a browser url-escapes entire address and converts it to a search query, e.g:
    ipfs://QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR → https://www.google.com/search?q=ipfs%3A%2F%2FQmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR&ie=utf-8&oe=utf-8&client=firefox-b-ab
• So what happens if you:
  • detect requests with URL containing url-encoded :/ ( %3A%2F)
  • find ones that match a query starting with one of custom protocols: /=(ipfs|ipns|dweb)%3A%2F(%2F[^&]+)/
  • extract IPFS path and check if passes IsIpfs.path test
    • if so, replace request to search engine with request to resource at public gateway 'https://ipfs.io/' + extractedIpfsPath
    • if not, return original request

Well.. it just works 🚀 🙃 (not sure if I am proud or ashamed, probably both)

Expect PR in near future. My plan is to polish and ship this workaround as the default behavior, so that there is no functional regression when users update from v1.6.0 to v2.x.x.

Of course there will be a preference that disables this: to pass review at Mozilla and in case someone really wants to search for ipfs://QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR

Update 2017-12-27: this hack does not work on Android – see #348

[lidel]

FYSA some recent developments (Q3 2018):

• Experimental Protocol Handler API for WebExtensions is being designed as a part of mozilla/libdweb:

  The Protocol API allows you to handle custom protocols from your Firefox extension. This is different from the existing WebExtensions protocol handler API in that it does not register a website for handling corresponding URLs but rather allows your WebExtension to implement the handler.
  More: https://github.com/mozilla/libdweb/#protocol-api

  • Still experimental, requires a special build, but the goal is for this to land in regular builds of Firefox Nightly at some point in the future ✨

  • We are working on a PoC handler that loads data over raw IPFS and keeps ipfs:// in address bar. It can be found in libdweb branch. More info in PR [PoC] libdweb experiment: protocol handler API #533 and /libdweb/docs/libdweb.md

• Safelisting DWeb Protocols | Safelisting DWeb Protocols arewedistributedyet/arewedistributedyet#23

  • tl;dr The goal is to be able to register it in all browsers without web+ prefix.
    (Chrome 67 requires web+ prefix for non-safelisted protocols)
  • Firefox already allows non-prefixed version, Chrome published intent-to-implement.

[lidel]

If DWeb protocols get safelisted, we need manifest.json/protocol_handlers feature parity to solve UX issues mentioned in arewedistributedyet/arewedistributedyet#23 (comment)

Filled a Chrome bug: Extensions API should implement manifest.json/protocol_handlers.

[ingokeck]

According to the URI syntax in RFC3986, it should not be "ipfs://{cidv1b32}", instead it should be "ipfs:{cidv1b32}". Same for {hash}. The extention should then use its default IPFS instance to retrieve the object {cidv1b32}. See https://tools.ietf.org/html/rfc3986#section-3

'//' denotes a specific instance. So an IPFS service on localhost could be addressed with "ipfs://localhost/{cidv1b32}", while IPFS running on ipfs.io would be "ipfs://ipfs.io/{cidv1b32}"

[lidel]

Those are good points, thanks for raising them @ingokeck
We would love to be as compliant with RFCs as possible, as long we don't compromise Origin isolation (details below), so this needs deeper analysis.

• 💚 For the sake of UX, I believe both should work. For example, you can use ipfs:bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi in Firefox today and if you have IPFS Companion installed, it will resolve just fine. If we flip the default, the reverse should be true as well.

• 💛 Removal of // needs discussion with Brave, Opera, and other community members, but I suspect the only reason for // is technical debt in web browsers and being pragmatic. Browsers are HTTP-centric and existing implementation leverage this property of RFC3986:

  Non-validating parsers (those that merely separate a URI reference into
  its major components) will often ignore the subcomponent structure of
  authority, treating it as an opaque string from the double-slash to
  the first terminating delimiter, until such time as the URI is dereferenced.

  For example Brave v1.19 uses CID as "authority" component and build Origin based on it. This provides us with security sandbox per content root (CID), but has a side-effect of // in the address bar.

  TLDR Origin isolation is way more important than being visually compliant with RFC – // can be removed only if Origin isolation per CID is maintained, which I don't believe is possible atm.

• 💔 ipfs://localhost/{cidv1b32} is incompatible with our security model given how addressing is implemented in browsers

  • Each CID needs to create own Origin, and this will put all CIDs under single Origin, so a big NO.
  • Yes, this could be special-cased, but is very unlikely any vendor will want to touch critical code paths for the sake of cosmetics. In the past we looked into something simpler with Suborigins and even that did not go anywhere: Suborigins  in-web-browsers#66

[lidel]

5 years after Firefox, Google Chrome is now willing to implement the protocol_handler for registering custom handlers via web extensions: w3c/webextensions#317 (comment)

[lidel]

Igalia is working on Chromium improvement to allow redirect-based handler registration with protocol_handlers in browser extension manifest. Completed the design document (The protocol_handlers Web Extension's Manifest key) with the permissions management and conflict resolutions, and shared the design document with Google for feedback.

Assuming that eventually lands in Chromium-based browsers, the next step will be to allow using the API for pointing protocol scheme at Service Worker, instead of URL. This will allows for keeping ipfs://cid in the address bar, and use it for Origin-based logic natively.

This will take even more time, for now the idea is tracked in:

• ServiceWorker-like protocol handlers for WebExtensions in-web-browsers#212