Hosting Web Apps on IPFS

[cwaring]

Special Edition: IPFS Docs v2 featuring VuePress

Thanks to @hugomrdias and @lidel for chatting through all various scenarios yesterday, below are a few thoughts and next steps balanced against time/effort and overall impact.

The first (and perhaps most important) thing to note is that this problem is not isolated to VuePress, and is currently causing many common web applications with client-side routing and absolute paths to appear broken when loaded via a gateway URL, such as http://gateway.ipfs.io/ipfs/<cidv0>.

In order to make IPFS a first-class citizen for hosting web apps, the consensus is that we should move to subdomain model as soon as possible (e.g. https://<cidv1b32>.ipfs.dweb.link).

• This provides origin isolation for app security and session storage, while also fixing client-side routing and asset issues that people are facing with some of the most popular webpack application bundlers that currently rely on absolute paths (a few examples are Next, Gatsby, create-react-app, Nuxt, and VuePress).

• Additionally, this change will provide familiarity to new users attempting to host these type of web apps on IPFS — thus lowering the barrier to adoption and increasing our reach.

The subdomain pattern is a familiar one for developers. Most hosting services use a https://<subdomain>.<service>.<ext> pattern, and with the recent efforts/refactors across our projects we are very close to making this a reality for IPFS too. To that end, we believe we should invest our effort towards that direction instead of patching frameworks on a case-by-case basis.

With that said, here is a summary of our current options:

Option 1: Make IPFS Companion "redirect to gateway" opt-in and default to dweb links

---

[certainty: high, effort: medium, impact: high]

This option ensures that IPFS Companion will not automatically push users to load a resource via their local gateway node by default, leaving people with a broken experience when an app fails to load correctly. @lidel also suggests that we should move to an opt-in model, so that users have the option to explore sites via their local gateway instead of forcing this as default behaviour. Ultimately, an in-browser proxy would allow us to use cidv1 urls locally alongside dweb links; however, this won't be the case until further work has landed in the Chrome/Firefox vendor space.

For this to work successfully, the following tasks must be completed:

• Make ipfs-dns-deploy use cidv1b32 by default with a dweb preview link.
• Change default behaviour of IPFS Companion to disable auto redirects on DNSLink websites. PR in progress..
• Design a bootloader plugin for VuePress that detects the latest deployed CID and redirects to a cidv1b32 dweb link.

This does mean that a VuePress site will not run locally until we have a subdomain gateway in go-ipfs that we can use as a proxy in ipfs-desktop. However, we believe this is an acceptable tradeoff in the meantime, as this is a niche use case — particularly considering that the main app will provide PWA functionality with offline caching via service workers (enhancing the experience on all platforms, including mobile).

Related issues/efforts:

• CID in subdomain: CID as a Subdomain ipfs/in-web-browsers#89
• Hostnames for localhost HTTP gateway: Hostnames for localhost HTTP gateway ipfs/in-web-browsers#109
• Act as HTTP PROXY for http://.ipfs.localhost: Act as HTTP PROXY for http://<cidv1b32>.ipfs.localhost ipfs/kubo#5982
• Subdomain gateway: Subdomain Gateway ipfs/kubo#6498
• Disable default redirect of DNSLink websites if it breaks Origin: Disable default redirect of DNSLink websites if it breaks Origin ipfs/ipfs-companion#701

Option 2: Create a plugin to rewrite VuePress compiled output

---

[certainty: medium, effort: medium, impact: low]

As sketched out in this gist, we could build a post-process plugin that rewrites all the absolute paths to relative ones, making an app work regardless of path depth. However, this could have a number of unknown and potentially experience-breaking side effects such as interfering with interactivity, routing, failure of offline caching via PWA, and uncertainty around how plugins may respond.

This is approach would be similar to gatsby-plugin-ipfs and may be acceptable for simple sites — but post-processing compiled assets with regex is a fragile and messy solution. We believe this should only be done as an absolute last resort or a stopgap option.

Option 3: Fix the problem upstream in VuePress

---

[certainty: low, effort: high, impact: medium]

We could apply development effort to fix this issue upstream in VuePress. Such work would help resolve additional absolute-path-related issues for other users; however, it's strongly unlikely that we could solve the key issues, patch affected plugin dependencies, and land a merged PR in a timely fashion without unnecessarily delaying the launch of IPFS docs and losing valuable beta user testing time.

The effort here is complex, involves consensus with multiple third parties, and only adds value to a niche use-case. If absolutely necessary, we could investigate pursuing this in the new year, but our recommendation is to avoid postponing beta launch by pursuing one of the other options in the meantime.

Related issues:

• Build with Relative URLs: Build with Relative URLs vuejs/vuepress#796
• Publish source code for vuepress-html-webpack-plugin or unfork it: Publish source code for vuepress-html-webpack-plugin or unfork it vuejs/vuepress#1303

Option 4: Use GitHub Pages, Netlify or similar (stopgap)

---

[certainty: max, effort: zero, impact: 🤷‍♂️]

We are setup this way at the moment and it has enabled us to focus on delivering end user functionality in the docs beta until there was a solution in place to fully migrate to cidv1b32.

Note that Filecoin and IPLD docs are currently configured like this too.

Summary

We are keen to pursue option 1 for maximum overall impact, however I'd like to hear your thoughts. 👂✨

ref: #271

[terichadbourne]

I love the idea of pursuing the goal that helps the most real-world developers host sites on IPFS without having to alter the rest of their stack, including in the very common pattern of client-side routing.

Quick clarifying question on this aspect of Option 1, and apologies in advance because I know my vocabulary isn't up to asking it properly:

This option ensures that IPFS Companion will not automatically push users to load a resource via their local gateway node by default, leaving people with a broken experience when an app fails to load correctly. @lidel also suggests that we should move to an opt-in model, so that users have the option to explore sites via their local gateway instead of forcing this as default behaviour.

Is this about...

(A) letting people choose between accessing the site on "THE gateway" - the thing we run ourselves that is sometimes down - versus "THEIR gateway node" - which I assume is the equivalent of a local/offline cache, only available if they've visited and pinned the site before?

(B) giving people the option to view the site on the "normal internet" as a fallback if "the gateway" (access to IPFS) is down?

Kudos in advance to whoever can parse my question. :)

[olizilla]

how do you like these apples ipfs-shipyard/ipfs-dns-deploy#7

[jessicaschilling]

@olizilla I like these apples very much, they are delicious

[olizilla]

@terichadbourne Yes! it's about always loading the "normal internet" (aka http) version first, and informing the user that an ipfs flavour version of the site is available if they would rather fetch it that way. I assume it would also mean "fallback to ipfs" if http is unavailable.

[lidel]

Starting with IPFS Companion v2.9.0.872 user will load DNSLink website in "normal" way by default (over original HTTP server), but still get context actions including new one "View on Gateway" for opening via Gateway (public or local) in a new tab.

If docs.ipfs.io (or any other DNSLink website) is down, IPFS Companion will recover from Gateway in a new tab (public or local, if user runs their own), but by default we utilize original HTTP server (a public gateway) for the best UX.

To borrow illustration from ipfs/ipfs-companion#826 (comment):

[momack2]

I’m not sure I’ve fully processed option #1 - but my understanding is the suggestion means the vuepress site will still be broken when loaded over ipfs, but to effectively turn off loading over ipfs without explicit user interaction for most people browsing with companion/desktop so they’re less likely to experience this issue in our docs site. Is that an accurate summary? In that model, what would happen if I tried to reload the site from my local gateway (via the button you describe), cohost it using ipfs-cohost, share it with someone else by ipns or ipfs cid, or otherwise fetch the site over the ipfs network? Is there a different answer if I’m using an embedded js-ipfs node vs ipfs-desktop?

More broadly, I don’t think we want to default ipfs-powered sites to use http on first load. Can you explain more why you think that is necessary (@lidel)? Of note, I don’t think a performance reason should apply here - this feedback loop is actually extremely critical for users of companion (frequently developers dogfooding) to understand how long it takes to load from ipfs and we don’t want to remove that dogfooding option from our network use. The solution to that problem is making ipfs more performant, not relying on http for first load. No one is going to regularly use a drop down menu in an extension, even if we make some sort of alert that”this site is ipfs-powered!” - clicking that button would just reload the site slowly, making ipfs look even worse in comparison - so I don’t see that as a mollifying replacement. Most concerningly, this transfers a large number of requests from the actual ipfs network (right now resolved with js-ipfs in companion or go-ipfs in ipfs-desktop) to the ipfs http gateway - which is the opposite direction than we want to be heading and significantly reducing the frequency with which users would interface directly with the ipfs network.

I plan to run the “default-redirect in companion” decision by @jbenet for clear BDFL direction-setting on how companion, embedded nodes, and the gateway should fit together because this would be a large, noticeable, and and high-impact change that has other longer-term strategy implications. Will try to get an answer by Monday so we can fully decide between option 1 and option 2 (agree option 3 is nice, but not achievable on this timeline). Please add context to help clarify options for his review!

Edited: let's have a meeting on Monday CET to discuss the pros/cons of this change before surfacing to @jbenet (IPFS BDFL) for approval/ack.

[olizilla]

I've wanted this for a long time ipfs/ipfs-companion#710

[terichadbourne]

I too look forward to hearing a little more about the order of operations for Option 1. In my experience with standard Offline First patterns and PWAs, I've found that the best user experience is to load first from local cache if you have it (fastest possible load time and avoiding getting stuck in LieFi state but possibly outdated information), then immediately try to pull an updated version from the network. What counts as "the network" is the challenge here. 😂 (And I'm super excited to see the goal of making our docs accessible offline via a PWA! 🎉)

I'd also love clarity on what the Option 1 experience would be like if I visited the new docs site having never installed IPFS or IPFS Companion. We've pushed a lot of content that would normally be on a marketing site into the docs site, so I don't think we should assume that the only folks referencing it are IPFS aficionados.

Just a few thoughts from the vantage of someone who's a self-appointed advocate for newer users, among whom I see "(How) can I host my web site / web app on IPFS?" as one of the most frequently occurring questions...

By pure numbers, there are far more traditional web developers out there than there are DWeb/IPFS enthusiasts who already build on IPFS and have all of our tools installed, and I happen to be on the "more familiar with traditional web dev, new to IPFS" end of that spectrum. The audience of traditional but DWeb-curious web devs -- including those rallying around Offline First and PWAs -- is a huge potential market for adoption in my opinion, and one we should keep in mind as we make decisions. What would they do if faced with our current decision-making process, and how can we best serve them moving forward?

If a traditional web dev were already familiar with tools that met her needs in web app development (VuePress for a documentation site, tools that easily support client-side routing, traditional PWA stacks, etc.), and she bumped into a poor fit between IPFS and those trusted tools, which would win out... her curiosity to try out hosting on IPFS or her reliance on trusted tools of the trade? Recognizing the use cases for which IPFS isn't currently the ideal solution, and responding by making IPFS compatible with a range of existing tools to resolve that (but only when we consider that use case to be a key one for us) is key to increasing adoption and improving our user experience, and I really appreciate the careful thought being put into long-term solutions that will meet the needs of diverse users. ❤️

My concern with creating a plugin for Vuepress (Option 2) or solving things upstream in Vuepress (Option 3) is that both approaches solve a compatibility issue within a very limited scope (as I understand, in a way that won't also apply to the many other stacks with this client-side routing challenge), leaving the problems of hosting web apps on IPFS unaddressed overall. I would definitely lean towards finding solutions within IPFS itself where can have a greater and broader long-term impact. Unless we expect future users to individually create plugins that make IPFS compatible with specific widely-used tools for building web apps, the process of doing so ourselves strikes me as a bit contrary to the dogfooding concept of putting ourselves in our users' shoes. ¯_(ツ)_/¯

As a newer IPFS user from the traditional web dev community, I'm really excited about the possibilities offered by Option 1 as I understand them, including:

• resolving long-term issues with web app routing and paths to increase adoption
• offering great offline access to docs immediately
• collecting user feedback on the new site design as quickly as possible

Thanks to all of the less-new users who are taking the time to dig into all of these options so thoughtfully. You rock! 🎉

[momack2]

I think the proposed changes to companion call for a design review meeting. What we decide and spec out from that review should also get input from @jbenet to ensure we have longer-term roadmap alignment. Let's keep surfacing details and constraints on this thread, and synthesize into prioritized decision making criteria to refine during that meeting. I propose @lidel as ultimate decision-owner responsible for synthesizing input from key parties and making the final call. @lidel - can you propose a deadline by which we can surface and synthesize feedback & make a call on this decision?

I'll be in CET next week - can you schedule a design review call for ~Monday/Tuesday to unpack this more? Thanks!!

[lidel]

(I'll try to be brief 🙈 )

TL;DR

I believe the underlying idea behind Option 1 is to double down on our efforts to move towards subdomain gateways and Origin-based security to make it possible to use existing web tools like vuepress as-is, and stop patching other people's software, or requiring people to use custom build configuration when publishing on IPFS (hugo, gatsby, vuepress.. what will be next? we could be spending this energy elsewhere 🚀).

On changes in ipfs-companion v2.9.0.872

I don’t think we want to default ipfs-powered sites to use http on first load. Can you explain more why you think that is necessary (@lidel)?

A side effect of ipfs-companion change helps with docs site, but it was not the main motivation, only a signal to do the change sooner than later.

Context:
There is a known security issue of inconsistency around respecting Origin sandbox.
Details in ipfs/ipfs-companion#667: in short, we should not redirect if the act of doing so decreases isolation guarantees of original request.

The change means ipfs-companion should stop redirecting DNSLink websites until switch to CIDv1 is done and we have native support for subdomain gateway in go-ipfs that works with *.ipfs.localhost and *.ipns.localhost

Note: even without immediate redirect we could decide to prefetch data to IPFS node in the background, making it available in offline contexts.

this feedback loop is actually extremely critical for users of companion (frequently developers dogfooding) to understand how long it takes to load from ipfs and we don’t want to remove that dogfooding option from our network use.

I believe the change proposed in v2.9.0.872 will make performance eyeballing bit easier: user will see the difference between public gateway load and one produced by View on (local) Gateway.

For constant dogfooding, the old behavior can be restored via Preferences, but this is now a conscious decision that the user is okay with lifting Origin isolation guarantees from all DNSLink websites.

I believe this temporary change is an overall net gain for Companion users (not only improves security of the default state, but gives us some breathing room until DHT is improved).

On vuepress loaded from local gateway

vuepress site will still be broken when loaded over ipfs [..]

IIRC the idea for Option 1 is to run a small JS snippet to detect when page is loaded from /ip(f|n)s/.. (a path gateway) and redirect to a subdomain gateway instead.

Caveat: we don't expose subdomain gateway on localhost (yet), so for now we would redirect to public one (/ipfs/ to {cid}.ipfs.dweb.link and /ipns/docs.ipfs.io to docs.ipfs.io).

Design Review Call (Tuesday 4pm CET)

@lidel - can you propose a deadline by which we can surface and synthesize feedback & make a call on this decision?

@momack2 ack, I created a meeting in IPFS Community calendar on Tuesday 4PM CET to give everyone time to provide feedback in this thread and via other means before we synthesize during the call.

On my end the decision blocks releasing Companion to the Stable channel, but there are no critical fixes in it so it can wait: publishing to Chrome Web Store takes about a week anyway. Would be great to resolve it before Friday 13th tho 💀 🙈

[momack2]

Great, let’s make Thursday our official decision deadline then to allow a release on Friday. SGTY?

[jessicaschilling]

Thanks so much, @lidel, for all this context. I’ve unfortunately got an immovable dentist appointment during that time, but not sure how much value I have to add to the meeting that can’t already be provided by @cwaring, @johnnymatthews and @terichadbourne. Much appreciated!

[jessicaschilling]

PS - just confirming Tues, not Thurs, as that’s what I currently see on the calendar.
cc @momack2

[autonome]

I believe the underlying idea behind Option 1 is to double down on our
efforts to move towards subdomain gateways and Origin-based security

I want to emphasize that this particular bit is not optional for the project as a whole.

Bypassing the same-origin policy is a serious security concern, and IPFS HTTP gateways are configured with this problem out of the box. We've long known this, and as the various linked issues this thread show, we intend to fix it.

The CID-in-subdomain part of option 1 is required for our HTTP compatibility story, docs website decisions aside. This is an opportunity to accelerate getting there.

Most concerningly, this transfers a large number of requests from
the actual ipfs network (right now resolved with js-ipfs in companion
or go-ipfs in ipfs-desktop) to the ipfs http gateway - which is the
opposite direction than we want to be heading and significantly reducing
the frequency with which users would interface directly with the ipfs network.

This feels like the metrics tail wagging the vision dog (wow, apologies for that abomination of a sentence!). We should prioritize the design and implementation of a safe and functional IPFS in order to grow long term adoption over short term numbers.

IIRC also we've got ~25k users of Companion, based on the install numbers from Chrome/Firefox extension stores? That seems pretty small compared to overall gateway traffic figures. And of those Companion users, do we know how many have Desktop installed as well?

[daviddias]

I know there is a lot of people here are doing some deep thinking in the best solution to this challenge and I understand that there are new ideas to this old problem. I do want to share some notes from the dinner convo I had with @cwaring yesterday. I believe we yielded some great litmus tests that can aid to make our final decision. Take the following just as things to consider :)

1. We should be careful when we make design decisions that favor a a centralized architecture. The Web is in this state because in the past some really smart people thought that designing in some shortcuts and patches "to make things work" was a good idea and that they could come back later to fix it, however, things tend to stick for longer than we would hope for and worse, they tend to crystallize. Consider this carefully when designing a solution that required an Internet connection (e.g. Using DNS for name resolution, Routing to the IPFS Gateway or even expecting the user to install IPFS Companion & IPFS Desktop)

2. There is no IPFS Desktop & IPFS Companion on the mobile platform, the most widely adopted platform. If you create a solution that asks the user to load Websites only from a Desktop computer, than it will fail to the majority of the users.

3. One of IPFS super powers is that it is offline first and distributed first. This means that a resource should be equally loadable from a IPFS Gateway or from the neighbors machine when there is no Internet to connect to (and just some shared mesh). If the Website breaks when loading it on my local gateway (localhost:8080// or ipfs:// loading from my node), then it is not distributed.

4. Think of the PeerPad magic demo. PeerPad can be loaded by peerpad.net, ipfs.io/ipfs/PeerPadCID, localhost:8080/ipfs/PeerPadCID or even friendsIP:8080/ipfs/PeerPadCID and in all of these scenarios (if WebRTC or PubSub Relay is present), all of those users can be collaborating on the same pad in an authenticated and secure way, it is mind blowing, magical even!. Now with Signed Exchanges in Chrome(https://youtu.be/zYjsVe4s1JA), we can not only validate the integrity of an application, we can validate that it was published by the author we trust. This is huge. If our strategy for deployment breaks this, than we fail to deliver one of IPFS super powers.

[jessicaschilling]

Design decision has been made, and will be issue-ized and worked shortly.

[lidel]

I'll post meeting notes and summary of next steps on Companion end later today/tomorrow morning.

[lidel]

Thank you all for contributing to this conversation in text form and during the call ❤️

Meeting notes can be found at ipfs/team-mgmt#1081

Below is my understanding of decisions made.
I had to jump to another meeting, so please correct me if I got anything wrong 🙏

Decisions

In a spirit of disagree and commit, our path to move forward is:

• (1) Docs team: do what is necessary for docs.ipfs.io to work when loaded from local gateway (http://127.0.0.1:8080/ipns/docs.ipfs.io/)
  • solve relative path problem with a plugin
    or fix upstream vuepress and use a fork for now
• (2) Browsers/GUI: do not change the current behavior of ipfs-companion
  • continue redirecting DNSLink websites to local gateway by default (ipfs-companion/pull/831)
  • fix origin problem when subdomain gateway for local gateway lands (go-ipfs/issues/6498)

In the future, we will try to eat smaller slices of the decision cake 🍰

[momack2]

Thanks for the great preparation, summary, and ownership @lidel!! Appreciate all your great work and preparation making sure we came into a deeply technical and interwoven topic with the needed context and background! 👍❤️🎉

[jessicaschilling]

Closing this issue, since work is being executed in a variety of other locations. See #271 for work on hosting the docs site on IPFS.