[UC] DECOUPLING between API PROTOCOL(s) and INTEROP specs

[niko-ng]

Apologies for not respecting the format of a "Use-Case" that is apparently mandated.

As some of you may know, I am working on local-first and E2EE RDF since more than 3 years, with the project NextGraph.org that I presented to the Solid Community in 3 instalments recently.
We then created a group specialised on CRDT4RDF https://lists.w3.org/Archives/Public/public-crdt4rdf/
and a Matrix one to talk about E2EE in Solid: https://matrix.to/#/#solid-e2ee:matrix.org

NextGraph in itself is not compatible with Solid, not because it doesn't want to, but because it cannot, in the current state of the specs. And I will explain here why.

Nevertheless, and thanks to our collaboration with ActivityPods.org, a federated server based on ActivityPub and Solid, we can bridge the gap and offer Solid endpoints to our users and developers. And reversely, ActivityPods' users will be able to benefit from local-first (CRDT) and end-to-end encryption somehow.

In fact, NextGraph, based on RDF and the Semantic Web, and very much inspired by the Giant Global Graph vision of Tim Berners-Lee, is another implementation of this "Web of Data" that we all want, together with the Solid Project, ActivityPods, and several other solutions.

But NextGraph is local-first and needs to offer APIs locally (in the "client-side" if you will), that will never connect to any server, because in NextGraph, there is no server, at least not in the traditional sense.

That's a big difference compared to Solid, that is based on a client-server architecture, and that mandates the use of the HTTP protocol.

The aim of our collaboration with ActivityPods, is to offer client-server HTTP endpoints on top of the P2P architecture of NextGraph. This will work well for programmers who want to use the Solid APIs, but they will have to install a server and manage it, while this requirement would not exist if the purely local APIs of NextGraph had been used instead.

Our APIs are very similar to what Solid tries to do: users and developers can read and write Resources (RDF Documents) and also upload and download binary files. They can change permissions and share this data with other users, and more importantly, with some applications, by following the SAI mechanism.

But in NextGraph, APIs are Javascript-based, because they are available locally only (they can also be accessed in Rust, and soon in Dart, Swift, and more languages used in the front-end).

The proposal of this "Use-Case" is about bringing the APIs of Solid compatible with NextGraph, or with any other implementation that would not be based on HTTP or on a client-server architecture, with local-first and E2EE as the direct purpose.

The concepts and ideas embodied by the Solid Specifications, are larger and more generic than the HTTP protocol.

The decentralized Web of Data, Giant Global Graph, and the lofty goals we all have in common, cannot be restricted to be dependent on one transport protocol only, specially today, with the diversity of technical solutions available to us.

In the context of local-first and/or end-to-end encryption, we believe that the Solid APIs should be abstracted, so they can then be implemented in various ways, be it with HTTP, WebSocket, binary protocols, or plain Javascript function calls.

All the amazing work of specification and research that has been done by the Solid Community along the years, is very important and could benefit to a larger audience, if those restrictions were to be lifted.

At NextGraph, we strongly believe in RDF and the Semantic Web, as you do, and would love to be able to participate or influence the upcoming writing of the final Solid specifications, as undertaken by the newly-formed W3C Linked Web Storage working group.

This proposal, named "DECOUPLING between API PROTOCOL(s) and INTEROP specs" is about identifying in all the specs of Solid, which parts concern the API definitions, and which parts concern the data and meta-data formats exchanged and/or stored by the POD provider.

In our proposal, the API parts (the exchange protocols) should be abstracted so that implementers can choose how exactly they will offer those APIs; while the rest of the specs (about Interop, data, meta data, etc...) can stay untouched.

To say it simply: everything related to HTTP should be abstracted, while anything related to RDF can remain as it is (but without a requirement for URLs to be HTTP based).

The abstraction that we propose could be of the following form: For each API endpoint, the specification would detail the type of API, and its semantics (accepted parameters, returned value, etc...). We could distinguish 3 main types of APIs :

• RPC (Request with payload/argument, followed by one Response: error, OK or content)

• Stream (one Request + Many responses coming over time + option to cancel. usually used in Pub/Sub mechanisms)

• Inbox (Sends one message. don't expect any response, or maybe async response later, coming back thru sender's Inbox)

Which syntax exactly to use in order to describe those APIs, is left unanswered for now.

Then another specification could detail how an implementation based on HTTP would be done (you just have to copy paste all your existing HTTP specifications for that matter). That specification would not be local-first (and probably not E2EE neither) but that's OK, as not everybody needs that.

And I would be more than happy to work with you all on formalising an "implementation specification" for local-first and E2EE encryption based on DIDs like the one designed in NextGraph.

Other groups could work on other implementation specifications as well.

And eventually, implementers would just pick which "implementation specs" they want to follow, while always remaining compatible with the "abstraction specs" anyway.

I noticed that the Solid Notifications spec is already giving some freedom about which exact protocol to use to implement the notification mechanism: WebSocket, WebHooks, etc... So the abstraction I am talking about would be this sort of freedom, generalized to all APIs in Solid.

If that was possible to achieve in the course of the W3C LWS WG undertaking, I believe that Solid would truly become a universal specifications with a much higher potential for adoption.

NextGraph would then be one of the first implementation available that offers local-first and E2EE out of the box.

A bit more details on local-first and E2EE.

I have seen at least 2 attempts at implementing those features, on top of a Solid POD provider, namely, with Noel de Martin proposal called Aerogel for CRDTs, and from the Australian National University team building E2EE on top of a normal POD provider.

E2EE is related to local-first, in the sense that once the RDF data is encrypted end-to-end, its content is only available in the client-side, and the server (POD provider) cannot operate on it at all anymore. In that scenario, the POD provider is just used as a relay of encrypted blobs.

In local-first and specially when CRDT are used (which is a must if you want to avoid merge conflicts), the same kind of problematic situation emerges: the Solid POD is not dealing with normal RDF resources anymore, but with an "append only log of operations" that it won't understand. In order to obtain the effective RDF resource, it is the client-side that will need to fetch the full log of operations, and materialize the current state of the RDF resource, locally. Hence, that RDF resource will only be available in the client-side, and the server will be deprived of all its features, and will only behave as a relay of "log of operations".

Then you can combine the 2 approaches and the POD provider server only deals with an "encrypted operation log" that it cannot decrypt nor understand.

That's not what the POD provider was intended to do. And I find this "twisting and stretching" of the POD provider usage, to be in fact an abuse of its functionalities, that is detrimental to the whole ecosystem.

My understanding is that using the POD provider in this unorthodox way, the data it holds is "polluted" and the features it was supposed to offer, become unavailable. All the logic of dealing with RDF data is then transferred on the client-side only, that in addition will also need to deal with low level mechanisms related to CRDT and E2EE (materializing the state, and encrypting/decrypting). Something that Solid clients are not meant to deal with.

I believe the engine implementing the user-facing APIs should already be dealing with all that, and the user/dev should only have to deal with normal/clean RDF resources, even when CRDT and E2EE are involved. This is why in NextGraph, all the metadata and processing related to E2EE and CRDT are hidden from the end-user, and stand much lower in the stack.

Using a POD provider just as a "sync server", is twisting and stretching the real purpose of a POD provider, that should normally be able to see all the RDF resources and act on them (enforce complex access control, sharing with others, search and query, etc...).

NextGraph offers those POD features in the client-side, because it implements a local POD provider locally, notably by following the SAI specification and offering a SPARQL endpoint locally, as well as filter, pagination and search, among other services. But those services aren't possibly following the Solid specs, because they are written in Javascript, while Solid specs mandate HTTP...

The internals on how the "append only log of operations" is handled, or on how the encryption happens, are not of interest to the end-user and app developers, who should both enjoy, from their point of view, a clean API about regular RDF resources, offered to them locally, in javascript.

Ideally, how the data is encrypted or synchronized between clients, should be out of the scope of the "abstract Solid specification", and should only be detailed in the "DID implementation specification" that NextGraph is an implementation of, in Rust.

NextGraph by example, has its own binary protocol for all the sync protocol and E2EE.

If I want to make a simple analogy, when someone uses a filesystem, they do not know all the details about the implementation, how exactly the binary data is stored, if it is encrypted at rest or not, nor how it is transferred between machines (in the case of NFS by example). The POSIX Filesystem API is abstract enough. Many implementation exist. Still, the FS API can be used to build compatible applications on top of it, without caring about the implementation details.

With this analogy, RDF, SAI and other Solid specifications related to Interop and Access control are forming the abstract APIs, and the implementations can be based on HTTP, DIDs, NextGraph binary protocol, or anything else too, depending on the needs.

App developers would use only the abstract APIs, and would not care about the underlying implementation details.

Eventually, I believe that this proposed change in the Solid specifications would truly bring about features like local-first or E2EE to Solid, and by building upon the extensive specification work already achieved by so many researchers along the years, would usher in a higher level of abstraction, and make the Solid specifications truly universal and future proof.

Moved by the common goal of advancing our dream of a decentralized Web of Data where users regain control over their privacy and digital lives, and in a spirit of collaboration, I remain available for any questions and clarifications on this proposal.

Niko
niko@nextgraph.org

decoupling SOLID.pdf

[hzbarcea]

Thanks @niko-ng for creating this. It will probably take me a bit to untangle, it looks like there are a few stories here. We touched on this in the CG meetings and what I think this means is that you're advocating for a data-format part of the spec, which you refer to as interop. But it may be more than that. I assume we'll have a longer conversation here. Thanks again!

[elf-pavlik]

This is excellent input for the conversation about design principles. I think it should still be possible to write down a few use cases that show what this approach could enable.
For E2EE, it would be helpful to write down a scenario or two for when individuals and groups want their data to be E2EE. For someone who has been working on that for years, it might seem like an obvious question to answer but for the general audience, it will be helpful to illustrate few scenarios where such requirements come up.
Similarly, there must be a few common scenarios that can illustrate the need for CRDTs. There is some offline-first related work that was happening at W3C, for example, https://wicg.github.io/background-sync/spec/
Those use cases would allow us to demonstrate how the system can satisfy them using E2EE and/or CRDT. At the same time, it will enable everyone to propose an alternative approach using different techniques to meet the requirements of those use cases. Last but not least, it could showcase why the current inputs from Solid CG can't support those use cases and which specific requirements aren't satisfied.
At the same time, I anticipate some use cases, for example, related to search and indexing, where E2EE and CRDT shouldn't be used, but there still needs to be a conformant implementation method.

[uvdsl]

There is a lot to digest, and I am not sure if I get everything straight. So let me ask some clarifying questions:

The proposal of this "Use-Case" is about bringing the APIs of Solid compatible with NextGraph, or with any other implementation that would not be based on HTTP or on a client-server architecture.

What kind of "other implementation" are you referring to? Implementations of what?
The use case could be renamed to "Make Solid compatible with Nextgraph" then?

The concepts and ideas embodied by the Solid Specifications, are larger and more generic than the HTTP protocol.

The Solid Protocol seems to be what survived the community discussions (over many years) as the lowest common denominator regarding the overarching vision of Solid.

This proposal, named "DECOUPLING between API PROTOCOL(s) and INTEROP specs" is about identifying in all the specs of Solid, which parts concern the API definitions, and which parts concern the data and meta-data formats exchanged and/or stored by the POD provider.

I am not sure what you refer to using the terms "API protocol(s)" and "INTEROP specs".
What is an "API protocol"? Are you referring to transport protocols like HTTP, Websockets, SMTP or something else?
What are "INTEROP specs"? Are you referring to the Solid Application Interoperability specification?

In our proposal, the API parts (the exchange protocols) should be abstracted so that implementers can choose how exactly they will offer those APIs; while the rest of the specs (about Interop, data, meta data, etc...) can stay untouched.

I am not sure if I grasp your idea: "implementers can choose how exactly they will offer those APIs".
Do you want to design an API that is agnostic to the transport protocol?
Considering you mentioned offering a Javascript/Rust/anyOtherProgrammingLanguage API, why is it not possible to offer such a language-specific API around the HTTP-based API of the current Solid Protocol? Even locally, I can interact with a Solid Pod hosted on my machine using a HTTP library in the language of my choice (or just cURL).

To say it simply: everything related to HTTP should be abstracted, while anything related to RDF can remain as it is (but without a requirement for URLs to be HTTP based).

You suggest to not follow the 2nd Linked Data principle, correct?

And eventually, implementers would just pick which "implementation specs" they want to follow, while always remaining compatible with the "abstraction specs" anyway.

This sounds familiar to what the International Dataspaces (IDS) protocol does with its "bindings".
They already design an abstract data transport protocol, where the actual data transport is left open to implementation (binding).
Why does their protocol not satisfy your needs?
What is missing?

---

In a meta discussion, I want to highlight particular wording in the issue description (as of now):

At NextGraph, we strongly believe in RDF and the Semantic Web, as you do, [...]
But NextGraph is local-first [...]
But in NextGraph, APIs are [...]
The proposal of this "Use-Case" is about bringing the APIs of Solid compatible with NextGraph [...]
[...] all your existing HTTP specifications [...]

While I appreciate the description of what is desired (and not yet supported) by existing specifications, I can't help but receive the subtle message that Nextgraph is a product (and not a use case) and that this WG should (please) align the (existing) specifications on the WG's side with that particular product.

I try to stay clear of that impression for the technical discussion, and would like to suggest that for discussions around local-first and end-to-end encryption a corresponding issue is created.
For these two topics, I can imagine use cases could be extracted from this current issue, or could further be provided by the NextGraph team.

[acoburn]

@niko-ng thank you for creating this ticket. As mentioned already, there is a lot of valuable information here.

Without commenting on the specific details described in this ticket, I want to highlight that the LWS working group has a well-defined scope, as outlined in our charter document. The first sentence is reiterated here:

The scope of the Working Group is to define a web protocol that applications can use to authenticate users and store their data with user-chosen identity and storage servers.

We will need to make sure that all of the cases submitted here fit within that scope. We already have an ambitious timeline, and we have to be careful not to add scope to the group.

[niko-ng]

Thanks for reading my proposal!

The proposal of this "Use-Case" is about bringing the APIs of Solid compatible with NextGraph, or with any other implementation that would not be based on HTTP or on a client-server architecture.

What kind of "other implementation" are you referring to? Implementations of what? The use case could be renamed to "Make Solid compatible with Nextgraph" then?

I am referring to other implementations of the Solid specifications (CSS, ESS, ActivityPods, etc.., NextGraph maybe one day, if the specs allow it).
This use case is not strictly about NextGraph. It is only meant to free the specs from the HTTP and client-server requirement. This will help NextGraph to be compliant with Solid, but it can help other implementations too.

The concepts and ideas embodied by the Solid Specifications, are larger and more generic than the HTTP protocol.

The Solid Protocol seems to be what survived the community discussions (over many years) as the lowest common denominator regarding the overarching vision of Solid.

and it is based on HTTP. agreed. Still I believe what Solid wants to achieve, could be done without HTTP (I already prove that with NextGraph).

This proposal, named "DECOUPLING between API PROTOCOL(s) and INTEROP specs" is about identifying in all the specs of Solid, which parts concern the API definitions, and which parts concern the data and meta-data formats exchanged and/or stored by the POD provider.

I am not sure what you refer to using the terms "API protocol(s)" and "INTEROP specs". What is an "API protocol"? Are you referring to transport protocols like HTTP, Websockets, SMTP or something else? What are "INTEROP specs"? Are you referring to the Solid Application Interoperability specification?

You are right that the terms I use are ambiguous. And it all boils down to mutual understanding of the terms, right.
The whole issue tried to elaborate on one concept, maybe unsuccessfully.
Yes the "INTEROP specs" include SAI, but only the parts in SAI that relate to defining ontologies and schemas (about AccessGrant etc...). To the contrary, the rest of SAI belongs to the "API part", by example when it says here "To discover a corresponding Agent Registration the requesting Agent may perform an HTTP HEAD or HTTP GET request on the IRI of the Authorization Agent for the target Social Agent." here it refers to HTTP HEAD and HTTP GET that are the things I would like to remove and replace with a more abstract terminology, by example:

"To discover a corresponding Agent Registration the requesting Agent may perform an RPC call named "AgentRegistrationDiscover" containing one parameter called "AuthorizationAgent" of type string with value the IRI of such Authorization Agent, and the expected reply is a triple relating the AgentRegistration to the Agent making the request, via the registeredAgent predicate". Or something similar. Then there will be an "HTTP implementation specification of SAI" that would detail how to implement the RPC AgentRegistrationDiscover with HTTP, and it would look like what we currently see in the spec at this chapter 7.1.4
Then the DID:NG implementation specs would detail how they implement such RPC.
And finally, the JS library implementer for DID would document that the "abstract RPC call AgentRegistrationDiscover" is implemented in the library as sai.agentRegistrationDiscover(authorizationAgent: string): Triple or similar.

The "INTEROP parts" remain the same, namely all the ontology describing the metadata of AccessGrants etc...
Only the "API parts" would be abstracted.

And this can be done for any Solid spec, I believe (like LDP, WAC, ACP, Solid Notifications, etc)

In our proposal, the API parts (the exchange protocols) should be abstracted so that implementers can choose how exactly they will offer those APIs; while the rest of the specs (about Interop, data, meta data, etc...) can stay untouched.

I am not sure if I grasp your idea: "implementers can choose how exactly they will offer those APIs".
Do you want to design an API that is agnostic to the transport protocol?
yes! because in a local-first setting, the API might not even need to transport anything because everything is handled locally.
Considering you mentioned offering a Javascript/Rust/anyOtherProgrammingLanguage API, why is it not possible to offer such a language-specific API around the HTTP-based API of the current Solid Protocol? Even locally, I can interact with a Solid Pod hosted on my machine using a HTTP library in the language of my choice (or just cURL).

Your proposal already exists. there are JS libraries that help connect and interact with a POD server, and that implement the HTTP specs.
What I am trying to convey is the opposite idea. The new specs would not mandate anything about the protocol and its transport. And API implementations, in JS or other languages, might use HTTP or any other transport protocol, or might not use any transport at all, depending on how they implement the API.

To say it simply: everything related to HTTP should be abstracted, while anything related to RDF can remain as it is (but without a requirement for URLs to be HTTP based).

You suggest to not follow the 2nd Linked Data principle, correct?

Exactly ! the two first principles could be reformulated as

1. Use URIs as identifiers for things
2. Use dereferenceable URIs so that people can look up those identifiers.

And eventually, implementers would just pick which "implementation specs" they want to follow, while always remaining compatible with the "abstraction specs" anyway.

This sounds familiar to what the International Dataspaces (IDS) protocol does with its "bindings". They already design an abstract data transport protocol, where the actual data transport is left open to implementation (binding). Why does their protocol not satisfy your needs? What is missing?

Thanks for relating the concept of "abstraction" to the one of "binding" in IDSA.
But my proposal is not regarding "transport protocol" only. Sorry if the wording I used was not precise enough.
I am mainly talking about the need for abstract APIs.
And the concept that need abstraction relates more to a "service" than to a "transfer protocol".
RPC is the closest analogy I could find in order to abstract what is now an "HTTP request" in the specs.
But other kind of services exist too, appart from RPC: namely Inbox and Stream.
None of them imply that a transport is needed, or that a protocol is needed neither.
You don't know what will happen under the hood in the Javascript function implementation.
By example in NextGraph, there will most probably be no transport, no connection, no protocol involved whatsoever, because most of the API calls can be answered by the system locally. Only when the system detects that something is missing, will it use a protocol and a TCP connection to a remote server. but that part should be totally out of the scope of the "abstract spec".

In a meta discussion, I want to highlight particular wording in the issue description (as of now):

At NextGraph, we strongly believe in RDF and the Semantic Web, as you do, [...]
But NextGraph is local-first [...]
But in NextGraph, APIs are [...]
The proposal of this "Use-Case" is about bringing the APIs of Solid compatible with NextGraph [...]
[...] all your existing HTTP specifications [...]

While I appreciate the description of what is desired (and not yet supported) by existing specifications, I can't help but receive the subtle message that Nextgraph is a product (and not a use case) and that this WG should (please) align the (existing) specifications on the WG's side with that particular product.

I try to stay clear of that impression for the technical discussion, and would like to suggest that for discussions around local-first and end-to-end encryption a corresponding issue is created. For these two topics, I can imagine use cases could be extracted from this current issue, or could further be provided by the NextGraph team.

Thanks for pointing out that the wording is not the best. I tried to convey an idea, I am happy it generates a discussion.

NextGraph is not a product, it isn't even an implementation of Solid.
NextGraph is an open source project that, for what concerns us here, offers a local SPARQL endpoint (via some JS functions) with E2EE and CRDT features backed by a local quadstore (also encrypted at rest). Under the hood, there is a sync protocol that can, also, be useful when some of the data is not available locally and needs to be retrieved, or when live collaboration happens, by example.

How we arrived to this UC issue is a bit explained in the introduction. NextGraph has been invited to present its work in the plenary meeting of CG few weeks ago, and then also in 2 Special meetings related to E2EE and CRDT, separately.
A Community Group has then be created under the name CRDT4RDF (which in the name, mentions neither the NextGraph "product" nor the Solid "product").
Then a Matrix group was established about E2EE in Solid, to support discussion with the Australian community working on the same.

NextGraph does not come forward here with the aim of "placing a product".
I used NextGraph as an example implementation of the desired abstract specification.
NextGraph does not ask for the LWS WG to align with its own work, that would be quite pretentious.
With our proposal, we just see an opportunity for Solid, to be open to local first environments, which from our experience, it isn't for now, due to the strong requirement in the specs on HTTP which is a client-server protocol (client-server and local-first are mutually exclusive. NextGraph did not make this up. As a consequence, NextGraph cannot be compliant with Solid)
As a matter of facts, NextGraph does not need to be compliant with Solid.
But that prospect would be interesting.
And more generally, I have heard from the Solid community members, that local-first is a strongly desired feature, and I felt that it was important to share with you all my insights on how it would be difficult to be local-first while keeping the HTTP requirement.

Thank you for your efforts at reading the issue I imperfectly wrote and trying to make sense out of it.
This issue is related very much to local-first (and E2EE indirectly), I think you got my point.

I could indeed also formulate separate issues about use cases on E2EE and local-first, as elf Pavlik suggested. Anybody else can also do that and we can discuss there was use cases really exist.

The present use case is more generic, but is a precondition, I believe for enabling local-first or E2EE features in Solid.

[niko-ng]

Hello @acoburn

The scope of the Working Group is to define a web protocol that applications can use to authenticate users and store their data with user-chosen identity and storage servers.

You are right to point out that the scope mentions both a protocol and servers, I didn't check it beforehand.

In that regard, you could discard the whole issue I wrote.

I believe that the Solid Project, envisioned by Tim and carried upon the shoulders of many engineers along the years, has evolved too, but its essential purpose is very much aligned with what NextGraph is also busy with. I let you pounder on that.

I wouldn't let the words "protocol" and "servers" come in the way of our fruitful exchanges.
We are moved by very similar goals.
Servers are everywhere, they might not be the entity the "user" of your specs will face directly. The users of Solid specs are not necessarily HTTP clients. That's my point. If you truly want local-first, you will have to integrate this concept, even maybe by changing the scope of the WG.
Protocols are conventions. Are they about transport of data between two computers connected by a network?
Are they more abstract? You know well the definition of the term is open.

Maybe using the term "storage providers" would have been more open and future proof.

I hope the use-case I formulated can somehow contribute to the success of the LWS WG goals, and of the Solid Project in general!

[niko-ng]

Also @uvdsl, regarding the Design of LinkedData from 2006,

You suggest to not follow the 2nd Linked Data principle, correct?

I see that it mentions SPARQL in point 3, while this amazing query language has been, unfortunately, completely removed from Solid specifications long time ago (while we do keep SPARQL in NextGraph and ActivityPods/SemApps).

So I wouldn't pay too much attention at following precisely the letter of those 4 rules, that nevertheless are good guiding principles.

2006 is quite far in the past. And those 4 rules are not written in stones, as far as I know.

[niko-ng]

Yesterday I had a very nice phone call with Rahul, following his invitation, and we were able to go over many interesting subjects. I will summarise here what is related to my "decoupling Solid specs" proposal and the questions he previously raised, about HTTP compliance.

First of all, my proposal is not so much about allowing different transport protocols to implement the APIs (API understood as, by example, the LDP GET method to fetch a Resource). The abstraction I am talking about is needed in order to remove the client-server requirement that is currently present in the specs a bit everywhere. For this reason I propose to describe the APIs in abstract terms (similar maybe to what IDL Interface Description Language could express) and then let implementers choose how exactly those APIs will work, and how they will be presented to the developer. The easiest way to provide those APIs while removing the client-server constraint, is to implement them as Javascript libraries (or any other language readily available in the client APP). Then each "flavor" of the library will do completely different things, depending on the implementation spec that is followed.
An HTTP implementer would use a window.fetch() call. And a DID implementer would use some other mechanisms to retrieve the data, in the case of NextGraph: if the data is not already present locally, it would use some WebSocket to contact some Brokers, do a sync with them and possibly subscribe to a topic in the Pub/Sub to stay updated on that resource. The protocol that is used here is specific to NextGraph. it is binary and openly specified, but not standardised (yet). The DID method will be registered to W3C soon though.

Now the main concern of Rahul, and he clearly made his point during the call, and I thank him for that, is that "sticking" to HTTP is primarily for the sake of performance and scalability. Being based on HTTP allows us to reuse the gigantic worldwide infrastructure of HTTP servers and more importantly, to benefit from the scalability mechanisms of services like Cloudflare companies can provide, namely: distributing content to the edge of the network, and transparently routing to the closest server.

This explains also the approach of BRAID in trying to stick to HTTP even for synchronizing CRDT updates.
Fair enough.

What I then was able to explain to Rahul is that:

• NextGraph does have its own mechanisms for scalability built in the protocol itself, as the "content delivery" is based on pub/sub broadcast mechanism centered on overlays. Brokers maintain the pub/sub and the routing tables between subscribers and publishers, forming a general (cyclic) undirected graph called the core network. This core network can reroute its connectivity according to rules ranging from load-balancing, multicasting, latency optimisation, or clustering of pub/sub topics (brokers who interact on similar pub/sub topic clusters will establish and keep direct connections). Replication/distribution of content is also easily done thanks to the Content Addressing feature employed pervasively in the protocol.
• In any case, scalability concerns are only relevant to public content, which is a valid usecase for Solid and NextGraph alike, of course. But we should always keep in mind that permissionned content will never need to scale. For this reason, we clearly separate in NextGraph the datasets that are public from those that are permissionned, for optimization purposes.
• eventually, we should also take into account that the "scalability issues" we want to address might be generated by the general architecture of our information systems and social networks that are currently highly centralised. And that once the data and usages are decentralised (which is clear goal pursued by all of us), the need for scalability might be drastically reduced.
  Do we really want to be able to offer technical solutions for Social Network platforms that foment the emergence of "super-star influencers"? Do we want to continue supporting an economical system that is highly concentrated and monopolized with big corporations that need, indeed, HA and scalability to sustain their business models? Or is Solid and our decentralized web approach more oriented towards keeping the scale of operations under control? What about virality in the networks? Do we want to support that? Is it a desirable feature?
  I believe we should have ways to address those needs when they arise (like national or international broadcast, emergency response, etc...) by example with the scalable broadcast available in NextGraph (and of course also with HTTP scalability offered by Cloudflare, to name the most famous one). But we should also understand I believe, that most scalability concerns come from the centralized paradigm we try to move away from, anyway.

The takeaway is probably that: HTTP is not the only way to offer scalability. abstraction is about removing the client-server constraint in the specs, and scalability should probably not be the main driver of our endeavour. I am more in favor of reaching adoption by the masses, who like you and me, are not super-stars, and still need decentralized, privacy-preserving solutions outside of Big Tech monopolies.

[CxRes]

I gather that @niko-ng has a very expansive vision for decentralized internet that has implications for many layers of the protocol stack and definitely should be input for LWS.

If we want LWS uses to benefit from Local-First technologies, we definitely need to think deeply about how best to hide client-server architecture (which is not going away any time soon), so that end user/application has access to data local first without having to worry about protocol level details, which is managed in the background for the user. This may or may not be within the perview of LWS, but it is a requirement that should certainly inform its design.

A broader discussion, which will involve P2P protocol stakeholders will need to examine how internet infrastructure can be decentralized. But that's a more ambitious project and is likely outside LWS' very limited charter.

[michielbdejong]

@niko-ng see also my comment in Matrix - about similarity between your proposal and things we're already doing in (Solid) Data Modules, PDS Interop, etc., that might be of interest to you and that we might be able to collaborate on!