What content types would you like to see?

[galligan]

With XIP-5 (Message Content Types), a framework for interoperable support of different content types in XMTP messages was introduced.

This flexible content type framework was set up to:

• enable iterative, community-led development of new types
• establish a standard to encourage interoperability
• carve a path for features that could improve XMTP's usability and functionality
  • and reduce the dependency on a central authority for adding them
• give developers creative freedom, and a path to get widespread adoption of the content type

Developers can create any new content type that they'd like—the only requirement being that a client needs to know how to render it. If a developer wishes to provide end-to-end support for a content type within their own client, awesome! After all, it's all just part of the payload of encrypted messages.

So…what kind of content types would be interesting? Let's jam on different types that could be developed, and how they might evolve the web3 messaging UX with XMTP:

👇 Leave a comment with a description of what the content would be, and how XMTP clients would interact/support it.
(One new content type idea per comment please!)

[galligan]

Personally I'd love to see a Markdown content type.

Just as I sit here and use markdown to style my text I'd love to be able to do the same in messages with XMTP.

Markdown support would:

• Allow for rich (enough) formatting for clarity of content
• Support for things like lists, code snippets, etc.
• Parity with popular chat platforms like Slack & Discord (think the #annoucements channel in so many servers)

The contentFallback would also be important here.

• Sending clients could strip back any formatting and add the Markdown-less text as the fallback
• Receiving clients could present the content in whatever format it can support

Also, if there was an image content type, then it could take advantage of XIP-9's introduction of the Composite type and see images nested within the Markdown.

[saulmc]

I wonder how closely a Markdown type might resemble the existing Text implementation. Should Markdown strings also be encoded in utf8?

If it's just a matter of declaring the type, then it will only need to be registered with the XMTP client in order for applications to know they should handle these messages with a Markdown renderer like react-markdown.

[galligan]

It might be a good idea for someone building the rendering components of this to fork or wrap react-markdown to improve some potential security UX issues.

My thinking here is that some clients may choose to render all Markdown formatting except links.

Links with aliases would be a very easy attack vector, especially if viewing those on a mobile client where you might not have a hover state to inspect the link. Socializing this issue as a part of an XRC makes sense, and going through greater lengths to make it easy for devs to implement such security measures may really help.

[killthebuddh4]

The next content type that I'm probably going to get started prototyping for Relay is Reply. It's pretty straight-forward I think, probably not much more than a parent Id + a payload. We'll be using Reply to implement a number of features, but the most salient are reactions and message threads.

[galligan]

@killthebuddh4 love that. Reply is definitely one that I've been keen to see for some time. We've tossed the idea around internally, but it's great to see someone from the community looking for it too.

parentId + payload is how I'd been thinking about it as well. What do you think about that being its own contentType first…upon which content types like reply or reaction and more could be built?

I'd been thinking about a type that I call target which has those two fields. From there you'd do a composite type and include a reply type. Or perhaps reply is simply another datapoint in target?

target could be used for:

• Replies
• Reactions
• Threads
• Deletions
  • Note: this is more of a future curiosity…would it be possible to direct another front-end to interact with messages based on some command? It would likely need to be directed by the owner of the messages, but still…something to think about.

[PanelSean]

target would be super useful, for all those operations. Relatedly, a general way to link to a specific message could allow for deep-linking?

[galligan]

target would be super useful, for all those operations. Relatedly, a general way to link to a specific message could allow for deep-linking?

Deep link would be awesome @PanelSean. Maybe warrants another discussion so we can dig into it? I could see it being super useful as it relates to notifications.

[leeadkins]

target also kinda seems like it could be used for something I've been thinking about around some sort of Read Receipt or Cursor content type.

Ideally a client would make those opt-in (for read receipts like we know today), but if we could direct a read receipt message to something like just your intro topic or some place other than the conversation topic, they could also serve as a kind of "cursor" so clients could keep track of the last location you read up to, which would help UX around unread counts.

Otherwise, if they just go out like standard messages, clients could use them and style them as normal "seen" markers.

[killthebuddh4]

One higher-level question I've been mulling over is how to think about the idea of a content type itself. For example in HTTP a "content type" is mostly (I think) about serialization whereas it seems (to me at least) that the more interesting XMTP content types are about applications. So in addition to something like an application/json content type that is all about serialization, we'd also have a more specific content type called Reply that uses JSON to implement something like

interface Reply {
  parent: MessageId;
  payload: any;
}

For example I've implemented a content type that can be used to do group conversation and I see "group conversation" as an "application".

So to be more specific/concrete, I'm wondering: What "sorts of things" belong in a "content type"? How is everybody else thinking about this? Does an "application" layer content type maybe make sense on an individual XMTP client level but not in terms of community standardization?

[galligan]

What "sorts of things" belong in a "content type"?

The original motivation behind a flexible framework for message content was that we couldn't possibly come up with all the use cases, nor did we believe that any central authority should be responsible for them.

Application-specific content types were envisioned. We'd even tossed around the idea that a turn-based game might want to communicate moves between players, by leveraging a content type that expressed those moves. A front-end would interpret those moves and change the state of the game accordingly. Perhaps the "fallback" in this case would simply be the description of what happened, or that a client may simply choose to not render it at all.

Considering this, just as you'd described…we'd anticipated developers producing content types that included structured payloads such as JSON.

How is everybody else thinking about this? Does an "application" layer content type maybe make sense on an individual XMTP client level but not in terms of community standardization?

Ultimately I think this is the beauty of the open content format. If you've got a novel approach, build it. See if other developers are interested in it (I'm envisioning a discussion in this forum), or put it straight up as an XIP/XRC proposal. That way it can be socialized, and perhaps adopted as a standard (not unlike how ERC-721 was adopted).

[killthebuddh4]

Awesome! All that sounds good to me.

The "turn-based game" idea leads more generally to something that is pretty incredible to consider IMO which is something like zero-effort integrations or maybe "cross-sectional applications". In web2 we have "integrations" between applications but in web3 the integration layer could be so powerful and effortless that the "integrations" start looking more like inter-application applications or "horizontal" applications that cut across more traditional "vertical" applications.

[tg44]

@galligan I'm joining in to this convo and thought-process! If I want to render a widget (let's say an opensea nft widget for example), I will need to add parameters to render it correctly. The "data" I will send is probably a json (here I would use something like {"tokenAddress": "0x...", "tokenId": "4", "network": "mainnet"}).

Should I use a JSON content-type, or my "custom" widget content type?

The JSON will help nothing to the other party about how this should be rendered, or what it is actually. If I choose JSON I will need to come up with a new protocol inside the json where I can define if this is an ens/opensea/uniswap widget, and after the decode, the other party should decide if they can render it and if not what should they show instead (which is already solved in content-types if the client has no codecs for the given type).

On the other hand If I start to use a new custom content-type, for every widget I implement, we will have a LOT of content types FAST, and we should start a content-type library asap.

[neekolas]

If there is clear differentiation for how you want to render each type of widget that can't be made universal, I think a content type per widget makes sense. Makes it easier for the application layer to do the right thing and render them all using separate components. Also makes it easier to perform validation, since each type presumably has a different schema you would want to enforce before sending/rendering.

You can bundle them all as one library or let people import them piecemeal

[saulmc]

The Image type is the one mentioned most frequently in my conversations with developers.

A few teams have even expressed interest in writing an XRC! Has anyone started work on a proof-of-concept yet?

[PanelSean]

I'm going to talk with @killthebuddh4 about this and share some libraries / paths to take to Images

[leeadkins]

Images would certainly be welcome.

Our current proof of concept around them is a pretty basic custom codec that is aware of base64-encoded image data URIs. Mostly just a Text clone that checks mime types. This works for our experimentation, but you can see how multiple kinds of encoding (or even on-network vs off-network) and additional metadata (image size, smaller on-network preview data in the event of off-network images) could be useful.

We use it in combination with the Composite type as a way to connect simple Text captions/messages to an Image.

I'm looking forward to seeing how other teams who might have been thinking about it a bit longer than I have are approaching it.

[saulmc]

Thanks for sharing that, @leeadkins, and welcome! That approach aligns with how I imagined an Image MVP would take shape.

Someone who I know has been thinking about this for a long time is @KishanBagaria. As I understand the TextsHQ SDK, IMG is abstracted as a type of MessageAttachment that includes some of the metadata you mentioned (size, posterImg) which might also be relevant to VIDEO or AUDIO attachments.

[seanonchain]

Hey @leeadkins , we're looking at adding Images to relay.cc chat and wonder if you have any further thoughts on the Composite type?

[polmaire]

I don't know if it enters the "ContentType" thread but one very important piece of information about messages is the read / unread status.
For now some clients use their own read / unread status which is cool locally but does not include any portability.

As XMTP is mostly about conversations, I see a very simple MVP for this: either a conversation is read or not. If it's not read, which message was last seen?

It seems like most web2 messaging services handle it that way (Whatsapp, Telegram, etc.) and it covers most of the use cases.

This comment is a very first introduction to the topic, I'd be happy to work with the community in order to make it become a reality. Reply to this comment with your thoughts!

[saulmc]

Portable read status would enhance UX at the app-level and reinforce the value and behavior of the network to end-users. @tg44 has a proposal here for a read receipt ContentType that might unblock this type of experience for recipients, while also providing valuable information to senders. What are your thoughts on this approach? Should we bring these conversations together or keep them separate?

[neekolas]

In my dream world, read receipts would be stored on a separate topic from messages themselves. That way they could be fetched independently, and it would be possible to only fetch the most recent message in a "meta" topic for each party in the conversation and know the most recently read message.

Third party apps having to display all these read receipts (since they are just messages) feels like a bad UX.

That would require some level of protocol support from us, which we'd have to prioritize against all of the other protocol upgrades.

I think it ties to some of the other conversations we've been having about the need for "meta messages" that are handled specially by the SDK so that they are never rendered in client applications.

[tg44]

I don't think the rendering is the main problem. The main problem is the computation need of the decoding. So if we have a lot of "non-renderable" messages mixed with "renderable" messages, the UX will be that the user clicks on a conversation and it loads really-really-really slow. If we move them to a separate convo, the UX in the read receipt case will be that the messages loaded as fast as they could and a little later you get the "last seen" line and the read icons when your computer catches on.

I would welcome some directions from the team, about what convo name/topic/id would be ok for these read-receipts. I think we should decouple these "meta" type of messages from each other, because it will hurt the loading performance of the read-receipts if somebody builds something spammy as a meta and uses the same convo as we. Also, if we have multiple meta, we can prioritize which features should load faster than others. (Like I want the messages first, the nicknames second, and the read-receipts third (I just come up with a probably silly idea).)

[polmaire]

Thank you everyone for your comments! @saulmc we can merge the two conversations for sure, they seem to cover the same use cases.
Adding @nmalzieu to this conversation for technical implementation.

[killthebuddh4]

IMO the "meta messages" conversation that @neekolas mentioned is where all these ideas come together. I personally think of "meta messages" as "application data" (as opposed to user-to-user messages). I think the 2 key details are:

• The reverse-chronological-message-stream architecture is not great for application data.
• Some application data can be scoped to a single client, but some should be portable (and therefore standardized).

I'm thinking the solution is somewhere in the direction of making conversation ids a first-class citizen, that we should think about standardizing conversation ids very similarly to how we think about standardizing content types. I'm imagining a situation where every wallet-to-wallet conversation is augmented by well-known "feature conversations". As a developer, if I load a conversation between 0xAlice and 0xBob I know that I can also load the conversations 0xA-0xB/receipts, 0xA-0xB/media, 0xA-0xB/permissions, etc.

[cboscolo]

Processing different content types without throwing errors

We have been experimenting with creating a content type based on the Composite content type XIP-9. Our thought was to have it contain a Text part, along with additional parts that carry additional meta data about the message. We notice that the default decoder for the composite type throws an error when it encounters types for which it doesn't have a codec. So, when you view this content type in the sample xmtp-inbox app, the only thing you see is a bunch of error messages.

Has anyone been thinking about how we ensure that different clients respond properly when receiving content types for which it doesn't have a codec? Should the community of builders align on some base types that are guaranteed to be processed and best practices when encountering unknown types?