Add a View property to the info section to change the perspective of subscribe and publish operations. #390
Comments
|
Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue. |
|
Hey Michael, it's an interesting and at least for me a confusing point too. I did indeed notice you had the switch in the paho-python and thought it was a good idea. IMHO, The ability to define the point of view is a MUST. In my day to day job most of my conversations are around how a client can interact with a service, rather than the actual server definition so being available in the core spec and therefore consistent would be a great solution. |
|
Thanks for your support. I put the same flag in the java-spring-cloud-stream template, and it was recently added to the java-spring-template with the name 'inverseOperations.' |
|
Hey Michael, i'm agree with the idea, but as i explained i other issues\PR I would be very careful with ...And regarding |
|
I agree the terms aren't perfect, those are the terms that our group in Solace came up with. I'll take this back to them for further discussion. Ultimately it's up to Fran to decide, should he agree to put this in the spec. |
|
Hi Michael, I agree with the idea. Couple of questions w.r.t "provider" perspective:
|
|
+1 to this feature. What if we use Consumer / Provider instead of Client / Provider? |
|
I would rather not have two versions of a spec based on a field (in this case, 'view' is being proposed to indicate which perspective you are viewing). Has anyone considered a shift in the language for the spec from being "application-centered" to being channel-centered? In that way, the channel defines what it accepts and what it publishes, leaving the spec to describe the message structures and assoc bindings for generators to leverage based on if the generator is designed to support publishing, consuming, or both. |
I understand where you are coming from and this is where I started as well. My fear was we needed something for the way the spec is written today (possibly a band-aid) and really think this through (in my opinion) for the next major release. But, to your point, if it was channel-centered, we could say a channel has a 1..* relationship with a schema and not use a verb (because to me, a channel is a pipe and is worthless unless someone put something into it, and someone pulls it out ;) ). I think then you could have "Application" specifications that reference channels and use verbs that say "what it does" (super helpful for internal enterprise ICD's and code gen) and "what you could do" which would be useful for B2B (external) use cases.... So my only point is that I like your proposal to have something channel centered, but could leverage that for "Application-centric" modeling as well. -- I will also in another reply, explain in better detail what we were thinking around view for the community to review as a near-term solution since we get tons of questions from our customers on this aspect/view (no pun intended). |
|
I like the debate this is generating. In my opinion, this flaw of the spec raises deeper intrinsic problems that need to be solved in a solid way. Let me expose my point of view here: View property on the info sectionI like this proposal for what it represents, however, we should take into account possible side-effects it can cause. This flag does not only change the meaning of the publish/subscribe verbs but the way protocol bindings are interpreted. For instance, HTTP, WebSockets, NATS, and other protocols supporting the concept of client and server or the concept of request and reply will be affected by this flag too. As an example, it's not the same to say that a Not going to say this is impossible to solve but definitely is not just about adding a flag on the info object. It has a lot of side-effects. And that's precisely what I would try to avoid in the spec at all costs: side-effects. Why avoiding side-effects is key?The spec is built with reusability in mind, even though it has its flaws on this area too. Let me explain this by example. Given the following AsyncAPI document: asyncapi: 2.0.0
info:
title: My user management service
version: 1.0.0
channels:
user/registered:
description: In this channel, we get a message every time a user registers on our platform.
subscribe:
description: Subscribe to this channel to get a message every time a user signs up.
message: ...If we add the asyncapi: 2.0.0
info:
title: My user management service
version: 1.0.0
view: provider
channels:
user/registered:
description: In this channel, we get a message every time a user registers on our platform.
subscribe:
description: Subscribe to this channel to get a message every time a user signs up.
message: ...The description message asyncapi: 2.0.0
info:
title: My user management service
version: 1.0.0
view: provider
channels:
user/registered:
$ref: 'shared.yaml#/channels/userRegistered'Suddenly, it's not that obvious. And it gets worse as more files depend on the What's the purpose of this flag?If I understood correctly, the purpose is to band-aid the spec to solve a common misconception regarding the Don't get me wrong, I'm all for fixing it but the more I think about a solution the less likely it seems to me it's going to happen in a minor version. This deserves an elegant solution and we shouldn't be afraid of releasing version 3.0.0 soon. Channel-centered vs Application-centeredI think @jhigginbotham pointed us in a great direction. Not only with the comment above but in this tweet about OpenAPI newest version (mate, you got me thinking). The problem of channel reusability and this problem can actually be addressed by going channel-centered. However, we'd lose the value of an application-centered definition. So, why not do both? And why not making operations first-class citizens instead-of/as-well-as channels? Maybe this discussion is out of the scope of this issue. Or maybe not, that really depends on how much do we want to change. Everything is better with examples, so here goes some food for thought. Please, don't treat it as a proposal. I just want everyone's mind to be as open as possible to solve this problem: Application (asyncapi.yaml) asyncapi: 3.0.0
kind: application # Optional. Defaults to "application". Can also be "library".
info:
title: Trains API
description: Real-time train updates
version: 1.0.0
operations:
publishTrainUpdate:
type: publish
channel:
$ref: 'library.yaml/channels/trainUpdates'
subscribeTrainUpdate:
type: clientSubscribe
channel:
$ref: 'library.yaml/channels/trainUpdates'Library (library.yaml) asyncapi: 3.0.0
kind: library
channels:
trainUpdates:
name: trains/updates # Not sure about "name". Maybe "path"? "address"?
message:
payload: ...I'll elaborate on this idea as soon as I find some time but please don't hesitate to leave your thoughts and/or concerns here. |
|
I think your last examples are what I was thinking in that what you have with a "Library" actually becomes sort of a catalog of available channels. In the kind "application" what does the verb represent? We know that in the kind "library" that you could publish or subscribe to any channel and that it would be access control rules that would limit what a client would actually be able to do (thus not described in the spec, just like openapi). Therefore the operations represent what the application does or the operations available for another application to perform? I think that brings us back to the crux of the discussion in that the Library is list of what channels are available and what you could do, but I would like the kind "Application" to be the operations performed by an application. Thoughts? I think this discussion is super interesting and good! |
Yes, sorry, I should have clarified this point. In this example, publish/subscribe means what the application is doing (opposite as now). Just so we don't screw the other point of view that's sometimes necessary, we should add something like @jhigginbotham and other people are questioning if we should have operation verbs at all. I think we should but let's keep our minds open and understand when is this information valuable and why. |
|
Good explanation on the potential additional confusion for using a view band aid. Couple of points that occur to me. First a question, what is the value of retaining publish, subscribe as is ? rather than having 4 verbs: applicationPublish, applicationSubscribe, clientPublish and clientSubscribe or something similar. Would continuing to use publish & subscribe still lead to confusion when you first pick up the spec? And actually taking a step back, does going down the route of adding verbs run the risk of making it to verbose? i.e. if I understood the point made about NATS in the community call, would you potentially need more verbs to support its req/response via the same channel? or did I misunderstand the point being made. |
|
I have a question: I was under the impression that one would define a yml file at the application level to document what the application does and allow for code generation at the application level. At the system level with many applications, the various yml files would point to a "library" of message definitions to support the publishing and consuming of the defined messages. My vote would be to keep current publish & subscribe verbs but document them as being from the application's perspective... aka messages application publishes and/or subscribes to. Great spec and very useful for document our current system. Many thanks to the group. Great work. |
|
This issue has been automatically marked as stale because it has not had recent activity 😴 |
|
By separating channel from operation, it looks like Ignoring NATS - what if the
@cappelaere I've been (incorrectly?) treating AsyncAPI as a way to describe the broker/event backbone - e.g "This channel contains weather data from IOT sensors - publish to it if you're a new device, subscribe to it if you want to process the data". However, I think the existing paradigm as documented (https://www.asyncapi.com/docs/tutorials/streetlights) makes sense from an application view too - you're stating "to use this application publish new messages here, and if you want to see output you need to subscribe here". Also, this means that you can continue (if acceptable) to use the spec to also describe the broker/event backbone - the inversion would be even more confusing - "this is a Kafka broker, any new messages will be published to the channel, but if you want to add a message you need to use the subscribe operation" To me, while I think "subscribe to the published messages" makes sense, "produce to the subscribe endpoint" is harder to wrap your head round as a new user: What use cases have you seen where the inverse is needed for applications? |
|
Or - is the problem fundementally regarding code generation - that tools need to be able to understand the perspective (i.e generating mocks of applications, vs generating consumers)? |
|
In my case, I have dozens of applications. Each one of them has a unique YML file describing what each application generates (publish) or ingest (subscribe). So the YML file is application centric. It allows for discovery of the API's (description of the services) and generation of code for applications. As a developer, this helps me keeping track of who does what and who needs what. I am the user of those YAML files :) |
|
I, too, see an AsyncAPI document as formalizing the (message-based) contract of an application, and i think it is wise not to depart from this clear application-centric perspective almost accidentally, as a side-effect of clarifying the meaning of
(This could work well with a solution to #415 .) |
|
I can't fault @lornajane 's, @fmvilas , and @jmenning-solace 's analyses, but TBH it makes me very uneasy to radically change perspectives and concepts of a spec that just recently reached version 2 - presumably a sign of maturity. I also feel that AsyncAPI is actually very effective at describing the application-level view of messaging (some confusion about the meaning of "publish"/"subscribe" notwithstanding). Maybe we could introduce a second, system-level spec that defers to individual AsyncAPI 2.x docs for the application-level view?:
That way we could rescue the application-level meaning and definition of AsyncAPI 2.x, yet still introduce that higher-level description that seems to be missing? |
|
I think that the problem for me here is that we would want an AsyncAPI file per endpoint, not a single central file. In this context whilst both the producer and consumer know about the channel - neither knows about the other. From a perspective of messaging practice, we don't want to couple the producer and consumer into using a single file. It should be possible to add a new consumer, by the act of describing the consumer, and the channel it consumes from. From a perspective of microservices, this is important to 'independent deployability' we want Team A writing the producer and Team B writing the consumer to work independently of each other, including in having separate repos. This allows Team B to deploy a consumer, independently of Team A. Now, I appreciate some folks might want to have producer and consumer in the same file, perhaps for a work queue, but this model, putting both in the same file won't work for us. At a deal breaking level. |
|
For me should separate between:
As Tooling can look in files, produce a list of who produces to and consumes from a file, there is no need to explicitly state that. AsynAPI could agree a format for such a 'graph' and provide tooling to generate one for sure, but I don't think its how we should describe an endpoint. I would stick to the ABCs for that. This applies to IaC too - we are explicitly using it in that context. |
|
I'd also suggest that endpoint is a far better term than actor as it is less loaded: https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageEndpoint.html Although WCF is mostly dead, it was authored by folks like Clemens Vasters, who have credibility in this space and represents and earlier attempt at this. It's definition of an endpoint and ABCs remains useful prior art IMO: https://docs.microsoft.com/en-us/dotnet/framework/wcf/feature-details/endpoints-addresses-bindings-and-contracts |
totally agreed - but that's the current situation, isn't it?: You define an AsyncAPI 2.x doc for an application, either for one of its endpoints, or some, or all. The granularity of AsyncAPI 2.x docs is up to you. But whatever the choice, the AsyncAPI 2.0 docs are for that application only. |
Agreed. During pre-2.0, I had proposed the following:
AsyncAPI was designed from the application perspective, which creates confusion with certain system topologies especially when they aren't client-broker. The specification tries to clarify this in multiple places, as it gets confusing when defining a channel but from an application-centric perspective. The choice so far has been to optimize for the tooling, rather than a description file format that centers on the channel. Thus, the answer so far has been to share the channel definitions across applications to better support the code generators. That seems to have worked so far, but that doesn't fit my desired model. I'd prefer to have a file format for a channel-centric definition with bindings that are independent of system topology. The channel becomes the focal point and declares what messages it will accept and publish - independent of the system topology that makes it happen. A separate file may be created to reference this channel definition and declare a topology, which could be used by code generators and other tooling. I think we could keep most of the backward compatibility of we define these two separate file formats and allow teams to select which one or both they wish to use. In addition, there could be multiple files of each format that make up a much larger system, allowing for decomposition. Unfortunately, I have no time to pull together a strawman of what this would look like. If someone else likes this idea, feel free to run with it and let's see what comes from it. Otherwise, we may need to remain application-centric for now as that seems to be the consensus barring any improvements to the existing specification to allow files to be channel-centric. |
Is this then the flexibility from @lornajane original post #390 (comment) - it provides:
An AsyncAPI document becomes actor centric - and tooling can combine documents to build the interconnected graph (which can also be described using the same document format). Channels then purely define the "pipework" between actors and have little substance themselves. My concern is that we don't want to lose sight of AsyncAPI for describing a consumable API - I should be able to read a spec - know what "channel" I need to connect to, and understand what "message" I should be sending/receiving to interact with an actor. Along with being able to document all existing actors so I can gain an understanding of the overall interactions in the system. |
agreed that it's important to not lose the "simple", application-centric view of AsyncAPI 2.x: after all, this captures the contract for a given message-based API as exposed by a given application. The system-wide view is important, and insightful, but on a higher level - and sometimes not pragmatically attainable or even desired: there still are siloed projects that are served best by the application-centric contract and don't care about the whole. Hence my proposal to compose the optional system-level doc by referencing individual AsyncAPI 2.x docs. |
@iancooper , great point. I updated actor -> endpoint |
|
@iancooper , thanks for the comments, I owe you some on your bindings proposal.
This a very common request from our clients as well. In the example above, the endpoints In the following examples, the endpoint, channel and message entities are presented as separate files. This is solely for the sake of consistency. The use of references allows for a wide range of implementation options including
Definitely. I think the example shows how that could be accomplished through refs. (As a side note, I think considering endpoints as either producers or consumers perhaps moves us too close to a synchronous, client-server view of the world. Most endpoints serve as both producers and consumers, and not necessarily in a 1:1 fashion where everything produced by an endpoint is consumed by an inverse endpoint.)
Instead of endpoints, in this proposal the location of the
IMHO, that needs to be an implementation decision. But as seen by the channel reuse problem, the current spec forces people to adopt solution #4. |
@GeraldLoeffler, this may be more than you want to attempt, but could you mock up an example of these files? My concern is that because v2 combines channel and the application into a single entity, the reusability of "shared resources like ... channels" would not be possible at the proposed system level. Likewise, requiring references to v2 application docs (which are in turn required to include channel definitions) seems like it would prevent reuse of channels and leave the verb confusion. |
|
@jmenning-solace Thanks for the detailed reply, the discussion is clarifying for me at least
I guess it depends on whether you think that a process or service Foo has just one Endpoint or many, if it has different roles. An open question for us internally, and I suspect that we would treat those as separate files. In the first 'Thinking Out Loud' I believe this idea was raised as something definable in the Info object - what endpoint is this.
Yes, reuse of a channel definition is interesting as a problem because multiple endpoints use the channel definition. I don't dislike @lornajane's idea of explicitly surfacing endpoints (it's more using the name actors over endpoints that concerns me). I'd assume something like: and then within endpoints, you would swallow up much of the operation object today, but moves it from under the channel. I'd presume we might want to define objects for send and receive (and keep publish/subscribe with existing semantics for migration or needs of WebSockets folks I don't fully grok). You still need bindings, payload, header et al here still of course. (I don't think you can put the message on the channel - not all channels are datatype channels that only support a single message type) Within that you could just reference the channel by name - assuming you need some sort of unique naming scheme anyway, otherwise we need some sort of namespace. So an endpoint (and you could have many) would look something like: And ultimately that seems to be the key observation from Lorna's talk - that we need to lift the 'operation' into a top level object, whether we call that endpoint or actor |
|
So overall @jmenning-solace I think I am pretty aligned, as always 'details', but surfacing endpoints, which I think is the kernel of @lornajane's proposal does seem to fix issues for us |
I agree - but we'd need to think about the https://www.asyncapi.com/docs/specifications/v2.0.0#channelItemObject - at this point, what is there to reuse - or does the channel essentially become metadata for tooling to build a "system wide" view (to collate endpoints by channel?) Because originally I understood the re-use problem with channels was that the channel owned the message definition. If instead we have the |
Well, pragmatically a channel may need bindings to help teams work with IaC or code generation. But yes, for messaging it is really a logical address that lets you route between senders and receivers.
|
|
Would I be simplifying the problem too much if I were to state that, What if we simplified the spec to consist of channel definitions along with their message structures alone ? It is upto the consumer of the spec to decide what sort of code they want to generate when interacting with a given channel. From spec standpoint if one can define a channel, its bindings and it message contract, wouldn't that suffice and make it simple. The concept of an actor and the concept of the verb / action performed by the actor holds significance from a code generation / tooling standpoint. The spec definitely should accommodate tooling concerns, but from how I look at it currently, it feels like the tooling concerns are defining how the spec should be defined. If we look at OpenAPI spec as an example, it is defined as a list of paths and the messages on those paths. |
This hits the same problem with channel reuse - "who" is in charge of writing that document? The From my understanding the aim of this discussion is to provide an answer to the following questions:
In OpenAPI - you can answer 1 and 2 by looking at the spec, but not 3 because you only talk about servers and not clients. In AsyncAPI you can only answer 1,2 and 3 by writing a combined EDA document - which you can definitely do in AsyncAPI 2.0 - but needs the channel reuse and pub/sub to be solved for tooling to be able to combine documents to get that answer |
you're right, sadly i don't have the bandwidth to do that, sorry.
the "verb confusion" would remain, true: it can only be resolved by a backwards-incompatible renaming: trivial to do and trivial to automate "upgrade" tooling for AsyncAPI docs. but it seems that all other restrictions of the v2 spec can be resolved by introducing fully backwards-compatible relaxations of what's required (say, in general, AsyncAPI 2.1.0 still very much operates by associating objects through containment in the YAML (channels contain their operations which contain their messages), but this can backwards-compatibly be relaxed by making the contained objects optional, allowing them on the top level, and allowing visually, nested objects can be transformed to: or whatever is more convenient for decoupling the objects |
It may not be your intent, but note that a channel is not necessarily constrained to a particular message schema. Yes, the DataType Channel pattern would constrain a channel to a single message type, but I don't think that AsyncAPI should constrain to just one type. The key issue is that two endpoints communicating over a channel need to share a reference to that channel, and implementers wishing to support IaC will want to create a binding on the definition of that channel. And those endpoints may be in separate files. Hence many of the suggestions introducing a new concept 'endpoint' (originally proposed as actor) to hold the operation being performed on the channel separately from the channel. An endpoint might be able to be described as one of a different set of types that give metadata about the type of interaction, and we might possibly need to support both broker based contexts and point-to-point scenarios. |
@GeraldLoeffler, thanks, I think I understand what you're getting at. It seems very similar to @jhigginbotham suggestion as well. Let me spend some time today working through the implications and writing up some examples to solidify things a bit more in my mind. It's helpful to understand what's "easy" and "hard" in terms of breaking changes. @nictownsend and @dalelane , how much would changing |
Breaking changes in the spec can definitely be accounted for |
I agree with @iancooper that we shouldn't constrain a channel to one data type. This is particularly important with Avro over Kafka, where LightMeasured v1 schema and LightMeasured v2 schema might both be presented on the same topic. However, I do think for decoupling purposes, data types should be properties of the channel, not the endpoint. If the purpose of a channel is to eliminate dependencies between information providers and information consumers, forcing consumers to determine what datatypes each provider sends would undermine that. |
Interesting point, as that brings up the definition of a channel and message type, including their relationship cardinality. Clarifying these definitions and providing copious examples will likely clarify intent across this discussion, avoid confusion in the future, and lay the groundwork for future spec documentation + examples. It sounds like a few people are doing that now, so as part of the work, I'd recommend defining what you mean with each term introduced to ensure alignment as part of the write-ups and examples being produced. In some cases, we may end up talking about the same thing, but in other cases we may be using the words in different contexts. |
|
@GeraldLoeffler So, I've been trying to piece together an example. I think there is general agreement that publish and subscribe are not generic enough for all async use cases. You suggested replace The as-is API representation of the Streetlights Control Hub then becomes: In this use, it is assumed the API describes:
In other words, the endpoint may choose tosend For those with different assumptions, requirements, use cases and/or perspectives, in proposed v3 it becomes possible--though not mandatory--to explicitly create endpoints and/or classes of endpoints. In the optional v3 usage the API describes how either
To accomplish this--and this basically just restating @GeraldLoeffler--:
As @iancooper stated, the devil is in the details, but this seems like a way to make the spec evolve towards more enterprise functionality, while (mostly) maintaining backwards compatibility through reasonable assumptions. However, I'd be interested in the opinions of those making tooling. Does having this flexibility make it harder/impossible to generate code, for instance? |
|
To make the issue title better reflective of the primary topic of conversation, I have opened a new issue centered on the |
|
Leaving here the discussion @dalelane and I had during the last ThinkingOutLoud episode: https://youtu.be/Qsu_yC-5YYM?t=64. I hope it serves as a basement for those who are still reluctant to get involved in the discussion. Key takeaways
|
|
Yo folks. As a result of the Thinking Out Loud episodes, I've come up with a proposal of how to solve this problem. Hope it makes sense and captures the feedback and ideas that y'all gave me: #618. |
The specification regards publishing and subscribing from the point of view of what a client is allowed to do. For example, if the spec has a publish operation, then applications will actually be subscribing. This is consistent with OpenAPI server side code, where if one sees a GET, it is not the server that does the GET but rather other applications.
However some users of AsyncAPI find it more intuitive to consider things from the opposite perspective. If they have a document with a publish operation and want to generate code, they expect that the code will publish, not subscribe.
I propose adding a parameter called 'view' that can have two values:
'client' - this is consistent with the current specification. It means that the document says what a client of the application can do.
'provider' - this means that the document describes what the provider of the service is doing, i.e. a subscribe operation means that the application is subscribing.
Can't it be tackled using specification extensions?
It can be done using specification extensions, but so far there are three generator templates using their own extensions and terminology - it would be nice to make this consistent.
Describe the solution you'd like
I would like to see an addition to the info section that supports this parameter, e.g.
Describe alternatives you've considered
The java-spring-cloud-stream and paho-python templates currently support this through the use of specification extentions and parameters. The java-template also has this feature but uses different terminology.
The text was updated successfully, but these errors were encountered: