chore: add support for asyncapi 3.0.0 operations and channels #654

fmvilas · 2022-10-13T15:40:21Z

This PR contains the following changes:

Move all the collections from v2 directory to the models directory because their implementation is always the same. Therefore we can reuse them for v3 and beyond. Check out 0791adc for more details.
Adds v3 models to work with v3 operations and channels.
Adds npm run dev so it recompiles on file change.

I split all the changes into meaningful commits to make the review more manageable.

…same implementation

fmvilas · 2022-10-13T15:41:23Z

src/custom-operations/index.ts

@@ -20,7 +21,7 @@ async function operationsV2(parser: Parser, document: AsyncAPIDocumentInterface,
  anonymousNaming(document);

  if (options.applyTraits) {
-    applyTraitsV2(detailed.parsed);
+    applyTraitsV2(detailed.parsed as v2.AsyncAPIObject);


You'll find a bunch of these. Since now a DetailedAsyncAPI can be either v2 or v3, we need to make it explicit here.

fmvilas · 2022-10-13T15:42:10Z

src/models/base.ts

@@ -3,7 +3,7 @@ import type { DetailedAsyncAPI } from '../types';

 export interface ModelMetadata {
  asyncapi: DetailedAsyncAPI;
-  pointer: string;
+  pointer?: string;


I found some cases in which pointer will be undefined and had to do this. Happy to hear about alternatives.

In which places? Could you point to them in this PR?

https://github.com/asyncapi/parser-js/pull/654/files#diff-d7ef7ffb6795ee82f8ee52244fb91a58a582e6b65a273d0ec162b0d04ae4c340R71-R73

src/models/channel-parameters.ts

fmvilas · 2022-10-13T15:54:28Z

src/models/operation-trait.ts

-  id(): string;
-  hasOperationId(): boolean;
-  operationId(): string | undefined;
+  id(): string | undefined;
+  hasId(): boolean;


I changed this because I think it's confusing. An operation shouldn't have an id and an operationId. The operationId is the id of the operation 😅 So yeah, changed it and changed tests too.

Makes sense 👍 . Would you mind adding such a change into asyncapi/parser-api#71 ?

id was treated as a key in the collection and not as operationId. The fact that in v3 operationId is mandatory (but not for operations in components - here keys are interpreted as a collection's key and not operationId), this remains a problem with v2 and how we filter collections -

parser-js/src/models/operations.ts

Line 12 in 46889bf

return this.collections.find(operation => operation.id() === id);

We have to think how to handle it.

I don't think I understand 😅 Just to clarify, in v3 operationId is not mandatory, it just doesn't exist. The id comes from the key in the operations object.

from v2 spec point of view it makes sense and that's the problem 😄

I don't think it does, even from v2 point of view. operationId was there to mean the id of the operation. If it's present, it should be the operation identifier. If it's not, ok, we can generate one if you want, but it was never meant to be an "extra" id.

Hmm 🤔 I have a problem in general because in the v3 version people will be able to define operations in components and how to distinguish this identifier in components.operations[x] from operationId - that's why we have these two methods.id() method is more of a method used to filter operations by identifier (used as key in the components.operations[x] and operations[x]), but... we'll see how to figure it out and id() should return operationId. Can we fix it in next PRs? We need also to fix that in parser-api and remove operationId() method.

To clarify: I wrote about components.operations[x] and operations[x], because in parser-api we have method allOperations() which should return all operations defined in given AsyncAPI document.

I realized no one updated the parser-api with any change regarding this.

All is fixed and there is no need to add anything else.

fmvilas · 2022-10-13T16:02:25Z

BTW, this PR is not yet done. It's still missing some implementations (challenge you to find it 😄) and, of course, tests. I hate writing tests so if someone in their divine presence wants to help this old weary man, please do it by writing some tests.

jonaslagoni · 2022-10-13T16:29:00Z

I hate writing tests so if someone in their divine presence wants to help this old weary man

If you really hate writing tests, I can only suggest you try writing them first before the implementation. That way they don't become a chore but a positive experience 😄 At least in theory 😛

fmvilas · 2022-10-14T10:21:46Z

I hate writing tests so if someone in their divine presence wants to help this old weary man

If you really hate writing tests, I can only suggest you try writing them first before the implementation. That way they don't become a chore but a positive experience 😄 At least in theory 😛

Dude, I've been around for a while. Don't preach the TDD shit at me, please ✋ 😝

magicmatatjahu

Good! Do not forget to refer to my comments and please add unit tests for new v3 models - for channel and operation will be enough.

EDIT: I will write some tests 😄

package.json

magicmatatjahu · 2022-10-17T07:34:33Z

src/models/base.ts

@@ -3,7 +3,7 @@ import type { DetailedAsyncAPI } from '../types';

 export interface ModelMetadata {
  asyncapi: DetailedAsyncAPI;
-  pointer: string;
+  pointer?: string;


In which places? Could you point to them in this PR?

src/custom-operations/index.ts

src/models/channel-parameters.ts

magicmatatjahu · 2022-10-17T07:52:35Z

src/models/operation-trait.ts

-  id(): string;
-  hasOperationId(): boolean;
-  operationId(): string | undefined;
+  id(): string | undefined;
+  hasId(): boolean;


id was treated as a key in the collection and not as operationId. The fact that in v3 operationId is mandatory (but not for operations in components - here keys are interpreted as a collection's key and not operationId), this remains a problem with v2 and how we filter collections -

parser-js/src/models/operations.ts

Line 12 in 46889bf

return this.collections.find(operation => operation.id() === id);

magicmatatjahu · 2022-10-17T07:53:07Z

src/models/operation-trait.ts

-  id(): string;
-  hasOperationId(): boolean;
-  operationId(): string | undefined;
+  id(): string | undefined;
+  hasId(): boolean;


We have to think how to handle it.

magicmatatjahu · 2022-10-17T07:55:10Z

src/models/v3/channel.ts

+  operations(): OperationsInterface {
+    const operations: OperationInterface[] = [];
+    // ['publish', 'subscribe'].forEach(operationAction => {
+    //   const id =  this._json[operationAction as 'publish' | 'subscribe'] && (this._json[operationAction as 'publish' | 'subscribe'] as v2.OperationObject).operationId || `${this.meta().id  }_${  operationAction}`;
+    //   if (this._json[operationAction as 'publish' | 'subscribe']) {
+    //     operations.push(
+    //       this.createModel(Operation, this._json[operationAction as 'publish' | 'subscribe'] as v2.OperationObject, { id, action: operationAction as OperationAction, pointer: `${this._meta.pointer}/${operationAction}` }),
+    //     );
+    //   }
+    // });
+    return new Operations(operations);
+  }


Do you have a problem with this implementation? I can help.

Yes 😄 #654 (comment)

Now seriously, just time. I wanted to share something as soon as possible for review because this PR was getting too big already.

src/models/v3/channel.ts

magicmatatjahu · 2022-10-17T08:10:38Z

src/models/v3/operation.ts

+  channels(): ChannelsInterface {
+    const channels: ChannelInterface[] = [];
+    const pointer = this._json.channel.address ? tilde(`/channels/${this._json.channel.address}`) : undefined;
+    channels.push(
+      this.createModel(Channel, this._json.channel, { id: this._json.channel['x-parser-id'], address: this._json.channel.address, pointer })
+    );
+    return new Channels(channels);
+  }


Why are you doing it this way? Since we define with $ref channel in the operation, we only need to pass it to the collection without additional operations. Only problem is to retrieve proper pointer:

Suggested change

channels(): ChannelsInterface {

const channels: ChannelInterface[] = [];

const pointer = this._json.channel.address ? tilde(`/channels/${this._json.channel.address}`) : undefined;

channels.push(

this.createModel(Channel, this._json.channel, { id: this._json.channel['x-parser-id'], address: this._json.channel.address, pointer })

);

return new Channels(channels);

}

channels(): ChannelsInterface {

const channels = this._meta?.asyncapi?.parsed?.channels || {};

let pointer: string = '';

let channelId: string = '';

for (const [channelName, channel] of Object.entries(channels)) {

if (channel === this._json.channel) {

channelId = channelName;

pointer = `/channels/${channelName}`;

}

}

const channel = this.createModel(Channel, this._json.channel, { id: channelId, pointer })

return new Channels(channel);

}

If there is anything you don't understand then let me know :)

Or I don't understand something.

Did you try this implementation? I have a couple of concerns with this:

You're only checking for existence in this._meta?.asyncapi?.parsed?.channels but the reality is that a $ref can be pointing anywhere, even another file. Actually, most probably another file in many cases.

if (channel === this._json.channel) { is this strict equality gonna work? With === the two have to be the same object instance in memory, right? Or am I missing something?

You're only checking for existence in this._meta?.asyncapi?.parsed?.channels but the reality is that a $ref can be pointing anywhere, even another file. Actually, most probably another file in many cases.

And that's my problem here, because I don't understand if operation's channel should be also included in the channels section of AsyncAPI document. If it won't be, how to check if this channel is involved in the described application? It seems to me that if a given operation is used on a given channel, it should also be defined in the document, even through the ref in the channels section.

if (channel === this._json.channel) { is this strict equality gonna work? With === the two have to be the same object instance in memory, right? Or am I missing something?

Yes, two objects should point to this same memory reference to check equality. We will check for example channel.servers in this same way to check if given server(s) is/are "available" in given application.

Ok, I pushed changes to the PR in latest commit - I know that there are quite a lot of changes, but we had a lot of missed methods or models didn't implement correctly interfaces. I will write unit tests and fix sonarcloud issues, don't worry. If you wanna, you can check implementation and ask if you have questions.

Co-authored-by: Maciej Urbańczyk <urbanczyk.maciej.95@gmail.com>

sonarqubecloud · 2022-10-21T10:31:04Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
20 Code Smells

No Coverage information
2.5% Duplication

magicmatatjahu

@fmvilas Ok, I wrote tests and also fixed all interfaces and corresponding models for v2 and v3 versions. I think we can merge it and fix all errors (especially sonar issues) in next PRs.

Also I changed title of PR to the chore, because:

we don't validate new version of spec
we don't have all implemented models

so package now is "broken". I hope that you understand :) Of course if you have any questions about implementation, why etc, feel free.

cc @smoya @jonaslagoni @derberg

magicmatatjahu · 2022-10-24T09:23:50Z

src/models/operation-trait.ts

-  id(): string;
-  hasOperationId(): boolean;
-  operationId(): string | undefined;
+  id(): string | undefined;
+  hasId(): boolean;


from v2 spec point of view it makes sense and that's the problem 😄

magicmatatjahu · 2022-10-24T09:26:22Z

src/models/v3/operation-trait.ts

+    return this.action() === 'receive';
+  }
+
+  channels(): ChannelsInterface {


Now that function returns empty collection, because we should discuss asyncapi/spec#856 problem. Also in current implementation that function creates circular references between import:

OperationTrait -> Channel -> Operation -> OperationTrait

I will try to fix it if we will agree what about channel and action in Operation Trait Object.

fmvilas · 2022-10-26T15:15:27Z

@magicmatatjahu what do you mean that the package is broken now? Will it fail if I try to simply import it? Or you mean that not everything is implemented?

magicmatatjahu · 2022-10-26T15:21:23Z

@fmvilas I wrote:

we don't validate new version of spec

we don't have all implemented models

so the package is not broken in this sense that you cannot use it, but due to the fact that we don't validate and parse custom schemas and don't apply traits then the returned model is useless and for now there is no point in using it at all, so I would prefer that the release 3.0.0-next-major-specX of parser be done when we will at least have 90% of the models implemented and validation (at least) based on JSON Schema.

fmvilas · 2022-10-26T15:41:07Z

Oh ok, I don't think that's a problem. Yeah, implementation is incomplete but that's expected in this branch.

magicmatatjahu · 2022-10-26T15:44:42Z

For me it is also not a big problem to release, but why would anyone need a non-working model? Also simple validation is needed, because the parser-api itself is written in such a way that we have sure that the spec is valid.

magicmatatjahu · 2022-10-26T17:36:08Z

@fmvilas So what? Can we merge it?

fmvilas · 2022-10-26T17:44:29Z

For me it is also not a big problem to release, but why would anyone need a non-working model? Also simple validation is needed, because the parser-api itself is written in such a way that we have sure that the spec is valid.

Nobody, except us, is using this version of the parser. It's not just that Parser is incomplete but v3 is also incomplete. It's ok, we're not gonna release it anytime soon 😄

So yeah, I approve it but I created this PR so I'm not sure I should be approving it 😅

magicmatatjahu · 2022-10-26T17:52:05Z

@fmvilas I forgot about it 🤣 Sorry!

jonaslagoni

Did not see anything jumping out, only smaller formatting issues, but since we have no code formatting configuration it's something that can be changed later 😄

pssst, smaller PRs preferred 😉

magicmatatjahu · 2022-10-26T18:02:58Z

@jonaslagoni Thanks!

magicmatatjahu · 2022-10-26T18:03:07Z

/rtm

asyncapi-bot · 2023-07-14T10:17:53Z

🎉 This PR is included in version 2.1.0-next-major-spec.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

asyncapi-bot · 2023-08-17T08:59:14Z

🎉 This PR is included in version 2.2.0-next-major-spec.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

fmvilas added 5 commits October 13, 2022 17:22

Move collections out of the v2 directory because they are always the …

0791adc

…same implementation

Adding v3 models

56b9fd1

Fixing various things found during v3 development

1af3ca8

Fixing v2 tests

73b009e

Adding npm run dev

5f3c1be

fmvilas requested review from magicmatatjahu, jonaslagoni, derberg and asyncapi-bot-eve as code owners October 13, 2022 15:40

fmvilas commented Oct 13, 2022

View reviewed changes

src/models/channel-parameters.ts Show resolved Hide resolved

fmvilas commented Oct 13, 2022

View reviewed changes

Fix linting errors

46889bf

This was referenced Oct 13, 2022

Implement new operations object in JS parser fmvilas/shapeup#9

Open

Implement new channels object in JS parser fmvilas/shapeup#7

Open

magicmatatjahu requested changes Oct 17, 2022

View reviewed changes

fmvilas and others added 10 commits October 17, 2022 12:32

Use import type instead of import

7361334

Co-authored-by: Maciej Urbańczyk <urbanczyk.maciej.95@gmail.com>

Remove meta.address from v3 model

c58a021

Co-authored-by: Maciej Urbańczyk <urbanczyk.maciej.95@gmail.com>

replace import with import type

aa6cf51

Fix some incomplete changes after review

80dd7ea

fix models from spec perspective

25e84bd

add unit tests

d900b06

add more tests

715b97d

add more tests

9d9c514

fix sonarcloud errors

cfff064

create AsyncAPIDocumentV3

df73fec

fix linter errors

a70e365

magicmatatjahu approved these changes Oct 24, 2022

View reviewed changes

magicmatatjahu changed the title ~~feat: add support for asyncapi 3.0.0 operations and channels~~ chore: add support for asyncapi 3.0.0 operations and channels Oct 24, 2022

jonaslagoni approved these changes Oct 26, 2022

View reviewed changes

asyncapi-bot added the ready-to-merge label Oct 26, 2022

asyncapi-bot merged commit f259f4a into asyncapi:next-major-spec Oct 26, 2022

magicmatatjahu mentioned this pull request Oct 26, 2022

[3.0.0] Clarify operation object channel reference asyncapi/spec#863

Closed

fmvilas deleted the v3-operations branch October 27, 2022 14:11

magicmatatjahu mentioned this pull request Nov 9, 2022

Work on 3.0 release asyncapi/spec#691

Closed

asyncapi-bot added the released on @next-major-spec label Jul 14, 2023

-  channels(): ChannelsInterface {
-    const channels: ChannelInterface[] = [];
-    const pointer = this._json.channel.address ? tilde(`/channels/${this._json.channel.address}`) : undefined;
-    channels.push(
-      this.createModel(Channel, this._json.channel, { id: this._json.channel['x-parser-id'], address: this._json.channel.address, pointer })
-    );
-    return new Channels(channels);
-  }
+  channels(): ChannelsInterface {
+    const channels = this._meta?.asyncapi?.parsed?.channels || {};
+    let pointer: string = '';
+    let channelId: string = '';
+    for (const [channelName, channel] of Object.entries(channels)) {
+      if (channel === this._json.channel) {
+        channelId = channelName;
+        pointer = `/channels/${channelName}`;
+      }
+    }
+    const channel = this.createModel(Channel, this._json.channel, { id: channelId, pointer })
+    return new Channels(channel);
+  }

chore: add support for asyncapi 3.0.0 operations and channels #654

chore: add support for asyncapi 3.0.0 operations and channels #654

Conversation

fmvilas commented Oct 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmvilas commented Oct 13, 2022

jonaslagoni commented Oct 13, 2022

fmvilas commented Oct 14, 2022

magicmatatjahu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Oct 21, 2022

magicmatatjahu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magicmatatjahu Oct 24, 2022 • edited Loading

Choose a reason for hiding this comment

fmvilas commented Oct 26, 2022

magicmatatjahu commented Oct 26, 2022

fmvilas commented Oct 26, 2022

magicmatatjahu commented Oct 26, 2022

magicmatatjahu commented Oct 26, 2022

fmvilas commented Oct 26, 2022

magicmatatjahu commented Oct 26, 2022

jonaslagoni left a comment • edited Loading

Choose a reason for hiding this comment

magicmatatjahu commented Oct 26, 2022

magicmatatjahu commented Oct 26, 2022

asyncapi-bot commented Jul 14, 2023

asyncapi-bot commented Aug 17, 2023

magicmatatjahu left a comment •

edited

Loading

magicmatatjahu left a comment •

edited

Loading

magicmatatjahu Oct 24, 2022 •

edited

Loading

jonaslagoni left a comment •

edited

Loading