Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Library for easier integration testing of code generators #752

Open
derberg opened this issue Feb 20, 2022 · 27 comments
Open

Library for easier integration testing of code generators #752

derberg opened this issue Feb 20, 2022 · 27 comments
Labels
enhancement New feature or request stale

Comments

@derberg
Copy link
Member

derberg commented Feb 20, 2022

Reason/Context

It is not so hard to write code generator, it is harder to maintain it ~ Abraham Lincoln

When you write a code generator, you always write it with some use case in mind. You test it manually quite a lot before the initial release. You do it basing on some AsyncAPI file, sometimes crafted for the purpose of the code generator, in most cases it is a specific version of the official streetlight example.

The problem comes with the next PRs that follow. New features that are added, etc. You get contributions about the new protocol that gets supported or support for some feature of AsyncAPI that was not yet supported. You can check if unit tests are added for specific code generation to check if good code will be generated. You can even add snapshot tests to check if generated files look exactly as you wanted them to look like. The problem is that you are never sure if generated code will still work for the previous scenario unless you manually generate an app and test it with a given broker.

Description

As a maintainer of specific template that can generate code, I'd like to have feature in Generator or maybe a separate library that I can easily enable in my repository and enable integration tests for my template:

  • Might be that we should just enable this testing feature through https://github.com/asyncapi/generator/blob/master/docs/authoring.md#configuration-file but also might be we need another standalone tool for it (this is most probable)
  • As template developer I do not want to write tests like https://github.com/asyncapi/nodejs-template/blob/master/test/integration.test.js. I want it all provided to me by default:
    • by default have test snapshot test that check if generated files match previous snapshot
    • I should be able to specify location of AsyncAPI file that I want to test against
    • I should be able to specify template parameters used in a given test
    • have the option to opt out certain files from test
    • have the option to specify that I expect a specific file to contain specific "text" inside generate file
  • As a template developer I want to have a solution in place that will take my AsyncAPI file and generate "application", start a broker if needed, and perform a test operation that will evaluate if generated application is really sending or receiving expected message. Maybe we can integrate https://microcks.io/ ?

For GSoC participants

  • you will code with JS or TS
  • you will work on a solution that will be used by template maintainers across AsyncAPI org
  • you will have a chance to learn in details how to write testing library
  • you will have a chance to work with docker, virtualization and testing automation
@derberg derberg added enhancement New feature or request gsoc This label shoudl be used for issues or discussions related to ideas for Google Summer of Code labels Feb 20, 2022
@sudoshreyansh
Copy link

This sounds interesting! I would like to work on this.

@sudoshreyansh
Copy link

I am wondering if I can merge this idea with asyncapi/glee#255. AsyncAPI doesn't have any automated testing tool which can perform automated tests based on the specification files, so I can make one and then integrate it with both Glee and Generator. The tool would be able to be used as a standalone tool, similar to the one mentioned in asyncapi/glee#255. It will also expose the necessary functionality needed for integration.

@derberg @fmvilas I need your opinions on this.

@fmvilas
Copy link
Member

fmvilas commented Feb 25, 2022

🤔 @sudoshreyansh I'm not sure we're talking about the same thing here. If I understood correctly, this issue is about generating integration tests for code generators and the one at Glee is about running integration tests with a specific syntax for Glee. As you say in the other issue "While writing tests, developers should only focus on the message being sent to the server and the response provided by the server or the actions the server does."

That said, it may be me who is not understanding properly, so I'm not saying we shouldn't merge the two issues somehow. However, my experience says it's often preferable to start separated and then merge if we think it makes sense. If we start with a single solution for both and we overlooked some details, we'll end up being conditioned on what's possible in the Generator templates or what's possible in the Glee tests. That's just my 2 cents.

@sudoshreyansh
Copy link

@fmvilas Thanks! I get you. It would be better if both are separate solutions for now.

@Sihamtahi
Copy link

Hello, I want to work on this project. I want to apply my technical knowledge of testing in a real case and have a deep knowledge of this discipline.

@derberg
Copy link
Member Author

derberg commented Jun 6, 2022

@sudoshreyansh this idea was selected for AsyncAPI Mentorship 2022 -> asyncapi/community#376 (comment)

you are the first one that volunteered for this issue + you also contributed to the project a lot by cracking google calendar API for me when I worked on AsyncAPI Meetings automation. It would be great if you decide to be an AsyncAPI-sponsored mentee for this issue 🙏🏼 but I also saw another idea where you volunteered and the idea was selected, so have a look, contact mentor and decide what issue you want to work on during next months

@sudoshreyansh
Copy link

@derberg I thought we had to re-pick our ideas for AsyncAPI mentorship after GSoC.
Sorry for the confusion. I would like to go with my application on asyncapi/glee#27

@likitarai1
Copy link

Can I work on this issue??

@derberg
Copy link
Member Author

derberg commented Jun 7, 2022

let us wait for @fmvilas decision on asyncapi/glee#27

@derberg
Copy link
Member Author

derberg commented Jun 7, 2022

because @sudoshreyansh works on asyncapi/glee#27 and @Sihamtahi will work on asyncapi/extensions-catalog#78 I will remove (resign as mentor) this issue from the current Mentorship Program in favor of asyncapi-archived-repos/design-system#4 that is complex and requires 2 mentees (and has 2 mentors)

@likitarai1 sorry but the rule was that you need to apply before deadline, so we get people that are really interested in a specific issue. I also do not recall you contributing to the project before, and it is different in the case of folks from asyncapi-archived-repos/design-system#4

@github-actions
Copy link
Contributor

github-actions bot commented Oct 6, 2022

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Oct 6, 2022
@derberg
Copy link
Member Author

derberg commented Oct 11, 2022

Still valid

@github-actions github-actions bot removed the stale label Oct 12, 2022
@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Feb 10, 2023
@derberg derberg removed the stale label Mar 27, 2023
@chinma-yyy
Copy link

Hey @derberg , Can we again propose this for the mentorship program this year? I would love to work on this project.

@derberg
Copy link
Member Author

derberg commented May 23, 2023

I'm not really sure, this is a big topic, and I proposed 2 others already + have another GSoC topic

Best if someone else from the community could mentor it

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Sep 21, 2023
@AyushNautiyalDeveloper
Copy link

@derberg could i work on this issue.

@github-actions github-actions bot removed the stale label Oct 25, 2023
@AayushSaini101
Copy link
Contributor

@derberg I am interested in this project Because I recently worked on one of the major features in the generator and we understood the issue that we faced due to testing the changes.

@derberg derberg removed the gsoc This label shoudl be used for issues or discussions related to ideas for Google Summer of Code label Feb 13, 2024
@derberg
Copy link
Member Author

derberg commented Feb 13, 2024

@AayushSaini101 I removed gsoc label. It is not a good issue for mentorship. It is way more complex than I thought when I defined the issue. We learned how complex it is when we tried to do some good integration testing in nodejs template

Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Jun 13, 2024
@derberg
Copy link
Member Author

derberg commented Aug 1, 2024

we should try to look into it again once we start onboarding first templates in generator

@Gmin2
Copy link
Collaborator

Gmin2 commented Aug 1, 2024

Hey @derberg can you suggest me some next steps that i can look into, i am deeply interested in it

@github-actions github-actions bot removed the stale label Aug 2, 2024
Copy link
Member Author

derberg commented Aug 3, 2024

@Gmin2 but next steps regarding onboarding templates, or regarding testing lib?

@Gmin2
Copy link
Collaborator

Gmin2 commented Aug 4, 2024

Regarding testing libraries 🙂

@Gmin2
Copy link
Collaborator

Gmin2 commented Aug 18, 2024

any thoughts @derberg

Copy link
Member Author

derberg commented Aug 19, 2024

yeah, we first need to onboard template to generator, and then we can think of more sophisticated end2end testing

Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request stale
Projects
None yet
Development

No branches or pull requests

9 participants