Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run only selective tests #1343

Open
GagandeepSharma060 opened this issue Oct 8, 2024 · 9 comments · May be fixed by #1349
Open

Run only selective tests #1343

GagandeepSharma060 opened this issue Oct 8, 2024 · 9 comments · May be fixed by #1349
Assignees
Labels
enhancement Improvement to an existing feature

Comments

@GagandeepSharma060
Copy link

Hi team , the filters thing that is mentioned in the documentation. Is it also possible if I am not using junit/java for my tests and if i just have the yaml files -> mention it in the specmatic.json and then run it through command line. I tried multiple ways but was not able to do so

"start": "cross-env-file -p ./token.json  specmatic test --testBaseURL=https://abc.com  --junitReportDir=./test-output && node cleanup"

This is the start command I am using, need urgent help

@GagandeepSharma060
Copy link
Author

And when we write --filter-name "xyz" , where should this xyz be present for that endpoint (example) to be hit or not hit. Please provide example. It will be great as I am stuck for this implementation

@harikrishnan83 @nashjain @rakeshpai

@pranavgawri
Copy link
Contributor

@GagandeepSharma060 thanks for sharing issue you are facing. We will update you shortly.

@GagandeepSharma060
Copy link
Author

@pranavgawri found out that the filters are applied on the path name itself. It is great but can we have some tagging kind of mechanism like we have in Cypress.

Because filtering on the path names does not seem to be a great idea when dealing with a lot of services.

Just a thought.

@pranavgawri
Copy link
Contributor

pranavgawri commented Oct 9, 2024

@GagandeepSharma060, Thank you for sharing your experience and suggestion. We're glad you found a workable solution with path name filtering.
Your idea about implementing a tagging system similar to Cypress is interesting and could indeed be beneficial for filtering tests across multiple services. We've noted this as a enhancement for future consideration.
We appreciate your feedback as it helps us improve Specmatic. Thanks.

@pranavgawri pranavgawri added the enhancement Improvement to an existing feature label Oct 9, 2024
@pranavgawri pranavgawri self-assigned this Oct 14, 2024
@pranavgawri
Copy link
Contributor

Hi @GagandeepSharma060 ,

We have enhanced our --filter option and I hope it should be able to help you better in your use case. Here is a quick overview:

The new filtering system provides powerful capabilities that align well with API contract testing:

  1. Filter by HTTP Method and Path:
specmatic test --filter="METHOD=POST" --filter="PATH=/api/employees"
  1. Filter by Status Codes (including pattern matching):
specmatic test --filter-not="STATUS=4xx"  # Excludes all client errors
  1. Advanced Combinations:
specmatic test --filter="PATH=/api/employees" --filter="METHOD=POST" --filter-not="STATUS=400,401,403"

These filters work directly with your API contract structure, making it natural to organize and run your tests based on your API's design.

Please find the full details in our documentation here: https://specmatic.io/documentation/contract_tests.html#selectively-running-tests-in-ci

Let us know how this works for you!

Thanks,
Pranav

@GagandeepSharma060
Copy link
Author

GagandeepSharma060 commented Oct 24, 2024

Thanks for the quick response and change @pranavgawri , will try it out and get back to you for feedback or any issues .

I believe this feature would help others too

Just a question, can we filter through description of the test .... as in that way it will give a huge flexibility

By description I mean we can have a dedicated field or use the fields like description or summary for a path
image

@pranavgawri
Copy link
Contributor

Hi @GagandeepSharma060,

Thanks for the feedback about filtering by description/summary fields. I've opened a discussion thread to explore this further: Filtering Tests by Description/Summary Fields

Please take a look and share your thoughts there. Once you've reviewed the discussion, please feel free to go ahead and close this original issue.
Appreciate you helping us improve Specmatic!

Thanks,
Pranav

@GagandeepSharma060
Copy link
Author

Hi @pranavgawri ...Thanks for the response. Justwanted to let you know that filtering by the Example name is not working for me..

Let me explain

--filter="EXAMLPE_NAME=ABC" is not working however. --filter-not="EXAMLPE_NAME=ABC" is working fine.

Can u please look into it

@pranavgawri
Copy link
Contributor

Hi @GagandeepSharma060 ,

Thanks for sharing the issue you are facing. We are able to replicate it at our end, and will push the fix soon.

I will update you on the same.

Thanks,
Pranav

@pranavgawri pranavgawri linked a pull request Dec 11, 2024 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement to an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants