-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
exit code for spec failures is the same as exit code for any errors #154
Comments
This seems plausible on the surface, but I have a few questions and concerns I'd like to talk about more before we go forward with this kind of change:
I'm open to doing something about this, but wanted to make sure everyone involved understands the issues and limitations with a potential implementation before putting more effort in. Hope this helps. Thanks for using Jasmine! |
For e.g. Karma + Jenkins, this seems to have been handled by Making this a documented configuration option that exits with code 0 in the
Spec expectation failures are a normal part of the development workflow, they're expected and the CI tooling needs to be able to identify and cleanly report those to the developer that submitted the build, so that they can easily identify and fix them. For complex CI builds with multiple spec suites (backend/frontend, jasmine/others), it's also desirable to continue the build in the case of normal spec failures, so that all of the spec failures can be reported to the developer, rather than stopping the entire build and only reporting the first spec suite failure. The way this is currently handled is to ignore the Test tooling failures caused by e.g. the build machine running out of disk space are not normal or expected, and they should (ideally) immediately abort the entire build. They're also not the developer's fault, and they're not necessarily the ones responsible for fixing those issues, so having a separate build status makes more sense. |
When running jasmine tests in a CI environment, it would be useful to distinguish between test failures and any other errors in test execution. Currently, it seems like the
jasmine
exit status is always 1 for both spec failures as well as unhandled nodejs exceptions.Ideally (using Jenkins terms), things like syntax errors in the test specs should result in a red FAILURE status, whereas spec failures should result in an yellow UNSTABLE status with associated JUnit test results.
This would be easy to implement if the exit code for spec failures were to be e.g. 2, instead of the default exit code 1 that node uses for unhandled errors (process Event uncaughtException).
Exit code with spec failures
This is presumably from the jasmine runner, and could be customized to use a different exit code?
jasmine-npm/lib/jasmine.js
Lines 217 to 222 in dd00f4b
Exit code with e.g. syntax error
This is presumably the default node
uncaughtException
->exit 1
case.The text was updated successfully, but these errors were encountered: