Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ShellTest is non-deterministic #107

Closed
julianhyde opened this issue Mar 1, 2022 · 0 comments
Closed

ShellTest is non-deterministic #107

julianhyde opened this issue Mar 1, 2022 · 0 comments

Comments

@julianhyde
Copy link
Collaborator

Some of the tests in ShellTest fail with non-deterministic errors. Here is an example:

Error:  testEmptyLines  Time elapsed: 0.194 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: is "(* a comment followed by empty *)\r\n\r\n;\r\n- (* a comment followed by empty *)\r\r\n- \r\r\n- ;\r\r\n- \r\r\n"
     but: was "- (* a comment followed by empty *)\r\n\r\n;\r\n(* a comment followed by empty *)\r\r\n- \r\r\n- ;\r\r\n- \r\r\n"
	at net.hydromatic.morel.ShellTest.testEmptyLines(ShellTest.java:134)

The cause seems to be a race-condition in JLine. Usually the echoed command appears first in the output, but occasionally the output appears first. Note that the "- " prompt has changed positions in the expected versus actual output.
julianhyde added a commit that referenced this issue Mar 1, 2022
@julianhyde
[MOREL-107] ShellTest is non-deterministic
4093a65 
The problem seems to be that JLine has a race-condition.

The solution is for the GitHub action to set an environment
variable 'morel.ci' when running 'mvn test'; the
non-deterministic tests in ShellTest check that variable and
skip if it is set.

When run from the command-line, all tests in ShellTest will
run as normal. This ensures that they get run occasionally.
Non-determinism seems to be less of a problem from the
command-line, perhaps because there are more cores available,
and perhaps because it is easier to re-run.

Fixes #107
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@julianhyde