Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed 13 flaky tests #107

Merged
merged 2 commits into from
Mar 31, 2022
Merged

Conversation

alany9552
Copy link
Contributor

@alany9552 alany9552 commented Dec 1, 2021

Description of the problem

These tests are found flaky:

io.castle.client.model.CastleContextTest.minimalContextAsJson
io.castle.client.model.CastleDeviceTest.fullBuilderJson
io.castle.client.model.CastleDeviceTest.jsonSerialized
io.castle.client.model.CastleMessageTest.fullBuilderJson
io.castle.client.model.CastleMessageTest.jsonSerialized
io.castle.client.model.CastleMessageTest.properties
io.castle.client.model.CastleUserAddressTest.jsonSerialized
io.castle.client.model.CastleUserDeviceContextTest.fullBuilderJson
io.castle.client.model.CastleUserDevicesTest.fullBuilderJson
io.castle.client.model.CastleUserDeviceTest.fullBuilderJson
io.castle.client.model.CastleUserDeviceTest.jsonSerialized
io.castle.client.model.CastleUserTest.jsonSerialized
io.castle.client.model.DeviceUserAgentTest.fullBuilderJson

when running mvn -pl ./ edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=TESTNAME .

Here is an example of the error log:

[ERROR] Failures:

[ERROR]   CastleMessageTest.fullBuilderJson:72 expected:<"{"[created_at":"2018-01-01","timestamp":"2018-01-01","device_token":"1234","event":"event","properties":{"key":"val"},"review_id":"2345","user_id":"3456","user_traits":{"key":"val"}]}"> but was:<"{"[user_id":"3456","user_traits":{"key":"val"},"timestamp":"2018-01-01","review_id":"2345","properties":{"key":"val"},"event":"event","device_token":"1234","created_at":"2018-01-01"]}">

Cause of Flaky

The reason of the flaky tests is because the Gson() library. It can not guarantee the order of the generated string when calling Gson.toJson(), it shuffles the order for each NonDex run.

What I have changed

To avoid the flakiness, I change the testing logic from comparing the hard coded string to comparing the expected and actual value as JsonObject by using JsonParser(). In this case, the order will be ignored and two jsonObject will be compared, to avoid the flakiness.

@alany9552 alany9552 changed the title Fixed 6 flaky tests Fixed 13 flaky tests Dec 2, 2021
@sebastiansimson sebastiansimson merged commit e221156 into castle:master Mar 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants