You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I ran Acto for 3 Hours. It ran 65 tests, and produced 25 alarms
All of the alarms are from our system state oracle.
For 19 out of 25: Acto didn't find any matching field in system state deltas for the input delta.
For 6 out of 25 alarms: Acto found some matching fields, but the value change is different
18 false alarms caused by no matching field in system state deltas
3 are caused by changing a complex object: When we change a complex object to null, changes are reflected on a lower level.
Concretely, consider the following example where we changed the secretBackend from null to a new object.
Acto tries to find a matching field based on the input delta's path ['spec']['secretBackend'], but the system state delta is at lower level.
In the system state delta, the path is ...['annotations']['key']. To match these two fields, we need to flatten the dict in the input delta before field matching.
10 are caused because we performed a change which is rejected by the operator: scale down and shrink volumn, key-value delimiter not found.
1 is caused by changing from a default value to null (this is effectively no change, but we were not aware of the default value). Need to be aware of default values
1 is caused by that field does not affect application's state (configuration of the operator itself)
2 are caused by a bug in our input generation
1 need further inspection
6 false alarms caused by value mismatch
3 are caused by lack of canonicalization when comparing dictionaries: easy to fix, canonicalize fields when comparing dict
Need canonicalization when comparing dictionaries requiredDuringSchedulingIgnoredDuringExecution != required_during_scheduling_ignored_during_execution
I ran Acto for 3 Hours. It ran 65 tests, and produced 25 alarms
All of the alarms are from our system state oracle.
For 19 out of 25: Acto didn't find any matching field in system state deltas for the input delta.
For 6 out of 25 alarms: Acto found some matching fields, but the value change is different
1 true alarm
See here: #39
18 false alarms caused by no matching field in system state deltas
3 are caused by changing a complex object: When we change a complex object to null, changes are reflected on a lower level.
Concretely, consider the following example where we changed the
secretBackend
fromnull
to a new object.Then we have the following system state delta:
Acto tries to find a matching field based on the input delta's path
['spec']['secretBackend']
, but the system state delta is at lower level.In the system state delta, the path is
...['annotations']['key']
. To match these two fields, we need to flatten thedict
in the input delta before field matching.10 are caused because we performed a change which is rejected by the operator: scale down and shrink volumn, key-value delimiter not found.
1 is caused by changing from a default value to null (this is effectively no change, but we were not aware of the default value). Need to be aware of default values
1 is caused by that field does not affect application's state (configuration of the operator itself)
2 are caused by a bug in our input generation
1 need further inspection
6 false alarms caused by value mismatch
3 are caused by lack of canonicalization when comparing dictionaries: easy to fix, canonicalize fields when comparing dict
Need canonicalization when comparing dictionaries
requiredDuringSchedulingIgnoredDuringExecution
!=required_during_scheduling_ignored_during_execution
2 are caused by comparing null to default value: need to be aware of default values when comparing with null
resulted:
1 is caused by
0 != '0'
: easy to fixThe text was updated successfully, but these errors were encountered: