If Safeguard probe is invalid, CTK just ignores it and continues running. #7
Comments
|
Oh right. The ctk validator is unaware that this particular control uses probes to achieve its goal. So they are not validated. I can't make the validator support that but I could bring the validation into the control itself, when we configure it. |
Sure, as long as it stops the experiment and throws an error, would be fantastic :) |
|
Yeap. I can't work on this today but I'll try tomorrow, ping me otherwise :) |
|
Awesome thank you! @Lawouach |
Signed-off-by: Sylvain Hellegouarch <sh@defuze.org> #7
|
Hey @WixoLeo could you try the latest commit on master and see if that works? |
Hi, sorry for late response, I tested it, and still the same problem for some reason |
|
Python: __all__ = [
"some_probe"
]
def some_probe():
print("Hello mate")
return Trueexp.json {
"version": "1.0.0",
"title": "test",
"description": "test",
"configuration": {
"myarg": "exp.json",
"another_arg": "another.json"
},
"variables": {
"selected_instance": null
},
"tags": [
"network",
"lala"
],
"controls": [
{
"name": "Safeguards",
"provider": {
"type": "python",
"module": "chaosaddons.controls.safeguards",
"arguments": {
"probes": [
{
"name": "Non existing probe",
"description": "Probe that doesn't exist",
"type": "probe",
"provider": {
"type": "python",
"module": "testoplugin.testo.probes1",
"func": "some_probe",
"arguments": {}
},
"tolerance": true
}
]
}
}
}
],
"steady-state-hypothesis": {
"title": "Check something",
"probes": [
{
"name": "Non existing probe",
"description": "Probe that doesn't exist",
"type": "probe",
"provider": {
"type": "python",
"module": "testoplugin.testo.probes",
"func": "some_probe",
"arguments": {}
},
"tolerance": true
}
]
},
"method": [
{
"name": "My probe",
"description": "My best probe",
"tolerance": {
"type": "regex",
"pattern": "exp.json",
"target": "stdout"
},
"type": "probe",
"provider": {
"type": "process",
"path": "echo",
"arguments": "${myarg}"
}
}
],
"rollbacks": [
{
"name": "My probe",
"description": "My best probe",
"tolerance": {
"type": "regex",
"pattern": "exp.json",
"target": "stdout"
},
"type": "probe",
"provider": {
"type": "process",
"path": "echo",
"arguments": "${myarg}"
}
}
]
}``` |
|
Found the reason why my fix wasn't enough. I'll have to update the ctk core lib and release it for this. On it. |
|
Once this one is merged and released, i'll update this repo accordingly chaostoolkit/chaostoolkit-lib#225 |
Awesome, thank you @Lawouach, absolute pro! |
|
Hey @WixoLeo I have updated the master branch with the changes from ctklib 1.20.0, feel free to give it a spin :) |
So I've tested it, and it seems like it's still not working.. |
|
Could you upload a ctk log file here and your experiment again? |
{
"version": "1.0.0",
"title": "test",
"description": "test",
"configuration": {
"myarg": "exp.json",
"another_arg": "another.json"
},
"variables": {
"selected_instance": null
},
"tags": [
"network",
"lala"
],
"controls": [
{
"name": "Safeguards",
"provider": {
"type": "python",
"module": "chaosaddons.controls.safeguards",
"arguments": {
"probes": [
{
"name": "Non existing probe",
"description": "Probe that doesn't exist",
"type": "probe",
"provider": {
"type": "python",
"module": "bla",
"func": "some_probe",
"arguments": {}
}
}
]
}
}
}
],
"steady-state-hypothesis": {
"title": "Check something",
"probes": []
},
"method": [],
"rollbacks": []
} |
|
The log says:
Maybe you're not updated properly? |
|
I've done a clean install, this happened: If I remove the probe and leave just the control in, it's going good |
|
Good and bad. Good because it uses now the chaosaddon as expected, bad because that chaosaddon master is broken in your case. Let me look. |
|
I think this is fixed in 98bcd4a |
|
Amazing! It's working! Thank you very much! @Lawouach Absolute pro! |
Here is an example of a simple experiment:
{ "version": "1.0.0", "title": "test", "description": "test", "configuration": { "myarg": "exp.json", "another_arg": "another.json" }, "tags": [ "network", "lala" ], "controls": [ { "name": "Safeguards", "provider": { "type": "python", "module": "chaosaddons.controls.safeguards", "arguments": { "probes": [ { "name": "Non existing probe", "description": "Probe that doesn't exist", "type": "probe", "provider": { "type": "python", "module": "blablabla", "func": "bla", "arguments": {} }, "tolerance": 2 } ] } } } ], "steady-state-hypothesis": { "title": "Check something", "probes": [ { "name": "My probe", "description": "My best probe", "tolerance": { "type": "regex", "pattern": "exp.json", "target": "stdout" }, "type": "probe", "provider": { "type": "process", "path": "echo", "arguments": "${myarg}" } } ] }, "method": [ { "name": "My probe", "description": "My best probe", "tolerance": { "type": "regex", "pattern": "exp.json", "target": "stdout" }, "type": "probe", "provider": { "type": "process", "path": "echo", "arguments": "${myarg}" } } ], "rollbacks": [ { "name": "My probe", "description": "My best probe", "tolerance": { "type": "regex", "pattern": "exp.json", "target": "stdout" }, "type": "probe", "provider": { "type": "process", "path": "echo", "arguments": "${myarg}" } } ] }``` Notice the probe with module and function as "blabla" Obviously it doesn't exists. But if you run the experiment, it just runs normally by ignoring that probe. If I put the real probe there, it works fine. I think the expected result would be some kind of an exception and abort everything. We've been running experiments for a long time without knowing that we've had a non working probe.The text was updated successfully, but these errors were encountered: