Support '--info' to pipe info logs to stdout #261
Conversation
|
It looks like @chia7712 hasn't signed our Contributor License Agreement, yet.
You can read and sign our full Contributor License Agreement here. Once you've signed reply with Appreciation of efforts, clabot |
There was a problem hiding this comment.
I actually like the direction of the patch generally to shift away from debug parameters to more general log_level parameters, though test.py changes in particular would need compatibility tests since I'm pretty sure the change of parameter names would be breaking (e.g. a test against kafka's existing tests; but I haven't checked the exact function calls). Even more generally, rather than continuing to add to the public interface w/ more command line args that are specific to log levels, having a way to feed in a more general logging config would probably be better long term.
That said, we should separate the issue from the solution. The bug report has to do just with timeouts, which can also be solved in other ways. In this case it seems lack of regular output is the issue. Outputting more stuff might help, but does it ultimately fix the problem or just hack around it temporarily? For example, if parallelism=1 and a test takes > 10min, I'm not sure this helps. Additionally, outputting test INFO logs to stdout could have a significant impact on log size for a test run.
In contrast, it seems the need here is simply for something (possibly with a newline and ensuring flushing, which is a common requirement) to be written to stdout periodically to indicate progress. Another way of accomplishing this would be periodic status updates, even if they just repeat the same info (e.g. Running test 12 of 52...). It's been awhile since I looked at that code, but that might be doable in just a few lines of runner.py if timeouts were applied to certain calls, specifically the ones waiting for the test runners to complete (the recv call I believe).
Have we looked into that option? Can we just ensure TestRunner emits something periodically that will keep CI and other systems happy knowing it is doing something?
|
@ewencp Thanks for your quick response! For this timeout issue, it seems to me the control of outputting something to make CI happy should be up to ducktape users, because they are able to set the suitable/meaningful output. It is nice that TestRunner emits something periodically for Kafka but it could be noisy to other users who don't require those periodical outputs. Personally, adding a "configurable" option to enable/disable much/less messages is more flexible and it can be transparent to "non-kafka" users. WDYT?
oh, I didn't notice related function calls. Will check them later. |
|
I think both --info and periodic printing of status updates are useful in their own ways. When running tests locally via ducker-ak, for example, it would be useful to see all info output. So if I understand the discussion above correctly, there are two open quesions:
|
|
|
fix #260