forked from rsyslog/rsyslog
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
11890 lines (11858 loc) · 662 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
------------------------------------------------------------------------------
Version 8.34.0 [v8-stable] 2018-04-03
- omelasticsearch: add support for CA cert, client cert auth
This allows omelasticsearch to perform client cert based authentication
to Elasticsearch.
adds parameters: tls.cacert, tls.mycert, tls.myprivkey
Thanks to Richard Megginson for the patch.
- stream/bugfix: memory leak on stream open if filename as already generated
this can happen if imfile reads a state file. On each open, memory for the
file name can be lost.
We detected this while working on imfile refactoring, so there is no related
bug report. No specific test has been crafted, as the refactored imfile
tests catch it (as soon as they are merged).
- bugfix imfile: memory leak in readMode 0
closes https://github.com/rsyslog/rsyslog/issues/2421
- bugfix omfile: some error messages had parameters in wrong order
which made the message look strange, but still readable
Thanks to Hans Rakers for the patch.
- bugfix testbench: some test were accidently not executed
Thanks to Kasumi Hanazuki for the patch.
- fix some cosmetic issues found by lgtm.com static code analyzer
e.g. header file guard not correctly set - if you really are interested in
details, check git log
- CI
* add lgtm.com static analyzer (automatically called via GitHub PR)
* improved stability of global-umask.sh test, which unnecessarily used
wildcards for test output file checking.
Thanks to Kasumi Hanazuki for the patch.
* added some test for omprog with transactional interface
Thanks to Joan Sala for the new tests.
------------------------------------------------------------------------------
Version 8.33.1 [v8-stable] 2018-04-03
- 8.33.0 tarball release was actually pre-8.33.0
... and so did not contain all features. This alone made a re-release
necessary.
- devcontainer: use some more sensible defaults
and add ability to specify generic docker run options
this makes integration into CI (and other scripting) easier
- fix problems with make dist
Thanks to Thomas D. (whissi), and Michael Biebl for alerting us on the
problem, providing advise and some of the patches.
------------------------------------------------------------------------------
Version 8.33.0 [v8-stable] 2018-02-20
- auto-detect if running inside a container (as pid 1)
adjust defaults in this case to be more container-friendly
- config: add include() script object
This permits to include files (like legacy $IncludeConfig) via a
script object. Needless to say, the script object offers more
features:
- include files can now be
- required, with rsyslog aborting when not present
- required, with rsyslog emitting an error message but otherwise
continuing when not present
- optional, which means non-present include files will be
skipped without notice
This is controlled by the "mode" parameter.
- text can be included form e.g. an environment variable
--> ex: include(text=`echo $ENVVAR`)
This finally really obsoletes $IncludeConfig.
closes https://github.com/rsyslog/rsyslog/issues/2151
- template: add option to generate json "container"
This enables easy JSON generation via template.
This commit also corrects an issue with the constant "jsonf"
format. That was recently added, and the implementation problem
only became visible when used inside a larger json object. No
officially released code is affected, thought - so it rellay
is just a side-note.
closes https://github.com/rsyslog/rsyslog/issues/2347
- core/template: add format jsonf to constant template entries
closes https://github.com/rsyslog/rsyslog/issues/2348
- config: add ability to disable config parameter ("config.enabled")
For auto-generated configs, it is useful to have the ability to disable some
config constructs even though they may be specified inside the config. This
can now be done via the ```config.enabled``` parameter, applicable to all
script objects. If set to ```on``` or not specified, the construct will be
used, if set to any other value, it will be ignored. This can be used
together with the backtick functionality to configure enable and disable
from either a file or environment variable.
closes https://github.com/rsyslog/rsyslog/issues/2431
- script: permit to use environment variables during configuration
new constant string type "backticks", inspired by sh
(sample: `echo $VARNAME`).
- new global config parameter "shutdown.enable.ctlc"
permits to shutdown rsyslog via ctl-c; useful e.g. in containers
- config optimizer: detect totally empty "if" statements and optimize
them out
- template: constant entry can now also be formatted as json field
This enhancements permits to craft clean templates that generate JSON,
e.g. for ElasticSearch consumption (or any other REST API)
- omstdout: support for new-style configuration parameters added
- core: set TZ on startup if not already set
In theory, TZ should be set by the OS. Unfortuantely, this seems
to be not the case any longer on many Linux distros. We now check
it and set it appropriate if not already given.
Thanks to github user JPvRiel for providing an excellent explanation
of the reasoning for this and how to work around it.
closes https://github.com/rsyslog/rsyslog/issues/2054
- imjournal bugfix: file handle leak during journal rotation
Thanks to Peter Portante for the patch
see also: https://github.com/rsyslog/rsyslog/pull/2437
- lmsig_ksils12 bugfix: dirOwner and dirGroup config was not respected
- script bugfix: replace() function worked incorrectly in some cases
If the end of the message was similar to the replacement string, parts
of the string could (not always) be missing.
Thanks to Yaroslav Bo for the patch.
- build system bugfix: --disable-libcurl did not work
Thanks to Dan Molik, Thomas D. (whissi), and Michael Biebl for the patches.
- fixed build issues on Alpine Linux
- core bugfix: misadressing in external command parser
This parser is used whenever a module (e.g. omprog) needs to process
command lines. If command parameters were given, memory misadressing
occured. This could lead to a segfault.
This is a regression in 8.32.0.
closes https://github.com/rsyslog/rsyslog/issues/2408
- core bugfix: small memory leak in external command parser
This parser is used whenever a module (e.g. omprog) needs to process
command lines. On each action definitions for actions that use the
parser a small amount of memory was leakd. It is an uncritical leak
as it only occurs during config parsing. So it leaks a couple of
KiB during startup but does not grow during actual message processing.
This is a regression in 8.32.0.
- core bugfix: string not properly terminated when RFC5424 MSGID is used
This could lead to misadressing when the jsonmesg property was used.
closes https://github.com/rsyslog/rsyslog/issues/2396
- bugfix: strndup() compatibility layer func copies too much
The function did not obey the upper limit, effectively becoming
a strdup(). This was only noticed when the compatibility layer
was required, most importantly on Solaris 10.
- CI system
- we now use well-defined containers for parts of the CI runs
- now also build test under Alpine Linux
- test added for omprog feedback feature
------------------------------------------------------------------------------
Version 8.32.0 [v8-stable] 2018-01-09
- NEW BUILD REQUIREMENTs:
* libfastjson 0.99.8 is now required; older versions lead to bugs in rsyslog
* libczmq >= 3.0.2 is now required for omczmq
This was actually required for quite some while, but not properly checked
during configure run. If the lib was older, build failed. Now configure
detects this and emits the appropiate error message.
* libcurl is now needed for rsyslog core
due to new script function http_request(). This can be turned off by the
new configure option --disable-libcurl. If so, http_request() is not
available.
- rsyslogd: add capability to specify that no pid file shall be written
Command line option -iNONE provides this capability. This utilizes the
pre-existing -i option, but uses the special name "NONE" to turn of the
pid file check feature. Turning off is useful for systems where this no
longer is needed (e.g. systemd based).
closes https://github.com/rsyslog/rsyslog/issues/2143
- ompgsql: considerable enhancements
The PostgreSQL output module was woefully out-of-date the following
list is changes made to update the module to current Rsyslog standards.
* allow for v6 configuration syntax
* configurable ports
* support transactional interface
* push db connection into workers (libpq is threadsafe)
* enable module testing on travis
* ensure configuration syntax backwards compatibility
* formatting around postgres core templating
* use new test conventions
* add new configuration syntax test
* add valgrind tests for new and old syntax
* add threading tests
* add action queue long running tests
* add action queue valgrind test
Thanks to Dan Molik for contributing this great enhancement!
- build system: removed --enable-rtinst configure option
This was a historic debugging option which has been superseeded by
newer tools like valgrind, helgrind, ASAN, TSAN, etc...
- pmrfc3164: support for headerless messages
pmrfc3164 now detects if the first non-SP, non-HT character is either
'{' or '[' and if so assume that no header (TAG, HOSTNAME, DATE) is
given. If so, it uses defaults for these values. The full message is
used as the MSG part in that case. Note that an initial PRI may still
be specified.
This follows the trend to send JSON messages via syslog without any
header. We use default header values in this case.
This right now is a bit experimental; we may roll it back if
problems show up in practice.
closes https://github.com/rsyslog/rsyslog/issues/2030
- omhiredis: add option to use RPUSH instead of LPUSH
see also https://github.com/rsyslog/rsyslog/issues/1625
- mmexternal improvements
* better error reporting if to-be-executed script cannot be executed
* some general cleanup
* better redirection of stdin/out/err for the executed script
* bugfix: argv[0] of the script was missing if command line parameters
were not specified
- omprog: refactored, code shared with mmexternal moved to common object
- logctl tool: refactor to support newer rsyslog standards
* Made the logctl usertool ISO C90 compliant.
* Made the logctl usertool use a homogeneous coding style.
Thanks to Hugo Soszynski for contributing this work (as well as
suggesting some workarounds related to libmongoc/libbson).
- imfile: added support for Solaris File Event notification (FEN)
also improves performance under Solaris, with FEN imfile provides
features equivalent to inotify mode
- core/action: new parameter "action.errorfile"
permits to write failed messages to an "error file" so that they
can be reviewed and potentially be reprocessed
- imfile: added new module parameter "sortFiles"
This permits to process newly created files in sorted order.
- imuxsock: improved status reporting: socket name received from systemd
Providing an indication of what we got from systemd facilitates problem
analysis.
- build system: added new testbench configure switches
now --enable-testbench1 and --enable-testbench2 exists which permit
to enable/disable parts of the testbench. By default, both are on
when --enable-testbench is given. For full testbench coverage, both
options must be given. These options are necessary because under
Travis we hit the max runtime for tests and needed to split tests
over multiple incarnations.
- mmpstrucdata: new parameter "sd_name.lowercase"
to permit preserving case for structured data identifiers
Special thanks to github user alanrobson for the intial commit that
preseves case (on which we based other work).
- omfile: add module-global option "dynafile.donotsuspend"
this permits to enable SUSPENDing dynafile actions. Traditionally,
SUSPEND mode was never entered for dynafiles as it would have blocked
overall processing flow. Default is not to suspend (and thus block).
closes https://github.com/rsyslog/rsyslog/issues/2236
- testbench: add a capability to turn off libfaketime tests via configure
Unfortunately, libfaketime becomes more and more problematic in newer
versions and causes aborts on some platforms. This provides the ability
to turn it off via --disable-libfaketime.
In the longer term, we should consider writing our own replacement.
- testbench: name valgrind tests consistently
all valgrind tests now end in -vg.sh
- RainerScript: add function parse_json()
- RainerScript: add function substring()
- RainerScript: add function http_request()
- RainerScript: add function previous_is_suspended()
This function returns a boolean indicating if the previous action is
suspended (0 - no, 1 - yes). This is useful if arbitrary processing
(other than calling actions) should be made depending on that state.
A prime example for this is probably calling a ruleset.
closes https://github.com/rsyslog/rsyslog/issues/1939
- Patches from BSD projects have been imported
... as far as they still apply. Some patches that patched BSD-specific
code were broadened to be more generic.
- script bugfix: invalid function names were silently ignored
no error message was produced
thanks to Matt Ezell for providing the patch.
- rainerscript: add int2hex() function
- rainerscript: add is_time() function
Thanks to Stephen Workman for implementing this.
- RainerScript: add function script_error() and error-reporting support
This permits script functions that could fail to report errors back, so
that the user can handle them if desired. We use an errno-style of
interface. That means script_error() needs to be called after functions
that supports it. It will return either 0 (all OK) or something else
(an error condition).
The commit also modifies the parse_time() function to use the new
interface. First of all, this is useful for users, and secondly we
needed a capability to craft a testbench test.
closes https://github.com/rsyslog/rsyslog/issues/1978
- testbench: fixed build problem of testbench tools under Alpine Linux
- added --enable-libsystemd configure option to enforce use of libsystemd
so we can fail the build on platforms where this is required
- core/glbl: remove long-unused option $optimizeforuniprocessor
This was still available, but had no effect (for ~10 years or so). The
plumbing is now removed. If someone tries to use the option, an
error message is generated.
closes https://github.com/rsyslog/rsyslog/issues/2280
- core/queue: emit better status messages at rsyslog shutdown
this helps to diagnose issue - unfortunately we need more work to ensure
that the messages always make it to the user. This is a start and
hopefully useful at least for the testbench, possibly more.
- fixed a couple of build issues with gcc-7 (in less frequently used modules)
- fixed a couple of build issues on the arm platform (actually raspbian)
- impstats: fix invalid counter definitions for getrusage() reporting
some of the counters were defined as int (32 bit) vs. intctr_t (64 bit).
On some platforms "long" seems to be 64bit, and getrusage() provides
what we store as int via long. So this caused truncation and/or overflow.
This had undefined effects. Most often, everything worked fine
for values smaller than 2^31 but sometimes we got negative values.
closes https://github.com/rsyslog/rsyslog/issues/1517
- imudp bugfix: potential segfault in ratelimiting
The rate-limiter inside imudp was not set to be thread safe, but was
used across multiple threads. This worked in default configuration,
but failed when RepeatedMsgReduction was set to "on".
Note that it in general is a bug to use a rate-limiter in
non-threadsafe mode across multiple threads. This also causes invalid
rate limiting counts in the default case.
closes https://github.com/rsyslog/rsyslog/issues/441
fixes https://github.com/rsyslog/rsyslog/issues/2132
- imptcp bugfix: access to free'ed memory
When notifyconnectionclose was on, a string buffer was accessed immediately
after it was freed (as part of the connection close operation).
Detected by LLVM ASAN.
- mmanon bugfix: fix wrong ipv6 embedded recognition
mmanon recognized IPv6 with embedded IPv4 that have too few (16 bit) fields.
example: 13:abd:45:0.0.0.0
closes https://github.com/rsyslog/rsyslog/issues/2357
- imfile bugfix: not detecting files in directory when wildcards are used.
When directories and files are created at the same time,
imfile may missed subdirs or file if the machine is on high load.
The handling has been enhanced to scan newly created directories ALWAYS for
matching files.
fixes https://github.com/rsyslog/rsyslog/issues/2271
However there still exist problems with multilevel directory configurations.
Details are discussed in https://github.com/rsyslog/rsyslog/issues/2354
Fixes for the remaining issues are expected for 8.33.0.
- script bugfix: improper string-to-number conversion for negative numbers
- core/action bugfix: 100% CPU utilization on suspension of output module
Triggering condition:
* output module using the legacy transaction interface
(e.g. omelasticsearch, omlibdbi)
* output module needs to suspend itself
In these cases, rsyslog enters a busy loop trying to resolve the
suspend condition. The bug is rooted in rsyslog core action code.
This patch fixes it by inserting a 1-second sleep during calls
to the resume handler.
Note: we cannot sleep exactly as long as tryResume needs. This
would require larger refactoring, which probably is not worth for
the legacy interface. The current solution is almost as good, as
the one second sleep has very little overhead on a real system.
Thus we have choosen that approach.
This patch now also ensures that failed messages are properly
handled and do not cause eternal hang.
closes https://github.com/rsyslog/rsyslog/issues/2113
- core/variables bugfix: bare $! cannot be used in set statement
fixes https://github.com/rsyslog/rsyslog/issues/326
- core bugfix: auto commit of actions improperly handled
The internal state variable bHadAutoCommit was handled in thread-unsafe way
and most probably caused (hard to see) issues when action instances were
run on multiple worker threads. It looks like the state variable was
forgotten to move over to worker state when action workers were introduced.
closes https://github.com/rsyslog/rsyslog/issues/2046
- core bugfix: filename length limitation of 199 bytes
file names (including path names) longer than 199 bytes could not be
handled at many places. This has now been uplifted to 4KiB after careful
review for the largest size supported among all relevant platforms.
- core bugfix: undefined behavior due to integer overflow
when searching strings, we may have an (unsigned) interger overflow
which can lead to misadressing.
Detected by clang ubsan.
- core bugfix: race on LocalHostIP property during startup
The way the default was set could lead to a race if e.g. two internal
messages were emitted at startup when the property was not yet set. This
has been seen to happen in practice. It had no bad effect except a very
small stationary memory leak, but made using thread analyzers unreliable
(as it was rightfully detected as a problem).
closes https://github.com/rsyslog/rsyslog/issues/2012
- bugfix: potential segfault on startup
timezone info table was "sorted" even though it may be NULL. There is
no practical case known where this lead to an actual abort, but in
theory it would be possible. If so, it would happen immediately on
startup.
Detected by clang ubsan.
- omhiredis bugfix: rsyslog segfault on startup if no template is specified
- omprog bugfix: argv[0] not set when using binary without arguments
When using the omprog plugin with a binary without arguments, argv[0] (the
name of the binary) is not set, causing binaries that depend on this value
being set to crash or misbehave.
This patch also mildly refactors omprog argv[] creations, albeit some more
refactoring would be useful.
closes https://github.com/rsyslog/rsyslog/issues/1858
- core: refactoring of rsyslog's cstr "class"
Function cstrGetSzStrNoNULL shall modified the string buffer on each call,
albeit it is considered a "read-only" function. It currently adds a '\0'
at the end. This is bad, especially when multiple threads access the same
string. As always the same data is written, it was not causing real issues
(except unnecessary cache writes), but it polluted the thread debugger and
as such prevent more elaborate automatted tests.
- parent directory creation function refactored
This should not cause any change of behavior, but is worth noting in case
we see a regression not caught by the CI system.
- mmsnmptrapd bugfix: potential misadressing
This only occured in situations when the system was totally out of memory.
- imkafka: fix potential small ressource leak
If rdkafka handle cannot fully populated, cleanup is added. Previously, we
could leak a handle if e.g. no brokers were avaiable. Note that this was
a cosmetic leak, as proper processing is not possible in any case and the
leak is once per listener, so not growing. But we now also ensure that
proper error reporting and handling is done in any case. Previously, we
may have some misleading error messages if the defunct kafka handle was
used.
closes https://github.com/rsyslog/rsyslog/issues/2084
- imkafka bugfix: do not emit error message on regular state
This was misunderstood as an error state and could spam the system
log considerably. Regression from 8.31.0.
- omkafka: expose operational status to user where useful
omkafka emits many useful operational status messages only to the debug
log. After careful review, we have exposed many of these as user error
and warning message (ex: librdkafka queue full, so user knows why we
suspend the plugin temporarily). This may have made the module too
chatty. If so, one can always filter out messages via configuration. And
if we really went overboard, we can undo those parts with the next
release. So it's better to give a bit more than less, as this definitely
eases troubleshooting for users.
closes https://github.com/rsyslog/rsyslog/pull/2334
- omkafka bugfix: potential message duplication
If a message that already failed in librdkafka was resubmitted and that
resubmit also failed, it got duplicated.
- omkafka: fix multithreading
omkafka has several issue if multiple worker instances are used. This commit
actually make the module use a single worker thread at max. Reasoning:
Librdkafka creates background threads itself. So omkafka basically needs to move
memory buffers over to librdkafka, which then does the heavy hauling. As such, we
think that it is best to run max one wrkr instance of omkafka -- otherwise we just
get additional locking (contention) overhead without any real gain. As such,
we use a global mutex for doAction which ensures only one worker can be active
at any given time. That mutex is also used to guard utility functions (like
tryResume) which may also be accessed by multiple workers in parallel.
Note: shall this method be changed, the kafka connection/suspension handling needs
to be refactored. The current code assumes that all workers share state information
including librdkafka handles.
closes https://github.com/rsyslog/rsyslog/issues/2313
- omkafka bugfix: potential misadressing
The failed message list is improperly cleaned. This is a regression
from recent commit 4eae19e089b5a83da679fe29398c6b2c10003793, which
was introduced in 8.31.0.
This problem is more likely to happen under heavy load or bad
connectivity, when the local librdkafka queue overruns or message
delivery times out.
closes https://github.com/rsyslog/rsyslog/issues/2184
closes https://github.com/rsyslog/rsyslog/issues/2067
- omkafka bugfix: build fails with older versions of librdkafka
closes https://github.com/rsyslog/rsyslog/issues/2168
- omgssapi bugfix: fix compiler warnings with gcc-7
closes https://github.com/rsyslog/rsyslog/issues/2097
- dnscache bugfix: entries were cached based on IP AND port number
That hash key which is used to find out already cached dns entry gets
incorrectly computed from the whole sockaddr_storage
(sockaddr_in/sockaddr_in6) structure including a sin_port (which doesn't
have a static value) instead of only an address, thus creating redundant
dns cache entries/consuming more space. This lead to potentially high memory
usage and ineffectiveness of the case. It could be considered a memory leak.
Thanks to Martin Osvald for the patch.
see also: https://github.com/rsyslog/rsyslog/pull/2160
- omkafka bugfix: fixed memory leak
a memory leak occurred when librdkafka communicated error/status information
to omkafka. this seems to happen relatively frequently, so this leak
could be pretty problematic.
- mmdblookup bugfix: replace thread-unsafe strtok() by thread-safe counterpart
Many thanks to Will Storey (github user @horgh) for mentioning this and
his phantastic help in debugging this rsyslog problem!
- pmnormalize bugfix: remove unsave "strcat" implementation
- rainerscript bugfix: ltrim() and rtrim function misadressing
This could lead to a segfault and was triggerred by certain input data
schemes. For example, a ltrim(" a") could lead to the fault.
- imklog bugfix: local host IP was hardcoded to 127.0.0.1
This is now taken from the global localHostIP setting, which is used
consistent accross all modules.
Also, the removed (2012?) directive $klogLocalIPIF has been added
again but directly marked as removed. That way, an informative error
message is generated if someone tries to use it.
closes https://github.com/rsyslog/rsyslog/issues/2276
- cleanup: remove obsolete pre-KSI GuardTime signature interface
this is no longer functional starting Jan 2018 as it reached end of life
closes https://github.com/rsyslog/rsyslog/issues/2128
- cleanup: obsolete defintion SOL_TCP replaced by newer IPPROTO_TCP
this should not have any effect at all except better portability, but is
worth mentioning in the ChangeLog nevertheless.
- lookup tables: fixed undefined behavior detected by UBSan
- CI testing
- ARM (via Raspberry PI) added to CI system
- Debian 9 added to CI system
- omgssapi and usertools components are now also tested in Travis
- test coverage on BSD has been enhanced
------------------------------------------------------------------------------
Version 8.31.0 [v8-stable] 2017-11-28
- NEW BUILD DEPENDENCY: ommongodb now requires libmongo-c
instead of deprecated libmongo-client.
- remove systemd embedded code, use libsystemd instead
Since the early days rsyslog used the orginal systemd embedded
interface code. This version now uses libsystemd where available.
If we do not find libsystemd, we assume the system does not use
systemd, which is a safe assumption nowadays. This ensures we use the
fresh interface code and also removes bloat from our project.
closes https://github.com/rsyslog/rsyslog/issues/1933
- mmanon: add support for IPv6 adresses with embedded IPv4 address
While this format is uncommon, it may still be present in logs and as
such should be supported. It is configurable via individual settings,
though. Especially the number of bits to anonymize may be desired to
be different than in pure IPv6.
- ommongodb: big refactoring, more or less a feature-enhanced rewrite
New features are :
* Handle TLS connexion to mongodb
* Handle MongoDB Replicasets
* Added the 'ssl_ca' and 'ssl_cert' directives to configure tls connexion
* Added 'uristr' directive to configure the connection uri in the form
of 'mongodb://...'
Now uses the official mongo-c-driver library instead of the deprecated
mongo-client library
Special thanks to Hugo Soszynski and Jérémie Jourdin for there hard work
to make this a reality!
See also: https://github.com/rsyslog/rsyslog/pull/1917
- rainerscript: add parse_time() function
Thanks to Stephen Workman for implementing this.
- omelasticsearch: add LF to every elastic response for error file
error file content was written without LF after each message, making
it hard to work with and monitor.
Thanks to Yaroslav Bo for the patch.
- omelasticsearch: add pipeline support
supports static and dynamic ElasticSearch pipeline parameter.
closes https://github.com/rsyslog/rsyslog/issues/1411
Thanks to github users scibi and WaeCo for the implementation.
- lmsig_ksi_ls12: support asynchronous mode of libksi
- omprog: added error handling and transaction support for external plugins
This permits much better integration of external output modules.
Special thanks to Joan Sala for providing this work!
- imzmq3/omzmq3: marked as deprecated, modules will be remove in v8.41
see also: https://github.com/rsyslog/rsyslog/issues/2100
- imzmq3/omzmq3: fixed build issues with gcc-7
- core: emit error message on abnormal input thread termination
This indicates a serious issue of which the user should be notified.
Was not done so far and caused grief when troubleshooting issues.
- core: refactored locking for json variable access
refactored the method so that it consistent in all functions and easier
to follow. Most importantly, now an as simple as possible locking paradigm
of lock ... unlock within the function. Hopefully easier to understand
both for humans and static code analyzers.
- core: refactored creation of UDP sockets
was very old, hard to follow code; streamlined that a bit
- core/dnscache: refactor locking
keep a simple lock ... unlock scheme within the function. That is
easier to follow for both humans as well as static analyzers.
Also removes Coverity scan CID 185419
- rainerscript: use crypto-grade random number generator for random() function
We now use /dev/urandom if available. If not, we fall back to the weak PRNG.
- imkafka: improve error reporting and cleanup refactoring
- imkafka bugfix: segfault if "broker" parameter is not specified
Now emits warning message instead and uses hardcoded default
(which usually matches where the kafka broker can be found).
fixes https://github.com/rsyslog/rsyslog/issues/2085
- omkafka: improve error reporting
- omkafka: slight speedup do to refactoring of LIST class
double-linked list was used for temporarily undeliverable message tracking
where singly-linked list was sufficient. Changed that.
- TCP syslog: support SNI when connecting as a client
This is done if a hostname is configured and turned off if an IP is used.
Thanks to Art O Cathain for the patch.
see also https://github.com/rsyslog/rsyslog/pull/1393
- msg variable bugfix: potential segfault on variable access
A segfault is likely to happen if during processing a variable with
more than one path component is accessed (e.g. $!path!var) and NO
variables oft hat kind (local, message, global) are defined at all.
closes https://github.com/rsyslog/rsyslog/issues/1920
- ratelimiting bugfix: data race in Linux-like ratelimiter
access to the Linux-like ratelimiter was not properly synchronized, and
the system rate-limiter was not using it in any case.
This could lead to the rate-limit settings not being properly
respected, but no "hard error".
- core/template bugfix: potential NULL pointer access at config load
could happen if the config was loaded and the parameters could not properly
be processed. If so, this occured only during the startup phase.
Detected by Coverity scan, CID 185318
- core/json var subsystem bugfix: segfault in MsgSetPropsViaJSON
Invalid libfastjson API use lead to double-free. This was most importantly
triggerred by mmexternal (but could potentially be triggered by other
uses as well)
closes https://github.com/rsyslog/rsyslog/issues/1822
- core/wrkr threads bugfix: race condition
During e.g. shutdown it could happen that a worker thread was started
and immediately be requested to terminate. In such situations there was
a race the prevented proper initialization. This could lead to follow-on
problems.
We believe (but could not proof) that this also could lead to a hang of
the termination process. Thus we claim to close an issue tracker down
here below, but are not 100% certain it really closes it (we hope for
user feedback on this). In any case, the thread debuger showed invalid
operations and this has been fixed, so it is useful in any case.
closes https://github.com/rsyslog/rsyslog/issues/1959
- core/wtp: potential hang during shutdown
when the wtp worker is cancelled in the final stage of shutting down
while the mutex is locked and there is one worker left, the system
will hang. The reason is that the cancelled thread could not free the
mutex that the other needs in order to shut down orderly.
Deteced with clang thread sanitizer.
- omfwd bugfix: generate error message on connection failure
- imtcp bugfix: "streamdriver.mode" parameter could not be set to 0
- imjournal bugfix: module was defunctional
The open function was broken by commit 92ac801, resulting in
no data being ever read from the journal.
patch bases on the idea of Radovan Sroka given here:
https://github.com/rsyslog/rsyslog/issues/1895#issuecomment-339017357
but follows the current imjournal-paradigm of having the journal
handle inside a global variable.
see also https://github.com/rsyslog/rsyslog/issues/1895
closes https://github.com/rsyslog/rsyslog/issues/1897
- imjournal: refactor error handling, fix error messages
imjournal did not try to recover from errors and used the error state
returned by journal functions incorrectly, resulting in misleading
error messages. Fixed this and also increased the number of error
messages so that it now is easier to diagnose problems with this module.
Also a little bit of internal brush-up.
-mmdblookup bugfix: fix potential segfault due to threading issues
libmaxminddb seems to have issues when running under multiple threads. As
a first measure, we prevent mmdblookup now from running on more than one
thread concurrently.
see also: https://github.com/rsyslog/rsyslog/issues/1885#issuecomment-344882616
- omelasticsearch bugfix: operational error messages not passed to user
This lead to sometimes hard to diagnose problem. Note that for obvious
raesons the amont of messages from omelasticsearch has increased; this
is not a sign of a problem in itself -- but it may reveal problems that
existed before and went unnoticed. Also note that too-verbose messages
can be filtered out via regular rsyslog methods (e.g. message discarding
based on content).
- omkafka bugfixes
* statistics counter maxoutqsize could not be reset
Thanks to github user flisky for the patch.
* potential hang condition
omkafka did not release a mutex under some error conditions (most
notably out of memory on specific alloc calls). This lead to a hang
during actively processing messages or during shutdown (at latest).
This could happen only if dynamic topics were configured.
Detected by Coverity Scan, CID 185781 (originally 185721, detected
as a different issue by Coverity after code refactoring done in regard
to 185721 -- then the problem became more obvious).
* file handle leak, which could occur when local buffering
of messages was needed
* invalid load of failedmsg file on startup if disabled
error "rsyslogd: omkafka: could not load failed messages from "
"file (null) error -2027 - failed messages will not be resend."
occurs because, depite `keepFailedMessages="off"` as a default,
omkafka still tries to check for and load a `(none)` file which
triggers an IO error of sorts according to the 2027 link above.
Obviously, it probably shouldn't try load the file if
`keepFailedMessages` is not enabled.
Thanks to github user JPvRiel for a very good error report and
analysis.
closes https://github.com/rsyslog/rsyslog/issues/1765
* various config parameters did not work
These were not even recognized when used and lead to a config startup
error message:
~ closeTimeout
~ reopenOnHup
~ resubmitOnFailure
~ keepFailedMessages
~ failedMsgFile
closes https://github.com/rsyslog/rsyslog/issues/2052
* considerable memory leak
Whenever a message could (temporarily) not be delivered to kafka,
a non-trivial amount of memory was leaked. This could sum up to
quite a big memory leak.
fixes https://github.com/rsyslog/rsyslog/issues/1991
* some small memory leaks fixed
most of them cosmetic or a few bytes statically (not growing as
omkafka was used) -- thus we do not mention each one explicitely.
For more details, see git commit log or this pull request:
https://github.com/rsyslog/rsyslog/pull/2051
- kafka bugfix: problem on invalid kafka configuration values
omkafka ended up in an endless loop and high cpu.
imkafka tried to subscribe to a not connected kafka server.
closes https://github.com/rsyslog/rsyslog/issues/1806
- [io]mgssapi: fix build problems (regression from 8.30.0)
- [io]czmq: fix build problems on some platforms (namely gcc 7, clang 5)
- tcpsrv bugfix: potential hang during shutdown
- queue bugfix: potential hang during shutdown
- queue bugfix: NULL pointer dereference during config processing
If the queue parameters were incorrectly given, a NULL pointer derefernce
could happen during config parsing. Once post that stage, no problem could
occur.
Detected by Coverity scan, CID 185339
- imczmq bugfix: segfault
happened in a call to
371: zcert_destroy(&serverCert) called from rcvData().
Thanks to ~achiketa Prachanda for the patch.
- imfile: some small performance enhancements
Thanks to Peter Portante for the patch
- omfile: hande file open error via SUSPEND mode
For a while, an open file error lead to suspension as the error was
not detected by the rule engine. This has changed with fixes
in 8.30.0. I asked users what they prefer (and expect) and
everyone meant it should still be handled via suspension. See
github tracker below for more details.
closes https://github.com/rsyslog/rsyslog/issues/1832
- omfile bugfix: race during directory creation can lead to loop
There was a race where two threads were competing for directory creation
which could lead to none succeeding and a flood of error message like this
"error during config processing: omfile: creating parent directories for
file". This has been solved.
Thanks to Niko Kortström for the patch.
- imudp: improve error reporting
When udp listener cannot be created, an error message containing
the ip-address and port is put out.
closes https://github.com/rsyslog/rsyslog/issues/1899
- omrelp bugfix: incorrect error handling
if librelp with TLS but without Authentication was included, librelp
did not emit the correct error message due to invalid error code
check. It also did not err-out but instead suspended itself.
Detected by Coverity scan, CID 185362
- [io]mrelp bugfix: segfault on startup if configured cert not readable
When the certificate file specified in the omrelp/imrelp configuration
can't be accessed, e.g. because it doesn't exist or you don't have
permission to do so, a Segmentation Fault will appear when you start
Rsyslog. This commit fixes that problem.
closes https://github.com/rsyslog/rsyslog/issues/1869
- mmanon fix: make build under gcc 7
Thanks to William Dauchy for the patch
- mmpstrucdata bugfix: formatting error of ']' char
This was invalidly formatted as '"'. Thanks to github user
wu3396 for the error report including the patch idea.
closes https://github.com/rsyslog/rsyslog/issues/1826
- mmexternalb bugfix: memory leak
- core/stats bugfix: memory leak if sender stats or tracking are enabled
- core bugfix: potential segfault during startup
A NULL pointer could be accessed if there was a problem with the
timezone parameters. Affects only startup, once started, no problem
existed.
Detected by Coverty scan; CID 185414
- core bugfix: potential race in variable handling
Root of variable tree is accessed prior to locking access to it.
This introduces a race that may result in various kinds of
misadressing.
Found while reviewing code, no bug report exists.
- core bugfix: potential segfault when shutting down rsyslog
when rulesets are nested a segfault can occur when shutting down
rsyslog. the reason is that rule sets are destructed in load order,
which means a "later" ruleset may still be active when an "earlier"
one was already destructed. In these cases, a "call" can invalidly
call into the earlier ruleset, which is destructed and so leads to
invalid memory access. If a segfault actually happens depends on the
OS, but it is highly probable.
The cure is to split the queue shutdown sequence. In a first step,
all worker threads are terminated and the queue set to enqOnly.
While some are terminated, it is still possible that the others
enqueue messages into the queue (which are then just placed into the
queue, not processed). After this happens, a call can no longer
be issued (as there are no more workers). So then we can destruct
the rulesets in any order.
closes https://github.com/rsyslog/rsyslog/issues/1122
- core/action bugfix: potential misadressing when processing hard errors
For batches that did fail in an output module, the rsyslog core
tries to find offending messages that generate hard (non-recoverable)
errors. During this process, the action templates for each message
are copied into a temporary buffer. That copy was invalidly sized,
actually copying only the first template string. As such, outputs
that requested more template strings AND had erros in batch submission
received uninitialized pointers. This could lead to all sorts of
problems.
see also https://github.com/rsyslog/rsyslog/issues/1885
closes https://github.com/rsyslog/rsyslog/issues/1906
- template object bugfix: NULL pointer access on invalid parameters
could happen only during startup
Detected by Coverity scan, CID 185376
- omjournal bugfix: NULL pointer access on invalid parameters
could happen only during startup
- omelasticsearch bugfix: configured credentials not used during health check
Authentication credentials were not applied during health check,
permission to use unsigned CERTS was not applied to regular data post.
closes https://github.com/rsyslog/rsyslog/issues/1949
- omelasticsearch bugfix: abort on unavailable ES server
Depending on the state of unavailability (libcurl return code),
omelasticsearch tries to process a NULL return message, what
leads to a segfault.
This fixes the problem and introduces better error handling and
better error messages.
see also https://github.com/rsyslog/rsyslog/issues/1885
- omelasticsearch: fix memory leak and potential misadressing
Commit 96b5fce introduced regressions, leading to potential misadressing
and a very probable memory leak. This commit fixes the issues and
hardens the module to better detect some error cases in the
future.
It also adds valgrind-based testbench tests which ensure that no
pointer errors exist. If they had been in place, the regressions
would never have been undetected.
Note that commit 96b5fce was introduced in 8.23.0 on Nov, 15th 2016.
Many thanks to Yaroslav Bo for alerting me on the root problem and
providing a very good analysis and guidance.
see also https://github.com/rsyslog/rsyslog/issues/1906
see also https://github.com/rsyslog/rsyslog/issues/1964
closes https://github.com/rsyslog/rsyslog/issues/1962
- omelasticsearch bugfix: output from libcurl to stdout
omelasticsearch made libcurl output messages to stdout. This
commit fixes that. It also automatically enables libcurl verbose
mode during debug runs - it needs to be seen if this is smart or
not (previously, code needed to be commented in).
closes https://github.com/rsyslog/rsyslog/issues/1909
- iczmq bugfix: potential memory leak
- imptcp bugfix: potential misadressing
When during a connection request the remote peer could not be
identified, imptcp could misadress memory if it is configured
to emit messages on connection open.
Detected by clang 5.0 static analyzer.
- imptcp: potential buffer overflow
if the local hostname or IP is larger than NI_MAXHOST-1, an internal
buffer is overflowed. This is irrespective of any input data.
Detected by Coverity scan, CID 185390
- core/nsd_gtls: fix potential unitialized data access
could occur during certificate check
found by clang 5.0 static analyzer
- stats bugfix: potential program hang
due to missing unlock. This could only occur if pstats was set to
CEE-format logging (very uncommon) AND if the system runs out of
memory (in which case other things go pretty bad as well).
found by Coverty scan
- omfwd bugfix: memory leak if network namespaces are used
very seldom used feature, occurs only during error case
found by Coverty scan.
- core: potential misadressing when accessing JSON properties
When a JSON property is accessed in template processing, memory may
have been misadressed or a double-free may occur while obtaining the
propety.
This was detected by a Coverty scan.
- gcry crypto provider bugfixes: potential misadressing and memory leak
If the config parameters were invalid, a misadressing could occur. If so,
this happens only during startup.
Also, a memory leak existed when the crypto provider errored out. This could
build up if it were used in frequently-changing dynafiles. This was
detected by Coverity scan, CID 185360.
- core/file stream object bugfix: memory leak
If a multiline read errored out, a line buffer could be leaked.
Detected by Coverity scan, CID 185328
- imdiag bugfix: double mutex unlock when working with stats
Note: while this could potentially lead to a program hang, it affected
only testbench execution as imdiag is a testbench-only tool.
Detected by Coverity scan, CID 185348 and 185350
- fixed several minor and cosmetic issues found by Coverty scan
includding false positives. For details see "$ git log". All noteworthy
issues are seperately mentioned in this ChangeLog. The ones not mentioned
are for example problems that can only occur during out of memory
conditions, under which it is extremely likely tha the rsyslog process
will be killed in any case
- testbench:
* added compile-check for [io]mgssapi, mmcount
* harden tests against hanging previous instances
* re-enable RELP tests on Solaris
* added basic test for imjournal
* added threading tests via valgrind's helgrind tool
* added valgrind test for kafka modules
* added capability to run elasticsearch tests with
a) different ElasticSearch versions
b) independently from OS-installed version
This also sets base to enable more elaboreate ES tests
* further relaxed timing of dynstats tests, as they tend to create
false positives on slow machines
- CI: improved test coverage on FreeBSD
- Travis: clang static analyzer 5.0 now run over all source files
- build: make compile warning-free under gcc 7
------------------------------------------------------------------------------
Version 8.30.0 [v8-stable] 2017-10-17
- NEW BUILD REQUIREMENTS
* libfastjson 0.99.7 is now mandatory
the new version is required to support case-insensitive variable
comparisons, which are now the default
* when building imjournal, libsystemd-journal >= 234 is now recommended
This is to support the imjournal enhancement. Note that it is possible
to build with lower version, but this will degrade imjournal functionality.
- CHANGE OF BEHAVIOUR: all variables are now case-insensitive by default
Formerly, JSON based variables ($!, $., $/) were case-sensitive.
Turn old default back on: global(variables.casesensitve="on")
See ChangeLog entry below for more details.
- core: handle (JSON) variables in case-insensitive way
The variable system inside rsyslog is JSON based (for easy consumption
of JSON input, the prime source of structured data). In JSON, keys
("variable names") are case-sensitive. This causes constant problems
inside rsyslog configurations. A major nit is that case-insensitivity
option inside templates (even if turned on) does not work with JSON
keys because they of inner workings*1.
It is much more natural to treat keys in a case-INsensitive way (e.g.
"$!Key" and "$!key" are the same). We do not expect any real problems
out of this, as key names only differing in case is highly unlikely.
However, as it is possible we provide a way to enable case-sensitivity
via the global(variables.casesensitve="on") global config object.
Note that the default is to do case-insensitive matches. The reason
is that this is the right thing to do in almost all cases, and we do
not envision any problems at all in existing deployments.
*1 Note: case-insensitivity in rsyslog is achieved by converting all
names to lower case. So that the higher speed of strcmp() can be used.
The template option does actually that, convert the template keys to
lower case. Unfortunately, this does not work with JSON, as JSON keys
are NOT converted to lower case.
closes https://github.com/rsyslog/rsyslog/issues/1805
- imjournal: made switching to persistent journal in runtime possible
with this patch imjournal can continue logging after switch to
persistent journal without need to restart rsyslog service
Thanks to github user jvymazal for the patch
- mmanon: complete refactor and enhancements
- add pseudonymization mode
- add address randomization mode
- add support for IPv6 (this also supports various replacement modes)
closes https://github.com/rsyslog/rsyslog/issues/1614
also fixes bugs
- in IPv4 address recognition
closes https://github.com/rsyslog/rsyslog/issues/1720
- in IPv4 simple mode to-be-anonymized bits can get wrong
closes https://github.com/rsyslog/rsyslog/issues/1717
- imfile: add "fileoffset" metadata
This permits to send the offset from which the message was read together
with the message text.
Thanks to github user derekjsonar for the initial implementation which
we extended to use the message start offset.
- RainerScript: add ltrim and rtrim functions
closes https://github.com/rsyslog/rsyslog/issues/1711
- core: report module name when suspending action
Thanks to Orion Poplawski for the patch.
- core: add ability to limit number of error messages going to stderr
This permits to put a hard limit on the number of messages that can
go to stderr. If for nothing else, this capability is helpful for the
testbench. It permits to reduce spamming the test log while still
providing the ability to see initial error messages. Might also be
useful for some practical deployments.
global parameter: global(errorMessagesToStderr.maxNumber)
- tcpsrv subsystem: improvate clarity of some error messages
operating system error message are added to some messages, providing
better advise of what actually is the error cause
- imptcp: include module name in error msg
- imtcp: include module name in error msg
- tls improvement: better error message if certificate file cannot be read
- omfwd: slightly improved error messages during config parsing
They now contain config file/line number information where this was missing.
- ommysql improvements
* Return specifc code for unrecoverable errors. This makes retry processing
more performant and robust.
* error messages improved
* Update to utilize native v8 transactional interface. Previously, it used
the v7 interface with v8 emulation.
* treat server and client-generated messages differently
Server-generated error messages are considered non-recoverable, while
client generated once point into connection problems (which simply can
be retried). This is part of the improvements towards better
message-induced errors. Previous commits did only handle SQL parsing
errors, now we aim to address all of the message-induced error. We assume
that all server-generated errors are such - mysql API unfortunately does
not provide a clear indication of what type if error this is and it is
out of question to check for hundereds of error codes.
closes https://github.com/rsyslog/rsyslog/issues/1830
- ommysql bugfix: do not duplicate entries on failed transaction
If a multi-message batch contains data errors, messages may be
duplicated as connection close is implicit commit (not rollback).
This patch introduces a specific rollback request in those cases.
closes https://github.com/rsyslog/rsyslog/issues/1829
- imtcp bugfix: parameter priorityString was ignored
defaults were always used
- template/bugfix: invalid template option conflict detection
This prevented "option.casesenstive" to be used with the SQL and JSON
options.
- core/actions: fix handling of data-induced errors
Rsyslog core should try to detect data-induced (unrecoverable) errors
during action processing. An example of such is invalid SQL statements.
If the action returns a data error, rsyslog shall retry individual
messages from a batch in an effort to log those without errors. The others
shall be dropped.
This logic was temporarily disabled after the switch to the new v8
transaction interface. Now this bug is fixed and the testbench has been
ammended to detect problems in the future.
closes https://github.com/rsyslog/rsyslog/issues/974
- core/action bugfix: no "action suspended" message during retry processing
The action engine does not emit "action suspended" messages but "resumed"
ones in retry processing. This is wrong, as they are a strong indication
that something does not work correctly. Nevertheless, "resumed" messages
were emitted, which was very puzzling for the user.
This patch fixes it so that "suspend" messages are given during retry
processing. These do not contain a retry timestamp, providing evidence
that a retry is currently being tried.
coses https://github.com/rsyslog/rsyslog/issues/1069
- core/ratelimit bugfix: race can lead to segfault
There was a race in iminternalAddMsg(), where the mutex is
released and after that the passed message object is accessed.
If the mainloop iterates in between, the msg may have already
been deleted by this time, resulting in a segfault.
Most importantly, there is no need to release the mutex lock
early, as suggested by current code. Inside mainloop the mutex
is acquired when it is time to do so, so at worst we'll have a
very slight wait there (which really does not matter at all).
This only happens if a large number of internal messages are emitted.
closes https://github.com/rsyslog/rsyslog/issues/1828
- core bugfix: rsyslog aborts if errmsg is generated in early startup
Note that the segfault can occur only during early startup. Once
rsyslog has started, everything works reliably. This condition can
especially be triggerred by specifying invalid TLS default certificates.
closes https://github.com/rsyslog/rsyslog/issues/1783
closes https://github.com/rsyslog/rsyslog/issues/1786
- core bugfix: informational messages was logged with error severity
When the main loop reaped a child process (a normal action), this was
reported as an error. This caused user confusion. Now it is reported as
an informational message.
- core bugfix: --enable-debugless build was broken
This was a regression from the v8.29.0 debug enhancements
Thanks to Stephen Workman for the patch.
- queue bugfix: file write error message was incorrect
when a queue was restarted from disk file, it almost always
emitted a message claiming
"file opened for non-append write, but already contains xxx bytes"
This message was wrong and did not indicate a real error condition.
The predicate check was incorrect.
closes https://github.com/rsyslog/rsyslog/issues/170 (kind of)
- omrelp bugfix: segfault when rebindinterval parameter is used
- imudp bugfix: UDP oversize message not properly handled
When a message larger than supported by the UDP stack is to be sent,
EMSGSIZE is returned, but not specifically handled. That in turn
will lead to action suspension. However, this does not make sense
because messages over the UDP max message size simply cannot be sent.
closes https://github.com/rsyslog/rsyslog/issues/1654
- core bugfix: memory corruption during configuration parsing
when omfwd is used with the $streamdriverpermittedpeers legacy
parameter, a memory corruption can occur. This depends on the
length of the provided strings and probably the malloc subsystem.
Once config parsing succeeds, no problem can happen.
Thanks to Brent Douglas for initially reporting this issue and
providing great analysis.
Thanks to github user bwdoll for analyzing this bug and providing
a suggested fix (which is almost what this commit includes).
closes https://github.com/rsyslog/rsyslog/issues/1408
closes https://github.com/rsyslog/rsyslog/issues/1474
- core bugfix: race on worker thread termination during shutdown