Rsyslog integration

[domhel]

Hello,

I have some devices that currently use rsyslog to log to Elastic / Kibana.
Is it possible to log to parseable with rsyslog as well?

Thanks

[nitisht]

Thanks for reporting @dippa-1 we'll take a look and get back.

[nitisht]

@dippa-1 have you seen / tried https://www.rsyslog.com/doc/configuration/modules/omhttp.html

AFAICS a config like should work. Happy to debug async and get this to working state

 action(
        type="omhttp"
        server="[Parseable-IP]"
        serverport="[Parseable-port]"
        restpath="api/v1/ingest"
        template="tpl_echo"
        uid="[Parseable-User]"
        pwd="[Parseable-Password]"
        httpheaders=[
            "X-P-Stream: [Parseable-stream-to-send-events-to]",
        ]
        batch="on"
        batch.format="jsonarray"
        batch.maxsize="10"

        retry="on"
        retry.ruleset="rs_retry_forever"
    )

[domhel]

Thanks for the response. I tried it out but sadly we don't have omhttp in rsyslog in our Yocto Linux.
If I find some time I will try adding it to the build configuration. Nonetheless, this would require all of our devices' OS to be updated to be able to log to parseable.

Ideally there would be some kind of compatibility layer for incoming elastic search like packets that parseable would turn into its own format.

[nitisht]

Thanks for the update @dippa-1 . Do you mind sharing a sample of such packet? We can take a deeper look

[domhel]

I don't have the exact packet but this is the configuration we use:

# Provides UDP syslog reception
module(load="imudp")
input(type="imudp"
  port="514")
# Provides TCP syslog reception
module(load="imtcp")
input(type="imtcp"
  port="514")

module(load="mmutf8fix")
action(type="mmutf8fix" replacementChar="?")

# JSON conversion
reset $.msg = $msg;
set $.msg = replace($.msg, "\\", "\\\\");
set $.msg = replace($.msg, "\"", "\\\"");
set $.msg = replace($.msg, "#011", "\\t");
set $.msg = replace($.msg, "#012", "\\n");

# drop .local suffix
reset $.hostname = replace($hostname, ".local", "");

# ElasticSearch output
module(load="omelasticsearch")
template(name="device-syslog" type="list") {
    constant(value="{")
    constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")        property(name="$.hostname")
    constant(value="\",\"severity-num\":")  property(name="syslogseverity")
    constant(value=",\"facility-num\":")    property(name="syslogfacility")
    constant(value=",\"severity\":\"")      property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
    constant(value="\",\"syslogtag\":\"")   property(name="syslogtag")
    constant(value="\",\"message\":\"")     property(name="$.msg")
    constant(value="\"}")
}
template(name="device-index" type="string" string="device-%$YEAR%.%$MONTH%.%$DAY%")
action(type="omelasticsearch"
  server="elasticsearch.our.domain.com"
  serverport="1337"
  uid="device"
  pwd="blurred"
  usehttps="on"
  template="device-syslog"
  searchIndex="device-index"
  dynSearchIndex="on"
  bulkmode="on")
# error file disabled as it is unbounded in size!
#  errorOnly="on"
#  errorfile="/var/log/omelasticsearch.log")

# also target rapid7
#$IncludeConfig /data/customize/logentries.conf

Does this help you?

[domhel]

Tested it again. Had to include the omhttp module manually in Yocto Linux.

The config is basically like you said:

# template stuff
template(name="my-template" type="list") {
   ...
}

module(load="omhttp")
action(
        type="omhttp"
        server="my-server-ip.com"
        serverport="my-port"
        restpath="api/v1/ingest"
        template="my-template"
        uid="admin"
        pwd="admin"
        httpheaders=[
            "X-P-Stream: my-stream"
        ]
        batch="on"
        batch.format="jsonarray"
        batch.maxsize="10"
        retry="on"
        retry.ruleset="rs_retry_forever"
)

Note that I had to remove a comma in the httpheaders brackets

[nitisht]

This is very useful info, thanks @dippa-1 . We'll add this in the documentation.