How hard would adding persistence be?

[nodesocket]

Perhaps the server could store lines to something like RethinkDB?

[kilianc]

@nodesocket good question and thanks for contributing.

Adding persistence requires a lot of work, mostly on the UI side. A robust implementation of an infinite scroll and pull/search api is needed. I am aware it would make this a better tool and cover a lot of use cases.

The direction this project is heading ATM is to sit and wait what people think it should do and we can move it from there. v0.2.0 is on its way out, but other than small improvements nothing big is planned.

Could you elaborate a bit more? What exactly is your use case?

[kilianc]

To be more specific, why aren't you using loggly or logentries, just cost ?

[SwaroopH]

@kilianc I prefer self hosted & open source if possible.

So @adityavm and I have been hacking with rTail today to see if we can get some persistence.

1. Local: Use local webstorage to store whatever the browser has received so can at least maintain some local persistence. Of course, closing the tab loses any data server has received since.
2. Permanent: Store everything on the server side on some nosql db (Redis/Mongo/Rethink). This coupled with some local storage can make it really quick. My only concern is storage over time but I am sure we can set a buffer for that too :)

Speaking of buffer, we tried increasing the server buffer to maintain a bigger log but it eventually kills the tab. I suppose infinite scrolling will take care of the issue so we don't store everything in DOM.

Thoughts?

P.S. Good job on rTail so far!

[kilianc]

@SwaroopH you guys are heading to the right direction, what I would do or better what I envision is a pluggable storage with a contract towards a public (super simple) interface. Basically storage adapters.

I would personally go for redis and other key/value storage as first attempt.

Again storing data is the easy part, I am more focused on gathering info about use cases and what you guys are trying to achieve.

This is what I think having persistency would be useful for:

• Auth
• Account based settings
• Search by time range / pattern
• Better stream filter so you can actually have all errors for example (the alternative is to | grep | rtail)
• Scroll up in order to have better context when bad things happens or you nee to use rtail as a "file" viewer, I am thinking about npm i or make outuput kind of thing.
• Regexp alerts on determined patterns
• Stats based on patters like mixpanel

To be honest is big step for this project and it needs some more validation, but that shouldn't stop hacking or feedback.

What do you guys need? How many streams you handle? Wha are the bottlenecks you are facing.

[SwaroopH]

Today was day 1 of rTail :) We are in the middle of a product launch and our data scientist wanted access to the logs. We've been wanting a better way to look at our logs as well and that's where rTail fit in.

• Auth was my biggest concern. I have it running behind a nginx with Authentication - not ideal but good for now. I like Github integration as suggested on add github signin integration #44 but I agree with you, this will all require a bit of effort.
• Same as above.
• Time sorting would be great and would work well with persistence. I am guessing Redis supports regex scans?
• I suppose some people wouldn't want everything in a log to be pushed to rTail. We use colour and codes like info/warn/and error in our custom logger to make pattern matching easy – don't mind the entire log being dumped to the stream.
• You mean some more information than what gets pushed to the stream?
• Alerts would be fantastic; we were just talking about how we wanted out customer logger to ping us when it sees a particular code like [error].
• Nice, I was eventually going to suggest stats that because I am obsessed with stats - that's our day job/startup :)

I agree this is a huge step for the project. May be we can get people to nominate and vote for what they are looking for. We definitely will need some more time to figure what we actually need. We currently only handle 6 (2-3 busy) streams but expect it to go up.

[kilianc]

@SwaroopH thanks for the fantastic insight 💯

[cleaton]

I like the idea of a lightweight log interface. There are already larger heavier solutions for logging available like https://www.graylog.org/ that can be used in self host environment (and is open source)

[kilianc]

@cleaton while I see your point, everything else out there takes forever to setup and leaves this in the hands of the devop team. The main goal of rtail is to remain developer friendly and empower everyone with centralized terminal output, not just logs.

Persistency is somehow a "grown up" thing and if we make it, we need to make it right. The project needs to stay lightweight and a pipe away no matter what feature we add.

We are using UDP, and talking about persistency... I mean if this thing ends up being used a lot we need a --tcp option 👍

[cleaton]

But if people are investigating adding more persistence to rtail, I think my suggestion would be kafka. One way could be to have rtail server display kafka topics as streams and consume from them. Or modify the ui to to fetch data from a restful kafka source like the one provided by confluent. There is no need to modify the rtail client as there are already powerful solutions to tail logs and forward to kafka, which also includes compression. Ex: kafkacat .

[mingfang]

+1
We need persistence.
We recently switched from Kibana to Graylog but they are not as good as rtail.
I also think kafka may be a good solution.
But agree that the challenge is in the UI changes to control the replay of logs.

[kilianc]

+1 :D

[megastef]

Hi my optinion is that rtail is great for realtime view and storing logs can be done in parallel - rtail should focus on best realtime log view experience including a short history of logs. I see a lot of work in other things like better rendering options for structured/JSON logs - like field selection in the display (just one example).

E.g.:
I use https://github.com/sematext/logagent-js for parsing and storing logs in Logsene/Elasticsearch and pipe the output to rtail client. So one can watch realtime logs in rtail Web UI, have logs stored and access to analytics/charts in Logsene UI or Kibana or Grafana (supports Elasticsearch datasources in v2.6)

sudo logagent  -t INDEX_OR_LOGSENE_TOKEN -g /var/log/*log | rtail -p 9090 --name maclogs

Little screen recording best of all:

• parsing logs with logagent-js
• storing in Logsene / Elasticsearch with logagent-js
• realtime view in rtail
• analytics/charts with Logsene, Kibana, Grafana with stored data from ES index.
• @SwaroopH Alerts on Logs
  

The command line tools logagent and rtail play well together - and indexing large amounts of logs plus the visualisation options are great.

Rtail WebUI authentication:
@SwaroopH For production I would wrap rtail into a docker container behind nginx reverse proxy for authentication. Or add https://github.com/jaredhanson/passport to the rtail webserver.