Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add processor to set timezone for an event #3902

Merged
merged 13 commits into from
Apr 7, 2017
Merged

Add processor to set timezone for an event #3902

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a45355d
Add add_locale processor
martinscholz83 Apr 4, 2017
b2ad649
register processor
martinscholz83 Apr 4, 2017
88cc4e7
Add tests
martinscholz83 Apr 4, 2017
7d2782e
Change handling of getting local timezone
martinscholz83 Apr 4, 2017
7f84cb3
change test
martinscholz83 Apr 4, 2017
dd74400
Remove unnecassary config
martinscholz83 Apr 4, 2017
eb4dea6
Simplified get zone
martinscholz83 Apr 4, 2017
a06dd06
Update changelog
martinscholz83 Apr 4, 2017
e6353be
Add description for config.asciidoc
martinscholz83 Apr 5, 2017
6914c9a
Fix grammar config.asciidoc
martinscholz83 Apr 5, 2017
ed01967
Add timezone to fields.common.yml
martinscholz83 Apr 5, 2017
94dca42
Fetch timezone in run method
martinscholz83 Apr 7, 2017
f5f4b4c
Add benchmark test
martinscholz83 Apr 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions libbeat/beat/beat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ import (
// Register default processors.
_ "github.com/elastic/beats/libbeat/processors/actions"
_ "github.com/elastic/beats/libbeat/processors/add_cloud_metadata"
_ "github.com/elastic/beats/libbeat/processors/add_locale"

// Register default monitoring reporting
_ "github.com/elastic/beats/libbeat/monitoring/report/elasticsearch"
Expand Down
47 changes: 47 additions & 0 deletions libbeat/processors/add_locale/add_locale.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package actions

import (
"time"

"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/processors"
"github.com/pkg/errors"
)

type addLocale struct {
timezone string
}

func init() {
processors.RegisterPlugin("add_locale", newAddLocale)
}

func newAddLocale(c common.Config) (processors.Processor, error) {
config := struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole config definition and the Unpack call can now be removed.

TimeZone string `config:"timezone"`
Copy link
Member

@andrewkroh andrewkroh Apr 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this just be completely automatic using time.Now().Location()? Example: https://play.golang.org/p/2Lpzo3KvWJ

Copy link
Contributor Author

@martinscholz83 martinscholz83 Apr 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what i not understand correctly. I thought we give the user the option to set the timezone. If not so you are right.

Copy link
Member

@andrewkroh andrewkroh Apr 4, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think idea is to report the local timezone. It probably shouldn't be configurable at all. Just detect the local TZ and report it in the event.

Once you have the timezone, you can use Logstash to do further transformations on the data (like interpreting syslog dates that are missing the timezone) or converting @timestamp to the machine's local timezone (not recommended IMHO).

A more specific use case is analyzing logon events, the timezone can be used to determine if the event occurred during normal working hours. Then you can alert on people working after hours and send them on a vacation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there's an even more direct way: time.Local.String().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

time.Local.String() only gives you Local. I have changed the method to get the the timezone

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, am I misunderstanding the docs at https://golang.org/pkg/time/#Location ?

Local represents the system's local time zone.

var Local *Location = &localLoc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On golang Playground i get UTC. On my machine i get

"beat": {
    "hostname": "4201halwsd00001",
    "name": "4201halwsd00001",
    "timezone": "Local",
    "version": "6.0.0-alpha1"
  },

???

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, you are right. I get the same thing "Local". So zone, _ := time.Now().Zone() should be sufficient because Now() always returns local time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Changed up to time.Now().Zone(). I was not sure if Now() always returns local time.

}{}

err := c.Unpack(&config)
if err != nil {
return nil, errors.Wrap(err, "failed to unpack add_locale config")
}

l := addLocale{timezone: config.TimeZone}

return l, nil
}

func (l addLocale) Run(event common.MapStr) (common.MapStr, error) {
zone, err := time.LoadLocation(l.timezone)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Run method should be as simple and fast as possible. In newAddLocale() you I think can do this work once and store the result so that Run just becomes a map.Put() operation.


if err != nil {
return event, err
}

event.Put("beat.timezone", zone.String())
return event, nil
}

func (l addLocale) String() string {
return "add_locale=" + l.timezone
}
60 changes: 60 additions & 0 deletions libbeat/processors/add_locale/add_locale_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
package actions

import (
"testing"

"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
"github.com/stretchr/testify/assert"
)

func TestExportTimeZone(t *testing.T) {
var testConfig, _ = common.NewConfigFrom(map[string]interface{}{
"timezone": "America/Curacao",
})

input := common.MapStr{}

actual := getActualValue(t, testConfig, input)

expected := common.MapStr{
"beat": map[string]string{
"timezone": "America/Curacao",
},
}

assert.Equal(t, expected.String(), actual.String())
}

func TestExportDefaultTimeZone(t *testing.T) {
var testConfig, _ = common.NewConfigFrom(map[string]interface{}{
"timezone": "",
})
input := common.MapStr{}

actual := getActualValue(t, testConfig, input)

expected := common.MapStr{
"beat": map[string]string{
"timezone": "UTC",
},
}

assert.Equal(t, expected.String(), actual.String())
}

func getActualValue(t *testing.T, config *common.Config, input common.MapStr) common.MapStr {
if testing.Verbose() {
logp.LogInit(logp.LOG_DEBUG, "", false, true, []string{"*"})
}

p, err := newAddLocale(*config)
if err != nil {
logp.Err("Error initializing add_locale")
t.Fatal(err)
}

actual, err := p.Run(input)

return actual
}