feat(config): Global Default Log Schemas #1769

Hoverbear · 2020-02-10T23:20:21Z

This addresses #1446.

It adds a log_schema field to the config.global. It goes into config.global since this already gets passed to the SourceConfig::build event. It also exists in the $SOURCE_source function args as this is used in some tests.

Example:

[hoverbear@obsidian:~/git/vector]$ cat test.toml 
data_dir = '/var/lib/vector/'
dns_servers = []

[log_schema]
host_key = "instance" # default "host"
message_key = "info" # default "message"
timestamp_key = "datetime" # default "timestamp"

[sources.source0]
max_length = 102400
type = 'stdin'
[sinks.sink0]
healthcheck = true
inputs = ['source0']
type = 'console'
encoding = "json"

[sinks.sink0.buffer]
type = 'memory'
max_events = 500
when_full = 'block'


[hoverbear@obsidian:~/git/vector]$ cargo run -- --config test.toml -qqq
# ....
Feb 13 04:24:30.902  INFO vector::topology: Starting sink "sink0"
Feb 13 04:24:30.902  INFO source{name=source0 type=stdin}: vector::sources::stdin: Capturing STDIN
asd
{"info":"asd","datetime":"2020-02-13T12:24:32.632275500Z","instance":"OBSIDIAN"}

binarylogic · 2020-02-10T23:55:56Z

The config syntax looks good! But shouldn't the resulting payload in your ouput have both the datetime and info keys since you didn't explicitly supply those as part of your input?

Hoverbear · 2020-02-11T17:52:51Z

Ok! That clarifies it for me! I wasn't really sure if only the fields the source had configs for would be affected, or all would.

bruceg

I have left a few questions inline.

AFAICT the kafka sink also appears to use these keys. Should it also be modified to use the new config?

src/sources/file.rs

src/sources/stdin.rs

src/topology/config/mod.rs

Hoverbear · 2020-02-11T18:19:34Z

Thanks for this early review @bruceg :)

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Jeffail

LGTM, but need to initialize the globals before running unit tests as well.

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear · 2020-02-13T14:45:51Z

@Jeffail I did it the easy way. :)

Hoverbear · 2020-02-13T14:46:19Z

src/event/mod.rs

+    // TODO: Help Rust project support before_each
+    // Support uninitialized schemas in tests to help our contributors.
+    // Don't do it in release because that is scary.
+    #[cfg(debug_assertions)]


I am not 100% sure on this.

src/event/mod.rs

LucioFranco · 2020-02-13T16:40:35Z

src/topology/config/mod.rs

@@ -332,6 +335,35 @@ impl Config {
        self.global.dns_servers.sort();
        self.global.dns_servers.dedup();

+        // If the user has multiple config files, we must *merge* log schemas until we meet a
+        // conflict, then we are allowed to error.
+        let default_schema = event::LogSchema::default();


nit: prob could just do if with.global.log_schema = LogSchema::default()

We use it later though :o

src/event/mod.rs

LucioFranco

Overall this looks fantastic, turned out much better than I thought it would :)

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear · 2020-02-13T18:21:06Z

Cargo.toml

@@ -143,6 +143,7 @@ evmap = { version = "7", features = ["bytes"] }
 logfmt = "0.0.2"
 notify = "4.0.14"
 once_cell = "1.3"
+getset = "0.1.0"


When you get to import your own library!

Hoverbear · 2020-02-13T18:21:32Z

@LucioFranco Take a look again?

Signed-off-by: Ana Hobden <operator@hoverbear.org>

LucioFranco · 2020-02-13T21:02:04Z

src/sources/syslog.rs

@@ -434,6 +437,7 @@ mod test {
    }

    #[test]
+    #[ignore]


why are these ignored?

Indeed why!

LucioFranco

Just have one question about those ignored tests otherwise LGTM

Signed-off-by: Ana Hobden <operator@hoverbear.org>

binarylogic · 2020-02-14T03:04:57Z

Nice work on this! I just wanted to couple doc changes with it. I'll follow up with a doc change PR.

* Global log schemas Signed-off-by: Ana Hobden <operator@hoverbear.org> * Tests working and some lints cleaned. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix unix tests Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup of remaining unix tests Signed-off-by: Ana Hobden <operator@hoverbear.org> * Use correct naming. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add better documentation for handling conflicting log_schemas. Signed-off-by: Ana Hobden <operator@hoverbear.org> * fmt Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup schema lookup in logdna Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix benches Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fmt and fix lints Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix some tests that are dead on Windows. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Docker nits Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix naming Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix a test Signed-off-by: Ana Hobden <operator@hoverbear.org> * Missed one! Signed-off-by: Ana Hobden <operator@hoverbear.org> * Unignore tests Signed-off-by: Ana Hobden <operator@hoverbear.org> * Reduce visiblility! Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear added this to the Initial schema support milestone Feb 10, 2020

Hoverbear requested review from bruceg, LucioFranco and lukesteensen as code owners February 10, 2020 23:20

Hoverbear self-assigned this Feb 10, 2020

Hoverbear changed the title ~~[Draft] Global Default Log Schemas~~ feature(config): Global Default Log Schemas [Draft] Feb 10, 2020

Hoverbear mentioned this pull request Feb 10, 2020

Global options for schema field names (message, host, timestamp, etc) #1446

Closed

3 tasks

bruceg reviewed Feb 11, 2020

View reviewed changes

Hoverbear requested a review from a user February 12, 2020 00:47

Hoverbear requested a review from MOZGIII as a code owner February 12, 2020 00:47

binarylogic requested review from bruceg and removed request for MOZGIII, lukesteensen, a user and LucioFranco February 12, 2020 06:50

binarylogic assigned bruceg and unassigned Hoverbear Feb 12, 2020

Global log schemas

1944719

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear force-pushed the templating-add-field branch from de54b42 to 1944719 Compare February 12, 2020 21:52

Hoverbear added 6 commits February 12, 2020 15:57

Tests working and some lints cleaned.

5177208

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fix unix tests

2f4c525

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fixup of remaining unix tests

91c977e

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Use correct naming.

853e83f

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Add better documentation for handling conflicting log_schemas.

9ced442

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Merge branch 'master' into templating-add-field

343e879

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear requested a review from Jeffail as a code owner February 13, 2020 02:04

Hoverbear changed the title ~~feature(config): Global Default Log Schemas [Draft]~~ feature(config): Global Default Log Schemas Feb 13, 2020

Hoverbear changed the title ~~feature(config): Global Default Log Schemas~~ feat(config): Global Default Log Schemas Feb 13, 2020

Hoverbear added 3 commits February 12, 2020 18:05

fmt

b5e11a1

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fixup schema lookup in logdna

e2bfef2

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fix benches

ef7bcc6

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Jeffail reviewed Feb 13, 2020

View reviewed changes

Hoverbear added 3 commits February 13, 2020 04:15

Fmt and fix lints

85fc5c1

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fix some tests that are dead on Windows.

a4b00bb

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Docker nits

bf9c2ca

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear commented Feb 13, 2020

View reviewed changes

LucioFranco reviewed Feb 13, 2020

View reviewed changes

src/event/mod.rs Outdated Show resolved Hide resolved

LucioFranco reviewed Feb 13, 2020

View reviewed changes

src/event/mod.rs Outdated Show resolved Hide resolved

LucioFranco approved these changes Feb 13, 2020

View reviewed changes

Fix naming

6cda319

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear commented Feb 13, 2020

View reviewed changes

Hoverbear added 2 commits February 13, 2020 10:32

Fix a test

37109b8

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Missed one!

8515def

Signed-off-by: Ana Hobden <operator@hoverbear.org>

LucioFranco reviewed Feb 13, 2020

View reviewed changes

LucioFranco approved these changes Feb 13, 2020

View reviewed changes

Hoverbear added 2 commits February 13, 2020 13:33

Unignore tests

036e160

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Reduce visiblility!

d6b02e8

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear merged commit 15e69c8 into vectordotdev:master Feb 14, 2020

bruceg mentioned this pull request Feb 20, 2020

Benchmarks are failing with Schema was not initialized #1872

Closed

ktff mentioned this pull request Feb 21, 2020

fix: (kubernetes source) Change timestamp_key back to message_key #1887

Merged

ghost mentioned this pull request Feb 21, 2020

Benefits and downsides of Atoms #1891

Closed

Hoverbear mentioned this pull request Feb 27, 2020

Sources need a context parameter, similar to sinks and transforms #1954

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(config): Global Default Log Schemas #1769

feat(config): Global Default Log Schemas #1769

Hoverbear commented Feb 10, 2020 •

edited

Loading

binarylogic commented Feb 10, 2020

Hoverbear commented Feb 11, 2020

bruceg left a comment

Hoverbear commented Feb 11, 2020

Jeffail left a comment

Hoverbear commented Feb 13, 2020

Hoverbear Feb 13, 2020

LucioFranco Feb 13, 2020

Hoverbear Feb 13, 2020

LucioFranco left a comment

Hoverbear Feb 13, 2020

Hoverbear commented Feb 13, 2020

LucioFranco Feb 13, 2020

Hoverbear Feb 13, 2020

LucioFranco left a comment

binarylogic commented Feb 14, 2020

feat(config): Global Default Log Schemas #1769

feat(config): Global Default Log Schemas #1769

Conversation

Hoverbear commented Feb 10, 2020 • edited Loading

binarylogic commented Feb 10, 2020

Hoverbear commented Feb 11, 2020

bruceg left a comment

Choose a reason for hiding this comment

Hoverbear commented Feb 11, 2020

Jeffail left a comment

Choose a reason for hiding this comment

Hoverbear commented Feb 13, 2020

Hoverbear Feb 13, 2020

Choose a reason for hiding this comment

LucioFranco Feb 13, 2020

Choose a reason for hiding this comment

Hoverbear Feb 13, 2020

Choose a reason for hiding this comment

LucioFranco left a comment

Choose a reason for hiding this comment

Hoverbear Feb 13, 2020

Choose a reason for hiding this comment

Hoverbear commented Feb 13, 2020

LucioFranco Feb 13, 2020

Choose a reason for hiding this comment

Hoverbear Feb 13, 2020

Choose a reason for hiding this comment

LucioFranco left a comment

Choose a reason for hiding this comment

binarylogic commented Feb 14, 2020

Hoverbear commented Feb 10, 2020 •

edited

Loading