Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fields labels #10119

Merged
merged 5 commits into from
Jan 22, 2019
Merged

Add fields labels #10119

merged 5 commits into from
Jan 22, 2019

Conversation

urso
Copy link

@urso urso commented Jan 16, 2019

The add_labels processor was more similar to the fields setting.
This change introduces the add_fields processor to behave like the
fields setting, but changes/limits the add_labels procesor to the use
case of adding a flat set of labels to ECS compatible events.

The add_labels processor will:

  • flatten keys by joining nested names with dots
  • always writes labels to the events 'labels' field. The target field is
    not configurable and labels can not be written to the root of the
    document

@urso urso added review needs_backport PR is waiting to be backported to other branches. labels Jan 16, 2019
@urso urso requested a review from a team as a code owner January 16, 2019 18:03
@urso urso requested a review from ruflin January 16, 2019 18:03
libbeat/processors/actions/add_labels.go Outdated Show resolved Hide resolved
@urso urso requested a review from kvch January 18, 2019 04:24
@urso
Copy link
Author

urso commented Jan 19, 2019

jenkins, test this

Copy link
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will need rebase.

The error is related to TLS on hearthbeat, first time I've seen it.

23:20:25 --- FAIL: TestHTTPSx509Auth (1.32s)
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:40
23:20:25         	Error:      	mapval could not validate map
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25         	Messages:   	13 errors validating source: 
23:20:25         	            	(common.MapStr) (len=4) {"error":{"message":"Get https://127.0.0.1:45821: read tcp 127.0.0.1:39022-\u003e127.0.0.1:45821: i/o timeout (Client.Timeout exceeded while awaiting headers)","type":"io"},"monitor":{"duration":{"us":1013787},"ip":"127.0.0.1","status":"down"},"tcp":{"rtt":{"connect":{"us":2023}}},"url":{"domain":"127.0.0.1","full":"https://127.0.0.1:45821","port":255,"scheme":"https"}}
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'error': unexpected field encountered during strict validation
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'monitor.status': objects not equal: actual(down) != expected(up)
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'tls.rtt.handshake.us': expected this key to be present
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'http.rtt.validate.us': expected this key to be present
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'http.rtt.content.us': expected this key to be present
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:25         	            	@path 'http.rtt.write_request.us': expected this key to be present
23:20:25         	Test:       	TestHTTPSx509Auth
23:20:25     <autogenerated>:1: 
23:20:25         	Error Trace:	mapvaltest.go:48
23:20:25         	Error:      	Received unexpected error:
23:20:26         	            	@path 'tls.certificate_not_valid_before': expected this key to be present
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'tls.certificate_not_valid_after': expected this key to be present
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'http.response.status_code': expected this key to be present
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'http.rtt.response_header.us': expected this key to be present
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'error.message': unexpected field encountered during strict validation
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'http.rtt.total.us': expected this key to be present
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26     <autogenerated>:1: 
23:20:26         	Error Trace:	mapvaltest.go:48
23:20:26         	Error:      	Received unexpected error:
23:20:26         	            	@path 'error.type': unexpected field encountered during strict validation
23:20:26         	Test:       	TestHTTPSx509Auth
23:20:26 FAIL

urso added 5 commits January 21, 2019 19:09
The add_labels processor was more similar to the fields setting.
This change introduces the add_fields processor to behave like the
fields setting, but changes/limits the add_labels procesor to the use
case of adding a flat set of labels to ECS compatible events.

The add_labels processor will:
- flatten keys by joining nested names with dots
- always writes labels to the events 'labels' field. The target field is
not configurable and labels can not be written to the root of the
document
@urso urso force-pushed the add-fields-labels branch from 2d79ded to fb9b01b Compare January 21, 2019 18:09
@urso urso merged commit 2ec5e1a into elastic:master Jan 22, 2019
@urso urso removed the needs_backport PR is waiting to be backported to other branches. label Jan 31, 2019
@urso urso deleted the add-fields-labels branch February 19, 2019 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants