Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Migrate php_fpm to ECS #10366

Merged
merged 5 commits into from
Jan 29, 2019
Merged

[Metricbeat] Migrate php_fpm to ECS #10366

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
67d4fba
[Metricbeat] Add TestFetch method to php_fpm tests
ruflin Jan 28, 2019
e4ae0a3
migrate to ECS
ruflin Jan 28, 2019
abdd1a6
update ecs-migratoin, fields yml, changelog
ruflin Jan 29, 2019
d8b6eaa
update tests and apply review feedback
ruflin Jan 29, 2019
a1e4a79
Merge branch 'master' into php-fpm-reporter
ruflin Jan 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Removed Prometheus stats metricset in favor of just using Prometheus collector {pull}9948[9948]
- Adjust Redis.info metricset fields to ECS. {pull}10319[10319]
- Rename http.request.body field to http.request.body.content. {pull}10315[10315]
- Adjust php_fpm.process metricset fields to ECS. {pull}10366[10366]

*Packetbeat*

Expand Down
27 changes: 27 additions & 0 deletions dev-tools/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1208,6 +1208,33 @@

### Redis

- from: php_fpm.status.pid
to: process.pid
alias: true
beat: metricbeat

- from: php_fpm.status.request_method
to: http.request.method
alias: true
beat: metricbeat

- from: php_fpm.status.request_uri
to: url.original
alias: true
beat: metricbeat

- from: php_fpm.status.content_length
to: http.response.body.bytes
alias: true
beat: metricbeat

- from: php_fpm.status.user
to: http.response.user.name
alias: true
beat: metricbeat

### Redis

- from: redis.info.server.version
to: service.version
alias: true
Expand Down
20 changes: 15 additions & 5 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18851,7 +18851,9 @@ process contains the metrics that were obtained from the PHP-FPM process.
*`php_fpm.process.pid`*::
+
--
type: integer
type: alias

alias to: process.pid

The PID of the process

Expand Down Expand Up @@ -18913,7 +18915,9 @@ The duration in microseconds (1 million in a second) of the current request (my
*`php_fpm.process.request_method`*::
+
--
type: keyword
type: alias

alias to: http.request.method

The request method (GET, POST, etc) (of the current request)

Expand All @@ -18923,7 +18927,9 @@ The request method (GET, POST, etc) (of the current request)
*`php_fpm.process.request_uri`*::
+
--
type: text
type: alias

alias to: url.original

The request URI with the query string (of the current request)

Expand All @@ -18933,7 +18939,9 @@ The request URI with the query string (of the current request)
*`php_fpm.process.content_length`*::
+
--
type: integer
type: alias

alias to: http.response.body.bytes

The content length of the request (only with POST) (of the current request)

Expand All @@ -18943,7 +18951,9 @@ The content length of the request (only with POST) (of the current request)
*`php_fpm.process.user`*::
+
--
type: keyword
type: alias

alias to: user.name

The user (PHP_AUTH_USER) (or - if not set) (for the current request)

Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/php_fpm/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 15 additions & 9 deletions metricbeat/module/php_fpm/pool/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,21 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"beat": {
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"event": {
"dataset": "php_fpm.pool",
"duration": 115000,
"module": "php_fpm"
},
"metricset": {
"host": "phpfpm:81",
"module": "php_fpm",
"name": "pool",
"rtt": 115
"name": "pool"
},
"php_fpm": {
"pool": {
"connections": {
"accepted": 1,
"accepted": 10,
"listen_queue_len": 0,
"max_listen_queue": 0,
"queued": 0
Expand All @@ -28,8 +30,12 @@
"total": 2
},
"slow_requests": 0,
"start_since": 670,
"start_time": 1512631228
"start_since": 600,
"start_time": 1548749474
}
},
"service": {
"address": "127.0.0.1:81",
"type": "php_fpm"
}
}
}
17 changes: 17 additions & 0 deletions metricbeat/module/php_fpm/pool/pool_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,27 @@ import (
"os"
"testing"

"github.com/stretchr/testify/assert"

"github.com/elastic/beats/libbeat/tests/compose"
mbtest "github.com/elastic/beats/metricbeat/mb/testing"
)

func TestFetch(t *testing.T) {
compose.EnsureUp(t, "phpfpm")

f := mbtest.NewReportingMetricSetV2(t, getConfig())
events, errs := mbtest.ReportingFetchV2(f)

assert.Empty(t, errs)
if !assert.NotEmpty(t, events) {
t.FailNow()
}

t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(),
events[0].BeatEvent("haproxy", "info").Fields.StringToPrint())

}
func TestData(t *testing.T) {
compose.EnsureUp(t, "phpfpm")
f := mbtest.NewReportingMetricSetV2(t, getConfig())
Expand Down
58 changes: 37 additions & 21 deletions metricbeat/module/php_fpm/process/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,25 @@
{
"@timestamp": "2018-09-25T16:47:04.998Z",
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"event": {
"dataset": "php_fpm.process",
"duration": 115000,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webmat I just realised now that we have here 2 values for event.duration. The event duration from metricbeat which is how long it took to get the event and the event duration reported by the metricset. Probably that means for consistency that the php_fpm duration should stay inside php_fpm?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depends on the semantics of the whole metricset. If there's 1 event per http request served, I think event.duration should be the http request duration.

If rather this is more of a sampling (e.g. get metrics every 10s, and report on the last http request), then the Metricbeat event duration is the one that should go to event.duration.

"module": "php_fpm"
},
"http": {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webmat Could you have a look if this is what you have expected? (check the full data.json)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, looks pretty good. One comment about method, but that's it

"request": {
"method": "get"
},
"response": {
"body": {
"bytes": 0
}
}
},
"metricset": {
"module": "php_fpm",
"host": "phpfpm:81",
"rtt": 24681267,
"name": "process"
},
"php_fpm": {
Expand All @@ -13,25 +29,25 @@
"process": {
"last_request_cpu": 0,
"last_request_memory": 2097152,
"request_duration": 1404,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was the request duration of the request processed by this process, that in this case happens to be the request to the status endpoint, but it could be any other thing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert this change as the two durations are different: https://github.com/elastic/beats/pull/10366/files#r251730563

"request_duration": 204,
"requests": 6,
"script": "-",
"pid": 6,
"start_time": 1537974596,
"user": "-",
"state": "Idle",
"start_since": 3946,
"requests": 28,
"request_method": "GET",
"request_uri": "/status?json=",
"content_length": 0
"start_since": 128,
"start_time": 1548769887,
"state": "Idle"
}
},
"beat": {
"name": "host.example.com",
"hostname": "host.example.com",
"version": "7.0.0-alpha1"
"process": {
"pid": 17
},
"service": {
"address": "127.0.0.1:81",
"type": "php_fpm"
},
"url": {
"original": "/status?full=\u0026json="
},
"host": {
"name": "DESKTOP-RFOOE09"
"user": {
"name": "-"
}
}
}
20 changes: 15 additions & 5 deletions metricbeat/module/php_fpm/process/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
release: ga
fields:
- name: pid
type: integer
type: alias
path: process.pid
migration: true
description: >
The PID of the process
- name: state
Expand All @@ -30,19 +32,27 @@
description: >
The duration in microseconds (1 million in a second) of the current request (my own definition)
- name: request_method
type: keyword
type: alias
path: http.request.method
migration: true
description: >
The request method (GET, POST, etc) (of the current request)
- name: request_uri
type: text
type: alias
path: url.original
migration: true
description: >
The request URI with the query string (of the current request)
- name: content_length
type: integer
type: alias
path: http.response.body.bytes
migration: true
description: >
The content length of the request (only with POST) (of the current request)
- name: user
type: keyword
type: alias
path: user.name
migration: true
description: >
The user (PHP_AUTH_USER) (or - if not set) (for the current request)
- name: script
Expand Down
49 changes: 34 additions & 15 deletions metricbeat/module/php_fpm/process/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package process

import (
"encoding/json"
"strings"

"github.com/elastic/beats/metricbeat/mb"

Expand Down Expand Up @@ -55,22 +56,40 @@ func eventsMapping(r mb.ReporterV2, content []byte) {
}
//remapping process details to match the naming format
for _, process := range status.Processes {
proc := common.MapStr{
"pid": process.PID,
"state": process.State,
"start_time": process.StartTime,
"start_since": process.StartSince,
"requests": process.Requests,
"request_duration": process.RequestDuration,
"request_method": process.RequestMethod,
"request_uri": process.RequestURI,
"content_length": process.ContentLength,
"user": process.User,
"script": process.Script,
"last_request_cpu": process.LastRequestCPU,
"last_request_memory": process.LastRequestMemory,
event := mb.Event{
RootFields: common.MapStr{
"http": common.MapStr{
"request": common.MapStr{
"method": strings.ToLower(process.RequestMethod),
},
"response": common.MapStr{
"body": common.MapStr{
"bytes": process.ContentLength,
},
},
},
"user": common.MapStr{
"name": process.User,
},
"process": common.MapStr{
"pid": process.PID,
},
"url": common.MapStr{
"original": process.RequestURI,
},
},
MetricSetFields: common.MapStr{
"state": process.State,
"start_time": process.StartTime,
"start_since": process.StartSince,
"requests": process.Requests,
"request_duration": process.RequestDuration,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request duration should go to event.duration

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

"script": process.Script,
"last_request_cpu": process.LastRequestCPU,
"last_request_memory": process.LastRequestMemory,
},
}
event := mb.Event{MetricSetFields: proc}

event.ModuleFields = common.MapStr{}
event.ModuleFields.Put("pool.name", status.Name)
r.Event(event)
Expand Down
20 changes: 19 additions & 1 deletion metricbeat/module/php_fpm/process/process_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,32 @@ import (
"os"
"testing"

"github.com/stretchr/testify/assert"

"github.com/elastic/beats/libbeat/tests/compose"
mbtest "github.com/elastic/beats/metricbeat/mb/testing"
)

func TestFetch(t *testing.T) {
compose.EnsureUp(t, "phpfpm")

f := mbtest.NewReportingMetricSetV2(t, getConfig())
events, errs := mbtest.ReportingFetchV2(f)

assert.Empty(t, errs)
if !assert.NotEmpty(t, events) {
t.FailNow()
}

t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(),
events[0].BeatEvent("php_fpm", "process").Fields.StringToPrint())

}

func TestData(t *testing.T) {
compose.EnsureUp(t, "phpfpm")
f := mbtest.NewReportingMetricSetV2(t, getConfig())
err := mbtest.WriteEventsReporterV2(f, t, "")
err := mbtest.WriteEventsReporterV2(f, t, ".")
if err != nil {
t.Fatal("write", err)
}
Expand Down