This tutorial will guide you through the process of developing a custom integration for New Relic Infrastructure in the Go language. To simplify the process, New Relic provides the following tools:
- nr-integrations-builder: command line tool that generates an integration "scaffold", with basic integration files in the correct directory structure.
- Integration Golang SDK: a Golang package containing a set of useful functions and types for creating metrics and inventory data structure.
For a simple overview of what Infrastructure integrations are and how they work, see the Intro to the Integrations SDK.
This tutorial is compatible with nr-integration-builder
v1.0.x and Integration Golang SDK
v1.0.x.
To successfully complete this tutorial you must:
- Be familiar with Golang
- Install the Vendor Tool for Go
- Have access to a supported Linux OS
- Install Go
- Install Redis
- Install the New Relic Infrastructure Agent
Step1: Install nr-integrations-builder
Install nr-integrations-builder
$ go get github.com/newrelic/nr-integrations-builder
This command will create the nr-integrations-builder
executable. Depending on your Go tools settings it could be placed inside $GOBIN
or $GOPATH/bin
. Make sure the directory of the folder with your Go binaries is placed in your $PATH
environment variable and run
$ nr-integrations-builder --help
If the Golang binary folder is not in your $PATH
you will have to run
$ $GOBIN/nr-integrations-builder --help
or
$ $GOPATH/bin/nr-integrations-builder --help
This tutorial assumes that your $GOBIN
or $GOPATH/bin
has been added to your $PATH
environment variable.
Step 3: Check govendor tool
Before initializing the integration with nr-integrations-builder
you have to check that the govendor
tool (used for managing dependencies) is successfully installed. Run the following command:
$ govendor
You should receive the description about the govendor
tool with the list of accepted commands. More information about the usage can be found in the README.md.
Step 4: Initialize the integration
To see the list of the parameters that you can specify for nr-integrations-builder
, type
$ nr-integrations-builder init --help
You will receive the following output
Initialize an integration generating a scaffold.
Usage:
nr-integrations-builder init [integration name] [flags]
Flags:
-n, --company-name string Company name (required)
-c, --company-prefix string Company prefix identifier (required)
-p, --destination-path string Destination path for initialized integration (default "./")
-h, --help help for init
Global Flags:
--config string config file (default is $HOME/.nr-integrations-builder.yaml)
--verbose verbose output
It's obligatory to specify company-name
and company-prefix
flags. Otherwise, the nr-integrations-builder
will not initialize the integration.
To initialize the integration and generate the scaffold, run
$ nr-integrations-builder init integration_name --company-name "your-company-name" --company-prefix "your-company-prefix"
After initializing the integration you should receive information that the scaffold was successfully created. If it failed you will get an error message.
Your current directory will be used as the default destination. The following structure of the files and folders will be created:
- integration_name
- CHANGELOG.md
- LICENSE
- README.md
- Makefile
- company-prefix-integration_name-config.yml.sample
- company-prefix-integration_name-definition.yml
- src
- integration_name.go
- integration_name_test.go
- vendor
- vendor.json
- external_packages_name
Step1: Create the directory where you want to place the Redis integration (it needs to be under $GOPATH/src
)
$ mkdir $GOPATH/src/myorg-integrations/
$ cd $GOPATH/src/myorg-integrations/
Step2: Initialize the integration
$ nr-integrations-builder init redis --company-prefix "myorg" --company-name "myorganization"
Step 3: Go to the directory you created. Before building the executable file you have to format the Go source code using gofmt tool. If you don't do this, the make
command will fail and the executable file will not be created.
$ cd redis/
$ go fmt src/redis.go
Step 4: Build the executable file and test that the integration was created properly
$ make
$ ./bin/myorg-redis -pretty
The following JSON payload will be printed to stdout:
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample"
}
],
"inventory": {},
"events": []
}
This is the basic JSON data format that is expected by the Infrastructure agent. The main logic is placed in src/redis.go
, which is the source that will be compiled into the integration executable file.
When the integration is initialized with nr-integrations-builder
, the
executable file builds the JSON output data with three header fields (name,
protocol_version,
integration_version),
metrics data (with
the mandatory event_type field), and an empty structure for inventory and events data.
The SDK package contains a function called NewIntegration
, which initializes a new instance of integration data. If you run the following simplified main
function that calls NewIntegration
:
func main() {
integration, err := sdk.NewIntegration(integrationName, integrationVersion, &args)
fatalIfErr(err)
// the code for populating Inventory and Metrics omitted
fatalIfErr(integration.Publish())
}
you will receive the following output:
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [],
"inventory": {},
"events": []
}
The complete files for the Redis integration can be found in tutorial-code.
Let's start by defining the metric data. MetricSet
is the basic structure for storing metrics. The NewMetricSet
function returns a new instance of MetricSet with its sample attached to the integration data.
Next, if you think it's necessary, modify the argument for NewMetricSet
in the
code. By default, nr-integrations-builder
generates an Event Type
automatically using the company-prefix
flag (that you specified initializing the integration), name of the integration and the word: 'Sample'. Your main
function should look like:
func main() {
integration, err := sdk.NewIntegration(integrationName, integrationVersion, &args)
fatalIfErr(err)
// the code for populating Inventory omitted
if args.All || args.Metrics {
ms := integration.NewMetricSet("MyorgRedisSample")
fatalIfErr(populateMetrics(ms))
}
fatalIfErr(integration.Publish())
}
After building, formatting the source code and executing the integration the following output is returned:
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample"
}
],
"inventory": {},
"events": []
}
In order to define the metric value, we will use the function SetMetric
from the SDK package. nr-integrations-builder
automatically generates the populateMetrics
function:
func populateMetrics(ms *metric.MetricSet) error {
// Insert here the logic of your integration to get the metrics data
// Ex: ms.SetMetric("requestsPerSecond", 10, metric.GAUGE)
// --
return nil
}
Uncomment the line that calls the SetMetric
function:
func populateMetrics(ms *metric.MetricSet) error {
// Insert here the logic of your integration to get the metrics data
ms.SetMetric("requestsPerSecond", 10, metric.GAUGE)
// --
return nil
}
and build, format Go source code (using gofmt
tool) and execute the integration. You will receive the following output:
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample",
"requestsPerSecond": 10
}
],
"inventory": {},
"events": []
}
The function SetMetric
requires three arguments. The first one is the metric name, the second is the metric value, and the last one is the source type of the metric.
The metric source type can be one of the following: GAUGE, RATE, DELTA or ATTRIBUTE. Continue reading to understand how to use the different source types.
The redis-cli info
command returns a list of redis performance and health metrics.
If you run:
redis-cli info | grep instantaneous_ops_per_sec:
you will receive:
instantaneous_ops_per_sec:4
This is the number of commands processed per second. This is a numeric value
that may increase or decrease and it should be stored as-is. Use the GAUGE
source type in these cases. For metric names, it is recommended that you use a prefix to categorize
them (check the currently used prefixes), innerCamelCase naming format, and specify the measurement unit using a unit suffix, i.e. PerSecond. In this case, for the metric data key, use query.instantaneousOpsPerSecond
:
func populateMetrics(ms *metric.MetricSet) error {
cmd := exec.Command("/bin/sh", "-c", "redis-cli info | grep instantaneous_ops_per_sec:")
output, err := cmd.CombinedOutput()
if err != nil {
return err
}
splittedLine := strings.Split(string(output), ":")
if len(splittedLine) != 2 {
return fmt.Errorf("Cannot split the output line")
}
metricValue, err := strconv.ParseFloat(strings.TrimSpace(splittedLine[1]), 64)
if err != nil {
return err
}
ms.SetMetric("query.instantaneousOpsPerSecond", metricValue, metric.GAUGE)
return nil
}
In order to continue to build the source, you'll need to add the needed packages to the import statemement at the top of the program:
import (
"fmt"
sdkArgs "github.com/newrelic/infra-integrations-sdk/args"
"github.com/newrelic/infra-integrations-sdk/log"
"github.com/newrelic/infra-integrations-sdk/metric"
"github.com/newrelic/infra-integrations-sdk/sdk"
"os/exec"
"strconv"
"strings"
)
After building, formatting the source code, and executing the integration, you should receive:
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample",
"query.instantaneousOpsPerSecond": 3
}
],
"inventory": {},
"events": []
}
Using the command:
redis-cli info | grep total_connections_received:
you will receive:
total_connections_received:111
This provides information about the total number of connections accepted by the server. This is an ever-growing value which might be reset. In a case like this it is more useful to store the change rate instead of the as-is value. We use the RATE type and the SDK will automatically compute the change rate.
Modify the populateMetrics
function to process the metric data using the RATE type:
func populateMetrics(ms *metric.MetricSet) error {
// ...
// previous code for getting `instantaneous_ops_per_sec` omitted
// ...
cmd = exec.Command("/bin/sh", "-c", "redis-cli info | grep total_connections_received:")
output, err = cmd.CombinedOutput()
if err != nil {
return err
}
splittedLine = strings.Split(string(output), ":")
if len(splittedLine) != 2 {
return fmt.Errorf("Cannot split the output line")
}
metricValue, err = strconv.ParseFloat(strings.TrimSpace(splittedLine[1]), 64)
if err != nil {
return err
}
ms.SetMetric("net.connectionsReceivedPerSecond", metricValue, metric.RATE)
return nil
}
Build, format the source code, and execute the integration, and then check the output (note: your metric values may vary.)
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample",
"net.connectionsReceivedPerSecond": 2,
"query.instantaneousOpsPerSecond": 3
}
],
"inventory": {},
"events": []
}
The calculations for a given metric source type are handled by SetMetric
; the only information you need to provide is the desired source type. Besides RATE and GAUGE type, there is a DELTA type, which is similar to RATE, but it is calculated as the difference between samples, not as a rate change, and the ATTRIBUTE type, which is used for string values.
This method of fetching data, shown above is not very efficient. You will want to fetch a set of data all at once, but this example just shows how to use the SetMetric
function and the source types.
Let's look now at the definition file of the redis integration. In the file myorg-redis-definition.yml under command you can specify common arguments for all instances (that you will define in the config file) that you want to monitor. In this case we have just one common argument: --metrics
.
name: com.myorganization.redis
description: Reports status and metrics for redis service
protocol_version: 1
os: linux
commands:
metrics:
command:
- ./bin/myorg-redis
- --metrics
interval: 15
# configuration for the inventory omitted
The config file generated by nr-integrations-builder
is a sample.
integration_name: com.myorganization.redis
instances:
- name: <INSTANCE IDENTIFIER>
command: metrics
arguments:
arg1: <ARG_VALUE>
labels:
key1: <LABEL_VALUE>
# configuration for the inventory omitted
Rename it to myorg-redis-config.yml
. It is required to specify instances that you want to monitor. Arguments and labels parameters are not mandatory. For fetching metric data for the redis integration there is no argument needed. But we can specify the label with the environment name and the role. Make sure that you use valid YAML file format.
integration_name: com.myorganization.redis
instances:
- name: redis-server-metrics
command: metrics
labels:
env: production
role: cache
# configuration for the inventory omitted
This configuration is only for metric data. The configuration for inventory will be done further along in this tutorial.
The last configuration step for metrics is to place the integration file in the directory used by the Infrastructure agent. Place the executable and the definition file in /var/db/newrelic-infra/custom-integrations/
$ sudo cp $GOPATH/src/myorg-integrations/redis/myorg-redis-definition.yml /var/db/newrelic-infra/custom-integrations/myorg-redis-definition.yaml
$ sudo cp -R $GOPATH/src/myorg-integrations/redis/bin /var/db/newrelic-infra/custom-integrations/
Place the integration config file in /etc/newrelic-infra/integrations.d/
$ sudo cp $GOPATH/src/myorg-integrations/redis/myorg-redis-config.yml /etc/newrelic-infra/integrations.d/myorg-redis-config.yaml
When all the above steps are done, restart the agent.
For more information, see the configuration file and definition file documentation.
When the integration and the Infrastructure agent are communicating correctly, you can view your metric data in New Relic Insights.
Below are example NRQL queries for the MyorgRedisSample
event type.
NRQL> SELECT average(`net.connectionsReceivedPerSecond`) FROM MyorgRedisSample TIMESERIES
NRQL> SELECT average(`query.instantaneousOpsPerSecond`) FROM MyorgRedisSample TIMESERIES
For more about creating NRQL queries, see Introduction to NRQL. For more on where to find integration data in New Relic products, see Find and use integration data.
If you do not see your metric data in New Relic Insights, please check configuration of the Infrastructure agent.
The populateInventory
function is used to fetch inventory data:
func populateInventory(inventory sdk.Inventory) error {
// Insert here the logic of your integration to get the inventory data
// Ex: inventory.SetItem("softwareVersion", "value", "1.0.1")
// --
return nil
}
Notice that in the code above we use the Inventory
type from the SDK package for inventory data. The SetItem
method stores a value into the inventory data structure. The first argument is the name of the inventory item, and the other two are a field name and the inventory data value.
Let's assume that we want to collect configuration information for Redis. For example, let's say we'd like to capture the value of the dbfilename
parameter. The command
redis-cli CONFIG GET dbfilename
gives the following result
1) "dbfilename"
2) "dump.rdb"
To parse this output and create the proper inventory data structure, modify the populateInventory
function:
func populateInventory(inventory sdk.Inventory) error {
cmd := exec.Command("/bin/sh", "-c", "redis-cli CONFIG GET dbfilename")
output, err := cmd.CombinedOutput()
if err != nil {
return err
}
splittedLine := strings.Split(string(output), "\n")
if splittedLine[0] == "dbfilename" {
inventory.SetItem(splittedLine[0], "value", splittedLine[1])
}
return nil
}
After building, formatting the source code and executing the integration (with just inventory data)
$ ./bin/myorg-redis -pretty -inventory
we receive
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [],
"inventory": {
"dbfilename": {
"value": "dump.rdb"
}
},
"events": []
}
Let's extend our populateInventory
function in order to collect the bind configuration parameter from Redis. By executing
redis-cli CONFIG GET bind
we get
1) "bind"
2) "127.0.0.1"
To add this to our integration we can update the populateInventory
function:
func populateInventory(inventory sdk.Inventory) error {
cmd := exec.Command("/bin/sh", "-c", "redis-cli CONFIG GET dbfilename")
output, err := cmd.CombinedOutput()
if err != nil {
return err
}
splittedLine := strings.Split(string(output), "\n")
if splittedLine[0] == "dbfilename" {
inventory.SetItem(splittedLine[0], "value", splittedLine[1])
}
cmd = exec.Command("/bin/sh", "-c", "redis-cli CONFIG GET bind")
output, err = cmd.CombinedOutput()
if err != nil {
return err
}
splittedLine = strings.Split(string(output), "\n")
if splittedLine[0] == "bind" {
inventory.SetItem(splittedLine[0], "value", splittedLine[1])
}
return nil
}
Finally, build, format the source code and execute the integration to fetch all inventory and metric data.
$ go fmt src/redis.go
$ ./bin/myorg-redis -pretty
{
"name": "com.myorganization.redis",
"protocol_version": "1",
"integration_version": "0.1.0",
"metrics": [
{
"event_type": "MyorgRedisSample",
"net.connectionsReceivedPerSecond": 0.5,
"query.instantaneousOpsPerSecond": 1
}
],
"inventory": {
"bind": {
"value": "127.0.0.1"
},
"dbfilename": {
"value": "dump.rdb"
}
},
"events": []
}
In the definition file, myorg-redis-definition.yml
, we want to increase the interval
value for the inventory. This is because the changes in the inventory data are not as frequent as in metrics data.
name: com.myorganization.redis
description: Reports status and metrics for redis service
protocol_version: 1
os: linux
commands:
metrics:
command:
- ./bin/myorg-redis
- --metrics
interval: 15
inventory:
command:
- ./bin/myorg-redis
- --inventory
prefix: config/myorg-redis
interval: 60
myorg-redis-definition.yml
contains a prefix
parameter that defines the source of the inventory data and can be used as a filter to see your inventory data on the Infrastructure Inventory page.
The prefix
is customizable, typically of the form category/integration-name. You can select maximum two levels (i.e. if you use three levels: config/myorg-redis/custom
, you won't be able to view your inventory data.)
Next, let's look at the config file:
integration_name: com.myorganization.redis
instances:
- name: redis-server-metrics
command: metrics
labels:
env: production
role: cache
- name: <OTHER INSTANCE IDENTIFIER>
command: inventory
arguments:
arg1: <ARG_VALUE>
labels:
key1: <LABEL_VALUE>
Specify the name of the instance and add two arguments: hostname
and port
. Make sure that you use YAML's indentation correctly.
integration_name: com.myorganization.redis
instances:
- name: redis-server-metrics
command: metrics
labels:
env: production
role: cache
- name: redis-server-inventory
command: inventory
arguments:
hostname: localhost
port: 6379
labels:
env: production
role: cache
The arguments are made available to an integration as a set of environment variables. It's necessary to modify the argumentList
type. This is what it looks like before the change:
type argumentList struct {
sdkArgs.DefaultArgumentList
}
And after the required change:
type argumentList struct {
sdkArgs.DefaultArgumentList
Hostname string `default:"localhost" help:"Hostname or IP where Redis server is running."`
Port int `default:"6379" help:"Port on which Redis server is listening."`
}
To finish the inventory configuration place the executable and the definition file in /var/db/newrelic-infra/custom-integrations/
$ sudo cp $GOPATH/src/myorg-integrations/redis/myorg-redis-definition.yml /var/db/newrelic-infra/custom-integrations/myorg-redis-definition.yaml
$ sudo cp -R $GOPATH/src/myorg-integrations/redis/bin /var/db/newrelic-infra/custom-integrations/
Place the integration config file in /etc/newrelic-infra/integrations.d/
$ sudo cp $GOPATH/src/myorg-integrations/redis/myorg-redis-config.yml /etc/newrelic-infra/integrations.d/myorg-redis-config.yaml
When all the above steps are done, restart the agent.
Inventory data can be viewed in New Relic Infrastructure on the Inventory page. Filter by prefix config/myorg-redis
(which was specified in the definition file) and you will see the inventory data collected by the redis integration and labels that you specified in the config file.
See more about how inventory data shows up in the New Relic UI in Find integration inventory data.