Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Ganglia metrics to Riemann event stream processor #124

Merged
merged 17 commits into from
Nov 7, 2013

Conversation

satterly
Copy link
Member

@satterly satterly commented Nov 3, 2013

Previously, to forward metrics from Ganglia to Riemann the Graphite carbon protocol had to be used. This was not ideal because string metrics weren't supported and much of the meta data associated with a Ganglia metric were lost.

It is possible to encode some of this meta data (like grid, cluster and host) into the carbon metric name but this does not work for the host IP address, location data or any tags configured for that cluster or host. In addition there is no way to pass arbitrary key-value attributes with a metric.

Using the native Riemann protocol based on Google Protocol Buffers solves these issues and mapping Ganglia metric attributes to Riemann fields is straight-forward.

# +-------------------+----------------+
# |  Ganglia          |  Riemann       |
# |-------------------|----------------|
# |  grid             |  grid          |
# |  cluster          |  cluster       |
# |  host             |  host*         |
# |  ip               |  ip            |
# |  metric           |  service*      |
# |  value(int,float) |  metric*       |
# |  type             |  (internal)    |
# |  units            |  description*  |
# |  value(string)    |  state*        |
# |  reported         |  time*         |
# |  tags(comma-sep)  |  tags*         |
# |  location         |  location      |
# |  tmax             |  ttl*          | 
# +-------------------+----------------+

For example, a Ganglia metric like the following:

<HOST NAME="myhost" IP="10.0.2.15" TAGS="os:linux,cloud=AWS/EC2" REPORTED="1383446781" TN="3" TMAX="20" DMAX="86400" LOCATION="madrid" GMOND_STARTED="1383446781">LOCATION="madrid" GMOND_STARTED="1383444832">
<METRIC NAME="swap_free" VAL="391768" TYPE="float" UNITS="KB" TN="13" TMAX="180" DMAX="0" SLOPE="both">
</METRIC>
</HOST>

Will be marshalled by Riemann as:

#riemann.codec.Event{
  :host "myhost",
  :service "swap_free",
  :state nil,
  :description "KB",
  :metric 391768.0,
  :tags ["os:linux" "cloud=AWS/EC2"],
  :time 345861311493/250,
  :ttl 180.0,
  :environment "PROD",
  :customer "Acme Corp",
  :location "madrid",
  :ip "10.0.2.15",
  :cluster "unspecified",
  :grid "MyGrid"
}

The support for forwarding metrics to Riemann is a compile-time option to the configure script. ie. "--with-riemann".

jbuchbinder added a commit that referenced this pull request Nov 7, 2013
Add support for Ganglia metrics to Riemann event stream processor
@jbuchbinder jbuchbinder merged commit faef257 into ganglia:master Nov 7, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants