Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add HDFS check using snakebite #551

Merged
merged 6 commits into from
Jul 5, 2013
Merged

add HDFS check using snakebite #551

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d767032
add HDFS check using snakebite
dcrosta Jun 22, 2013
6ff3a94
add 'in_use' (percent used) gauge
dcrosta Jun 22, 2013
d2ae1e2
HDFSCheck: safely import snakebite.client
dcrosta Jul 2, 2013
bc904f3
HDFSCheck: raise exception rather than logging and returning
dcrosta Jul 2, 2013
1e40685
HDFSCheck: split hostport into 2 configuration items
dcrosta Jul 2, 2013
1165b07
whitespace
dcrosta Jul 2, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions checks.d/hdfs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
from checks import AgentCheck

import snakebite.client
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this import within the check method so it won't crash if the module is missing ?

Something like

try:
    import snakebite.client
except ImportError:
    raise Exception("Snake bite module is not installed. Please install it using easy install")


class HDFSCheck(AgentCheck):
"""Report on free space and space used in HDFS.
"""

def check(self, instance):
if 'namenode' not in instance:
self.log.info('Missing key \'namenode\' in HDFSCheck config')
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of ```python
return


you should do:

```python
raise Exception("Missing key 'namenode' in HDFSCheck config")

Doing so will display this message in the info page.


hostport = instance['namenode']
if ':' in hostport:
host, _, port = hostport.partition(':')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

string.partition is not 2.4 compatible.

Wouldn't it be easier to have a dedicated field in the config for the port ?

port = int(port)
else:
host = hostport
port = 8020

hdfs = snakebite.client.Client(host, port)
stats = hdfs.df()
# {'used': 2190859321781L,
# 'capacity': 76890897326080L,
# 'under_replicated': 0L,
# 'missing_blocks': 0L,
# 'filesystem': 'hdfs://hostname:port',
# 'remaining': 71186818453504L,
# 'corrupt_blocks': 0L}

self.gauge('hdfs.used', stats['used'])
self.gauge('hdfs.free', stats['remaining'])
self.gauge('hdfs.capacity', stats['capacity'])
self.gauge('hdfs.in_use', float(stats['used']) / float(stats['capacity']))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be surrounded by a try except to catch a possible ZeroDivisionError or the capacity will never be 0 ? (I don't really know hdfs)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do that if you like. It's very unlikely for the capacity (in bytes) to be 0 -- that would probably only happen if the hadoop cluster were seriously broken.

self.gauge('hdfs.under_replicated', stats['under_replicated'])
self.gauge('hdfs.missing_blocks', stats['missing_blocks'])
self.gauge('hdfs.corrupt_blocks', stats['corrupt_blocks'])

if __name__ == '__main__':
check, instances = HDFSCheck.from_yaml('./hdfs.yaml')
for instance in instances:
check.check(instance)
print "Events: %r" % check.get_events()
print "Metrics: %r" % check.get_metrics()

6 changes: 6 additions & 0 deletions conf.d/hdfs.yaml.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
init_config:
# HDFS check does not require any init_config

instances:
# Each instance requires a namenode hostname:port
- namenode: namenode.example.com:8020