loggly-api

Description

This library allows a user to work with Loggly's APIs through simple function calls.

Note: As of version 0.3.0, this library only supports Loggly's API v2 as described here. Additionally, only API tokens are supported as an authentication type. Username/password (HTTP Basic Auth) support has been ripped out. Other breaking changes have also been implemented, such as the change of the loggly.token variable to loggly.customer_token to be consistent with Loggly's terminology. RequestErrors no longer represent HTTP responses. They are now called ResponseErrors. A RequestError is now something completely different, and is documented below. InvalidStatError no longer exists, and has been converted into a RequestError.

If your code relies on these features and you cannot update your code, install version 0.2.3 of this library.

Installation

pip install loggly-api

You're done.

Unless you need an old version, in which case, you should try:

pip install loggly-api<0.3.0

Usage

import loggly

You can set various access credentials; different ones are needed for different operations.

loggly.subdomain = 'your_subdomain'
loggly.customer_token = 'your_customer_token'
loggly.api_token = 'your_api_token'

Ship One Event, or a Multiline Event

You need a customer token to ship logs (as opposed to an API token).

The Content-Type header of the request will be automatically determined, so you can ship JSON by passing in a dict or a JSON-parseable string. Anything else gets treated as plaintext.

loggly.submit('Event data')       # Plaintext
loggly.submit({ 'foo': 'bar' })   # JSON
loggly.submit('{ "foo": "bar" }') # JSON

All forms of submit support tagging the events by setting the tag=whatever argument.

loggly.submit('Event data', tag='production')

Ship Bulk Data

You need a customer token to ship logs.

You can pass in a list of events...

loggly.bulk_submit([
    'Event one',
    'Event two',
    'Event three'])

...or a single string.

with open('/path/to/a/big/log.file', 'r') as fh:
    loggly.bulk_submit(fh.read())

Both forms support tagging.

loggly.bulk_submit(eventlist, tag='production')

Bear in mind there are restrictions to bulk shipping. At the time of this writing, you can't send single events larger than 1 MB each, and no more than 5 MB per request.

Search Events

To search events, you need a subdomain and an API token.

Note: Loggly has deprecated their single-block retrieval API endpoint, and this library does not support it. It doesn't matter, though, since the library obscures the method of retrieval and only presents the outcome of the search.

You can start a search like this, and the process will hang until search results are returned by the API, at which point you'll have a list of events:

events = loggly.search('foo:"bar"', frm='-1D', til='now')

Or you can set paginate=True to instead get a SearchIterator back immediately. Each "next" call will cause a request for more results to come from Loggly. This is essentially a less-automated way of accomplishing the same thing as above, except you can run other code between pages of results. Consider:

search = loggly.search('foo:"bar"', frm='-1D', til='now', paginate=True, pagesize=100)
events = list()
for page in search:
    events.extend(page)
    print('{} events collected so far!'.format(len(events)))

pagesize is the number of results to return per page. Loggly limits this to 1,000 events at most, but this can sometimes cause problems as well. If you find yourself getting inexplicable errors from Loggly, try making your page size smaller. The effects of this setting are not always obvious when paginate is False, although the page size is applied in that case, fewer API calls are made, and the function may return faster. But when it's True, you'll get back that many events at a time from the iterator.

Event Count

To count events, you need a subdomain and an API token.

Event counts support the same querying and time-boxing options as searching. It also supports the include_volume option, which reports the byte size of the events queried. This function call returns a dictionary that always contains a count field, and which also contains a volume field if you've asked for it. For example:

counts = loggly.count('foo:"bar"', frm='-1D', til='now', include_volume=True)

>>> {'count': 2574134, 'volume': 3095600785}

Account Info

Retrieving account info requires a subdomain and an API token.

info = loggly.account_info()

Calling account_info() also updates some convenience properties in the module:

loggly.customer_tokens
loggly.retention
loggly.subdomain
loggly.volume_limit

Statistics

The stats API requires a subdomain and an API token.

You can call the stats function to get statistical data about numerical fields in your log events. In the fictional example below, you'd get back the average request time for your app.

stats = loggly.stats(stat='avg', field='json.log.request_time', query='*', frm=None, til=None)

Refer to the documentation for a complete list of valid statistics you can call for.

Volume Metrics

The volume metrics API requires a subdomain and an API token.

First of all, this doesn't appear to be a documented feature. I've recieved a PDF of docs from an account representative that I can't find a corresponding website for.

This call lets you gather broad aggregate statistics about the number of events and their total size within a period of time, filtered by the host and "app" that generated the events. The following call would return the stats from the past day.

metrics = loggly.volume_metrics(frm='-1d', til='now')

Other parameters that are supported:

• group_by (str)
• host (str)
• app (str)
• measurement_types (list of str)

Exceptions

The base class for all exceptions in this module is loggly.LogglyError, but it's nothing special and is useless except to verify that some other error is a loggly error.

print(isinstance(loggly.RequestError, loggly.LogglyError))

Output:
  True

A ResponseError is thrown anytime this module makes a request to Loggly and gets something other than a 200 OK in response. It will contain a requests response object as well as some convenience members which are extracted from it:

try:
    stats = loggly.stats(field='json.log.bleh')
except loggly.ResponseError as e:
    print('{} {}\n{}'.format(e.status_code, e.reason, e.message))

Output:
 400 BAD REQUEST
 The requested field 'json.log.bleh' is unknown.

A RequestError, by contrast, is thrown when the library is able to detect a problem in the formation of a request before the request is made. It contains a reason field which will contain a plaintext explanation of the problem.

try:
    loggly.bulk_submit({'adict': 'is invalid here'})
except loggly.RequestError as e:
  print(e.reason)

Output:
  events must be a list or string

An AuthenticationError is raised when you try to make an API call without providing sufficient credentials for the operation.

try:
    loggly.subdomain = 'mysub'
    events = loggly.search()
except AuthenticationError as e:
    print('You didn't set an api_token!')