simpledb

If you're using this library, feel free to contact me on twitter if you have any questions! :) @rjrodger

Current Version: 0.0.7

Tested on: node 0.4.4

A user-friendly fault-tolerant library for Amazon AWS SimpleDB access. The core SimpleDB actions are mapped to functions:

var simpledb = require('simpledb')
var sdb      = new simpledb.SimpleDB({keyid:'YOUR_AWS_KEY_ID',secret:'YOUR_AWS_SECRET_KEY'})

sdb.createDomain( 'yourdomain', function( error ) {

  sdb.putItem('yourdomain', 'item1', {attr1:'one', attr2:'two'}, function( error ) {
  
    sdb.getItem('yourdomain', 'item1', function( error, result ) {
      console.log( 'attr1 = '+result.attr1 )
      console.log( 'attr2 = '+result.attr2 )
    })
  })
})

Any given SimpleDB request has a non-trivial chance of failing. This library implements the exponential back-off retry algorithm as recommended in the SimpleDB developer guide.

This library depends on the excellent aws-lib module: aws-lib

Key Features:

• simple API
• fully configurable
• detailed logging
• all request attributes can be overridden
• fully tested

Core Functions:

• createDomain ("CreateDomain")
• domainMetadata ("DomainMetadata")
• listDomains ("ListDomains")
• deleteDomain ("DeleteDomain")
• putItem ("PutAttributes")
• batchPutItem ("BatchPutAttributes")
• batchDeleteItem ("BatchDeleteAttributes")
• getItem ("GetAttributes")
• deleteItem ("DeleteAttributes")
• select ("Select")
• request (any action)

This is still an early version so there's probably some wierdness - use at your risk. Secure connections are not supported on node 0.3.x.

Installation

npm install simpledb

And in your code:

var simpledb = require('simpledb')

Or clone the git repository: git clone git://github.com/rjrodger/simpledb.git

The simpledb module depends on the aws-lib module. npm will install this automatically.

Usage

This module uses the node.js-style callback convention. All functions take a callback function as their last argument. This callback function should accept three arguments:

callback( error, result, meta )

Where error is an object ({Code:'...',Message:'...'}) describing any errors that occured. If the function was successful then error is null.

So, you check if error is null to see if you can continue working:

sdb.listDomains( function( error, result, meta ) {
  if( error ) {
    console.log('listDomains failed: '+error.Message )
  }
  else {
    // do stuff with result, an array of domain names
  }
})

The result parameter contains the results of a successful action and what the result parameter is depends on the action. It could be a string, an array or an object.

The meta parameter contains a description of the request, including the underlying details from Amazon. Take a look with:

console.log( JSON.stringify(meta) )

Conventions

Where possible, the SimpleDB naming style is preserved: CamelCaseBaby. Names of functions and their parameters also mostly match SimpleDB.

The simpledb.SimpleDB wrapper options (maxtry, secure, etc) are not directly related to Amazon, and so have their own names.

It is sometimes necessary to embed meta-directives into the Amazon query or result objects. These non-Amazon attributes always begin with the $ character, but are in CamelCase. For example: $AsArrays.

This wrapper is based on the REST API. I looked at the SOAP API but... yeah. No X.509 for you. Yet.

API

For the API examples, assume the following lines of code at the top of your source code file:

var simpledb = require('simpledb')

var sdb = new simpledb.SimpleDB(
  {keyid:'YOUR_AWS_KEY_ID',secret:'YOUR_AWS_SECRET_KEY'},
  simpledb.debuglogger
)

This gives you the standard wrapper, with a basic debugger that prints to STDOUT.

You should really also read the Amazon SimpleDB documentation so that you understand how SimpleDB works: Amazon SimpleDB Developer Guide

As a get-out-of-jail, you can provide request attribute overrides. You supply these in an optional override argument just before the callback argument. You can use an override on any of the SimpleDB action wrapper functions.

sdb.getItem('domain','itemname', {ConsistentRead:'false'} ,function(err,res,meta){ ... })

In the above code, {ConsistentRead:"false"} is the optional override argument.

simpledb.SimpleDB: simpledb.SimpleDB( options, logger )

• options: (required) set of options; keyid and secret are required

• logger: (optional) callback for log events

  var sdb = new simpledb.SimpleDB( options, logger )

Create a new SimpleDB wrapper. The options argument sets general options for the requests. The logger argument receives logging events so that you can debug and/or record SimpleDB interactions.

options: required

• keyid: (required), your Amazon AWS Key ID
• secret: (required), your Amazon Secret Key

For further options, see the section on options below

logger: optional

See the section on logging below

createDomain: sdb.createDomain(domain,override,callback)

• domain: (required) the name of the domain
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Create a domain. A domain is like a SQL table, sort of.

sdb.createDomain('<domain>',function(err,res,meta){
  if( !err ) {
    console.log('Mul-ti-pass!')
  }
}

Where <domain> is the name of your domain.

domainMetadata: sdb.domainMetadata(domain,override,callback)

• domain: (required) the name of the domain
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Get some statistics about your domain, such as a count of items and how much storage it is using (you pay Amazon for this!).

sdb.domainMetadata('<domain>',function(err,res,meta){
   console.log('Mmm, floor pie! '+JSON.stringify(res) )
}

Where <domain> is the name of your domain.

listDomains: sdb.listDomains(override,callback)

• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Returns a list of your domain names as an array of strings. Restricted to the specified SimpleDB host (default=sdb.amazonaws.com). See the simpledb.SimpleDB options to change this.

sdb.listDomains(function(err,res,meta){
   console.log('You hear that? That's market bacon hitting the pan: '+JSON.stringify(res) )
}

deleteDomain: sdb.deleteDomain(domain,override,callback)

• domain: (required) the name of the domain
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete a domain. Cannot be undone!

sdb.deleteDomain('<domain>',function(err,res,meta){
  if( !err ) {
    console.log('God made the world, but we made the field.')
  }
}

Where <domain> is the name of your domain.

putItem: sdb.putItem(domain,itemname,attrs,override,callback)

• domain: (required) the name of the domain
• itemname: (required) the unique name of the item in the domain
• attrs: (required) the item attributes to store
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Store an item in SimpleDB.

sdb.putItem('<domain>','<itemname>', 
  {
    <attr>:'<value>',
    ...
  },
  function(err,res,meta){
    console.log("Memories, you're talking about memories: "+JSON.stringify(res)) 
  })

Where <itemname> is the unique name of your item, and <attr>:"<value>" are the attribute-value pairs for your item. The value must be either a string or an array of strings.

If you want to use conditional puts, you'll need to add some override values:

sdb.putItem('<domain>','<itemname>', 
  {
    <attr1>:'<value>',
    <attr2>:['<value1>','<value2>',...]
    ...
  },
  {
    'Expected.1.Name':'VersionNumber',
    'Expected.1.Value':'1'
  },
  function(err,res,meta){
    console.log("Nobody expects the spanish inquistion! "+JSON.stringify(res)) 
  })

batchPutItem: sdb.batchPutItem( domain, items, override, callback )

• domain: (required) the name of the domain
• items: (required) the list of items to store
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Store multiple items in the same request. This is more efficient. The items argument is an array of item objects. Each item object must have a $ItemName meta-attribute that specifies the name of the item.

sdb.batchPutItem('<domain>',
  [ 
    { $ItemName:'<itemname1>', <attr>:'<value>', ...}, 
    { $ItemName:'<itemname2>', <attr>:'<value>', ...}
  ],function(err,res,meta){
    console.log("And what was your ownership share diluted down to?"+JSON.stringify(res)) 
  })

batchDeleteItem: sdb.batchDeleteItem( domain, items, override, callback )

• domain: (required) the name of the domain
• items: (required) the list of items to delete
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete multiple items in one request. This is more efficient. The items argument is an array of item objects. Each item object must have a $ItemName meta-attribute that specifies the name of the item.

sdb.batchDeleteItem('<domain>',
  [ 
    { $ItemName:'<itemname1>', <attr>:'<value>', ...}, 
    { $ItemName:'<itemname2>', <attr>:'<value>', ...}
  ],function(err,res,meta){
    console.log("Done"+JSON.stringify(res)) 
  })

getItem: sdb.getItem( domain, itemname, override, callback )

• domain: (required) the name of the domain
• itemname: (required) the unique name of the item in the domain
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Get an item from SimpleDB using the item's unique name. The values of the item's attributes are returned as strings. You can provide an $AsArrays meta-directive in the override argument. When true, all attribute values are returned as arrays. As SimpleDb is schemaless, it is not possible to know in advance if an attribute is multi-valued. In the default case, {$AsArrays:false}, multiple values are returned as string, with the value list comma-separated. SimpleDB returns multiple values in alphabetical order.

sdb.getItem('<domain>','<itemname>',function( error , result, meta ){
  console.log("Those are good burgers, Walter: "+JSON.stringify(res)) 
})

sdb.getItem('<domain>','<itemname>',{$AsArrays:true},function( error, result, meta ){
  console.log("I've been watching television so much the shows are starting to run together: "+JSON.stringify(res)) 
})

By default, simpledb uses consistent reads. For improved performance, if this is suitable for your application, you can set the consistent option to false when creating simpledb.SimpleDB. Or you can set it on a case-by-case basis, using an override: {ConsistentRead:"false"}

deleteItem: sdb.deleteItem( domain, itemname, attrs, override, callback )

• domain: (required) the name of the domain
• itemname: (required) the unique name of the item in the domain
• attrs: (optional) the attributes to delete
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete an item from SimpleDB. The attrs argument is optional, and can be:

• an array of attribute names: all matching attributes will be deleted
• an object whose properties are attribute names: attributes of the item will be deleted if they have the same value as the object properties. Values can be either a single string, or an array of string values, in which case all matching attributes are deleted.

If no attributes are specified, the item is completely removed. If present, only the specified attributes are removed. If all the attributes of an item are removed, then it will also be completely deleted.

sdb.deleteItem('<domain>','<itemname>',function( error, result, meta ){
  console.log("Well, Ted, like I said the last time: it won't happen again: "+JSON.stringify(res)) 
})

sdb.deleteItem('<domain>','<itemname>',[ '<attr>', ... ], function( error, result, meta ){
  console.log("I felt like destroying something beautiful. "+JSON.stringify(res)) 
})

sdb.deleteItem('<domain>','<itemname>',
  { '<attr1>': '<value1>', 'attr2': ['<value2>, ... ], ... }, 
  function( error, result, meta ){
    console.log("I don't know what to write about. "+JSON.stringify(res)) 
  }
)

select: sdb.select( query, override, callback )

• query: (required) SimpleDB select expression
• override: (optional) SimpleDB attributes to override function defaults
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Perform a SELECT-style query on a SimpleDB domain. The syntax is almost-but-not-quite SQL. You should read the Amazon documentation: Using Select

The results are returned as an array of items. Each item contains an $ItemName meta-attribute providing you with the name of the item.

If you need to handle NextToken you'll need to do this manually with the override argument. You can get the NextToken from the meta parameter to your callback.

sdb.select("select * from <domain> where <attribute> = '<value>'",function( error, result, meta ){
  console.log("I'll get you, my pretty, and your little dog too! "+JSON.stringify(result)+" "+JSON.stringify(meta)) 
})

request: sdb.request( action, attrs, callback )

• action: (required) SimpleDB action
• attrs: (required) SimpleDB request attributes
• callback: (required) callback function accepting parameters callback(error, result, metadata)

Make a direct request to SimpleDB. You're on your own! Again, read Amazon SimpleDB Developer Guide Unlike the other functions above, the request function is not a SimpleDB action wrapper. Use it when the wrapper functions have painted themselves into a corner.

sdb.request("<action>", 
  {
    <attribute>:"<value>",
    ...
  },
  function( error, result, meta ){
    console.log("Gotta keep 'em separated: "+JSON.stringify(res)) 
  })

Where <action> is the SimpleDB action, such as GetItem, and <attribute>:"<value>" are the SimpleDB REST request attribute pairs.

client: sdb.client

The aws-lib client object. Use this to send raw requests. Go hardcore.

handle: sdb.handle( start, action, query, tryIndex, last, response, stop, callback, )

• start: Date object, start time of request
• action: string, name of SimpleDB action
• query: full SimpleDB query
• tryIndex: number of tries attempted, up to maxtry
• last: true if this is the last request that will be made
• response: result from SimpleDB
• stop: stop(true|false), function to stop retries in case of errors
• callback: action-specific callback, as provided by functions like getItem, putItem, etc.

Replace this with your own implementation to change the handling of SimpleDB responses. Most useful is to modify the response in some way and then call the original function. Also good for testing.

This example counts the number of requests made:

var resultcount = 0    

var orighandle = sdb.handle
sdb.handle = function(start,action,query,tryIndex,last,response,stop,callback){
  res.$ResultCount = resultcount++
  orighandle(start,action,query,tryIndex,last,response,stop,callback)
}

Options

The additional options that can be given to simpledb.SimpleDB are:

• secure: (optional, default=false), if true, use HTTPS
• consistent: (optional, default=true), if true, ask for consistent reads
• test: (optional, default=false), if true, don't actually send anything to SimpleDB
• host: (optional, default=sdb.amazon.com), SimpleDB host
• path: (optional, default=/), SimpleDB path
• version: optional), default=2009-04-15, SimpleDB API version
• maxtry: (optional, default=4), maximum number of retries when SimpleDB fails
• delaymin: (optional, default=0), minimum delay in milliseconds
• delayscale: (optional, default=100), delay multiplier, in milliseconds
• randomdelay: (optional, default=true), apply a random delay multiplier between 0 and 1
• expbase: (optional, default=4), exponent base, for the formula that calculates delay time when SimpleDB fails

Logging

You can provide a logger callback when you are creating the simpledb.SimpleDB object to get notifications of request processing events. A simple logger that prints to STDOUT is provided by simpledb.debuglogger:

var sdb = new simpledb.SimpleDB( {...}, simpledb.debuglogger )

The logger callback accepts the following arguments: logger( type, date, ... )

• type: string, one of create, request, handle, error, status
• date: a Date object
• ...: remaining arguments depend on type

For type=create, fired when the simpledb.SimpleDB object is created, the arguments are:

• opts: options object
• awsopts: aws-lib options

For type=request, fired just before a request is made to SimpleDB, the arguments are:

• start: Date object, start time of request
• action: string, name of SimpleDB action
• query: full SimpleDB query

For type=handle, fired after each response from SimpleDB, the arguments are:

• start: Date object, start time of request
• action: string, name of SimpleDB action
• query: full SimpleDB query
• tryIndex: number of tries attempted, up to maxtry
• last: true if this is the last request that will be made
• response: result from SimpleDB

For type=error, fired after any response with an error, the arguments are:

• start: Date object, start time of request
• action: string, name of SimpleDB action
• query: full SimpleDB query
• tryIndex: number of tries attempted, up to maxtry
• last: true if this is the last request that will be made
• retry: true if a retry will be attempted
• err: the error that occurred, an object like {Code:'...',Message:'...'}, where Code is the Amazon error code
• res: the result
• meta: the request meta data

For type=status, fired after each retry, the arguments are:

• done: true if request has finally succeeded
• tryIndex: count of attempts
• last: true if this was the last attempt
• delay: delay in milliseconds until this attempt
• err: any error that occurred

Testing

The unit tests use expresso

npm install expresso
npm install eyes

To configure your keys, edit the test/keys.js file. The tests are in test/simpledb.test.js

Amazon AWS SimpleDB

Here's some more information on SimpleDB:

Amazon AWS SimpleDB Developer Guide

Hacking around with real time charts

Chartaca Hit Chart