Simple Django cache backend for Amazon ElastiCache (memcached based). It uses pylibmc and sets up a connection to each node in the cluster using auto discovery.
- pylibmc
- Django 1.5+.
It was written and tested on Python 2.7 and 3.4.
Get it from pypi:
pip install django-elasticache
or github:
pip install -e git://github.com/gusdan/django-elasticache.git#egg=django-elasticache
Your cache backend should look something like this:
CACHES = { 'default': { 'BACKEND': 'django_elasticache.memcached.ElastiCache', 'LOCATION': 'cache-c.draaaf.cfg.use1.cache.amazonaws.com:11211', 'OPTIONS': { 'IGNORE_CLUSTER_ERRORS': [True,False], 'behaviors': { # pylibmc behaviors, passed to underlying client 'ketama': True, 'receive_timeout': 50, # milliseconds, socket timeout for memcached reads 'send_timeout': 50, # milliseconds, socket timeout for memcached writes }, }, 'DISCOVERY_TIMEOUT': 0.1, # seconds, Elasticache discovery connection timeout 'TIMEOUT': 600, # seconds, default memcached key expiration time if not specified in set() } }
By the first call to cache it connects to cluster (using LOCATION
param),
gets list of all nodes and setup pylibmc client using full
list of nodes. As result your cache will work with all nodes in cluster and
automatically detect new nodes in cluster. List of nodes are stored in class-level
cached, so any changes in cluster take affect only after restart of working process.
But if you're using gunicorn or mod_wsgi you usually have max_request settings which
restart process after some count of processed requests, so auto discovery will work
fine.
The IGNORE_CLUSTER_ERRORS
option is useful when LOCATION
doesn't have support
for config get cluster
. When set to True
, and config get cluster
fails,
it returns a list of a single node with the same endpoint supplied to LOCATION
.
DISCOVERY_TIMEOUT controls how long to wait for response to any command during the discovery sequence, including initial connection and any subsequent commands. This is passed to the underlying socket used in the Telnet connection for communicating with the ElastiCache cluster. Measured in seconds.
Django-elasticache does not change default pylibmc params. The user should set performance-related params in the cache configuration.
ElastiCache provides memcached interface so there are three solution of using it:
In this case your application will randomly connect to nodes in cluster and cache will be used with not optimal way. At some moment you will be connected to first node and set item. Minute later you will be connected to another node and will not able to get this item.
CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', 'LOCATION': 'cache.gasdbp.cfg.use1.cache.amazonaws.com:11211', } }
It will work fine, memcache client will separate items between all nodes and will balance loading on client side. You will have problems only after adding new nodes or delete old nodes. In this case you should add new nodes manually and don't forget update your app after all changes on AWS.
CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache', 'LOCATION': [ 'cache.gqasdbp.0001.use1.cache.amazonaws.com:11211', 'cache.gqasdbp.0002.use1.cache.amazonaws.com:11211', ] } }
It will connect to cluster and retrieve ip addresses of all nodes and configure memcached to use all nodes.
CACHES = { 'default': { 'BACKEND': 'django_elasticache.memcached.ElastiCache', 'LOCATION': 'cache-c.draaaf.cfg.use1.cache.amazonaws.com:11211', } }
Difference between setup with nodes list (django-elasticache) and connection to only one configuration Endpoint (using dns routing) you can see on this graph:
Run the tests like this:
nosetests