Home

If you would like, you can start with a simple example:

• WordCount

How-to documentations:

• Write your mappers and reducers
• Read and write data
• Launch your mapreducer
• Write Matlab mappers

What mincepie is for

Mincepie is a lightweight mapreduce engine purely written in Python for some of my research code. It is meant for simple distribution of jobs that:

• can be easily separated to map calls, such as extracting one feature for each image,
• does not involve a lot of communication, i.e. the keys and values being transmitted are not huge,
• the main computation time is spent inside the map() or reduce() function, not communication.

Also, the simplified system

• holds everything in memory - the input, the keys, and the values in every stage of the mapreduce run.
• does not handle server errors. If the server is down, you have to restart mapreduce.
• partially tolerates client failure. If a client is down, its last map or reduce operation will simply be re-run on another client.
• provides a simple support to run Matlab scripts as a subprocess call.

I have used it to perform feature extractions on ImageNet and it works pretty well for our research use. But it may or may not fit your use case.