Jug: A Task-Based Parallelization Framework

What is Jug?

Jug allows you to write code that is broken up into tasks and run different tasks on different processors.

It currently has two backends. The first uses the filesystem to communicate between processes and works correctly over NFS, so you can coordinate processes on different machines. The second is based on redis so the processes only need the capability to connect to a common redis server.

Jug also takes care of saving all the intermediate results to the backend in a way that allows them to be retrieved later.

Examples

Short Example

Here is a one minute example. Save the following to a file called primes.py:

from time import sleep

from jug import TaskGenerator


@TaskGenerator
def is_prime(n):
    sleep(1.)
    for j in range(2, n - 1):
        if (n % j) == 0:
            return False
    return True


primes100 = [is_prime(n) for n in range(2, 101)]

Of course, this is only for didactical purposes, normally you would use a better method. Similarly, the sleep function is so that it does not run too fast.

Now type jug status primes.py to get:

Task name                     Waiting       Ready    Finished     Running
-------------------------------------------------------------------------
primes.is_prime                     0          99           0           0
.........................................................................
Total:                              0          99           0           0

This tells you that you have 99 tasks called primes.is_prime ready to run. So run jug execute primes.py &. You can even run multiple instances in the background (if you have multiple cores, for example). After starting 4 instances and waiting a few seconds, you can check the status again (with jug status primes.py):

Task name                     Waiting       Ready    Finished     Running
-------------------------------------------------------------------------
primes.is_prime                     0          63          32           4
.........................................................................
Total:                              0          63          32           4

Now you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they will all finish and you can inspect the results with jug shell primes.py. This will give you an ipython shell. The primes100 variable is available, but it is an ugly list of jug.Task objects. To get the actual value, you call the value function:

In [1]: primes100 = value(primes100)

In [2]: primes100[:10]
Out[2]: [True, True, False, True, False, True, False, False, False, True]

More Examples

There is a worked out example in the tutorial, and another, fully functioning in the examples/ directory.

How do I get Jug?

You can either get the git repository at

http://github.com/luispedro/jug

Or download the package from PyPI. You can use easy_instal jug or pip install jug if you’d like.

What do I need to run Jug?

It is a Python only package. Jug is continuously tested with Python 2.6 and up (including Python 3.3 and up).

How does it work?

Read the tutorial.

What’s the status of the project?

Since version 1.0, jug should be considered stable.

Indices and tables