Types in JugΒΆ
Any type that can be pickle()d
can be used with jug tasks. However, it
might sometimes be wiser to break up your tasks in ways that minimise the
communication necessary. For example, consider the following image processing
code:
from glob import glob
from mahotas import imread
def process(img):
# complex image processing
files = glob('inputs/*.png')
imgs = [Task(imread, f) for f in files]
props = [Task(process, img) for img in imgs]
This will work just fine, but it will save too much intermediate data. Consider rewriting this to:
from glob import glob
def process(f):
from mahotas import imread
img = imread(f)
# complex image processing
files = glob('inputs/*.png')
props = [Task(process, f) for f in files]
I have also moved the import of mahotas.imread
to inside the process
function. This is another micro-optimisation: it makes jug status
and
friends just that little bit faster (as they do not need to perform this import
to do their jobs).