grain.multiprocessing module

grain.multiprocessing module#

APIs to work with multiprocessing.

List of Members#

class grain.multiprocessing.MultiprocessingOptions(num_workers=0, per_worker_buffer_size=1, enable_profiling=False)#

Options for using Python multiprocessing.

Parameters:

  • num_workers (int)

  • per_worker_buffer_size (int)

  • enable_profiling (bool)

num_workers#

Number of Python worker processes. More processes can speed up the pipeline if it’s compute bound and bottlenecked on the CPython’s GIL. The default value of 0 means no Python multiprocessing, and as a result all data loading and transformation will run in the main Python process.

Type:

int

per_worker_buffer_size#

Size of the buffer for preprocessed elements that each worker maintains. These are elements after all transformations. If your transformations include batching this means a single element is a batch.

Type:

int

enable_profiling#

If True, profiling info is logged. This is only available when num_workers >= 1.

Type:

bool

class grain.multiprocessing.SharedMemoryArray(shape, dtype=<class 'float'>)#

A NumPy array subclass which is backed by shared memory.

This should be used in combination with Python multiprocessing. Compared with the normal NumPy ndarray it avoids expensive serialization when sending the array to another Python process (on the same machine). It also doesn’t require a copy on the receiving side.

The last processes using the array must call unlink_on_del()! Otherwise the memory will not be freed.

Parameters:

  • shape (Iterable[int])

  • dtype (npt.DTypeLike)

classmethod enable_async_del(num_threads=1, max_outstanding_requests=50)#

Enables asynchronous deletion of shared memory arrays.

Parameters:

  • num_threads (int) – The number of threads to use for deletion.

  • max_outstanding_requests (int) – The maximum number of outstanding requests to close/unlink shared memory. A larger value may make the __del__ method faster, but it may also lead to OOM errors or hitting file descriptor limits, since max_outstanding_requests shared memory objects and their associated file descriptors may be buffered before deletion.

Return type:

None

Mark this object responsible for unlinking the shared memory.

Return type:

None

Contents