Skip to main content
Warning: You are using the test version of PyPI. This is a pre-production deployment of Warehouse. Changes made here affect the production instance of TestPyPI (testpypi.python.org).
Help us improve Python packaging - Donate today!

Distributed and Parallel Computing Framework with/for Python.

Project Description

dispy is a rather comprehensive, yet easy to use framework for creating and using compute clusters to execute computations in parallel across multiple processors in a single machine (SMP), among many machines in a cluster, grid or cloud. dispy is well suited for data parallel (SIMD) paradigm where a computation is evaluated with different (large) datasets independently with no communication among computation tasks (except for computation tasks sending intermediate results to the client). If communication/cooperation among tasks is needed, asyncoro framework could be used.

dispy works with Python versions 2.7+ and 3.1+. It has been tested with Linux, OS X and Windows; it may work on other platforms too.

Features

  • dispy is implemented with asyncoro, an independent framework for asynchronous, concurrent, distributed, network programming with coroutines (without threads). asyncoro uses non-blocking sockets with I/O notification mechanisms epoll, kqueue and poll, and Windows I/O Completion Ports (IOCP) for high performance and scalability, so dispy works efficiently with a single node or large cluster(s) of nodes. asyncoro itself has support for distributed/parallel computing, including transferring computations, files etc., and message passing (for communicating with client and other computation tasks), although it doesn’t include job scheduling.

  • Computations (Python functions or standalone programs) and their dependencies (files, Python functions, classes, modules) are distributed automatically.

  • Computation nodes can be anywhere on the network (local or remote). For security, either simple hash based authentication or SSL encryption can be used.

  • After each execution is finished, the results of execution, output, errors and exception trace are made available for further processing.

  • Nodes may become available dynamically: dispy will schedule jobs whenever a node is available and computations can use that node.

  • If callback function is provided, dispy executes that function when a job is finished; this can be used for processing job results as they become available.

  • Client-side and server-side fault recovery are supported:

    If user program (client) terminates unexpectedly (e.g., due to uncaught exception), the nodes continue to execute scheduled jobs. If client-side fault recover option is used when creating a cluster, the results of the scheduled (but unfinished at the time of crash) jobs for that cluster can be retrieved later.

    If a computation is marked reentrant (with ‘reentrant=True’ option) when a cluster is created and a node (server) executing jobs for that computation fails, dispy automatically resubmits those jobs to other available nodes.

  • dispy can be used in a single process to use all the nodes exclusively (with JobCluster - simpler to use) or in multiple processes simultaneously sharing the nodes (with SharedJobCluster and dispyscheduler program).

Installation

To install dispy for Python 2.7+, run:

pip install dispy

or to install dispy for Python 3.1+, run:

pip3 install dispy

Dependencies

dispy is implemented with asyncoro for concurrent, asynchronous network programming. Part of asyncoro package needed for dispy is included in dispy package itself. However, other modules in asyncoro support asynchronous pipes, distributed programming/computing, so users interested in those features may want to install asyncoro as well.

Under Windows efficient polling notifier I/O Completion Ports (IOCP) is supported only if pywin32 is installed; otherwise, inefficient select notifier is used.

Authors

  • Giridhar Pemmasani
Release History

Release History

This version
History Node

3.22-4

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
dispy-3.22-4.tar.gz (249.1 kB) Copy SHA256 Checksum SHA256 Source Apr 15, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting