River is a Python-based framework for rapid prototyping of
reliable parallel run-time systems. The current quest for
new parallel programming models is hampered, in part, by
the time and complexity required to develop dynamic
run-time support and network communication. Integrating
mechanisms for fault tolerance, such as checkpointing or
mirroring, further complicates an implementation. The
simplicity of the River core combined with Python's dynamic
typing and concise notation makes it possible to go from a
design idea to a working implementation in a matter of days
or even hours. With the ability to test and throw away
several implementations, River allows researchers to
explore a large design space. In addition, the River core
and new extensions can be used directly to develop parallel
applications in Python.
The River core interface is based on a few fundamental
concepts that enables the execution of code on multiple
virtual machines and provides a flexible mechanism for
communication. These concepts are supported by the River
run-time system, which manages automatic discovery,
connection management, and naming. River can be used
directly by an application programmer to implement parallel
programs or it can be used as a framework for implementing
programming models.
A River program consists of one or more virtual resources
(VRs) that execute on River virtual machines (VMs). VMs can
exist locally or remotely and VRs are named using UUIDs. A
VM is simply a Python interpreter running the River
run-time system. An initiating VR can discover existing VMs
and deploy itself or other VRs onto selected VMs. Once
running, VRs communicate with each other using a mechanism
called super flexible messaging (SFM) for sending and
receiving dynamically-typed packets.
Virtual Resources
A virtual resource (VR) is a process-like abstraction for
Python programs. A VR is specified by subclassing
VirtualResource and each VR has its own thread of
execution. Therefore, a VR encapsulates state using the
standard Python class type. In addition, each VR is named
using a UUID. A message queue is associated with each VR
and is used to support our super flexible messaging
mechanism. The VirtualResource base class provides methods
for VM discovery, VR creation, and message passing.
Super Flexible Messaging
In River we introduce a novel message passing mechanism
called super flexible messaging (SFM). Our mechanism
leverages Python's dynamic typing and named arguments to
send structured data as (attribute,value) pairs between
VRs. Our mechanism allows for an arbitrary number of
attributes and the values can be any Python data type that
can be serialized using the pickle module.