PyPVM ===== Greg Baker gregb@ifost.org.au What is it? =========== PyPVM is a set of Python language bindings for writing programs that exploit the Parallel Virtual Machine (PVM). Python is a very readable interpretive object-oriented language which is freely available from http://www.python.org/. PVM is a software system that enables a collection of heterogeneous computers to be used as a coherent and flexible concurrent computational resource. The PVM web page is at http://www.epm.ornl.gov/pvm/pvm_home.html . If you're impatient, skip down to the installation section below, install it, and try it out. How is it related to Stefane Fermigier's pvmmodule.c ? ======================================================= Not greatly. I started with his pvmmodule.c and got it to work with PVM3.3.11 & Python1.5 (I've got patches if you want them). He's not working on pvmmodule.c any more. I suppose PyPVM is the successor to his package, but is not backward compatible. pvmmodule.c defined only about 11 functions, PyPVM has about 50 out of the total 86. Stefane was very helpful and enthusiastic and encouraged me to develop this version. It was at his suggestion that I started from scratch using swig. Then I took the output and cleaned it up. Licensing ========= I don't really mind what you do with this, use it in any way that you want to, as long as the copyright notice stays in place. Naturally, there's no warranty. What does it run on? ==================== I've only tested it on Linux, but pretty much any UNIX should work. PVM3.3.11 runs on WinNT/95, as does Python. I suppose that should mean that PyPVM should work there as well but I have no way of testing this. Can anyone help here? PVM is nicely heterogeneous, so a program running on one architecture has no problems talking to a program on another. What does it need? ================== PVM3.3.11, at least. I would expect PyPVM to work with 3.3.x where x>=11 . A little bit of tweaking should make 3.3.10 possible as well, but I haven't had a chance to try this. You will need to have python 1.5 _including source_. I don't think there are any particular dependencies on 1.5; 1.3 or 1.4 would seem likely to work, but again, I haven't tested it. A lot of people run RedHat Linux, these folks might have received a binary without realising they have source. Look for a file called python-1.5.src.rpm and install it with rpm --install . At the moment, I can't see any way of generating a dynamic module (PVM compiles as a static library. Anyone know how to force this otherwise?), so you'll need to build PyPVM into your python. I developed portions of this interface using swig1.1p5, but I used it as a base and have developed the remainder by hand. Installation ============ We need to know where the Modules directory is from your Python source. It could be /usr/local/src/Python-1.5/Modules for a "typical" setup, /home/yourname/src/Python-1.5/Modules if you build python for yourself or /usr/src/redhat/SOURCES/python-1.5 for a RedHat linux system. Whereever it is, we'll call it $PYTHON_MODS_DIR. cp PyPVM.c $PYTHON_MODS_DIR cp pvm3special.c $PYTHON_MODS_DIR echo "pvmc pvm3special.c pvm3_wrap.c -I/usr/local/pvm3/include -L/usr/local/pvm3/lib/LINUX -lpvm3 -lgpvm3" >> $PYTHON_MODS_DIR/Setup # You'll need to change /usr/local/pvm3 to whatever $PVM_ROOT is. cd $PYTHON_MODS_DIR ./makesetup cd .. make make install # You may need to do the "make install" step as root. One day I might get terribly intelligent and write a script that will automate that. What should I know? =================== pvm_setopts(PvmAutoErr,0) gets run in the module initialisation. You already get a complete traceback, and thrown exceptions anyway, so I don't know why you'd want to have pvm splurge messages out to stderr. If you want to have them, you'll need to explicitly do a pvm_setopts back again. As of the 0.1 release, however, there is no python interface into pvm_setopts. This will be fixed for 0.2 (which will probably be a little after Easter). I don't necessarily intend to keep backward compatibility with this release, so don't start writing that 10000 line number cruncher yet. In fact, I'm interested in feedback from people to tell me what they did and didn't like in this interface. What is and isn't implemented? ============================== There are 86 functions in the PVM3.3.11 library. As of release 0.1 of PyPVM: 46 are implemented directly 7 have their functionality implemented somehow else 32 are unimplemented 1 is undocumented (pvm_tickle) i.e this is 62% of the way to being a complete PVM3.3.11 API. Here's the general way I've worked. Constants get stored in a couple of dictionaries, with names somehow related to their C equivalents pvm.data holds parameters for pvm.initsend() and pvm.mkbuf() pvm.spawnOpts holds parameters for pvm.spawn() pvm.exceptions holds the complete set of PVM exceptions pvm.notify is badly named and holds constants for pvm_notify, which isn't implemented A function called pvm_yyy() is called yyy() in Python. The wrapper function is called pypvm_yyy(). Since I'm sure that you'll do an import pvm (as opposed to an from pvm import * ), this leads to the nice syntax pvm.yyy() If the C function returns an integer less than zero to indicate an error, that has been turned into an raised exception. The exception name is XXXException where the C #define was PvmXXX . These exceptions can be named through the pvm.exceptions dictionary. If that's all the function would have returned, then it returns None. Some functions return negative numbers on error, and a positive number on success. pvm_mytid is an example. In Python, this becomes mytid(), which can raise a variety of exceptions. If it returns anything, it _is_ a tid, and is guaranteed to be positive. Some C functions take an argument of a pointer to some storage space in which data will be returned. These all return negative numbers on failure. pvm_spawn is an example (it returns the task ids of the tasks it spawns). In these cases, we will raise exceptions if we had a negative return value; otherwise we will return a python list of the passed in data. PyPVM is smart, it allocates the storage space that it's going to need to do it. Some functions take or return structures; these become python tuples with components in the same order as the structure. Where a function has some logical default (e.g. pvm_initsend should default to PvmDataDefault), the python bindings will use that. I should really put together a complete list of them somewhere. This is divided up into: C : implement is a direct mapping from C, using the above rules F : functionality exists, but done differently somehow U : unimplemented, sorry C language name Python language name Nature --------------------------------------------------- pvm_archcode archcode C pvm_barrier barrier C pvm_bcast bcast C pvm_bufinfo bufinfo C pvm_config configuration F number_of_architectures U pvm_delete delete C pvm_delhosts U pvm_exit exit C pvm_export export C pvm_freebuf freebuf C pvm_gather U pvm_getfds U pvm_getinst getinst C pvm_getmwid getmwid C pvm_getopt U pvm_getrbuf getrbuf C pvm_getsbuf getsbuf C pvm_gettid gettid C pvm_gsize gsize C pvm_halt halt C pvm_hostsync U pvm_initsend initsend C pvm_insert insert C pvm_joingroup joingroup C pvm_kill kill C pvm_lookup lookup C pvm_lvgroup lvgroup C pvm_mcast U pvm_mkbuf mkbuf C pvm_mstat mstat C pvm_mytid mytid C pvm_notify U pvm_nrecv nrecv C pvm_packf pack F pvm_parent parent C pvm_perror perror C pvm_pkbyte U pvm_pkcplx U pvm_pkdcplx pack F pvm_pkdouble pack F pvm_pkfloat U pvm_pkint U pvm_pklong pack F pvm_pkshort U pvm_pkstr pack F pvm_pkuint U pvm_pkulong U pvm_pkushort U pvm_precv U pvm_probe probe C pvm_psend U pvm_pstat pstat C pvm_recv recv C pvm_recvf U pvm_reduce U pvm_reg_hoster U pvm_reg_rm U pvm_reg_tasker register_tasker C pvm_scatter U pvm_send send C pvm_sendsig sendsig C pvm_setmwid setmwid C pvm_setopt U pvm_setrbuf setrbuf C pvm_setsbuf setsbuf C pvm_spawn spawn C pvm_start_pvmd U pvm_tasks tasks F pvm_tickle ? pvm_tidtohost tidtohost C pvm_trecv U pvm_unexport unexport C pvm_unpackf U pvm_upkbyte U pvm_upkcplx U pvm_upkdcplx U pvm_upkdouble upkdouble C pvm_upkfloat U pvm_upkint U pvm_upklong upklong C pvm_upkshort U pvm_upkstr upkstr C pvm_upkuint C pvm_upkulong C pvm_upkushort C pvm_version version C How do I use it? ================ To be written. If you're using this kind of early release, then I'm assuming you already know how to use PVM (for either Fortran or C). If you can do either of these, you should find the Python version pretty easy to pick up. Sorry about this, I'll start working on a proper introduction after I've implemented the full API. Thanks, credits, etc. ===================== Thanks to CSIRO Radiophysics, my long-ago employer who gave me a network of machines and let me learn PVM. Thanks also to Brett Hardman, my boss at Apt Consulting, who didn't complain when I burst forth with a couple of days of all-consuming interest in getting this out. Greg Baker, 9th April, 1998. (Somewhere between Brisbane and Sydney).