Accessing pyre components from C
From DANSE
GOAL: To be able to access (i.e. run) pyre components when 'main' lives in C.
Pyre is built on python, thus to drive pyre from C is not the intended way to use a pyre component or run a pyre application. The following, however, is a simple solution for using pyre within an environment that suffers from these restrictions.
There are (at least) two possibilities for configuration:
- normal python (and pyre) installation is allowed
- python (and thus pyre) is not installed as an executable
We will first explain both variants for the simple case of python (not yet involving pyre whatsoever).
| Table of contents |
Requirements
The only hard requirements here are:
- a C compiler
- a python source distribution
- a pyre source distribution (for the pyre-specific stuff)
Additional requirements for the following examples are:
- the python package Numeric
- the python package Matplotlib
- freetype (required by Matplotlib)
- libpng (required by Matplotlib)
- zlib (required by Matplotlib)
I assume the C compiler, the three libraries that Matplotlib requires, and any libraries that python requires are already installed. Everything else, may or may not be installed, depending on each example below.
We will use the same directory for executing all of the test code below:
DEMO_DIR = $(HOME)/demo
We will also assume that the python source code is found at:
PYTHON_SRCDIR = $(HOME)/src/Python-$(PYTHON_VERSION)
In my case, PYTHON_VERSION = 2.3; however, if you have python > 2.3 things should also work.
Another thing to right now is to add '.' to the beginning of your PYTHONPATH. This is not necessary when you are executing python with the python executable, however we are running code from C in most cases. So, in your $HOME/.bash_profile (or at the shell prompt) make sure to do this now.
export PYTHONPATH=.:$PYTHONPATH
We will also use a few environment variables in this tutorial, so that we can have abstraction in our files. Please set these as you see fit to correspond to your python and pyre installations.
- PYTHON_DIR -- default is often /usr/local
- PYTHON_INCDIR -- generally is $(PYTHON_DIR)/include/python$(PYTHON_VERSION)
- PYTHON_LIBDIR -- generally is $(PYTHON_DIR)/lib/python$(PYTHON_VERSION)
Calling python from python
For the first example, we will show a simple example of a standard python use of the Numeric package.
Requirements:
- the python executable is installed in $(PYTHON_DIR)/bin; and $(PYTHON_DIR)/bin is included on the system PATH.
- Numeric is installed under the python 'site-packages' directory $(PYTHON_LIBDIR)/site-packages
These requirements are satisfied by the standard install for both python and Numeric, whence the installed executables are correctly added to your system paths.
We will create a very minimalistic python script, demo-numeric.py, to demonstrate standard invocation of Numeric.
#!/usr/bin/env python #========================= # demo-numeric.py #========================= # import Numeric # import the Numeric package print Numeric.pi # print the value of 'pi' x = Numeric.array([1,2,3]) # set x as a 1-D NumArray print x # print the value of 'x'
When we execute demo-numeric.py from the commandline, the program is run by the installed python executable in $(PYTHON_DIR)/bin.
>$ python demo-numeric.py 3.14159265359 [1 2 3]
Calling python from C, acting on python files
Now we can take a simple step forward, by calling python from C. This has the same requirements as the above section, and since our script is so small, we will just rewrite everything in C.
// -*- C -*-
//
/*************************
* demo-numeric.c
*************************
*/
#include <Python.h> // include Python/C API
main() {
printf("demo-numeric\n");
Py_Initialize(); // allow calls to libpython
PyRun_SimpleString("import Numeric"); // import the Numeric package
PyRun_SimpleString("print Numeric.pi"); // print the value of 'pi'
PyRun_SimpleString("x = Numeric.array([1,2,3])"); // set x as a 1-D NumArray
PyRun_SimpleString("print x"); // print the value of 'x'
}
This code is fairly simple, in that it does not attempt to pass variables between python and C. However, doing so does not change the implementation of the example's code, so for this tutorial, we will keep things simple.
There is a bit more work to do before we can have python driven by C code. We must write a Makefile to help build and link our C code.
# Makefile for embedded Python.
# The C compiler you are using
CC= gcc
# Top of the python build and source tree
# (Location of python static library and initial python executable)
blddir= $(PYTHON_SRCDIR)
srcdir= $(PYTHON_SRCDIR)
# Python version
VERSION= $(PYTHON_VERSION)
# Compiler flags to provide initial link to python
OPT= -g
INCLUDES= -I$(srcdir)/Include -I$(blddir)
CFLAGS= $(OPT)
CPPFLAGS= $(INCLUDES)
# The Python static library
LIBPYTHON= $(blddir)/libpython$(VERSION).a
# libraries required by your python build
# edit these to match the required libs in your $(blddir)/Makefile
LIBS= -lnsl -ldl -lreadline -ltermcap -lieee -lpthread -lutil
LDFLAGS= -Xlinker -export-dynamic
SYSLIBS= -lm
MODLIBS= -ltk8.3 -ltcl8.3 -L/usr/lib -L/usr/X11R6/lib -lX11 -lgdbm
ALLLIBS= $(LIBPYTHON) $(MODLIBS) $(LIBS) $(SYSLIBS)
#######################################################################
# Executables to build
MODULES= demo-numeric demo-numzip run_orig run_pyre
# Build all the applications
all: $(MODULES)
demo%: demo%.o
$(CC) $(LDFLAGS) demo$*.o $(ALLLIBS) -o demo$*
run%: run%.o
$(CC) $(LDFLAGS) run$*.o $(ALLLIBS) -o run$*
# Administrative targets (cleanup)
clean:
-rm -f *.o core
restore: clean
-rm -f *~ @* '#'* $(MODULES) *.pyc
Adapting this makefile to your installation of python should happen above the divider line in the file -- and in particular on the lines that describe the LIBS that python needed to be built. If those I've provided don't work, then it will take some digging in the Makefile that came with your python source.
We can now compile and link demo-numeric.c
>$ make demo-numeric
This produces an executable 'demo-numeric', which we can run with
>$ ./demo-numeric demo-numeric 3.14159265359 [1 2 3]
As we can see, the C code, produces the same output from python as the original python code. Note that demo-numeric.c makes a call to the python static library instead of the python executable. It is only the "import Numeric" statement that uses python code in the 'standard' form. If we didn't call the installed Numeric package in our example, we would not need to have installed python on our system.
Calling python from C, acting on library and zip files
So let's take that next step of removing the installed Numeric library from the process. Since python can import packages from not only .py, .pyc, and .pyo files, but also .so, .a, and .zip files, we will take advantage of that functionallity. To completely remove intalled python from our execution flow, we need to either build the Numeric package into a shared library or a zip file. Actually, we will chose to do a little of both...
First off, Numeric uses a bit of rare python magic, so we have to deal with that. For python to recognize a directory as a package, it must include a __init__.py file. This file need not have any content in it, but it's mere presence allows a directory to be imported into python. There are some exceptions to this rule, however. The $PYTHON_LIBDIR/site-packages directory is a 'special' directory for python, and provides such an exception. If python detects a .pth file within the 'site-packages' directory, then the path that is held within the .pth file is added to the system's PATH -- if the directory and the .pth file have the same name. Thus $PYTHON_LIBDIR/site-packages/Numeric.pth points to "Numeric" within the 'site-packages' directory, and invoking python temporarily appends $PYTHON_LIBDIR/site-packages/Numeric to your system's PATH. We won't bother with this magic, so we will first create a simple __init__.py file and for Numeric place it in $PYTHON_LIBDIR/site-packages/Numeric
# this __init__.py file replaces Numeric.pth
Simple file, eh? We now move this file into $PYTHON_LIBDIR/site-packages/Numeric
>$ mv __init__.py $PYTHON_LIBDIR/site-packages/Numeric
Now we can proceed on with the easiest route, which is to go to $(PYTHON_LIBDIR)/site-packages/Numeric and then make a zip archive of the whole directory.
>$ cd $PYTHON_LIBDIR/site-packages/Numeric >$ zip -r Numeric *
This pulls the entire installed distribution of Numeric into a single zipfile. Note that we could have excluded any .pyc or .pyo files from our new zipfile "Numeric.zip", but having the precompiled python makes our code run faster so we left them in. This speed difference is even more noticable when python is importing from a .zip file. The Numeric package includes a few .so files for things like its interaction with lapack, and these libraries are imported by select files (like Numeric/Numeric.py) within the Numeric distribution. Since python cannot import a static/shared library from within a zipfile, we must do something extra with these libraries. We will create a new directory, 'lib', and dump a copy of all of these libraries into it.
>$ mkdir $DEMO_DIR/lib >$ cp *.so */*.so $DEMO_DIR/lib >$ mv Numeric.zip $DEMO_DIR >$ cd $DEMO_DIR
We want to have python use Numeric.zip and the shared libraries in the 'lib' directory when it runs from our C code, so we have C tell python to add these two items to our system PATH, as shown in the code below.
// -*- C -*-
//
/*************************
* demo-numzip.c
*************************
*/
#include <Python.h> // include Python/C API
main() {
printf("demo-numzip\n");
Py_Initialize(); // allow calls to libpython
PyRun_SimpleString("import sys"); // import sys package from libpython
PyRun_SimpleString("sys.path.insert(0, 'Numeric.zip')"); // add Numeric.zip to head of PATH
PyRun_SimpleString("sys.path.insert(0, 'lib')"); // add ./lib to head of PATH
PyRun_SimpleString("import Numeric"); // import the Numeric package (from zip)
PyRun_SimpleString("print Numeric.pi"); // print the value of 'pi'
PyRun_SimpleString("x = Numeric.array([1,2,3])"); // set x as a 1-D NumArray
PyRun_SimpleString("print x"); // print the value of 'x'
}
Notice right now if we were to start the python interpreter, and try to import Numeric, that we would still succeed because Numeric is still installed.
>$ python >>> import Numeric >>>
Even though our C code will use the .zip version of Numeric, we can for the purpose of demonstration, 'uninstall' the version on Numeric within $PYTHON_LIBDIR/site-packages/Numeric. If we just rename $PYTHON_LIBDIR/site-packages/Numeric.pth to $PYTHON_LIBDIR/site-packages/Numeric.pthX, then python will not be able to find this version of Numeric.
>$ mv $PYTHON_LIBDIR/site-packages/Numeric.pth $PYTHON_LIBDIR/site-packages/Numeric.pthX >$ python >>> import Numeric Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named Numeric
We can use the same Makefile as in the previous section to compile the new C code. We then will also test it.
>$ make demo-numzip >$ ./demo-numzip demo-numzip 3.14159265359 [1 2 3]
Thus, we have now compiled and executed a script that drives python from C -- and can run without any .py files explicitly being installed on our system.
Calling pyre from python
Now that we've seen how to access python from C, and do so without having any python files explicitly installed, we should know enough to repeat these same steps for a more complex package such as pyre. Pyre is not only a python package, but it an entire component-based framework. There are two main features of pyre that we will concern ourselves with here:
- pyre as a container and lifecycle manager
- pyre as a component interaction manager
Since this is well seperated within pyre, we can parallel this seperation in our examples. We can have two ways we can try to invoke the code within pyre components: with and without driving the code from a pyre script.
Another view of these two options is that if we drive pyre components from a simple python script, then we only use pyre as a lifecycle manager for our components. If we actually control our components from within a pyre script (i.e. inherit from pyre.applications.Script.Script), then we are going the "full monty" for pyre, and using it as a component interaction manager as well.
- FIXME: only code is provided for the below sections; text needs fleshing out.
components interact through python
Phonon DOS data file 'pfrout3.txt' in 3-Column ascii format.
#!/usr/bin/env python
#=========================
# run_orig.py
#=========================
#
def run():
from pyIO import Reader
rt = Reader()
rt.read('pfrout3.txt',intoRows=False,dialect='ascii')
xlabel = rt.header()[0]
ylabel = rt.header()[1]
xdata = rt.data()[0]
ydata = rt.data()[1]
edata = rt.data()[2]
from graphics.Matplotlib import pylab
mp = pylab()
mp.errorbar(xdata,ydata,edata,ecolor='r')
mp.xlabel(xlabel)
mp.ylabel(ylabel)
mp.show()
return
if __name__ == '__main__':
run()
We can execute with python...
>$ python run_orig.py data read by column from 'pfrout3.txt'
and the plot shown above is produced.
components interact through pyre
#!/usr/bin/env python
#=========================
# run_pyre.py
#=========================
#
from pyre.applications.Script import Script
import run_orig as orig
class runPyre(Script):
def main(self, *args, **kwds):
orig.run()
return
def __init__(self):
Script.__init__(self, 'run_pyre')
return
def _configure(self):
Script._configure(self)
return
# main
if __name__ == '__main__':
app = runPyre()
app.run()
We can execute with python...
>$ python run_pyre.py data read by column from 'pfrout3.txt'
and the plot shown above is again produced.
Calling pyre from C, acting on python files
We could rewrite the core of the python code as in our previous examples...
// -*- C -*-
//
/*************************
* run_orig.c
*************************
*/
#include <Python.h>
main() {
printf("run_orig\n");
Py_Initialize();
PyRun_SimpleString("from pyIO import Reader");
PyRun_SimpleString("rt = Reader()");
PyRun_SimpleString("rt.read('pfrout3.txt',intoRows=False,dialect='ascii')");
PyRun_SimpleString("xlabel = rt.header()[0]");
PyRun_SimpleString("ylabel = rt.header()[1]");
PyRun_SimpleString("xdata = rt.data()[0]");
PyRun_SimpleString("ydata = rt.data()[1]");
PyRun_SimpleString("edata = rt.data()[2]");
PyRun_SimpleString("from graphics.Matplotlib import pylab");
PyRun_SimpleString("mp = pylab()");
PyRun_SimpleString("mp.errorbar(xdata,ydata,edata,ecolor='r')");
PyRun_SimpleString("mp.xlabel(xlabel)");
PyRun_SimpleString("mp.ylabel(ylabel)");
PyRun_SimpleString("mp.show()");
}
However, if we were smart, we would just reuse 'run_orig.py' (and include it in our .zip file if necessary)... we will follow this easier path in the below examples.
components interact through python
// -*- C -*-
//
/*************************
* run_orig.c
*************************
*/
#include <Python.h>
main() {
printf("run_import\n");
Py_Initialize();
PyRun_SimpleString("import run_orig");
PyRun_SimpleString("run_orig.run()");
}
We can use the same Makefile as above.
>$ make run_orig >$ ./run_orig run_orig data read by column from 'pfrout3.txt'
and the plot is produced as previously.
components interact through pyre
We have to add two new lines to get the pyre script to run. Since pyre scripts will always process the commandline args, we need to supply them to python -- even if they are "None". These two new lines are useful, in any case, and could have been added to all examples in this tutorial. By passing argv and argc to python, this allows pyre to maintain the full functionality of manipulating the inventory, as done directly from python.
// -*- C -*-
//
/*************************
* run_pyre.c
*************************
*/
#include <Python.h>
main(int argc, char **argv) {
printf("run_pyre\n");
Py_SetProgramName(argv[0]); //pass name of program to python
Py_Initialize();
PySys_SetArgv(argc, argv); //pass argc & argv to python
PyRun_SimpleString("from run_pyre import runPyre");
PyRun_SimpleString("app = runPyre()");
PyRun_SimpleString("app.run()");
}
We can use the same Makefile as above.
>$ make run_pyre >$ ./run_pyre run_pyre data read by column from 'pfrout3.txt'
and the plot is produced as previously.
Calling pyre from C, acting on library and zip files
- TODO: John, this is your section. Use the code from the rest of the tutorial.
components interact through python
components interact through pyre
Appending to libpython.a
- TODO: John, you can also take the lead here. Feel free to rename this section.
