Writing C extensions for Python
From DANSE
| Table of contents |
First, some sources of information written by vastly more competent authorities: the extension and embedding manual, and the Python-C API reference manual. These documents are also available at the Python website: http://python.org, or just look in the 'ext' and 'api' subdirectories of your Python distribution's Doc directory.
This how-to was written to supplement those documents, first by giving an example of dynamically allocating C++ objects and keeping track of them, and second by introducing some of the API functions for working with aggregate types like tuples and lists, for which the built in Python-C conversion tools won't work.
Note: In the following, C++ is used to mean both C and C++. The Python-C API is written in C, but this introduces few complications.
Introduction
Why write C++ extensions to Python? To reuse existing code, and to gain better performance. A great deal of software has already been written in C and C++ (not to mention FORTRAN), and, at least at the present, nothing beats compiled languages for performance in the numerically intensive codes that DANSE supports. Extending Python allows us to turn all that code into building blocks for solutions. Making libraries of extensions packages building blocks into toolkits that users can adapt to solve their problems.
Of course you can do all that without Python. So why use Python? Python has an easier learning curve than C++, making it available to a wider set of users. Moreover, once a library of extensions is available on a platform, users can call out to that library, mixing and matching components and testing combinations, without the overhead of compiling and linking. (This immediacy should not be underestimated.) And with all its standard library packages, Python can be manipulated to do some pretty amazing things that would be more difficult to realize in C++, like parsing XML documents, setting up computations, etc.
Writing C++ extensions for Python can be learned in an afternoon, especially if there are some examples to follow. The idea is this: you have a C++ class or function, and you'd like to make it callable from the Python interpreter. You'll need to
- write some wrapper code in C++, and then
- compile it into a C++ library that the Python interpreter can dynamically load and use. You then typically
- write a Python module that loads this library, and provides checks on what's being sent to the C++ library. The end user will interact with this Python module, instead of having to load up the C++ library themselves. This makes life pretty plush for those users who don't want to be bothered with the details.
That's it.
More details
Here are those three easy steps again, in slightly more detail:
Write the bindings
There are three essential components to Python-C bindings:
- C/C++ wrapper function(s): One for each function you want callable from Python. The wrapper typically calls a function or a class method, or it creates/destroys a heap object. Once you learn how to write one wrapper, you know how to do it, because all wrappers do the same three things.
- The method table: One entry in the table for each function you want callable from Python. The method table is a table of contents: it tells the Python interpretter which C++ functions it can call from the module and where to find them. There are several ways to structure the method table (of course we have our preferences).
- init function: One per module. This is how the interpreter gets started with a module.
These three components are implemented with generous aid from the Python-C extension API.
Compile/link the bindings
This is slightly platform dependent (Michael Aivazis's system for processing source code removes this platform dependence for UNIX flavors, including cygwin; Windows is in the works). The idea is to compile into a shared object library (unix) or a dynamically linked library (the much beloved Windows dll). Here are pages with details of compiling and linking the Numbers example for Windows and Linux.
Call it from Python
One typically writes a Python module that acts as a layer between the user and the C++ library. By doing things like providing Python classes that mirror the C++ classes, one can make the experience quite similar. Or dissimilar. The choice is yours.
Advanced topics
Exceptions
The API gives a painless way to send exceptions to the Python interpreter. This is an important way to communicate error conditions to a Python user. Here are details on raising exceptions.
The package template and Python extension modules
The "package" shell script lays out a working Python extension module. ... more soon
C++ template functions
Working with template functions requires a little more work. Here's how to do it.
