pyinterface.md

Openage C++ <-> Python interface

Openage consists of python modules, which contain the program entry point, and the library libopenage.so, which contains all C++ code.

Cython is used for glue code.

Cython crash course

Cython modules are written in .pyx files, and roughly equivalent to .py files. In addition to regular Python syntax, .pyx files allow you to define typed functions and objects:

cdef int square(int x):
    return x * x

cdef cppclass Rectangle:
    int h, w

    int size():
        return this.h * this.w

def foo():
    cdef Rectangle r
    r.h = 5
    r.w = 6
    return square(r.size())

.pyx files are translated to .cpp by Cython ("cythonized") as part of the openage build process; syntax errors are shown in this step.

Each .cpp file is then compiled to a Python extension module, which may be used from everywhere.

- `def` functions and `cdef class` classes can be used from Python.
- `cdef` functions are suitable for storage in a `C` function pointer.

$ cython --cplus -3 test.pyx
$ g++ -shared -fPIC -I/usr/include/python3.4m test.cpp -o test.so
$ python3
>>> import test
>>> test.foo()
900

Cython can use any regular C++ function or type; for that purpose, they need to be declared in .pxd files (which are analogous to C++ .h files):

cdef extern from "<stdlib.h>":
    int atoi(const char *s)

To use this function declared in foo.pxd,

from foo cimport atoi as c_atoi

def atoi(s):
    return c_atoi(s)

Openage has a helper, pxdgen, which auto-generates .pxd files for .h files in the libopenage/ subdirectory, from pxd: annotations in these files, as part of the build system.

The pxd annotations are really simple; just have a look at some of the headers. You'll find the generated .pxd files next to the C++ header files.

To cimport a class Foo that was pxd-annotated in util/foo.h, type

from libopenage.util.foo cimport Foo

Cython ships .pxd files for most C, C++ and CPython functions:

from libc.math cimport sin
from libcpp.vector cimport vector

cdef vector[float] vector_sin(vector[float] args):
    return [sin(arg) for arg in args]

print(vector_sin(range(10)))

From time to time, it may be useful to have a look at the generated .cpp file, especially if you require more exotic functionality or something doesn't work out as you expected. For that purpose, .html files are generated next to the .cpp files.

Calling C++ functions from Python

To make a C++ function available for calling from Cython code, annotate the header file with pxd comments:

libopenage/example.cpp

namespace openage {

int foo(int arg0, std::string arg1) {
        return 5;
}

} // openage

libopenage/example.h

// pxd: from libcpp.string cimport string
#include <string>

namespace openage {

/**
 * The famous foo function. Warning: might bar occasionally.
 *
 * pxd:
 *
 * int foo(int arg0, string arg1) except +
 */
int foo(int arg0, std::string arg1);

} // openage

Always pxd-declare your functions as except + ("may potentially rise a C++ exception"), unless the C++ function is marked noexcept itself. If a function that is not declared except + throws anyways, the entire CPython interpreter is likely to be shredded.

cmake must be informed about the pxd-annotated header file:

libopenage/CMakeLists.txt

pxdgen(example.h)

The function is now available from Cython. To make it available for pure-python modules, write a wrapper:

openage/foo.pyx

from libopenage.foo cimport foo as c_foo

def foo(int arg0, str arg1):
    with nogil:
        return c_foo(arg0, arg1)

openage/bar.py

from openage.foo import foo

print(foo(10, "test"))

Calling Python functions from C++

The interface works one-way: Cython can access libopenage, but not the other way round. Thus, libopenage must provide PyIfFunc function pointers that are filled by Cython during initialization.

Any cdef functions may be stored in C++ function pointers; the openage::pyinterface::PyIfFunc type has been created for this purpose; it allows binding arguments and makes sure that the pointer is properly initialized (instead of invoking undefined behavior if called before initialization).

We will call this pure-Python function from C++:

openage/bar.py

def bar(arg0, arg1):
    """
    This function involves rainbows and unicorns.

    arg0 shall be an integer, and arg1 a string.
    """
    return 6.283185307179586

Declare, define and pxd-export the function pointer:

libopenage/foo.cpp

#include "foo.h"

PyIfFunc<float, int, std::string> bar;

libopenage/foo.h

// pxd: from libcpp.string cimport string
#include <string>

#include "pyinterface/functional.h"

// pxd: PyIfFunc2[float, int, string] bar
extern PyIfFunc<float, int, std::string> bar;

Wrap the python function in a cdef function, and define a method setup(), which binds the cdef function to the PyIfFunc object.

openage/foo.pyx

from libopenage.foo cimport bar as c_bar

from .bar import bar as py_bar

cdef float bar(int arg0, string arg1) except * with gil:
    return py_bar(arg0, <str> arg1)

def setup():
    c_bar.bind0(bar)

PyIfFunc2 means that the function takes 2 arguments, and bind0 means that 0 arguments are bound (this is needed because Cython currently doesn't support variadic template arguments).

Add a call to openage.foo.setup() to openage.cppinterface.setup.setup.

openage/pyinterface/setup.pyx

def setup():
    # (...)

    from openage.foo import setup
    setup()

    # (...)

If you forget to do that, openage.pyinterface.setup.setup will raise a fatal exception.

libopenage/bar.cpp

#include "foo.h"

std::cout << openage::bar(5, "test") << std::endl;

The only way of accessing Python code from C++ should be via the Func or PyIfFunc function wrappers, as those guarantee that exceptions are properly translated, among other things.

Real-life examples

For code that wraps a C++ class for Python, see openage/cabextract/lzxd.pyx and libopenage/util/lzxd.h. For code that wraps a Python class for C++, see openage/util/fslike_cpp.pyx and libopenage/util/fslikeobject.h.

Notes on the GIL

The GIL must be acquired for any Python functionality (even as simple as PyErr_Occurred).

However, GIL-safety is guaranteed by the combination of Cython, the pyinterface code, and the fact that libopenage doesn't link against Python itself / include any Python headers.

Any code in libopenage can safely be run without the GIL.

Only functions that are marked with gil can be bound to PyIfFunc or Func objects; this ensures that the GIL is always re-acquired when jumping into Cython code.

Never use raw function pointers in the interface; always use the PyIfFunc or Func objects; otherwise, you'll lose all safety guarantees.