Building a Python C Extension Module

Understanding fputs()

fputs() is the C library function that you’ll be wrapping:

int fputs(const char *, FILE *)

This function takes two arguments:

1. const char * is an array of characters.
2. FILE * is a file stream pointer.

fputs() writes the character array to the file specified by the file stream and returns a non-negative value. If the operation is successful, then this value will denote the number of bytes written to the file. If there’s an error, then it returns EOF. You can read more about this C library function and its other variants in the manual page entry.

Writing the C Function for fputs()

This is a basic C program that uses fputs() to write a string to a file stream:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main() {
    FILE *fp = fopen("write.txt", "w");
    fputs("Real Python!", fp);
    fclose(fp);
    return 1;
}

This snippet of code can be summarized as follows:

1. Open the file write.txt.
2. Write the string "Real Python!" to the file.

In the following section, you’ll write a wrapper for this C function.

Wrapping fputs()

It might seem a little weird to see the full code before an explanation of how it works. However, taking a moment to inspect the final product will supplement your understanding in the following sections. The code block below shows the final wrapped version of your C code:

 1#include <Python.h>
 2
 3static PyObject *method_fputs(PyObject *self, PyObject *args) {
 4    char *str, *filename = NULL;
 5    int bytes_copied = -1;
 6
 7    /* Parse arguments */
 8    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {
 9        return NULL;
10    }
11
12    FILE *fp = fopen(filename, "w");
13    bytes_copied = fputs(str, fp);
14    fclose(fp);
15
16    return PyLong_FromLong(bytes_copied);
17}

This code snippet references three object structures which are defined in Python.h:

1. PyObject
2. PyArg_ParseTuple()
3. PyLong_FromLong()

These are used for data type definition for the Python language. You’ll go through each of them now.

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Snip */

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {
 7        return NULL;
 8    }
 9
10    /* Snip */

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {
 7        return NULL;
 8    }
 9
10    FILE *fp = fopen(filename, "w");
11    bytes_copied = fputs(str, fp);
12    fclose(fp);
13
14    return PyLong_FromLong(bytes_copied);
15}

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {
 7        return NULL;
 8    }
 9
10    FILE *fp = fopen(filename, "w");
11    bytes_copied = fputs(str, fp);
12    fclose(fp);
13
14    return PyLong_FromLong(bytes_copied);
15}

Writing the Init Function

You’ve written the code that makes up the core functionality of your Python C extension module. However, there are still a few extra functions that are necessary to get your module up and running. You’ll need to write definitions of your module and the methods it contains, like so:

static PyMethodDef FputsMethods[] = {
    {"fputs", method_fputs, METH_VARARGS, "Python interface for fputs C library function"},
    {NULL, NULL, 0, NULL}
};


static struct PyModuleDef fputsmodule = {
    PyModuleDef_HEAD_INIT,
    "fputs",
    "Python interface for the fputs C library function",
    -1,
    FputsMethods
};

These functions include meta information about your module that will be used by the Python interpreter. Let’s go through each of the structs above to see how they work.

static PyMethodDef FputsMethods[] = {
    {"fputs", method_fputs, METH_VARARGS, "Python interface for fputs C library function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef fputsmodule = {
    PyModuleDef_HEAD_INIT,
    "fputs",
    "Python interface for the fputs C library function",
    -1,
    FputsMethods
};

PyMODINIT_FUNC PyInit_fputs(void) {
    return PyModule_Create(&fputsmodule);
}

Putting It All Together

Now that you’ve written the necessary parts of your Python C extension module, let’s take a step back to see how it all fits together. The following diagram shows the components of your module and how they interact with the Python interpreter:

When you import your Python C extension module, PyInit_fputs() is the first method to be invoked. However, before a reference is returned to the Python interpreter, the function makes a subsequent call to PyModule_Create(). This will initialize the structures PyModuleDef and PyMethodDef, which hold meta information about your module. It makes sense to have them ready since you’ll make use of them in your init function.

Once this is complete, a reference to the module object is finally returned to the Python interpreter. The following diagram shows the internal flow of your module:

The module object returned by PyModule_Create() has a reference to the module structure PyModuleDef, which in turn has a reference to the method table PyMethodDef. When you call a method defined in your Python C extension module, the Python interpreter uses the module object and all of the references it carries to execute the specific method. (While this isn’t exactly how the Python interpreter handles things under the hood, it’ll give you an idea of how it works.)

Similarly, you can access various other methods and properties of your module, such as the module docstring or the method docstring. These are defined inside their respective structures.

Now you have an idea of what happens when you call fputs() from the Python interpreter. The interpreter uses your module object as well as the module and method references to invoke the method. Finally, let’s take a look at how the interpreter handles the actual execution of your Python C extension module:

Once method_fputs() is invoked, the program executes the following steps:

1. Parse the arguments you passed from the Python interpreter with PyArg_ParseTuple()
2. Pass these arguments to fputs(), the C library function that forms the crux of your module
3. Use PyLong_FromLong to return the value from fputs()

To see these same steps in code, take a look at method_fputs() again:

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {
 7        return NULL;
 8    }
 9
10    FILE *fp = fopen(filename, "w");
11    bytes_copied = fputs(str, fp);
12    fclose(fp);
13
14    return PyLong_FromLong(bytes_copied);
15}

To recap, your method will parse the arguments passed to your module, send them on to fputs(), and return the results.

Building Your Module

Now that you have your setup.py file, you can use it to build your Python C extension module. It’s strongly advised that you use a virtual environment to avoid conflicts with your Python environment.

Navigate to the directory containing setup.py and run the following command:

$ python3 setup.py install

This command will compile and install your Python C extension module in the current directory. If there are any errors or warnings, then your program will throw them now. Make sure you fix these before you try to import your module.

By default, the Python interpreter uses clang for compiling the C code. If you want to use gcc or any other C compiler for the job, then you need to set the CC environment variable accordingly, either inside the setup script or directly on the command line. For instance, you can tell the Python interpreter to use gcc to compile and build your module this way:

$ CC=gcc python3 setup.py install

However, the Python interpreter will automatically fall back to gcc if clang is not available.

Running Your Module

Now that everything is in place, it’s time to see your module in action! Once it’s successfully built, fire up the interpreter to test run your Python C extension module:

>>> import fputs
>>> fputs.__doc__
'Python interface for the fputs C library function'
>>> fputs.__name__
'fputs'
>>> # Write to an empty file named `write.txt`
>>> fputs.fputs("Real Python!", "write.txt")
13
>>> with open("write.txt", "r") as f:
>>>     print(f.read())
'Real Python!'

Your function performs as expected! You pass a string "Real Python!" and a file to write this string to, write.txt. The call to fputs() returns the number of bytes written to the file. You can verify this by printing the contents of the file.

Also recall how you passed certain arguments to the PyModuleDef and PyMethodDef structures. You can see from this output that Python has used these structures to assign things like the function name and docstring.

With that, you have a basic version of your module ready, but there’s a lot more that you can do! You can improve your module by adding things like custom exceptions and constants.

Raising Exceptions From C Code

While you can’t raise exceptions in C, the Python API will allow you to raise exceptions from your Python C extension module. Let’s test this functionality by adding PyErr_SetString() to your code. This will raise an exception whenever the length of the string to be written is less than 10 characters:

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &fd)) {
 7        return NULL;
 8    }
 9
10    if (strlen(str) < 10) {
11        PyErr_SetString(PyExc_ValueError, "String length must be greater than 10");
12        return NULL;
13    }
14
15    fp = fopen(filename, "w");
16    bytes_copied = fputs(str, fp);
17    fclose(fp);
18
19    return PyLong_FromLong(bytes_copied);
20}

Here, you check the length of the input string immediately after you parse the arguments and before you call fputs(). If the string passed by the user is shorter than 10 characters, then your program will raise a ValueError with a custom message. The program execution stops as soon as the exception occurs.

Note how method_fputs() returns NULL after raising the exception. This is because whenever you raise an exception using PyErr_*(), it automatically sets an internal entry in the exception table and returns it. The calling function is not required to subsequently set the entry again. For this reason, the calling function returns a value that indicates failure, usually NULL or -1. (This should also explain why there was a need to return NULL when you parse arguments in method_fputs() using PyArg_ParseTuple().)

Raising Custom Exceptions

You can also raise custom exceptions in your Python C extension module. However, things are a bit different. Previously, in PyMODINIT_FUNC, you were simply returning the instance returned by PyModule_Create and calling it a day. But for your custom exception to be accessible by the user of your module, you need to add your custom exception to your module instance before you return it:

static PyObject *StringTooShortError = NULL;

PyMODINIT_FUNC PyInit_fputs(void) {
    /* Assign module value */
    PyObject *module = PyModule_Create(&fputsmodule);

    /* Initialize new exception object */
    StringTooShortError = PyErr_NewException("fputs.StringTooShortError", NULL, NULL);

    /* Add exception object to your module */
    PyModule_AddObject(module, "StringTooShortError", StringTooShortError);

    return module;
}

As before, you start off by creating a module object. Then you create a new exception object using PyErr_NewException. This takes a string of the form module.classname as the name of the exception class that you wish to create. Choose something descriptive to make it easier for the user to interpret what has actually gone wrong.

Next, you add this to your module object using PyModule_AddObject. This takes your module object, the name of the new object being added, and the custom exception object itself as arguments. Finally, you return your module object.

Now that you’ve defined a custom exception for your module to raise, you need to update method_fputs() so that it raises the appropriate exception:

 1static PyObject *method_fputs(PyObject *self, PyObject *args) {
 2    char *str, *filename = NULL;
 3    int bytes_copied = -1;
 4
 5    /* Parse arguments */
 6    if(!PyArg_ParseTuple(args, "ss", &str, &fd)) {
 7        return NULL;
 8    }
 9
10    if (strlen(str) < 10) {
11        /* Passing custom exception */
12        PyErr_SetString(StringTooShortError, "String length must be greater than 10");
13        return NULL;
14    }
15
16    fp = fopen(filename, "w");
17    bytes_copied = fputs(str, fp);
18    fclose(fp);
19
20    return PyLong_FromLong(bytes_copied);
21}

After building the module with the new changes, you can test that your custom exception is working as expected by trying to write a string that is less than 10 characters in length:

>>> import fputs
>>> # Custom exception
>>> fputs.fputs("RP!", fp.fileno())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
fputs.StringTooShortError: String length must be greater than 10

When you try to write a string with fewer than 10 characters, your custom exception is raised with a message explaining what went wrong.