Python's Flying Circus

Lately I’ve been working on integrating (or, more accurately, attempting to integrate) the Python scripting language into some plugins for an application we develop plugins for. We’ve wrapped many libraries with varying levels of success, so this one wasn’t going to be much different. Or, so we thought. Technically, we’re interested in “Embedding” Python — We have a native Objective-C application that needs to make use of the Python interpreter at various stages of execution. The Python scripts are user-supplied, and are exceptionally free-form: the functions they write can take variable numbers of inputs, produce variable numbers of outputs, and can use any Python modules they have installed.

Unfortunately, this makes the embedding process somewhat complicated: From Objective-C, we need to be able to parse the script, find all its functions, and get all those functions’ input and output parameters. From this data, we can expose the Python module’s interface usefully. Unfortunately, Python does not appear to allow this kind of introspection from the outside (and it’s questionable whether or not it’s even possible from the inside.)

In searching for information on this, Numerous Documents are Found that Needlessly Complicate or Obfuscate the difference between Embedding and Extending. As if it’s really that difficult (embed means “fix firmly and deeply in a surrounding mass” while extend means “cause to cover a larger area; make longer or wider”. From these obvious definitions, we can infer that “Embedding” means putting something inside, while “Extending” means adding functionality or abilities. Maybe it’s not so clear for non-native english speakers. I don’t know.)

Another annoying side-trip of this research was an overwhelmingly smug idea that almost reeks of Java. This side effect is the recorded smarmy discussion of why one should “Extend” rather than “Embed” (these reasons are then used to explain away why there isn’t any good documentation on actually embedding Python). The arguments go something like this:

  • Python’s so good, cross-platform, and flexible that it’s actually more cost-effective to Throw Out All Your C/C++/ObjC Code And Rewrite Everything In Python.
  • Embedding is so cumbersome to code, and so difficult compared to embedding Python in Python, that you should Throw Out All Your C/C++/ObjC Code And Rewrite Everything In Python.
  • High-performance code can be written in C, and then called from within Python’s runtime when you actually need to performance boost. Why not just Throw Out All Your C/C++/ObjC Code And Rewrite Everything In Python Except for the fast bits?
  • If you embed Python, you’ll annoy Python developers who can’t access the modules they’re used to using. Why not Throw Out All Your C/C++/ObjC Code And Rewrite Everything In Python so you don’t annoy your Python developers?

These arguments cover some pretty diverse ground there. Unfortunately, there are 2 fatals flaws in the list above.

First, embedded Python Can in fact use installed modules just like raw Python, so point 4 above is flagrantly incorrect. There are some suggestions that it’s simply annoying to develop in such an environment because namespaces are all strange (or wrong) and nothing works quite right. Guess What: Welcome To Plugin Development! It’s like these Python developers have never worked on real projects before or something.

Second, All of the above points assume that it’s possible to discard your entire code-base and rewrite everything. While this is technically possible, it’s not very likely when you don’t have access to the source of the application to be discarded (in our case). The unfortunate downside is that All The Listed “Solutions” Hinge On This One Idea.

What’s with these non-solutions from allegedly flexible languages? Even Objective-C, a compiled language, and the (totally undocumented) JavaVM-ObjC bridge, offer enough introspection to at least find methods and parameter counts without too much hassle. Maybe Python does this just as simply, but if so, no one’s talking about it…