Python and Linux kernel 3.0: sys.platform != 'linux2'

It's getting more and more challenging to compile Python. Half a year ago Python 2.x's build system broke caused by multiarch support in Ubuntu Natty. Now Linux kernel 3.0 is going to reveal yet another issue in Python's configure script.

If you compile Python under kernel 3.0, sys.platform changes to 'linux3'. The altered platform string introduces bugs in several libraries and in our softwares stack, too. We and a lot of other people check for Linux with sys.platform == "linux2".

>>> import sys
>>> sys.platform
'linux3'

It turns out the 'configure' script causes the problem. It takes the lower case kernel name (uname -s) and first digit of the kernel release (uname -r) to fill the variable MACHDEP. The issue is discussed in http://bugs.python.org/issue12326 to a create length and addressed in upcoming releases 2.7.3, 3.2.2 and 3.3. The 2.7 and 3.2 series announce a 3.0 Linux kernel as linux2 platform. Starting with Python 3.3 sys.platform will be 'linux' for Kernel 2.x and 3.x.

However 2.7.3 isn't out yet. Worse older versions of Python are in maintenance mode and will only see security fixes. I'm going to show you, how you can work around the issue.

Change your software

I recommend that you replace all code like sys.platform == "linux2" with sys.platform.startswith("linux"). It causes the least trouble and is future compatible with Python 3.3 as well.

Alter the configure script

If you compile your own version of Python on Linux, you can alter the configure script before running it.

       case $MACHDEP in
      cygwin*) MACHDEP="cygwin";;
      darwin*) MACHDEP="darwin";;
      atheos*) MACHDEP="atheos";;
      irix646) MACHDEP="irix6";;
      linux*) MACHDEP="linux2";; # add this line
      '')     MACHDEP="unknown";;
      esac

Run make with MACHDEP=linux2

I find it easier to run make a different MACHDEP variable. It requires no patching.

    ./configure
   make MACHDEP=linux2
   make altinstall

Good luck!

smc.freeimage 0.0.2

A while ago I noticed that Christoph Gohlke has unofficial Windows builds of my smc.freeimage wrapper on his hosting site. It made me aware that some people actually use my image processing software.

A few minutes ago I synced our internal SVN repository with the project site on berlios.de. The recent version uses Cython 0.14 to wrap most of FreeImage 3.15.0 and a limited subset of LCMS 2.1. It can read over 30 image formats including subsets like G3 and G4 compressed TIFFs, which aren't supported by PIL. smc.freeimage also supports limited ICC transformation with embedded or external ICC profiles (for now 24bpp RGB images only) and introspection of ICC profiles. The ICC transformation is optimized with cached transformations and no-copy processing.

smc.freeimage wraps only functionality we actually need at work. The core features are heavily tested in production. I estimate that we have processed betwann 300 TB to about half a Petabyte of data, mostly uncompressed TIFF images but also compressed TIFFs, PNGs and JPEGs.

If you are interested on adding features or building a more general solution, feel free to contact me.

How to compile Python on Ubuntu 11.04

At work we deploy and compile our own Python environment on all server in order to have full control over versions, patches and libraries. Three days ago I stumbled upon a problem in our build process on Ubuntu Natty. Several modules like zlib weren't available. It took me a while to figure out the problem. Python's setup.py simply couldn't find libz.so in it's usual search paths like /usr/lib.

Natty has introduced a new feature called multiarch. Some shared libraries are installed in architecture specific directories, e.g. /usr/lib/x86_64-linux-gnu/libz.so instead of /usr/lib/libz.so. The dynamic linker ld.so has been modified to look for libraries in the new locations. If you wonder how, /etc/ld.so.conf.d/x86_64-linux-gnu.conf does the trick. However Python's setup.py uses hard coded paths and doesn't know about the new feature. Barry's posting [1] has some insight information.

The problem has been dealt with for Python 2.7, 3.1, 3.2 and newer versions, but 2.6 and earlier won't see any fixes. Current Python releases (2.7.1, 3.2.0) suffer from the issue, too. Don't be battle-weary! The solution to the issue is rather simple.

   $ make distclean
   $ export LDFLAGS="-L/usr/lib/$(dpkg-architecture -qDEB_HOST_MULTIARCH)"
   $ ./configure
   $ make
   $ make install
   $ unset LDFLAGS

This adds an additional library search path (-L/usr/lib/x86_64-linux-gnu on my box). Now setup.py knows about the multiarch lib directory and builds zlib and all the other missing modules just fine.

Actually my build system a bit more paranoid. It also adds /lib/x86_64-linux-gnu as library search path and /usr/include/x86_64-linux-gnu as header include path for C and C++.

   $ export arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
   $ export LDFLAGS="-L/usr/lib/$arch -L/lib/$arch"
   $ export CFLAGS="-I/usr/include/$arch"
   $ export CPPFLAGS="-I/usr/include/$arch"
   $ ./configure
   $ make
   $ make install
   $ unset arch LDFLAGS CFLAGS CPPFLAGS

[1] https://lists.ubuntu.com/archives/ubuntu-devel/2011-April/033049.html

Microsoft gives MSDN Premium subscription to PSF members

This almost went through unnoticed. Steve Holden (Python Software Foundation) has worked out a fantastic deal with Sam Ramji and Tom Hanrahan (both leading Open Source guys at Microsoft). Microsoft has given fourteen MSDN Premium subscriptions to Python core developers and PSF members. I'm one of the lucky few. [1], [2]

The premium subscription includes licenses and downloads of almost every Microsoft product from MS-DOS 6.22 to Windows 7, all versions of Visual Studio and many more stuff. This is very useful for Python core developers. Every developer with a subscription can finally set up multiple virtual boxes with 32 and 64bit versions of XP, Vista and Windows 7 to test and debug issues. 64bit versions of Windows were hard and costly to come by.

I'll keep my Ubuntu boxes for daily work and I'll still be skeptical about Microsoft's open source politics. However I'm glad that their paradigm towards Open Source is changing into the right direction. Python (more precisely IronPython) is going to become more important to Microsoft. I'll put my subscription into good use.

Thanks to Sam, Tom and Steve!

[1] http://mail.python.org/pipermail/python-dev/2009-July/090704.html
[2] http://mail.python.org/pipermail/python-dev/2009-August/091020.html

How to add a new module search path

Once in a while Python users are asking how to add some directories to sys.path permanently. Usually a solution like the PYTHONPATH env variable are suggested to the op. Other solutions require root privileges or modify the search path for all users. PEP 370 adds another way that is more clean and easy to use. It doesn't require root privileges and it doesn't suffer from other issues. PYTHONPATH causes trouble for multiple Python versions. C extensions only work for one version of Python, most Python modules won't work on Python 2 and 3.

My preferred way adds additional search pathes just for one version of Python and just for me. It uses a .pth file as explained in the site module manual. .pth files only work in site-packages directories, either the global or the user specific directories.

The Python way

$ python2.6
>>> import os
>>> import site
>>> site.USER_SITE

'/home/heimes/.local/lib/python2.6/site-packages'

Create the directory if it doesn't exist yet

>>> if not os.path.isdir(site.USER_SITE):
... os.makedirs(site.USER_SITE)
...

mypath.pth is going to contain my list of addition search path

>>> mypth = os.path.join(site.USER_SITE, "mypath.pth")
>>> path_to_add = ["/home/heimes/modules", "/home/heimes/other_modules"]

Add a list of search paths line by line, also make sure we end with an empty line

>>> with open(mypth, "a") as f:
... f.write("\n".join(path_to_add))
... f.write("\n")
...
>>>

The bash way

$ python2.6 -m site --user-site
/home/heimes/.local/lib/python2.6/site-packages
$ mkdir -p $(python2.6 -m site --user-site)
$ echo "/home/heimes/more_modules" >> $(python2.6 -m site --user-site)/mypath.pth

Let's check if it works

check the pth file

$ cat $(python2.6 -m site --user-site)/mypath.pth
/home/heimes/modules
/home/heimes/other_modules
/home/heimes/more_modules

Let's see if the modules are in the new search path ... they aren't because the directories don't exist yet.

$ python2.6 -m site
sys.path = [
'/home/heimes',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/home/heimes/.local/lib/python2.6/site-packages',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/PIL',
'/usr/lib/python2.6/dist-packages/gst-0.10',
'/var/lib/python-support/python2.6',
'/usr/lib/python2.6/dist-packages/gtk-2.0',
'/var/lib/python-support/python2.6/gtk-2.0',
'/var/lib/python-support/python2.6/pyinotify',
'/usr/lib/python2.6/dist-packages/wx-2.6-gtk2-unicode',
'/usr/local/lib/python2.6/dist-packages',
]
USER_BASE: '/home/heimes/.local' (exists)
USER_SITE: '/home/heimes/.local/lib/python2.6/site-packages' (exists)
ENABLE_USER_SITE: True

create one example directory

$ mkdir /home/heimes/modules
$ python2.6 -m site
sys.path = [
'/home/heimes',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/home/heimes/.local/lib/python2.6/site-packages',
'/home/heimes/modules',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/PIL',
'/usr/lib/python2.6/dist-packages/gst-0.10',
'/var/lib/python-support/python2.6',
'/usr/lib/python2.6/dist-packages/gtk-2.0',
'/var/lib/python-support/python2.6/gtk-2.0',
'/var/lib/python-support/python2.6/pyinotify',
'/usr/lib/python2.6/dist-packages/wx-2.6-gtk2-unicode',
'/usr/local/lib/python2.6/dist-packages',
]
USER_BASE: '/home/heimes/.local' (exists)
USER_SITE: '/home/heimes/.local/lib/python2.6/site-packages' (exists)
ENABLE_USER_SITE: True

Easy, isnt' it?

libxml2 crash on 64bit Ubuntu

I've spent the last couple of hours debugging a really strange segfault. Our application stack had a reproduceable crash in libxml2 -- but only with self compiled versions of libxml2. Ubuntu's 2.6.32 worked like a charm, my self compiled 2.6.32 didn't. The very same version works on several other Debian, Redhat and SuSE boxes, 32 and 64bit, too. WTF!?

The crash always occured in xmlIO.c:__xmlParserInputBufferCreateFilename() with xmlGzfileOpen() as open handler. After several gdb debugging sessions and several recompiles I noticed a suspicious message in the make output:

xmlIO.c: In function 'xmlGzfileOpen_real':
xmlIO.c:1132: warning: implicit declaration of function 'gzopen64'
xmlIO.c:1132: warning: nested extern declaration of 'gzopen64'
xmlIO.c:1132: warning: assignment makes pointer from integer without a cast
xmlIO.c: In function 'xmlGzfileOpenW':
xmlIO.c:1200: warning: assignment makes pointer from integer without a cast

The message only occured during my own compiles but not during "apt-get source -b libxml2" . Apparently Ubuntu has patched the sources to fix the issue. The changelog contains yet another hint:

* libxml.h: define _LARGEFILE64_SOURCE to properly get gzopen64 defines in zlib.h. Closes: #439843. Thanks Dann Frazier.

That's the solution to my problem! CFLAGS="-D_LARGEFILE64_SOURCE" ./configure and both the compiler warning and the crash is gone.

multiprocessing 2.6.2.1 released

A new version of the multiprocessing backport to Python 2.4 and 2.5 has been released. It contains all fixes from the Python 2.6 branch. As usually the release is available as tar.gz and Windows installer for Python 2.4 and 2.5 on PyPI.