Freitag, 20. Mai 2011

How to compile Python on Ubuntu 11.04

At work we deploy and compile our own Python environment on all server in order to have full control over versions, patches and libraries. Three days ago I stumbled upon a problem in our build process on Ubuntu Natty. Several modules like zlib weren't available. It took me a while to figure out the problem. Python's setup.py simply couldn't find libz.so in it's usual search paths like /usr/lib.

Natty has introduced a new feature called multiarch. Some shared libraries are installed in architecture specific directories, e.g. /usr/lib/x86_64-linux-gnu/libz.so instead of /usr/lib/libz.so. The dynamic linker ld.so has been modified to look for libraries in the new locations. If you wonder how, /etc/ld.so.conf.d/x86_64-linux-gnu.conf does the trick. However Python's setup.py uses hard coded paths and doesn't know about the new feature. Barry's posting [1] has some insight information.

The problem has been dealt with for Python 2.7, 3.1, 3.2 and newer versions, but 2.6 and earlier won't see any fixes. Current Python releases (2.7.1, 3.2.0) suffer from the issue, too. Don't be battle-weary! The solution to the issue is rather simple.

$ make distclean
$ export LDFLAGS="-L/usr/lib/$(dpkg-architecture -qDEB_HOST_MULTIARCH)"
$ ./configure
$ make
$ make install
$ unset LDFLAGS

This adds an additional library search path (-L/usr/lib/x86_64-linux-gnu on my box). Now setup.py knows about the multiarch lib directory and builds zlib and all the other missing modules just fine.

Actually my build system a bit more paranoid. It also adds /lib/x86_64-linux-gnu as library search path and /usr/include/x86_64-linux-gnu as header include path for C and C++.

$ export arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
$ export LDFLAGS="-L/usr/lib/$arch -L/lib/$arch"
$ export CFLAGS="-I/usr/include/$arch"
$ export CPPFLAGS="-I/usr/include/$arch"
$ ./configure
$ make
$ make install
$ unset arch LDFLAGS CFLAGS CPPFLAGS

[1] https://lists.ubuntu.com/archives/ubuntu-devel/2011-April/033049.html

Kommentare:

  1. Hmm, that may also be why my dev builds broke after the upgrade.

    AntwortenLöschen
  2. Indeed, that was the problem - thanks for the tip!

    AntwortenLöschen
  3. Yep, thanks for the blog posting. I was actually surprised about the change too, and have had some discussions both private and public about the way multiarch was rolled out. I don't disagree with the need for the feature, which had actually been planned for several cycles, but it was difficult for the multiarch advocates to know everything that would be affected by it. Sadly, 2.6 was already in security-fix only mode by then so the change couldn't be backported.

    AntwortenLöschen
  4. Thanks for the blog post...just spent half the day trying to figure out why it wasn't grabbing the zlib, and the frustrated search of "why is it so hard to compile python on linux today" returned your blog post lol

    AntwortenLöschen
  5. Great tip, thanks.

    Also, some more details regarding _ssl: http://askubuntu.com/questions/21547/what-are-the-packages-libraries-i-should-install-before-compiling-python-from-so

    AntwortenLöschen
  6. Thank you - solved my problem today!

    AntwortenLöschen
  7. Thank you a lot! That problem has spoiled a half of my day. But you've saved a second one!

    AntwortenLöschen