Building my Python toolchain with MinGW

Update: GDAL is quite broken when built with my hand crafted MinGW (segfaults, memory leaks, etc...), so I'll use VS2008 versions of everything until a proper installer is released.

My old Python toolchain for Windows started to give signs of weakness recently (strange segfaults and errors on my py2exe'd PyQt apps), so I wanted to rebuild it from scratch with the latest and greatest MinGW.

The first problem I encountered was downloading MinGW itself. While the old distribution had a nice automated installer, you now have to download each and every package individually. This was going to take ages, especially with the clumsy sourceforge "file browser", so I wrote a small script to automate the process. It simply browses a SF mirror and tries to download the latest versions of everything.

You still have to download the "core" packages for MSYS and MinGW. For MSYS I chose the latest installer available from the sourceforge download pages ("MSYS-1.0.10.exe", located in "MSYS/BaseSystem/msys-1.0.10" at the time of writing). For the MinGW part, I chose GCC 4.4 (in "MinGW/BaseSystem/GCC/Version4/Previous Release gcc-4.4.0") because I had problems building Qt with the latest 4.5 version. You'll also have to download make for both MinGW and MSYS.

Once you have all the package files, start by installing MSYS with the installer, and then extract the packages where you installed it (normally "c:\msys\1.0").

We can now start building the libraries. Important: the libs using mingw32-make need to be built from a non MSYS shell, or you'll get error when doing "mingw32-make install".

proj.4 4.7.0

./configure --without-mutex --prefix=/mingw
make
make install

GDAL 1.7.2

Start by installing numpy, either from source or the binary package.

GDAL doesn't like being built from a mixed case path on Windows, so start by moving the source tree to a lower case location (e.g. "c:\gdal"). It also has problems with MinGW's libtool and didn't like my expat version (2.0.1), so I disabled them.

If numpy is installed, the python bindings build will fail with an error message saying that sprintf() was defined twice. The solution is to edit "swig/python/setup.py", search for the "array_module" extension definition and add "-D __USE_MINGW_ANSI_STDIO" to its compilation args:

array_module = Extension('osgeo._gdal_array',
        sources=['extensions/gdal_array_wrap.cpp'],
        extra_compile_args=["-D __USE_MINGW_ANSI_STDIO"],
        extra_link_args=extra_link_args)

You can now build and install GDAL:

./configure --without-libtool --without-expat --without-python \
    --prefix=/mingw --includedir=/mingw/include/gdal
make
make install
cd swig/python
python setup.py build_ext -c mingw32
python setup.py install

Qt 4.6.3

Extract Qt source distribution to somewhere standard, e.g. "c:\Qt\4.6.3". It is important to choose this location well before compiling, as the path will be hardcoded in qmake.

./configure.exe -platform win32-g++ -opensource
mingw32-make && mingw32-make install

Notes:

• be careful to use single hyphens to specify options
• explicitly specify "configure.exe", not "configure"
• the make and install phases can be very long, that's why I prefer to chain them with '&&'

SIP 4.10.2

python configure.py --platform win32-g++
mingw32-make
mingw32-make install

PyQt 4.7.3

python configure.py
mingw32-make
mingw32-make install

potrace 1.8

CPPFLAGS=-I/include LDFLAGS=-L/lib ./configure --prefix=/mingw
make
make install
cp src/libpotrace.a /mingw/lib
cp src/potracelib.h /mingw/include

AGG 2.5

First make sure you have pkg-config (you need both the "Tool" and "Dev" packages, and also glib available on the same page) in your path.

make
mkdir /mingw/include/agg2
cp -r include/* /mingw/include/agg2
cp src/libagg.a /mingw/lib

(Yes, there is a set of autotools files... but I've not been able to make them work with my msys install)

Setting up Hudson on Ubuntu

Basic setup

First install the Hudson package, as explained in these instructions:

wget -O /tmp/key http://hudson-ci.org/debian/hudson-ci.org.key
sudo apt-key add /tmp/key
wget -O /tmp/hudson.deb http://hudson-ci.org/latest/debian/hudson.deb
sudo apt-get install daemon gij-4.3 openjdk-6-jre
sudo dpkg --install /tmp/hudson.deb

If all went well, Hudson is now accessible at http://localhost:8080/.

The next thing to do is to install a plugin for your favorite SCM. I use git so I went to Manage Hudson > Manage plugins and installed Hudson GIT plugin. Also install the Chuck Norris plugin for extra control.

You can now create a new job for your project. For this example I will setup a Python project that uses nose for its tests.

Go to New job, give it a name and select Build a free-style software project.

In the job settings, select GIT and enter the (read-only) URL of your project. You can check Poll SCM to trigger builds when the repository is updated, in that case you must define the poll frequency in the schedule text area, e.g. "* * * * *" to poll every minute.

Click the Add build step button, choose Execute shell and make it launch nosetests:

nosetests --with-xunit --verbose

Finally, check the Publish JUnit test result report option, and type **/nosetests.xml in there.

And you're done ! You can now start a build and check that it works fine. If it fails, look at the build's console output for details.

Advanced: setting up a VNC server for tests that require a display

Some tests, like the ones using the QtTest module, need a display connection to work and will fail:

cannot connect to X server :0.0

Hudson runs as a daemon and doesn't have access to a physical display, so we need to fake one.

Fortunately there is a plugin handling that problem in Hudson. Go to the plugins installation page and install the Hudson Xvnc plugin. You also need to install a VNC server on your machine. I chose to use tightvnc:

sudo apt-get install tightvncserver

Go to the Hudson configuration page (Manage Hudson > Configure system) and enter this command line in the Xvnc section:

/usr/bin/Xvnc :$DISPLAY_NUMBER

Return to your job's configuration and check the Run Xvnc during build option. Your tests will use the VNC server for their display, and should hopefully pass.

About the blog name

Thanks to Google's CAPTCHA for generating this fine name.