Plumbling Life's Depths

Have some time before a client meeting this afternoon... so here's the issue I've been turning over in my mind since it was brought up at the last PyGTA.  We've run into the same issue at multiple clients and while the "right" solution would likely be to make easy_install/distutils record the information, I've spent way too much of my life inside the distutils code-base already (we used to have custom distutils code in PyOpenGL).

So here's the problem/use-case:

You run easy_install to install TurboGears (or any other huge multi-package system) into a virtualenv, it downloads 45 (or so) eggs.  You run setup.py on your TG application... another 4 (or so) eggs.  You hand the package to someone else, they run the setup... and it doesn't work, some dependency changed version ever so slightly, you now get to track down which one and fix it.

Buildout, PoachEggs and the like seem to want you to manually specify all 49 packages (with the idea that you do it once and then everyone shares the joy).  Problem is, I'm a lazy cuss.  I may have issues with easy_install, but it's the closest thing we've got to a package manager for Python eggs at the moment, so how do I just ask easy_install to do "exactly the same install" as it just did on some other machine?

If you're me, you hack up a dumb script that processes the output of easy_install looking for "Downloading " lines.  You make the script download those files into a directory (and record an index file so it knows the order), you then add an option to the same script to replay the install with extremely restrictive easy_install options (i.e. no hosts, no dependency resolution, etceteras).

Obviously this falls into the "expediant hack" rather than "properly engineered" pile, but it does look like it would work.  It generates an editable file you can tweak, just as you might tweak a "recipe" base system's configuration, but it automates the initial generation of that recipe.

The project is on LaunchPad, as is the code (bzr branch lp:recordeggs) and the downloads are in PyPI.

To use it:

easy_install -i http://www.turbogears.org/2.0/downloads/2.0.1/index tg.devtools 2>&1 | recordeggs.py ../egg-sources
recordeggs -p ../egg-sources

The "redirect stderr to stdout" stuff is one of those things I always need to look up, so maybe I should just run easy_install as a captive process, but for now this seems to work.

[Update] usage is now:

recordeggs ../egg-sources [easy_install options/args]
recordeggs -p ../egg-sources

which feels a lot more natural, but I haven't yet got the pipes aligned so you could see the easy_install output as it processes (at the moment it kicks all the output out at the end of the run).

Spent the afternoon testing PyOpenGL 3.0.1, dull, really.  Found a few errors in the Demo where code was trying to use OpenGL before there is a context (recent Linux drivers are getting really picky about that).  Added a flag to the root OpenGL namespace to control unpacking of single-length arrays, thinking that PyOpenGL 3.1 should finally break that API to return arrays instead.

Got bored with that and added primitive textured-sprite support to OpenGLContext's PointSet objects.  Basically if you have a texture in the shape the PointSet will enable sprites, however it doesn't currently have any mechanisms for configuring them (i.e. setting points sizes or attenuation).  The test/demo just hard-codes that at the application level for now.  Upshot is that the demo now has big fat water droplets instead of single-pixel points .

Idly trying to get my (Lenovo W500) to suspend again this morning as I wait for the washing machine to finish.  Biggest problem is that there are so many sites all with outdated information so I have to filter through hundreds of posts most of which are discussing e.g. Ubuntu 8.10 (which suspended fine with compiz disabled), or how to get compiz+suspend working (I don't need the effects).

Seems we need a tracker somewhere that "moves" with the current version of Ubuntu, where we can install, test each component and report status and any work-arounds needed.  You'd be able to look at the 8.10, but 9.04 would be what's there by default.  You could search within each component's history...

Or we could all just write nice-o-grams to Lenovo to tell them to support Linux properly on their machines.

Ah well, laundry is finished, so no suspense for me...

 Began work on simplifying the PyQNet API.  Got rid of blocking/non-blocking versions (everything is non-blocking now, though there's a "wait" operation to block until incoming messages are received).  Also simplified the callback-based (server) interface.  Added (optional) zlib compression to the packages and added support for arbitrarily sized messages.

Probably next up would be getting the Pygame event sending version up and then building a default "portal" application for matching up players.  Not really something people should be integrating into their games yet, but it's beginning to shape up a little.

I spent far too long today playing with two different networking systems.  The first was RabbitMQ and py-amqplib.  This seems pretty nice, you can send hundreds of messages in a fraction of a second... though it then seems to take forever to process them.  Appears to force in-order delivery, including an explicit ack from the client code... probably not suitable for low-latency (game) applications.

So for some reason I started playing with PyQNet.  It was (by contrast) processing around 3000 messages/second, though the messages in PyQNet can get loaded into a single package/packet and send together, so there would be fewer packets, though that was with 500B packages, so not all that many fewer.  Thing is, it doesn't matter how fast it is, as no-one uses it.

PyQNet has a traditional "struct" based interface.  You define message-types (structs) and the structs are linearized/delinearized on sending/receiving.  But what newbie user wants to define structs like that?  So this evening, after PyGTA, I sat down and coded up a simple experiment to use compressed json as the wire encoding for data... still preliminary, but it seems to slow down to about 1/10th of the speed, but it does feel better as an API.

Ah well, have spent far too much time playing with this stuff today, maybe some other day I'll actually have a game to write and writing a network library for it will come up again.

Behdad is going to be presenting how to use PyCairo.  PyCairo is a immediate-mode 2D rendering library.  It can target all sorts of back-ends including OpenGL, X windows, PDF, etceteras.

[Update] We'll still be on Wednesday, but turns out Behdad is unavailable due to the turmoil in Iran.

I've just released the Python 2.6 compatible version of SimpleParse.  The only changes of note were the 2.6 fixes and a renaming of the simpleparse.xml module to simpleparse.xmlparser, as under Python 2.6 import semantics the "xml" module caused an import conflict.

I've released only the 2.5 win32 egg, as I'm using my Vista machine to build and I don't yet have a MSVC compiler chain on there (I have mingw32, and that creates unloadable libraries when I compile the package for Python 2.6).