Plumbing Life's Depths

Problem that looks like it will be coming up soon... you are always wanting to be able to test your code against a (huge, PostgreSQL) LIVE DB before you release/promote the code. Loading a DB dump can take hours. Your test code can't all just run in a single transaction.

First approach: run the DB server in a virtual machine, sync the LIVE DB down to the virtual machine, snapshot just before you run tests, throw away the snapshot.

Second approach: run the DB server on real hardware, sync the LIVE DB down to a template DB on the real hardware, drop the database and recreate based on the template (note: you have to stop the sync before you create the new db). In my spike test the create-from-template approach still takes a few minutes, but it's far faster than loading an SQL dump.

create database moo with template moo_template;

Both approaches, of course, require a DB sync operation... that's the part I still need to research.

Was looking up something for a backup operation and saw a command where someone was catting from ssh... I'd never thought of doing it before, but it does make sense:

ssh dbhost.example.com "pg_dumpall | gzip -c" > ~/cluster.sql.gz

should connect and dump the entire DB cluster (gzipped) to your local machine.

ssh dbhost.example.com "gunzip -c | pg_restore" < ~/cluster.sql.gz

should rebuild the database on the machine (assuming you've rebuilt the cluster in the meantime). One could, for that matter, pipe the result of the downloaded db directly into pg_restore on another machine (potentially also going over an ssh link). Of course, if your ssh link goes away, sucks to be you, but still a cool effect.

x.split( '.' )

Is an idiom that shows up all over the place, particularly in list comprehensions. Run your code through 2to3 and it fails when x is a str (bytes) object, as 3.x thinks the '.' is unicode. It fails with a rather cryptic TypeError: Type str doesn't support the buffer API error as it's apparently trying to convert '.' using the Py_Buffer protocol rather than up-converting x to unicode. The number of places this needs to be tracked down (I constantly sling strings around and there are lots of string literals around in the codebase) to continue porting PyOpenGL and OpenGLContext has put me into "meh" on the project for now.

At work I need to interact with an svn (1.4) repository, but honestly I'm so spoiled by bzr that it just drives me nuts, particularly the (frustratingly frequent) situations where I miss a particular change-set/revision when merging into my tree and wind up reverting someone's changes.  The VCS should bloody-well track what revisions are missing, says I.

So, I've been using bzr-svn and bzr-rebase to bridge the gap, and so far it has been quite workable, with a few caveats.  The pattern:

• bzr init-repo # create a repository which shares data for all your branches/checkouts
• bzr co http://server/path/to/repo/trunk
• bzr branch trunk working

the co takes an extremely long time (for some of my personal projects it takes hours, enough to make it worth converting the project to bzr as soon as it finishes), but it eventually finishes.  Now, your colleagues are going to be committing their changes to trunk, to pull them in (I suggest doing this frequently, which is not something I suggest with svn 1.4):

• cd trunk
• bzr up
• cd ../working
• bzr rebase

which moves your commits to being made against current trunk.  Be careful with bzr-rebase, read the docs and understand what you are doing, you can lose your changes if you are not careful. Keep hacking until you're ready to commit to trunk.

DO NOT PUSH or PULL!

While this would normally be fine with bzr, and seems to "work" with bzr-svn, bzr-svn appears to want to modify every revision in the whole SVN repository with its metadata so that other bzr-svn users can work with you using direct bzr to bzr operations.  Honestly I've never let it try this (ctrl-c ctrl-c ctrl-c argh!), as I'm pretty sure that would (rightly) tick off all the svn users.  Instead, use bzr merge to pull the changes in:

• cd trunk
• bzr up # if there are changes re-rebase
• bzr merge ../working
• bzr commit

since you've rebased onto trunk, your merge should be clean.  There are going to be a lot of extra properties added to the svn commit (enough to make other users go "yech" when they look at the Trac view), but that's more than made up for by not having to manually track merged change-sets.

If I didn't get annoyed by git's command-line interface I could use that instead (one of our other devs does that), but I want to reduce the total pain felt.

Decided to see how much work getting PyOpenGL ported to Python 3.x would be.  Notes so far:

• ubuntu has Python 3.1 packaged (yay)
• also has distribute and -dev, you'll need both
• virtualenv -- this is key for my porting work-flow (not just for PyOpenGL, for almost every project), had to use an hg branch to get it working
• numpy can be built from svn for Python 3.1
• just added a flag to setup so that 2to3 would get run for Python 3 installs via easy_install
• got lots of NameErrors where I was cleaning up (deleting) loop vars after the loops, added try:except NameError around those
• str array support will need to be reworked slightly, may as well support arbitrary buffers while we're at it (we use Python-API calls to
  get the string pointer, may as well allow passing any buffer-protocol-supporting instance)
• didn't try building the accelerator modules
• without pygame, the built-in tests won't run, didn't feel like building pygame from svn just to test this
• GLUT-only demos do run, all looks to be working properly, save that without string support shaders can't be compiled (they are passed as arrays-of-strings).
• Demo code isn't currently easy_installed, so we don't get free 2to3 support there

Which is about where I ran out of interest.  All the code changes to work with Python3 so far were compatible with Python2.  Updating/fixing the string/buffer support should be straightforward, I can actually write a different plug-in and only register that one for Python 3.x.  Cython should, I believe, be Python 3.x compatible already, so even the accelerator modules should be pretty easy to port.  Still doesn't seem that we're at a point where Python 3.x is ready for PyOpenGL, though, but when it does get here we should be in pretty good shape.  OpenGLContext will be a lot more work, as it requires PyVRML97 and SimpleParse, both of which have hand-coded C extensions that aren't even ported to 2.7 yet.

The backups did, indeed, work, though the /etc/ backup wasn't quite as recent as I'd have liked.  However, now another disk is corrupting (this time the home/var disk), and it looks like it's the motherboard's SATA controller at fault.  Replacing that will require new motherboard, memory, and CPU... and yet another day of working on it instead of the dozens of tasks on my todo list.  Going to have to shut down the machine until I get the parts and the time to fix it.

[Update] Motherboard + CPU + Memory or buy a PCI SATA card for $50... I am tempted...

Just log onto the server and see how I arranged the backups... duh!  Server's disk has corrupted irrecoverably.  Hope those backups worked (won't know until I can buy a new disk on Tuesday).