A few days of paying work (Just for a change...)

Switched back to the web-"spidering" application yesterday, a few more hours of work today on it. This is really just a web-checking engine that goes out and sees if any of a few tens of thousands of pages have changed in the last day or so and reports the results.

Nothing really complex there, but it's the first application I've created by re-using components from the application framework in Cinemon. The framework allows for using PyTable with Twisted by running database queries in a background thread. That's where just about all of the headaches have come so far.

First problem was that a Queue.Queue instance was somehow being created twice. Tracking that down took something like 10 hours, mostly because I'm not accustomed to working with other developers, so my normal strategy of debugging "in-head" wasn't picking up the change Tim had made that karked it up.

Basically, Tim had used a relative import to import the application module, which wound up creating two copies of the module. Duplicate loading of modules is always bad (e.g. isinstance( instance, cls ) will fail), but it's particularly bad when the module being duplicated is one providing a singleton object.

There's a PEP (Python Enhancement Proposal) that will make these problems go away. With that pep, instead of writing:
import scannerapplication

To get the module inside the package (somepackage.scannerapplication) from another module in the package (somepackage.somemodule) you would use:
import .scannerapplication

or, of course, use the (currently valid, and more explicit):
from somepackage import scannerapplication

which is what all of my code uses.

The second problem was because of a rather, ahem, stupid "feature" of PostgreSQL, where it was keeping locks open after a SELECT that never committed that were blocking another thread (after a few hundred queries). Sure, I should have committed/rollbacked the transaction, but SELECT should not have been blocking an insert. Oh well, blah.


  1. x

    x on 09/03/2004 7:35 a.m. #

    At least my kack wasn't checked in... or was it, incidentally, at that point combined with some other trivial update I made? The .pep seems like a good idea. Though i still think there's some conceptual line between application and library/package which isn't being addressed. Ah well, i guess just a shell script (or python script if one wants to keep it in the family) wrapper with executes/imports the app in the proper environment. Sort of what Zope does with instances (bin directory, etc) (verses the library collection which it mostly is, esp. Zope 3). <br />
    <br />

  2. Mike Fletcher

    Mike Fletcher on 09/07/2004 5:09 a.m. #

    I strongly favour the "bulk of the code is in a library that's called by a script" approach. But then at heart I'm just a library developer who's being forced to work on applications by a cruel and heartless world.

Comments are closed.


Pingbacks are closed.