Very long day today. Some template rework on the VoIP project. Lots of debugging on Cinemon to try to catch a hanging condition. Eventually decided to use the trick from James Knight to track down every blocking operation in the application.

So far only found one block of 5s or more; the point where the CMTS watcher has downloaded thousands of modem descriptions and is creating the thousands of modem watchers in response. That's done via a "parallel" call, but apparently there's too many instances going in parallel, the signal resetting operation doesn't get called for > 5s (likely just too many todo items in the reactor's queues, rather than an actual block/hang there).

I've written the code to log a traceback of the problematic context and continue processing, so hopefully by tomorrow I'll have a decent catalogue of all problematic areas. After that I may drop the acceptable block frequency down to 1s or .5s and see what shows up. That would mean, in essence, that we never pile up more than .5s of cummulative todo items in the queues... Not sure that kind of rigor would be worth the cost in programming, guess we'll see.

Which makes me think, maybe this stuff needs to be a Twisted tool, included as something like "twisted.internet.debugblock", with a single function that just takes a maximum acceptable block period and logs any failures of the application to process a periodic operation within that time. Not sure how portable it would be, but it might be useful for keeping apps "honest" about their asynchronous operation.

In other news; I helped Rosey rearrange her room for a few minutes (mostly just running the cable through the wall to connect her computer's TV card so she can get subtitled French TV and then reconnecting her (moved) machine). She seems very happy, it's apparently far better than she was expecting (which is probably why she procrastinated on getting her room rearranged to allow it for so long). Also got her laptop set up with putty and other tools so I can use it to work tomorrow (I'm going stir crazy in my room, so I'm thinking I'll spend the whole day down at Linux Caffe).