Bug turns out to be a logic problem with parallel operation (It only waits until an op is finished, if it finishes instantly it never waits)

Finally tracked down the hanging Twisted code. It was a big operation that happens every 20 minutes. The first time through every operation has to wait for a network connection or 2, so there's always a deferred. Subsequent times none of the (3000+) operations has to wait, but they are very heavy in terms of processing requirements. With the parallel iteration code, those 3000+ heavy operations were all occurring in a single "time slice" as it were. That froze the reactor for 3-4 minutes.

All of this is because I haven't found a decent way to say "run this task with low priority" within the Twisted system. Maybe I should just spelunk through the library for a while to expand my higher-level toolkit.


Comments are closed.


Pingbacks are closed.