Finally tracked down the hanging Twisted code. It was a big operation that happens every 20 minutes. The first time through every operation has to wait for a network connection or 2, so there's always a deferred. Subsequent times none of the (3000+) operations has to wait, but they are very heavy in terms of processing requirements. With the parallel iteration code, those 3000+ heavy operations were all occurring in a single "time slice" as it were. That froze the reactor for 3-4 minutes.
All of this is because I haven't found a decent way to say "run this task with low priority" within the Twisted system. Maybe I should just spelunk through the library for a while to expand my higher-level toolkit.
Pingbacks are closed.