Have been doing all of our testing/development work with a partially-specified hierarchy provided by ACI. The change last week, which I thought of as creating an over-specified hierarchy in the demo/testing code had actually come darn close to simulating precisely a normal hierarchy. The extreme slowdown turns out to be a real effect that we are going to have to fix.

The one fully-specified branch runs to 1 group for every 4 modems. The others are running 1 group for every couple of hundred modems. If we spec out the whole tree, we've got 2500 groups instead of 200. That totally throws all those carefully balanced heuristics out the window.

It's not a huge, or insurmountable problem, but it's a serious PITA to discover it this late in the game. Guess I should have been pushing ACI harder to get a full hierarchy done.

Anyway, went a considerable way toward solving the problem already (probably increased the number of groups we can handle by a factor of 2 or 3), but will need to find another 2 or 3x speedup to make it all work right with 2500 groups.