As Tim and I were leaving my place to head to the management meeting I realised that the symptoms Bryan had just described to me (and most of the mysteriously disappearing symptoms from yesterday) would all be explained if the Physical Group watchers were somehow managing to crash silently (i.e. without logging any errors).

When I got home, I started looking for said crash, then had one of those "duh!" moments. I'd originally intended that the hierarchy loader (a SimpleParse parser) would simply create the watcher objects, and whatever called it would be responsible for telling them to start watching.

As I was working on the code that calls the loader, however, I decided that would be too complex an operation on that side of the "wall", so I decided it should be done in the parser/interpreter (where it's a trivial call, since it already knows whether the group is new or not). Unfortunately, I forgot to go back and add that trivial call.

The result was that Cinemon has only been scanning the network groupings, not the physical groupings. Every modem still gets scanned (modems must belong to a network grouping), but it could take 6 or more hours to notice a small physical group that went offline, while it should take only three or four minutes.