Dipping a toe into pb connection pooling (Brr, these pools get cold in the Canadian autumn...)


Okay, this is a hack written in the middle of the night, so expect there to be problems, but it does seem to basically work. What it is is a collection of three object types.

The first, the ConnectionStats, is a simple utility object that tracks statistics and the semaphor used to throttle the number of simultaneous requests to any given server (PB doesn't seem to like getting hundreds of simultaneous requests, at least in my test system, the whole set just bogs down and calls never return at all... that might be because of the pysnmp stuff the server is doing in response to the requests, of course). Eventually should be tracking response times and factoring those into the choice of a "best" server for a query.

The second, the ClientFactory, is just a subclass of the standard pb.PBClientFactory that overrides methods to 1) store the connection parameters necessary to reconnect, 2) reconnect when the connection is lost, 3) store the perspective avatar from login, and 4) mark events on the ConnectionStats class.

The third, the MetaPerspective, is a perspective-like object that dispatches callRemote queries to any of its factories which currently have an active avatar. It raises a RuntimeError if there are no factories currently connected, and uses the stats in the ConnectionStats to determine which ClientFactory to use. It uses the semaphore in the ConnectionStats to throttle the number of simultaneous connections to any given ClientFactory.

Anyway, the code is in the scripts collection for the curious.

Comments

  1. Glyph

    Glyph on 12/17/2005 5:07 p.m. #


    Cool.<br />
    <br />
    Any thoughts to abstracting the load-balancing logic to other Twisted protocols?<br />
    <br />
    (There was a meme at one point that every protocol should have callRemote methods with domain-appropriate submethods depending on the protocol. Maybe that's the way to achieve this?)

  2. Mike Fletcher

    Mike Fletcher on 12/20/2005 12:26 p.m. #


    Haven't really given it much thought. I wound up creating a local adapter that let an avatar be run as though through a perspective to include the local process in the load balancing. Having every protocol become RPC-ish, though, seems a bit too rigid as an approach.

Comments are closed.

Pingbacks

Pingbacks are closed.