Ported back to Twisted (but with TurboGears)

For those who are wondering about the Twisted + TurboGears posts.  Yes, I did move ChatTrack back to Twisted, and I did go with an embedded WSGI-hosted TurboGears for the "webish" stuff.

As some of you may recall, I started off in December writing my little tool for real-time feedback using Athena... and wound up running smack into the hiccup that is the Nevow/Web2/Web issues in Twisted.  Since it was a personal project, I decided to play around and see what Tornado (and then CouchDB) would have to offer.

I got the whole project pretty-much finished in Tornado + CouchDB.  It worked perfectly fine, it was as fast as I could ask... but I'd written far too much of the stack myself, and there really didn't seem to be any interest from other people in working on e.g. the CouchDB wrapper, or the generic channel-subscription mechanism, with me.  Tossing "hey, here's something you could use" posts out and finding that no-one is interested is very clarifying... it tells you "hey, this isn't going to be fun to work on in the future".  If no-one else is going to contribute (or even use it), that's a lot of code you have to maintain going forward.

What's more, I ran smack into the query mechanisms of CouchDB... they're fine if you're doing "document" stuff, but they get painful when you want to do heavily-joined queries and the like.  I got to a point where I was looking at re-implementing query languages or having to create an async postgresql client (when what I really wanted was SQLAlchemy).  Those are not things I want to be maintaining, particularly as I don't see it as likely I'll be using them in another project.

Which is where running TurboGears under Twisted came in.  The Twisted code is really good at low-level multi-protocol "networking done right".  TurboGears is good for writing large, complex database (SQLAlchemy) driven web-sites rapidly.  What's more, I'm comfortable enough in TurboGears (I use it for many of my clients) to write solid, tested, code without even having to pause to think about it.

With this architecture, what's more, the Twisted and TurboGears "resources" are largely separable.  The Twisted channel protocol is in a separate project that doesn't have anything to do with the TurboGears site other than that it receives "send" commands from it.  I could readily put that into another process and have the TG site run under Apache and use RPC to do the sending.  That is, the web-site-development environment isn't tied to the low-level networking environment.

The Twisted implementation of the channel server is a trivial set of changes from the Tornado one (just a different sub-class of the same basic code that handles the different request-argument-parsing and callback-registration mechanism).  The real difference is that now I can hook up that implementation to X other protocols, that is, it is now sitting in an environment where people have written code to interact with dozens of other protocols... and I can use that code, instead of having to write my own.  Ironically, I likely won't have time to do that hooking up before PyCon, but whatever.

Tornado is a perfectly fine web framework.  But compared to TurboGears for writing web applications it's early in its game, and I can get patches into TurboGears reasonably easily (I'm mostly working on the docs, but I work with TG enough to hack on the internals).  I likely could have used WSGI to host TurboGears within Tornado... but honestly the callback-based networking in Tornado is nowhere near as elegant as the Twisted Deferred pattern (just my opinion), so if I'm going to use a WSGI abstraction anyway, I'd rather go with Twisted to write (and maintain) the trivial protocol.