Decided this morning that I wanted more experience with TurboGears. I decided to re-implement the http log file analysis project I did a while ago as a TurboGears project (previously it generated static HTML files). The project basically performs whole-set analysis on about 150MB of log files to allow me to view hierarchically organised and summarised usage on the various web-sites I run.
This took far longer than I'd anticipated (so long that the core code is not yet finished after 7.5 hours of work), as I ran into a number of problems where finding the elegant SQLObject solution took a while (or didn't happen at all). The data is loaded and can be viewed as a page-tree. It's very slow, but it does work. No viewer/user views yet, and nothing for sorting/searching the results either (which is the kind of stuff I wanted to play with).
In no particular order here's where I ran into issues:
- Couldn't use tg-admin sql create (same issue as before, tries to add constraints pointing to not-yet-defined tables). Write to file then use psql to execute.
- SQL generation isn't producing SQL code to create the indices defined in sqlmeta.indexes. Eventually just wrote an sql file to run after creation.
- Couldn't figure out how one is supposed to do a distinct query on anything other than the id of the table; that is, how to count the number of unique X where X is other than the table.id: eventually just used a raw SQL query to handle this
- Couldn't figure out how to make AND( ) take an SQL string and a field==value constraint together, just coded the SQL string operation in Python as post-processing of the data-set
- Took a while to find the accumulateMany method on sql result objects (to do multiple counting/reporting queries on a table with a single query)
- Speed is currently unusably slow, likely just an application design problem (currently is calculating the counts on every view, wanted to get the core code working before optimising that)
Dealing with the abstraction layer really frustrated me in a number of places, places where I write dozens of queries a day just like them, but figuring out how to do it with the library took 20 or 30 minutes of googling and/or spelunking through the code. That's the nature of learning abstraction layers with a bit of an impedence mismatch I suppose (and ORMs are known for their impedence mismatch). Of course, the point of the whole exercise was that I was going to go beyond SQLObject and out into the rest of the TurboGears stack... guess I still have to work on the foundation a bit first.
Pingbacks are closed.