A small enough task to explore PyPy without getting lost

Those who have spoken with me on the topic will recall that I'm rather enthused about PyPy's potential. But every time I've sat down to play with it I've run into a bit of a "small steps" issue... i.e. what small project do I have where I don't need Numpy (which excludes most of my performance-critical projects, i.e. PyOpenGL, OpenGLContext, etc) but where speed is a non-trivial concern. I generally find it easier to explore a new environment with such a project, rather than trying to jump in with a huge, complex project that might take years to prove.

Enter SimpleParse... currently it's written as a wrapper around a (forked, non-stack-based) version of mxTextTools... it has an exhaustive test-suite, so I know if I (or PyPy) have messed up. If I were to rewrite it in pure Python I'd have a good test case for working with PyPy IMO. The original is a very highly optimized C engine (think register declarations all over the place and a minute focus on making everything fast and small), so even getting close to its speed in PyPy would be a significant milestone.

So, I started working on the rewrite this afternoon. So far I have the Object Generator stuff (the code that actually does the parsing) passing all of the tests. I don't yet have a Boyer-Moore Search implementation, nor have I hooked it into the parser generator itself, but it was a pretty good afternoon's hacking, and hopefully when I'm done this will be a reasonable starting point for playing with PyPy (oh, and it should allow SimpleParse to have a pure-Python fallback in case the TextTools library isn't available).

I'm purposely defining the methods in a very generic way, rather than coding lots of special cases, as I'm hoping PyPy will be able to magic-away much of the generic code. That also means I'm winding up with a lot less code than before. Object Generator will likely be 1/2 the length of the original by the time I finish all of the generators (there's only a few left). I figure it will take at least one more hack-day to get to a working pure-Python version, then I can start playing with PyPy in my little sandbox.


  1. Peter Shinners

    Peter Shinners on 02/07/2011 8 p.m. #

    Waiting to hear about your results. I have also been looking for an excuse to Pypy.

  2. Victor

    Victor on 02/07/2011 8:58 p.m. #

    Once you get it working, you can jump into #pypy @ freenode to discuss how to squeeze maximum performance. There's also a wiki page about JIT friendliness: https://bitbucket.org/pypy/pypy/wiki/JitFriendliness

  3. sem media

    sem media on 02/08/2011 12:33 a.m. #


  4. fijal

    fijal on 02/08/2011 12:55 a.m. #

    The Jit Friendliness wiki is relatively new (a couple of days). You might also want to look at jitviewer and most definitely you want pypy trunk and not 1.4.1. Of course #pypy on freenode is the place to go once you want to explore performance more.


Comments are closed.


Pingbacks are closed.