Generic Binary-Data Pack/Unpack Module?

Plumbing Life's Depths

Calendar

Back September '10
Mo Tu We Th Fr Sa Su
Thu September 2 2010
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30      

Quicksearch

Archives

September 2010
August 2010
July 2010
Recent...
Older...

Categories



All categories

Syndicate This Blog

XML RSS 0.91 feed
XML RSS 1.0 feed
XML RSS 2.0 feed
ATOM/XML ATOM 0.3 feed
ATOM/XML ATOM 1.0 feed
XML RSS 2.0 Comments

Blog Administration

Open login screen

Comments

T. Middleton about Replicate live into a throw-away staging/dev db
2010-08-31 12:04
That template db idea is prett y neat. I have to say it's get ting pretty annoying downloadi ng database backups and [...]


Brian Harring about All those little spots where you use strings...
2010-08-23 20:57
Yeah, in a similar boat with t he code I write... upshot, I'v e got some pretty good compat code in snakeoil, just w [...]


Mike C. Fletcher about All those little spots where you use strings...
2010-08-22 08:40
Yeah, that's what I started do ing (the function to convert t o bytes), as I support down to 2.4. Thing is this is [...]


Brian Harring about All those little spots where you use strings...
2010-08-22 06:28
Your blog aparently likes does n't handle the less than char (<) conversion all that wel l.. everything following [...]


Brian Harring about All those little spots where you use strings...
2010-08-22 06:24
The annoying thing about suppo rting both py2k and py3k is th at you wind up having to get r ather explicit about you [...]


Dave Stott about What is the measure of a man? (Pondering while the unit-tests run...)
2010-08-16 00:48
The measure of a man, I have a lways thought, was - that he l oved. And was loved.


Mike C. Fletcher about bzr-svn for those stuck with svn servers...
2010-08-15 21:28
Hmm, hadn't realized that was what dpush was for. I guess I 'll have to make the trunk a b ranch rather than a chec [...]


Jelmer Vernooij about bzr-svn for those stuck with svn servers...
2010-08-15 20:43
For what it's worth, newer ver sions of bzr-svn will not remo ve existing revisions from you r mainline unless you ex [...]


Jelmer Vernooij about bzr-svn for those stuck with svn servers...
2010-08-15 20:40
bzr-svn supports not inserting any unusual revision properti es, it just means that pushing your bzr revisions into [...]


Mike Fletcher about bzr-svn for those stuck with svn servers...
2010-08-15 18:21
Hmm, maybe I missed something there. Do you mean merge supp ort as in being able to pull f rom N branches into your [...]


Brian Harring about bzr-svn for those stuck with svn servers...
2010-08-15 17:45
Lack of merge support kills th e main benefit of bzr-svn over just using svk I'd argue. Personally, I'm *extreme [...]


Dan Villiom Podlaski Christiansen about bzr-svn for those stuck with svn servers...
2010-08-15 11:46
You might want to check out hg subversion :-) The UI should b e even simpler, and we don't i nsert any unusual revi [...]


Jim about GIT is a PITA (Stupid, stupid, stupid source code control...)
2010-07-25 14:02
> and would have no Trac integ ration The trac-bzr plugin[ 1] seems to provide good integ ration between bzr and t [...]


Dr Clue about Curiouser and curiouser with Asterisk manager interface (pyst doesn't seem to realise that there are event-generating queries)
2010-07-13 21:47
I've always been fascinated wi th the Asterisk AMI interface. So much so that I married tha t fascination with the [...]


Mike C. Fletcher about Mocker's hidden gem...
2010-07-03 21:32
Yes, only references in dicti onaries are replaced, so hold ing references in lists, tuple s, etceteras keeps them alive.


Tuesday, March 30. 2010

Generic Binary-Data Pack/Unpack Module?

Having just written yet another module to pack/unpack data into a particular binary format, I'm wondering if someone has a more effective method they'd like to share?  I'm thinking some module that can handle RLE/TLV type stuff, complex "encodings" such as utf-8 continued charcters, struct-like pack/unpack for static elements, efficient loading of arrays, and bit-field pack/unpack.  Some way to customize handling of a given element within the format (so you could write constraints or what have you), some way to provide defaults, both static and calculated, some way to specify derivative fields, checksums and the like, some way to...

Whenever I look at generalizing this pattern, it always seems to turn into "you should have the equivalent of a compiler/parser-generator" produce this code, but if someone has a good, generic solution, I'd love to hear about it.

in Snaking at 01:29 | Comments (7)
Trackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Display comments as (Linear | Threaded)
This kind of sounds like Hachoir: http://bitbucket.org/haypo/hachoir/wiki/Home
#1 Dirkjan Ochtman (Homepage) on 2010-03-30 02:35 (Reply)
I'm not sure if it'll meet your requirements, but I worked with Construct a while ago and loved it. The way it outlines the data's structure so explicitly made the code 100% clearer.
#2 Charlie on 2010-03-30 03:53 (Reply)
As 'construct' is not a nice google keyword, I'd like to point interested visitors to the construct homepage: http://construct.wikispaces.com/
#2.1 Felix Schwarz on 2010-04-10 05:48 (Reply)
hdf5 library maybe?
#3 rdm on 2010-03-30 06:37 (Reply)
Construct looks like exactly the kind of project I was envisioning. Same basic interface as my current code (in a number of projects).

Hachoir actually looks sexier for an "out of box" collection of primitives (given it has 70 defined languages already).

Not sure what performance is like, but it looks like I've got some playing to do when I get the time :-) .
#4 Mike C. Fletcher (Homepage) on 2010-03-30 10:56 (Reply)
Sounds like you want protocol buffers, until you mention that you don't like code-generation. In that case, Avro might fit your bill [http://hadoop.apache.org/avro/], however, you pay a performance penalty when you use Avro's dynamic packing/unpacking. Avro does have code generation as an optimization feature, though.
#5 bs on 2010-03-30 12:25 (Reply)
I'm fine with code generation, as I mentioned, it looks like a compiler/parser would be the appropriate path :-) . Avro and Protocol Buffers are, AFAIK, particular binary formats, I'm looking for a mechanism that lets you interact with already-defined binary formats (efficiently, with a reasonable API).
#5.1 Mike Fletcher (Homepage) on 2010-03-30 12:33 (Reply)
Add Comment
Note: If you're here from a DoFollow list, please go elsewhere. This blog is actively monitored and spam removed, and it is now NoFollow rather than DoFollow.
Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA