Shared storage mechanism (Temporary GBs of ganged storage)
Written by
on
in
Young Coders.
As I was walking home I was considering the question of large-media "special events", such as wanting to watch a multi-GB movie off the network at school. If the class wants to record an event (school play) at high enough resolution to do a good job of editing where would the children store the gigabytes of video?
Would be nice to have a storage mechanism user mode fs that allowed the children to gang together 128MB (or whatever) each of their RAM into a buffer file system for storing big files temporarily. You'd create a "buffer activity" and then invite in your friends to share the load with you.
You'd want to be able to determine how much of your memory you can dedicate to the task somewhere. We'd want to allow for some robustness in the face of lost nodes as an option as well. Would want to be able to link it to a given downloading mechanism (activity), so likely want it to appear as a low-level filesystem to which an activity can be granted access (Journal environment with a mounted filesystem as a child).
Anyway, just a thought.
Comments
Comments are closed.
Pingbacks
Pingbacks are closed.
Mike C. Fletcher, the VRPlumber, is a philosopher of design turned computer programmer based in Toronto, Canada. He works primarily in the Python programming language, and maintains a large number of Open Source software packages. You can reach him at: mcfletch@vrplumber.com.
Jim McCoy on 06/06/2007 11:08 p.m. #
Sounds like you want an ad-hoc distributed hash table of some sort, but you will need to deal with write-access issues that most DHTs gloss over; I am guessing that just going with an "owner is the person who setup the initial group" and letting that node handle all writes is the easiest option. There are a variety of error correction codes that can deal with lost nodes, but given the CPU constraints you are dealing with it might be easier to go with simple replication over erasure coding. You mentioned editing this shared chunk of storage as a goal, are you thinking about editing over the shared space or just using the shared space as a buffer which will dump to a more suitable large chunk of storage somewhere for the actual editing process?
Mike Fletcher on 06/07/2007 12:02 a.m. #
For the approach I'm considering I'd like to be able to avoid the need to load onto "something else" in order to edit or view the file. That is, assume that the kids will have to use their laptops to do the work and that they may not have sufficient space on the school server (the likely only "large" system nearby) to do the work.<br />
<br />
The CPU constraints likely aren't that huge a concern here (the CPU's are relatively fast in the B3 and beyond), nor is robustness in the face of bad actors (you and your friends decided to do something together), but robustness in the face of a machine or two going down would be useful.<br />
<br />
It's quite reasonable to imagine wanting to dump 3GB of raw (but likely downsized) video for recording a play, then splice together the elements you want in a non-linear editor to produce a movie that you then encode (using an efficient codec) down to a few hundred MBs (which you can store on one or two nodes).<br />
<br />
Consider also the case of using the system to view a downloaded video. Everyone in the group wants to be able to play the video (at some point), but there isn't enough space on the (overworked) school server to store 3GB. If we (the students) want to do something like this we should be able to pool our individual resources without needing to have extra resources allocated at the school.
anonymous on 06/07/2007 1:07 a.m. #
bittorrent
Mike Fletcher on 06/07/2007 11:19 a.m. #
BitTorrent is a transfer mechanism. This is an end-point-storage system. If you've "shared" the resource with your friends, they can just read the storage from each other's shared space (using whatever low-level protocol we use).<br />
<br />
While BitTorrent was partly the inspiration (the need) the present BitTorrent clients would not work well for these children (who will only have 1GB of uncompressed storage on disk (you can't compress compressed movies much)).