• Imagine you have 512M of RAM in your system, and you need to transfer 2048M to another machine via a network - once, and never again.
• When you read the first 512M of data, that data will commonly wipe out everything useful in your buffer cache; everything slows way down, because reading from disk is much slower than reading from the buffer cache.
• Then you read the next 512M, the next, and then the final 512M.
• At this point, your buffer cache is mostly full of that last 512M of data, which you're only going to use once. Nothing useful is in the buffer cache, even though the transfer is done, and it'll stay that way until you use your applications, one at a time, and each one feels like it's taking forever to wake back up.
• This odirect module allows you to prevent this; as you read the 4 blocks of 512M, they don't go into the buffer cache at all. Instead, they go directly to a buffer in your program.

• tests run:
  • I've tested the C library from a C program. This test only reads via odirect
  • I've tested the swig interface code via a python script that uses the swig interface directly. This test too only reads via odirect
  • I've tested the swig interface code via a python wrapper that gives module much more pythonic functionality. This test reads via both odirect and the usual Python file I/O, and compares
  • I've tested the pythonic interface wrapped with bufsock. This test only reads via odirect, but bufsock adds the ability to read sub-block pieces from a buffered full-length block
  • I've tested read support with C++
• Caveats:
  • Although the tests are pretty good, I've used this in only one real application so far - and that's my pyindex program
  • I've only written interfaces and tested with C, C++ and Python so far If you're interested in other languages, please drop me a note - ideal would be diffs for the Makefile and any supporting files, like odirect.whateverExtension wrappers.
  • 0.55 had no write support; only read. The next version, 0.65, had 100% untested write support. That's right - all that testing is only checking reading.
  • No "install" rule in the Makefile yet.
  • The various python modules insert "." on their python path from too many places; this should be done in the test programs themselves, not the module(s) lest we create (and leave) a security issue

You can e-mail the author with questions or comments: