reblock (download here) is a bit like dd, but it appears to work better when transferring data over a network, to a tape drive, in that it makes more of an effort to keep block sizes consistent. Why not just use "tar -b" or "dd obs"? Because tar -b is a producer, and if you write what's produced over a network via a pipe, your blocking may get messed up. As for "dd obs", it doesn't appear to know how to wait for the remainder of a block to arrive. Reblock goes a long way out of its way to ensure that the blocks it outputs are consistently sized, and it doesn't rush to write out data before a complete block has been received.

However, perhaps reblock's more common purpose now, is for getting a running tally of transfer throughput, like in these two examples. The first shows using reblock where filesize is available via fstat, and the second shows reblock usage where filesize info is not available via fstat.

Note: On 2014-08-04 I rewrote reblock for compatibility with CPython 3.x and 2.x, without 2to3 or similar tool. Previously, reblock only worked with CPython 2.x. This change also changed the interface a bit. The interface changes mostly reflect the fact that I haven't used a tape drive in years, and that reblock mostly is used for getting progress out of a pipeline today. It also provides rational defaults for the blocksize and timeout, rather than making you specify them.

Please note that you only need to provide an estimate of the transfer size if both of the following are true:

  1. You want an estimate of how much longer the transfer will take
  2. reblock cannot automatically use os.fstat() to determine how large the file you're transferring is. Usually, os.fstat will be able to get the size of a file, but not a pipe.

Usage is like:

reblock just writes a bunch of data followed by a carriage return, to give a running tally of how things are going. It will often write more than 80 characters on a line though, so if you see reblock scrolling really fast, you probably need to resize your terminal emulator (xterm, rxvt, gnome-terminal, konsole, &c) to be wide enough to hold the entire line.

Known bugs:

  1. If you control-Z reblock, and leave it that way for a while, the throughput measure will get all messed up, as will the completion time estimate. But the timeout doesn't occur.

Future possibilities:

  1. It'd be nice to be able to specify a file to feed to gzip, bzip2 or xz, in order to get an estimate on an archive's compression ratio... Or even an option that accepts a list of files on stdin, and a percentage of the files to compress for the estimate. We could then use that list of files on stdin, to use as input to tar/gtar/whatever.

  • This software is owned by The university of California, Irvine, and is not distributed under any version of the GPL. GPL is a fine series of licenses, but the owners of the software need it to be distributed under these terms.

    You can download it here.

    Related projects:




    2509

    Back to Dan's tech tidbits

    You can e-mail the author with questions or comments: