• copy lots of data - probably more than you really wanted to know :)
  • Linux software RAID (was "RHEL software RAID")
  • run-when-available
  • RHEL kernel builds
  • GFS procedures (derived from Francisco's document)
  • Generating ssh keys for passwordless, passphraseless ssh
  • Setting up unrouted networks for nagios monitoring using check_cluster2
  • Lustre Notes
  • Setting up a nagios screensaver
  • A couple of things to try on a crashy system
  • RHEL netdumps
  • magic-sysrq
  • dcs.nac crash trouble
  • SP switch stuff
  • Rate-limiting nagios notices
  • NBD doc (includes information about LVM2 and md as well)
  • ESMF in a nutshell - stuff for admins
  • SunOS stel during install
  • MPC and Grad-ea notes
  • VNC Information
  • Linksys BEFSR11 (and BEFSR41) information
  • evolution antispam filter
  • SunOS software RAID
  • 3ware raid tuning on rhel
  • setting up a new release of solaris for autoinstalling on autoinst.nacs
  • RHEL headless install
  • Setting up Xprint on Solaris 9
  • mtee: tee'ing data into multiple pipes
  • Turning on Jumbo frames
  • Verifying data transfers from a suspect source
  • Palm Applications
  • Connecting to SSL services for testing
  • Efforts to get a working geometry for CCSM3 on the ESMF
  • Setting up a PXE server, courtesy of Tim Cook
  • Getting a constant hostname without paying for static IP addresses with your ISP.
  • deep-ssh, or automatically chaining ssh commands through routing discontinuities due to unrouted networks.
  • sshgui
  • rpc-health
  • bufsock.py - a buffered socket python module
  • Sun diskless boot doc
  • Cobble documentation
  • ESMF storage nodes doc
  • reblock
  • find-sym
  • NFS troubleshooting
  • dspam notes
  • fix-html is a stunningly simple program for cleaning up messy html (EG, automatically generated HTML) based on BeautifulSoup
  • Debugging with system call tracers
  • notify-when-up - gives you a notice when a system comes back up (say, after a reboot, but you didn't want to set up nagios/omniscient for the system.
  • pdialog - a very simple, high level X/curses GUI for python applications
  • A new contextual script, for adding lines to a file before or after a particular shell regular expression.
  • Setting up mutt to do a decent job of displaying html
  • env-search - give it a good environment, a bad environment, and a script that can distinguish between a good run and a bad run, and env-search will:
    1. Identify whether it is likely that the problem is attributable to an environment difference.
    2. Identify a single environment variable that appears to be related to the problem.
    3. Verifies whether the problem is due to more than one environment variable.
  • QEMU notes - a noncommercial alternative to VMWare, which unlike Xen, does not require a "port" of the guest OS.
  • NX notes - an impressive X11 protocol compressor. I use it with VNC from home, and find it quite a bit faster than both native VNC or VNC+zebedee.
  • sendmail notes
  • New Palm memo "On getting command line arguments not to show up in ps"
  • New dup-label program for copying SunOS disklabels from one disk to another, in a way that should prove a little bit less confusing to future admins of a system.
  • New xfree86 configurations notes palm memo.
  • Added a top-level view of RAID in a RAID notes palm memo.
  • slowdown - a program to make another program run slower; more effective at alleviating problems due to buffer cache irrelevancy than "nice" and similar programs that only/mostly impact CPU usage.
  • Verifying a data transfer. A list of steps to perform manually, at least until this can be coded.
  • web-extract: automatically converting palm memos to published web pages.
  • ethereal notes palm memo.
  • ESMF data recovery - what we've tried, some of the things we could try.
  • E250 notes - describes Sun Enterprise 250 hardware, which pertains particularly to meter, alpha.ddm and e3.oac.
  • Converting binary data for one platform to binary data for another, including a taxonomy of 3 different kinds of conversions, and some conversion program examples.
  • fallback-reboot - finally given a top-level link
  • Unofficial page describing mounting dcslib. Covers the variety of methods that DCS autoinstall uses. The official link (one of?) appears at the bottom of this page.
  • Page about recovering data from a flakey filesystem, including the "try-copying-up-to-n-times" script I wrote to facilitate ESMF data recovery (or more generally, data recovery in an inconsistently-flakey (IE, it flakes different ways different times)) filesystems, after giving up on the "rsync in a for loop" method.
  • New "What program is active on that port?" palm memo.
  • New article on "Recovering e-mail". Links to a related page: recursive search.
  • New, very lightweight "Backup.remote" script for disk to disk backups, being used in production from the ESMF across ucinet to a linux machine running Ubuntu Linux. It's basically just tar | ssh 'cat', but it also keeps 2 daily backups, 2 weeklies, and 2 monthlies.
  • Network performance - Provides definitions, ways of assessing performance, and ways of improving network performance.
  • AIX shared libraries - They aren't your usual Sun-like shared libraries found on nearly all *ix's and (modern) linuxes.
  • chunkup - a program to tack a pipe-like interface onto a program using only an MS-DOS-like (IE: no true pipes) interface. Written to make it "possible" to pipe into an "Scommand" (EG: "Sput"), which is part of SRB. SRB in turn, is a very mtools-like series of programs.
  • convert-database: a program for converting data in one database format to another. Could also be used for rebuilding a partially-broken database back into the same (but internally-consistent) format.
  • A new article about why it's important to check error conditions early (particularly when writing programs that allow unsafe memory references). The language used for the examples given is FORTRAN, but the same general ideas apply to C, sometimes C++, and a host of other languages.
  • AoE notes, AKA "ATA over Ethernet", which makes it a tad like iSCSI or NBD/ENBD.
  • Pidof, which makes it easier to look up the pid of a process (believe it or not ).
  • New pdftopdb program - for converting pdf files into plucker format - for use on constrained devices like PalmOS or Linux PDA's, or most anything else with a plucker viewer.
  • New Linux USB drives page - hopefully applies to USB disk drives as well as USB flash drives
  • Script for finding the highest numbers in a list: highest. Also useful for ASCII order, and finding lowest quantities. It's much faster than the traditional: ...on very large collections of data, as long as "-10" doesn't get too large.
  • evolution-sledgehammer script. It's for unconfusing evolution , EG if evolution always dies as soon as you start it up.
  • nfs-test script, for tuning NFS performance
  • kill-duplicate-processes script - can be used to, for example, kill all but one imap server process for a given user
  • Finding the largest directories in one or more filesystems using largest-dirs and dir-sizes - useful for finding filesystem performance filesystems due to huge directories, especially in conjunction with backups
  • *ix accounting
  • sum-by - for very easy report generation from a whitespace delimited flat file database source file
  • modunits.py - for performing common modular units conversions, EG seconds, minutes, hours, days, weeks, months, years, centuries
  • fast-pseudorandom program for benchmarking and other processes that need a constant but high-entropy data source
  • Sun stuff page has some new items, as well as most of the older things I wrote up that have Sun stuff too (including the promised stuff about Solaris 10)
  • New "backup notes" palm memo. Includes information about Networker, diagnosing backup problems (courtesy of David), other interesting backup programs based on rsync and/or rdiff, and the relationship between backups and compression and encryption.
  • An overview of what forms of data recovery are doable on *ix, from grep'ing the device file, to making an image backup, to using a data recovery service.
  • How to easily find info about a program you don't (yet) know the first thing about.
  • pnetcat (my ancient "packetpasser" script updated and generalized, which is also a sort of fast reimplementation of netcat in python)
  • stddev script, for computing an average and standard deviation from a list of numbers on stdin
  • net-perf-framework (similar to the sort of transfer-size related benchmarking seen in nfs-test, but for a variety of protocols, not including NFS)
  • Why you should use ADT's in your code.
  • IQS - an implicit queuing system (an old program recently ported to AIX, and recently documented much better).
  • Methods of dividing files up into equivalence classes and performance (classify, equivs, equivs2)
  • An overview of how X11 works
  • merging uids programs for merging NIS domains, /etc/passwd's, and similarly-formatted password databases.
  • penalize-users, for making the jobs of some users run slower (in case they're using too many resources).
  • drs-rewrite, for informing users of where a set of URL's have moved to
  • "malloc and sbrk" page, about the relationship between malloc and sbrk, and why free'ing memory sometimes doesn't shrink your process.
  • "redhat-to-AIX-routes" script for automatically converting Redhat static route configuration data to AIX static routes (both for the routing table and the ODM).
  • New max-tcp-window script, for determining your system's maximum tcp window size.
  • Packaged up my old pbmonherc program for displaying .pbm images on a hercules adapter.
  • Packaged up my old hashing routines in C, to be released as drs-hash. Includes a statically typed version that's less memory-efficient, and a weakly typed version that's very memory-efficient.
  • opendap stuff - A software system for efficiently serving up large scientific (for example) datasets.
  • setpgrp-and-exit - some code I wrote to explore the relationship between setpgrp, fork, exit and defunct processes.
  • Finally gave loop a web page. It's for parallelizing jobs, without running all jobs concurrently, by maintaining a pool of n nearly-always-running jobs out of m total.
  • equivs2 now has a -d option for specifying an output delimeter, and treats nonexistent files as empty files. It also now has the ability to read filenames from stdin separated by a null byte, in addition to its previous ability to do the same with newline termination.
  • drsbackup should handle -s correctly now (series of tar archives, one per filesystem)
  • evolution-sledgehammer, when yanking on ~/.evolution, will back up not to ~/.evolution.old, but rather to a directory name that includes the date.
  • Debugging with system call tracers now has an additional section about how to contend with getting a meaningful syscall trace out of apache.
  • Optiputer discussion, especially with regard to TCP Window Size Scaling (RFC 1323).
  • Rebooting a hung Sun - something we have to talk clients through from time to time...
  • WDS notes (linksys wifi/DSL router range extending)
  • Running one command on many hosts - a survey of software for doing it
  • Backup.remote page now also has my Backup.local and Backup.rsync scripts described in it (for disk to disk backup on the same system, and rsync incrementals locally or to another system). rsync backups or similar could really make our backup costs a lot lower, both in terms of storage requirements and network network bandwidth requirements. I did Backup.remote and Backup.local for the ESMF, and Backup.rsync for for my backups at home. The ESMF probably should be switched to Backup.rsync though. rsync backups apparently are excellent both for individual file recovery (*ix, MacOS X and windows) as well as disaster recovery (*ix, maybe MacOS X, probably not windows).
  • Efforts to write a device driver for an Olympus VN480PC voice recorder - stuck on a mystery (Proprietary? Containerless?) codec. Protocol mostly deciphered though.
  • A new page presenting a couple of disk use summary scripts
  • A program to facilitate indexing ext3 filesystems without having to umount and fsck: redo-dirs, along with info about the fsck method.
  • pyindex can now easily index over 79,000 e-mail messages
  • New section in fallback-reboot page about testing for a failure to fork+exec using telnet, finger, smtp, portmap/rpcbind and fallback-reboot.
  • SunOS software RAID page now has a section about determining which disk you're booted from, when you have an SVM-based root mirror.
  • context-split finally documented a teensy bit. It's for splitting a large file into many small files on regex boundaries.
  • network performance URL now has a list of nice performance-related links from "prg".
  • Backup.remote page now has Backup.SRB, in addition to the previous Backup.remote, Backup.local and Backup.rsync.
  • New test-network script for empirically determing whether 10/100, half/dull work best.
  • A writeup of autoinst's recent network trouble
  • Seems the maxtime link may have disappeared, so here it is.
  • Finally documented my merge-libs script, that's intended to take some of the sting out of AIX library unusualness, when you need a library with both 32 bit and 64 bit objects in it.
  • cachedb.py - for caching python database entries - write-through, write-back, and with disk-to-core and core-to-disk transparent format conversion
  • base255.py - a python module for converting numbers to strings and vice versa, but the strings have the interesting property that they can be null terminated and concatentated, without increasing ambiguity.
  • count is listed again - link disappeared?
  • mysql.perms
  • head0
  • Finally documented stamp a bit
  • NBD doc page now has info about NBD and ENBD (as before), but also GNBD (without GFS) now.
  • odirect - for reading data without going through the buffer cache, hence deterring cache poisoning when reading data that you know you won't be reading again anytime soon. Includes C and Python support, but the python support is done via SWIG, so it should be easy to do other languages.
  • equivs2 can now skip the byte-for-byte comparisons that were required sometimes previously, instead trusting hash comparisons. This optional behavior results in a huge performance gain, at the expense of a small difference in accuracy.
  • refile-mail-lists
  • upward
  • readline0.py - read data in python until an arbitrary character is found
  • Still more OPeNDAP stuff (new URL)
  • Ted Tso's December, 2005 Usenix/LISA presentation about data recovery in ext3
  • ubcd2usb-linux: converting a UBCD iso to something you can boot from a USB thumb drive, using Linux.
  • Using mplayer without the xvideo X server extension


    Hits: 4894
    Timestamp: 2023-02-05 06:08:54 PST

    Back to Dan's tech tidbits

    You can e-mail the author with questions or comments: