NBD doc

Note: This page needs a table of contents, but the gist is there's stuff here about NBD, ENBD and GNBD

The goal is to get a stable, > 2 terrabyte filesystem using disks on a number of different systems, combined into one large filesystem, using nbd, the "network block device", or enbd, the "extended network block device" kernel modules and userspace utilities.

This page has two major sections:

NBD stuff. This is fairly detailed, but doesn't work that well.
ENBD stuff. This is pretty sketchy, but will hopefully work better eventually.

NBD stuff.

External doc:
- nbd-client.8.txt
- nbd-server.1.txt
What I tried that didn't work (packetpasser is just something that copies packets from one port to another, to get past routing discontinuities):
- nbd server (RHEL kernel 2.4.21-15.0.4.EL, 32 bit x86) -> packetpasser on AIX 5.1 -> nbd client (RHEL kernel 2.4.21-15.0.4.EL on 64 bit opteron) -> md (RHEL kernel 2.4.21-15.0.4.EL on 64 bit opteron).
  - This combination did not support larger than 2 terrabytes.
- nbd server (RHEL kernel 2.4.21-15.0.4.EL, 32 bit x86) -> packetpasser on AIX 5.1 -> nbd client (RHEL kernel 2.4.21-15.0.4.EL on 64 bit opteron) -> LVM (RHEL kernel 2.4.21-15.0.4.EL on 64 bit opteron).
  - This combination did not support larger than 2 terrabytes.
- nbd server (RHEL kernel 2.4.21-15.0.4.EL, 32 bit x86) -> packetpasser on AIX 5.1 -> nbd client (FC2 kernel 2.6.6-1.435, 32 bit x86) -> md (FC2 kernel 2.6.6-1.435, 32 bit x86).
  - This combination did not support larger than 2 terrabytes.
The closest thing I had to something that works:
- nbd server (RHEL kernel 2.4.21-15.0.4.EL, 32 bit x86) -> packetpasser on RHEL kernel 2.4.21-15.0.4.EL, 32 bit x86 -> nbd client (FC2 kernel 2.6.6-1.435, 32 bit x86) -> LVM (FC2 kernel 2.6.6-1.435, 32 bit x86).
- What I did:
  - "go" script on esmfsn02, the machine with the big disks (RAID cards actually) and the nbd server
  - "go" script on esmfgw (You most likely won't need this, but it's presented here to fill in all the variables I was looking at. It just passes packets between esmfsn02 and tesuji here)
  - "go" script on tesuji, the machine on which we do LVM, and run the NBD client
- Notes
  - When I used essentially this combination with mkfs.ext3, it ran for a long time, but eventually crashed. mkfs.ext3 never completed for this large filesystem.
  - When I did an mkfs.jfs, it errored out
  - When I did the same with mkfs.xfs, the mkfs completed very quickly, and gave a useable filesystem. However, upon attempting to copy lots of data into it, it eventually crashed too.
Future directions
- Someone should try enbd! I'm pretty busy, and my manager probably didn't want me sinking the time I have into this already (it was on my own time though, all but 2 hours), but I'm in communication with some folks who sounded pretty stoked on trying this with enbd. I hope to have some information from them later.

ENBD

Peter Breuer's hints:
- Use 2.4.32pre with 2.6.x kernels.
- Userspace portion should be compiled against 2.4 kernel headers.
What I've tried
- Building enbd on Redhat 9... I had to turn off SSL, and I didn't need SMP...
- To get device files, I ran nbd/scripts/MAKEDEV (part of the enbd distribution).
- I tried to add ENBD to the 2.6 .config in a simple-minded way, but got:
  So now I'm just going ahead and doing a kernel build, and hoping that I got an enbd module.
- The build procedure appears to be ignoring enbd. I've now added:
  ...to the drivers/block/Makefile to see if that gets it built.
- The Makefile that comes with 2.4.32pre is incompatible with 2.6's build system. I've restored the original drivers/block/Makefile, and added enbd's .o's to it.
- That didn't work, so now I've created:
  I also created a drivers/block/Kconfig entry. This looks useful. The build system recognized that ENBD was new, and asked y/m/n about it. The actual entry I added to Kconfig was:
- make bzImage didn't build the enbd module, but then, I configured it as a module. :) Trying make modules.
- BTW, parallel make (make -jN) doesn't appear to work at all. :-S
The build system finally tried to make enbd, but then I got tons of errors. Upon describing the problem to Peter, it's sounding like I may have come upon the wrong version of the sources!
Mon Sep 19, 2004: "yum update" on an FC2 system gives a kernel of 2.6.8.1, however the kernel-source package was only for 2.6.5. "yum list" showed that a 2.6.8.1 kernel source package was available. "yum upgrade kernel-source" gave me 2.6.8.1 kernel sources in /usr/src.
Turns out you need to apply a kernel patch for 2.6.x. Go to the directory containing the big, misleading file named "THESE_ARE_NOT_KERNEL_PATCHES", and apply the kernel patch that looks the most suitable.
"make M=drivers/block/enbd" speeds things along - so you don't have to build all the modules.
FC2's 2.6.8* kernel did not work with the enbd-2.6.7.pat - it applied pretty cleanly, but gave numerous compile-time errors when attempting to build the enbd code.
Linus' 2.6.7 kernel accepted the enbd-2.6.7.patch nicely: No .rej's, only one fuzz, and "make M=drivers/block/enbd" completed promptly without error.
Linus' 2.6.7 defaults to compiling ext3 as a module. Word on google is that this is why my mount of / is failing at boot time. :)
The enbd man page for enbd-client seems to have an incorrect example in it, at least if the example init script is correct. In particular, it looks like the device name does not come after the hostname:port. It comes after all the options; it must be the last argument, after all the arguments with a "-". In fact, it appears to do most of the communication over ports I didn't actually specify to use.
Firing up a tethereal reveals that enbd-client is using more ports than just the port you give it. Things finally started to work when I fired up packetpasser's on 1100-1108, 5051, and 5052.
I fired up a process to copy 200+ copies of Fedora Core 2 into a JFS over LVM2 over enbd. It did 1.5, and then JFS detected filesystem corruption and marked itself read only.
Later in the day, the system I was using as an enbd client (with JFS and LVM2) crashed.
The scripts I used:
- "go" script on esmfsn022, the machine with the big disks (RAID cards actually) and the enbd server.
- "go" script on esmfgw (You most likely won't need this, but it's presented here to fill in all the variables I was looking at. It just passes packets between esmfsn01 and tesuji here). This is actually two scripts, bundlewd up in a shar archive.
- "go" script on tesuji, the machine on which we do LVM2, and run the enbd clients

DRB (Domingos Begalli) Notes :)

PSCSG of UCI has found nbd stable with the in-kernel 2.4.x version.
He suggests keeping the slices of identicall size.
They're using md, not LVM{,2}.
They have a 400G, 800G and nearly 1.8T nbd+md slices, formatted with ext2 (not ext3).

From Tony Battersby on the LBD mailing list:

> As someone who is using LBDs in production, I'd like to know what the
> risk exposure is for data loss at this point.  If there are really
> numerous flaws such as the ones you patched, are we at great risk?

That's a good question.  I'm sure it depends on the filesystem you are
using.  I see _lots_ of LBD problems in ext3 and reiserfs.  I haven't
looked too closely at other filesystems, but xfs looks to be mostly
correct (the only xfs LBD bug I found was in a debugging function that
might not even be compiled by default).  I assume that the first patch
that I sent impacts all filesystems, but not being a filesystem expert I
am not sure of the implications for data safety.

My best guess is that you are probably OK with xfs, especially if it has
been working up till now.  If using ext3 or reiserfs, you would have
probably already seen data corruption.

Mail about GNBD (which is part of GFS, but apparently doesn't require GFS) from binky123 at excite dot com on Feb 27 and 28, 2006:

Initial message:

Does anybody have any suggestions on what I can use to coalesce partitions together from 3 PC's?  I didn't have much luck over
the weekend on my tests.

Testing Summary on 3 PCs with 2 2TB real partitions each:
local 2TB                   + LVM(2 devices)     : OK
GNBD 2TB(4 imports 2 PCs)   + LVM(6 devices)     : FAIL(mkfs xfs)
GNBD 2TB(2 imports 2 PCs)   + LVM(2 devices)     : FAIL(dd ok,
                                                   mkfs xfs fails)
GNBD 2TB(2 imports 2 PCs)   + MD(2 devices)      : FAIL(dd ok,
                                                   mkfs xfs fails)
GNBD 2TB(2 imports same PC) + LVM(2 devices)     : FAIL(mkfs xfs)
GNBD 2TB(2 imports same PC) + MD(2 devices)      : FAIL(dd 100000 ct ok,
                                                   mkfs xfs fails,
                                                   mkfs ext3 fails)
GNBD 4TB(1 import linear array) + LVM(1 device)  : FAIL(mkfs xfs)
GNBD 4TB(1 import linear array)                  : FAIL(dd 5000 ct ok,
                                                   mkfs xfs fails)
GNBD 2TB(1 import)    + LVM(1 device)  : OK(dd,mkfs xfs on LVM device)
GNBD 2TB(1 import)                     : OK(dd,mkfs xfs on gnbd device)

Testing Summary on 1 PC with 2 2TB partitions and 1 PC with 4 .37GB partitions:
local 2TB                   + LVM(2 devices)     : OK
GNBD .37GB(4 imports 2 PCs) + LVM(6 devices)     : not tested
GNBD .37GB(2 imports same PC) + LVM(2 devices)   : FAIL(dd ok, mkfs xfs fails,
                                                   mkfs xfs waiting for fsync(),
                                                   mkfs reiserfs ok,
                                                   mount reiserfs fails)
GNBD .37GB(2 imports same PC) + MD(2 devices)    : OK(dd,mkfs xfs on MD device,
                                                   mkfs reiserfs ok,
                                                   mount reiserfs fails)
GNBD .74GB(1 import linear array) + LVM(1 device): FAIL(mkfs xfs fails,
                                        dd ok)
GNBD .74GB(1 import linear array)      : OK(dd,mkfs xfs on gnbd device)
GNBD .37GB(1 import)  + LVM(1 device)  : OK(dd,mkfs xfs on LVM device)
GNBD .37GB(1 import)                   : OK(dd,mkfs xfs on gnbd device)

A followup message:

Hits: 4901
Timestamp: 2024-12-27 07:59:39 PST

Back to Dan's tech tidbits

You can e-mail the author with questions or comments: