This post edited to correct my mistakes
ATA-over-Ethernet, the high performance low-cost SAN protocol developed by Sam Hopkins over at Coraid, never gets any love from Red Hat Enterprise Linux. The AoE kernel modules included with any given release are always egregiously out of date, and don’t even seem to be contemporary with the kernels they’ve been distributed with. If you want to use up-to-date drivers, you have to either wrap up coraid’s kernel module sources with DKMS and support a complete build chain, or create kernel-version independent modules, or else you’ll get system breakage from routine yum updates. Either way you have to muck about building RPMs so you won’t break package dependency and inventory tracking, and you have to set things up to get rid of the old Red Hat module each time you get a kernel update.
And despite the inclusion of an (elderly) aoe module in Red Hat, they don’t provide any officially blessed aoe-tools package. Coraid maintains a set of simple aoe management utilities, a remote console app for their aoe devices, and a throughput testing app, all free open source software. The basic aoe-tools package has been a part of Fedora for some time now, and lately coraid has been bundling the sources for the tools with the driver sources.
In my shop, we’ve been running more than 12 terabytes of AoE storage infrastructure on Red Hat EL 3, 4, and 5 for eight years or more now. Currently more than 150 TB in production. I had some pretty major problems with AoE on RHEL6, and openly blamed Red Hat’s out of date drivers for them, but those problems have been resolved. We had a bad VLAN trunk, basically, so it was really our fault (my sincere apologies to those I mistakenly accused).
You know, it’s always seemed to me that ATA-over-Ethernet should be a natural win for Red Hat. The simplicity of the protocol, when compared to Red Hat’s approved iSCSI, reminds me strongly of the value proposition that linux represented ten years ago when compared to proprietary unixes. You can build an AoE SAN that’s faster and more reliable than an iSCSI SAN for considerably less money; why would anyone purposely choose the less cost-effective solution? That’s why so many IT shops left HP-UX, SunOS, and MVS for Red Hat linux – because linux delivered the capabilities we needed for less cash.
Strangely, though, Red Hat treats AoE like an unloved and ugly stepchild, at best neglecting it entirely. Whenever a Fedora release is repackaged for publication as a Red Hat Enterprise Linux release, the aoe-tools package is removed; bleeding-edge Fedora probably provides a more stable AoE SAN platform than the flagship product. It’s deeply weird behavior and counterproductive for Red Hat.
I wrote an initscript for AoE under Red Hat that works under RHEL 3 through 6. It’s in the software section.
There’s an interesting discussion that references this blog post here.