Red Hat just doesn’t get AOE

This post edited to correct my mistakes

ATA-over-Ethernet, the high performance low-cost SAN protocol developed by Sam Hopkins over at Coraid, never gets any love from Red Hat Enterprise Linux. The AoE kernel modules included with any given release are always egregiously out of date, and don’t even seem to be contemporary with the kernels they’ve been distributed with. If you want to use up-to-date drivers, you have to either wrap up coraid’s kernel module sources with DKMS and support a complete build chain, or create kernel-version independent modules, or else you’ll get system breakage from routine yum updates. Either way you have to muck about building RPMs so you won’t break package dependency and inventory tracking, and you have to set things up to get rid of the old Red Hat module each time you get a kernel update.

And despite the inclusion of an (elderly) aoe module in Red Hat, they don’t provide any officially blessed aoe-tools package. Coraid maintains a set of simple aoe management utilities, a remote console app for their aoe devices, and a throughput testing app, all free open source software. The basic aoe-tools package has been a part of Fedora for some time now, and lately coraid has been bundling the sources for the tools with the driver sources.

In my shop, we’ve been running more than 12 terabytes of AoE storage infrastructure on Red Hat EL 3, 4, and 5 for eight years or more now. Currently more than 150 TB in production. I had some pretty major problems with AoE on RHEL6, and openly blamed Red Hat’s out of date drivers for them, but those problems have been resolved. We had a bad VLAN trunk, basically, so it was really our fault (my sincere apologies to those I mistakenly accused).

You know, it’s always seemed to me that ATA-over-Ethernet should be a natural win for Red Hat. The simplicity of the protocol, when compared to Red Hat’s approved iSCSI, reminds me strongly of the value proposition that linux represented ten years ago when compared to proprietary unixes. You can build an AoE SAN that’s faster and more reliable than an iSCSI SAN for considerably less money; why would anyone purposely choose the less cost-effective solution? That’s why so many IT shops left HP-UX, SunOS, and MVS for Red Hat linux – because linux delivered the capabilities we needed for less cash.

Strangely, though, Red Hat treats AoE like an unloved and ugly stepchild, at best neglecting it entirely. Whenever a Fedora release is repackaged for publication as a Red Hat Enterprise Linux release, the aoe-tools package is removed; bleeding-edge Fedora probably provides a more stable AoE SAN platform than the flagship product. It’s deeply weird behavior and counterproductive for Red Hat.

I wrote an initscript for AoE under Red Hat that works under RHEL 3 through 6. It’s in the software section.

There’s an interesting discussion that references this blog post here.

3 thoughts on “Red Hat just doesn’t get AOE

  1. Hi Charlie,

    Just to clarify, my kmod-aoe package, which you link to above, allows you to just ‘yum install’ and use, you don’t have to “muck about building RPMs” or worry about updates with new kernel versions – hopefully it should all Just Work ™. My repo also has current versions of aoetools and cec available:

    But I do agree that it would be nicer if aoe was properly supported by RedHat.


    • Hmmm… I guess I wasn’t very clear, at that! It’s very kind of you to share your work – you mucked around with RPM so others won’t have to. I feel inspired to go get my aoe initscript online now.

  2. Man, what do you want? CoRaid’s builds are all crazy, no conformance to anybody else’s, and they aren’t pushing required mods upstream to Linus. Of course Red Hat isn’t going to touch that shit.