Red Hat Enterprise Linux 6.4 installs samba4-libs by default

I’m arguing with Red Hat again… the latest downloadable DVD of RHEL6 by default installs part of samba 4, which is supposed to be an unsupported “technology preview” and not a mainline package. In what world does it make sense for your flagship product, for which you sell expensive support contracts, to depend on a chunk of code you decline to support? How is that not bad craziness?

If you try to tear it out with rpm -e you’ll get sssd dependency errors. And ghods, I hate the way RHEL6 and up basically force you to run half-baked name and authentication service caching daemons – my networks worked faster and better without caching, because we actually had a high performance LDAP infrastructure that didn’t need such Microsofty complications. But that’s another rant entirely.

ANYway, if you say OK I will upgrade to Samba 4 to avoid dependency hell, you trigger bug 984727 which Red Hat has set to CLOSED WONTFIX.

Update: Andreas Schneider of Red Hat and the Samba Team has clarified the matter. Since FreeIPA (Red Hat’s Active Directory implementation) and sssd (Red Hat’s new authentication daemon, much like PADL’s PAM and NSS modules only rawer and more oriented toward caching) both require the samba4-libs library in RHEL6, that single package is now officially supported – although version 4 of the Samba Suite is otherwise still a “technology preview”.

dkms patches go live

My fixes for Dell’s Dynamic Kernel Module System made it into their git tree.

Mario’s still reviewing my rewrite of the autoinstall loop, but that’s not actually very important from a functional standpoint. Presumably there will be a fresh release as soon as he’s rejected or accepted it.

I’ve already distributed RPMs at work with the fixed AoE and DKMS packages, so we’re stable at this point.

dkms conquered?

I found the problem. I don’t understand the rationale for the way DKMS is behaving, but here’s what it does…

Let’s say you are running kernel B, and you have kernels A & C also sitting around. And then you install an updated schmegadriver. During the install, the old schmegadriver gets saved off somewhere safe, and the updated schmegadriver gets compiled for kernel B. So far so good! But then, dkms will create links in a “weak-updates” directory from the updated schmegadriver for each of kernels A & C.

Now these weak updates are completely useless, as far as I can see, except to cause pain and suffering. They won’t get used, because your kernels A & C already have the distro schmegadriver installed (the evil one, that causes an endless loop at bootup and occasionally sets local orphanages on fire out of sheer spite) and the weak-updates are links to a schmegadriver that is incompatible with kernels A & C anyway (since it was specifically compiled for kernel B).

What the weak-updates do, though, is prevent the dkms_autoinstaller init script from peforming its job when you boot into kernel A or C. It says, “Oh-ho, there’s a weak update here, no need to build a proper kernel module, let’s burn up all the orphans!” and away you go in a handbasket.

The easiest fix was to hack the dkms_autoinstaller script to delete all “weak updates” automatically. It’s like a anti-feature.

I suppose a proper fix would be to either prevent the weak update links from being created in the first place, or make –autoinstall stop turning itself off when it sees one. I wonder what purpose they were actually intended for? I’m pretty sure the linux guys at Dell don’t hate orphans. Well, reasonably sure.

aoe/dkms/rhel6 redux

Situation as it stands:

The aoe v22i kernel module distributed with RHEL5 is not a major problem. It occasionally has boot issues, where it’s unable to find and mount volumes, but a reboot usually clears this right out, so I think it’s a minor timing bug, probably a race condition; the protocol uses fixed intervals which are not based on primes. We can ignore all that for the moment since it only happens occasionally at boot.

However, the aoe v47 kernel module distributed with RHEL6 causes an infinite loop at boot when used with the Coraid VSX appliances. The only way out of the loop appears to be hard power off. This is obviously a major issue!

Dell’s Dynamic Kernel Module System offers a way to use updated drivers that should prevent the system from blowing up every time a sysadmin types “yum update”. So we should, in theory, be able to use the latest greatest aoe module with dkms and be happy… provided both DKMS and the l.g.a.m. actually work. Hilarity ensues.

The aoe v6-79 kernel module currently available on the coraid and sourceforge sites works reasonably well. It spits out a screen or two of “unsupported ioctl” warnings shortly after boot, but these do not appear to affect function. It has another bug that will never affect disk I/O, but which is a major problem for DKMS. If you do a “modinfo aoe” the output is formatted incorrectly, and DKMS uses that output to determine kernel module version.

With help from others, I made a little patch set that fixes the modinfo problem with aoe6-79 and Ed Cashin at is receptive to including it in the next version release. In the meantime I have built the patched version and integrated it with DKMS.

The DKMS package that comes with Acronis, that we have installed on most of our machines, is very broken. We need to replace it. I don’t know how to backfeed changes to Acronis, but for the moment I’m just going to make replacing Acronis’s package a requirement for installing my aoe6-79 package.

The DKMS package currently being distributed by Dell is also broken, at least on current Red Hat. I am trying to figure out how to patch that as well. I’ve already patched the dkms-autoinstaller init script, but now I need to figure out why –autoinstall does not work. The way DKMS reverses normal unix program output conventions is irritating – DKMS is chatty when it works, and silent when it breaks. This transgression of one of the most basic rules of *nix makes the bearded Dennis Ritchie sad.

Work is being done, progress is being made, and a breakthrough is inevitable, as Mr. Z. would say.

udev under RHEL6 on a Dell m1000e blade server

The udev subsystem dynamically responds to udev events generated by the kernel and creates device nodes on the fly. This solves the old *nix problem of too many files in /dev that don’t have anything to do with the hardware you actually have. If you see a file named /dev/mouse, the system really truly has a mouse attached to it. Or at least that’s the plan. The kernel calls udev whenever you activate or deactivate any device, not just USB and PCMCIA cards, and this lets linux support absolutely any kind of hot-swappable hardware, even processors.

This is not the first dynamic hardware mapping scheme to hit linux – I think Red Hat EL4 used devfs and there was at least one other one before that. And people have tried to do it using the HAL daemon too.

The udev daemon reads “rules” that tell it how you want your devices named. This is so that different distributions that have historically used different names can all use udev, they just have their own rule sets. This also means that hardware invented tomorrow is trivial to add – just make a new rule.

In Red Hat Enterprise Linux v6, udev rules are read from multiple places in order to be maximally confusing and infuriating.

/lib/udev/rules.d is a “library” of standard rules that udev uses by default.

/etc/udev/rules.d is a place where you can put your own system-specific rules that will override the library. This is based on the names of the files, not the rules themselves.

/dev/.udev/rules.d is a secret hidden directory that seems to exist mostly to make you angry. I haven’t found any clear documentation of precedence of rule files found here. You have to experiment on a test system. The rules in here are usually referred to as “temporary rules” which may very well mean “permanent overrides” in normal English.

udev automatically detects changes to rules files, so changes take effect immediately without requiring udev to be restarted. However, the rules are not re-triggered automatically on already existing devices, so you might have learn to use the udevadm command (or just reboot) if you write any new rules.

But wait! There’s more! It was starting to make sense, and we can’t have that!

If you have a Dell system, a program called biosdevname will be invoked by rules in the library file /lib/udev/rules.d/71-biosdevname.rules whenever the system becomes aware that a network interface exists. The Dell white papers on the subject tell me that this program will rename your network devices and disk drives in a way that’s both counter-intuitive and inaccurately documented, but in reality it just renames the ethernet interfaces on a specific set of Dell systems.

Now, honestly, on some models of Dell server this behaviour makes a decent amount of sense; it labels the embedded ethernet ports on the system’s motherboard as “em1” and “em2” which is a good idea since Dell labeled the ports 1 & 2 (instead of the expected 0 and 1 – Real Computer Scientists can’t count past nine on their fingers). Then it will label any additional interfaces by the PCI bus positions, which is sort of uselessly informative but at least consistent.

On a Dell M1000e blade server, though, running biosdevname is worse than doing nothing. Instead of following their reasonably sane idea of making the labels on the chassis match the labels on the interfaces, the emn and pxpy naming scheme is mapped pseudo-randomly across the network ports. Net fabric A is arbitrarily designated as “embedded” (remember, on the M1000e none of the interfaces are embedded in the blades themselves), P3 is considered fabric B, and P1 is assigned to fabric C – even the order doesn’t match. So, on blade five, the interface names end up looking like this:

Blade Chassis Name -> Linux Interface Name
A1-port5 -> em1
A2-port5 -> em2
B1-port5 -> p3p1
B2-port5 -> p3p2
C1-port5 -> p1p1
C2-port5 -> p1p2

This is like purposeful obfuscation of location, and much worse than just nailing down eth0 through eth6 to specific MAC addresses (which RHEL6 would happily do if Dell’s biosdevname software wasn’t sticking its member into the pie).

The best solution I’ve found so far is to create a file named /etc/udev/rules.d/71-biosdevname.rules (thus overriding /lib/udev/rules.d/71-biosdevname.rules) and put a series of udev rules in it that associate specific names with specific MAC addresses. I map the actual Dell chassis labels like so:

Blade Chassis Name -> Linux Interface Name
A1-port5 -> ethA1p5
A2-port5 -> ethA2p5
B1-port5 -> ethB1p5
B2-port5 -> ethB2p5
C1-port5 -> aoeC1p5
C2-port5 -> aoeC2p5

With this setup, you will know which wire the telco guy unplugged as soon the system starts screaming about something gone wrong on ethA1p5 – it’s the Cat 6 ethernet wire in slot A1 port 5, obviously. You can also see which interfaces are intended to be used for our high-speed ATA-over-Ethernet SAN infrastructure.

That’s enough for today…