Don’t be a .local yokel

Wikipedia has a nice technical write up that explains why you should never, ever use the .local suffix the way Microsoft has frequently recommended.

But I like this politically incorrect version better:

Microsoft: “Gee, nobody is using the .local piece of the globally shared Internet namespace, so let’s tell all our customers that it’s best practice to use it for our totally super cool version of Kerberized LDAP service called Active Directory!”

Novell: “Oh noes, Microsoft has made an inferior competitor to our flagship technology! It’ll probably destroy our market advantage just like their inferior networking stack did!”

Linux/Unix: “Oh noes, when somebody attaches the new Microsoft technology to an existing mature standards-based network, Kerberos breaks!”

Microsoft: “HA HA HA HA HA HA HA we are totally following the standard, lusers!”

Linux/Unix: “grumble whine we will patch Kerberos even though we don’t agree.”

Microsoft: “whatevs. Did you notice we broke your DNS too? :)”

Apple: “Hey, IETF, we have this cool new zeroconf technology. We want to reserve the .local namespace for it.”

IETF: “OK, sure, you’ve filled out all the forms and attended all the meetings and there’s two independent implementations so you’ve done everything correctly. We have no valid reason to deny this allocation.”

Novell: “Hey, we were using SLP already, what did you just do?”

Apple: “Oh, whoopsie, did we just eat your lunch? HA HA HA HA HA”

Microsoft: “Hey, what just happened?”

Apple: “HA HA HA HA HA HA HA HA HA HA HA RFC6762, lusers!”

Linux/Unix: “grumble mumble whatevs. We can do mDNS.”

Microsoft customers: “OH NOES WE ARE SCREWZ0RRED”

Microsoft: “Meh, you didn’t really want Apple products on your networks anyway.”


Microsoft customers: “How much would it cost to fix this network?”

Microsoft: “What, were you talking to us? Everything’s fine here. Windows 10 forever!”

Comodo up to more tricks

People occasionally ask me who they should buy security certificates from. I absolutely will not recommend anyone in particular – even the most honest and honorable Certificate Authorities are inherently swindlers, because the trade itself is pretty much a legalized extortion scheme – but I am willing to say who I don’t recommend – Comodo is the worst CA, hands down. Witness their latest hijinks:

When you install Comodo Internet Security, by default a new browser called Chromodo is installed and set as the default browser. Additionally, all shortcuts are replaced with Chromodo links and all settings, cookies, etc are imported from Chrome. They also hijack DNS settings, among other shady practices.
[Link to Chromodo download elided]
Chromodo is described as “highest levels of speed, security and privacy”, but actually disables all web security. Let me repeat that, they ***disable the same origin policy***…. ?!?..

This certainly isn’t the first time Comodo’s been caught doing things they shouldn’t, but somehow they still control around a third of the world’s certificate issuance. People need to stop giving business to known bad actors, even when it’s unclear whether the actions stem from malice or incompetence.

Query all non-subscribed RHEL7 repos at once

The old Red Hat Network was simple and easy to use. The RHN website presented a list of systems in your web browser, with counts of outstanding patches and outdated packages. You could click on a specific system name and do various things like subscribe to specific repositories (channels) etc.

The current Red Hat Network is a glittering javascript tour-de-force that multiplies the number of clicks and the amount of specialized knowledge you will need to manage your systems. You can pay extra for add-on capabilities such as the ability to select groups of systems and apply a set of operations to all of them, which is almost certainly necessary if you have a large number of systems. It’s a sad travesty of the much-maligned system it replaced.

If you’re completely entangled in the new RHN with your Red Hat Enterprise Linux 7 systems (by which I mean that you haven’t managed to exit the Red Hat ecosystem for a more cost-effective infrastructure yet) you might want to do something like figure out which of the various poorly named repos (such as -extras, -optional, and -supplementary) contains some particular package you want.

Command line to the rescue! Ignore all RHN’s useless beauty and use ugly, reliable Gnu awk. This, for example, finds the repo where the git-daemon package has been hidden away.

subscription-manager repos --list | gawk '/^Repo ID/{print "yum --showduplicates list available --disablerepo=\"*\" --enablerepo=" $3}' | bash | grep -i git-daemon

After several minutes (there’s a lot of network traffic involved) you’ll find that versions of git-daemon are in five different repos.

git19-git-daemon.x86_64 1.9.4-2.el7 rhel-server-rhscl-7-eus-rpms
git19-git-daemon.x86_64 1.9.4-3.el7 rhel-server-rhscl-7-eus-rpms
git19-git-daemon.x86_64 1.9.4-3.el7.1 rhel-server-rhscl-7-eus-rpms
git-daemon.x86_64 rhel-7-server-optional-fastrack-rpms
git-daemon.x86_64 rhel-7-server-optional-rpms
git-daemon.x86_64 rhel-7-server-optional-rpms
git-daemon.x86_64 rhel-7-server-optional-rpms
git19-git-daemon.x86_64 1.9.4-2.el7 rhel-server-rhscl-7-rpms
git19-git-daemon.x86_64 1.9.4-3.el7 rhel-server-rhscl-7-rpms
git19-git-daemon.x86_64 1.9.4-3.el7.1 rhel-server-rhscl-7-rpms
git-daemon.x86_64 rhel-7-server-optional-beta-rpms

So, you query the Red Hat Package Manager, rpm, to find out what version of git you have.

rpm -q git

Since matches the latest version of git-daemon available from the rhel-7-server-optional-rpms repository, that’s the one you need to add in order to load git-daemon.

subscription-manager repos --enable rhel-6-server-optional-rpms
yum install git-daemon

This process is much easier than using the Red Hat Network web gui, and requires less specialized knowledge. Which is pretty sad, considering how arcane these incantations are.

rsyslog & systemd

The ancient Berkeley syslog is a functionally impoverished logging mechanism, but the protocol is well understood and widely supported. You can use a modern version of the daemon (Ranier’s rsyslog or syslog-ng for example) and work around the shortcomings of the protocol itself.

I’ve been working with a Red Hat Enterprise Linux version 7 spin-up, and since systemd is basically a Red Hat product it should come as no surprise that RHEL7 thoroughly embeds systemd.

Here’s a section of the documentation that describes how the error logging works:

Some versions of systemd journal have problems with database corruption, which leads to the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resources get exhausted. This can be somewhat mitigated by using proper rate-limiters, but even then there are spikes of old data which are endlessly repeated. By default, ratelimiting is activated and permits to process 20,000 messages within 10 minutes, what should be well enough for most use cases.

Firefox annoyance #5: redirect caching

Firefox Annoyances:

1) Sync
2) pocket
3) hello
4) everything else, other than the plug-in API itself, that isn’t a paper-thin shell around gecko
5) 301 redirect caching

To clear the 301 redirect cache for a single page, go to the “View” menu and light up the “History” sidebar (yeah, of course you forgot about that, nobody uses it), find the site you’re working on, right-click and select “forget about this site”.

annoying git

I’ve been installing git on some corporate servers with the idea of converting existing CVS and ad-hoc code management systems into something reasonably fast and modern.

It’s been somewhat tedious and painful, but supposedly once I’m done the installation will be stable and maintainable. For an enterprise SCM that’s a lot more important than ease of installation, at least in theory. (I ran OpenLDAP for a decade or more, so I can appreciate the value of putting all the pain up front.)

Today’s annoyance is that the gitolite documentation and web site refer to a “hosting user” but the toolset and other web sites describing gitolite installation talk about an “admin user”. After wasting several hours with Google trying to find out exactly what the difference was, I created a new user account for the admin user and executed the commands – at which point it became immediately obvious that THOSE ARE THE SAME DAMN THING.

Curse you, gitolite. I WANTED US TO BE FRIENDS.

ISP hacked, blog savaged

Our ISP,, was hacked and an amateurish attempt was made to plant various forms of malware on this site. Fortunately for my non-existent readers, the hackers weren’t particularly competent. Unfortunately for me, the same might be said of my ISP…

User registrations are disabled, for the nonce, which again will be a trial for my non-existent audience.

Foswiki dependency hell

I really wanted to run Foswiki, because it seems like most of the TWiki devs ended up there, and because my employers want to run an enterprise wiki with fine-grained access and revision control driven from a corporate directory. Since Foswiki is written in perl, and Graham Barr’s excellent perl-LDAP modules can easily handle arbitrarily complex directory integration, I figured I’d just rip out all the code that checked users and groups against the Foswiki DB and replace it with appropriate LDAP calls, then send my mods upstream to the Foswiki devs. They seem like a good crowd, they’d probably appreciate a non-caching LDAP module.

But we’re heavily federally regulated, and we can’t run unmaintainable code. The number of unpackaged dependencies I’d need to run Foswiki on Red Hat Enterprise Linux is just unsupportable. I can’t find an audited, securely maintained package of File::Copy::Recursive, for example, anywhere. And there’s quite a few more (although some are available from EPEL).

I’d love to find a wiki engine that used real LDAP, instead of just caching copies of data retrieved by LDAP in a local database.

I can have a page named NUL in linux, though.

Excellent article, but he forgot my favorite, CLOCK$. I used to have a web page with a big, shiny red button linked to <A HREF=”c:\clock$\clock$”> and the message “don’t click the button or your computer will be destroyed and all your files deleted”. It didn’t really do that, but it would instantly crash any Microsoft system prior to Win98SE or thereabouts. People did click on it, which still kind of amazes me.

14-yr-old imprisoned for changing wallpaper

Zero tolerance means no harmless prank shall go unpunished.

This kid’s prank demonstrated that children in his school have the ability to easily see the questions to be used for the state’s standardized tests, because teachers there apparently have full administrator access to the school’s network, and their passwords are their upcased last names.

But nobody cares about that… I think they’re basically freaked out because it’s Florida, and the kid’s prank was to put a picture of men kissing on his teacher’s desktop. Education be damned, we must punish the gaiety! The teenager has since been released into his parents’ custody.

Terminology: routes and gateways

Originally, back when the ARPAnet merged with SRI, BBN, NSFnet and MERIT to become the Internet, and dinosaurs still roamed the earth, there was no such thing as a “network router”. How can that be? Meh, it’s just semantics. The terminology has evolved.

Internet-connected systems that routed traffic (which was most of them, back in the day) usually ran a program called “gated” (that’s the GATEway Daemon, written at MERIT) that routed IP traffic between networks. A lot of those oldtimey networks were connected by UUCP dial-up links that were only live between 11pm and midnight to save money, so the code was written to support poor quality network links that came and went somewhat randomly.

Any physical network connection that would accept packets bound for some remote network was called a gateway. Gateways were defined by their network addresses. A data structure was created to hold information about which gateways led to which networks – this is called the routing table. The individual entries in that table are created by specifying a set of target IP addresses (using a network address and a mask), a target gateway, and which physical connection to use to reach that target gateway. That terminology is still in use in some commands, such as the “route” command. The individual routing table entries quickly came to be called routes.

At some point somebody at Stanford or MIT came up with the concept of the default gateway. This was a hack, that has become a crucially important networking concept today. No matter what kind of OS they were running, network-connected computers already had routing tables that held networks, masks, and gateways – so a special “fake network” was defined for the purpose of putting a default gateway into the existing tables. It has an address/mask pair that makes no sense at all – – this is intentional, so the fake network entry can’t possibly interfere with any real networks.

The network stacks of all modern systems (post 1979) will look for a route to a target address, and if they don’t find one, they will use the route defined by the routing table entry. It’s a wild swing, the hail mary pass, you just throw it out there and hope for the best.

Since the default route fits the format that is used for all other routes (it just has an impossible ip/netmask pair) it can be carried on any dynamic routing protocol – BGP, EIGRP, OSPF, RIPv2, you name it. This usually causes more problems than it’s worth, so most places do not distribute default routes dynamically. Instead they are configured by DHCP or defined manually, and cannot fluctuate.

Anyway, today, individual people have their own computers, instead of sharing a computer with 500 other people using dumb terminals, so most of our hosts don’t route, so their routing tables are almost empty. They will typically have two entries:

1) the default route, still called the default gateway in many implementations
2) the route to the local net, which is specified by the host’s IP address and mask, and uses the physical ethernet port as the gateway.

A host that has no default route can only talk to machines on networks for which it holds specific routes.

Multicast-capable hosts (like linux and Windows machines) may also have multicast routes in their routing tables, but that is something you usually only see on servers at this point. It will become more common on end user desktops in the future, though; MacOSX and Ubuntu already have multicast capabilities turned on from the factory.

So today any network-capable widget might have static routes, defined by the system administrators, and those static routes might include a default route. It might also have dynamic routes, learned by communicating over the network with other systems, and those dynamic routes might include a default route. You can still call the target of the default route the default gateway if you wish, or you can call it the default route’s next hop, but most networking pros will just say default route or default gateway interchangeably. We’re a little sloppy with the language.

Oddly, over time computers have become less and less capable of dealing with multiple default routes. The pre-v2 linux kernels handled it effortlessly, but modern linux is just as bad in this respect as Windows.

Language evolves, although not always for the better. I personally have found it advantageous to adopt or at least be fluent in the terms and notations used by the youngest generation of technologists. I try to say folder instead of directory, for instance, because directory now means a backend database accessed by LDAP, instead of an on-disk filesystem data structure. I insist on using only international date notation. And I would like to train myself to pronounce router the same as rooter – which is almost certainly going to be the standard pronunciation before I manage to retire – but I haven’t got that programmed into my wetware yet. And I try to always say route instead of gateway whenever possible. The only time I want to use the word gateway is when I’m specifically talking about the target of a route. It’s not that the term is wrong in all other contexts, it’s just that it’s somewhat sloppy and very old-fashioned; it’s like calling your car a flivver instead of a beater.

Office not so 365

Microsoft’s Azure Cloud service failed at almost exactly midnight last night, taking down hundreds of websites who may have thought that hardware redundancy could magically protect them from sysadmin oopses, as well as users of Xbox live and Microsoft’s flagship service Office 365.

Viva Zorggroep, a Dutch healthcare organisation with 4,000 employees, said it had also been affected as a consequence of adopting Microsoft’s online apps.

“At this time, our supporting departments such as finance, HR, education, IT et cetera are working with Office 365,” said Dave Thijssen, an IT manager at the company.

“This morning these servers were unresponsive, which means users were not able to log in to Office 365.

“As a result they had no access to email, calendars, or – most importantly – their documents and Office Online applications.

“We also had trouble reporting the outage to our users as most of digital communication – email, Lync, intranet/Sharepoint – was out.

The outage persisted for over five hours for some customers and apparently there are still latency issues at this time. This is of course a violation of the Service Level Agreement… so you can keep a nickel or two of your monthly rent, I bet.

Internet soft spots

Want to build a ginormous botnet without doing a lot of work? Compromise one of the Internet’s soft spots.

If you take over, you can use the site to inject malware in 1.3 million visitors. Chump change! How about, or, or whatever? Face it, you’re not going to get more than 15 million suckers. It’s just too much effort for a lazy man; you’d still be doing a lot of hard work to recruit a paltry few million zombies.

So, you take over, or Now you’re cooking with gas! It’s become common practice for websites to use remotely sourced scripts – so there are thousands of sites that will blindly push out whatever is in the file jquery.js at, and all that site’s visitors will run it just as blindly. So if you take over a popular script or advertisement source, you can leverage that into billions of individual attacks, quite easily.

And that’s my Halloween horror story for this year.

DIY Ground-based Ion Cannon

Hobbit’s netcat can be used to vomit forth network traffic as fast as your machine can generate it. We don’t need no steenkin’ LOIC!

Anyway, I needed to test a WAN pipe to see if Comcast was delivering the bandwidth we’re paying for – we’re supposed to have a 200 Mbps link to Boston.

[root@monster ~]# yes | nc -4 -u -v -v -n 9

The yes command just screams “yes!” incessantly, like a teenage boy’s dream girlfriend. We pipe the output to netcat, and force it to use UDP and IPv4 to send all the yes traffic to a host in Boston. UDP port 9 is the “discard” service, of course, so the machine at the other end just throws the traffic away. We already constantly monitor all the routing nodes in the path so we can see and graph what happens to the packets in real time.

Turns out the host can generate 80Mbps, sustainable indefinitely. That goes into the 200Mbps Comcast pipe… and only 4Mbps comes out the other end! Thanks, netcat! Time to call Comcast!

Don’t do this if you aren’t ready to deal with the repercussions of completely smashing your network. Saturating interfaces, routers and pipes will severely impact normal business routines, and should be saved as a last resort.

0+12=13 in DBMspeak

In the field of Computer Science, there are many sub-disciplines, and there are varying shades of technical opinion. One of the shadiest of these is that of the database managers. An outspoken group on many subjects, always willing to force academic ideals of data integrity on hapless junior programmers, critical of any engine that caters primarily to real-world use cases, and always willing to compromise any such principles in any situation that affects them personally. Phil Ochs fans are laughing, everyone else is confused.

To illustrate: Everybody knows C programmers can’t count to ten on their fingers, because they start at zero. But if you ask a C programmer to provide ten items, he will – they’ll just be numbered from zero to nine, that’s all. A Visual Basic or FORTRAN programmer will give you the same absolute number of items, although they’ll be numbered from one to ten. Two boxes twice is always four boxes, in the world of workers getting things done, no matter the language nor what the labels on the boxes say.

Perhaps only in the field of database management would a list of “Ted Codd’s 12 Rules” include 13 items numbered zero through twelve. I suspect that in any other field, this would be considered a typographical error and quickly corrected in the proofreading stage.

Fix uart boot errors on M1000e blade chassis

Somebody else figured it out in 2009, and I’m late to the party.

Basically, if you are running Red Hat Enterprise Linux (or one of it’s clone siblings) on a Dell M600 blade, you’ll need to modify the default BIOS settings in a non-intuitive way or you’ll get an error on every boot-up.

IRQ handler type mismatch for IRQ 12

Call Trace:
[] setup_irq+0x1b7/0x1cf
[] serial8250_interrupt+0x0/0xfe
[] request_irq+0xb0/0xd6
[] serial8250_startup+0x43d/0x5dc
[] uart_startup+0x76/0x16c
[] uart_open+0x19e/0x427
[] tty_open+0x1e8/0x3b0
[] chrdev_open+0x14d/0x183
[] open_namei+0x2be/0x6ba
[] chrdev_open+0x0/0x183
[] __dentry_open+0xd9/0x1dc
[] do_filp_open+0x2a/0x38
[] do_sys_open+0x44/0xbe
[] tracesys+0xd5/0xdf

I knew this was a serial port issue from the second and fifth lines of the trace, but I couldn’t figure out why it was involving IRQ 12, which is normally used for SCSI cards or PS/2 mice.