Don’t be a .local yokel

Wikipedia has a nice technical write up that explains why you should never, ever use the .local suffix the way Microsoft has frequently recommended.

But I like this politically incorrect version better:

Microsoft: “Gee, nobody is using the .local piece of the globally shared Internet namespace, so let’s tell all our customers that it’s best practice to use it for our totally super cool version of Kerberized LDAP service called Active Directory!”

Novell: “Oh noes, Microsoft has made an inferior competitor to our flagship technology! It’ll probably destroy our market advantage just like their inferior networking stack did!”

Linux/Unix: “Oh noes, when somebody attaches the new Microsoft technology to an existing mature standards-based network, Kerberos breaks!”

Microsoft: “HA HA HA HA HA HA HA we are totally following the standard, lusers!”

Linux/Unix: “grumble whine we will patch Kerberos even though we don’t agree.”

Microsoft: “whatevs. Did you notice we broke your DNS too? :)”

Apple: “Hey, IETF, we have this cool new zeroconf technology. We want to reserve the .local namespace for it.”

IETF: “OK, sure, you’ve filled out all the forms and attended all the meetings and there’s two independent implementations so you’ve done everything correctly. We have no valid reason to deny this allocation.”

Novell: “Hey, we were using SLP already, what did you just do?”

Apple: “Oh, whoopsie, did we just eat your lunch? HA HA HA HA HA”

Microsoft: “Hey, what just happened?”

Apple: “HA HA HA HA HA HA HA HA HA HA HA RFC6762, lusers!”

Linux/Unix: “grumble mumble whatevs. We can do mDNS.”

Microsoft customers: “OH NOES WE ARE SCREWZ0RRED”

Microsoft: “Meh, you didn’t really want Apple products on your networks anyway.”


Microsoft customers: “How much would it cost to fix this network?”

Microsoft: “What, were you talking to us? Everything’s fine here. Windows 10 forever!”

I can have a page named NUL in linux, though.

Excellent article, but he forgot my favorite, CLOCK$. I used to have a web page with a big, shiny red button linked to <A HREF=”c:\clock$\clock$”> and the message “don’t click the button or your computer will be destroyed and all your files deleted”. It didn’t really do that, but it would instantly crash any Microsoft system prior to Win98SE or thereabouts. People did click on it, which still kind of amazes me.

Terminology: routes and gateways

Originally, back when the ARPAnet merged with SRI, BBN, NSFnet and MERIT to become the Internet, and dinosaurs still roamed the earth, there was no such thing as a “network router”. How can that be? Meh, it’s just semantics. The terminology has evolved.

Internet-connected systems that routed traffic (which was most of them, back in the day) usually ran a program called “gated” (that’s the GATEway Daemon, written at MERIT) that routed IP traffic between networks. A lot of those oldtimey networks were connected by UUCP dial-up links that were only live between 11pm and midnight to save money, so the code was written to support poor quality network links that came and went somewhat randomly.

Any physical network connection that would accept packets bound for some remote network was called a gateway. Gateways were defined by their network addresses. A data structure was created to hold information about which gateways led to which networks – this is called the routing table. The individual entries in that table are created by specifying a set of target IP addresses (using a network address and a mask), a target gateway, and which physical connection to use to reach that target gateway. That terminology is still in use in some commands, such as the “route” command. The individual routing table entries quickly came to be called routes.

At some point somebody at Stanford or MIT came up with the concept of the default gateway. This was a hack, that has become a crucially important networking concept today. No matter what kind of OS they were running, network-connected computers already had routing tables that held networks, masks, and gateways – so a special “fake network” was defined for the purpose of putting a default gateway into the existing tables. It has an address/mask pair that makes no sense at all – – this is intentional, so the fake network entry can’t possibly interfere with any real networks.

The network stacks of all modern systems (post 1979) will look for a route to a target address, and if they don’t find one, they will use the route defined by the routing table entry. It’s a wild swing, the hail mary pass, you just throw it out there and hope for the best.

Since the default route fits the format that is used for all other routes (it just has an impossible ip/netmask pair) it can be carried on any dynamic routing protocol – BGP, EIGRP, OSPF, RIPv2, you name it. This usually causes more problems than it’s worth, so most places do not distribute default routes dynamically. Instead they are configured by DHCP or defined manually, and cannot fluctuate.

Anyway, today, individual people have their own computers, instead of sharing a computer with 500 other people using dumb terminals, so most of our hosts don’t route, so their routing tables are almost empty. They will typically have two entries:

1) the default route, still called the default gateway in many implementations
2) the route to the local net, which is specified by the host’s IP address and mask, and uses the physical ethernet port as the gateway.

A host that has no default route can only talk to machines on networks for which it holds specific routes.

Multicast-capable hosts (like linux and Windows machines) may also have multicast routes in their routing tables, but that is something you usually only see on servers at this point. It will become more common on end user desktops in the future, though; MacOSX and Ubuntu already have multicast capabilities turned on from the factory.

So today any network-capable widget might have static routes, defined by the system administrators, and those static routes might include a default route. It might also have dynamic routes, learned by communicating over the network with other systems, and those dynamic routes might include a default route. You can still call the target of the default route the default gateway if you wish, or you can call it the default route’s next hop, but most networking pros will just say default route or default gateway interchangeably. We’re a little sloppy with the language.

Oddly, over time computers have become less and less capable of dealing with multiple default routes. The pre-v2 linux kernels handled it effortlessly, but modern linux is just as bad in this respect as Windows.

Language evolves, although not always for the better. I personally have found it advantageous to adopt or at least be fluent in the terms and notations used by the youngest generation of technologists. I try to say folder instead of directory, for instance, because directory now means a backend database accessed by LDAP, instead of an on-disk filesystem data structure. I insist on using only international date notation. And I would like to train myself to pronounce router the same as rooter – which is almost certainly going to be the standard pronunciation before I manage to retire – but I haven’t got that programmed into my wetware yet. And I try to always say route instead of gateway whenever possible. The only time I want to use the word gateway is when I’m specifically talking about the target of a route. It’s not that the term is wrong in all other contexts, it’s just that it’s somewhat sloppy and very old-fashioned; it’s like calling your car a flivver instead of a beater.

I hope you’re not reading this with Internet Explorer

If you ever built a website that is only useable with a specific browser you should be ashamed of yourself. Get a job where quality doesn’t matter, OK? Be a banker or something.

The Internets are awash with reports that the US and UK governments are recommending nobody use Microsoft’s Internet Explorer web browser until CVE-2014-1776 is fixed.

And that’s great advice! Use Firefox or Chrome. They are free and work at least as well as Microsoft’s products do.

But various idiots have built systems that only work with IE… unsurprisingly, many of these idiots work for the government, and many of the systems that require IE were built with your tax dollars. A little more surprisingly, many of the hospitals I work with have purchased systems that require IE, although given the increasing reliance of modern medicine on high technology you’d hope that hospitals would know better than to buy any system that isn’t OS- and browser-agnostic. You’d hope in vain, unfortunately.

Stained Glass of Chartres Cathedral

Dennis Aubrey has been photographically documenting the restoration of the Cathedral Notre Dame de Chartres.

The restoration of the stained glass at the cathedral is one of the great accomplishments of this project. Long considered the finest ensemble of medieval glass, we are just now beginning to see the windows in their original condition again. The years have not been kind and the accumulation of inside dust and outside pollution had made the windows almost opaque.

Shots of the newly cleaned windows are at Via Lucis.

SSL/TLS certificates, formats and file types

This stuff is a stack. You can’t skip the middle part and expect to understand any of it.

SSL (Secure Socket Layer) is a type of secure communications channel that you can push anything you want through. It is mostly used by web browsers to talk to web servers but it has infinite other uses. It was invented so that you could use a credit card online, and that is still the #1 use for it.

When a web address starts with “HTTPS” instead of “HTTP” it’s using SSL. You might see a little padlock icon in your browser when you go there.

SSL and TLS (Transport Layer Security) are pretty much the same thing. Everything I say here about SSL also applies to TLS.

PKI really means Paired Key Infrastructure even though officially the “P” stands for “Public”. I use lots of different PKIs, you probably do too. SSH uses one, SSL uses a different one, etc.

X.509 is a PKI standard for using linked pairs of cryptographic keys to ensure two separate things: #1, that you are talking to exactly who you think you are talking to, not some random criminal, and, #2 nobody can listen in on the conversation.

The security and reliability of x.509 depends on the non-existent virtuousness of commercial Certificate Authorities, so it’s not as great as you could hope, but good enough for buying stuff on Amazon or protecting PHI. The NSA and Unit 8200 are totally inside it all the time, but they don’t care about your Amazon wish list.

X.509 specifies only how key pairs are used, and not how they are stored on your disk drive. There are many formats for storage, but we have to stack up some more knowledge before we can talk intelligently about that.

As usual in paired key crypto, one key is chosen to be “public” (doesn’t matter which one) and one key is chosen to be “private”. Data encrypted with one can only be decrypted with the other, and vice versa. Bigger keys are better. Most people aren’t using big enough keys.

X.509 adds the extra wrinkle that the key chosen to be public will be time-stamped and signed by a Certificate Authority. A signed, stamped public key is called a certificate. The time stamp is there so CAs can charge absurdly high fees when certificates expire; it serves no other real purpose and don’t let them tell you different.

Don’t worry about what “signed” means. All that matters is that your web browser can always tell if your certificate was signed by a real commercial CA, or by your employer’s private CA, or is self-signed, or was signed by some random unknown system that might be criminal, or is expired.

When certificates are passed around from one system to another on the wires (like, from Amazon to your web browser, or in a Certificate Signing Request submitted to a CA, or whatever) they use Abstract Syntax Notation One’s Distinguished Encoding Rules (ASN.1 DER). If you really want to understand everything about standardized arbitrary data structure representation go to Wikipedia and start reading at ASN.1, which is sort of the ground rules everything else rests on. But you don’t really need to know the air:fuel mixture in your car is 16:1 to fix a carburetor, and you won’t need to know ASN.1 or DER to build a great web service.

Major point here: When you say “SSL certificate” you are saying “X.509 ASN.1 DER timestamped signed public key”, in the same way that when you say “living woman” you are saying “breathing mammalian human female person”. You don’t add any information by saying DER or X.509, those are already known when you say “SSL certificate”. Which is why I get annoyed whenever I read vendor documentation to see what format they want their certs in, because they always say something useless like “DER” or “X.509”. I already knew that!

Certificates and keys can be stored on disk in an bewildering number of different formats. Tomcat/Java, Apache, IIS/AD, and HP-UX’s webserver all use different formats with mostly stupid names following no particularly obvious pattern.

I’m only going to talk about the storage formats you might actually need to use, and I’m going to ignore lots of details.

PEM (used by lots of stuff) is the easiest way to store certs and keys and the least secure. You have to be super careful when you use PEM; making minor mistakes with file permissions or user privileges can be equivalent to leaving the root password written on a postit stuck to the side of your keyboard. Poorly written software may require you to put both the (public) certificate and the (private) key in a single PEM file which is unnecessarily dangerous. There are no non-printable characters in a PEM cert, it’s all human-readable gibberish that you can cut and paste.

PKCS#12 (Public Key Cryptography Standard number 12, the “Personal Information Exchange Syntax Standard”) is a password-protected format that can hold multiple sets of both (public) certs and (private) keys. The encryption is not marvelously strong so you still have to protect a PKCS#12 file, but it’s strong enough that you sure don’t want to lose the password! It’s a very good format for moving certificates and keys from system to system and used by many Microsoft products.

JKS (Java Keystore) is supposedly PKCS#12… but in my experience, using various versions of Tomcat, you have to build your Java keystore with the Java keytool that came with the version of the Java SDK that was used to build your Java application (such as Tomcat) which is a pain in the butt. It’s password-protected, so you need the passphrase used to build it in order to use it. The Java keytool can’t extract the private key to another file but there are plenty of other tools that can, so it’s not like this adds any real extra security, it’s mostly just annoying.

PKCS#7 (Public Key Cryptography Standard number seven, the “Cryptographic Message Syntax Standard”) is used a lot in the deep deep infrastructure. It cannot hold private keys, only certs, but it can hold a “cert chain” of any length, so for example CertX signed by CertZ, plus CertZ signed by some CA, plus the CA cert all in one file. I occasionally need to put certificates into this format for stuff like complex multi-OS LDAP architectures, and CAs use it, but most people will never need to work with it.

<Curmudgeonly Digression> An unfortunate result of Microsoft’s market dominance is that otherwise well-informed people often think that the last four characters of file names are deeply magical. This is because Apple used to have better filesystems than Microsoft (and arguably they still do). Apple filesystems implemented a resource fork as an extension to file metadata; the resource fork allows users, applications or operating systems to mark what program(s) should be used to process a file, so that you can just click on a file created by Excel and it will open in Excel, or whatever. Microsoft made a really crappy lame fake of this capability by creating a list of three-character codes and assigning each one to a piece of software, so that when you click on a file ending in .xls the operating system fires up Excel. If you think about this really deeply, you’ll realize it’s is a truly horrible idea that Microsoft’s success has conditioned everyone to believe is reasonable – sort of like the way people used to be conditioned to think it was totally reasonable to test for witchcraft by dunking people in water. Nowadays Microsoft takes this stupidity a step further by hiding the last four characters from the user (unless you change the file viewer settings, which you definitely should), mostly likely because they are ashamed of the utter boneheadedness of it.
</End Digression>

So anyway, although file “types” aren’t really types at all, but merely arbitrary strings preceded by dots on the ends of file names, that are used in Microsoft systems to do Dumb Things™, we humans generally use names and labels to encode useful hints to other humans and that’s all very well and good. I always end my perl sources with .pl for example, even though the perl interpreter couldn’t care less. It’s a useful hint to my co-workers about content.

These are the most commonly used file types for x.509:

something.key = PEM format private key for something
something.csr = PEM format “certificate signing request” to submit to a CA
something.crt = PEM format signed certificate

whatever.p7s = PKCS#7 format certificate chain

whatever.p12 = PKCS#12 password-protected keystore
whatever.pfx = either a PKCS#12 keystore or an obsolete Microsoft PFX keystore
tomcat.jks = a Java Keystore, probably for Tomcat, possibly PKCS#12 format

Unfortunately, there are hundreds of exceptions to the common usages – and Netscape Security Services, which is used in Firefox and HP-UX and lots of other places, can use files with names like cert7.db, secmod.db, key3.db, that use formats I haven’t even bothered to explain (use PEM format to import and export certs and keys into NSS and don’t worry about it).

Here are the takeaways:

#1 Crypto isn’t simple. Every vendor believes they are doing it right and nobody else is, although really they are pretty much all doing it partly wrong… in various different ways.

#2 If you start thinking .cer or .der or .spc means something outside a very limited space, you aren’t doing yourself any favors. File names are poor hints only. Never ask someone for a .DER formatted file, it makes you sound like an idiot.

#3 You can use well known vendor-independent language that does have real meaning – Here’s a list of the PKCS number standards and what they are used for. If you use that language, you can communicate effectively (and also sound like you might know what you’re talking about).

#4 Make sure you thoroughly document any non-standard formats that you’re forced to use by vendors so your co-workers aren’t cursing your name whenever you’re on vacation.

#5 Be fanatical about securing your private keys, and don’t lose the passwords to your keystores.

invoking a powershell command with an SSH key

I wanted to use SSH (from any version of any operating system) to execute a specific command in response to a specific cryptographic key.

This is pretty trivial in linux using OpenSSH, but the target server was an MS-Windows 2008r3 box running the Tectia SSH daemon, so I was forced to learn a few new tricks.

Here’s a screen grab of me doing it with my admin account instead of with a dedicated key:

[charlie@linus ~]$ ssh 'powershell -c Write-Output \"Hello World\" <NUL'

Password Authentication:
admin_charlie's password:

Hello World

[charlie@linus ~]$

Simple, eh? The local shell invokes an ssh client, which authenticates to the ssh server, which starts a DOS session, which invokes powershell, which executes a cmdlet, which produces output, which is sent back over the ssh connection.

The single quotes tell the local bash shell to pass everything inside them to the ssh client program as a single parameter without doing normal shell expansion (splitting it on spaces into multiple parameters, interpreting metacharacters, etc). Bash strips away the single quotes silently as it does this, so nothing else ever sees them.

The slashes tell the DOS session not to remove the double quotes before it executes the rest of the line. The slashes are stripped by cmd.exe (the DOS interpreter) so nothing else ever sees them.

The double quotes are there so that the powershell interpreter does not split the literal string “Hello World” into a list of two literal strings, but instead passes them to the Write-Output cmdlet as a single parameter. If Write-Output thought it was receiving a list of parameters (instead of a single one with a space in it) it would print each one on a separate line followed by a carriage return/line feed pair (which ssh would silently convert to a POSIX newline since we ran this from a linux machine). We want to print Hello World on a single line. Powershell strips away the double quotes lalalalala you’re used to that tune by now.

The “powershell -c” is because I am too lazy to type “powershell.exe -Command” with a full path just for testing. If I was writing this for real I’d use the long form because the short form is undocumented.

The redirection of input to come from the NUL device (Microsoft equivalent of POSIX /dev/null, which always returns End-Of-File if you perform any kind of READ operation against it) prevents the powershell interpreter from hanging around waiting for an EOF or “exit” command. If we didn’t do that, the ssh session would stay open, because the DOS session would stay alive, because the powershell session would remain alive, until we typed exit or control-Z.

It’s actually easier if you are submitting a script file rather than just firing off a command like that, since the EOF on the script file will tell the Powershell interpreter to let go of the SSH session and you won’t need the NUL.

And even easier still if you are calling an existing powershell script on the host, because then you don’t have to worry about pushing the script across the SSH connection.

Not every user account can necessarily do this kind of thing against any old server. The server I used for the proof of concept above is a primary domain controller, and it won’t let anybody SSH in except designated administrator accounts.

fix the date format in windows

Windows, like most things made in the USA for mass consumption, likes to use the insanely stupid American date format, for which we are famous the world around.

Of course, the actual American standard for writing dates is ISO 8601 format, as recommended by the National Institute for Standards and Technology (as well as every science and engineering standards body in the world) but American schools still teach the brain-dead m/d/yy format to our children, so understandably the world thinks that’s what we use.

Here’s a link to instructions to fix window’s date representation.

Here’s a link to a possibly useful rant I wrote on this subject