Docker lessons learned 1 year in

A little under a year ago, I started doing devops work for a startup (the Company) with very specialized needs. As it operates in a highly regulated sector, the company's access to their infrastructure is extremely restricted, to prevent accidental or malicious disclosure of protected information. Their in-house web apps and off-the-shelf on-prem software are deployed on a compliant PaaS (I'll call them "the Host", even though they offer vastly more than just hosting), which is very similar to Heroku and uses Docker exclusively for all applications deployed on their private EC2 cloud. I knew about Docker but had never used it, and it's been an interesting few months, so I thought I'd write up some observations in case they help someone.

Topsy Turvy

If you're coming to Docker from a traditional ops shop, it's important to keep in mind that many of your old habits and best practices either don't apply or are flipped upside down in a Docker environment. For example, you're probably going to use config management with Chef or Ansible a lot less, and convert your playbooks into Dockerfiles instead. Ansible/Chef/etc is based on the assumption that infrastructure has some level of permanence: you stand up a box, set it up with the right services and configuration, and it will probably be there and configured when you get around to deploying your app to it. By contrast, in the Docker world, things are much more just-in-time: you stand up and configure your container(s) while deploying your app. And when you update your app, you just toss the old containers and build new ones.

Another practice that may feel unnatural is the foregrounding of (the main) processes. On a traditional web server, you'd typically run nginx, some kind of app server, and your actual app, all in the background. Docker, on the other hand, tends to use a one-service-one-container approach, and because a container dies when its main process does, you have to have something running in the foreground (not daemonized) for your container to stay up. Typically that'll be your main process (e.g. nginx), or you'll daemonize your main process and have an infinite tail -f /some/log as your main process.

As a corollary, while traditional server setups often have a bunch of backgrounded services all logging to files, a typical Dockerized service will only have one log you care about (the one for your main process), and because a container is usually an ephemeral being, its local file system is best treated as disposable. That means not logging to files, but to stdout instead. It's great for watching what's happening now, but not super convenient if you're used to hopping on a box and doing quick greps and counts or walking through past logs when troubleshooting something that happened an hour ago. To do that, you have to deploy a log management system as soon as your app goes live, not after you have enough traffic and servers that server-hopping, grep and wc has become impractical. So get your logstash container ready, because you need it now, not tomorrow.

It's a decidedly different mindset that takes some getting used to.

I was already on board with the "everything is disposable" philosophy of modern high-availability systems, so conceptually it wasn't a huge leap, but if you're coming from a traditional shop with bare-metal (or even VM) deployments, it's definitely a mental switch.

Twelve Factor App Conventions

This one is more specific to the Host than to Docker in general, but it's part of an opinionated movement in modern software dev shops that includes Docker (and Rails, and Heroku), so I'll list it here. The Twelve-Factor App manifesto is a practical methodology for building modern apps delivered over the web. There's a lot of good stuff in there, like the emphasis on explicit declarations or the importance of a dev/stage environment matching production closely. But there's also questionable dogma that I find technically offensive. Specifically, factor 3 holds that configuration must be stored in the environment (as opposed to config files or delivered over some service).

I believe this is wrong. The app is software that runs in user space; the environment is a safe, hands-off container for the app. The environment and the app live at different levels of resolution: all the app stuff is inward-looking, only for and about the app; while the environment is outward-looking, configured with and exposing the right data for its guests (the apps and services running in the environment). Storing app-level (userspace) data in the environment is like trusting the bartender in a public bar with your specific drink preferences, and asking her what you like to drink (yes, this is a bad simile).

In addition, the concerns, scope, skills, budget, toolsets, and personalities of the folks involved in app work tend to be different from those of people doing the environment (ops) stuff. And while I'm ecstatic that devs and ops people appear to finally be merging into a "devops" hybrid, there's a host of practical reasons to divide up the work.

In practical terms, storing configuration in the environment also has significant drawbacks given the tools of the trade: people like me use grep dozens of times every day, and grepping through a machine's environment comprehensively (knowing that env variables may have been set as different Unix users) is error-prone and labor-intensive for no discernible benefit. Especially when your app is down and you're debugging things under pressure. It's also very easy to deploy what's supposed to be a self-contained "thing" (your twelve-factor app) and see it fail miserably, because someone forgot to set the environment variables (which highlights the self-contradictory, leaky nature of that config-in-the-environment precept: if your app depends on something external to it (the environment), it's not self-contained).

Another driver for the config-in-the-environment idea is to make sure developers don't store sensitive information like credentials, passwords, etc. in code that winds up in source control (and thus on every dev's computer, and potentially accidentally left in code you helpfully decided to open-source on GitHub). That makes a ton of sense and I'm all for it. But for practical purposes, this still means every dev who wants to do work on their local machine needs a way to get those secrets onto their computer, and there aren't a lot of really easy-to-use, auditable, secure and practical methods to share secrets. In other words, storing configuration in the environment doesn't solve a (very real) problem: it just moves it somewhere else, without providing a practical solution.

You may find this distinction specious, backwards, antiquated, or whatever. That's fine. The environment is the wrong place to store userspace/app-specific information. Don't do it.

That was a long-winded preamble to what I really wanted to discuss, namely the fact that the Host embraces this philosophy, and in quite a few instances it's made me want to punch the wall. In particular, the Host makes you set environment variables using a command-line client that's kind of like running remote ssh commands, meaning that values you set need to be escaped, and they don't always get escaped or unescaped the way you expect when you query them. So if you set an environment variable value to its current value as queried by the command-line client, you'll double-escape the value (e.g. "lol+wat" gets first set as "lol\+wat"; looking it up returns "lol\+wat" (escaped); resetting it turns it into "lol\\\+wat"; i.e. a set-get-set operation isn't idempotent). All this is hard-to-debug, painfully annoying, and completely unnecessary if the model wasn't so stupid about using the environment for configuration.

Dev == Prod?

One of the twelve-factor tenets is that dev/stage should mirror production closely. This is a very laudable goal, as it minimizes the risk of unexpected bugs due to environment differences (aka "but it worked on my machine"). It's especially laudable as a lot of developers (at least in Silicon Valley) have embraced OSX/macOS as their OS of choice, even though nobody deploys web apps to that operating system in production, which means there's always a non-zero risk of stuff that works on dev failing on production because of some incompatibility somewhere. This also means every dev wastes huge amounts of time getting their consumer laptop to masquerade an industrial server, using ports and casks and bottles and build-from-source and other unholy devices, instead of just, you know, doing the tech work on the same operating system you're deploying on, because that would mean touching Linux and ewww that's gross.

Originally, the Company had wrapped its production apps into Docker container using the Host's standard Dockerfiles and Procfiles, but devs were doing work on their bare-metal Macs, which meant finding, installing and configuring a whole bunch of things like Postgres, Redis, nginx, etc. That's annoying, overwhelming for new employees (since the documentation or Ansible playbooks you have to do that work are always behind and out of date about what actually happens on dev machines), and a pain to keep up to date. Individual dev machines drift apart from each other, "it works on my machine (but nor on yours)" becomes a frequent occurrence, and massive amounts of time (and money) are wasted debugging self-inflicted problems that really don't deserve to be debugged when it's so easy to do it right with a Linux VM and Ansible playbooks, but that would mean touching Linux and ewww that's gross.

So I was asked to wrap the dev environment into Dockerfiles, and ideally we'd use the same Dockerfile as production, so that dev could truly mirror prod and we'd make all those pesky bugs go away. Good plan. Unfortunately, though, I didn't find that to be practical in the Company's situation: the devs use a lot of dev-only tools (unit test harnesses, linters, debuggers, tracers, profilers) that we really do not want to have available in production. In addition, starting the various apps and services is also done differently on dev and prod: debug options are turned on, logging levels are more verbose, etc. So we realized and accepted the fact that we just can't use the same Dockerfile on dev and on prod. Instead, I've been building a custom parent image that includes the intersection of all the services and dependencies used in the Company's various apps, and converting each app's Dockerfile to extend that new base image. This significantly reduces the differences and copy-pasta between Dockerfiles, and will give us faster deployments, as the base image's file system layers are shared and therefore more likely to be cached.

Runtime v. Build Time

Back to Docker-specific bits, this one was a doozy. When building the dev Dockerfiles, I had split the setup between system-level configuration (in the Dockerfile) and app-specific setup (e.g. pip installs, node module installation, etc), which lived in a bootstrap script executed as the Dockerfile's CMD. It worked well, but it felt inelegant (two places to look for information about the container), so I was asked to move the bootstrap stuff into the Dockerfile.

The devs' setup requirements are fairly standard: they have their Mac tools set up just right, so they want to be able to use them to edit code, while the code executes in a VM or a Docker container. This means sharing the source code folder between the Mac host and the Docker containers, using the well-supported VOLUME or -v functionality. Because node modules and pip packages are app-specific, they are listed in various bog-standard requirements.txt and package.json files in the code base (and hence in the Mac's file system). As the code base is in a shared folder mounted inside the Docker container, I figured it'd be easy to just put the pip install stuff in the Dockerfile and point it at the mounted directories.

But that failed, every time. A pip install -e /somepath/ that was meant to install a custom library in editable mode (so it's pip-installed the same way as on prod, but devs can live-edit it) failed every time, missing its setup.py file, which is RIGHT THERE IN THE MOUNTED FOLDER YOU STUPID F**KING POS. A pip install -r /path/requirements.txt also failed, even though 1) it worked fine in the bootstrap script, which is also in the same folder/codebase 2) the volumes were specified and mounted correctly (I checked from inside the container).

That's when I realized the difference between build time and runtime in Docker. The stuff in the Dockerfile is read and executed at build time, so your app has what it needs in the container at runtime. During build time, your container isn't really running--a bunch of temporary containers briefly run so various configuration steps can be executed, and they leave file system layers behind as Docker moves through the Dockerfile. The volumes you declare in your Dockerfile and/or docker-compose.yml file are mounted as you'd expect (you can ssh into your container and see the mount points); but they are only bound to the host's shared folders at runtime. This means that commands in your Dockerfile (which are used at build time) cannot view or access files in your shared Mac folder, because those only become available at runtime.

Of course you could just ADD or COPY the files you need from the Mac folder into the mounted directory, and do your pip install in the Dockerfile that way. It works, but it feels kinda dirty. Instead, what we'll do is identify which pip libraries are used by most services, and bake those into our base image. That'll shave a few seconds off the app deployment time.

Editorializing a bit, while I (finally) understand why things behaved the way they did, and it's completely consistent with the way Docker works, I feel it's a design flaw and should not be allowed by the Docker engine. It violates the least-surprise principle in a major way: it does only part of what you think it will do (create folders and mount points). I'd strongly favor some tooling in Docker itself that detects cases like these and issues a WARNING (or barfs altogether if there was a strict mode).

Leaky Abstractions and Missing Features

Docker aims to be a tidy abstraction of a self-contained black box running on top of some machine (VM or bare-metal). It does a reasonable job using its union file system, but the abstraction is leaky: the underlying machine still peeks through, and can bite you in the butt.

I was asked to Dockerize an on-prem application. It's a Java app which is launched with a fairly elaborate startup script that sets various command-line arguments passed to the JVM, like memory and paths. The startup script is generic and meant to just work on most systems, no matter how much RAM they have or where stuff is stored in the file system. In this case, the startup script sets the JVM to use some percentage of the host's RAM, leaving enough for the operating system to run. It does this sensibly, parsing /proc/meminfo and injecting the calculated RAM into a -Xmx argument.

But when Dockerized, the container simply refused to run: the Host had allocated some amount of RAM to it, and the app's launcher was requesting 16 times more, because the /proc/meminfo file was... the host EC2 instance's! Of course, you could say "duh, that's a layered file system, of course that's what it does" and you'd be right. But the point is that a Docker container is not a fully encapsulated thing; it's common enough to query your environment's available RAM, and a clean, encapsulated container system should always give an answer that's reflective of itself, not breaking through to the underlying hardware.

Curious "Features"

Docker's network management is... peculiar. One of its more esoteric features is the order in which ports get EXPOSEd. I was working on a Dockerfile that was extending a popular public image, and I could not make it visible to the outside world, even though my ports were explicitly EXPOSEd and mapped. My parent image was EXPOSing port 443, and I wanted to expose a higher port (4343). For independent reasons, the Host's system only exposes the first port it finds, even if several are EXPOSEd; and because there's no UNEXPOSE functionality, it seemed I'd have to forget about extending the public base image and roll my own so I could control the port.

But the Host's bottomless knowledge of Docker revealed that Docker exposes ports in lexicographic order. Not numeric. That means 3000 comes before 443. So I could still EXPOSE port a high port (3000) as long as lexicographically it appeared before the base image's port 443, and the Host would pick that one for my app.

I still have a bruise on my forehead from the violent D'OHs I gave that day.

On a slightly higher level than this inside-baseball arcana, though, this "feature" also shows how leaky the Docker abstraction is: a child image is not only highly constrained by what the parent image does (you can't close/unexpose/override ports the parent exposes), it (or its auhor) needs to have intimate knowledge of its parent's low-level details. Philosophically, that's somewhat contrary to the Docker ideal of every piece of software being self-contained. Coming at it from the software world, if I saw a piece of object-oriented code with a class hierarchy where a derived class had to know, be mindful of, or override a lot of the parent class's attributes, that'd be a code smell I'd want to get rid of pretty quickly.

Conclusion: Close, But Not Quite There

There is no question Docker is a very impressive and useful piece of software. Coupled with great, state-of-the-art tooling (such as the container tools available from AWS and other places), and some detailed understanding of Docker internals, it's a compelling method for deploying and scaling software quickly and securely.

But in a resource-constrained environment (a small team, or a team with no dedicated ops resource with significant Docker experience), I doubt I'd deploy Docker on a large scale until some of its issues are resolved. Its innate compatibility with ephemeral resources like web app instances also makes it awkward to use with long-running services like databases (also known as persistence layers, so you know they tend to stick around). So you'll likely end up with a mixed infrastructure (Docker for certain things, traditional servers for others; Dockerfiles here, Ansible there; git push deploys here, yum updates there), or experience the ordeal joy of setting up a database in Docker.

Adding to the above, the Docker ecosystem also has a history of shipping code and tools with significant bugs, stability problems, or non-backward-incompatible changes. Docker for Mac shipped out of beta with show-stopping, CPU-melting bugs. The super common use case of running apps in Docker on dev using code in a shared folder on the host computer was only resolved properly a few months ago; prior to that, inotify events when you modified a file in a shared, mounted folder on the host would not propagate into the container, and so apps that relied on detecting file changes for hot reloads (e.g. webpack in dev mode, or Flask) failed to detect the change and kept serving stale code. Before Docker for Mac came out, the "solution" was to rsync your local folder into its alter ego in the container so the container would "see" the inotify events and trigger hot reloads; an ingenious, effective, but brittle and philosophically bankrupt solution that gave me the fantods.

Docker doesn't make the ops problem go away; it just moves it somewhere else. Someone (you) still has to deal with it. The promise is ambitious, and I feel it'll be closer to delivering on that promise in a year or two. We'll just have to deal with questionable tooling, impenetrable documentation, and doubtful stability for a while longer.

DVR?

Comcast is aggressively pushing their DVR service. Their hold message says "if you want to record the Olympics, be sure to double-check the times, because they may change, so you'll have to reset your DVR."

I was scratching my head the whole time.

Why can't I just tell my computer "record all the javelin events" and have it do the work of recording the right thing at the right time, without my having to make changes?

Worse still, why do I have to record anything in the first place? Why not just let me pull any event I want whenever I want on my computer?

It's not like the technology to do all this isn't available already.

Personalize like it's 1995

This is what I see on Hulu when I'm not logged in:

And this is what I see when I am logged in:

Notice the difference? Me neither. I've never watched Cougar Town or any of the other shows on the home page. I never watch clips--only full-length episodes. And I only ever watch one show on Hulu--Law and Order SVU. Nothing else, ever. Personalizing my home page to my tastes would be completely trivial: just add a link to the last show I watched. They don't need a $1M recommendation algorithm to pull shows I may want to watch--just a list of shows I've watched in the past would be nice.

I wonder what Hulu's engagement metrics look like. I know every web site has priorities to juggle, and maybe recommendations aren't at the top of that list. But engagement and stickiness are crucial to all ad-supported sites, and especially media sites. So why isn't Hulu trying a little harder to get me to engage with their site by giving me ways to get to the shows I want?

Overloaded 404 part deux

I just noticed Backpack by 37Signals also 404s when you access a page you need to be logged in to view.

STOP THE MADNESS!

And of course, if I tell them it's bad UX, they're likely to tell me to go f*** myself.

The art of choosing fonts

Wells Fargo emails use a pretty wild array of fonts.




Reminds me of those vintage mixed-font posters (source).

This is why Apple wins on mobile devices

I recently switched to an Android phone because my non-evil carrier's data plan is not compatible with the iPhone. I like the MyTouch 4G fine--it's fast, and not hobbled by the same arbitrary restrictions Apple phones are. I like being able to install any email program I want and pick the best. But it's not as polished or slick as the iPhone, in terms of integration and general usability, and if the iPhone were available on non-evil carriers I'd seriously consider switching back.

This gauntlet of installation steps for the Amazon app store application is emblematic of the state of Android on mobile devices:

Download the Amazon Appstore app immediately by clicking [link] from your Android device, or follow the click-by-click guide below.

Click-by-Click Guide

You need to do this only once for each device. The clicks below should take less than 30 seconds.

Click 1

Open your device Settings and click "Applications".

Click 2

If unchecked, click "Unknown sources". If "Unknown sources" is already checked, skip to Click 4.

Note: AT&T Wireless does not support the Amazon Appstore for Android. See Help for more details.

Click 3

Click "OK" on the "Attention" dialog. "Unknown sources" will now have a green check.

Click 4

Open your notifications and click the e-mail message from Amazon Appstore.

Click 5

Click the link: [link]. The Amazon Appstore app will download to your device.

Click 6

Open your notifications and click "Amazon_Appstore.apk".

Click 7

Click "Install".

Click 8

Click "Open".

That's it! Sign in with your Amazon.com account and start enjoying thousands of apps for Android. You need to do this only once for each device.

"That's it"? Seriously?

Most of this is the price of freedom (in the free-software sense) and the all-purpose nature of Android, which is an "open-source software stack for mobile devices" (as opposed to a UI and operating system designed for exactly two devices, the iPhone and iPad): generic software is inevitably less integrated and smooth because it's meant to function in a wide variety of heterogeneous environments (think off-the-rack v. bespoke suits). What's sad is that most attempts at improving the native Android UI and integration seem to be driven more by silly branding and business deals than by a genuine concern for user experience.

Many of the most successful computing products have benefited from a (more-or-less) benevolent dictator making the hard choices about what is or isn't going into the final product: Apple, of course, but also Linux, MySQL and Python come to mind. Compare that to the confusing, inconsistent array of products (even open source) churned out by headless democratic nerd posses like the Mozilla Foundation with its multitude of browsers (Firefox, Camino, SeaMonkey), calendars (Sunbird, Lightning), extensions ("add-ons", "extensions", "plug-ins") and skinning engines (personas v. themes, which are "add-ons" themselves); or the Linux UI community, with Gnome and KDE and so many other options.

Maybe it's time for a strong leader to emerge and make the Android people focus on one optimal, uncluttered, integrated experience. Fat chance.

Mozilla: Please Fix This

All the tools I use professionally are open source, with the exception of MS Office and Visio. My browser is Firefox, my email client is Thunderbird, and I manage my calendars with whichever calendaring program wasn't abandoned by Mozilla. By and large Mozilla software it's pretty good. It does the job very well much of the time, and competently almost all the time; but I do wish Mozilla had a little more UI sense and discipline to make better, clearer choices in their product development.

Tabs, Tabs, Everywhere
Thunderbird makes a big deal of its tabs. "Now with tabs, better search, and email archiving" is the main selling point on the download page:



Now, I'm all for innovative UIs, but did the world really need tabs for email? Email has been around for decades, and mostly mainstream for 15 years. Most consumer desktop programs open a new window when you open an email. It's pretty much the standard paradigm for desktop clients; web and mobile clients are probably making this paradigm semi-obsolete, but millions of people use desktop email programs every day, so it's not dead yet. And I don't think tabs are the way to go for email.

Sure, you could make the same argument about browsers: before tabbed browsers started gaining market share, new pages opened in new windows, but tabs have proven themselves as a superior alternative for many people. No argument from me; I used a tabbed wrapper around IE back when Firefox (then Firebird) was more unstable than Jeffrey Dahmer. But a browser has one primary function (viewing web pages), whereas email clients have a long history of letting you manage more than just email, with calendaring, to-dos, notes, and various other things. And that is where the tabbed design breaks down, and why emails shouldn't be opening in tabs. Note I don't expect the Mozilla people to read, care about, agree with, or implement any of these suggestions; after all, it took them years to fix the horrible usability error whereby you couldn't disable the functionality that marks emails as read when displayed in the preview pane. This is the one problem that prevented me from using Thunderbird as my email client. It's now an option, and disabled by default, if memory serves--thanks! But I'm going to gripe^H^H^H^H^Hwrite about it anyway.

It's a pretty common occurrence for me to get an email trying to set up a meeting, with "Does Thursday at 3 work for you?" That's not exactly convenient if you use Thunderbird for calendaring with the Lightning plug-in--now you have an email in a tab, and your calendar in a tab, and so you can't look at them at the same time. You can't alt-tab between them. You have to be keyboard-savvy enough to know that ctrl-tab is how you flip through tabs in a tabbed MDI program. Opening an email in a tab is also not as dramatic visually as opening a new window, so it was pretty frequent for me to open an email multiple times because I hadn't seen the new tab.

Tabs work well for distinct top-level features in Thunderbird: email, calendar, notes, to-dos, etc. Using tabs for sub-features of any of these breaks the hierarchy.

To their credit, Thunderbird lets you disable tabs for email, and that's the first thing I did, but it shouldn't be the default.

Drag Queens
While on the subject of Thunderbird, when dragging attachments onto a message window, why does the main message body not register the drop, while the "To:" section does? What kind of sense does that make?



Conceptually, the attachment is closer to the body of the message; it has nothing to do with the recipients. I understand why the whole address section is a drop target--that's because the attachment summary opens a little square to the right of the address section when you've added an attachment. But that square is nowhere to be seen until you've added said attachment, making it very non-obvious how you're supposed to attach files, especially if you've tried to drop a file onto the message body and failed. Come on, folks, this isn't hard. Get it right.

Tangent #1
Speaking of drag and drop: why do bloggers and web content creators still put up with the pre-2000 model of having to explicitly upload their photos via a dedicated upload form, and (if they're lucky) move it into place in their rich-text editor? Why haven't enough smart techies come up with a standard way to let people just drag a photo from their desktop onto their rich-text editor and have it show up in the right place, without having their text reflowed or mangled beyond repair?

I know the technical answer; what I want to know is why people aren't angry enough about it to demand a fix. Software should be transparent; exposing the nasty technical guts of the software by not providing an integrated experience is a sad cop-out, and the web community should do better.

More T-Bird
While on the topic of Thunderbird, another little thing that wouldn't hurt is to add a little intelligence into the parsing and display of the email. Apple's terrible Mail program does this remarkably well by automatically highlighting words like "yesterday" and "tomorrow" and showing a context menu to create events in their otherwise execrable iCal program; it also highlights contact information like phone numbers detected inside an email.

What I'd like Thunderbird to do is be a little smarter in the message display. When someone receives a message and highlights a section of it, then right-clicks on it, why not show a Search for xxx in your browser (using the default browser and search settings)? Even better: if you can detect that the highlighted string is an address, add a link to a map (a Google map would guarantee even more money for the Mozilla foundation); a phone number? Show a context menu to add it to your address book (together with sender's name and email), or overlay the contact's info if you can find it in the address book.

Yes, I know, I can edit the code myself and probably finagle my changes into the trunk, but why should I have to?

Pick One
One last gripe: I wish the Mozilla folks would just focus a little. Why have plug-ins, add-ons and extensions? Why even use that awful phrase "add-on", when "extension" is pretty clear (extensions extend the basic functionality, get it?) and a lot less techy? Similarly, why have both themes and personas? And why have Firefox, Seamonkey and Camino? Why not call them Firefox, Firefox Suite and Firefox for Mac, if you must distinguish, since they share so much of their rendering technology? What are the differences within those product families? Is there a difference? Most importantly, why should the end-user care?

With double-digit market share, Firefox isn't the bailiwick of basement-dwelling, Gentoo-compiling geeks anymore, so the old hyper-specific tech terminology needs a revamp.

Make Sure Your Customers Can Pay You

If you're running an e-commerce site, chances are you want to extract money out of your visitors. Yet you wouldn't know it by looking at some sites like Best Western's reservation portal, which makes it incredibly difficult if not impossible to complete a paying transaction, as I recently discovered to my chagrin.

Invisible Form

Here's what happened: after selecting a date range, I pressed the big giant "Book it" button next to the room I was interested in (+1 for using big huge buttons for the primary action):

This led me to the following page, completely devoid of a form into which I could enter my credit card information:

After another failed attempt, I fired up Internet Explorer to see if the form was crucially incompatible with Firefox, and I got my answer: the credit card form is served in a popup window, which IE requested permission to open and Firefox quashed silently (I have the "be quiet about popups" setting turned on). Here's what it would have looked like with slightly more verbose Firefox settings:

Note that no one using a reasonably recent browser in its default settings gets to see the popup window unless they explicitly authorize it. In other words, one hundred percent of Best Western's juiciest, readiest-to-buy visitors are not shown the reservation form in the default use case.

I don't have to spell out how completely crazy this is.

To make matters worse, when you do complete the form and click the submit button, nothing happens in Explorer due to a javascript:void(0) issue in the form submit handler¹; in Firefox, the window simply closes with no confirmation the transaction went through.

I had to call the hotel to find out how many of my three form submission attempts had gone through (one had, which was soon confirmed by an email from the site).

Money For Nothing And Tests For Free

What this tells me is that the designer(s) and developer(s) in charge of the Best Western portal never did so much as basic hallway usability testing before they shipped their product. Maybe they don't know about it; maybe the product team is in Connecticut and the developers are in a different country. Either way, a 30-minute investment would have caught what is potentially costing the company tens of thousands of dollars a day.

Another notable omission is that the credit card form itself (for those of you lucky enough to see it) doesn't exactly scream "GIVE ME ALL YOUR MONEY NOW!" I'd bet its conversion hasn't been tested or optimized either.

Testing form conversion is hardly a novel or esoteric branch of rocket science; it's a simple, mature field with well-known best practices and automated testing systems to help you get the most out of your site. But a company that can't be bothered to test the basic usability of their purchase flow can't be expected to look at bounce rates or A/B test their funnel.

Given the bang for the buck you get from real-world testing with a tiny number of users, I submit Best Western could have earned an ROI on the order of 100,000% if they had just watched a half dozen travelers go through the booking process using a fresh, default install of IE or Firefox.

Help Me Buy What You're Selling

I will never know how much money Best Western is losing from this; presumably a lot of travelers wind up calling the hotel (whose phone number is displayed at the top) and maybe even pay the (higher) non-internet rate you get when reserving by phone. So for all I know Best Western is actually coming out ahead (I have my doubts).

What's really sad about this grossly inefficient system is that Best Western didn't even have to earn my business: all they had to do is keep it. Given the way RevPAR is trending, that's a gimme they don't get very often. I always stay at that hotel when traveling down south, because it's convenient, well located, reasonably quiet and clean, and its wireless internet occasionally functions. But I don't know how patient I'm going to be next time around.


---

1. Don't use javascript:void(0) to block a page reload event in an a tag. This will work:

<a href="#" onclick="do_something();return false;">text</a>